Guardrail 1 is now the process that runs before the AI is allowed to type: an Issue with a defined problem and proposed fix, an ADR reviewed by at least two people for anything non-obvious, then a design and implementation plan that says how it will be tested. The eight agent roles move here — they are what "then the AI takes over" means. Guardrail 5 replaces "I stayed the reviewer" with the test regime: quick test on every change, deep test via test-module.sh --deep, test plan authored in the ADR, and a full regression across every module after implementation. Test figures reconciled to one scope across the deck (both repos): 195 suites, 34,490 lines, 27% of source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
732 lines
29 KiB
Markdown
732 lines
29 KiB
Markdown
---
|
||
marp: true
|
||
theme: gaia
|
||
class: invert
|
||
paginate: true
|
||
title: TAPPaaS one year in — The Good, the Bad, and the Ugly
|
||
description: Sommerhack 2026, Taler Teltet — 27 August, 15:30
|
||
---
|
||
|
||
<style>
|
||
section { font-size: 26px; }
|
||
h1 { font-size: 1.5em; }
|
||
h2 { font-size: 1.15em; }
|
||
pre { font-size: 0.7em; line-height: 1.35; }
|
||
table { font-size: 0.78em; }
|
||
th, td { padding: 0.2em 0.55em; }
|
||
/* Unfinished content, loud on purpose: an unfinished deck should never be
|
||
presented by accident. Written in the markdown as `TODO: ...`. */
|
||
code.todo { color: #ff8a80; font-weight: 600; }
|
||
/* Mermaid renders at its natural size, which is far too small on a projector. */
|
||
div.mermaid { display: flex; justify-content: center; width: 100%; }
|
||
div.mermaid svg { width: 100% !important; height: auto !important; max-height: 500px; }
|
||
section.diagram h2 { margin-bottom: 0.1em; }
|
||
/* Hand-drawn diagrams (inline SVG) get the same full-width treatment. */
|
||
section.diagram > svg { display: block; width: 100%; height: auto; max-height: 520px; margin: 0 auto; }
|
||
/* Section dividers: gaia's `lead` does the centring; this only sets the scale. */
|
||
section.chapter h1 { font-size: 2.6em; margin-bottom: 0.1em; }
|
||
/* Punch lines: the slides that are one sentence, said loudly. */
|
||
section.punch h1 { font-size: 2.1em; line-height: 1.15; }
|
||
section.chapter p { font-size: 1.15em; opacity: 0.75; }
|
||
</style>
|
||
|
||
<!-- _paginate: false -->
|
||
|
||
# TAPPaaS — one year in
|
||
|
||
## The Good, the Bad, and the Ugly
|
||
|
||
**Lars Rossen** · Sommerhack 2026 · Taler Teltet
|
||
27 August, 15:30
|
||
|
||
<!--
|
||
Sixty minutes. Roughly: 8 opening, 18 Good, 14 Bad, 14 Ugly, 6 close.
|
||
Leave the last 5 for questions from the tent — this crowd will have them.
|
||
|
||
Tone: this is a confession, not a product pitch. The proof is the mess.
|
||
-->
|
||
|
||
---
|
||
|
||
## Last year, in this tent, I pitched a dream
|
||
|
||
> Everyone needs a cloud in their basement.
|
||
> A **T**rusted, **A**utomated, fully **P**rivate **P**latform **a**s **a** **S**ervice.
|
||
|
||
On commodity hardware:
|
||
|
||
- your data, your hardware
|
||
- no hidden fees
|
||
- no reliance on closed source
|
||
- **no reliance on the Internet**
|
||
|
||
A year later I am back with the messy proof that it is real —
|
||
and to make the case that you should build one too.
|
||
|
||
<!--
|
||
Read the four bullets slowly. The fourth is the one that sounds mad and turns
|
||
out to be the whole point — come back to it in the AI section.
|
||
-->
|
||
|
||
---
|
||
|
||
## The receipts, up front
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| First commit | **10 May 2025** — "Initial commit" |
|
||
| Commits when I submitted this abstract | **1,091** |
|
||
| Commits standing here today | **1,507** |
|
||
| Contributors | 3 (I am ~77% of it) |
|
||
| ADRs written | **24** |
|
||
| **Modules** — 8 foundation · 12 apps · 21 community | **41** |
|
||
| Lines of code (bash · TypeScript · Python · Nix) | **126,699** |
|
||
| Of which **test** code | **34,490** — 27% |
|
||
| Lines of documentation | **~39,500** |
|
||
|
||
Everything in this talk is in two public repositories. You can check my homework.
|
||
|
||
<!--
|
||
The 1091 → 1507 jump is the joke that lands: I wrote the abstract in the
|
||
spring and the project kept going. Point at it.
|
||
|
||
The 41 is the other one worth pausing on: 8 foundation + 12 apps in the
|
||
TAPPaaS repo, plus 21 modules other people wrote in the Community repo.
|
||
That last number is the one that says this stopped being my hobby.
|
||
|
||
Figures recomputed 2026-08-25 from the tree (a module = a directory with
|
||
its own <name>.json contract), using the methodology in src/STATISTICS.md.
|
||
Community repo: codeberg.org/TAPPaaS/Community.
|
||
-->
|
||
|
||
---
|
||
|
||
## What I am actually going to do to you
|
||
|
||
1. **The Good** — it runs. Here is what "it runs" means, and one surprise.
|
||
2. **The Bad** — what it really costs, and why that only works if it is shared.
|
||
3. **The Ugly** — the confession. I gave an AI root on every node.
|
||
4. **Your turn** — why you should build one, and how to start small.
|
||
|
||
I am not selling anything. There is nothing to buy.
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# The Good
|
||
|
||
It runs.
|
||
|
||
<!--
|
||
Deliberately short divider slide. Say "it runs" out loud and pause.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert diagram -->
|
||
|
||
## The iron: three boxes, a switch, an AP and a modem
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
n1["<b>tappaas1</b> — ASUS · EPYC 4464P<br/>12c/24t · 64 GB ECC<br/>tanka1 2×4 TB NVMe <b>mirror</b><br/>tankb1 12 TB HDD"]
|
||
n2["<b>tappaas2</b> — Minisforum MS-S1 MAX<br/>Ryzen AI MAX+ 395 · 16c/32t<br/><b>128 GB unified</b> · Radeon 8060S<br/>tanka1 2 TB NVMe"]
|
||
n3["<b>tappaas3</b> — TianBei WTR PRO<br/>Ryzen 7 5825U · 8c/16t · 32 GB<br/>tanka1 1 TB NVMe<br/>tankc1 5 TB HDD — PBS"]
|
||
sw["<b>UniFi USW Pro XG 10 PoE</b><br/>10.0.0.201<br/>trunk: mgmt untagged +<br/>VLAN 200 310 410 420 430 510 610"]
|
||
ap["UniFi Nano HD<br/>access point · port 9"]
|
||
modem["ISP modem<br/>port 5 · VLAN 100 = wan"]
|
||
n1 -- "10G · p12" --> sw
|
||
n2 -- "10G · p2" --> sw
|
||
n3 -- "2.5G · p3" --> sw
|
||
sw --- ap
|
||
sw --- modem
|
||
```
|
||
|
||
<!--
|
||
Every number here was read off the running cluster, not a wiki page.
|
||
Worth saying out loud: only tanka1 on tappaas1 is a mirror. tappaas2 and
|
||
tappaas3 run single NVMe — deliberate, they are rebuildable from backup.
|
||
The 10G links are direct-attach copper; tappaas1 and tappaas3 have a second
|
||
NIC on VLAN 100 straight to the modem, which is how OPNsense gets a WAN.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert diagram -->
|
||
|
||
## The software: foundation underneath, stacks on top
|
||
|
||
<svg viewBox="0 0 1200 470" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="TAPPaaS running VMs, layered"><defs><marker id="ar" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="#fdf6e3"/></marker></defs><rect x="0" y="0" width="700" height="212" rx="10" fill="rgba(255,255,255,0.05)" stroke="#fdf6e3" stroke-width="1.5" stroke-dasharray="6 4" opacity="0.9"/><text x="16" y="26" fill="#fdf6e3" font-size="18" font-weight="700" opacity="0.95">AI stack — tappaas2, on the GPU</text><rect x="14" y="46" width="214" height="148" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="121.0" y="76" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">vllm-amd · 312</text><text x="121.0" y="101" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">LXC · Radeon 8060S</text><text x="121.0" y="123" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">128 GB unified</text><rect x="240" y="46" width="214" height="148" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="347.0" y="76" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">litellm · 310</text><text x="347.0" y="101" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">gateway · keys</text><text x="347.0" y="123" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">4 GB</text><rect x="466" y="46" width="214" height="148" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="573.0" y="76" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">openwebui · 311</text><text x="573.0" y="101" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">the chat window</text><text x="573.0" y="123" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">4 GB</text><rect x="730" y="0" width="470" height="212" rx="10" fill="rgba(255,255,255,0.05)" stroke="#fdf6e3" stroke-width="1.5" stroke-dasharray="6 4" opacity="0.9"/><text x="746" y="26" fill="#fdf6e3" font-size="18" font-weight="700" opacity="0.95">Productivity — tappaas1</text><rect x="744" y="46" width="214" height="148" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="851.0" y="76" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">nextcloud · 340</text><text x="851.0" y="101" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">files · calendar</text><text x="851.0" y="123" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">8 GB · 80 GB</text><rect x="970" y="46" width="214" height="148" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="1077.0" y="76" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">euro-office · 343</text><text x="1077.0" y="101" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">documents</text><text x="1077.0" y="123" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">8 GB</text><line x1="350" y1="216" x2="350" y2="300" stroke="#fdf6e3" stroke-width="2.5" marker-end="url(#ar)"/><line x1="965" y1="216" x2="965" y2="300" stroke="#fdf6e3" stroke-width="2.5" marker-end="url(#ar)"/><rect x="0" y="306" width="1200" height="164" rx="10" fill="rgba(255,255,255,0.05)" stroke="#fdf6e3" stroke-width="1.5" stroke-dasharray="6 4" opacity="0.9"/><text x="16" y="332" fill="#fdf6e3" font-size="18" font-weight="700" opacity="0.95">Foundation — every app above rests on these</text><rect x="15" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="106.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">network · 110</text><text x="106.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">OPNsense</text><text x="106.5" y="421" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">8 GB</text><rect x="210" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="301.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">tappaas-cicd · 130</text><text x="301.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">mothership</text><text x="301.5" y="421" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">16 GB</text><rect x="405" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="496.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">identity · 140</text><text x="496.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">Authentik</text><text x="496.5" y="421" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">4 GB</text><rect x="600" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="691.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">logging · 150</text><text x="691.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">2 GB</text><rect x="795" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="886.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">unifi-os · 811</text><text x="886.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">switch · AP</text><text x="886.5" y="421" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">6 GB</text><rect x="990" y="344" width="183" height="110" rx="8" fill="rgba(255,255,255,0.10)" stroke="#fdf6e3" stroke-width="1.5"/><text x="1081.5" y="374" text-anchor="middle" fill="#fdf6e3" font-size="19" font-weight="700">backup · PBS</text><text x="1081.5" y="399" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">on tappaas3</text><text x="1081.5" y="421" text-anchor="middle" fill="#fdf6e3" font-size="16" opacity="0.85">tankc1</text></svg>
|
||
|
||
<!--
|
||
The point of the picture: the top two boxes are the only part anyone in my
|
||
house notices, and they are the small part. Everything below the line is
|
||
what makes them survive a year unattended.
|
||
vllm-amd is the one LXC in the estate — a container, not a VM, because the
|
||
GPU has to be passed through to reach the 8060S.
|
||
-->
|
||
|
||
---
|
||
|
||
## I promised you eight. There are forty-one.
|
||
|
||
| Where | # | What is in there |
|
||
| --- | ---: | --- |
|
||
| **Foundation** — TAPPaaS repo | 8 | `cluster` `network` `templates` `tappaas-cicd` `identity` `backup` `logging` `satellite` |
|
||
| **Apps** — TAPPaaS repo | 12 | `nextcloud` `euro-office` `vaultwarden` `litellm` `openwebui` `vllm-amd` `hass` `deconz` `coturn` `nextcloud-hpb` `netbird-client` `windows-server` |
|
||
| **Community** repo — *other people* | **21** | `immich` `jellyfin` `wordpress` `forgejo` `mailserver` `hosting` `sonos` `hue` `synology` `reolink` `solaredge` `alfen` `unifi` `shelly-fleet` … |
|
||
|
||
Twenty-one of those forty-one are **not mine**. Erik wrote 13, Andreas 6.
|
||
|
||
That is the number I did not dare put in the abstract.
|
||
|
||
<!--
|
||
The abstract promised eight services. Give the 41 a beat before explaining it.
|
||
Then be honest about what "in the tree" means versus "live in my basement":
|
||
a module existing and a module your family depends on are different claims.
|
||
The Community repo is codeberg.org/TAPPaaS/Community — one directory per
|
||
person, no gatekeeping, and it is the only slide in this deck that is
|
||
evidence the project outgrew me. Say that plainly; do not undersell it.
|
||
-->
|
||
|
||
---
|
||
|
||
## "It runs" is a bigger claim than it sounds
|
||
|
||
Every one of those services, without me:
|
||
|
||
- **updates itself** on a schedule — and the update is gated by a test
|
||
- **backs itself up** nightly to Proxmox Backup Server, and off-site
|
||
- **reports its own health**, so I find out before my family does
|
||
- **has a certificate** that renews
|
||
- **has one login** — my identity provider, not eight password fields
|
||
|
||
The interesting engineering is not installing Nextcloud.
|
||
It is Nextcloud still being there, patched, in eighteen months, unattended.
|
||
|
||
<!--
|
||
This is the core argument of the whole project. Slow down here.
|
||
Self-hosting is easy. Self-hosting you can forget about is not.
|
||
-->
|
||
|
||
---
|
||
|
||
## One idea does all the work: everything is a module
|
||
|
||
```text
|
||
nextcloud/
|
||
├── nextcloud.json # the contract — what it is, needs, provides
|
||
├── nextcloud.nix # the NixOS machine
|
||
├── install.sh # put it there (once)
|
||
├── update.sh # keep it patched (on schedule)
|
||
├── test.sh # prove it still works (gates the update)
|
||
└── README.md
|
||
```
|
||
|
||
The firewall is a module. The backup server is a module.
|
||
The mothership that installs the modules is a module.
|
||
|
||
**One model to learn, not eight.**
|
||
|
||
---
|
||
|
||
## `dependsOn` — the trick the whole thing rests on
|
||
|
||
```json
|
||
{
|
||
"description": "Nextcloud — files, calendar, contacts",
|
||
"vmname": "nextcloud", "vmid": 210,
|
||
"dependsOn": ["cluster:vm", "templates:nixos", "backup:vm",
|
||
"network:proxy", "identity:identity"],
|
||
"config": {
|
||
"cluster:vm": { "cores": 4, "memory": "8192", "diskSize": "64G" },
|
||
"network:proxy": { "proxyPort": 443 }
|
||
}
|
||
}
|
||
```
|
||
|
||
Install **order** is computed from these declarations. Nobody maintains a list.
|
||
Add a module, and the platform works out that it needs a VM, an OS, a VLAN,
|
||
a certificate, a login and a backup job — in that order.
|
||
|
||
<!--
|
||
If someone asks "is this just Ansible/Terraform?" — the answer is: those are
|
||
how you build one machine. This is about a machine estate that has a shape.
|
||
-->
|
||
|
||
---
|
||
|
||
## Reduce flexibility. On purpose.
|
||
|
||
> A key design goal is to **REDUCE** flexibility.
|
||
> There is value in decisions having been taken up front.
|
||
|
||
Some use cases will not fit TAPPaaS. That is the trade.
|
||
For what fits, it is dramatically easier.
|
||
|
||
Zones, VLANs, naming, storage roles, backup policy, identity model — decided.
|
||
You get to pick the applications, and where they live.
|
||
|
||
<!--
|
||
This is the least popular slide with hackers and the most important one.
|
||
Every hour you spend re-deciding VLAN layout is an hour not spent on services.
|
||
Expect pushback; welcome it.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# The surprise
|
||
|
||
My basement grew a brain.
|
||
|
||
---
|
||
|
||
## Local AI. Private, offline, mine.
|
||
|
||
| | |
|
||
| --- | --- |
|
||
| Silicon | AMD Ryzen AI MAX+ 395 — Radeon 8060S (Strix Halo) |
|
||
| Memory | **128 GB unified** — the GPU sees nearly all of it |
|
||
| Largest model tested | `gpt-oss-120b` — **120B parameters** |
|
||
| Speed | ~50 tok/s at 7B FP16 · ~20 tok/s at 30B 4-bit |
|
||
| API | OpenAI-compatible, on my own VLAN |
|
||
| Data leaving the building | **none** |
|
||
|
||
One commodity box. Not a rack, not a hyperscaler, not a monthly bill.
|
||
|
||
<!--
|
||
Stress "unified memory" — that is what makes a 120B model possible on a box
|
||
that fits under a desk. This is a genuinely new thing as of this year.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert punch -->
|
||
|
||
## The demo I have been waiting a year to do
|
||
|
||
<br>
|
||
|
||
# Pull the cable out.
|
||
|
||
<br>
|
||
|
||
Then ask it something.
|
||
|
||
<!--
|
||
DEMO. Physically unplug the uplink. Visible. Then a real question in OpenWebUI.
|
||
Have a screen-recorded fallback ready — the tent's projector will betray you.
|
||
If it works live, this is the moment of the talk. Do not rush it.
|
||
-->
|
||
|
||
---
|
||
|
||
## Why this is the point, not a party trick
|
||
|
||
Sovereignty is not a checkbox you tick at a vendor.
|
||
|
||
- The model runs on hardware **you** own
|
||
- Your documents are indexed on **your** VLAN
|
||
- Nobody re-prices it, deprecates it, or reads it
|
||
- It works when the fibre is cut, the account is suspended,
|
||
or the terms of service change on a Tuesday
|
||
|
||
`litellm` in front means apps ask for "a model" — local today,
|
||
someone else's tomorrow, **your choice, revocable**.
|
||
|
||
<!--
|
||
Tie back to bullet four from the dream slide: "no reliance on the Internet".
|
||
That was the mad-sounding one. Here it is, cashed in.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# The Bad
|
||
|
||
What it actually cost.
|
||
|
||
---
|
||
|
||
## The quiet months are the honest part
|
||
|
||
**September–October 2025: 15 commits in two months.**
|
||
|
||
`TODO: what actually happened here — say it plainly, it is the most relatable slide in the deck`
|
||
|
||
The lesson I take from it: a self-hosted platform that needs *you* every week
|
||
is not a platform, it is a pet.
|
||
|
||
Those two months are the real test. The services stayed up.
|
||
Because updates, backups and tests do not need me to be enthusiastic.
|
||
|
||
<!--
|
||
Strong point: the automation is not a nice-to-have, it is what makes the
|
||
project survivable by a human with a life.
|
||
-->
|
||
|
||
---
|
||
|
||
## June 2026: 351 commits. Something changed.
|
||
|
||
That is not me getting three times better at typing.
|
||
|
||
That is the month I leaned all the way into AI-assisted development —
|
||
which is exactly the confession in part three.
|
||
|
||
Hold that thought.
|
||
|
||
<!--
|
||
Deliberate hook into The Ugly. The commit spike is the evidence, and the
|
||
audience will already be suspicious. Good. Let them be.
|
||
-->
|
||
|
||
---
|
||
|
||
## The iceberg under every "simple" service
|
||
|
||
| | Files | Lines |
|
||
| --- | ---: | ---: |
|
||
| **Foundation** — the platform | 575 | **114,138** |
|
||
| **Apps** — the things you actually use | 144 | **15,726** |
|
||
|
||
For every line in an app module, there are **seven lines of platform underneath**.
|
||
|
||
`tappaas-cicd` alone — the mothership — is **81,942 lines**, 72% of the
|
||
foundation. Inside it: 8 TypeScript managers (36,920 lines) and the
|
||
controller layer that talks to Proxmox, OPNsense and the switch (29,352).
|
||
|
||
<!--
|
||
The 7:1 platform-to-app ratio is the single most useful number in this talk
|
||
for anyone thinking "I'll just spin up Docker Compose". That works — until you
|
||
want it to still work next year without you.
|
||
Source: src/STATISTICS.md, regenerated 2026-08-23.
|
||
-->
|
||
|
||
---
|
||
|
||
## It is a lot of work. That is the argument for sharing it.
|
||
|
||
Building this was not cheap, and I will not pretend otherwise:
|
||
**112,000 lines**, 1,507 commits, 24 ADRs, six unfamiliar stacks, a year of evenings.
|
||
|
||
But look at *where* that cost sits:
|
||
|
||
| Paid **once**, by whoever writes it | Paid by **you**, per site |
|
||
| --- | --- |
|
||
| the 8 foundation modules | a box and an afternoon |
|
||
| 195 test suites that gate every update | `module-manager module add <app>` |
|
||
| every install, update and repair script | nothing — patches arrive tested |
|
||
|
||
A hyperscaler amortises a datacentre across a million tenants.
|
||
**We amortise a platform across a thousand basements.**
|
||
|
||
Erik's 13 modules and Andreas's 6 cost me nothing and made my system better.
|
||
Yours would too. That is the economic case: not that self-hosting is cheap,
|
||
but that **the expensive part only has to happen once.**
|
||
|
||
<!--
|
||
This is the slide that turns The Bad from a complaint into an invitation.
|
||
The honest concession first — it IS a lot of work — then the pivot: almost
|
||
none of it is per-site work. Land on the last line and pause.
|
||
If someone objects "a thousand basements do not exist yet" — agree, and say
|
||
that is precisely why you are standing here.
|
||
TODO: your own hours-per-week number makes the concession land harder.
|
||
-->
|
||
|
||
---
|
||
|
||
## Did I get it all right?
|
||
|
||
**No. There have been at least two large rewrites — and a whole system before this one.**
|
||
|
||
- **The predecessor.** I ran a hand-built setup for a few years before TAPPaaS
|
||
existed. Every design decision here is an argument I already lost once, at home.
|
||
- **Rewrite 1 — ADR-007.** Foundation modules used to be *numbered*:
|
||
`05-ProxmoxNode`, `10-firewall`, `30-tappaas-cicd`. The number **was** the
|
||
install order. Retired: order is now computed from `dependsOn`.
|
||
- **Rewrite 2 — ADR-014.** Zone tiers lived in prose in a README. Now `tier` and
|
||
`isolated` are state, and four invariants are machine-checked on every reconcile.
|
||
|
||
Both times the same lesson: something I had encoded as a **convention**
|
||
had to become a **declaration**.
|
||
|
||
`TODO: name the predecessor, and say how long you ran it`
|
||
|
||
<!--
|
||
This is the credibility slide. Do not soften the "No" — an audience that has
|
||
just been told you gave an AI root needs to hear that you rebuild things when
|
||
they are wrong. The ADR-007 and ADR-014 stories are both in the repo if
|
||
anyone wants to check.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# The Ugly
|
||
|
||
The confession.
|
||
|
||
---
|
||
|
||
<!-- _class: invert punch -->
|
||
|
||
# I used AI to build it.
|
||
|
||
# And I gave it root on every node.
|
||
|
||
<br>
|
||
|
||
Not "AI-assisted autocomplete".
|
||
|
||
Root. `ssh`. `nixos-rebuild`. `qm`. `pvesh`. The firewall. The secrets.
|
||
|
||
<!--
|
||
Say it flatly and then be quiet for three full seconds. Let the tent react.
|
||
This is the slide people came for and the one they will argue with afterwards.
|
||
-->
|
||
|
||
---
|
||
|
||
## Why on earth would you do that
|
||
|
||
Because the alternative was that it never got built.
|
||
|
||
- One retiree, evenings and weekends, six unfamiliar technology stacks
|
||
- 112,000 lines of platform code for thirteen app modules
|
||
- The plumbing is *tedious*, not clever — the exact shape of work to hand over
|
||
|
||
The graph does not lie: **May 190, June 351.** That is what handing over the
|
||
tedium looks like.
|
||
|
||
And the honest part: I could not have hand-written the last third of this.
|
||
|
||
<!--
|
||
Do not be defensive. The result is on the projector; the method is the price.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert punch -->
|
||
|
||
# So the real question is not "did you"
|
||
|
||
# It is: **what did you fence it with?**
|
||
|
||
The guardrails are not vibes. They are written down, in the repo,
|
||
loaded on every single session, and they override anything the model
|
||
would otherwise default to.
|
||
|
||
<!--
|
||
This is the pivot from confession to engineering. Everything after this is
|
||
transferable to anyone in the tent using these tools.
|
||
-->
|
||
|
||
---
|
||
|
||
## Guardrail 1 — nothing starts with a prompt
|
||
|
||
By the time the AI is allowed to type, three humans-only artefacts exist:
|
||
|
||
1. **An Issue** — a well-defined problem and a proposed fix. No issue, no change.
|
||
2. **An ADR**, for anything non-obvious — reviewed by **at least two people**.
|
||
Usually me and Erik. 24 of them so far.
|
||
3. **A design and implementation plan** — including **how this will be tested**,
|
||
written *before* a line is written.
|
||
|
||
*Then* the AI takes over — as eight specialist roles: architect, bash, python,
|
||
nix, tester, security, infra, PM.
|
||
|
||
The model does not decide what to build, or what "done" means. It never has.
|
||
|
||
<!--
|
||
This is the most transferable slide in the talk, and the one people actually
|
||
need. The order matters: problem, decision, plan, and only then code.
|
||
Two reviewers on an ADR is the part that keeps me honest — Erik has killed
|
||
several of my ideas, and the ADR is where that argument is recorded.
|
||
-->
|
||
|
||
---
|
||
|
||
## Guardrail 2 — the line it may never cross
|
||
|
||
> **Never run `git commit` or `git push` — full stop.**
|
||
> The operator performs ALL commits and pushes themselves.
|
||
> This holds even when a request seems to imply it — "land it", "ship it",
|
||
> "move this to main" — and even when a previous turn involved committing.
|
||
> That is NOT standing authorization.
|
||
|
||
The AI may change any file on disk. It may not make a change **permanent**.
|
||
|
||
Every single line that entered history passed under my eyes.
|
||
|
||
<!--
|
||
Verbatim from CLAUDE.md. The distinction — mutate freely, persist never — is
|
||
the single most useful idea in this section. Say why: git is the undo button,
|
||
so the undo button is the thing it must not touch.
|
||
-->
|
||
|
||
---
|
||
|
||
## Guardrail 3 — the blast radius is designed
|
||
|
||
| Layer | What it bounds |
|
||
| --- | --- |
|
||
| **Modules** | A mistake lands in one VM, not "the server" |
|
||
| **Zones** | A compromised VM cannot reach what its VLAN forbids |
|
||
| **Proxmox snapshots** | Minutes-old rollback, per machine |
|
||
| **PBS + off-site** | Nightly, immutable, pull-based |
|
||
| **NixOS** | `nixos-rebuild test` before `switch` — a bad config dies at reboot |
|
||
| **34,490 lines of tests** | The update does not land unless the service proves it works |
|
||
|
||
Root access is only terrifying if the system underneath is a snowflake.
|
||
Mine is disposable by construction.
|
||
|
||
<!--
|
||
This is the actual answer to "you gave an AI root?!". The architecture that
|
||
makes unattended updates safe is the same architecture that makes an
|
||
over-eager agent survivable. Same property, two beneficiaries.
|
||
-->
|
||
|
||
---
|
||
|
||
## Guardrail 4 — confirm before the irreversible
|
||
|
||
The standing rules, as written:
|
||
|
||
- **Confirm before destructive ops** — deleting a VM it did not create,
|
||
dropping a storage pool, force-pushing `main`/`stable`, wiping `/etc/secrets`
|
||
- **Fix root causes, not symptoms** — no `--no-verify`, no silenced errors,
|
||
no bypassed CI to make an install "succeed"
|
||
- **Read before you rebuild** — `nixos-rebuild test` before `switch`
|
||
|
||
Note what these have in common: they are all rules about **honesty**,
|
||
not about capability.
|
||
|
||
<!--
|
||
The failure mode with a capable agent is not malice. It is an agent that
|
||
makes the red thing turn green by removing the check. Name that explicitly.
|
||
-->
|
||
|
||
---
|
||
|
||
## Guardrail 5 — the tests decide, not the agent
|
||
|
||
Every module ships **two** test levels:
|
||
|
||
| | When it runs | What it is for |
|
||
| --- | --- | --- |
|
||
| **quick** | every change, every scheduled update | is this service still itself? |
|
||
| **deep** | `test-module.sh <module> --deep` | the full behaviour, too slow for every commit |
|
||
|
||
And around them:
|
||
|
||
- the **test plan is written in the ADR**, before the implementation exists
|
||
- after implementation, a **full regression** across every module — not just the one touched
|
||
- **195 test suites · 34,490 lines · 27% of all source**
|
||
|
||
An agent that can edit code can also edit the test that would catch it.
|
||
That is exactly why a human writes the test plan first, and why the
|
||
regression sweep is the thing that says "done" — not the agent.
|
||
|
||
<!--
|
||
This closes the loop opened in Guardrail 1: the plan said how it would be
|
||
tested, and this is where that promise is collected.
|
||
The last two lines are the answer to the sharpest question in the room —
|
||
"how would you even know if it cheated?"
|
||
-->
|
||
|
||
---
|
||
|
||
## Am I still in control? Honestly.
|
||
|
||
**Yes — but "control" moved.**
|
||
|
||
I no longer control every line. I control:
|
||
|
||
- the **architecture** — modules, zones, contracts
|
||
- the **gate** — commits, pushes, releases
|
||
- the **proof** — 195 test suites, 27% of the source, that must be green
|
||
- the **exit** — it is all open source, on my hardware, in my hands
|
||
|
||
That is a real answer, not a comfortable one. Ask me the hard version in Q&A.
|
||
|
||
<!--
|
||
Do not claim more than this. If someone says "that's not control, that's
|
||
supervision" — agree, and say supervision with a hard gate and a working undo
|
||
is what control has always meant in operations.
|
||
-->
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# Your turn
|
||
|
||
Build one too.
|
||
|
||
---
|
||
|
||
## Why you, specifically, should
|
||
|
||
**European sovereignty is not a policy problem you can wait out.**
|
||
|
||
It is thousands of small boxes, in basements and back offices,
|
||
running software nobody can withdraw.
|
||
|
||
- Your data has to live *somewhere*. Somewhere can be here.
|
||
- A skill you own beats a subscription you rent.
|
||
- Every basement cloud makes the next one cheaper to build.
|
||
|
||
`TODO: your one-sentence version of this — say it in your own words, not mine`
|
||
|
||
---
|
||
|
||
## Start smaller than I did
|
||
|
||
| Step | What you get |
|
||
| --- | --- |
|
||
| 1. One box, evaluation tier | 4 cores / 16 GB / a disk. Nested virt is fine. |
|
||
| 2. Proxmox + OPNsense | Zones, VLANs, DNS, certificates that renew |
|
||
| 3. The mothership | `tappaas-cicd` — the thing that installs the rest |
|
||
| 4. **One** service | Vaultwarden. Small, useful, immediately missed. |
|
||
| 5. Backup **before** service two | Non-negotiable. Ask me why. |
|
||
|
||
No public IP? A satellite VPS is the escape hatch.
|
||
No GPU? Skip local AI, keep everything else.
|
||
|
||
---
|
||
|
||
## Where to find all of it
|
||
|
||
- **tappaas.org** — docs, work in progress, honest about it
|
||
- **codeberg.org/TAPPaaS/TAPPaaS** — the code, the 24 ADRs, the commit graph
|
||
- **sovereigncomputing.org** — the wider argument
|
||
- **This deck** — slides.makerfloss.eu/tappaas/one-year-in
|
||
- **MakerFLOSS** — Orange Makerspace, bi-weekly FLOSS jam. Come build one with us.
|
||
|
||
Contributions welcome. Open an issue before you open a pull request —
|
||
somebody may already be packaging your app.
|
||
|
||
---
|
||
|
||
<!-- _class: invert lead chapter -->
|
||
|
||
# Questions
|
||
|
||
The harder, the better.
|
||
|
||
<!--
|
||
Prepared answers to have loaded:
|
||
- "Isn't this just Ansible?" → estate vs. machine; dependsOn
|
||
- "Why NixOS?" → reproducible rebuild is the undo button
|
||
- "You gave an AI root — seriously?" → guardrail 2, blast radius by design
|
||
- "What if you get hit by a bus?" → open source, docs, ADRs, TODO: honest answer
|
||
- "Cheaper than Google?" → no. Ownable, though.
|
||
- "Can I run it on one Raspberry Pi?" → no. Evaluation tier, x86, be realistic.
|
||
-->
|