--- marp: true theme: gaia class: invert paginate: true title: TAPPaaS one year in — The Good, the Bad, and the Ugly description: Sommerhack 2026, Taler Teltet — 27 August, 15:30 --- # TAPPaaS — one year in ## The Good, the Bad, and the Ugly **Lars Rossen** · Sommerhack 2026 · Taler Teltet 27 August, 15:30 --- ## Last year, in this tent, I pitched a dream > Everyone needs a cloud in their basement. > A **T**rusted, **A**utomated, fully **P**rivate **P**latform **a**s **a** **S**ervice. On commodity hardware: - your data, your hardware - no hidden fees - no reliance on closed source - **no reliance on the Internet** A year later I am back with the messy proof that it is real — and to make the case that you should build one too. --- ## The receipts, up front | | | | --- | --- | | First commit | **10 May 2025** — "Initial commit" | | Commits when I submitted this abstract | **1,091** | | Commits standing here today | **1,507** | | Contributors | 3 (I am ~77% of it) | | ADRs written | **24** | | **Modules** — 8 foundation · 12 apps · 21 community | **41** | | Lines of code (bash · TypeScript · Python · Nix) | **126,699** | | Of which **test** code | **34,490** — 27% | | Lines of documentation | **~39,500** | Everything in this talk is in three public repositories — check my homework: - `codeberg.org/TAPPaaS/TAPPaaS` — the platform - `codeberg.org/TAPPaaS/Community` — everyone else's modules - `codeberg.org/TAPPaaS/Documentation` — tappaas.org --- ## What I am actually going to do to you 1. **The Good** — it runs. Here is what "it runs" means, and one surprise. 2. **The Bad** — what it really costs, and why that only works if it is shared. 3. **The Ugly** — the confession. I gave an AI root on every node. 4. **Your turn** — why you should build one, and how to start small. I am not selling anything. There is nothing to buy. --- # The Good It runs. --- ## The iron: three boxes, a switch, an AP and a modem ```mermaid flowchart TB n1["tappaas1 — ASUS · EPYC 4464P
12c/24t · 64 GB ECC
tanka1 2×4 TB NVMe mirror
tankb1 12 TB HDD"] n2["tappaas2 — Minisforum MS-S1 MAX
Ryzen AI MAX+ 395 · 16c/32t
128 GB unified · Radeon 8060S
tanka1 2 TB NVMe"] n3["tappaas3 — TianBei WTR PRO
Ryzen 7 5825U · 8c/16t · 32 GB
tanka1 1 TB NVMe
tankc1 5 TB HDD — PBS"] sw["UniFi USW Pro XG 10 PoE
10.0.0.201
trunk: mgmt untagged +
VLAN 200 310 410 420 430 510 610"] ap["UniFi Nano HD
access point · port 9"] modem["ISP modem
port 5 · VLAN 100 = wan"] n1 -- "10G · p12" --> sw n2 -- "10G · p2" --> sw n3 -- "2.5G · p3" --> sw sw --- ap sw --- modem ``` This is one basement. **There are 4 other known instances of TAPPaaS.** --- ## The same platform, two very different rooms

The MakerFLOSS rack at Orange Makerspace — **makerfloss.eu** — Orange Makerspace. Donated enterprise iron, a community lab.

The rack in my basement at home — **hrossen.dk** — my basement. Three small boxes, a switch, a modem.

Same modules. Same `install.sh`. Same tests. Same updates. --- ## The software: foundation underneath, stacks on top --- ## I promised you eight. There are forty-one. | Where | # | What is in there | | --- | ---: | --- | | **Foundation** — TAPPaaS repo | 8 | `cluster` `network` `templates` `tappaas-cicd` `identity` `backup` `logging` `satellite` | | **Apps** — TAPPaaS repo | 12 | `nextcloud` `euro-office` `vaultwarden` `litellm` `openwebui` `vllm-amd` `hass` `deconz` `coturn` `nextcloud-hpb` `netbird-client` `windows-server` | | **Community** repo — *other people* | **21** | `immich` `jellyfin` `wordpress` `forgejo` `mailserver` `hosting` `sonos` `hue` `synology` `reolink` `solaredge` `alfen` `unifi` `shelly-fleet` … | Twenty-one of those forty-one are **not mine**. Erik wrote 13, Andreas 6. That is the number I did not dare put in the abstract. --- ## "It runs" is a bigger claim than it sounds Every one of those services, without me: - **updates itself** on a schedule — and the update is gated by a test - **backs itself up** nightly to Proxmox Backup Server, and off-site - **reports its own health**, so I find out before my family does - **has a certificate** that renews - **has one login** — my identity provider, not eight password fields The interesting engineering is not installing Nextcloud. It is Nextcloud still being there, patched, in eighteen months, unattended. --- ## One idea does all the work: everything is a module ```text nextcloud/ ├── nextcloud.json # the contract — what it is, needs, provides ├── nextcloud.nix # the NixOS machine ├── install.sh # put it there (once) ├── update.sh # keep it patched (on schedule) ├── test.sh # prove it still works (gates the update) └── README.md ``` The firewall is a module. The backup server is a module. The mothership that installs the modules is a module. **One model to learn, not eight.** --- ## `dependsOn` — the trick the whole thing rests on ```json { "description": "Nextcloud — files, calendar, contacts", "vmname": "nextcloud", "vmid": 210, "dependsOn": ["cluster:vm", "templates:nixos", "backup:vm", "network:proxy", "identity:identity"], "config": { "cluster:vm": { "cores": 4, "memory": "8192", "diskSize": "64G" }, "network:proxy": { "proxyPort": 443 } } } ``` Install **order** is computed from these declarations. Nobody maintains a list. Add a module, and the platform works out that it needs a VM, an OS, a VLAN, a certificate, a login and a backup job — in that order. --- ## Reduce flexibility. On purpose. > A key design goal is to **REDUCE** flexibility. > There is value in decisions having been taken up front. Some use cases will not fit TAPPaaS. That is the trade. For what fits, it is dramatically easier. Zones, VLANs, naming, storage roles, backup policy, identity model — decided. You get to pick the applications, and where they live. --- # The surprise My basement grew a brain. --- ## Local AI. Private, offline, mine. | | | | --- | --- | | Silicon | AMD Ryzen AI MAX+ 395 — Radeon 8060S (Strix Halo) | | Memory | **128 GB unified** — the GPU sees nearly all of it | | Largest model tested | `gpt-oss-120b` — **120B parameters** | | Speed | ~50 tok/s at 7B FP16 · ~20 tok/s at 30B 4-bit | | API | OpenAI-compatible, on my own VLAN | | Data leaving the building | **none** | One commodity box. Not a rack, not a hyperscaler, not a monthly bill. --- ## The demo I have been waiting a year to do # Pull the cable out. Then ask it something. **But we cannot do that here today.** So instead — **let us kill a node.** `tappaas1` is running the firewall (`vm:110`), the mothership (`vm:130`) and identity (`vm:140`) — all HA, fenced, watchdog armed. I cut its power. They come back on another box. --- ## Why this is the point, not a party trick Sovereignty is not a checkbox you tick at a vendor. - The model runs on hardware **you** own - Your documents are indexed on **your** VLAN - Nobody re-prices it, deprecates it, or reads it - It works when the fibre is cut, the account is suspended, or the terms of service change on a Tuesday `litellm` in front means apps ask for "a model" — local today, someone else's tomorrow, **your choice, revocable**. --- # The Bad What it actually cost. --- ## The brutal commit graph of a side project ```text 2025-05 99 ██████████ ← the honeymoon 2025-06 21 ██ 2025-07 53 █████ 2025-08 141 ██████████████ ← Sommerhack 2025 2025-09 6 █ ← vacation 2025-10 9 █ 2025-11 41 ████ 2025-12 47 █████ 2026-01 66 ██████ 2026-02 141 ██████████████ 2026-03 33 ███ 2026-04 19 ██ 2026-05 190 ██████████████████ 2026-06 351 ██████████████████████████████████ ← something changed 2026-07 164 ████████████████ 2026-08 126 ████████████ (to the 25th) ``` Not a burndown chart. A heartbeat — 1,507 beats, and two months where it nearly flatlined. The services stayed up through the flat bits anyway, because updates, backups and tests do not need me to be enthusiastic. --- ## June 2026: 351 commits. Something changed. That is not me getting three times better at typing. That is the month I leaned all the way into AI-assisted development — which is exactly the confession in part three. Hold that thought. --- ## The iceberg under every "simple" service | | Files | Lines | | --- | ---: | ---: | | **Foundation** — the platform | 575 | **114,138** | | **Apps** — the things you actually use | 144 | **15,726** | For every line in an app module, there are **seven lines of platform underneath**. `tappaas-cicd` alone — the mothership — is **81,942 lines**, 72% of the foundation. Inside it: 8 TypeScript managers (36,920 lines) and the controller layer that talks to Proxmox, OPNsense and the switch (29,352). --- ## It is a lot of work. That is the argument for sharing it. Building this was not cheap, and I will not pretend otherwise: **112,000 lines**, 1,507 commits, 24 ADRs, six unfamiliar stacks, a year of evenings. But look at *where* that cost sits: | Paid **once**, by whoever writes it | Paid by **you**, per site | | --- | --- | | the 8 foundation modules | a box and an afternoon | | 195 test suites that gate every update | `module-manager module add ` | | every install, update and repair script | nothing — patches arrive tested | A hyperscaler amortises a datacentre across a million tenants. **We amortise a platform across a thousand basements.** Erik's 13 modules and Andreas's 6 cost me nothing and made my system better. Yours would too. That is the economic case: not that self-hosting is cheap, but that **the expensive part only has to happen once.** --- ## Did I get it all right? **No. There have been at least two large rewrites — and a whole system before this one.** - **The predecessor.** Before TAPPaaS existed I ran a hand-built setup for **five years, uninterrupted**. It never had a name. Every design decision here is an argument I already lost once, at home. - **Rewrite 1 — ADR-003.** Every module was a plain `install.sh` you ran in the right order, by hand. Replaced by `dependsOn` / `provides` contracts, resolved into an install order by the platform. - **Rewrite 2 — ADR-007.** The flat pile of sequenced scripts became a **manager / controller** paradigm: 7 TypeScript managers that own intent, 6 controllers that do the live I/O against OPNsense, Authentik, Proxmox, the switch. Both times the same mistake: I had written as a **sequence of steps** what needed to be a **structure** — dependencies the first time, roles the second. --- # The Ugly The confession. --- # I used AI to build it. # And I gave it root on every node.
Not "AI-assisted autocomplete". Root. `ssh`. `nixos-rebuild`. `qm`. `pvesh`. The firewall. The secrets. --- ## Why on earth would you do that Because the alternative was that it never got built. - One retiree, evenings and weekends, six unfamiliar technology stacks - 112,000 lines of platform code for thirteen app modules - The plumbing is *tedious*, not clever — the exact shape of work to hand over The graph does not lie: **May 190, June 351.** That is what handing over the tedium looks like. And the honest part: I could not have hand-written the last third of this. --- # So the real question is not "did you" # It is: **what did you fence it with?** The guardrails are not vibes. They are written down, in the repo, loaded on every single session, and they override anything the model would otherwise default to. --- ## Guardrail 1 — nothing starts with a prompt By the time the AI is allowed to **code**, 2–3 artefacts exist — carefully crafted and reviewed by humans: 1. **An Issue** — a well-defined problem and a proposed fix. No issue, no change. 2. **An ADR**, for anything non-obvious — reviewed by **at least two people**. Usually me and Erik. 24 of them so far. 3. **A design and implementation plan** — including **how this will be tested**, written *before* a line is written. *Then* the AI takes over — as eight specialist roles: architect, bash, python, nix, tester, security, infra, PM. The model does not decide what to build, or what "done" means. It never has. --- ## Guardrail 2 — the line it may never cross > **Never run `git commit` or `git push` — full stop.** > The operator performs ALL commits and pushes themselves. > This holds even when a request seems to imply it — "land it", "ship it", > "move this to main" — and even when a previous turn involved committing. > That is NOT standing authorization. The AI may change any file on disk. It may not make a change **permanent**. Every single **commit** that entered history passed under my eyes. --- ## Guardrail 3 — the blast radius is designed | Layer | What it bounds | | --- | --- | | **Modules** | A mistake lands in one VM, not "the server" | | **Zones** | A compromised VM cannot reach what its VLAN forbids | | **Proxmox snapshots** | Minutes-old rollback, per machine | | **PBS + off-site** | Nightly, immutable, pull-based | | **NixOS** | `nixos-rebuild test` before `switch` — a bad config dies at reboot | | **34,490 lines of tests** | The update does not land unless the service proves it works | Root access is only terrifying if the system underneath is a snowflake. Mine is disposable by construction. --- ## Guardrail 4 — confirm before the irreversible The standing rules, as written: - **Confirm before destructive ops** — deleting a VM it did not create, dropping a storage pool, force-pushing `main`/`stable`, wiping `/etc/secrets` - **Fix root causes, not symptoms** — no `--no-verify`, no silenced errors, no bypassed CI to make an install "succeed" - **Read before you rebuild** — `nixos-rebuild test` before `switch` Note what these have in common: they are all rules about **honesty**, not about capability. --- ## Guardrail 5 — the tests decide, not the agent Every module ships **two** test levels: | | When it runs | What it is for | | --- | --- | --- | | **quick** | every change, every scheduled update | is this service still itself? | | **deep** | `test-module.sh --deep` | the full behaviour, too slow for every commit | And around them: - the **test plan is written in the ADR**, before the implementation exists - after implementation, a **full regression** across every module — not just the one touched - **195 test suites · 34,490 lines · 27% of all source** An agent that can edit code can also edit the test that would catch it. That is exactly why a human writes the test plan first, and why the regression sweep is the thing that says "done" — not the agent. --- ## Am I still in control? Honestly. **Yes — but "control" moved.** I no longer control every line. I control: - the **architecture** — modules, zones, contracts - the **gate** — commits, pushes, releases - the **proof** — 195 test suites, 27% of the source, that must be green - the **exit** — it is all open source, on my hardware, in my hands That is a real answer, not a comfortable one. Ask me the hard version in Q&A. --- # Your turn Build one too. --- ## Why you, specifically, should **European sovereignty is not a policy problem you can wait out.** It is thousands of small boxes, in basements and back offices, running software nobody can withdraw. - Your data has to live *somewhere*. Somewhere can be here. - A skill you own beats a subscription you rent. - Every basement cloud makes the next one cheaper to build. --- ## Start smaller than I did | Step | What you get | | --- | --- | | 1. One box, evaluation tier | 4 cores / 16 GB / **two disks**. Nested virt is fine. | | 2. Proxmox + OPNsense | Zones, VLANs, DNS, certificates that renew | | 3. The mothership | `tappaas-cicd` — the thing that installs the rest | | 4. **One** service | Home Assistant. Small, useful, immediately missed. | | 5. Backup **before** service two | Non-negotiable. Ask me why. | No public IP? A satellite VPS is the escape hatch. No GPU? Skip local AI, keep everything else. --- ## Where to find all of it - **tappaas.org** — docs, work in progress, honest about it - **codeberg.org/TAPPaaS/TAPPaaS** — the code, the 24 ADRs, the commit graph - **sovereigncomputing.org** — the wider argument - **This deck** — slides.makerfloss.eu/tappaas/one-year-in - **MakerFLOSS** — Orange Makerspace, bi-weekly FLOSS jam. Come build one with us. Contributions welcome. Open an issue before you open a pull request — somebody may already be packaging your app. --- # Questions The harder, the better.