From 962f07ca6bb35b9fc790f0fe4bb8df3e39228ec4 Mon Sep 17 00:00:00 2001 From: sjat Date: Sun, 5 Jul 2026 17:50:20 +0200 Subject: [PATCH] incidents: mamba loses netbird when switch cable plugged in Switch mgmt-VLAN DHCP advertised a default gateway (192.168.88.1) that stole mamba's default route from WiFi, cutting the netbird underlay to askari. Fixed client-side with never-default/high-metric/ignore-auto-dns on the wired NetworkManager profile; WiFi stays default, tunnel holds. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...-05-mamba-loses-netbird-on-switch-cable.md | 70 +++++++++++++++++++ incidents/README.md | 4 +- 2 files changed, 73 insertions(+), 1 deletion(-) create mode 100644 incidents/2026-07-05-mamba-loses-netbird-on-switch-cable.md diff --git a/incidents/2026-07-05-mamba-loses-netbird-on-switch-cable.md b/incidents/2026-07-05-mamba-loses-netbird-on-switch-cable.md new file mode 100644 index 0000000..ad86cb8 --- /dev/null +++ b/incidents/2026-07-05-mamba-loses-netbird-on-switch-cable.md @@ -0,0 +1,70 @@ +# 2026-07-05 — mamba drops its netbird link when the switch cable is plugged in + +- **Host(s):** `mamba` (makerspace laptop), CRS310 `crs310-maker` +- **Reported by / observed:** sjat, on-site at the makerspace +- **Reach path used:** Claude on `ubongo` → netbird overlay → `mamba` + (ssh `sjat@100.99.101.182 -p 7576`). access.md §A + on-demand netbird. + +## Symptom + +sjat is working over a netbird overlay from `ubongo` to `mamba`. The moment the +ethernet cable to the CRS310 is plugged into `mamba` (`enp0s31f6`), the netbird +session drops and Claude loses `mamba`. + +## Investigation + +- netbird on `mamba` is **relayed** via `netbird.askari.wingu.me:443`, so all + overlay traffic egresses `mamba`'s **default route** — normally WiFi + `OrangeMakers` (`172.17.3.60`, `dev wlp0s20f3`, metric 600). +- On carrier-up, NetworkManager auto-activated **`Wired connection 1`** + (autoconnect=yes, DHCP), not the purpose-built static profile `crs310-bench` + (autoconnect=no, so it never won the race). +- The CRS310 mgmt-VLAN DHCP hands out a **default gateway**: + `backups/crs310-maker/export.rsc` → `/ip dhcp-server network add + address=192.168.88.0/24 gateway=192.168.88.1`. +- Wired links get a lower default metric (~100) than WiFi (600), so DHCP + installed `default via 192.168.88.1 dev enp0s31f6` and it **won**. netbird's + underlay to askari then routed through the switch — which by design has **no + gateway/DNS/internet** — so the tunnel died. + +## Root cause + +The wired DHCP profile on `mamba` accepted the switch's DHCP-provided default +gateway, stealing the default route from WiFi and cutting the netbird underlay. +The isolated mgmt VLAN advertising a gateway it can't route is the upstream smell. + +## Fix + +- **Live action taken (mamba, NetworkManager — client-side, this box only):** + ``` + sudo nmcli con mod "Wired connection 1" \ + ipv4.never-default yes ipv4.route-metric 700 ipv4.ignore-auto-dns yes + ``` + Wired link still gets a DHCP address on the mgmt subnet (reaches the switch) + but never installs a default route or hijacks DNS; WiFi stays the default and + the netbird lifeline holds. Not captured in any repo — lives in `mamba`'s NM + config. `crs310-bench` (static `.2`, `never-default yes`) remains as the + alternative sticky profile. +- **Source repo + commit:** none required for the client-side fix. Cleaner root + fix is switch-side (see Follow-ups) and would land in `MakerFLOSS_Mikrotik`. + +## Verification + +With the fix applied and the cable seated (link flapped once on a loose seat, +then stable at 100 Mbps): + +- `mamba` `enp0s31f6` → `192.168.88.253/24` (DHCP from mgmt pool). +- `ip route show default` → **WiFi only** (`172.17.3.1 … metric 600`); no + `192.168.88.1` default appeared. +- `ip route get 100.99.146.14` (ubongo) → still via `wt0` (netbird). +- `ping 192.168.88.1` → 0% loss, ~0.85 ms. Claude never lost `mamba`. + +## Follow-ups + +- **Switch-side root fix (preferred):** drop `gateway=192.168.88.1` from the + mgmt-VLAN `dhcp-server network` in `MakerFLOSS_Mikrotik` — an isolated mgmt + plane with "no gateway/NTP/DNS" should not advertise one. Then *no* client + needs the NM workaround. Branch + run-twice per that repo's lockout rules. +- Consider making `crs310-bench` the sticky wired profile on `mamba` + (`autoconnect yes` + higher priority, `Wired connection 1` off) so the static + `.2` is used deterministically for mgmt work. diff --git a/incidents/README.md b/incidents/README.md index d36c1c1..4326d47 100644 --- a/incidents/README.md +++ b/incidents/README.md @@ -30,4 +30,6 @@ what we found, what we changed (and in which source repo), outcome. ## Log -_(none yet)_ +- [2026-07-05 — mamba drops netbird when the switch cable is plugged in](2026-07-05-mamba-loses-netbird-on-switch-cable.md) + — switch mgmt-DHCP default gateway stole mamba's default route; fixed with + `never-default` on the wired NM profile.