docs: restore runbook (#628) + sparse STATUS.md [#628 #344]

Baseline backup 34a05d70 pulled off-box. Runbook covers 3 layers: HA-level, PBS VM-level (dev-lane drill rule), fresh-HAOS-from-git. Meat: https://projects.knownelement.com/issues/790#note-4970
2026-09-06 05:42:45 -05:00
parent c4ebb8aae7
commit 78a9258263
2 changed files with 94 additions and 0 deletions
@@ -0,0 +1,78 @@
+# pfv-bms restore runbook (#628)
+
+Three restore layers exist. Work **inside-out**: usually only layer 1 is
+needed. Restore drills run on the **dev lane only** — never on hosts
+running prod VMs (standing rule after the 2026-09-03 PBS drill that
+I/O-starved prod HA, #684 / t/298).
+
+## Layer 0 — what is NOT in git
+
+| Artifact | Where it lives | Covered by |
+|----------|----------------|------------|
+| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups |
+| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups |
+| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups |
+
+Git (this repo, `release` branch) is the source of truth for
+configuration **YAML only**. A fresh box restored from git still needs a
+`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes
+ badge/webhook ids) before HA will start.
+
+## Layer 1 — HA-level backup (config + settings, small, fast)
+
+Taken on-box via the `ha` CLI; produces a self-contained `.tar`:
+
+```bash
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups list'          # find slug
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>'   # verify size/state
+# OFF-BOX COPY (the on-box copy dies with the VM disk):
+scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/
+```
+
+HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup"
+entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box
+copy remains the manual step.
+
+Restore (same or fresh HAOS box):
+
+```bash
+scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'
+```
+
+Baseline on record: slug `34a05d70`, name
+`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after
+the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy:
+`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming
+(sha256 6ff2fa6bd5f3dd2f2a47…).
+
+## Layer 2 — VM-level backup (prod PBS, nightly)
+
+- All prod VMs — **including VM 100 (pfv-bms) on pfv-tsys1** — back up
+  nightly to prod PBS, staggered 21:00–04:30 Central, throttled
+  60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
+- Restore: PVE web UI or `qmrestore` — **target a scratch VM on the dev
+  lane** (verify the restore boots there), never restore in place over
+  the live VM without a Kuma maintenance window.
+- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on
+  tsys5 are unprotected; fine for beta, not for anything promoted.
+
+## Layer 3 — config-from-git (last resort, fresh HAOS)
+
+1. Flash/install HAOS, complete onboarding, join Tailscale.
+2. `git clone git.knownelement.com:KNEL/KNELBMS` → copy tree to `/config`
+   at the `release` HEAD the fleet expects.
+3. Re-provision `secrets.yaml` (Layer 0 table) — from keeper records.
+4. `ha core check` before starting; then watch the first boot: the
+   post-deploy validation automation pages if criticals fail to
+   materialize (#778).
+5. Verify the trail: deploy sensor matches `release` HEAD, Kuma
+   heartbeat green, doorman roster sensors populated.
+
+## Related
+
+- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check`
+  on-box before any release push.
+- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298
+  (Sep 3 wedge), t/320 (reload vs restart deploy semantics).