2026-09-06 12:21:42 +00:00
2 changed files with 94 additions and 0 deletions
@@ -0,0 +1,16 @@
+# STATUS — point-in-time agent state (sparse; history lives in Redmine)
+
+Updated: 2026-09-06 ~06:00 CDT · dev @ c4ebb8a · release @ 83b32d5
+
+| ticket | tier | state | item |
+|---|---|---|---|
+| #344 | prod | 🔄 | alert hygiene batch (7e13612) + #790 chain (c4ebb8a) on dev; release PR pending founder |
+| #790 | dev | ✅ | UPS shutdown chain landed, DRY RUN mode; exec wiring needs PVE tokens (NeedsInput) |
+| #628 | docs | ✅ | restore runbook + 788MB baseline backup (slug 34a05d70, off-box on ultix) |
+| #781 | infra | ⏳ | beta HA VM attempt in progress (HAOS on pfv-tsys5) |
+| #345 | prod | 🔄 | rest_command verified registered again; swap-back = founder re-test call |
+
+## Inbox
+- 2026-09-06: tsys6 ambient flap false-positive fixed via 5-min MA smoothing — watch trend sensors for 24h post-deploy
+- 2026-09-06: stlpc-artroom host down (raw+semantic sensors dead) — policy question in NeedsInput
+- gitea→HA deploy webhook still blocked (SSRF allowlist); sha-watch path is the pipeline
@@ -0,0 +1,78 @@
+# pfv-bms restore runbook (#628)
+
+Three restore layers exist. Work **inside-out**: usually only layer 1 is
+needed. Restore drills run on the **dev lane only** — never on hosts
+running prod VMs (standing rule after the 2026-09-03 PBS drill that
+I/O-starved prod HA, #684 / t/298).
+
+## Layer 0 — what is NOT in git
+
+| Artifact | Where it lives | Covered by |
+|----------|----------------|------------|
+| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups |
+| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups |
+| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups |
+
+Git (this repo, `release` branch) is the source of truth for
+configuration **YAML only**. A fresh box restored from git still needs a
+`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes
+ badge/webhook ids) before HA will start.
+
+## Layer 1 — HA-level backup (config + settings, small, fast)
+
+Taken on-box via the `ha` CLI; produces a self-contained `.tar`:
+
+```bash
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups list'          # find slug
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>'   # verify size/state
+# OFF-BOX COPY (the on-box copy dies with the VM disk):
+scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/
+```
+
+HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup"
+entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box
+copy remains the manual step.
+
+Restore (same or fresh HAOS box):
+
+```bash
+scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
+ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'
+```
+
+Baseline on record: slug `34a05d70`, name
+`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after
+the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy:
+`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming
+(sha256 6ff2fa6bd5f3dd2f2a47…).
+
+## Layer 2 — VM-level backup (prod PBS, nightly)
+
+- All prod VMs — **including VM 100 (pfv-bms) on pfv-tsys1** — back up
+  nightly to prod PBS, staggered 21:00–04:30 Central, throttled
+  60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
+- Restore: PVE web UI or `qmrestore` — **target a scratch VM on the dev
+  lane** (verify the restore boots there), never restore in place over
+  the live VM without a Kuma maintenance window.
+- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on
+  tsys5 are unprotected; fine for beta, not for anything promoted.
+
+## Layer 3 — config-from-git (last resort, fresh HAOS)
+
+1. Flash/install HAOS, complete onboarding, join Tailscale.
+2. `git clone git.knownelement.com:KNEL/KNELBMS` → copy tree to `/config`
+   at the `release` HEAD the fleet expects.
+3. Re-provision `secrets.yaml` (Layer 0 table) — from keeper records.
+4. `ha core check` before starting; then watch the first boot: the
+   post-deploy validation automation pages if criticals fail to
+   materialize (#778).
+5. Verify the trail: deploy sensor matches `release` HEAD, Kuma
+   heartbeat green, doorman roster sensors populated.
+
+## Related
+
+- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check`
+  on-box before any release push.
+- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298
+  (Sep 3 wedge), t/320 (reload vs restart deploy semantics).