diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..675cf60 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,16 @@ +# STATUS β€” point-in-time agent state (sparse; history lives in Redmine) + +Updated: 2026-09-06 ~06:00 CDT Β· dev @ c4ebb8a Β· release @ 83b32d5 + +| ticket | tier | state | item | +|---|---|---|---| +| #344 | prod | πŸ”„ | alert hygiene batch (7e13612) + #790 chain (c4ebb8a) on dev; release PR pending founder | +| #790 | dev | βœ… | UPS shutdown chain landed, DRY RUN mode; exec wiring needs PVE tokens (NeedsInput) | +| #628 | docs | βœ… | restore runbook + 788MB baseline backup (slug 34a05d70, off-box on ultix) | +| #781 | infra | ⏳ | beta HA VM attempt in progress (HAOS on pfv-tsys5) | +| #345 | prod | πŸ”„ | rest_command verified registered again; swap-back = founder re-test call | + +## Inbox +- 2026-09-06: tsys6 ambient flap false-positive fixed via 5-min MA smoothing β€” watch trend sensors for 24h post-deploy +- 2026-09-06: stlpc-artroom host down (raw+semantic sensors dead) β€” policy question in NeedsInput +- giteaβ†’HA deploy webhook still blocked (SSRF allowlist); sha-watch path is the pipeline diff --git a/docs/restore-runbook.md b/docs/restore-runbook.md new file mode 100644 index 0000000..afcac6f --- /dev/null +++ b/docs/restore-runbook.md @@ -0,0 +1,78 @@ +# pfv-bms restore runbook (#628) + +Three restore layers exist. Work **inside-out**: usually only layer 1 is +needed. Restore drills run on the **dev lane only** β€” never on hosts +running prod VMs (standing rule after the 2026-09-03 PBS drill that +I/O-starved prod HA, #684 / t/298). + +## Layer 0 β€” what is NOT in git + +| Artifact | Where it lives | Covered by | +|----------|----------------|------------| +| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups | +| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups | +| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups | + +Git (this repo, `release` branch) is the source of truth for +configuration **YAML only**. A fresh box restored from git still needs a +`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes ++ badge/webhook ids) before HA will start. + +## Layer 1 β€” HA-level backup (config + settings, small, fast) + +Taken on-box via the `ha` CLI; produces a self-contained `.tar`: + +```bash +ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"' +ssh -p 22222 root@pfv-bms.knel.net 'ha backups list' # find slug +ssh -p 22222 root@pfv-bms.knel.net 'ha backups info ' # verify size/state +# OFF-BOX COPY (the on-box copy dies with the VM disk): +scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/.tar' /backup/destination/ +``` + +HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup" +entries observed 2026-09-05/06) β€” layer 1 has a baseline; the off-box +copy remains the manual step. + +Restore (same or fresh HAOS box): + +```bash +scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/ +ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore --type full' +``` + +Baseline on record: slug `34a05d70`, name +`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after +the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy: +`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming +(sha256 6ff2fa6bd5f3dd2f2a47…). + +## Layer 2 β€” VM-level backup (prod PBS, nightly) + +- All prod VMs β€” **including VM 100 (pfv-bms) on pfv-tsys1** β€” back up + nightly to prod PBS, staggered 21:00–04:30 Central, throttled + 60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321). +- Restore: PVE web UI or `qmrestore` β€” **target a scratch VM on the dev + lane** (verify the restore boots there), never restore in place over + the live VM without a Kuma maintenance window. +- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet β€” beta/test VMs on + tsys5 are unprotected; fine for beta, not for anything promoted. + +## Layer 3 β€” config-from-git (last resort, fresh HAOS) + +1. Flash/install HAOS, complete onboarding, join Tailscale. +2. `git clone git.knownelement.com:KNEL/KNELBMS` β†’ copy tree to `/config` + at the `release` HEAD the fleet expects. +3. Re-provision `secrets.yaml` (Layer 0 table) β€” from keeper records. +4. `ha core check` before starting; then watch the first boot: the + post-deploy validation automation pages if criticals fail to + materialize (#778). +5. Verify the trail: deploy sensor matches `release` HEAD, Kuma + heartbeat green, doorman roster sensors populated. + +## Related + +- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check` + on-box before any release push. +- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298 + (Sep 3 wedge), t/320 (reload vs restart deploy semantics).