mrcharles
78a9258263
ci / lint (push) Successful in 58s
Baseline backup 34a05d70 pulled off-box. Runbook covers 3 layers: HA-level, PBS VM-level (dev-lane drill rule), fresh-HAOS-from-git. Meat: https://projects.knownelement.com/issues/790#note-4970
79 lines
3.4 KiB
Markdown
79 lines
3.4 KiB
Markdown
# pfv-bms restore runbook (#628)
|
||
|
||
Three restore layers exist. Work **inside-out**: usually only layer 1 is
|
||
needed. Restore drills run on the **dev lane only** — never on hosts
|
||
running prod VMs (standing rule after the 2026-09-03 PBS drill that
|
||
I/O-starved prod HA, #684 / t/298).
|
||
|
||
## Layer 0 — what is NOT in git
|
||
|
||
| Artifact | Where it lives | Covered by |
|
||
|----------|----------------|------------|
|
||
| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups |
|
||
| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups |
|
||
| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups |
|
||
|
||
Git (this repo, `release` branch) is the source of truth for
|
||
configuration **YAML only**. A fresh box restored from git still needs a
|
||
`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes
|
||
+ badge/webhook ids) before HA will start.
|
||
|
||
## Layer 1 — HA-level backup (config + settings, small, fast)
|
||
|
||
Taken on-box via the `ha` CLI; produces a self-contained `.tar`:
|
||
|
||
```bash
|
||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
|
||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups list' # find slug
|
||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>' # verify size/state
|
||
# OFF-BOX COPY (the on-box copy dies with the VM disk):
|
||
scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/
|
||
```
|
||
|
||
HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup"
|
||
entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box
|
||
copy remains the manual step.
|
||
|
||
Restore (same or fresh HAOS box):
|
||
|
||
```bash
|
||
scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
|
||
ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'
|
||
```
|
||
|
||
Baseline on record: slug `34a05d70`, name
|
||
`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after
|
||
the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy:
|
||
`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming
|
||
(sha256 6ff2fa6bd5f3dd2f2a47…).
|
||
|
||
## Layer 2 — VM-level backup (prod PBS, nightly)
|
||
|
||
- All prod VMs — **including VM 100 (pfv-bms) on pfv-tsys1** — back up
|
||
nightly to prod PBS, staggered 21:00–04:30 Central, throttled
|
||
60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
|
||
- Restore: PVE web UI or `qmrestore` — **target a scratch VM on the dev
|
||
lane** (verify the restore boots there), never restore in place over
|
||
the live VM without a Kuma maintenance window.
|
||
- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on
|
||
tsys5 are unprotected; fine for beta, not for anything promoted.
|
||
|
||
## Layer 3 — config-from-git (last resort, fresh HAOS)
|
||
|
||
1. Flash/install HAOS, complete onboarding, join Tailscale.
|
||
2. `git clone git.knownelement.com:KNEL/KNELBMS` → copy tree to `/config`
|
||
at the `release` HEAD the fleet expects.
|
||
3. Re-provision `secrets.yaml` (Layer 0 table) — from keeper records.
|
||
4. `ha core check` before starting; then watch the first boot: the
|
||
post-deploy validation automation pages if criticals fail to
|
||
materialize (#778).
|
||
5. Verify the trail: deploy sensor matches `release` HEAD, Kuma
|
||
heartbeat green, doorman roster sensors populated.
|
||
|
||
## Related
|
||
|
||
- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check`
|
||
on-box before any release push.
|
||
- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298
|
||
(Sep 3 wedge), t/320 (reload vs restart deploy semantics).
|