# pfv-bms restore runbook (#628)

Three restore layers exist. Work **inside-out**: usually only layer 1 is
needed. Restore drills run on the **dev lane only** — never on hosts
running prod VMs (standing rule after the 2026-09-03 PBS drill that
I/O-starved prod HA, #684 / t/298).

## Layer 0 — what is NOT in git

| Artifact | Where it lives | Covered by |
|----------|----------------|------------|
| `secrets.yaml` (tokens, webhook ids, unlock URL) | on-box `/config/secrets.yaml` | Layer 1 + 2 backups |
| `.storage/` (dashboards-in-storage, registry, auth) | on-box `/config/.storage/` | Layer 1 + 2 backups |
| Zigbee/HomeKit/BT pairings | on-box | Layer 1 + 2 backups |

Git (this repo, `release` branch) is the source of truth for
configuration **YAML only**. A fresh box restored from git still needs a
`secrets.yaml` re-provisioned from the keeper's records (~/.creds notes
+ badge/webhook ids) before HA will start.

## Layer 1 — HA-level backup (config + settings, small, fast)

Taken on-box via the `ha` CLI; produces a self-contained `.tar`:

```bash
ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
ssh -p 22222 root@pfv-bms.knel.net 'ha backups list'          # find slug
ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>'   # verify size/state
# OFF-BOX COPY (the on-box copy dies with the VM disk):
scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/
```

HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup"
entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box
copy remains the manual step.

Restore (same or fresh HAOS box):

```bash
scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'
```

Baseline on record: slug `34a05d70`, name
`pfv-bms-baseline-2026-09-06-post-790`, 788 MB compressed, taken after
the alert-hygiene + #790 batches (dev `c4ebb8a`). Off-box copy:
`~/projects/KNEL/PhysicalPlant/pfv-bms-backups/` on ultix-streaming
(sha256 6ff2fa6bd5f3dd2f2a47…).

## Layer 2 — VM-level backup (prod PBS, nightly)

- All prod VMs — **including VM 100 (pfv-bms) on pfv-tsys1** — back up
  nightly to prod PBS, staggered 21:00–04:30 Central, throttled
  60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
- Restore: PVE web UI or `qmrestore` — **target a scratch VM on the dev
  lane** (verify the restore boots there), never restore in place over
  the live VM without a Kuma maintenance window.
- Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on
  tsys5 are unprotected; fine for beta, not for anything promoted.

## Layer 3 — config-from-git (last resort, fresh HAOS)

1. Flash/install HAOS, complete onboarding, join Tailscale.
2. `git clone git.knownelement.com:KNEL/KNELBMS` → copy tree to `/config`
   at the `release` HEAD the fleet expects.
3. Re-provision `secrets.yaml` (Layer 0 table) — from keeper records.
4. `ha core check` before starting; then watch the first boot: the
   post-deploy validation automation pages if criticals fail to
   materialize (#778).
5. Verify the trail: deploy sensor matches `release` HEAD, Kuma
   heartbeat green, doorman roster sensors populated.

## Related

- Deploy pipeline + gate: README "Deployment pipeline"; `ha core check`
  on-box before any release push.
- Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298
  (Sep 3 wedge), t/320 (reload vs restart deploy semantics).