Files

T

ci / lint (push) Successful in 58s

Details

docs: restore runbook (#628 ) + sparse STATUS.md [#628 #344 ]

Baseline backup 34a05d70 pulled off-box. Runbook covers 3 layers:
HA-level, PBS VM-level (dev-lane drill rule), fresh-HAOS-from-git.

Meat: https://projects.knownelement.com/issues/790#note-4970

2026-09-06 05:42:45 -05:00

3.4 KiB

Raw Blame History

pfv-bms restore runbook (#628)

Three restore layers exist. Work inside-out: usually only layer 1 is needed. Restore drills run on the dev lane only — never on hosts running prod VMs (standing rule after the 2026-09-03 PBS drill that I/O-starved prod HA, #684 / t/298).

Layer 0 — what is NOT in git

Artifact	Where it lives	Covered by
`secrets.yaml` (tokens, webhook ids, unlock URL)	on-box `/config/secrets.yaml`	Layer 1 + 2 backups
`.storage/` (dashboards-in-storage, registry, auth)	on-box `/config/.storage/`	Layer 1 + 2 backups
Zigbee/HomeKit/BT pairings	on-box	Layer 1 + 2 backups

Git (this repo, release branch) is the source of truth for configuration YAML only. A fresh box restored from git still needs a secrets.yaml re-provisioned from the keeper's records (~/.creds notes

badge/webhook ids) before HA will start.

Layer 1 — HA-level backup (config + settings, small, fast)

Taken on-box via the ha CLI; produces a self-contained .tar:

ssh -p 22222 root@pfv-bms.knel.net 'ha backups new --name "pfv-bms-$(date +%F)"'
ssh -p 22222 root@pfv-bms.knel.net 'ha backups list'          # find slug
ssh -p 22222 root@pfv-bms.knel.net 'ha backups info <slug>'   # verify size/state
# OFF-BOX COPY (the on-box copy dies with the VM disk):
scp -P 22222 'root@pfv-bms.knel.net:/mnt/data/supervisor/backup/<slug>.tar' /backup/destination/

HAOS automatic backups are CONFIRMED LIVE (daily "Automatic backup" entries observed 2026-09-05/06) — layer 1 has a baseline; the off-box copy remains the manual step.

Restore (same or fresh HAOS box):

scp -P 22222 backup.tar root@pfv-bms.knel.net:/mnt/data/supervisor/backup/
ssh -p 22222 root@pfv-bms.knel.net 'ha backups restore <slug> --type full'

Baseline on record: slug 34a05d70, name pfv-bms-baseline-2026-09-06-post-790, 788 MB compressed, taken after the alert-hygiene + #790 batches (dev c4ebb8a). Off-box copy: ~/projects/KNEL/PhysicalPlant/pfv-bms-backups/ on ultix-streaming (sha256 6ff2fa6bd5f3dd2f2a47…).

Layer 2 — VM-level backup (prod PBS, nightly)

All prod VMs — including VM 100 (pfv-bms) on pfv-tsys1 — back up nightly to prod PBS, staggered 21:00–04:30 Central, throttled 60/30 MiB/s + job bwlimit 50 MiB/s (founder ruling t/321).
Restore: PVE web UI or qmrestore — target a scratch VM on the dev lane (verify the restore boots there), never restore in place over the live VM without a Kuma maintenance window.
Gap (open, OAM lane): pfv-tsys5 has no PBS job yet — beta/test VMs on tsys5 are unprotected; fine for beta, not for anything promoted.

Layer 3 — config-from-git (last resort, fresh HAOS)

Flash/install HAOS, complete onboarding, join Tailscale.
git clone git.knownelement.com:KNEL/KNELBMS → copy tree to /config at the release HEAD the fleet expects.
Re-provision secrets.yaml (Layer 0 table) — from keeper records.
ha core check before starting; then watch the first boot: the post-deploy validation automation pages if criticals fail to materialize (#778).
Verify the trail: deploy sensor matches release HEAD, Kuma heartbeat green, doorman roster sensors populated.

Deploy pipeline + gate: README "Deployment pipeline"; ha core check on-box before any release push.
Incident history: #684 (PBS drill), #780 (tsys1 storage), t/298 (Sep 3 wedge), t/320 (reload vs restart deploy semantics).

3.4 KiB Raw Blame History Unescape Escape