mrcharles
46fe0cb6a5
REBUILD.md maps a full rebuild (PVE qm spec incl. nested-virt flag, guest OS/accounts/dotfiles bootstrap, script order, package baseline, verification checklist). 14-capture-state.sh regenerates docs/state snapshots (guest apt/units/sysctl/fstab/docker + PVE 5111 config). Docs sync: stale multiqueue-owed text removed from NEXT/REPORT/RUNBOOK; README script list + open-work list current. Details: https://projects.knownelement.com/issues/615
37 lines
1.9 KiB
Markdown
37 lines
1.9 KiB
Markdown
# Runbook 2026-08-31 — COMPLETE (amended after the Oct replan)
|
||
|
||
Steps 1-3 below all landed and were verified read-only the same day (see
|
||
REPORT.md status banner + §8, and NEXT.md). Kept for the record. Paths
|
||
updated: repo moved ~/optimize → ~/projects/ultix.
|
||
|
||
**Remaining perf actions (as of 2026-09-01):**
|
||
1. Net multiqueue — DONE + verified 2026-09-01 morning (bounce
|
||
22:51->22:53; `ethtool -l` ens18 4/4, ens19 2/2).
|
||
2. Root growth 279G → ~505G — fully manual, step-by-step with checks:
|
||
`~/projects/ultix/GROW-ROOT-RUNBOOK.md` (#603). No script runs it; you
|
||
type every command yourself.
|
||
3. Rebuild-from-scratch path documented: REBUILD.md + docs/state
|
||
snapshots (regen: 14-capture-state.sh) — #615.
|
||
|
||
**Friday 2026-09-04 (#609) is TROUBLESHOOTING-ONLY**: USB SDR reseat +
|
||
zero-cost socket/PSU/BIOS checks. NO CPU/RAM swap, NO upgrades. All hardware
|
||
moved to the ~Oct 2026 window (#601: 2× X5675 + 6× 16GB → 192G; NFS outage
|
||
for -02 planned first; then q35 + 20 vCPU/128G + GPUs #606). Until then the
|
||
interim fences (accounts 3G/4.5G) and daynight interim values ARE the
|
||
operating config.
|
||
|
||
Original steps, for the record:
|
||
1. `sudo ~/projects/ultix/1-guest-prep.sh` — kernel tunables, BBR, THP,
|
||
cgroup slices, PSI collector, day/night timers, docker daemon.json, sddm
|
||
autologin. DONE + verified.
|
||
2. `ssh root@pfv-tsys5.knel.net 'bash -s -- --go' <
|
||
~/projects/ultix/2-host-one-shot.sh` — graceful stop of 5111,
|
||
ssd/discard/multiqueue-config/onboot flags, cpuunits 9000 over the
|
||
sectestbed fleet, start 5111. DONE + verified (16:01 bounce).
|
||
3. Verification — DONE (16:04-16:15 pass). Follow-up root cause: gateway
|
||
prod was DOWN at boot (bind race + live-restore endpoint loss);
|
||
permanently fixed same day by the ukrrs-gateway-ensure unit, both lanes.
|
||
|
||
Rollback notes unchanged: apply-guest backs up every file it touches; qm
|
||
flags are individually reversible.
|