mrcharles
74934b1095
REPORT.md gets a post-pass status banner + per-item STATUS tags + phase/QM checklist truth; RUNBOOK-TODAY amended for the Oct window replan (Friday is troubleshooting-only #609); all ~/optimize paths updated after the repo move to ~/projects/ultix. Adds the 22:00 night-flip watcher log and 6-remove-agent-stacks.sh (boot-time screen/crush relauncher teardown with backups to removed-agent-stacks/). 💘 Generated with Crush Assisted-by: Crush:glm-5.2 [#602]
2.0 KiB
Runbook 2026-08-31 — COMPLETE (amended after the Oct replan)
Steps 1-3 below all landed and were verified read-only the same day (see REPORT.md status banner + §8, and NEXT.md). Kept for the record. Paths updated: repo moved ~/optimize → ~/projects/ultix.
Remaining perf actions (as of 2026-08-31 evening):
- Net multiqueue activation — ONE graceful bounce of VM 5111:
~/projects/ultix/5-activate-multiqueue.sh(~4 min, kills every crush session on this VM). Human ruling 08-31 eve: system idle tonight — good window after the session's remaining tasks. Verify after:ethtool -l ens18→ Combined 4/4,ethtool -l ens19→ 2/2. - Root growth 279G → ~505G — fully manual, step-by-step with checks:
~/projects/ultix/GROW-ROOT-RUNBOOK.md(#603). No script runs it; you type every command yourself.
Friday 2026-09-04 (#609) is TROUBLESHOOTING-ONLY: USB SDR reseat + zero-cost socket/PSU/BIOS checks. NO CPU/RAM swap, NO upgrades. All hardware moved to the ~Oct 2026 window (#601: 2× X5675 + 6× 16GB → 192G; NFS outage for -02 planned first; then q35 + 20 vCPU/128G + GPUs #606). Until then the interim fences (accounts 3G/4.5G) and daynight interim values ARE the operating config.
Original steps, for the record:
sudo ~/projects/ultix/1-guest-prep.sh— kernel tunables, BBR, THP, cgroup slices, PSI collector, day/night timers, docker daemon.json, sddm autologin. DONE + verified.ssh root@pfv-tsys5.knel.net 'bash -s -- --go' < ~/projects/ultix/2-host-one-shot.sh— graceful stop of 5111, ssd/discard/multiqueue-config/onboot flags, cpuunits 9000 over the sectestbed fleet, start 5111. DONE + verified (16:01 bounce).- Verification — DONE (16:04-16:15 pass). Follow-up root cause: gateway prod was DOWN at boot (bind race + live-restore endpoint loss); permanently fixed same day by the ukrrs-gateway-ensure unit, both lanes.
Rollback notes unchanged: apply-guest backs up every file it touches; qm flags are individually reversible.