docs sync post-pass: REPORT v1.1 status, Oct replan runbook, repo path fix
REPORT.md gets a post-pass status banner + per-item STATUS tags + phase/QM checklist truth; RUNBOOK-TODAY amended for the Oct window replan (Friday is troubleshooting-only #609); all ~/optimize paths updated after the repo move to ~/projects/ultix. Adds the 22:00 night-flip watcher log and 6-remove-agent-stacks.sh (boot-time screen/crush relauncher teardown with backups to removed-agent-stacks/). 💘 Generated with Crush Assisted-by: Crush:glm-5.2 [#602]
This commit is contained in:
+32
-25
@@ -1,30 +1,37 @@
|
||||
# Today: two commands, one reboot wave
|
||||
# Runbook 2026-08-31 — COMPLETE (amended after the Oct replan)
|
||||
|
||||
**1. On the VM** (ultix-streaming, any time before step 2):
|
||||
```
|
||||
sudo ~/optimize/1-guest-prep.sh
|
||||
```
|
||||
Installs everything (kernel tunables, BBR, THP, cgroup slices, PSI collector,
|
||||
day/night timers, docker daemon.json, sddm autologin, no autolock). Docker
|
||||
restarts once. Touches NO storage.
|
||||
Steps 1-3 below all landed and were verified read-only the same day (see
|
||||
REPORT.md status banner + §8, and NEXT.md). Kept for the record. Paths
|
||||
updated: repo moved ~/optimize → ~/projects/ultix.
|
||||
|
||||
**2. From your desk** (dry-run first if you like, then execute):
|
||||
```
|
||||
ssh root@pfv-tsys5.knel.net 'bash -s' < ~/optimize/2-host-one-shot.sh
|
||||
ssh root@pfv-tsys5.knel.net 'bash -s -- --go' < ~/optimize/2-host-one-shot.sh
|
||||
```
|
||||
Graceful stop of 5111, ssd/discard/multiqueue/onboot flags, CPU priority over
|
||||
the sectestbed fleet, start 5111, then reboots every other running VM.
|
||||
No disk resizing, no host reboot (NFS role).
|
||||
**Remaining perf actions (as of 2026-08-31 evening):**
|
||||
1. Net multiqueue activation — ONE graceful bounce of VM 5111:
|
||||
`~/projects/ultix/5-activate-multiqueue.sh` (~4 min, kills every crush
|
||||
session on this VM). Human ruling 08-31 eve: system idle tonight — good
|
||||
window after the session's remaining tasks. Verify after:
|
||||
`ethtool -l ens18` → Combined 4/4, `ethtool -l ens19` → 2/2.
|
||||
2. Root growth 279G → ~505G — fully manual, step-by-step with checks:
|
||||
`~/projects/ultix/GROW-ROOT-RUNBOOK.md` (#603). No script runs it; you
|
||||
type every command yourself.
|
||||
|
||||
**3.** Say "done" here; I verify everything read-only and report.
|
||||
**Friday 2026-09-04 (#609) is TROUBLESHOOTING-ONLY**: USB SDR reseat +
|
||||
zero-cost socket/PSU/BIOS checks. NO CPU/RAM swap, NO upgrades. All hardware
|
||||
moved to the ~Oct 2026 window (#601: 2× X5675 + 6× 16GB → 192G; NFS outage
|
||||
for -02 planned first; then q35 + 20 vCPU/128G + GPUs #606). Until then the
|
||||
interim fences (accounts 3G/4.5G) and daynight interim values ARE the
|
||||
operating config.
|
||||
|
||||
**Separate, whenever you choose** (fully manual, step-by-step with checks):
|
||||
root growth 279G -> ~505G via `~/optimize/GROW-ROOT-RUNBOOK.md`. No script
|
||||
runs it; no boot triggers it; you type every command yourself.
|
||||
Original steps, for the record:
|
||||
1. `sudo ~/projects/ultix/1-guest-prep.sh` — kernel tunables, BBR, THP,
|
||||
cgroup slices, PSI collector, day/night timers, docker daemon.json, sddm
|
||||
autologin. DONE + verified.
|
||||
2. `ssh root@pfv-tsys5.knel.net 'bash -s -- --go' <
|
||||
~/projects/ultix/2-host-one-shot.sh` — graceful stop of 5111,
|
||||
ssd/discard/multiqueue-config/onboot flags, cpuunits 9000 over the
|
||||
sectestbed fleet, start 5111. DONE + verified (16:01 bounce).
|
||||
3. Verification — DONE (16:04-16:15 pass). Follow-up root cause: gateway
|
||||
prod was DOWN at boot (bind race + live-restore endpoint loss);
|
||||
permanently fixed same day by the ukrrs-gateway-ensure unit, both lanes.
|
||||
|
||||
Rollback: apply-guest backs up every file it touches; qm flags are
|
||||
individually reversible.
|
||||
|
||||
Friday still owns: CPU/RAM swap (host power-off -> plan the NFS outage for
|
||||
-02 first), second USB3 card reseat, q35 + 20 vCPU / 128G reshape, GPUs.
|
||||
Rollback notes unchanged: apply-guest backs up every file it touches; qm
|
||||
flags are individually reversible.
|
||||
|
||||
Reference in New Issue
Block a user