Files
ultix/RUNBOOK-TODAY.md
T
mrcharles f815149472 Repo tidy: logs consolidated, 15/16 scripts tracked, docs synced (#615/#611/#603)
Run outputs moved to logs/ (incl. sda partition-table backup). New from
the 15:57-16:03 parallel session: 16-grow-root-online.sh (RAN: / = 431G
online, swap removed by design — GROW-ROOT-RUNBOOK superseded) and
15-docker-to-data2.sh (staged, not run). README/NEXT/RUNBOOK/REBUILD
reflect current reality. Moot rdp-testprep timer removed from the VM.

Details: https://projects.knownelement.com/issues/615
2026-09-01 17:17:10 -05:00

1.9 KiB
Raw Blame History

Runbook 2026-08-31 — COMPLETE (amended after the Oct replan)

Steps 1-3 below all landed and were verified read-only the same day (see REPORT.md status banner + §8, and NEXT.md). Kept for the record. Paths updated: repo moved ~/optimize → ~/projects/ultix.

Remaining perf actions (as of 2026-09-01):

  1. Net multiqueue — DONE + verified 2026-09-01 morning (bounce 22:51->22:53; ethtool -l ens18 4/4, ens19 2/2).
  2. Root growth DONE ONLINE 2026-09-01 16:03 by a parallel crush session: 16-grow-root-online.sh — / is 431G, swap removed by design, no reboot needed (sfdisk backup in logs/ + /root/). #603 satisfied.
  3. Rebuild-from-scratch path documented: REBUILD.md + docs/state snapshots (regen: 14-capture-state.sh) — #615.

Friday 2026-09-04 (#609) is TROUBLESHOOTING-ONLY: USB SDR reseat + zero-cost socket/PSU/BIOS checks. NO CPU/RAM swap, NO upgrades. All hardware moved to the ~Oct 2026 window (#601: 2× X5675 + 6× 16GB → 192G; NFS outage for -02 planned first; then q35 + 20 vCPU/128G + GPUs #606). Until then the interim fences (accounts 3G/4.5G) and daynight interim values ARE the operating config.

Original steps, for the record:

  1. sudo ~/projects/ultix/1-guest-prep.sh — kernel tunables, BBR, THP, cgroup slices, PSI collector, day/night timers, docker daemon.json, sddm autologin. DONE + verified.
  2. ssh root@pfv-tsys5.knel.net 'bash -s -- --go' < ~/projects/ultix/2-host-one-shot.sh — graceful stop of 5111, ssd/discard/multiqueue-config/onboot flags, cpuunits 9000 over the sectestbed fleet, start 5111. DONE + verified (16:01 bounce).
  3. Verification — DONE (16:04-16:15 pass). Follow-up root cause: gateway prod was DOWN at boot (bind race + live-restore endpoint loss); permanently fixed same day by the ukrrs-gateway-ensure unit, both lanes.

Rollback notes unchanged: apply-guest backs up every file it touches; qm flags are individually reversible.