docs sync post-pass: REPORT v1.1 status, Oct replan runbook, repo path fix

REPORT.md gets a post-pass status banner + per-item STATUS tags + phase/QM
checklist truth; RUNBOOK-TODAY amended for the Oct window replan (Friday is
troubleshooting-only #609); all ~/optimize paths updated after the repo move
to ~/projects/ultix. Adds the 22:00 night-flip watcher log and
6-remove-agent-stacks.sh (boot-time screen/crush relauncher teardown with
backups to removed-agent-stacks/).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
[#602]
This commit is contained in:
2026-08-31 21:55:48 -05:00
parent bfc6f92a9f
commit 74934b1095
21 changed files with 178 additions and 74 deletions
+54 -23
View File
@@ -1,4 +1,13 @@
# ultix-streaming optimization report v1 (2026-08-31)
# ultix-streaming optimization report v1.1 (2026-08-31)
> **POST-PASS STATUS (2026-08-31 evening).** The guest plan (§4) is applied
> and verified live; the hot host flags (§8 checklist) are applied; net
> multiqueue is in the qm config and activates at the next VM start (one
> human-timed bounce: `~/projects/ultix/5-activate-multiqueue.sh`). §1 below
> is the PRE-PASS baseline, kept for reference. Hardware work moved to the
> ~Oct 2026 window (#601); Fri 2026-09-04 is troubleshooting-only (#609).
> Living state of record: NEXT.md + TRACKING.md. Repo moved: ~/optimize →
> ~/projects/ultix (paths updated throughout).
Scope: the ultix-streaming KVM guest on pfv-tsys5, tuned for ~9 autonomous agent
accounts + gateway fleet + builds + ETL + SDR + GPU, all concurrent. Inputs:
@@ -10,9 +19,9 @@ Companion files: `questions-v1.md` (answer inline), `host-audit.sh` (run on PVE)
One 2010-era 4-core Xeon currently serves this entire host: your dev
workstation plus 24 other running VMs, with zero resource limits anywhere in
the fleet, and the workstation does not even auto-start after a host reboot.
The fix is roughly $300 of used parts, one maintenance window, and the
configurations already staged in this directory.
the fleet. [2026-08-31: the software half of the fix is LIVE — onboot=1,
ssd=1/discard/iothread, cpuunits 9000/50, all guest tuning, gateway
boot-race fix; the hardware half (~$300 parts) moved to ~Oct 2026, #601.]
| spend item | rough cost (used, 2026) | effect |
|---|---|---|
@@ -35,36 +44,54 @@ backpressure wired end to end. All software pieces are staged and $0.
including 24 running VMs (load 7.5/8 at idle). CPU2 socket is empty.
Drop-in fix: 2× Xeon X5675 (LGA1366, 6C/12T each, 95W, ~$25 each used) = 24
threads, plus CPU2 DIMM slots unlock 192G. See section 8.
[STATUS: parts + window deferred to ~Oct 2026, #601; Fri 09-04 =
troubleshooting-only, #609.]
2. One maintenance window fixes VM shape: machine q35 (i440fx today, blocks clean
GPU passthrough), 20 vCPU / 128G / ballooning off, virtio-scsi-single +
iothread + ssd=1 + discard=on per disk (guest currently sees ALL disks as
ROTATIONAL: the SSD hint is not passed through), net0 multiqueue.
[STATUS: iothread/ssd/discard LIVE + verified (ROTA=0, discard 4K/1G);
multiqueue in qm config, activates at next VM start; q35/20c/128G → Oct
window #601.]
3. No resource limits exist anywhere in the fleet: 18 live containers, zero
cpus/mem/cgroup_parent in any compose, and `dev.sh check` builders default to
GOMAXPROCS=8 on an 8-vCPU box. Staged: cgroup v2 slice architecture
(ukrrs-{gateway,pmo,lsp,batch,rt} + per-account slices via mkacct.sh).
[STATUS: LANDED — slice units installed+enabled; compose cgroup_parent
wiring = #608.]
4. Docker daemon is 100% stock: unbounded json-file logs, live-restore off,
default address pool (~15 networks; you already have 12), no metrics socket.
Staged daemon.json + weekly builder-prune timer (build cache is 9.3G/187 entries).
[STATUS: LANDED + verified live 2026-08-31.]
5. Guest kernel: THP=always (bad for the two Postgres), 66M min_free_kbytes,
tiny socket buffers (212K) for long LLM streams, dirty ratios in % of 48G
(multi-second writeback stalls under ETL). Staged sysctl.d + THP=madvise.
[STATUS: LANDED + verified post-reboot; day/night runtime flips live.]
6. Signals: harness already reads loadavg/MemAvailable/PSI-io
(harness/internal/quota/resources.go:24). Missing: PSI cpu+memory, per-slice
pressure, docker awareness. Cheapest full stack: node_exporter as a compose
service into your existing mopac prometheus + PSI textfile script. Beszel
agent already runs (PID 940). k8s and Proxmox both have clean shedding APIs.
[STATUS: pending, #604.]
7. sddm-greeter burns ~22% of one core 24/7 (stuck on login screen). Fix:
autologin + lock. Sunshine is already installed; it becomes excellent once
the GPU lands (NVENC), replacing any xrdp idea.
[STATUS: FIXED — sddm autologin (Relogin=true) verified, greeter burn
gone. xrdp stays as the Jump/iPad quick-attach path per Q18; Sunshine
waits for the compute GPUs.]
8. Root growth to 500G is a safe online operation (MBR shuffle, §4.3).
[STATUS: pending — manual runbook, #603.]
9. Day/night dynamic profile (§5.5): staged timers flip batch weight, CPU pool,
memory fences and /data2 readahead at 22:00 and 05:00, matching your sleep
window (22:00-05:00) with the gateway's z.ai peak ladder (01:00-05:00 CST).
Guest has no CPU governor to remove (KVM: host owns P-states; set host
governor to performance once, done).
[STATUS: LANDED — day/night timers live; interim values are the operating
config until the Oct window; no cpufreq driver exists (audit §8), so the
profile is cgroup-only exactly as designed.]
## 1. Measured current state
## 1. Measured baseline (PRE-PASS, 2026-08-31 morning — kept for reference;
"now/measured" below means that morning, before anything was applied)
### Guest (ultix-streaming)
| aspect | measured |
@@ -154,10 +181,11 @@ optional device throttles if a specific ETL goes rogue.
## 3. Host plan (pfv-tsys5)
### 3.1 Audit (do this first)
My tooling cannot ssh (hard-banned), so run:
### 3.1 Audit (DONE 2026-08-31 — parsed into §8; the ssh tool-ban was lifted
later that day, crush reaches the host now)
Original instructions, for the record:
```
ssh root@pfv-tsys5.knel.net 'bash -s' < ~/optimize/host-audit.sh > ~/optimize/host-audit.out
ssh root@pfv-tsys5.knel.net 'bash -s' < ~/projects/ultix/host-audit.sh > ~/projects/ultix/host-audit.out
```
I parse the output and finalize §3.2-3.7 numbers. The script is read-only.
@@ -268,7 +296,7 @@ down idle sectestbed VMs entirely (frees host RAM too) and their disks are
already separate (local-lvm spinner vs your dedicated SSDs), so no I/O
arbitration needed. Optional host cmdline at next reboot: `iommu=pt`.
## 4. Guest plan (staged in ~/optimize/staged, apply via apply-guest.sh)
## 4. Guest plan (staged in ~/projects/ultix/staged, apply via apply-guest.sh)
### 4.1 Kernel (60-ukrrs-vm.conf)
| knob | now → set | why |
@@ -344,7 +372,7 @@ across the three physical devices + (rare) per-container device rate limits.
### 4.6 Desktop
Fix the greeter burn: sddm autologin into an UNLOCKED session (ruling
2026-08-31: no autolock, ever; instant Jump/iPad re-attach outranks lock
security on this box). `~/optimize/fix-kde.sh` does it (sudo, idempotent,
security on this box). `~/projects/ultix/fix-kde.sh` does it (sudo, idempotent,
optional --restart). Result: greeter stops rendering, session idles near 0%,
an always-alive unlocked KDE session is attachable instantly. Delete sddm-greeter
CPU cost entirely the day you stop wanting console KDE (multi-user.target) but
@@ -415,7 +443,7 @@ non-persistent; a reboot lands safely in day mode; values in
| batch AllowedCPUs | 0-5 (post 0-15) | 0-6 (post 0-17) |
| batch MemoryHigh | 12G (post 24G) | 30G (post 80G) |
| gateway CPUWeight | 900 | 500 (still far above idle accounts) |
| /data2 readahead | 256K | 1M (ETL sequential) |
| /data2 readahead | 128K (256 sectors) | 1M (2048 sectors, ETL sequential) |
| dirty_bytes / background | 1G / 256M | 2G / 512M |
Honest limit: the guest has no CPU governor to "take off"; KVM host owns
P-states. Set the host governor to performance once (audit confirms current
@@ -443,15 +471,18 @@ agents → ukrrs-batch.slice. Decide via Q15; no OS change either way, the
slice architecture absorbs it.
## 6. Sequencing and rollback
- Phase 0 (now, reversible, ~1h, no reboot): apply-guest.sh sysctl+thp+slices+
noatime+timers+daynight+desktop; daemon.json in a quiet window (one docker
restart).
- Phase 1: host audit (one ssh line), order CPUs/RAM (Q2/Q3).
- Phase 2 (window, VM off): q35 + resize + disk flags + multiqueue; then root
growth to 500G; then GPU + USB3 passthrough; then VM vCPU/RAM at new values.
- Phase 3 (repo tickets, independent): builder caps + cgroup_parent in compose,
resources.go PSI extension, gateway pressure input, #585 cop removal.
- Phase 4: k8s join, proxmox-ctl wiring.
- Phase 0 DONE 2026-08-31 (apply-guest.sh: sysctl+thp+slices+noatime+timers+
daynight+desktop; daemon.json; all verified live post-reboot).
- Phase 1 DONE 2026-08-31 (host audit parsed, §8; CPU/RAM ordering deferred
with the window slip).
- Phase 2 SPLIT by the Oct replan: disk flags + onboot + cpuunits DONE hot
2026-08-31; multiqueue in qm config, activates at next VM start; q35 +
resize + root growth (#603, manual) + GPU/USB3 + 20 vCPU/128G → ~Oct 2026
(#601/#606).
- Phase 3 (repo tickets, independent): builder caps + cgroup_parent in
compose (#608), resources.go PSI extension (#604), gateway pressure input,
#585 cop removal.
- Phase 4: k8s join + proxmox-ctl wiring (#605, blocked by #601).
Every phase independently revertible: sysctl.d and units are files, qm changes
are per-flag, partition work has dumps. Never a bare `down`, per house rules.
@@ -496,15 +527,15 @@ See questions-v1.md, answer inline; version the file if you want another round.
- Only **ONE** Renesas uPD720201 USB3 card visible (24:00.0). You said two.
Check the second: unseated, dead, or behind the (empty) Pericom PCI-X
bridge at 01:00.0.
- VM 5111 has **no onboot flag**: it will not auto-start after a host
reboot. debian13.iso still attached to ide2 (nit).
- VM 5111 had **no onboot flag** and ide2 still held the debian13.iso.
[FIXED 2026-08-31: onboot=1, startup order=10, ide2 deleted.]
- k8s on this host consolidated (2026-08-31): VM 500 pfv-k8s-wnode-tsys5 was
removed; ultix-streaming itself becomes the k8s worker on pfv-tsys5
(12G RAM + 2 vCPU of host pressure freed; timing/control plane per Q11).
- The 18-VM sectestbed fleet runs 24/7 on the same 8 threads and the 2TB
spinner. proxmox-ctl.sh from this VM is the natural on/off switch (Q12).
### QM checklist for the next touch (any time, hot where noted)
### QM checklist (APPLIED 2026-08-31, every line except the two deferred notes)
```
qm set 5111 --onboot 1 --startup order=10,up=180
qm set 5111 -scsi0 NVME:5111/vm-5111-disk-0.qcow2,iothread=1,ssd=1,discard=on,size=438G