ultix/REPORT.md

# ultix-streaming optimization report v1.1 (2026-08-31)

> **POST-PASS STATUS (2026-08-31 evening).** The guest plan (§4) is applied
> and verified live; the hot host flags (§8 checklist) are applied; net
> multiqueue is in the qm config and activates at the next VM start (one
> human-timed bounce: `~/projects/ultix/5-activate-multiqueue.sh`). §1 below
> is the PRE-PASS baseline, kept for reference. Hardware work moved to the
> ~Oct 2026 window (#601); Fri 2026-09-04 is troubleshooting-only (#609).
> Living state of record: NEXT.md + TRACKING.md. Repo moved: ~/optimize →
> ~/projects/ultix (paths updated throughout).

Scope: the ultix-streaming KVM guest on pfv-tsys5, tuned for ~9 autonomous agent
accounts + gateway fleet + builds + ETL + SDR + GPU, all concurrent. Inputs:
direct measurement of the guest, the ukrrs repo tree, and your answers in chat.
Companion files: `questions-v1.md` (answer inline), `host-audit.sh` (run on PVE),
`staged/` (ready-to-apply configs, gated by `RUN=1`).

## Exec summary

One 2010-era 4-core Xeon currently serves this entire host: your dev
workstation plus 24 other running VMs, with zero resource limits anywhere in
the fleet. [2026-08-31: the software half of the fix is LIVE — onboot=1,
ssd=1/discard/iothread, cpuunits 9000/50, all guest tuning, gateway
boot-race fix; the hardware half (~$300 parts) moved to ~Oct 2026, #601.]

| spend item | rough cost (used, 2026) | effect |
|---|---|---|
| 2× Xeon X5675 | $60-80 | host 8 → 24 threads |
| 6× 16GB DDR3 ECC RDIMM 1333 | $120-180 | 96G → 192G |
| Dell T7500 CPU2 heatsink + paste | $25-40 | required for socket 2 |
| **platform total** | **~$210-300** | **3× threads, dev VM 128G** |
| optional: 1T DRAM-buffered TLC NVMe | $60-100 | only if the Crucial P1 QLC throttles builds |
| 2× compute GPU (models pending, Q5) | ~$300-400 if 2× RTX A2000 12GB | CUDA pool, no PSU change (75W each) |

Expected end state: dev VM at 20 vCPU / 128G with cgroup-fenced account lanes,
gateway and PMO dispatch permanently protected, nightly 22:00-05:00 batch burn
window aligned with the z.ai trickle ladder, sectestbed fleet subordinate by
host-side CPU weights, SDRs on their own controller + RT cores, and PSI-driven
backpressure wired end to end. All software pieces are staged and $0.

## 0. TL;DR, ordered by leverage

1. Host CPUs: CONFIRMED by audit: one E5620 (4C/8T) serves the ENTIRE host,
   including 24 running VMs (load 7.5/8 at idle). CPU2 socket is empty.
   Drop-in fix: 2× Xeon X5675 (LGA1366, 6C/12T each, 95W, ~$25 each used) = 24
   threads, plus CPU2 DIMM slots unlock 192G. See section 8.
   [STATUS: parts + window deferred to ~Oct 2026, #601; Fri 09-04 =
   troubleshooting-only, #609.]
2. One maintenance window fixes VM shape: machine q35 (i440fx today, blocks clean
   GPU passthrough), 20 vCPU / 128G / ballooning off, virtio-scsi-single +
   iothread + ssd=1 + discard=on per disk (guest currently sees ALL disks as
   ROTATIONAL: the SSD hint is not passed through), net0 multiqueue.
   [STATUS: iothread/ssd/discard LIVE + verified (ROTA=0, discard 4K/1G);
   multiqueue in qm config, activates at next VM start; q35/20c/128G → Oct
   window #601.]
3. No resource limits exist anywhere in the fleet: 18 live containers, zero
   cpus/mem/cgroup_parent in any compose, and `dev.sh check` builders default to
   GOMAXPROCS=8 on an 8-vCPU box. Staged: cgroup v2 slice architecture
   (ukrrs-{gateway,pmo,lsp,batch,rt} + per-account slices via mkacct.sh).
   [STATUS: LANDED — slice units installed+enabled; compose cgroup_parent
   wiring = #608.]
4. Docker daemon is 100% stock: unbounded json-file logs, live-restore off,
   default address pool (~15 networks; you already have 12), no metrics socket.
   Staged daemon.json + weekly builder-prune timer (build cache is 9.3G/187 entries).
   [STATUS: LANDED + verified live 2026-08-31.]
5. Guest kernel: THP=always (bad for the two Postgres), 66M min_free_kbytes,
   tiny socket buffers (212K) for long LLM streams, dirty ratios in % of 48G
   (multi-second writeback stalls under ETL). Staged sysctl.d + THP=madvise.
   [STATUS: LANDED + verified post-reboot; day/night runtime flips live.]
6. Signals: harness already reads loadavg/MemAvailable/PSI-io
   (harness/internal/quota/resources.go:24). Missing: PSI cpu+memory, per-slice
   pressure, docker awareness. Cheapest full stack: node_exporter as a compose
   service into your existing mopac prometheus + PSI textfile script. Beszel
   agent already runs (PID 940). k8s and Proxmox both have clean shedding APIs.
   [STATUS: pending, #604.]
7. sddm-greeter burns ~22% of one core 24/7 (stuck on login screen). Fix:
   autologin + lock. Sunshine is already installed; it becomes excellent once
   the GPU lands (NVENC), replacing any xrdp idea.
   [STATUS: FIXED — sddm autologin (Relogin=true) verified, greeter burn
   gone. xrdp stays as the Jump/iPad quick-attach path per Q18; Sunshine
   waits for the compute GPUs.]
8. Root growth to 500G is a safe online operation (MBR shuffle, §4.3).
   [STATUS: pending — manual runbook, #603.]
9. Day/night dynamic profile (§5.5): staged timers flip batch weight, CPU pool,
   memory fences and /data2 readahead at 22:00 and 05:00, matching your sleep
   window (22:00-05:00) with the gateway's z.ai peak ladder (01:00-05:00 CST).
   Guest has no CPU governor to remove (KVM: host owns P-states; set host
   governor to performance once, done).
   [STATUS: LANDED — day/night timers live; interim values are the operating
   config until the Oct window; no cpufreq driver exists (audit §8), so the
   profile is cgroup-only exactly as designed.]

## 1. Measured baseline (PRE-PASS, 2026-08-31 morning — kept for reference;
"now/measured" below means that morning, before anything was applied)

### Guest (ultix-streaming)
| aspect | measured |
|---|---|
| CPU | 8 vCPU presented as 2 sockets × 4 cores × 1 thread; Xeon E5620 2.40GHz (Westmere-EP, LGA1366, 2010; no AVX); 1 NUMA node |
| RAM | 47G usable + 8.8G swap on sda5 (priority -2, unused) |
| Machine type | i440fx + SeaBIOS (blocks clean GPU passthrough; q35 preferred) |
| Kernel/OS | Debian 13 trixie, 6.12.105, cgroup v2 unified, systemd |
| Disks | sda 438G: sda1 root 279G ext4 (29% used), sda5 swap; sdb 400G ext4 /data2 (empty); sdc 200G ext4 /data1 (empty). MBR table. ALL show ROTA=1 |
| Sched/mount | scheduler `none` on all disks (good), readahead 128K, relatime everywhere, fstrim.timer present |
| Desktop | KDE + sddm, greeter idling at ~22% of a core since boot (118 min CPU); no autologin; Sunshine installed, unused; no xrdp; no nvidia userspace |
| PSI now | cpu some avg60 ~1.2%, memory/io ~0 (idle baseline) |
| Desktop default | readlink default.target empty (graphical) |

Key tunables (measured): swappiness 60, dirty_ratio 20/10 (percent-based),
min_free_kbytes 66M, max_map_count 1048576 (fine), aio-max-nr 65536, inotify
watches 386K / instances 128, somaxconn 4096 (fine), conntrack 262144 (fine),
ip_local_port_range 32768-60999, cubic only (bbr module not loaded),
rmem_max/wmem_max 212992, slow_start_after_idle=1, THP enabled=always,
autogroup=1 (good, keep for screen sessions), pid_max fine. tuned not installed.

### Docker
| aspect | measured |
|---|---|
| Version/driver | 29.7.2, overlayfs, systemd cgroup driver, cgroup v2 |
| Config | NO daemon.json: unbounded json-file logs, live-restore false, data-root /var/lib/docker on root fs |
| Footprint | 24 images 14.3G, build cache 9.3G (187 entries), 18 containers up, 6 volumes |
| Networks | 12 total (9 user bridges). Default pool = 172.17-31/16, caps at ~15 networks |
| Running | 2 gateway stacks + 2 postgres (prod+beta), 9 LSP, openwebui, cli-mred, 2 fakes |

### Fleet facts (from ukrrs tree)
- 9 harness daemons planned on THIS host (harness/deploy/accounts.tsv, idx 0-8:
  reachableceo, TSGBOD, TSGCOO, TSGCTO, TSGCCO, reachableceo-offstage,
  COSRCEO-Personal, COSRCEO-Biz, COSWFO). NOTE: /home currently shows different
  names (COS-RCEO, COS-TSG, ...) so the account map is not final (Q7).
- Gateway: fleet semaphore 15 parallel provider calls, policy ladder
  normal/peak/cooldown/hard recomputed every 30s, pacing cruise 90%.
- Harness gate order: resources → quota → peak → soft-defer. Resource reads:
  loadavg, MemAvailable, PSI io only. Known gaps: harness still polls retired
  cop :8110 (#585), credits are estimates, no PSI cpu/mem.
- Scale target: ~100 work streams, 24×7, 750 credits/hour.
- `./dev.sh check` = pinned golang builder, go build+vet+test, NO -p/cpus flags
  → each builder defaults to all 8 vCPUs. 42 MCP + 10 LSP always-on.
- Beszel agent running (PID 940); prometheus with 30d TSDB already in harness
  compose (mopac-harness-prometheus).

## 2. Bottleneck model and the policy

Who fights whom on this box:

| workload | cpu | ram | disk | net | latency sensitivity |
|---|---|---|---|---|---|
| LLM harness loops ×9 | low | low | low | 9 × long streams | medium (streaming) |
| gateway + 2× postgres | low-med | med (shared_buffers+cache) | WAL writes | all LLM traffic, via tailscale | HIGH |
| docker pulls (burst) | med (gzip) | low | layer writes (NVMe) | burst 100s MB | low |
| compiles (dev.sh check) | saturating, GOMAXPROCS=8 each | 2-6G each | cache reads (NVMe) | low | low |
| ETL | med | page cache heavy | sequential r/w (SSD) | low | low |
| LSP ×10 + MCP 42 | idle mostly | 0.3-1G total-ish | low | none | low (interactive bursts) |
| SDR DSP (future) | 1-2 cores steady | low | low | low | HARD real-time |
| Sunshine encode (future) | 2-4 cores CPU / ~0 with NVENC | low | low | LAN | medium |

Design in one sentence: schedule by intent, not by luck. Three CPU pools
(interactive/gateway, batch/build, rt/sdr), per-account fair shares with cgroup
v2 weights + memory fences, spindle separation per I/O class, and PSI as the
universal pressure signal your harness already half-reads.

```
                    +----------------- system.slice -----------------+
                    |  ukrrs-gateway.slice   CPUWeight 900  (gateway+pg stacks)
                    |  ukrrs-pmo.slice       CPUWeight 200  (9 PMO dispatch loops)
                    |  ukrrs-lsp.slice       CPUWeight 150  (LSP + MCP tier)
                    |  ukrrs-batch.slice     CPUWeight 25   AllowedCPUs pool A
                    |  ukrrs-acct-<9>.slice  CPUWeight 75   AllowedCPUs pool A
                    |  ukrrs-rt.slice        AllowedCPUs pool B (SDR, encode)
                    |  docker default (everything not tagged) = last
                    +---------------------------------------------------+
  interim 8 vCPU:  pool A = 0-5, pool B = 6-7
  post-upgrade 20: pool A = 0-15, pool B = 18-19 (gateway may use 0-19)
```

Fairness math: with 9 accounts each compiling at weight 75 and gateway at 900,
gateway keeps ~13% of CPU even under total compile storm; with weights alone an
idle machine still lets ONE account use every core (burstable, no quotas). I/O
weights are inert under the `none` scheduler, so fairness comes from separation
(NVMe = images/builds, /data1 = databases, /data2 = ETL/backup/scratch) plus
optional device throttles if a specific ETL goes rogue.

## 3. Host plan (pfv-tsys5)

### 3.1 Audit (DONE 2026-08-31 — parsed into §8; the ssh tool-ban was lifted
later that day, crush reaches the host now)
Original instructions, for the record:
```
ssh root@pfv-tsys5.knel.net 'bash -s' < ~/projects/ultix/host-audit.sh > ~/projects/ultix/host-audit.out
```
I parse the output and finalize §3.2-3.7 numbers. The script is read-only.

### 3.2 CPU upgrade (drop-in, LGA1366)
| option | result | notes |
|---|---|---|
| add 2nd E5620 (~$10) | 8C/16T | cheapest, matches existing |
| 2× X5675 (recommended) | 12C/24T @3.06/3.46 | 95W each, DDR3-1333, ~$25 each |
| 2× X5690 | 12C/24T @3.46/3.73 | 130W each, hot, marginal gain, PSU risk with GPUs |
| 2× L5640 | 12C/24T @2.27 | low power, slower |

Caveats the audit resolves: chassis must be dual-socket (T5500/T7500; T3500 is
single), BIOS revision (A17 supports 5600-series), current DIMM layout, PSU
rating. Westmere lacks AVX/AVX2: Go/Rust/CUDA fine, but some prebuilt binaries
(newer Node native modules, llama.cpp CPU builds, some distro packages) assume
AVX; plan on building from source or using GPU for inference.

### 3.3 RAM
96G now is CPU0 DIMMs only (6 slots). Second CPU unlocks 6 more slots → 192G.
Give the VM 128G, leave 64G for host + preprod VMs. Use DDR3 ECC RDIMM 1333
(12800R); mixing 10600R works but clocks down. Balanced population per branch.

### 3.4 VM re-shape (one maintenance window, VM off)
```
qm set <id> --machine q35                 # from i440fx; expect NIC rename in guest
qm set <id> --cpu host --sockets 2 --cores 5 --threads 2 --numa 1   # 20 vCPU
qm set <id> --memory 131072 --balloon 0
qm set <id> --scsihw virtio-scsi-single
# per disk (real disk ids from audit):
qm set <id> -scsi0 <store>:vm-<id>-disk-0,ssd=1,discard=1,iothread=1   # repeat sdb sdc
qm set <id> --net0 virtio=<mac>,bridge=<br>,multiqueue=4
```
q35 + NIC rename: prepare a systemd .link file or just let udev rename and
restart networking; tailscale state survives. Guest then sets 4 RSS queues
(ethtool -L, handled by a tiny oneshot unit, staged). If ballooning was on, off.
i440fx→q35 is required for the GPU plan; do it once, with console access.

### 3.5 Storage backend
Fixes: ssd=1 (kills the ROTA=1 lie the guest sees today), discard=on (with the
fstrim.timer already present), virtio-scsi-single + iothread=1 (one iothread per
disk spreads interrupts), cache=none on ZFS backend / writeback only on
LVM-thin with UPS. Audit reports backend + link speeds. Known ceiling: if the
two SSDs sit on the onboard ICH10R SATA2 ports they cap ~280MB/s each (3Gbps
link); the NVMe on a PCIe2.0 x4 card caps ~1.7GB/s. Fine for this fleet, just
set expectations for ETL throughput.

### 3.6 GPU passthrough
Now: old NVIDIA on host. Soon: two newer GPUs. Plan:
1. IOMMU: already active with clean per-device groups (audit section 8). No
   ACS override needed. Optional at next host reboot: add `iommu=pt` for
   cheaper DMA on passthrough devices.
2. The Quadro 4000: skip entirely. Fermi has no support in any CUDA toolkit
   this decade (compute capability 2.x was dropped at CUDA 9) and no driver
   that builds on kernel 6.x. It stays as the host console card.
3. The two newer GPUs are COMPUTE cards (per ruling 2026-08-31), which is
   easier than display passthrough: no dummy plug, usually no romfile, pass
   without x-vga (`qm set 5111 -hostpci0 0000:xx:00.0`), modern NVIDIA
   drivers (465+) officially support VMs. Remaining risk: this-era BIOS has
   no Above-4G-Decoding, so big-VRAM cards with large BARs may fail to map.
   Sweet spot for this box: 2× used RTX A2000 12GB (~$150-200 each): 75W
   slot power, no aux cables, no PSU change, 24G VRAM total, BARs fit. Larger
   cards (3060 12G / 4060Ti 16G class) need the PSU check + may hit the BAR
   wall. q35 recommended when they land; OVMF only if a UEFI-only vBIOS
   forces it.
4. In-guest: nvidia driver + nvidia-container-toolkit; builders get `--gpus
   all` in the batch slice (house-style digest-pinned CUDA image).
5. PSU budget if going bigger: 2× 95W CPU + 2× 200-250W GPUs + platform ≈
   750-850W sustained: needs the 875W PSU option (Q5 sticker check).
6. Sunshine (already installed) still benefits even though the cards are
   compute-only: capture the KDE session via X11/KMS on the emulated VGA,
   encode on the GPU with NVENC headless. CPU encode (current state,
   2-4 Westmere cores at 1080p60) remains tolerable-but-temporary. No xrdp:
   it gives a second synthetic session with no accel; Sunshine mirrors the
   real session with audio. SSH stays break-glass.

### 3.7 SDR passthrough
You have two PCI USB3 cards, SDRs load-balanced across them: pass the CARDS,
never devices. `qm set <id> -hostpci1 <id1> -hostpci2 <id2>`. Guest gets native
xHCI controllers with real MSI interrupts (emulated USB adds latency and jitter
that ruins tuner sample streams). SDR processes run in ukrrs-rt.slice
(AllowedCPUs pool B) so compile storms cannot preempt DSP. Bandwidth: tens of
MB/s per SDR, trivial for USB3. libusb latency in a VM is fine for SDR work.

### 3.8 Proxmox control from the harness
PVE API token (appendix B has exact pveum commands), then
`staged/proxmox-ctl.sh vms|shutdown|start|snapshot` with a hard PROTECTED list
containing this VM and anything untouchable. Natural uses: stand preprod VMs
down during compile storms, snapshot before risky agent work, bring up scratch
VMs. Scoping question Q12.

### 3.9 Host-side fleet arbitration (dev VM over sectestbed/preprod)
Yes, the audit pulled every VM config (26 VMs: 24 running, 1 stopped, 1
template). Two live levers, no reboots:
```
# priority: PVE maps cpuunits to cgroup v2 CPUWeight (clamped 1-10000)
qm set 5111 --cpuunits 9000
FLEET="5000 5101 5102 5103 5104 5105 5106 5107 5108 5109 51011 51012 51013 51014 51015 51016 515 53100 53101 53102 53103 53104 53105 53106 53107 53108"
for id in $FLEET; do qm set $id --cpuunits 50; done
# optional hard caps where the test fleet never needs burst:
#   qm set <id> --cpulimit 1
# (VM 500 k8s-wnode was removed 2026-08-31: ultix-streaming becomes the k8s worker)
```
At equal demand the dev VM holds ~87% of CPU; the fleet still bursts to full
idle capacity when the dev VM is quiet. Post-upgrade, physical partitioning
on 24 threads: `qm set 5111 --affinity 0-19` and the fleet `--affinity 20-23`
(so test VMs can never preempt dev cores at all). Add proxmox-ctl standing
down idle sectestbed VMs entirely (frees host RAM too) and their disks are
already separate (local-lvm spinner vs your dedicated SSDs), so no I/O
arbitration needed. Optional host cmdline at next reboot: `iommu=pt`.

## 4. Guest plan (staged in ~/projects/ultix/staged, apply via apply-guest.sh)

### 4.1 Kernel (60-ukrrs-vm.conf)
| knob | now → set | why |
|---|---|---|
| dirty_background_bytes / dirty_bytes | % ratios → 256M / 1G | bytes-based caps writeback stalls (ETL) regardless of RAM size |
| min_free_kbytes | 66M → 384M (768M post-upgrade) | survive bursty reclaim under compile+ETL |
| aio-max-nr | 65536 → 1M | postgres AIO / io_uring era defaults |
| inotify instances | 128 → 512; watches 386K → 1M | 9 harness daemons + LSPs + crush sessions watching repos |
| ip_local_port_range | 32768-60999 → 10240-65535 | 9 accounts × long-lived streams + tailscale + docker NAT |
| tcp_tw_reuse | 2 → 1 | outbound provider connection churn |
| tcp_slow_start_after_idle | 1 → 0 | LLM turns idle minutes between bursts on live sockets |
| rmem_max/wmem_max + tcp_r/wmem | 212K → 16M | large SSE/JSON streams through gateway |
| congestion control | cubic → bbr + fq | module load staged; falls back to cubic if absent |
| DefaultLimitNOFILE | 1024 soft → 65536 | harness daemons, many sockets/files |

Left alone deliberately: swappiness 60, overcommit 0, vfs_cache_pressure,
pid_max, somaxconn 4096, conntrack 262144, autogroup 1, page-cluster.
THP: always → madvise (oneshot unit, staged). Postgres dislikes always-THP;
Go/builds get THP via madvise anyway where it matters.

### 4.2 Mounts and root growth to 500G (MBR shuffle, online except swapoff)
Ruling 2026-08-31: storage operations are MANUAL ONLY, executed by the human,
one command at a time. Nothing automatic, nothing boot-triggered. The
step-by-step checklist with per-step verification is GROW-ROOT-RUNBOOK.md;
the commands below are reference for what it does.
Current MBR: sda1 root ends 279G, then extended sda2 holding sda5 swap 8.8G,
then ~150G unallocated (disk 438G). Target: root 500G → grow disk at PVE first:
`qm resize <id> scsi0 +80G` (→ 518G), then in-guest:
```
sfdisk --dump /dev/sda > /root/sda.sfdisk.bak; cp /etc/fstab /root/fstab.bak
swapoff /dev/sda5
sfdisk --delete /dev/sda 2            # removes extended + logical swap
growpart /dev/sda 1                   # cloud-guest-utils; grows root partition
resize2fs /dev/sda1                   # online ext4 grow → 505G
parted /dev/sda mkpart primary linux-swap 505GB 100%; mkswap /dev/sdaX
# fix fstab swap UUID (blkid), swapon -a; verify with lsblk + df -h /
```
Rollback: fstab + sfdisk dumps kept; original layout restorable offline. Also:
add `noatime` to /, /data1, /data2 (remount online; relatime today).

### 4.3 Swap
Keep sda5-style swap at disk tail (fresh 10G from the shuffle). Skip zram: old
CPU (compression burns cores) and RAM is tripling soon. Optional: 32G low-pri
swapfile on /data2 as OOM insurance post-upgrade (staged flag, default off).

### 4.4 Docker (daemon.json staged)
live-restore true (daemon restarts stop killing agent containers; aligns with
your "never bounce prod for convenience" rule), json-file logs capped 20m × 3
(9 accounts × 42 MCP without caps = quiet-log rule violation waiting to happen),
max-concurrent-downloads 6 (parallel pulls across accounts), address pool
172.16/12 as /24s (you are at 12 of ~15 default networks; the fleet adds one
per compose project), metrics on 127.0.0.1:9323 (experimental:true is required
by docker for the engine metrics endpoint; drop both lines if unwanted).
Weekly `docker builder prune --keep-storage 25GB` timer staged (9.3G today,
unbounded growth otherwise). Applying daemon.json restarts docker once (do it in
a quiet moment; live-restore protects future restarts only).

### 4.5 cgroup v2 architecture (the core deliverable)
Static slices staged: gateway (CPUWeight 900, MemoryMin 2G, MemoryHigh 12G),
pmo (200, High 1G/Max 1.5G; the dispatch control plane: cheap but weighted
above workers so a compile storm can never starve dispatch), lsp (150, High
6G/Max 8G), batch (25, High 12G/Max 16G, pool A cpus), rt (pool B cpus,
weight 10000). Night profile overrides batch/gateway at runtime (§5.5). Per-account slices via `mkacct.sh <name> <uid>`:
CPUWeight 75, TasksMax 4096, Memory fences (interim 48G: High 3G/Max 4.5G ×9;
post-upgrade 128G: High 10G/Max 12G ×9), plus a matching user-<uid>.slice
drop-in so the harness daemon process itself is fenced too.
Containers join their account slice via compose `cgroup_parent:` (snippet
printed by mkacct). Builders and ETL join ukrrs-batch via dev.sh/compose.
Everything untagged lands in plain docker scope = effectively lowest priority.
NOTE: with the `none` I/O scheduler, IOWeight is inert; fairness = separation
across the three physical devices + (rare) per-container device rate limits.

### 4.6 Desktop
Fix the greeter burn: sddm autologin into an UNLOCKED session (ruling
2026-08-31: no autolock, ever; instant Jump/iPad re-attach outranks lock
security on this box). `~/projects/ultix/fix-kde.sh` does it (sudo, idempotent,
optional --restart). Result: greeter stops rendering, session idles near 0%,
an always-alive unlocked KDE session is attachable instantly. Delete sddm-greeter
CPU cost entirely the day you stop wanting console KDE (multi-user.target) but
that kills Sunshine's session, so default plan keeps KDE.

KDE vs XFCE verdict: do not switch for resource reasons. Locked/idle Plasma is
~0% CPU and roughly 600-900M RSS vs XFCE ~350-450M; on a box going to 128G
that delta is noise. The 22% greeter burn was a greeter-stuck artifact, not
Plasma weight, and lightdm+XFCE can do the same trick. The one real argument
for XFCE later: it is X11-only, and X11 is still the most battle-tested
Sunshine capture path with the NVIDIA proprietary driver; Plasma Wayland
capture on NVIDIA is good in 2026 but younger. Sequence: keep KDE now, fix
autologin+lock, land the GPU, test Sunshine on Wayland+NVENC, switch to XFCE
only if capture disappoints. Revisit when Agent Zero lands (Q15): if it ever
runs headed-browser tasks, the desktop story matters more than RAM.

## 5. Signals and integrations

### 5.1 What the harness reads today and the minimal patch
resources.go reads loadavg, MemAvailable, PSI-io avg60, disk free, with clean
seams. Extend (same pattern, ~40 lines + tests, ticket per repo):
- generalize readIODelay → readPressure(res) for cpu and memory
- optional: read /sys/fs/cgroup/ukrrs-batch.slice/cpu.pressure to gate on "is
  the batch pool saturated" rather than host-wide load (loadavg counts 24
  post-upgrade vCPUs; PSI is the honest signal)
- suggested defer thresholds: cpu some avg60 > 20, io avg60 > 25, mem some
  avg60 > 10 (tune with real data; the metric names already exist
  harness_resource_load_avg_1m / _io_delay_pct)

### 5.2 Metrics stack (your "super lightweight" ask)
Nothing new to run on the host OS: node_exporter as a compose service
(read-only binds of / and /var/lib/node_exporter/textfile, house-compliant,
digest-pinned at adoption), scraped by the ALREADY RUNNING
mopac-harness-prometheus (30d TSDB). PSI textfile script (staged) adds
per-slice pressure gauges every 15s. Docker engine metrics land on
127.0.0.1:9323 via daemon.json. Beszel (already installed) keeps the human
dashboard role. Gateway choice (Q10): poll node_exporter directly in its 30s
tick (no new dependency, ~30 lines) vs PromQL against existing prom. Either
beats the retired cop :8110 (#585) it still polls.

### 5.3 k8s (yes, the APIs shed work cleanly)
- `kubectl cordon ultix-streaming` stops new scheduling; `kubectl drain
  --ignore-daemonsets --delete-emptydir-data --grace-period=120` evicts.
- Set all k8s workloads to a low PriorityClass + preemption policy, so burst
  pods die first under node pressure automatically.
- kubelet eviction thresholds (memory.available<2Gi, imagefs.available<10Gi)
  auto-shed when builds eat disk/RAM. Plus --system-reserved/--kube-reserved.
- Recommendation: join AFTER the host upgrade, as worker only, never control
  plane, kubelet+containerd units get CPUWeight 20 + MemoryHigh fences so the
  compose fleet always wins arbitration. Control plane location Q11.

### 5.4 Proxmox
See §3.8; skeleton staged, token + PROTECTED list via Q12.

### 5.5 Day/night dynamic profile + PMO dispatch backpressure
Windows (Q14): OS night profile 22:00-05:00 local (your sleep window); the
z.ai peak ladder stays 01:00-05:00 CST as the LLM-trickle subset
(gateway/config.yaml:21-26, harness mirror config.go:235-244). Net effect:
22:00 the system opens up for batch, 01:00 LLM drops to trickle while CPU/IO
burn maxes, 05:00 everything returns to day bias.

What flips at night (systemctl set-property --runtime + sysctl, all
non-persistent; a reboot lands safely in day mode; values in
/etc/ukrrs/daynight.conf, staged):
| object | day | night (48G interim / 128G post) |
|---|---|---|
| batch slice CPUWeight | 25 | 400 |
| batch AllowedCPUs | 0-5 (post 0-15) | 0-6 (post 0-17) |
| batch MemoryHigh | 12G (post 24G) | 30G (post 80G) |
| gateway CPUWeight | 900 | 500 (still far above idle accounts) |
| /data2 readahead | 128K (256 sectors) | 1M (2048 sectors, ETL sequential) |
| dirty_bytes / background | 1G / 256M | 2G / 512M |
Honest limit: the guest has no CPU governor to "take off"; KVM host owns
P-states. Set the host governor to performance once (audit confirms current
state); an always-busy 15-year-old Xeon saves nothing from dynamic P-states.
Night is also the maintenance window: builder prune Sun 22:30, fstrim,
postgres vacuum/reindex, image pre-pull for the morning fleet.

PMO backpressure: yes, gate at dispatch, it is the earliest and cheapest
point. Each account's PMO loop claims from Redmine only when, in order (all
zero-dependency reads, same pattern as resources.go):
1. gateway /status policy mode is normal (else trickle: ladder already
   handles 01:00-05:00),
2. /proc/pressure/{cpu,memory,io} avg60 under thresholds,
3. ukrrs-batch.slice cpu/io pressure under threshold (worker pool headroom),
4. wall clock: night = max parallel builders + bigger tickets (compiles,
   ETL, doc builds); day = fewer, interactive-sized tickets.
The 9 PMO loops live in ukrrs-pmo.slice so worker storms cannot starve the
dispatcher. This gate composes with the existing harness gate order
(resources → quota → peak → soft-defer, harness/README.md:298-312) instead of
replacing it.

Agent Zero: containerized python agent runtime. Placement: interactive
agents → their account slice (or gateway-adjacent weights), background
agents → ukrrs-batch.slice. Decide via Q15; no OS change either way, the
slice architecture absorbs it.

## 6. Sequencing and rollback
- Phase 0 DONE 2026-08-31 (apply-guest.sh: sysctl+thp+slices+noatime+timers+
  daynight+desktop; daemon.json; all verified live post-reboot).
- Phase 1 DONE 2026-08-31 (host audit parsed, §8; CPU/RAM ordering deferred
  with the window slip).
- Phase 2 SPLIT by the Oct replan: disk flags + onboot + cpuunits DONE hot
  2026-08-31; multiqueue in qm config, activates at next VM start; q35 +
  resize + root growth (#603, manual) + GPU/USB3 + 20 vCPU/128G → ~Oct 2026
  (#601/#606).
- Phase 3 (repo tickets, independent): builder caps + cgroup_parent in
  compose (#608), resources.go PSI extension (#604), gateway pressure input,
  #585 cop removal.
- Phase 4: k8s join + proxmox-ctl wiring (#605, blocked by #601).
Every phase independently revertible: sysctl.d and units are files, qm changes
are per-flag, partition work has dumps. Never a bare `down`, per house rules.

## 7. Open items
See questions-v1.md, answer inline; version the file if you want another round.

## 8. Host audit results (2026-08-31, parsed from host-audit.out)

### Confirmed
- Dell Precision T7500 (baseboard 06FW8P A02), dual socket, **CPU2 EMPTY**.
  One E5620 4C/8T (HT on) serves 24 RUNNING VMs (~57G/96G RAM allocated);
  host load was 7.5 on 8 threads while mostly idle. Your VM gets all 8
  threads only because the rest are quiet. The 2× X5675 upgrade is now
  urgent, not optional: 3× threads. DMI quirk: "Upgrade: Socket LGA771" is a
  Dell string error (E5620 exists only in LGA1366); still verify visually
  before ordering.
- RAM: 6× 16GB DDR3-1600 filling ALL six CPU1 slots. Second CPU unlocks six
  more: order 6× 16GB more → 192G. Runs at CPU max (1066 on E5620, 1333 on
  X5675).
- IOMMU active with CLEAN per-device groups: Quadro alone (20), Renesas
  USB3 alone (5), NVMe alone (21), NIC ports separate (3/4). **No ACS
  override needed** (better than section 3.6 assumed).
- No CPU frequency driver exists on this host (no /sys cpufreq at all, this
  kernel + 5520). There is no governor to tune anywhere, host or guest.
  Day/night dynamics are cgroup-only, exactly as staged in section 5.5.
- Host storage: PVE root on 2TB Hitachi spinner via LVM-thin (local-lvm,
  where the 24 sectestbed VMs live); SMART PASSED on everything; temps fine.

### Corrected
- Your "NVMe" = **Crucial P1 500G QLC (DRAM-less)** as qcow2 on ext4 dir
  storage (NVME:), dedicated to this VM, 59% used at storage level. /data1 =
  Samsung 860 PRO 256G, /data2 = SK hynix SC300B 512G, both dedicated, both
  on ICH10R SATA2 3Gbps links (~280MB/s ceiling each). None carry ssd=1 or
  discard (that is why the guest sees ROTA=1). Expect P1 sustained-write
  collapse under heavy builder churn (QLC, no DRAM): monitor; if it bites,
  the fix is a better NVMe, not config.
- Old GPU = **Quadro 4000 (Fermi, 2GB)**. Verdict: skip passthrough. Fermi
  has no NVENC (starts with Kepler) and no driver that builds on kernel 6.x
  (390xx is dead): it buys display outputs only. Keep it as host console.
  Sunshine stays software-encode until the new GPUs arrive; their clean
  IOMMU groups are already waiting.
- Only **ONE** Renesas uPD720201 USB3 card visible (24:00.0). You said two.
  Check the second: unseated, dead, or behind the (empty) Pericom PCI-X
  bridge at 01:00.0.
- VM 5111 had **no onboot flag** and ide2 still held the debian13.iso.
  [FIXED 2026-08-31: onboot=1, startup order=10, ide2 deleted.]
- k8s on this host consolidated (2026-08-31): VM 500 pfv-k8s-wnode-tsys5 was
  removed; ultix-streaming itself becomes the k8s worker on pfv-tsys5
  (12G RAM + 2 vCPU of host pressure freed; timing/control plane per Q11).
- The 18-VM sectestbed fleet runs 24/7 on the same 8 threads and the 2TB
  spinner. proxmox-ctl.sh from this VM is the natural on/off switch (Q12).

### QM checklist (APPLIED 2026-08-31, every line except the two deferred notes)
```
qm set 5111 --onboot 1 --startup order=10,up=180
qm set 5111 -scsi0 NVME:5111/vm-5111-disk-0.qcow2,iothread=1,ssd=1,discard=on,size=438G
qm set 5111 -scsi1 ssd2:5111/vm-5111-disk-0.qcow2,iothread=1,ssd=1,discard=on,size=400G
qm set 5111 -scsi2 SSD:5111/vm-5111-disk-0.qcow2,iothread=1,ssd=1,discard=on,size=200G
qm set 5111 --net0 virtio=BC:24:11:1A:8F:6F,bridge=vmbr0,multiqueue=4
qm set 5111 --net1 virtio=BC:24:11:E3:32:D9,bridge=datanet,multiqueue=2
qm set 5111 --delete ide2
# q35 + 20 vCPU + 128G + sockets/threads reshape: only at the CPU/RAM window (3.4)
# root growth to 500G: MANUAL only, GROW-ROOT-RUNBOOK.md (discard=on live first; fstrim reclaims qcow2)
# net multiqueue needs a guest-side ethtool -L oneshot after reboot
```

## Appendix A: staged file map
```
staged/60-ukrrs-vm.conf          → /etc/sysctl.d/ (kernel)
staged/modules-load.d/tcp_bbr.conf → /etc/modules-load.d/
staged/systemd/*.slice|*.service|*.timer → /etc/systemd/system/
staged/ukrrs-psi-textfile.sh     → /usr/local/sbin/ (with .service/.timer)
staged/ukrrs-daynight.sh         → /usr/local/sbin/ + day/night profile timers
                                  + /etc/ukrrs/daynight.conf (all staged values)
staged/mkacct.sh                 → run per account: name uid [high] [max] [cpus]
staged/apply-guest.sh            → installer, RUN=1 to mutate, subcommands
staged/docker/daemon.json        → /etc/docker/ (restarts docker once)
staged/proxmox-ctl.sh            → PVE API wrapper, PROTECTED guard
```

## Appendix B: repo patches (ticketed, not applied from here)
1. compose builder caps (each repo with a builder/check svc):
```
x-ukrrs-batch: &ukrrs-batch
  cgroup_parent: ukrrs-batch.slice
  cpus: 4
  mem_limit: 4g
services:
  check: { <<: *ukrrs-batch, ...existing... }
```
2. dev.sh one-shot builders: `docker run --rm --cgroup-parent ukrrs-batch.slice
   --cpus 4 --memory 4g ...` (GOMAXPROCS then follows cpuset/cpus).
3. resources.go: generalize PSI read to cpu/mem + optional slice pressure
   (fields + gates + tests, mirrors readIODelay at resources.go:90).
4. gateway pressure poller: in the 30s tick (gateway.go:802), GET
   127.0.0.1:9100/metrics, parse node_pressure_cpu_waiting_seconds... or the
   textfile gauges; feed the ladder as a new host-pressure input class.
5. PVE token:
```
pveum user add ukrrs-infra@pam
pveum user token add ukrrs-infra@pam harness -privsep 0 -expire 0
pveum acl modify /pool/<preprod-pool> -user ukrrs-infra@pam -role PVEVMUser
```
6. GPU vBIOS dump (on host, card must be unused):
```
cd /sys/bus/pci/devices/0000:0X:00.0; echo 1 > remove
cat rom > /root/gpu-<model>.rom; echo 0 > remove
```
7. node_exporter service (into harness deploy compose, digest pin at adoption):
```
node-exporter:
  image: prom/node-exporter@sha256:<pin>
  container_name: ukrrs-mopac-nodeexporter
  command: [--path.rootfs=/host, --collector.pressure,
            --collector.textfile.directory=/textfile]
  pid: host
  volumes: [/:/host:ro,rslave, /var/lib/node_exporter/textfile:/textfile:ro]
  network_mode: host
  restart: unless-stopped
```