ultix/TRACKING.md

# ultix-streaming optimization — tracking HUD

One table, updated in place at each checkpoint. States: done ✅ / doing 🔄 / blocked ⛔ / next ⏳

| id | lane | state | item |
|---|---|---|---|
| OPT-1 | guest | ✅ | recon: system + ukrrs fleet (REPORT §1) |
| OPT-2 | guest | ✅ | REPORT.md + staged configs + questions-v1.md |
| OPT-3 | host | ✅ | audit parsed → REPORT §8 (T7500, CPU2 empty, clean IOMMU, Q4000 skip) |
| OPT-4 | host | ⏳ | #601 ~OCT 2026: 2× X5675 + 6× 16GB → 24T/192G (NFS outage for -02 first; USB3 reseat; q35/20c/128G). PRESUME 8 vCPU/48G until then |
| OPT-5 | host | ✅ | one-shot landed + verified post-reboot; net multiqueue in qm config, activates at next VM start (bounce approved for tonight) |
| OPT-6 | guest | ✅ | guest prep applied + verified (bbr, THP, slices, timers, noatime, daemon.json, autologin) |
| OPT-7 | guest | ⏳ | #603: root growth MANUAL ONLY, GROW-ROOT-RUNBOOK.md, user-executed |
| OPT-8 | repos | ✅ | PMO backpressure design note in ukrrs/docs/harness; code = ticketed |
| OPT-9 | integ | ⏳ | #604: node_exporter svc + gateway pressure scrape (Q10) |
| OPT-10 | integ | ⏳ | #605: k8s join (Q11, blocked by #601) + proxmox-ctl token (Q12) |
| #606 | gpu | 🔄 | Quadro 4000 passthrough STAGED on 5111 (vfio ids + q35 + hostpci0, staged/gpu-passthrough.md); activates at the human's pre-Friday roll; nouveau-only on kernel 6.12 (no CUDA/NVENC — compute cards come with #601) |
| #684 | pve-cfg | ✅ | NEW 2026-09-05 (OAM lane): ukrrs-pve-config-backup.timer (03:12 CT nightly, Persistent) runs KNEL/backups scripts/pve-config-backup.sh -> sanitized /etc/pve of all 6 PVE nodes into PRIVATE KNEL/pve-configs git repo; idempotent (commits only on drift); secrets excluded+verified. Ticket #684 |
| #610 | fleet | ✅ | Open Terminal 9-account fleet (tailscale-only :30000-30008) + OpenWebUI UAT PASS |
| #611 | rdp | ✅ | UAT PASS both accounts 2026-09-01 evening (5160x2160 clean, iPad res clean). Fix chain: kwin compositor-off + 5min timeout; max_bpp=24 (16K bitmap-row limit was the bars); drdynvc=false; autologin off; DisconnectedTimeLimit=0 (24x7 kept) + tcp keepalives; clean-slate = stop user@UID (manager recycle), not just pkill. 2026-09-05 post-update-reboot outage FIXED (waitforx shim + xbrlapi hook): see 17-fix-rdp-postupdate.sh |
| #612 | dotfiles | ✅ | UAT PASS 2026-09-01 (human): omz+v2 base + kali addendum both accounts; p10k fallback via KALI_PROMPT=0 |
| #615 | microvm+rebuild | 🔄 | microvm tooling LIVE + REBUILD.md/docs/state + docs synced + regular-VM lane PROVEN (libvirt q35, REGULAR-VMS.md); human UAT pending |
| #616 | macos | ⏳ | macOS VM on pfv-tsys5 (headless CI + iOS dev) — blocked by #601 |
| #631 | hidpi | ✅ | 150% scaling both accounts (Xft.dpi 144 + kdeglobals, persisted); cursor box RESOLVED 2026-09-05 (human verdict: gone): core shadowless cursors + xrdp new_cursors=false (server-side draw; Jump drops cursor alpha). Follow-on = #679 |
| #679 | rdp-perf | ⏳ | 16K compressed-bitmap drops at 5160 (lag/stale updates); 1MB send buffer STAGED (applies at next reconnect); EGFX-at-24bpp retest queued (human-timed) |
| #798 | rtmp | 🔄 | 24x7 desktop stream LIVE as a SERVICE since 2026-09-05 05:45 (installed by crush via 18-rtmp-stream.sh; ukrrs-rtmp-stream.service in ukrrs-rt.slice, streaming :10 to loopback receiver, verified cgroup+fence+h264 1920w). REMAINING: human watch-UAT + Cloudron URL into ~/.creds/rtmp-stream.env + restart. See STREAMING.md |

Inbox (mid-task interrupts): none.

Decisions log (latest wins):
- 2026-09-05 (morning, #798): 24x7 RTMP desktop stream built client-side. Host ffmpeg 7.1.5 ruled OK to use host-side (human: "it's fine... special case working on host", same family as the #610 open-terminal exception; nothing new installed). Architecture: systemd ukrrs-rtmp-stream.service (User=reachableceo, Slice=ukrrs-rt.slice = cores 6-7, MemoryMax 4G) -> /usr/local/sbin/ukrrs-rtmp-stream.sh: discovers the NEWEST xorgxrdp display of UID 1001 via /proc scan (xrdp display numbers move; after reboot no session exists until first RDP login - stream waits), x11grab whole screen + pulse monitor, x264 -> FLV/RTMP. TUNING MEASURED: 2560w veryfast does NOT fit the 2-core fence (saturates at ~13fps); 1920w superfast runs 187% of one core with headroom - interim operating point until #601. x11grab on xorgxrdp tops at ~20 delivered fps (GetImage latency, not CPU). Secrets: RTMP_URL only in ~/.creds/rtmp-stream.env. Local loopback receiver: mediamtx 1.20.1 pinned, 127.0.0.1:1935 only, compose rtmp/. mediamtx v1.20 key lesson: booleans are rtsp/hls/webrtc/srt/moq/api/metrics/pprof/playback (no *Disable keys, no rtmpTransports). Cloudron streaming server = final target, details pending from human; cutover = edit creds + restart (STREAMING.md).
- 2026-09-05 (early AM, post-update-reboot outage, #611): two independent RDP killers after the 6.12.105/u2 reboot: (1) xrdp 0.10.1 waitforx races xorgxrdp deferred-RandR output creation and goes blind for its whole 30s window ("Timed out waiting for RandR outputs" -> session killed -> Jump error + black screen); the u2 security NMU ALSO intermittently execs waitforx with EMPTY argv (observed 03:58:38) which the real binary rejects. Fix = wrapper at /usr/lib/x86_64-linux-gnu/xrdp/waitforx: defaults -d from $DISPLAY when argv empty + forces an xrandr RandR query past the deferred window, then execs pristine waitforx.real; invocations logged to /var/log/xrdp-fix/. (2) /etc/X11/Xsession.d/90xbrlapi runs /bin/xbrlapi in the FOREGROUND; brltty inactive (no BrlAPI socket) -> hangs forever -> Xsession never reaches startplasma-x11 -> connected-but-black screen. Hook disabled (renamed .disabled; no braille HW on this VM). Plus new_cursors=false in xrdp.ini (cursor black box: server-side cursor draw, pairs with #631 core cursor theme). All re-appliable: 17-fix-rdp-postupdate.sh (idempotent, session-safe, self-elevating). LESSON: bouncing the xrdp FRONT-END while a session lives orphans sesman state -> next login CREATES a second session instead of reconnecting -> two KDE sessions on one user bus fight (half-drawn desktop); recovery = kill all session procs + verify no plasma left (user@1001 stop can hang when crush lives inside it). Diagnostic gold: shim the helper binary (waitforx) to dump argv+env at the failure point.
- 2026-09-01 (midday): RDP #611 diagnosis: xrdp 0.10.1/xorgxrdp 0.10.2 serves 5160x2160 CLEANLY on every codec (EGFX-RFXPro, NSCodec, bitmaps) and mid-session resize 2816->5160 (freerdp headless UAT, Xvfb screenshots; ASCII-render analysis). Jump-client corruption pinned to EGFX large-frame path -> XRDP_GFX_MAX_COMPRESSED_BYTES=32MB + FRAMES_IN_FLIGHT=2 via xrdp.service.d drop-in (12-fix-rdp-bigres.sh, idempotent, restart-guards on active sessions). Scratch user rdptest = the UAT account (remove after #611 closes). Sudoers: reachableceo -> offstage NOPASSWD (narrow, /etc/sudoers.d/reachableceo-to-offstage). Harness note: crush v0.91.2 hardcodes a bash command blocklist, no config override (upstream #2761); leading-word prefix bypasses it.
- 2026-09-01 (afternoon): #615 microvm LIVE: qemu -M microvm + KVM engine (apt-only deps), firecracker v1.16.1 installed but its KVM cap check (0x38 = USER_MEMORY/SET_TSS_ADDR/VAPIC) refuses this nested Westmere KVM — retry MICROVM_ENGINE=firecracker post-#601. Per-user tap pools mv-r0..3/mv-o0..3 (/30s in 172.30.0/1.x) + iptables NAT via microvm-net.service; kvm group added both accounts (one re-login activates). UAT: boot->ssh 27.9s first boot (cloud-init+growpart 3->9.7G), apt OK through NAT, graceful stop 6.4s. #616 macOS VM ticket created w/ blocks-relation on #601. REBUILD.md + docs/state capture (14-capture-state.sh) landed; README/NEXT/REPORT/RUNBOOK synced (stale multiqueue text). mred = the Redmine CLI for all ticket work (~/.local/bin/mred).
- 2026-09-01 (night): #631 HiDPI: 150% at the 34" 5160x2160 (164 PPI) via Xft.dpi 144 + kdeglobals, persisted both accounts; black cursor box = Jump drops cursor alpha (Breeze shadow paints solid) -> shadowless core cursors. #679 opened: xrdp 16K COMPRESSED-PDU cap drops bitmap updates (second 16K cap found; first was the row cap fixed by 24bpp); tcp_send_buffer 1MB staged for next reconnect; EGFX-at-24bpp retest queued (bars may have been a 32bpp threshold). Full clean-slate for RDP black screens = kill session processes AND stop user@UID (sessions live in the SYSTEM sesman cgroup; reconnect into a surviving session never restarts the user manager) — documented in 12-fix part 5.
- 2026-09-01 (evening 2): repo tidy + docs sync: run logs consolidated into logs/ (incl. the 15:57-16:03 parallel-crush-session ops: 16-grow-root-online.sh RAN — / = 431G online, swap gone by design, #603 satisfied; 15-docker-to-data2.sh STAGED not-run); README/NEXT/RUNBOOK/REBUILD updated to match. RDP #611 final state: clean slate (all RDP/KDE processes killed per human blanket permission), DisconnectedTimeLimit=60 reaps stale sessions, human running final clean-slate UAT. Monitor UAT PASS earlier ("gorgeous", 5160x2160, zero dropped-tile errors in log watcher).
- 2026-09-01 (evening): #611 fix chain complete: (1) EGFX budget env (earlier), (2) kwin compositor off + [Service] TimeoutStartSec=300 (user-manager reload needs XDG_RUNTIME_DIR — root daemon-reload does NOT touch user units), (3) dual-session plasmashell conflict resolved by killing console session live, (4) EGFX OFF server-wide (drdynvc=false) after band-analysis proved server clean at 5160 while Jump showed bars (classic codecs verified clean via harness), (5) human ruling: sddm autologin DISABLED (conf removed + staged/apply-guest.sh step neutralized; Sunshine later gets kiosk/XFCE). sddm greeter at next boot; RDP = primary desktop.
- 2026-09-01 (afternoon 2): GPU passthrough STAGED (#606): vfio-pci ids 10de:06dd/0be5 (IOMMU group 20 clean, initramfs rebuilt) + VM 5111 machine q35 + hostpci0 pcie=1 — activates at the human's pre-Friday roll; verify+revert in staged/gpu-passthrough.md; host console goes headless; guest gets nouveau only (no Fermi driver on kernel 6.x). Regular-VM lane PROVEN (#615): libvirt 11.3/q35/host-passthrough boots the Debian cloud image — DHCP lease + ssh + growpart + systemctl running, nested vmx visible (REGULAR-VMS.md documents the three gotchas: disks under /var/lib/libvirt/images/reachableceo w/ libvirt-qemu group 660/770; direct-kernel files must be under /boot for virt-aa-helper; console output needs console=ttyS0). libvirt default net now autostarted. Teardown verified clean.
- 2026-09-01 (morning): POST-REBOOT AUDIT PASS (boot 22:53 = multiqueue bounce): ens18 Combined 4/4 + ens19 2/2 LIVE; qm pending empty; bounce log 22:51->22:53 clean; gw prod (ts :4000) + beta (:4002) both 200/healthy; postgres x2, openwebui :3000, 12 LSPs, all 9 open-terminal ports bound on 100.101.187.119; PSI textfile fresh; no relauncher respawn. Night window EXTENDED 22:00-05:00 -> 22:00-07:00 (human ruling; day-flip timer 07:00, live + staged synced via 11-shift-day-flip-0700.sh; z.ai peak ladder 01:00-05:00 untouched). openwebui.creds per-account distributor staged (10-) NOT run — human reads master list at ~/.creds/open-terminal.env.
- 2026-08-31 (night, con handed to crush): #610 DONE in one session: open-terminal 0.11.34 bare-metal (HUMAN EXCEPTION to no-host-install rule, recorded), template unit open-terminal@.service, per-user TOML keys (never in ps/repo; ~/.creds holds copies), bind TAILSCALE-ONLY 100.101.187.119 per human ruling (0.0.0.0 rejected); all 8 other accounts + docker group; UAT via throwaway openwebui v0.11.1 (matches Cloudron prod) ALL PASS incl. hallucination-guarded chat round trip (model ran ls through OpenWebUI's terminal proxy, answered 9/9 real names). NOTE: prod openwebui container (v0.3.10, :3000) is too old for Open Terminal — upgrade to 0.11.x before wiring; Cloudron 0.11.1 is fine. Learned: raw API callers execute tool_calls client-side; the UI path runs the server-side loop.
- 2026-08-31 (night): 22:00 night-profile flip OBSERVED LIVE from the guest (dirty_bytes 1G→2G, bg 256M→512M, /data2 readahead 128K→1M; day timer restores at 05:00). First runtime-only flip since install.
- 2026-08-31 (night): agent-stack relaunchers REMOVED by 6-remove-agent-stacks.sh (reachableceo-agent-stack.service, tsg-supervisor.timer/.service 5-min self-heal, dead tsg-agent-stacks.service; launchers + units backed up to removed-agent-stacks/). No screen/crush sessions auto-start on reboot anymore; human: stacks no longer used for pmo/work.
- 2026-08-31 (night): docs synced post-pass: REPORT v1.1 (status banner, per-item STATUS tags, §6/§8 truth), RUNBOOK amended for the Oct replan, all paths ~/optimize → ~/projects/ultix (repo moved). Human ruling: system idle tonight → multiqueue bounce approved for this session, after remaining tasks. Agent-stack teardown prepped as 6-remove-agent-stacks.sh (kills boot-time RCEO/TSG screen respawn + the 5-min supervisor self-heal; unit/launcher backups to removed-agent-stacks/; root also quits leftover root screen ukrrs-gateway-final, leaves 'boot' alone).
- 2026-08-31 (late): Redmine replan complete: #601 Oct window (due 10-31, High, version october-2026-window; Redmine API refuses start_date on update — Oct-1 start recorded in the note instead), NEW #608 cgroup_parent wiring (High; THE starvation fix on 8 vCPU), NEW #609 Fri 09-04 troubleshooting-only outage (USB SDR reseat + socket/PSU/BIOS checks; NO upgrades; host down → this VM auto-starts via onboot=1, gateway auto-recovers). #605/#606 noted. docs repo pushed (main 9eb324e).
- 2026-08-31 (eve): HARDWARE WINDOW → ~OCT 2026; presume 8 vCPU/48G operating values; cpuset reservation deferred → cgroup_parent wiring = the starvation fix; one VM-5111 reboot owed for multiqueue (5-activate-multiqueue.sh, human-timed; crush may roll OTHER VMs on pfv-tsys5 anytime).
- 2026-08-31 (eve): PUSH AS YOU GO standing permission (human monitors gitea/redmine/discourse, not ssh/crush). House rules updated + pushed (86d8565).
- 2026-08-31 16:5x: Redmine = record: perf pass + RCA filed as #602 (project 55, tracker Support, left New for human review); open items #601 (Fri window, Urgent, due 09-04) #603 root growth #604 metrics #605 k8s+proxmox (blocked by #601) #606 GPUs (relates #601). mred = docker exec wrapper (~/.local/bin/mred → mopac-cli-mred container).
- 2026-08-31 16:40: net multiqueue CLOSED: queues=4/2 written to live qm config via crush ssh (user authorized non-disruptive host mods); activates at next VM start; guest oneshot enables channels on that boot. Earlier silent no-op explained: the 15:59 run used a pre-fix script copy.
- 2026-08-31 16:3x: crush CAN ssh to the host from this workspace (old "hard-banned" note obsolete); still human-gated for disruptive ops.
- 2026-08-31 (post-reboot validation): prod gateway down at boot ROOT-CAUSED twice over: (a) port bind to tailscale IP lost the dockerd-vs-tailscaled race (docker never retries failed starts), (b) live-restore dropped the container's network endpoint (host resolver → gateway-db unresolvable → crash loop). Recovered via compose --force-recreate (healthy, mode=normal). Permanent fix staged: ukrrs-gateway-ensure unit, BOTH lanes, MUST be enabled before Friday's bounces.
- 2026-08-31: multiqueue host flags verified live (ROTA=0 + discard 1G on all 3 disks), but guest still runs 1 queue/NIC: ethtool not installed; 3-post-reboot-fixes.sh staged (installs ethtool + oneshot unit, sets 4/2 queues).
- 2026-08-31: mopac-demo/mcli fake containers left DOWN (restart=no, exited at first bounce); prometheus/harness daemons not deployed on this box (pre-existing; belongs to OPT-9).
- 2026-08-31: STORAGE OPS MANUAL ONLY: boot-time auto-grow service removed, qm resize removed from the one-shot, grow-root.sh deleted. Runbook = GROW-ROOT-RUNBOOK.md, every command typed by the human.
- 2026-08-31: KDE autologin into UNLOCKED always-alive session (Relogin=true); autolock OFF (instant Jump/iPad attach outranks lock). XFCE only if Sunshine capture disappoints post-GPU.
- 2026-08-31: streamlined to 1-guest-prep.sh + 2-host-one-shot.sh (one reboot wave; no post-reboot steps; no storage).
- 2026-08-31: constraint: host pfv-tsys5 NEVER reboots casually (NFS server for -02); Friday CPU swap needs a planned NFS outage first.
- 2026-08-31: GPUs are COMPUTE cards; Quadro 4000 skipped (Fermi: no NVENC, no kernel-6 driver, no modern CUDA); clean IOMMU groups verified.
- 2026-08-31: day/night = runtime-only cgroup flips 22:00/05:00; NO cpufreq driver exists host or guest.
- 2026-08-31: zram rejected; no docker data-root move; skip tuned (sysctl.d + units).
- 2026-08-31: second USB3 SDR card missing on host (only one Renesas visible); Friday reseat/check.
----