KNELPerf v2 wave 1: Ultix scheduler breakout + loop skeleton + reports (#826)

- scheduler/: day/night engine (conf-driven redesign), psi/perfsnap collectors,
  proxmox-ctl, systemd timer templates — ported from ~/projects/ultix per the
  breakout map in scheduler/README.md
- loop/: perf-loop driver (baseline/audit/tweak/rebaseline), cpu-bench (new CPU
  leg), baseline-diff (tolerance compare) — smoke-tested locally
- docs/ARCHITECTURE.md: full program design (loop, resource groups, VM/spindle
  balancing, k8s-vs-Slurm, beszel+RAPL/iDRAC telemetry spine, workload classes,
  solar/HA + two-site power economics)
- docs/report-amt-power-telemetry.md + docs/report-moonlight-desktop.md

Redmine: https://projects.knownelement.com/issues/826#note-2

docs/report-moonlight-desktop.md

@@ -0,0 +1,68 @@
+# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
+
+Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826.
+Source research: web-surveyed 2026-09-06 (sources linked at the end).
+
+## Context constraints (verified)
+
+- Xeon E5-2630 v2 (pfv-tsys7 class) has **no iGPU / no Quick Sync** — hardware
+  video encode is impossible on the current CPU-only nodes. Software x264 (or a
+  future NVIDIA node's NVENC) is the only path.
+- Current access path is xrdp/KDE over Tailscale at ~36 ms RTT. Protocol choice
+  matters less than encoder cost and whether the Tailscale path is direct vs
+  DERP-relayed (DERP caps ~5 Mbps and adds latency — check `tailscale netcheck`
+  before blaming the desktop stack).
+- The cluster is CPU-only today, mixed with production workloads, flux gitops —
+  heavy sustained CPU encoding on shared nodes is an operational risk.
+
+## Comparison
+
+| | Sunshine+Moonlight | Selkies-GStreamer | Kasm Workspaces | Webtop / Guacamole | Tuned xrdp (baseline) |
+|---|---|---|---|---|---|
+| Transport | Moonlight protocol, H.264/265/AV1 | WebRTC (browser) | KasmVNC over WebSocket | VNC/RDP → WebSocket | RDP |
+| GPU needed | No (x264 soft mode) but costly | No (x264 soft) | No | No | No |
+| CPU cost on old Xeon | HIGH: 2–4 cores sustained @1080p60 | HIGH + WebRTC stack ~1 core | MODERATE (framebuffer diff, no video encode) | LOW–MODERATE (Guacamole ~15–27%/core per 1–2 users) | LOW |
+| Latency @36 ms RTT | Best-in-class on direct path; 100–200 ms if misconfigured | Low | Moderate; fine for desktop, visible on video | Moderate–high (protocol translation + browser) | Moderate (tuned RDP is decent) |
+| Client | Native Moonlight apps (excellent) — NOT browser | Any browser | Any browser | Any browser | Any RDP client |
+| k8s fit | Awkward: privileged pod + `/dev/uinput` hostDevice, dummy X | Good (purpose-built for k8s) | Heavy control plane for one desktop | Trivial pod + PVC (webtop); Guacamole = small extra stack | Stays a VM |
+| Persistence | StatefulSet + PVC | PVC | Disposable by design | webtop PVC-backed | Full VM (best) |
+| GitOps friendliness | Medium | High | Medium | High | N/A |
+
+## Recommendation
+
+1. **Now:** keep the tuned xrdp/KDE VM as baseline; do NOT put Sunshine/Moonlight
+   on CPU-only nodes — real-time x264 eats 2–4 cores next to production tenants.
+2. **Migration path:** deploy **linuxserver/webtop (KasmVNC variant)** as a
+   PVC-backed StatefulSet on a CPU-only node behind the existing ingress
+   (browser access; WAN via Tailscale). This is the containerized successor for
+   desktop-class use at 36 ms RTT: cheapest CPU, unprivileged pod, clean
+   flux fit.
+3. **WAN/browser fallback for the existing RDP VM:** small Apache Guacamole
+   stack (guacd + guacamole), accepting it is slower than native RDP clients.
+4. **When the NVIDIA node lands:** revisit **Selkies-GStreamer** (most
+   k8s-native low-latency option; NVENC removes the encoder cost) as the
+   premium tier. **Sunshine+Moonlight** earns its complexity (privileged pod,
+   /dev/uinput) only if gaming-grade latency on native clients becomes a hard
+   requirement — then pin it to the GPU node only, never shared CPU nodes.
+
+Rollout: tuned xrdp VM stays during migration → webtop StatefulSet UAT →
+retire VM after human UAT → Selkies on the future GPU node.
+
+## Sources
+
+- Intel Xeon Quick Sync support: https://www.intel.com/content/www/us/en/support/articles/000034104/processors/intel-xeon-processors.html
+- Sunshine advanced usage (encoder cost): https://docs.lizardbyte.dev/projects/sunshine/v0.23.0/about/advanced_usage.html
+- Moonlight setup guide: https://github.com/moonlight-stream/moonlight-docs/wiki/Setup-Guide
+- Tailscale DERP 5 Mbps gotcha: https://cfreeman.cloud/breaking-the-5-mbps-barrier-streaming-moonlight-over-tailscale-with-full-bandwidth/
+- LizardByte latency discussion: https://github.com/orgs/LizardByte/discussions/17
+- LXC headless Sunshine/uinput: https://discuss.linuxcontainers.org/t/headless-wayland-container-streaming-via-sunshine-sway-libinput-not-finding-uinput-devices/18852
+- steam-headless uinput issue: https://github.com/Steam-Headless/docker-steam-headless/issues/191
+- Selkies-GStreamer: https://github.com/selkies-project/selkies
+- Kasm desktop images: https://hub.docker.com/r/kasmweb/desktop
+- linuxserver/docker-webtop: https://github.com/linuxserver/docker-webtop
+- LSIO docker-kasm: https://docs.linuxserver.io/images/docker-kasm/
+- Guacamole web-client slowness: https://stackoverflow.com/questions/77224933/apache-guacamole-much-slower-on-the-web-client-than-microsoft-desktop
+- guacd CPU benchmarks: https://lists.apache.org/thread/zdlsnn2br4gvtx09q6zqtxym6dwlrjwc
+- Cendio on browser remote desktop limits: https://www.cendio.com/blog/apache-guacamole-alternatives/
+
+> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328