KNELPerf v2 wave 1: Ultix scheduler breakout + loop skeleton + reports (#826)

- scheduler/: day/night engine (conf-driven redesign), psi/perfsnap collectors, proxmox-ctl, systemd timer templates — ported from ~/projects/ultix per the breakout map in scheduler/README.md - loop/: perf-loop driver (baseline/audit/tweak/rebaseline), cpu-bench (new CPU leg), baseline-diff (tolerance compare) — smoke-tested locally - docs/ARCHITECTURE.md: full program design (loop, resource groups, VM/spindle balancing, k8s-vs-Slurm, beszel+RAPL/iDRAC telemetry spine, workload classes, solar/HA + two-site power economics) - docs/report-amt-power-telemetry.md + docs/report-moonlight-desktop.md Redmine: https://projects.knownelement.com/issues/826#note-2
2026-09-06 14:16:54 -05:00
parent 89a93359f0
commit 25299373f7
19 changed files with 839 additions and 1 deletions
@@ -1 +1 @@
-#709
+#826
@@ -0,0 +1 @@
+data/baselines/*/
@@ -0,0 +1,177 @@
+# KNELPerf v2 — architecture for dynamic, intelligent performance optimization
+
+Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end
+baseline → audit/benchmark → tweak → re-baseline loop, two-week program.
+
+## 1. The loop
+
+```
+        ┌──────────────────────────────────────────────────┐
+        │                                                  │
+   BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
+   (capture)     (diff + beszel/       (CR-gated,   (compare vs
+                 PSI/rapl context)     idempotent)  baseline; keep
+                                                     or roll back)
+```
+
+- **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
+  (cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines/<host>/<ts>/`.
+  Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg
+  (`loop/cpu-bench.sh`) is new — the repo previously had zero CPU coverage.
+- **audit**: `loop/perf-loop.sh audit <host>` re-runs the read-only probes and
+  diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based,
+  catches governor flips exactly). This is what the scheduled loop runs.
+- **tweak**: every mutation is a file under `tweaks/<id>.sh` (idempotent,
+  rollback helper included). Production tweaks ride a GLPI CR; the loop
+  refuses `tweak` on prod hosts without `.crush/active-cr`.
+- **re-baseline**: same capture, then `compare` — the delta table is the
+  evidence artifact that lands on the Redmine ticket.
+
+Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
+the audit itself is low-cost); re-baseline only after a tweak or hardware
+change. First full-fleet baseline wave is this week's deliverable.
+
+## 2. Scheduling architecture (Ultix breakout redesign)
+
+See `scheduler/README.md` for the port map and the lessons-learned list. Core
+redesign decisions:
+
+1. **Single source of truth per host** — `/etc/knelperf/daynight.conf`; no
+   value ever lives in a timer file, unit default, or script constant.
+2. **Reboot lands in day mode** — invariant carried from Ultix.
+3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so
+   PVE hosts tune `qemu.slice` children and host services with the same engine
+   the workstation uses.
+4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
+   payload; job templates own the rollout waves.
+
+## 3. Proxmox resource groups & VM balancing
+
+Current fleet (dmidecode-verified 2026-09-06):
+
+| node | platform | cores | RAM | storage character |
+|---|---|---|---|---|
+| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role |
+| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton |
+| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm |
+| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) |
+| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner |
+| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner |
+| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge |
+| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe |
+
+Directions:
+- **Proxmox resource groups / pools**: map pools to tenant classes (see §6),
+  so scheduler policy attaches to the pool, not per-VM.
+- **Balancing across hosts AND spindles**: nodes are standalone (no
+  corosync/HA), so "balancing" = placement policy at create/migrate time, not
+  live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6;
+  RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8,
+  tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with
+  one-job-per-spindle placement (fio baselines per spindle will quantify).
+  The audit loop's beszel/PSI data feeds a placement scorecard.
+- **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G —
+  largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.).
+  Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and
+  the core count to soak k8s bins; tsys6's older CPUs are fine for
+  steady-state VMs but it lacks DCMI telemetry on old firmware. Decision
+  needs founder ruling — will be proposed on #826 with a capacity table once
+  tsys8/2 land.
+
+## 4. Where k8s scheduling fits; Slurm?
+
+- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
+  (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
+  class 1–2 workloads; node labels/taints pin tenant classes to node pools.
+- **HFNOC batch (class 3)** = interruptible, deadline-loose, large — this is
+  HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics
+  (preemption via priority classes — fits "start and stop easily"); (b) Slurm
+  on a dedicated partition across idle capacity. Recommendation: start with
+  k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only
+  introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm
+  co-located with k8s on shared nodes via cgroup slices is possible
+  (knelperf-batch.slice) but doubles scheduling systems for little gain at
+  this fleet size.
+- The knelperf daynight engine gives the batch window (22:00–07:00) its
+  expanded resource envelope on each participating node.
+
+## 5. Telemetry spine: beszel + power
+
+- **Beszel has 100% VM+physical coverage** → it is the always-on signal for
+  the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
+  feed the placement scorecard; PSI textfile collector
+  (`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks.
+- **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts
+  come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5
+  needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the
+  un-metered UPS, everything else on the metered one) is compatible: the
+  nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.
+
+## 6. Workload priority classes (founder-set order)
+
+| class | tenants | profile | scheduling |
+|---|---|---|---|
+| 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7) |
+| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
+| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
+
+## 7. Solar + Home Assistant integration
+
+- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
+  Assistant exposes real-time production. The loop consumes: surplus watts =
+  green light to start class-3 batches (excess production is the cheapest
+  compute on earth); forecast + reservation calendar (KillBill) → promotion
+  timing advice ("run the Side Door promo the week of X: projected surplus +
+  idle capacity Y").
+- Integration path: HA REST/websocket (creds pattern exists:
+  `~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) → a small
+  exporter writing `knelperf_solar_surplus_watts` into the textfile
+  collector's directory; the batch launcher gates on it.
+- SITER-Solar math review (2026-09-06): found constant drift
+  ($301.08/23952 vs plan's $264.47/22614), a stale Dockerfile entrypoint
+  (deprecated python script), header/ROI inconsistencies, and fixed-cost
+  scenario assumptions — fixes landing in that repo.
+
+## 8. Two-site power economics (site 2: printer farm / laser / CNC)
+
+The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
+MUCH more expensive grid power than the main site. That asymmetry drives
+scheduling policy as much as raw performance:
+
+- **Marginal-price model**: every site has an effective $/kWh curve. Main
+  site = cheap grid + (soon) solar with export at $0.04 and self-use at
+  $0.085 (SITER-Solar contract). Site 2 = expensive grid. The scheduler's
+  objective becomes: run flexible load (compute AND manufacturing) where the
+  marginal $/kWh is lowest at that hour.
+- **Solar surplus is main-site-first**: excess production self-consumed at
+  the main site displaces the CHEAPEST power — class-3 batch compute and any
+  deferrable main-site load should chase the surplus curve.
+- **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and
+  often deferrable (prints especially — hours-long, unattended). Candidate
+  policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows
+  (if its tariff has TOU) or batch them for periods when main-site solar
+  surplus can't absorb more compute. Home Assistant + metered UPS legs give
+  the measured baseline; site 2 needs its own metering (smart plug / CT per
+  machine) before scheduling — queued as a data dependency.
+- **Never shift OLTP**: class-1 paid workloads stay on their reserved
+  capacity regardless of power price — revenue and latency beat energy cost.
+- KillBill reservation calendar × solar forecast × site-2 tariff = the
+  promotion/job-timing advisor output: "cheapest week to run X".
+
+## 9. Hardware plan hooks
+
+- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
+  are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
+  baseline each with fio, then one-batch-job-per-spindle).
+- tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory
+  row pending (PFVSystemInventory.csv not found in repo — founder to drop in).
+- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
+  with tsys3 for latency-sensitive class-1 storage.
+
+## 10. Organization
+
+Performance management org reports to VPTechOps, dotted line VPFacilities —
+identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
+groups). All work filed under #826; CRs for any prod change.
+
+> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
@@ -0,0 +1,84 @@
+# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
+
+Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via
+dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included).
+
+## Executive answers
+
+**Does AMT provide power usage data? Can it be pulled from the BIOS?**
+**No — no Intel AMT version (9.x through current) exposes real-time watt
+telemetry, via BIOS or any API.** Intel's own forum answer is categorical:
+AMT provides power *control* (on/off/cycle/boot-once), not power
+*measurement*. The Redfish `Power`/`PowerConsumedWatts` resource is a
+server-BMC feature (iDRAC/iLO wired to PSU metering hardware), not an AMT
+feature. So AMT is worth enabling only for remote KVM/power-control, never
+for watts.
+
+## Per-system table (hardware verified by dmidecode 2026-09-06)
+
+| System | Platform (verified) | AMT/vPro? | Watts via mgmt? | Best watt source |
+|---|---|---|---|---|
+| pfv-tsys1 | OptiPlex 9020, i7-4770 (Haswell), Q87 | YES — AMT 9.1 (Dell spec confirms vPro/iAMT 9.x) | NO | Linux RAPL: PKG + DRAM |
+| pfv-tsys3 | Precision 7510 (mobile WS), Xeon E3-1535M v5 (Skylake-H), CM236 | YES — AMT 11.x (patch post-CVE-2017-5689!) | NO | Linux RAPL: PKG (DRAM domain often absent on mobile SKL — verify) |
+| pfv-tsys4 | Precision T1700, Xeon E3-1246 v3 (Haswell), C226 | **NO AMT** — CPU is vPro-capable but C226 has no AMT ME stack (Haswell vPro needs Q87); no BMC either | NO | Linux RAPL: PKG + DRAM |
+| pfv-tsys5 | Precision T7500, Xeon E5620 (Westmere), 5520 chipset | NO — server-class chipset, pre-vPro era; no BMC | NO | **Nothing software-readable: Westmere has NO RAPL** (Sandy Bridge introduced it). Metered PDU / smart plug only |
+| pfv-tsys6 | PowerEdge R610, 2x E5530, iDRAC6 | NO AMT (servers = BMC) | **YES via iDRAC6** | `racadm getconfig -g cfgServerPower -o cfgServerActualPowerConsumption`; ipmitool sensor "System Level" (watts often x10). DCMI partial/absent on old firmware |
+| pfv-tsys7 | PowerEdge R620, 2x E5-2630 v2 (Ivy Bridge-EP), iDRAC7 | NO AMT (BMC) | **YES via iDRAC7** (full DCMI) | `ipmitool dcmi power reading`; `racadm getsysinfo -c`; plus host-side RAPL PKG+DRAM |
+| tsys9 | unknown (2013-17 OptiPlex/Precision) | Identify first: `dmidecode -t system` + `lspci \| grep -i MEI` + MEBx (Ctrl+P) | if AMT: still NO watts | RAPL if Haswell or newer |
+
+## Practical pull methods
+
+- **AMT 9.x/10.x/11.x (tsys1, tsys3):** WSMAN/SOAP via `wsman` CLI or
+  MeshCommander (port 16993 TLS) — power state, on/off, KVM, inventory. AMT
+  11.6+ also speaks Redfish at `https://<host>:16992/redfish/v1/` but with no
+  Power telemetry resource.
+- **Local MEI/HECI discovery:** `/dev/mei0`, `intelmetool` — tells you whether
+  AMT firmware is present/provisioned (use this for tsys9 triage).
+- **RAPL readers:** `turbostat` (most complete; PKG/cores/DRAM), powercap
+  sysfs `/sys/class/powercap/intel-rapl:*` (poll `energy_uj`, delta/t = W),
+  `perf stat -e power/energy-pkg/`. node_exporter has NO built-in RAPL
+  collector — deploy a small rapl exporter or scrape turbostat. Caveat: RAPL
+  measures CPU package (+DRAM), not whole-wall draw (~60-75% of wall on
+  desktops).
+
+## RAPL coverage by CPU generation
+
+- RAPL introduced Sandy Bridge (2011); **Westmere (E5620) and Nehalem (E5530)
+  have none — tsys5 and tsys6 hosts cannot RAPL.**
+- Haswell adds DRAM domain: tsys1 (i7-4770) and tsys4 (E3-1246 v3) get
+  PKG+PP0+DRAM.
+- Skylake-H (tsys3): PKG yes, DRAM typically absent on mobile.
+- Ivy Bridge-EP (tsys7): PKG+DRAM.
+
+## Recommendations
+
+1. Don't chase watts via AMT anywhere — the capability does not exist. Use
+   AMT (tsys1, tsys3) only for remote KVM/power control; patch tsys3's AMT 11
+   firmware for CVE-2017-5689 before enabling.
+2. Uniform host-side telemetry = RAPL exporter on tsys1, tsys3, tsys4, tsys7.
+3. Servers (tsys6, tsys7): pull BMC watts out-of-band (racadm/ipmitool DCMI);
+   update iDRAC6 firmware on tsys6 if DCMI commands fail.
+4. tsys5 (T7500): budget a metered PDU or smart plug — no software path
+   exists (fits the planned UPS re-cabling: put tsys5 on the metered leg).
+5. tsys9: run `dmidecode -t system` + `ls /sys/class/powercap` — that pair
+   answers both its AMT and RAPL questions; report back.
+6. UPS plan alignment: since only tsys6/7 (BMC) + RAPL hosts give per-node
+   watts, keep the metered UPS leg feeding the gear you most need wall-watts
+   for; per-node software telemetry covers the rest.
+
+## Sources
+
+- Intel Community — AMT and power usage monitoring: https://community.intel.com/t5/Intel-Business-Client-Software/AMT-and-power-usage-monitoring/td-p/853085
+- Intel Community — AMT Support Identification: https://community.intel.com/t5/Intel-vPro-Platform/Intel-AMT-Support-Identification/td-p/1275668
+- DMTF Redfish Telemetry DSP2051: https://www.dmtf.org/sites/default/files/standards/documents/DSP2051_1.1.0.pdf
+- OptiPlex 9020 spec (AMT 9.x): https://www.dell.com/support/manuals/en-us/optiplex-9020-desktop/opt9020sffom-v2/specifications
+- Precision 7510 spec sheet: https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/us/Dell-Precision-15-7000-Series-7510-Spec-Sheet.pdf
+- ARK E3-1246 v3: https://www.intel.com/content/www/us/en/products/sku/80916/intel-xeon-processor-e31246-v3-8m-cache-3-50-ghz/specifications.html
+- ServerFault Dell power CLI: https://serverfault.com/questions/736068/how-do-i-get-the-power-consumption-of-a-dell-poweredge-server-on-the-cli
+- jonamiki racadm power capping: https://jonamiki.com/2014/10/18/power-control-and-capping-on-dell-servers-using-racadm/
+- Dell InfoHub power reporting: https://infohub.delltechnologies.com/en-us/p/server-power-consumption-reporting-and-management/
+- Launchpad RAPL domains: https://bugs.launchpad.net/bugs/1467014
+- Hackenberg et al. Haswell RAPL survey: https://tu-dresden.de/zih/forschung/ressourcen/dateien/projekte/firestarter/2015_hackenberg_hppac.pdf
+- kernel mei/iamt docs: https://docs.kernel.org/driver-api/mei/iamt.html
+
+> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
@@ -0,0 +1,68 @@
+# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
+
+Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826.
+Source research: web-surveyed 2026-09-06 (sources linked at the end).
+
+## Context constraints (verified)
+
+- Xeon E5-2630 v2 (pfv-tsys7 class) has **no iGPU / no Quick Sync** — hardware
+  video encode is impossible on the current CPU-only nodes. Software x264 (or a
+  future NVIDIA node's NVENC) is the only path.
+- Current access path is xrdp/KDE over Tailscale at ~36 ms RTT. Protocol choice
+  matters less than encoder cost and whether the Tailscale path is direct vs
+  DERP-relayed (DERP caps ~5 Mbps and adds latency — check `tailscale netcheck`
+  before blaming the desktop stack).
+- The cluster is CPU-only today, mixed with production workloads, flux gitops —
+  heavy sustained CPU encoding on shared nodes is an operational risk.
+
+## Comparison
+
+| | Sunshine+Moonlight | Selkies-GStreamer | Kasm Workspaces | Webtop / Guacamole | Tuned xrdp (baseline) |
+|---|---|---|---|---|---|
+| Transport | Moonlight protocol, H.264/265/AV1 | WebRTC (browser) | KasmVNC over WebSocket | VNC/RDP → WebSocket | RDP |
+| GPU needed | No (x264 soft mode) but costly | No (x264 soft) | No | No | No |
+| CPU cost on old Xeon | HIGH: 2–4 cores sustained @1080p60 | HIGH + WebRTC stack ~1 core | MODERATE (framebuffer diff, no video encode) | LOW–MODERATE (Guacamole ~15–27%/core per 1–2 users) | LOW |
+| Latency @36 ms RTT | Best-in-class on direct path; 100–200 ms if misconfigured | Low | Moderate; fine for desktop, visible on video | Moderate–high (protocol translation + browser) | Moderate (tuned RDP is decent) |
+| Client | Native Moonlight apps (excellent) — NOT browser | Any browser | Any browser | Any browser | Any RDP client |
+| k8s fit | Awkward: privileged pod + `/dev/uinput` hostDevice, dummy X | Good (purpose-built for k8s) | Heavy control plane for one desktop | Trivial pod + PVC (webtop); Guacamole = small extra stack | Stays a VM |
+| Persistence | StatefulSet + PVC | PVC | Disposable by design | webtop PVC-backed | Full VM (best) |
+| GitOps friendliness | Medium | High | Medium | High | N/A |
+
+## Recommendation
+
+1. **Now:** keep the tuned xrdp/KDE VM as baseline; do NOT put Sunshine/Moonlight
+   on CPU-only nodes — real-time x264 eats 2–4 cores next to production tenants.
+2. **Migration path:** deploy **linuxserver/webtop (KasmVNC variant)** as a
+   PVC-backed StatefulSet on a CPU-only node behind the existing ingress
+   (browser access; WAN via Tailscale). This is the containerized successor for
+   desktop-class use at 36 ms RTT: cheapest CPU, unprivileged pod, clean
+   flux fit.
+3. **WAN/browser fallback for the existing RDP VM:** small Apache Guacamole
+   stack (guacd + guacamole), accepting it is slower than native RDP clients.
+4. **When the NVIDIA node lands:** revisit **Selkies-GStreamer** (most
+   k8s-native low-latency option; NVENC removes the encoder cost) as the
+   premium tier. **Sunshine+Moonlight** earns its complexity (privileged pod,
+   /dev/uinput) only if gaming-grade latency on native clients becomes a hard
+   requirement — then pin it to the GPU node only, never shared CPU nodes.
+
+Rollout: tuned xrdp VM stays during migration → webtop StatefulSet UAT →
+retire VM after human UAT → Selkies on the future GPU node.
+
+## Sources
+
+- Intel Xeon Quick Sync support: https://www.intel.com/content/www/us/en/support/articles/000034104/processors/intel-xeon-processors.html
+- Sunshine advanced usage (encoder cost): https://docs.lizardbyte.dev/projects/sunshine/v0.23.0/about/advanced_usage.html
+- Moonlight setup guide: https://github.com/moonlight-stream/moonlight-docs/wiki/Setup-Guide
+- Tailscale DERP 5 Mbps gotcha: https://cfreeman.cloud/breaking-the-5-mbps-barrier-streaming-moonlight-over-tailscale-with-full-bandwidth/
+- LizardByte latency discussion: https://github.com/orgs/LizardByte/discussions/17
+- LXC headless Sunshine/uinput: https://discuss.linuxcontainers.org/t/headless-wayland-container-streaming-via-sunshine-sway-libinput-not-finding-uinput-devices/18852
+- steam-headless uinput issue: https://github.com/Steam-Headless/docker-steam-headless/issues/191
+- Selkies-GStreamer: https://github.com/selkies-project/selkies
+- Kasm desktop images: https://hub.docker.com/r/kasmweb/desktop
+- linuxserver/docker-webtop: https://github.com/linuxserver/docker-webtop
+- LSIO docker-kasm: https://docs.linuxserver.io/images/docker-kasm/
+- Guacamole web-client slowness: https://stackoverflow.com/questions/77224933/apache-guacamole-much-slower-on-the-web-client-than-microsoft-desktop
+- guacd CPU benchmarks: https://lists.apache.org/thread/zdlsnn2br4gvtx09q6zqtxym6dwlrjwc
+- Cendio on browser remote desktop limits: https://www.cendio.com/blog/apache-guacamole-alternatives/
+
+> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# KNELPerf baseline diff — compares two baseline bundles (or live-audits one).
+#
+#   baseline-diff.sh <dirA> <dirB>   compare two stored bundles
+# Values are TSV metric/value/unit rows from cpu-bench.tsv. Numeric metrics
+# compare with a tolerance (default 10%); non-numeric metrics compare for
+# equality (governor flips are exactly what we want to catch).
+set -euo pipefail
+TOL_PCT=${KNELPERF_DIFF_TOL:-25}  # 1-thread openssl on a busy host swings >10% run-to-run
+[ $# = 2 ] || { echo "usage: $0 <dirA> <dirB>" >&2; exit 2; }
+A=$1; B=$2
+[ -r "$A/cpu-bench.tsv" ] || { echo "no cpu-bench.tsv in $A" >&2; exit 3; }
+[ -r "$B/cpu-bench.tsv" ] || { echo "no cpu-bench.tsv in $B" >&2; exit 3; }
+
+join_tsv() { awk -F'\t' 'NR==FNR{a[$1]=$2;next} {print $1"\t"a[$1]"\t"$2"\t"$3}' "$1" "$2"; }
+
+# shellcheck disable=SC2034  # exit-status reserved for future strictness
+status=0
+join_tsv "$A/cpu-bench.tsv" "$B/cpu-bench.tsv" | while IFS=$'\t' read -r metric va vb unit; do
+  [ -n "$va" ] && [ -n "$vb" ] || continue
+  if [ "$va" = "$vb" ]; then
+    printf 'OK    %-28s %s = %s %s\n' "$metric" "$va" "$vb" "$unit"
+  elif [ "$va" = "${va#[-+0-9.]}" ] || [ "$vb" = "${vb#[-+0-9.]}" ]; then
+    printf 'DRIFT %-28s %s -> %s %s\n' "$metric" "$va" "$vb" "$unit"
+  else
+    delta=$(awk -v a="$va" -v b="$vb" 'BEGIN{d=(b-a)/((a==0)?1:a)*100; printf "%+.1f", d}')
+    flag=$(awk -v d="${delta#+}" -v t="$TOL_PCT" 'BEGIN{m=d<0?-d:d; if(m>t) print "DELTA"; else print "OK"}')
+    if [ "$flag" = OK ]; then
+      printf 'OK    %-28s %s -> %s (%s%%) %s\n' "$metric" "$va" "$vb" "$delta" "$unit"
+    else
+      printf 'DELTA %-28s %s -> %s (%s%%) %s\n' "$metric" "$va" "$vb" "$delta" "$unit"
+    fi
+  fi
+done
+echo "tolerance: ${TOL_PCT}% (set KNELPERF_DIFF_TOL to change)"
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+# KNELPerf CPU benchmark — the missing CPU leg of the perf loop (#826).
+# Runs ON the target host (or locally). Zero-install friendly: uses openssl
+# (always present) plus sysbench when available. Output = TSV tag/value.
+#
+# Design notes:
+#   - openssl speed = single+multi core crypto throughput (stable, comparable)
+#   - sysbench cpu (if installed) = integer event rate
+#   - /proc-derived context: governor, PSI at capture time
+set -u
+DUR=${KNELPERF_CPUBENCH_SECS:-10}
+
+echo -e "metric\tvalue\tunit"
+
+gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo none)
+echo -e "governor\t$gov\t-"
+nproc=$(nproc)
+echo -e "nproc\t$nproc\t-"
+
+# openssl: 1 thread then all threads, aes-256-cbc (avx-sensitive) and sha256.
+# Parse: LAST line whose first field is the algo; value = last numeric field
+# (per-thread rows precede the aggregate row in -multi output).
+for algo in aes-256-cbc sha256; do
+  v=$(openssl speed -seconds "$DUR" "$algo" 2>/dev/null \
+    | awk -v a="$algo" '$1==a{v=$NF} END{print v}')
+  [ -n "${v:-}" ] && echo -e "openssl_${algo}_1t\t$v\tk/s"
+  # -multi: per-thread rows precede the aggregate; keep the LAST match
+  v=$(openssl speed -multi "$nproc" -seconds "$DUR" "$algo" 2>/dev/null \
+    | awk -v a="$algo" '$1==a{v=$NF} END{print v}')
+  [ -n "${v:-}" ] && echo -e "openssl_${algo}_${nproc}t\t$v\tk/s"
+done
+
+# sysbench (optional, if installed)
+if command -v sysbench >/dev/null 2>&1; then
+  ev=$(sysbench cpu --time=$((DUR*1000)) --threads=1 run 2>/dev/null \
+    | awk -F'/s|events per second:/{for(i=1;i<=NF;i++) if($i ~ /^[0-9.]+$/){print $i; exit}}')
+  [ -n "${ev:-}" ] && echo -e "sysbench_cpu_1t\t$ev\tevents/s"
+  ev=$(sysbench cpu --time=$((DUR*1000)) --threads="$nproc" run 2>/dev/null \
+    | awk -F'/s|events per second:/{for(i=1;i<=NF;i++) if($i ~ /^[0-9.]+$/){print $i; exit}}')
+  [ -n "${ev:-}" ] && echo -e "sysbench_cpu_${nproc}t\t$ev\tevents/s"
+else
+  echo -e "sysbench_cpu\tabsent\t-"
+fi
+
+# PSI snapshot at capture time (context for the numbers above)
+for res in cpu memory io; do
+  v=$(awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' \
+    "/proc/pressure/$res" 2>/dev/null || echo 0)
+  echo -e "psi_${res}_avg60\t$v\t%"
+done
+exit 0
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# KNELPerf loop driver — baseline -> audit/benchmark -> tweak -> re-baseline.
+#
+# The heart of KNELPerf v2 (#826). One host, one cycle phase at a time:
+#
+#   perf-loop.sh baseline <host>       capture baseline bundle (cpu/net/storage/psi)
+#   perf-loop.sh audit <host>          read-only audit vs baseline (alerts on drift)
+#   perf-loop.sh tweak <host> <id>     apply a tweak (from tweaks/) after CR gate
+#   perf-loop.sh rebaseline <host>     capture post-tweak bundle + diff report
+#   perf-loop.sh compare <host> <a> <b>  diff two stored bundles
+#
+# Bundles live in data/baselines/<host>/<ts>/ and are plain TSV/JSON so diff
+# and jq stay the toolchain. Nothing here mutates a host except `tweak`.
+set -euo pipefail
+REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)
+DATA_DIR=${KNELPERF_DATA_DIR:-$REPO_ROOT/data/baselines}
+SSH_OPTS=(-o ConnectTimeout=8 -o BatchMode=yes)
+
+usage() { sed -n '2,12p' "$0"; exit 2; }
+[ $# -ge 2 ] || usage
+cmd=$1; host=$2
+
+now() { date +%Y%m%d-%H%M%S; }
+host_dir() { mkdir -p "$DATA_DIR/$1"; }
+
+run_on_host() { # host script-path args...
+  local h=$1 script=$2; shift 2
+  case $h in
+    local) bash "$REPO_ROOT/$script" "$@" ;;
+    *) ssh "${SSH_OPTS[@]}" "root@$h" "bash -s" < "$REPO_ROOT/$script" "$@" ;;
+  esac
+}
+
+case "$cmd" in
+baseline|rebaseline)
+  ts=$(now); dir="$DATA_DIR/$host/$ts"; host_dir "$host"; mkdir -p "$dir"
+  echo "== KNELPerf $cmd: $host -> $dir"
+  # shellcheck disable=SC2043  # single-part list by design; grows as legs are added
+  for part in cpu-bench; do
+    run_on_host "$host" "loop/$part.sh" > "$dir/$part.tsv" 2> "$dir/$part.err" \
+      || echo "WARN: $part failed (see $dir/$part.err)"
+  done
+  # storage + network reuse the existing bench harness outputs
+  run_on_host "$host" "bench/bench-run.sh" > "$dir/bench-run.tsv" 2> "$dir/bench.err" \
+    || echo "WARN: bench-run failed (see $dir/bench.err)"
+  echo "$cmd $host $ts" > "$dir/MANIFEST"
+  echo "done: $(ls "$dir")"
+  ;;
+audit)
+  latest=$(find "$DATA_DIR/$host" -maxdepth 1 -mindepth 1 -type d -printf '%f\n' 2>/dev/null | sort | tail -1)
+  [ -n "$latest" ] || { echo "no baseline for $host" >&2; exit 3; }
+  echo "== KNELPerf audit: $host vs baseline $latest"
+  "$REPO_ROOT/loop/baseline-diff.sh" "$DATA_DIR/$host/$latest" "$host"
+  ;;
+tweak)
+  [ $# -ge 3 ] || usage; tweak_id=$3
+  echo "== KNELPerf tweak $tweak_id on $host (CR gate applies - see AGENTS.md)"
+  run_on_host "$host" "tweaks/$tweak_id.sh"
+  ;;
+compare)
+  [ $# -ge 4 ] || usage
+  "$REPO_ROOT/loop/baseline-diff.sh" "$DATA_DIR/$host/$3" "$DATA_DIR/$host/$4"
+  ;;
+*) usage ;;
+esac
@@ -0,0 +1,51 @@
+# KNELPerf scheduler — breakout from Ultix (#826)
+
+The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`)
+was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains
+the live workstation deployment; this module is the going-forward home.
+
+## What was ported and what changed
+
+| Ultix source | KNELPerf destination | Redesign delta |
+|---|---|---|
+| `staged/ukrrs-daynight.sh` | `daynight/knelperf-daynight.sh` | Slice targets, devices, and every knob are conf-driven lists (was: hardcoded `reachableceo-*` slices, `/dev/sdb`, user 1001). Slice specs are `slice:weight:cpus:memhigh` so any host participates. |
+| `staged/etc-ukrrs-daynight.conf` | `daynight/knelperf-daynight.conf.example` | Self-documenting example; post-upgrade values live per-host, not in comments of the engine. |
+| `staged/systemd/reachableceo-{day,night}profile.*` | `daynight/systemd/knelperf-*` | Generic units; the conf — never a sed-edit of a live timer — is the only value source. |
+| `staged/reachableceo-perfsnap.sh` | `collectors/knelperf-perfsnap.sh` | Watched cgroups/interfaces are env-driven lists; no baked-in uids/NICs. Same pipe-delimited log format. |
+| `staged/ukrrs-psi-textfile.sh` | `collectors/knelperf-psi-textfile.sh` | Cgroup glob + metric prefix env-driven; emits for any slice namespace. |
+| `staged/proxmox-ctl.sh` | `proxmox/proxmox-ctl.sh` | Ported as-is (was already generic + guarded: PROTECTED vmids, graceful shutdown only, pvesh local-or-token). |
+
+Not ported (stay workstation-local in ultix): `mkacct.sh` account tiers,
+`21-priority-ladder.sh` GUI-first cgroup ladder (host-specific by design),
+`gateway-ensure.sh`, RTMP stack.
+
+## Lessons learned carried forward (from ultix REPORT.md / TRACKING.md)
+
+1. `Slice=` is create-time only — `set-property` cannot set it; caps go in unit
+   drop-ins AND live via `set-property --runtime`.
+2. `CPUWeight` on user-manager units is inert until `cpu` is in the parent's
+   `cgroup.subtree_control` — enable it explicitly.
+3. IOWeight is inert under the `none` I/O scheduler (virtio) — don't bother.
+4. Guests have no cpufreq driver — governor tuning is PVE-host-side only.
+5. Reboot must land in DAY mode: slice unit files carry day defaults; the
+   profile engine only does runtime mutations. Non-negotiable invariant.
+6. `Persistent=true` timers replay missed flips at boot (intentional).
+7. Duplicated truth is the failure mode: one conf file per host is the only
+   place values live (the 07:00 sed-edit of a live timer was the wart that
+   motivated this redesign).
+8. Measure before touching; collectors install data, they tune nothing.
+
+## Fleet rollout model
+
+- Workstation (ultix-streaming): stays on its live ultix units until this
+  module is UAT'd, then converges onto knelperf-* units.
+- PVE nodes: daynight engine applies to host-level services + VM cgroup
+  scopes (`qemu.slice` children) via the same conf mechanism; timers install
+  from `daynight/systemd/`.
+- Everything deploys through AWX job templates (KNELIAC) per the 100%-IaC
+  mandate; this repo supplies the scripts, not ad-hoc ssh.
+
+See `docs/ARCHITECTURE.md` for the full baseline → audit → tweak → re-baseline
+program this module feeds.
+
+> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# KNELPerf perfsnap — periodic /proc+cgroupfs snapshot collector.
+# Ported from ultix staged/reachableceo-perfsnap.sh (#826); generalizations:
+#   - watched cgroups and interfaces are env/conf-driven lists (no baked-in
+#     user ids or NIC names)
+#   - log dir env-driven; same pipe-delimited format for awk analysis
+#
+# Env (or /etc/knelperf/perfsnap.conf):
+#   KNELPERF_PERF_LOGDIR   default /var/log/knelperf-perf
+#   KNELPERF_SNAP_CGROUPS  space-separated "label:/sys/fs/cgroup/path" list
+#   KNELPERF_SNAP_IFACES   space-separated interface names
+# Fields (snap.log): ts|l1|l5|l15|run|blocked|memavail_kb|psi_cpu|psi_mem|
+#   psi_io|<cgroup usage_usec per label>|<iface rx tx per label>
+# psi_* = avg60 "some" percentages. *_usec = cgroup cpu.stat usage_usec
+# (cumulative; analyze deltas).
+set -u
+CONF=/etc/knelperf/perfsnap.conf
+[ -r "$CONF" ] && . "$CONF"
+LOGDIR=${KNELPERF_PERF_LOGDIR:-/var/log/knelperf-perf}
+SNAP_CGROUPS=${KNELPERF_SNAP_CGROUPS:-}
+SNAP_IFACES=${KNELPERF_SNAP_IFACES:-}
+mkdir -p "$LOGDIR"
+ts=$(date +%s)
+
+read -r l1 l5 l15 _ < /proc/loadavg
+run=$(awk '/procs_running/{print $2}' /proc/stat)
+blocked=$(awk '/procs_blocked/{print $2}' /proc/stat)
+memavail=$(awk '/MemAvailable/{print $2}' /proc/meminfo)
+
+psi() { awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "/proc/pressure/$1" 2>/dev/null || echo 0; }
+
+line="$ts|$l1|$l5|$l15|$run|$blocked|$memavail|$(psi cpu)|$(psi memory)|$(psi io)"
+
+cgusec() { awk '/^usage_usec/{print $2}' "$1/cpu.stat" 2>/dev/null || echo 0; }
+for spec in $SNAP_CGROUPS; do
+  line="$line|$(cgusec "${spec#*:}")"
+done
+
+nd() { awk -v i="$1" '$1==i":":{print $2" "$10}' /proc/net/dev 2>/dev/null || echo "0 0"; }
+for ifc in $SNAP_IFACES; do
+  read -r rx tx <<< "$(nd "$ifc")"
+  line="$line|$rx|$tx"
+done
+
+echo "$line" >> "$LOGDIR/snap.log"
+
+ps -eo pcpu,pid,comm --sort=-pcpu --no-headers | head -5 \
+  | awk -v t="$ts" '{print t"|"$1"|"$2"|"$3}' >> "$LOGDIR/procs.log"
+
+# retention: 14 days; rotate anything over 50MB
+find "$LOGDIR" -name '*.log' -mtime +14 -delete 2>/dev/null
+for f in "$LOGDIR"/*.log; do
+  [ "$(wc -c <"$f" 2>/dev/null || echo 0)" -gt 52428800 ] && mv "$f" "$f.1"
+done
+exit 0
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector.
+# Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations:
+#   - cgroup glob is env-driven (any slice namespace, not just ukrrs-*)
+#   - metric prefix is env-driven (default knelperf_)
+set -u
+OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile}
+CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup}
+CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice}
+PREFIX=${KNELPERF_PREFIX:-knelperf}
+mkdir -p "$OUT_DIR"
+tmp=$(mktemp "$OUT_DIR/.knelperf.XXXXXX")
+
+some60() { # file -> prints avg60 value or nothing
+  awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "$1" 2>/dev/null
+}
+
+emit() { # name cgroup_path
+  local res v
+  for res in cpu memory io; do
+    [ -r "$2/$res.pressure" ] || continue
+    v=$(some60 "$2/$res.pressure")
+    [ -n "${v:-}" ] && echo "${PREFIX}_slice_pressure_some60{slice=\"$1\",res=\"$res\"} $v"
+  done
+}
+
+echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp"
+echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp"
+for cg in $CG_GLOB; do
+  [ -d "$cg" ] && emit "${cg##*/}" "$cg"
+done
+[ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice"
+
+echo "# HELP ${PREFIX}_host_pressure_some60 PSI some avg60 (percent) host-wide" >>"$tmp"
+echo "# TYPE ${PREFIX}_host_pressure_some60 gauge" >>"$tmp"
+for res in cpu memory io; do
+  [ -r "/proc/pressure/$res" ] || continue
+  v=$(some60 "/proc/pressure/$res")
+  [ -n "${v:-}" ] && echo "${PREFIX}_host_pressure_some60{res=\"$res\"} $v"
+done
+
+mv "$tmp" "$OUT_DIR/${PREFIX}_pressure.prom"
+chmod 0644 "$OUT_DIR/${PREFIX}_pressure.prom"
@@ -0,0 +1,28 @@
+# /etc/knelperf/daynight.conf — knelperf-daynight.sh configuration.
+# Copy to /etc/knelperf/daynight.conf and edit per host. Every field is
+# optional; omit slices/sections you do not manage on this host.
+#
+# Slice spec format (space-separated list):
+#   slice[:CPUWeight[:AllowedCPUs[:MemoryHigh]]]
+# Empty field = skip that property. Day AND night lists are required for
+# any slice you list (reboot-safety invariant: day values == unit-file
+# defaults).
+
+# Example (workstation, 8 vCPU — mirrors the ultix values):
+# DAY_SLICES="reachableceo-batch.slice:25:0-5:12G reachableceo-gateway.slice:900::"
+# NIGHT_SLICES="reachableceo-batch.slice:400:0-6:30G reachableceo-gateway.slice:500::"
+
+# Example (PVE host with a batch pool):
+# DAY_SLICES="knelperf-batch.slice:25:0-3:8G"
+# NIGHT_SLICES="knelperf-batch.slice:400:0-7:24G"
+
+# Block devices for night readahead boost (partitions auto-included):
+# BLOCK_DEVS="/dev/sdb"
+
+# Defaults shown in the script; override only what differs:
+# DATA_RA_DAY=256
+# DATA_RA_NIGHT=2048
+# DIRTY_DAY=1073741824
+# DIRTY_NIGHT=2147483648
+# DIRTY_BG_DAY=268435456
+# DIRTY_BG_NIGHT=536870912
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# KNELPerf day/night resource profile engine.
+#
+# Broken out of ultix staged/ukrrs-daynight.sh (2026-09-06, #826) and
+# redesigned: slice names, devices, and all numeric knobs are CONF-driven
+# (no host-specific hardcoding); slice targets are a list so any host can
+# run it. Runtime-only changes (set-property --runtime + sysctl + readahead):
+# a reboot always lands safely in day mode, because slice unit files carry
+# the day defaults (unchanged ultix invariant).
+#
+# Usage: knelperf-daynight.sh [--conf FILE] day|night
+set -euo pipefail
+
+CONF=${KNELPERF_DAYNIGHT_CONF:-/etc/knelperf/daynight.conf}
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --conf) CONF=$2; shift 2 ;;
+    day|night) MODE=$1; shift ;;
+    *) echo "usage: $0 [--conf FILE] day|night" >&2; exit 2 ;;
+  esac
+done
+[ "${MODE:-}" ] || { echo "usage: $0 [--conf FILE] day|night" >&2; exit 2; }
+[ -r "$CONF" ] && . "$CONF"
+
+# Targets: "slice:CPUWeight:AllowedCPUs:MemoryHigh" per mode, space-separated.
+# Empty fields mean "skip that property". See knelperf-daynight.conf.example.
+: "${DAY_SLICES:=}";  : "${NIGHT_SLICES:=}"
+# Block devices for readahead tuning (space-separated; partitions auto-appended).
+: "${BLOCK_DEVS:=}"
+: "${DATA_RA_DAY:=256}";          : "${DATA_RA_NIGHT:=2048}"
+: "${DIRTY_DAY:=1073741824}";     : "${DIRTY_NIGHT:=2147483648}"
+: "${DIRTY_BG_DAY:=268435456}";   : "${DIRTY_BG_NIGHT:=536870912}"
+
+log() { echo "[knelperf-daynight] $*"; }
+ra=$DATA_RA_DAY; dirty=$DIRTY_DAY; dirtybg=$DIRTY_BG_DAY; slices=$DAY_SLICES
+[ "$MODE" = night ] && { ra=$DATA_RA_NIGHT; dirty=$DIRTY_NIGHT; dirtybg=$DIRTY_BG_NIGHT; slices=$NIGHT_SLICES; }
+
+apply_slice() { # "slice:weight:cpus:memhigh"
+  local spec=$1 slice weight cpus memhigh
+  IFS=: read -r slice weight cpus memhigh <<< "$spec"
+  [ -n "$slice" ] || return 0
+  local -a props=()
+  [ -n "$weight" ]  && props+=(CPUWeight="$weight")
+  [ -n "$cpus" ]    && props+=(AllowedCPUs="$cpus")
+  [ -n "$memhigh" ] && props+=(MemoryHigh="$memhigh")
+  if [ ${#props[@]} -gt 0 ]; then
+    systemctl set-property --runtime "$slice" "${props[@]}" \
+      || log "WARN: set-property failed for $slice (slice not loaded?)"
+  fi
+}
+
+for spec in $slices; do apply_slice "$spec"; done
+
+if [ -n "$BLOCK_DEVS" ]; then
+  for dev in $BLOCK_DEVS; do
+    blockdev --setra "$ra" "$dev" 2>/dev/null || log "readahead skip: $dev"
+    for part in "${dev}"[0-9]*; do
+      [ -e "$part" ] && blockdev --setra "$ra" "$part" 2>/dev/null || true
+    done
+  done
+fi
+sysctl -q -w vm.dirty_bytes="$dirty" vm.dirty_background_bytes="$dirtybg"
+log "profile $MODE applied $(date -Is)"
@@ -0,0 +1,24 @@
+# systemd units — day/night profile timers
+
+Install per host (adjust names to your unit namespace if not knelperf-):
+
+    install -m 0644 knelperf-dayprofile.service  /etc/systemd/system/
+    install -m 0644 knelperf-dayprofile.timer    /etc/systemd/system/
+    install -m 0644 knelperf-nightprofile.service /etc/systemd/system/
+    install -m 0644 knelperf-nightprofile.timer   /etc/systemd/system/
+    systemctl daemon-reload
+    systemctl enable --now knelperf-dayprofile.timer knelperf-nightprofile.timer
+
+Design notes (carried from ultix, keep true here):
+
+- `Persistent=true` — a missed flip (host powered off) replays at next boot,
+  so the host can never get stuck in the wrong profile across a reboot cycle.
+- Services are `Type=oneshot` calling the profile engine with the SAME conf
+  file; the conf — not the timer — is the only place values live (fixes the
+  ultix wart where the day-flip time was sed-edited on the live timer).
+- Day schedule 07:00, night 22:00 local time (batch window 22:00–07:00).
+  Change the OnCalendar here when a host needs a different window; the timers
+  stay trivial and identical in shape.
+
+Reboot-safety invariant: slice unit files must carry the DAY values as their
+static defaults, so an unattended reboot always lands in day mode.
@@ -0,0 +1,9 @@
+[Unit]
+Description=KNELPerf day resource profile (interactive-first)
+After=multi-user.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/knelperf-daynight.sh day
+# Never restart a profile flip; the timer owns scheduling
+Restart=no
@@ -0,0 +1,10 @@
+[Unit]
+Description=KNELPerf day profile flip (07:00 local)
+
+[Timer]
+OnCalendar=*-*-* 07:00:00
+Persistent=true
+Unit=knelperf-dayprofile.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,8 @@
+[Unit]
+Description=KNELPerf night resource profile (batch window 22:00-07:00)
+After=multi-user.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/knelperf-daynight.sh night
+Restart=no
@@ -0,0 +1,10 @@
+[Unit]
+Description=KNELPerf night profile flip (22:00 local)
+
+[Timer]
+OnCalendar=*-*-* 22:00:00
+Persistent=true
+Unit=knelperf-nightprofile.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# UKRRS Proxmox control wrapper for harness/human use. HARD RULES baked in:
+#   - vmids listed in PROTECTED are never touched (list THIS VM first).
+#   - graceful shutdown only; no reset, no hard stop, no bare down (house rule).
+# Env:
+#   PVE_LOCAL=1                      run ON the pve host itself (pvesh direct)
+#   or PVE_HOST + PVE_TOKEN          remote, token format: user@realm!tokenid=secret
+#   PVE_NODE                         node name (required for vm ops)
+#   PROTECTED                        required, comma-separated vmids
+# Usage:
+#   proxmox-ctl.sh vms
+#   proxmox-ctl.sh shutdown <vmid>
+#   proxmox-ctl.sh start <vmid>
+#   proxmox-ctl.sh snapshot <vmid> <name>
+# Token setup (on the pve host, see docs/REPORT.md appendix B):
+#   pveum user add ukrrs-infra@pam
+#   pveum user token add ukrrs-infra@pam harness -privsep 0 -expire 0
+#   pveum acl modify /pool/<preprod-pool> -user ukrrs-infra@pam -role PVEVMUser
+set -euo pipefail
+: "${PROTECTED:?set PROTECTED=vmid1,vmid2,... (this VM must be in the list)}"
+
+pvesh_() {
+  if [ "${PVE_LOCAL:-0}" = 1 ]; then
+    command pvesh "$@"
+  else
+    : "${PVE_HOST:?}" "${PVE_TOKEN:?}"
+    command pvesh --host "$PVE_HOST" --api-token "$PVE_TOKEN" "$@"
+  fi
+}
+
+guard() {
+  case ",$PROTECTED," in
+    *",$1,"*) echo "REFUSED: vmid $1 is PROTECTED" >&2; exit 3 ;;
+  esac
+}
+
+cmd=${1:-}; shift || true
+case "$cmd" in
+  vms)
+    pvesh_ get /cluster/resources --type vm ;;
+  shutdown)
+    [ $# = 1 ] || { echo "usage: $0 shutdown <vmid>" >&2; exit 2; }
+    guard "$1"; : "${PVE_NODE:?}"
+    pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/shutdown" --timeout 120 ;;
+  start)
+    [ $# = 1 ] || { echo "usage: $0 start <vmid>" >&2; exit 2; }
+    guard "$1"; : "${PVE_NODE:?}"
+    pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/start" ;;
+  snapshot)
+    [ $# = 2 ] || { echo "usage: $0 snapshot <vmid> <name>" >&2; exit 2; }
+    guard "$1"; : "${PVE_NODE:?}"
+    pvesh_ create "/nodes/$PVE_NODE/qemu/$1/snapshot" snapname="$2" ;;
+  *)
+    echo "usage: $0 vms|shutdown <vmid>|start <vmid>|snapshot <vmid> <name>" >&2
+    exit 2 ;;
+esac