KNEL/KNELPerf
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

KNELPerf v2 wave 1: Ultix scheduler breakout + loop skeleton + reports (#826)
ci / audit (push) Failing after 26s
Details

Browse Source 
- scheduler/: day/night engine (conf-driven redesign), psi/perfsnap collectors,
  proxmox-ctl, systemd timer templates — ported from ~/projects/ultix per the
  breakout map in scheduler/README.md
- loop/: perf-loop driver (baseline/audit/tweak/rebaseline), cpu-bench (new CPU
  leg), baseline-diff (tolerance compare) — smoke-tested locally
- docs/ARCHITECTURE.md: full program design (loop, resource groups, VM/spindle
  balancing, k8s-vs-Slurm, beszel+RAPL/iDRAC telemetry spine, workload classes,
  solar/HA + two-site power economics)
- docs/report-amt-power-telemetry.md + docs/report-moonlight-desktop.md

Redmine: https://projects.knownelement.com/issues/826#note-2
This commit is contained in:
mrcharles
2026-09-06 14:16:54 -05:00
parent 89a93359f0
commit 25299373f7
19 changed files with 839 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+177
View File
@@ -0,0 +1,177 @@
# KNELPerf v2 — architecture for dynamic, intelligent performance optimization
Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end
baseline → audit/benchmark → tweak → re-baseline loop, two-week program.
## 1. The loop
```
┌──────────────────────────────────────────────────┐
│ │
BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
(capture) (diff + beszel/ (CR-gated, (compare vs
PSI/rapl context) idempotent) baseline; keep
or roll back)
```
- **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
(cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines/<host>/<ts>/`.
Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg
(`loop/cpu-bench.sh`) is new — the repo previously had zero CPU coverage.
- **audit**: `loop/perf-loop.sh audit <host>` re-runs the read-only probes and
diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based,
catches governor flips exactly). This is what the scheduled loop runs.
- **tweak**: every mutation is a file under `tweaks/<id>.sh` (idempotent,
rollback helper included). Production tweaks ride a GLPI CR; the loop
refuses `tweak` on prod hosts without `.crush/active-cr`.
- **re-baseline**: same capture, then `compare` — the delta table is the
evidence artifact that lands on the Redmine ticket.
Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
the audit itself is low-cost); re-baseline only after a tweak or hardware
change. First full-fleet baseline wave is this week's deliverable.
## 2. Scheduling architecture (Ultix breakout redesign)
See `scheduler/README.md` for the port map and the lessons-learned list. Core
redesign decisions:
1. **Single source of truth per host** — `/etc/knelperf/daynight.conf`; no
value ever lives in a timer file, unit default, or script constant.
2. **Reboot lands in day mode** — invariant carried from Ultix.
3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so
PVE hosts tune `qemu.slice` children and host services with the same engine
the workstation uses.
4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
payload; job templates own the rollout waves.
## 3. Proxmox resource groups & VM balancing
Current fleet (dmidecode-verified 2026-09-06):
| node | platform | cores | RAM | storage character |
|---|---|---|---|---|
| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role |
| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton |
| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm |
| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) |
| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner |
| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner |
| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge |
| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe |
Directions:
- **Proxmox resource groups / pools**: map pools to tenant classes (see §6),
so scheduler policy attaches to the pool, not per-VM.
- **Balancing across hosts AND spindles**: nodes are standalone (no
corosync/HA), so "balancing" = placement policy at create/migrate time, not
live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6;
RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8,
tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with
one-job-per-spindle placement (fio baselines per spindle will quantify).
The audit loop's beszel/PSI data feeds a placement scorecard.
- **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G —
largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.).
Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and
the core count to soak k8s bins; tsys6's older CPUs are fine for
steady-state VMs but it lacks DCMI telemetry on old firmware. Decision
needs founder ruling — will be proposed on #826 with a capacity table once
tsys8/2 land.
## 4. Where k8s scheduling fits; Slurm?
- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
(ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
class 1–2 workloads; node labels/taints pin tenant classes to node pools.
- **HFNOC batch (class 3)** = interruptible, deadline-loose, large — this is
HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics
(preemption via priority classes — fits "start and stop easily"); (b) Slurm
on a dedicated partition across idle capacity. Recommendation: start with
k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only
introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm
co-located with k8s on shared nodes via cgroup slices is possible
(knelperf-batch.slice) but doubles scheduling systems for little gain at
this fleet size.
- The knelperf daynight engine gives the batch window (22:00–07:00) its
expanded resource envelope on each participating node.
## 5. Telemetry spine: beszel + power
- **Beszel has 100% VM+physical coverage** → it is the always-on signal for
the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
feed the placement scorecard; PSI textfile collector
(`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks.
- **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts
come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5
needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the
un-metered UPS, everything else on the metered one) is compatible: the
nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.
## 6. Workload priority classes (founder-set order)
| class | tenants | profile | scheduling |
|---|---|---|---|
| 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7) |
| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
## 7. Solar + Home Assistant integration
- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
Assistant exposes real-time production. The loop consumes: surplus watts =
green light to start class-3 batches (excess production is the cheapest
compute on earth); forecast + reservation calendar (KillBill) → promotion
timing advice ("run the Side Door promo the week of X: projected surplus +
idle capacity Y").
- Integration path: HA REST/websocket (creds pattern exists:
`~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) → a small
exporter writing `knelperf_solar_surplus_watts` into the textfile
collector's directory; the batch launcher gates on it.
- SITER-Solar math review (2026-09-06): found constant drift
($301.08/23952 vs plan's $264.47/22614), a stale Dockerfile entrypoint
(deprecated python script), header/ROI inconsistencies, and fixed-cost
scenario assumptions — fixes landing in that repo.
## 8. Two-site power economics (site 2: printer farm / laser / CNC)
The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
MUCH more expensive grid power than the main site. That asymmetry drives
scheduling policy as much as raw performance:
- **Marginal-price model**: every site has an effective $/kWh curve. Main
site = cheap grid + (soon) solar with export at $0.04 and self-use at
$0.085 (SITER-Solar contract). Site 2 = expensive grid. The scheduler's
objective becomes: run flexible load (compute AND manufacturing) where the
marginal $/kWh is lowest at that hour.
- **Solar surplus is main-site-first**: excess production self-consumed at
the main site displaces the CHEAPEST power — class-3 batch compute and any
deferrable main-site load should chase the surplus curve.
- **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and
often deferrable (prints especially — hours-long, unattended). Candidate
policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows
(if its tariff has TOU) or batch them for periods when main-site solar
surplus can't absorb more compute. Home Assistant + metered UPS legs give
the measured baseline; site 2 needs its own metering (smart plug / CT per
machine) before scheduling — queued as a data dependency.
- **Never shift OLTP**: class-1 paid workloads stay on their reserved
capacity regardless of power price — revenue and latency beat energy cost.
- KillBill reservation calendar × solar forecast × site-2 tariff = the
promotion/job-timing advisor output: "cheapest week to run X".
## 9. Hardware plan hooks
- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
baseline each with fio, then one-batch-job-per-spindle).
- tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory
row pending (PFVSystemInventory.csv not found in repo — founder to drop in).
- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
with tsys3 for latency-sensitive class-1 storage.
## 10. Organization
Performance management org reports to VPTechOps, dotted line VPFacilities —
identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
groups). All work filed under #826; CRs for any prod change.
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
docs/report-amt-power-telemetry.md
+84
View File
@@ -0,0 +1,84 @@
# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via
dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included).
## Executive answers
**Does AMT provide power usage data? Can it be pulled from the BIOS?**
**No — no Intel AMT version (9.x through current) exposes real-time watt
telemetry, via BIOS or any API.** Intel's own forum answer is categorical:
AMT provides power *control* (on/off/cycle/boot-once), not power
*measurement*. The Redfish `Power`/`PowerConsumedWatts` resource is a
server-BMC feature (iDRAC/iLO wired to PSU metering hardware), not an AMT
feature. So AMT is worth enabling only for remote KVM/power-control, never
for watts.
## Per-system table (hardware verified by dmidecode 2026-09-06)
| System | Platform (verified) | AMT/vPro? | Watts via mgmt? | Best watt source |
|---|---|---|---|---|
| pfv-tsys1 | OptiPlex 9020, i7-4770 (Haswell), Q87 | YES — AMT 9.1 (Dell spec confirms vPro/iAMT 9.x) | NO | Linux RAPL: PKG + DRAM |
| pfv-tsys3 | Precision 7510 (mobile WS), Xeon E3-1535M v5 (Skylake-H), CM236 | YES — AMT 11.x (patch post-CVE-2017-5689!) | NO | Linux RAPL: PKG (DRAM domain often absent on mobile SKL — verify) |
| pfv-tsys4 | Precision T1700, Xeon E3-1246 v3 (Haswell), C226 | **NO AMT** — CPU is vPro-capable but C226 has no AMT ME stack (Haswell vPro needs Q87); no BMC either | NO | Linux RAPL: PKG + DRAM |
| pfv-tsys5 | Precision T7500, Xeon E5620 (Westmere), 5520 chipset | NO — server-class chipset, pre-vPro era; no BMC | NO | **Nothing software-readable: Westmere has NO RAPL** (Sandy Bridge introduced it). Metered PDU / smart plug only |
| pfv-tsys6 | PowerEdge R610, 2x E5530, iDRAC6 | NO AMT (servers = BMC) | **YES via iDRAC6** | `racadm getconfig -g cfgServerPower -o cfgServerActualPowerConsumption`; ipmitool sensor "System Level" (watts often x10). DCMI partial/absent on old firmware |
| pfv-tsys7 | PowerEdge R620, 2x E5-2630 v2 (Ivy Bridge-EP), iDRAC7 | NO AMT (BMC) | **YES via iDRAC7** (full DCMI) | `ipmitool dcmi power reading`; `racadm getsysinfo -c`; plus host-side RAPL PKG+DRAM |
| tsys9 | unknown (2013-17 OptiPlex/Precision) | Identify first: `dmidecode -t system` + `lspci \| grep -i MEI` + MEBx (Ctrl+P) | if AMT: still NO watts | RAPL if Haswell or newer |
## Practical pull methods
- **AMT 9.x/10.x/11.x (tsys1, tsys3):** WSMAN/SOAP via `wsman` CLI or
MeshCommander (port 16993 TLS) — power state, on/off, KVM, inventory. AMT
11.6+ also speaks Redfish at `https://<host>:16992/redfish/v1/` but with no
Power telemetry resource.
- **Local MEI/HECI discovery:** `/dev/mei0`, `intelmetool` — tells you whether
AMT firmware is present/provisioned (use this for tsys9 triage).
- **RAPL readers:** `turbostat` (most complete; PKG/cores/DRAM), powercap
sysfs `/sys/class/powercap/intel-rapl:*` (poll `energy_uj`, delta/t = W),
`perf stat -e power/energy-pkg/`. node_exporter has NO built-in RAPL
collector — deploy a small rapl exporter or scrape turbostat. Caveat: RAPL
measures CPU package (+DRAM), not whole-wall draw (~60-75% of wall on
desktops).
## RAPL coverage by CPU generation
- RAPL introduced Sandy Bridge (2011); **Westmere (E5620) and Nehalem (E5530)
have none — tsys5 and tsys6 hosts cannot RAPL.**
- Haswell adds DRAM domain: tsys1 (i7-4770) and tsys4 (E3-1246 v3) get
PKG+PP0+DRAM.
- Skylake-H (tsys3): PKG yes, DRAM typically absent on mobile.
- Ivy Bridge-EP (tsys7): PKG+DRAM.
## Recommendations
1. Don't chase watts via AMT anywhere — the capability does not exist. Use
AMT (tsys1, tsys3) only for remote KVM/power control; patch tsys3's AMT 11
firmware for CVE-2017-5689 before enabling.
2. Uniform host-side telemetry = RAPL exporter on tsys1, tsys3, tsys4, tsys7.
3. Servers (tsys6, tsys7): pull BMC watts out-of-band (racadm/ipmitool DCMI);
update iDRAC6 firmware on tsys6 if DCMI commands fail.
4. tsys5 (T7500): budget a metered PDU or smart plug — no software path
exists (fits the planned UPS re-cabling: put tsys5 on the metered leg).
5. tsys9: run `dmidecode -t system` + `ls /sys/class/powercap` — that pair
answers both its AMT and RAPL questions; report back.
6. UPS plan alignment: since only tsys6/7 (BMC) + RAPL hosts give per-node
watts, keep the metered UPS leg feeding the gear you most need wall-watts
for; per-node software telemetry covers the rest.
## Sources
- Intel Community — AMT and power usage monitoring: https://community.intel.com/t5/Intel-Business-Client-Software/AMT-and-power-usage-monitoring/td-p/853085
- Intel Community — AMT Support Identification: https://community.intel.com/t5/Intel-vPro-Platform/Intel-AMT-Support-Identification/td-p/1275668
- DMTF Redfish Telemetry DSP2051: https://www.dmtf.org/sites/default/files/standards/documents/DSP2051_1.1.0.pdf
- OptiPlex 9020 spec (AMT 9.x): https://www.dell.com/support/manuals/en-us/optiplex-9020-desktop/opt9020sffom-v2/specifications
- Precision 7510 spec sheet: https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/us/Dell-Precision-15-7000-Series-7510-Spec-Sheet.pdf
- ARK E3-1246 v3: https://www.intel.com/content/www/us/en/products/sku/80916/intel-xeon-processor-e31246-v3-8m-cache-3-50-ghz/specifications.html
- ServerFault Dell power CLI: https://serverfault.com/questions/736068/how-do-i-get-the-power-consumption-of-a-dell-poweredge-server-on-the-cli
- jonamiki racadm power capping: https://jonamiki.com/2014/10/18/power-control-and-capping-on-dell-servers-using-racadm/
- Dell InfoHub power reporting: https://infohub.delltechnologies.com/en-us/p/server-power-consumption-reporting-and-management/
- Launchpad RAPL domains: https://bugs.launchpad.net/bugs/1467014
- Hackenberg et al. Haswell RAPL survey: https://tu-dresden.de/zih/forschung/ressourcen/dateien/projekte/firestarter/2015_hackenberg_hppac.pdf
- kernel mei/iamt docs: https://docs.kernel.org/driver-api/mei/iamt.html
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
docs/report-moonlight-desktop.md
+68
View File
@@ -0,0 +1,68 @@
# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826.
Source research: web-surveyed 2026-09-06 (sources linked at the end).
## Context constraints (verified)
- Xeon E5-2630 v2 (pfv-tsys7 class) has **no iGPU / no Quick Sync** — hardware
video encode is impossible on the current CPU-only nodes. Software x264 (or a
future NVIDIA node's NVENC) is the only path.
- Current access path is xrdp/KDE over Tailscale at ~36 ms RTT. Protocol choice
matters less than encoder cost and whether the Tailscale path is direct vs
DERP-relayed (DERP caps ~5 Mbps and adds latency — check `tailscale netcheck`
before blaming the desktop stack).
- The cluster is CPU-only today, mixed with production workloads, flux gitops —
heavy sustained CPU encoding on shared nodes is an operational risk.
## Comparison
| | Sunshine+Moonlight | Selkies-GStreamer | Kasm Workspaces | Webtop / Guacamole | Tuned xrdp (baseline) |
|---|---|---|---|---|---|
| Transport | Moonlight protocol, H.264/265/AV1 | WebRTC (browser) | KasmVNC over WebSocket | VNC/RDP → WebSocket | RDP |
| GPU needed | No (x264 soft mode) but costly | No (x264 soft) | No | No | No |
| CPU cost on old Xeon | HIGH: 2–4 cores sustained @1080p60 | HIGH + WebRTC stack ~1 core | MODERATE (framebuffer diff, no video encode) | LOW–MODERATE (Guacamole ~15–27%/core per 1–2 users) | LOW |
| Latency @36 ms RTT | Best-in-class on direct path; 100–200 ms if misconfigured | Low | Moderate; fine for desktop, visible on video | Moderate–high (protocol translation + browser) | Moderate (tuned RDP is decent) |
| Client | Native Moonlight apps (excellent) — NOT browser | Any browser | Any browser | Any browser | Any RDP client |
| k8s fit | Awkward: privileged pod + `/dev/uinput` hostDevice, dummy X | Good (purpose-built for k8s) | Heavy control plane for one desktop | Trivial pod + PVC (webtop); Guacamole = small extra stack | Stays a VM |
| Persistence | StatefulSet + PVC | PVC | Disposable by design | webtop PVC-backed | Full VM (best) |
| GitOps friendliness | Medium | High | Medium | High | N/A |
## Recommendation
1. **Now:** keep the tuned xrdp/KDE VM as baseline; do NOT put Sunshine/Moonlight
on CPU-only nodes — real-time x264 eats 2–4 cores next to production tenants.
2. **Migration path:** deploy **linuxserver/webtop (KasmVNC variant)** as a
PVC-backed StatefulSet on a CPU-only node behind the existing ingress
(browser access; WAN via Tailscale). This is the containerized successor for
desktop-class use at 36 ms RTT: cheapest CPU, unprivileged pod, clean
flux fit.
3. **WAN/browser fallback for the existing RDP VM:** small Apache Guacamole
stack (guacd + guacamole), accepting it is slower than native RDP clients.
4. **When the NVIDIA node lands:** revisit **Selkies-GStreamer** (most
k8s-native low-latency option; NVENC removes the encoder cost) as the
premium tier. **Sunshine+Moonlight** earns its complexity (privileged pod,
/dev/uinput) only if gaming-grade latency on native clients becomes a hard
requirement — then pin it to the GPU node only, never shared CPU nodes.
Rollout: tuned xrdp VM stays during migration → webtop StatefulSet UAT →
retire VM after human UAT → Selkies on the future GPU node.
## Sources
- Intel Xeon Quick Sync support: https://www.intel.com/content/www/us/en/support/articles/000034104/processors/intel-xeon-processors.html
- Sunshine advanced usage (encoder cost): https://docs.lizardbyte.dev/projects/sunshine/v0.23.0/about/advanced_usage.html
- Moonlight setup guide: https://github.com/moonlight-stream/moonlight-docs/wiki/Setup-Guide
- Tailscale DERP 5 Mbps gotcha: https://cfreeman.cloud/breaking-the-5-mbps-barrier-streaming-moonlight-over-tailscale-with-full-bandwidth/
- LizardByte latency discussion: https://github.com/orgs/LizardByte/discussions/17
- LXC headless Sunshine/uinput: https://discuss.linuxcontainers.org/t/headless-wayland-container-streaming-via-sunshine-sway-libinput-not-finding-uinput-devices/18852
- steam-headless uinput issue: https://github.com/Steam-Headless/docker-steam-headless/issues/191
- Selkies-GStreamer: https://github.com/selkies-project/selkies
- Kasm desktop images: https://hub.docker.com/r/kasmweb/desktop
- linuxserver/docker-webtop: https://github.com/linuxserver/docker-webtop
- LSIO docker-kasm: https://docs.linuxserver.io/images/docker-kasm/
- Guacamole web-client slowness: https://stackoverflow.com/questions/77224933/apache-guacamole-much-slower-on-the-web-client-than-microsoft-desktop
- guacd CPU benchmarks: https://lists.apache.org/thread/zdlsnn2br4gvtx09q6zqtxym6dwlrjwc
- Cendio on browser remote desktop limits: https://www.cendio.com/blog/apache-guacamole-alternatives/
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328