diff --git a/.crush/active-ticket b/.crush/active-ticket index 312f716..0e839d4 100644 --- a/.crush/active-ticket +++ b/.crush/active-ticket @@ -1 +1 @@ -#709 +#826 diff --git a/data/baselines/.gitignore b/data/baselines/.gitignore new file mode 100644 index 0000000..396917e --- /dev/null +++ b/data/baselines/.gitignore @@ -0,0 +1 @@ +data/baselines/*/ diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..1a79459 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,177 @@ +# KNELPerf v2 — architecture for dynamic, intelligent performance optimization + +Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end +baseline → audit/benchmark → tweak → re-baseline loop, two-week program. + +## 1. The loop + +``` + ┌──────────────────────────────────────────────────┐ + │ │ + BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘ + (capture) (diff + beszel/ (CR-gated, (compare vs + PSI/rapl context) idempotent) baseline; keep + or roll back) +``` + +- **baseline**: `loop/perf-loop.sh baseline ` stores a bundle + (cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines///`. + Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg + (`loop/cpu-bench.sh`) is new — the repo previously had zero CPU coverage. +- **audit**: `loop/perf-loop.sh audit ` re-runs the read-only probes and + diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based, + catches governor flips exactly). This is what the scheduled loop runs. +- **tweak**: every mutation is a file under `tweaks/.sh` (idempotent, + rollback helper included). Production tweaks ride a GLPI CR; the loop + refuses `tweak` on prod hosts without `.crush/active-cr`. +- **re-baseline**: same capture, then `compare` — the delta table is the + evidence artifact that lands on the Redmine ticket. + +Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring: +the audit itself is low-cost); re-baseline only after a tweak or hardware +change. First full-fleet baseline wave is this week's deliverable. + +## 2. Scheduling architecture (Ultix breakout redesign) + +See `scheduler/README.md` for the port map and the lessons-learned list. Core +redesign decisions: + +1. **Single source of truth per host** — `/etc/knelperf/daynight.conf`; no + value ever lives in a timer file, unit default, or script constant. +2. **Reboot lands in day mode** — invariant carried from Ultix. +3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so + PVE hosts tune `qemu.slice` children and host services with the same engine + the workstation uses. +4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the + payload; job templates own the rollout waves. + +## 3. Proxmox resource groups & VM balancing + +Current fleet (dmidecode-verified 2026-09-06): + +| node | platform | cores | RAM | storage character | +|---|---|---|---|---| +| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role | +| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton | +| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm | +| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) | +| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner | +| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner | +| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge | +| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe | + +Directions: +- **Proxmox resource groups / pools**: map pools to tenant classes (see §6), + so scheduler policy attaches to the pool, not per-VM. +- **Balancing across hosts AND spindles**: nodes are standalone (no + corosync/HA), so "balancing" = placement policy at create/migrate time, not + live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6; + RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8, + tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with + one-job-per-spindle placement (fio baselines per spindle will quantify). + The audit loop's beszel/PSI data feeds a placement scorecard. +- **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G — + largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.). + Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and + the core count to soak k8s bins; tsys6's older CPUs are fine for + steady-state VMs but it lacks DCMI telemetry on old firmware. Decision + needs founder ruling — will be proposed on #826 with a capacity table once + tsys8/2 land. + +## 4. Where k8s scheduling fits; Slurm? + +- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes + (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for + class 1–2 workloads; node labels/taints pin tenant classes to node pools. +- **HFNOC batch (class 3)** = interruptible, deadline-loose, large — this is + HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics + (preemption via priority classes — fits "start and stop easily"); (b) Slurm + on a dedicated partition across idle capacity. Recommendation: start with + k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only + introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm + co-located with k8s on shared nodes via cgroup slices is possible + (knelperf-batch.slice) but doubles scheduling systems for little gain at + this fleet size. +- The knelperf daynight engine gives the batch window (22:00–07:00) its + expanded resource envelope on each participating node. + +## 5. Telemetry spine: beszel + power + +- **Beszel has 100% VM+physical coverage** → it is the always-on signal for + the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics + feed the placement scorecard; PSI textfile collector + (`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks. +- **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts + come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5 + needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the + un-metered UPS, everything else on the metered one) is compatible: the + nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes. + +## 6. Workload priority classes (founder-set order) + +| class | tenants | profile | scheduling | +|---|---|---|---| +| 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7) | +| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 | +| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 | + +## 7. Solar + Home Assistant integration + +- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home + Assistant exposes real-time production. The loop consumes: surplus watts = + green light to start class-3 batches (excess production is the cheapest + compute on earth); forecast + reservation calendar (KillBill) → promotion + timing advice ("run the Side Door promo the week of X: projected surplus + + idle capacity Y"). +- Integration path: HA REST/websocket (creds pattern exists: + `~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) → a small + exporter writing `knelperf_solar_surplus_watts` into the textfile + collector's directory; the batch launcher gates on it. +- SITER-Solar math review (2026-09-06): found constant drift + ($301.08/23952 vs plan's $264.47/22614), a stale Dockerfile entrypoint + (deprecated python script), header/ROI inconsistencies, and fixed-cost + scenario assumptions — fixes landing in that repo. + +## 8. Two-site power economics (site 2: printer farm / laser / CNC) + +The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on +MUCH more expensive grid power than the main site. That asymmetry drives +scheduling policy as much as raw performance: + +- **Marginal-price model**: every site has an effective $/kWh curve. Main + site = cheap grid + (soon) solar with export at $0.04 and self-use at + $0.085 (SITER-Solar contract). Site 2 = expensive grid. The scheduler's + objective becomes: run flexible load (compute AND manufacturing) where the + marginal $/kWh is lowest at that hour. +- **Solar surplus is main-site-first**: excess production self-consumed at + the main site displaces the CHEAPEST power — class-3 batch compute and any + deferrable main-site load should chase the surplus curve. +- **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and + often deferrable (prints especially — hours-long, unattended). Candidate + policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows + (if its tariff has TOU) or batch them for periods when main-site solar + surplus can't absorb more compute. Home Assistant + metered UPS legs give + the measured baseline; site 2 needs its own metering (smart plug / CT per + machine) before scheduling — queued as a data dependency. +- **Never shift OLTP**: class-1 paid workloads stay on their reserved + capacity regardless of power price — revenue and latency beat energy cost. +- KillBill reservation calendar × solar forecast × site-2 tariff = the + promotion/job-timing advisor output: "cheapest week to run X". + +## 9. Hardware plan hooks + +- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins + are placement + spindles (tsys4's 3 USB toasters join the spindle farm: + baseline each with fio, then one-batch-job-per-spindle). +- tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory + row pending (PFVSystemInventory.csv not found in repo — founder to drop in). +- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair + with tsys3 for latency-sensitive class-1 storage. + +## 10. Organization + +Performance management org reports to VPTechOps, dotted line VPFacilities — +identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI +groups). All work filed under #826; CRs for any prod change. + +> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328 diff --git a/docs/report-amt-power-telemetry.md b/docs/report-amt-power-telemetry.md new file mode 100644 index 0000000..7842ad8 --- /dev/null +++ b/docs/report-amt-power-telemetry.md @@ -0,0 +1,84 @@ +# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7) + +Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via +dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included). + +## Executive answers + +**Does AMT provide power usage data? Can it be pulled from the BIOS?** +**No — no Intel AMT version (9.x through current) exposes real-time watt +telemetry, via BIOS or any API.** Intel's own forum answer is categorical: +AMT provides power *control* (on/off/cycle/boot-once), not power +*measurement*. The Redfish `Power`/`PowerConsumedWatts` resource is a +server-BMC feature (iDRAC/iLO wired to PSU metering hardware), not an AMT +feature. So AMT is worth enabling only for remote KVM/power-control, never +for watts. + +## Per-system table (hardware verified by dmidecode 2026-09-06) + +| System | Platform (verified) | AMT/vPro? | Watts via mgmt? | Best watt source | +|---|---|---|---|---| +| pfv-tsys1 | OptiPlex 9020, i7-4770 (Haswell), Q87 | YES — AMT 9.1 (Dell spec confirms vPro/iAMT 9.x) | NO | Linux RAPL: PKG + DRAM | +| pfv-tsys3 | Precision 7510 (mobile WS), Xeon E3-1535M v5 (Skylake-H), CM236 | YES — AMT 11.x (patch post-CVE-2017-5689!) | NO | Linux RAPL: PKG (DRAM domain often absent on mobile SKL — verify) | +| pfv-tsys4 | Precision T1700, Xeon E3-1246 v3 (Haswell), C226 | **NO AMT** — CPU is vPro-capable but C226 has no AMT ME stack (Haswell vPro needs Q87); no BMC either | NO | Linux RAPL: PKG + DRAM | +| pfv-tsys5 | Precision T7500, Xeon E5620 (Westmere), 5520 chipset | NO — server-class chipset, pre-vPro era; no BMC | NO | **Nothing software-readable: Westmere has NO RAPL** (Sandy Bridge introduced it). Metered PDU / smart plug only | +| pfv-tsys6 | PowerEdge R610, 2x E5530, iDRAC6 | NO AMT (servers = BMC) | **YES via iDRAC6** | `racadm getconfig -g cfgServerPower -o cfgServerActualPowerConsumption`; ipmitool sensor "System Level" (watts often x10). DCMI partial/absent on old firmware | +| pfv-tsys7 | PowerEdge R620, 2x E5-2630 v2 (Ivy Bridge-EP), iDRAC7 | NO AMT (BMC) | **YES via iDRAC7** (full DCMI) | `ipmitool dcmi power reading`; `racadm getsysinfo -c`; plus host-side RAPL PKG+DRAM | +| tsys9 | unknown (2013-17 OptiPlex/Precision) | Identify first: `dmidecode -t system` + `lspci \| grep -i MEI` + MEBx (Ctrl+P) | if AMT: still NO watts | RAPL if Haswell or newer | + +## Practical pull methods + +- **AMT 9.x/10.x/11.x (tsys1, tsys3):** WSMAN/SOAP via `wsman` CLI or + MeshCommander (port 16993 TLS) — power state, on/off, KVM, inventory. AMT + 11.6+ also speaks Redfish at `https://:16992/redfish/v1/` but with no + Power telemetry resource. +- **Local MEI/HECI discovery:** `/dev/mei0`, `intelmetool` — tells you whether + AMT firmware is present/provisioned (use this for tsys9 triage). +- **RAPL readers:** `turbostat` (most complete; PKG/cores/DRAM), powercap + sysfs `/sys/class/powercap/intel-rapl:*` (poll `energy_uj`, delta/t = W), + `perf stat -e power/energy-pkg/`. node_exporter has NO built-in RAPL + collector — deploy a small rapl exporter or scrape turbostat. Caveat: RAPL + measures CPU package (+DRAM), not whole-wall draw (~60-75% of wall on + desktops). + +## RAPL coverage by CPU generation + +- RAPL introduced Sandy Bridge (2011); **Westmere (E5620) and Nehalem (E5530) + have none — tsys5 and tsys6 hosts cannot RAPL.** +- Haswell adds DRAM domain: tsys1 (i7-4770) and tsys4 (E3-1246 v3) get + PKG+PP0+DRAM. +- Skylake-H (tsys3): PKG yes, DRAM typically absent on mobile. +- Ivy Bridge-EP (tsys7): PKG+DRAM. + +## Recommendations + +1. Don't chase watts via AMT anywhere — the capability does not exist. Use + AMT (tsys1, tsys3) only for remote KVM/power control; patch tsys3's AMT 11 + firmware for CVE-2017-5689 before enabling. +2. Uniform host-side telemetry = RAPL exporter on tsys1, tsys3, tsys4, tsys7. +3. Servers (tsys6, tsys7): pull BMC watts out-of-band (racadm/ipmitool DCMI); + update iDRAC6 firmware on tsys6 if DCMI commands fail. +4. tsys5 (T7500): budget a metered PDU or smart plug — no software path + exists (fits the planned UPS re-cabling: put tsys5 on the metered leg). +5. tsys9: run `dmidecode -t system` + `ls /sys/class/powercap` — that pair + answers both its AMT and RAPL questions; report back. +6. UPS plan alignment: since only tsys6/7 (BMC) + RAPL hosts give per-node + watts, keep the metered UPS leg feeding the gear you most need wall-watts + for; per-node software telemetry covers the rest. + +## Sources + +- Intel Community — AMT and power usage monitoring: https://community.intel.com/t5/Intel-Business-Client-Software/AMT-and-power-usage-monitoring/td-p/853085 +- Intel Community — AMT Support Identification: https://community.intel.com/t5/Intel-vPro-Platform/Intel-AMT-Support-Identification/td-p/1275668 +- DMTF Redfish Telemetry DSP2051: https://www.dmtf.org/sites/default/files/standards/documents/DSP2051_1.1.0.pdf +- OptiPlex 9020 spec (AMT 9.x): https://www.dell.com/support/manuals/en-us/optiplex-9020-desktop/opt9020sffom-v2/specifications +- Precision 7510 spec sheet: https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/us/Dell-Precision-15-7000-Series-7510-Spec-Sheet.pdf +- ARK E3-1246 v3: https://www.intel.com/content/www/us/en/products/sku/80916/intel-xeon-processor-e31246-v3-8m-cache-3-50-ghz/specifications.html +- ServerFault Dell power CLI: https://serverfault.com/questions/736068/how-do-i-get-the-power-consumption-of-a-dell-poweredge-server-on-the-cli +- jonamiki racadm power capping: https://jonamiki.com/2014/10/18/power-control-and-capping-on-dell-servers-using-racadm/ +- Dell InfoHub power reporting: https://infohub.delltechnologies.com/en-us/p/server-power-consumption-reporting-and-management/ +- Launchpad RAPL domains: https://bugs.launchpad.net/bugs/1467014 +- Hackenberg et al. Haswell RAPL survey: https://tu-dresden.de/zih/forschung/ressourcen/dateien/projekte/firestarter/2015_hackenberg_hppac.pdf +- kernel mei/iamt docs: https://docs.kernel.org/driver-api/mei/iamt.html + +> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328 diff --git a/docs/report-moonlight-desktop.md b/docs/report-moonlight-desktop.md new file mode 100644 index 0000000..efbfd0b --- /dev/null +++ b/docs/report-moonlight-desktop.md @@ -0,0 +1,68 @@ +# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s + +Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826. +Source research: web-surveyed 2026-09-06 (sources linked at the end). + +## Context constraints (verified) + +- Xeon E5-2630 v2 (pfv-tsys7 class) has **no iGPU / no Quick Sync** — hardware + video encode is impossible on the current CPU-only nodes. Software x264 (or a + future NVIDIA node's NVENC) is the only path. +- Current access path is xrdp/KDE over Tailscale at ~36 ms RTT. Protocol choice + matters less than encoder cost and whether the Tailscale path is direct vs + DERP-relayed (DERP caps ~5 Mbps and adds latency — check `tailscale netcheck` + before blaming the desktop stack). +- The cluster is CPU-only today, mixed with production workloads, flux gitops — + heavy sustained CPU encoding on shared nodes is an operational risk. + +## Comparison + +| | Sunshine+Moonlight | Selkies-GStreamer | Kasm Workspaces | Webtop / Guacamole | Tuned xrdp (baseline) | +|---|---|---|---|---|---| +| Transport | Moonlight protocol, H.264/265/AV1 | WebRTC (browser) | KasmVNC over WebSocket | VNC/RDP → WebSocket | RDP | +| GPU needed | No (x264 soft mode) but costly | No (x264 soft) | No | No | No | +| CPU cost on old Xeon | HIGH: 2–4 cores sustained @1080p60 | HIGH + WebRTC stack ~1 core | MODERATE (framebuffer diff, no video encode) | LOW–MODERATE (Guacamole ~15–27%/core per 1–2 users) | LOW | +| Latency @36 ms RTT | Best-in-class on direct path; 100–200 ms if misconfigured | Low | Moderate; fine for desktop, visible on video | Moderate–high (protocol translation + browser) | Moderate (tuned RDP is decent) | +| Client | Native Moonlight apps (excellent) — NOT browser | Any browser | Any browser | Any browser | Any RDP client | +| k8s fit | Awkward: privileged pod + `/dev/uinput` hostDevice, dummy X | Good (purpose-built for k8s) | Heavy control plane for one desktop | Trivial pod + PVC (webtop); Guacamole = small extra stack | Stays a VM | +| Persistence | StatefulSet + PVC | PVC | Disposable by design | webtop PVC-backed | Full VM (best) | +| GitOps friendliness | Medium | High | Medium | High | N/A | + +## Recommendation + +1. **Now:** keep the tuned xrdp/KDE VM as baseline; do NOT put Sunshine/Moonlight + on CPU-only nodes — real-time x264 eats 2–4 cores next to production tenants. +2. **Migration path:** deploy **linuxserver/webtop (KasmVNC variant)** as a + PVC-backed StatefulSet on a CPU-only node behind the existing ingress + (browser access; WAN via Tailscale). This is the containerized successor for + desktop-class use at 36 ms RTT: cheapest CPU, unprivileged pod, clean + flux fit. +3. **WAN/browser fallback for the existing RDP VM:** small Apache Guacamole + stack (guacd + guacamole), accepting it is slower than native RDP clients. +4. **When the NVIDIA node lands:** revisit **Selkies-GStreamer** (most + k8s-native low-latency option; NVENC removes the encoder cost) as the + premium tier. **Sunshine+Moonlight** earns its complexity (privileged pod, + /dev/uinput) only if gaming-grade latency on native clients becomes a hard + requirement — then pin it to the GPU node only, never shared CPU nodes. + +Rollout: tuned xrdp VM stays during migration → webtop StatefulSet UAT → +retire VM after human UAT → Selkies on the future GPU node. + +## Sources + +- Intel Xeon Quick Sync support: https://www.intel.com/content/www/us/en/support/articles/000034104/processors/intel-xeon-processors.html +- Sunshine advanced usage (encoder cost): https://docs.lizardbyte.dev/projects/sunshine/v0.23.0/about/advanced_usage.html +- Moonlight setup guide: https://github.com/moonlight-stream/moonlight-docs/wiki/Setup-Guide +- Tailscale DERP 5 Mbps gotcha: https://cfreeman.cloud/breaking-the-5-mbps-barrier-streaming-moonlight-over-tailscale-with-full-bandwidth/ +- LizardByte latency discussion: https://github.com/orgs/LizardByte/discussions/17 +- LXC headless Sunshine/uinput: https://discuss.linuxcontainers.org/t/headless-wayland-container-streaming-via-sunshine-sway-libinput-not-finding-uinput-devices/18852 +- steam-headless uinput issue: https://github.com/Steam-Headless/docker-steam-headless/issues/191 +- Selkies-GStreamer: https://github.com/selkies-project/selkies +- Kasm desktop images: https://hub.docker.com/r/kasmweb/desktop +- linuxserver/docker-webtop: https://github.com/linuxserver/docker-webtop +- LSIO docker-kasm: https://docs.linuxserver.io/images/docker-kasm/ +- Guacamole web-client slowness: https://stackoverflow.com/questions/77224933/apache-guacamole-much-slower-on-the-web-client-than-microsoft-desktop +- guacd CPU benchmarks: https://lists.apache.org/thread/zdlsnn2br4gvtx09q6zqtxym6dwlrjwc +- Cendio on browser remote desktop limits: https://www.cendio.com/blog/apache-guacamole-alternatives/ + +> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328 diff --git a/loop/baseline-diff.sh b/loop/baseline-diff.sh new file mode 100755 index 0000000..fd760b9 --- /dev/null +++ b/loop/baseline-diff.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# KNELPerf baseline diff — compares two baseline bundles (or live-audits one). +# +# baseline-diff.sh compare two stored bundles +# Values are TSV metric/value/unit rows from cpu-bench.tsv. Numeric metrics +# compare with a tolerance (default 10%); non-numeric metrics compare for +# equality (governor flips are exactly what we want to catch). +set -euo pipefail +TOL_PCT=${KNELPERF_DIFF_TOL:-25} # 1-thread openssl on a busy host swings >10% run-to-run +[ $# = 2 ] || { echo "usage: $0 " >&2; exit 2; } +A=$1; B=$2 +[ -r "$A/cpu-bench.tsv" ] || { echo "no cpu-bench.tsv in $A" >&2; exit 3; } +[ -r "$B/cpu-bench.tsv" ] || { echo "no cpu-bench.tsv in $B" >&2; exit 3; } + +join_tsv() { awk -F'\t' 'NR==FNR{a[$1]=$2;next} {print $1"\t"a[$1]"\t"$2"\t"$3}' "$1" "$2"; } + +# shellcheck disable=SC2034 # exit-status reserved for future strictness +status=0 +join_tsv "$A/cpu-bench.tsv" "$B/cpu-bench.tsv" | while IFS=$'\t' read -r metric va vb unit; do + [ -n "$va" ] && [ -n "$vb" ] || continue + if [ "$va" = "$vb" ]; then + printf 'OK %-28s %s = %s %s\n' "$metric" "$va" "$vb" "$unit" + elif [ "$va" = "${va#[-+0-9.]}" ] || [ "$vb" = "${vb#[-+0-9.]}" ]; then + printf 'DRIFT %-28s %s -> %s %s\n' "$metric" "$va" "$vb" "$unit" + else + delta=$(awk -v a="$va" -v b="$vb" 'BEGIN{d=(b-a)/((a==0)?1:a)*100; printf "%+.1f", d}') + flag=$(awk -v d="${delta#+}" -v t="$TOL_PCT" 'BEGIN{m=d<0?-d:d; if(m>t) print "DELTA"; else print "OK"}') + if [ "$flag" = OK ]; then + printf 'OK %-28s %s -> %s (%s%%) %s\n' "$metric" "$va" "$vb" "$delta" "$unit" + else + printf 'DELTA %-28s %s -> %s (%s%%) %s\n' "$metric" "$va" "$vb" "$delta" "$unit" + fi + fi +done +echo "tolerance: ${TOL_PCT}% (set KNELPERF_DIFF_TOL to change)" diff --git a/loop/cpu-bench.sh b/loop/cpu-bench.sh new file mode 100755 index 0000000..37ceaf3 --- /dev/null +++ b/loop/cpu-bench.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# KNELPerf CPU benchmark — the missing CPU leg of the perf loop (#826). +# Runs ON the target host (or locally). Zero-install friendly: uses openssl +# (always present) plus sysbench when available. Output = TSV tag/value. +# +# Design notes: +# - openssl speed = single+multi core crypto throughput (stable, comparable) +# - sysbench cpu (if installed) = integer event rate +# - /proc-derived context: governor, PSI at capture time +set -u +DUR=${KNELPERF_CPUBENCH_SECS:-10} + +echo -e "metric\tvalue\tunit" + +gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo none) +echo -e "governor\t$gov\t-" +nproc=$(nproc) +echo -e "nproc\t$nproc\t-" + +# openssl: 1 thread then all threads, aes-256-cbc (avx-sensitive) and sha256. +# Parse: LAST line whose first field is the algo; value = last numeric field +# (per-thread rows precede the aggregate row in -multi output). +for algo in aes-256-cbc sha256; do + v=$(openssl speed -seconds "$DUR" "$algo" 2>/dev/null \ + | awk -v a="$algo" '$1==a{v=$NF} END{print v}') + [ -n "${v:-}" ] && echo -e "openssl_${algo}_1t\t$v\tk/s" + # -multi: per-thread rows precede the aggregate; keep the LAST match + v=$(openssl speed -multi "$nproc" -seconds "$DUR" "$algo" 2>/dev/null \ + | awk -v a="$algo" '$1==a{v=$NF} END{print v}') + [ -n "${v:-}" ] && echo -e "openssl_${algo}_${nproc}t\t$v\tk/s" +done + +# sysbench (optional, if installed) +if command -v sysbench >/dev/null 2>&1; then + ev=$(sysbench cpu --time=$((DUR*1000)) --threads=1 run 2>/dev/null \ + | awk -F'/s|events per second:/{for(i=1;i<=NF;i++) if($i ~ /^[0-9.]+$/){print $i; exit}}') + [ -n "${ev:-}" ] && echo -e "sysbench_cpu_1t\t$ev\tevents/s" + ev=$(sysbench cpu --time=$((DUR*1000)) --threads="$nproc" run 2>/dev/null \ + | awk -F'/s|events per second:/{for(i=1;i<=NF;i++) if($i ~ /^[0-9.]+$/){print $i; exit}}') + [ -n "${ev:-}" ] && echo -e "sysbench_cpu_${nproc}t\t$ev\tevents/s" +else + echo -e "sysbench_cpu\tabsent\t-" +fi + +# PSI snapshot at capture time (context for the numbers above) +for res in cpu memory io; do + v=$(awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' \ + "/proc/pressure/$res" 2>/dev/null || echo 0) + echo -e "psi_${res}_avg60\t$v\t%" +done +exit 0 diff --git a/loop/perf-loop.sh b/loop/perf-loop.sh new file mode 100755 index 0000000..981232f --- /dev/null +++ b/loop/perf-loop.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# KNELPerf loop driver — baseline -> audit/benchmark -> tweak -> re-baseline. +# +# The heart of KNELPerf v2 (#826). One host, one cycle phase at a time: +# +# perf-loop.sh baseline capture baseline bundle (cpu/net/storage/psi) +# perf-loop.sh audit read-only audit vs baseline (alerts on drift) +# perf-loop.sh tweak apply a tweak (from tweaks/) after CR gate +# perf-loop.sh rebaseline capture post-tweak bundle + diff report +# perf-loop.sh compare diff two stored bundles +# +# Bundles live in data/baselines/// and are plain TSV/JSON so diff +# and jq stay the toolchain. Nothing here mutates a host except `tweak`. +set -euo pipefail +REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) +DATA_DIR=${KNELPERF_DATA_DIR:-$REPO_ROOT/data/baselines} +SSH_OPTS=(-o ConnectTimeout=8 -o BatchMode=yes) + +usage() { sed -n '2,12p' "$0"; exit 2; } +[ $# -ge 2 ] || usage +cmd=$1; host=$2 + +now() { date +%Y%m%d-%H%M%S; } +host_dir() { mkdir -p "$DATA_DIR/$1"; } + +run_on_host() { # host script-path args... + local h=$1 script=$2; shift 2 + case $h in + local) bash "$REPO_ROOT/$script" "$@" ;; + *) ssh "${SSH_OPTS[@]}" "root@$h" "bash -s" < "$REPO_ROOT/$script" "$@" ;; + esac +} + +case "$cmd" in +baseline|rebaseline) + ts=$(now); dir="$DATA_DIR/$host/$ts"; host_dir "$host"; mkdir -p "$dir" + echo "== KNELPerf $cmd: $host -> $dir" + # shellcheck disable=SC2043 # single-part list by design; grows as legs are added + for part in cpu-bench; do + run_on_host "$host" "loop/$part.sh" > "$dir/$part.tsv" 2> "$dir/$part.err" \ + || echo "WARN: $part failed (see $dir/$part.err)" + done + # storage + network reuse the existing bench harness outputs + run_on_host "$host" "bench/bench-run.sh" > "$dir/bench-run.tsv" 2> "$dir/bench.err" \ + || echo "WARN: bench-run failed (see $dir/bench.err)" + echo "$cmd $host $ts" > "$dir/MANIFEST" + echo "done: $(ls "$dir")" + ;; +audit) + latest=$(find "$DATA_DIR/$host" -maxdepth 1 -mindepth 1 -type d -printf '%f\n' 2>/dev/null | sort | tail -1) + [ -n "$latest" ] || { echo "no baseline for $host" >&2; exit 3; } + echo "== KNELPerf audit: $host vs baseline $latest" + "$REPO_ROOT/loop/baseline-diff.sh" "$DATA_DIR/$host/$latest" "$host" + ;; +tweak) + [ $# -ge 3 ] || usage; tweak_id=$3 + echo "== KNELPerf tweak $tweak_id on $host (CR gate applies - see AGENTS.md)" + run_on_host "$host" "tweaks/$tweak_id.sh" + ;; +compare) + [ $# -ge 4 ] || usage + "$REPO_ROOT/loop/baseline-diff.sh" "$DATA_DIR/$host/$3" "$DATA_DIR/$host/$4" + ;; +*) usage ;; +esac diff --git a/scheduler/README.md b/scheduler/README.md new file mode 100644 index 0000000..90a8e78 --- /dev/null +++ b/scheduler/README.md @@ -0,0 +1,51 @@ +# KNELPerf scheduler — breakout from Ultix (#826) + +The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`) +was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains +the live workstation deployment; this module is the going-forward home. + +## What was ported and what changed + +| Ultix source | KNELPerf destination | Redesign delta | +|---|---|---| +| `staged/ukrrs-daynight.sh` | `daynight/knelperf-daynight.sh` | Slice targets, devices, and every knob are conf-driven lists (was: hardcoded `reachableceo-*` slices, `/dev/sdb`, user 1001). Slice specs are `slice:weight:cpus:memhigh` so any host participates. | +| `staged/etc-ukrrs-daynight.conf` | `daynight/knelperf-daynight.conf.example` | Self-documenting example; post-upgrade values live per-host, not in comments of the engine. | +| `staged/systemd/reachableceo-{day,night}profile.*` | `daynight/systemd/knelperf-*` | Generic units; the conf — never a sed-edit of a live timer — is the only value source. | +| `staged/reachableceo-perfsnap.sh` | `collectors/knelperf-perfsnap.sh` | Watched cgroups/interfaces are env-driven lists; no baked-in uids/NICs. Same pipe-delimited log format. | +| `staged/ukrrs-psi-textfile.sh` | `collectors/knelperf-psi-textfile.sh` | Cgroup glob + metric prefix env-driven; emits for any slice namespace. | +| `staged/proxmox-ctl.sh` | `proxmox/proxmox-ctl.sh` | Ported as-is (was already generic + guarded: PROTECTED vmids, graceful shutdown only, pvesh local-or-token). | + +Not ported (stay workstation-local in ultix): `mkacct.sh` account tiers, +`21-priority-ladder.sh` GUI-first cgroup ladder (host-specific by design), +`gateway-ensure.sh`, RTMP stack. + +## Lessons learned carried forward (from ultix REPORT.md / TRACKING.md) + +1. `Slice=` is create-time only — `set-property` cannot set it; caps go in unit + drop-ins AND live via `set-property --runtime`. +2. `CPUWeight` on user-manager units is inert until `cpu` is in the parent's + `cgroup.subtree_control` — enable it explicitly. +3. IOWeight is inert under the `none` I/O scheduler (virtio) — don't bother. +4. Guests have no cpufreq driver — governor tuning is PVE-host-side only. +5. Reboot must land in DAY mode: slice unit files carry day defaults; the + profile engine only does runtime mutations. Non-negotiable invariant. +6. `Persistent=true` timers replay missed flips at boot (intentional). +7. Duplicated truth is the failure mode: one conf file per host is the only + place values live (the 07:00 sed-edit of a live timer was the wart that + motivated this redesign). +8. Measure before touching; collectors install data, they tune nothing. + +## Fleet rollout model + +- Workstation (ultix-streaming): stays on its live ultix units until this + module is UAT'd, then converges onto knelperf-* units. +- PVE nodes: daynight engine applies to host-level services + VM cgroup + scopes (`qemu.slice` children) via the same conf mechanism; timers install + from `daynight/systemd/`. +- Everything deploys through AWX job templates (KNELIAC) per the 100%-IaC + mandate; this repo supplies the scripts, not ad-hoc ssh. + +See `docs/ARCHITECTURE.md` for the full baseline → audit → tweak → re-baseline +program this module feeds. + +> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328 diff --git a/scheduler/collectors/knelperf-perfsnap.sh b/scheduler/collectors/knelperf-perfsnap.sh new file mode 100755 index 0000000..d2734e8 --- /dev/null +++ b/scheduler/collectors/knelperf-perfsnap.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +# KNELPerf perfsnap — periodic /proc+cgroupfs snapshot collector. +# Ported from ultix staged/reachableceo-perfsnap.sh (#826); generalizations: +# - watched cgroups and interfaces are env/conf-driven lists (no baked-in +# user ids or NIC names) +# - log dir env-driven; same pipe-delimited format for awk analysis +# +# Env (or /etc/knelperf/perfsnap.conf): +# KNELPERF_PERF_LOGDIR default /var/log/knelperf-perf +# KNELPERF_SNAP_CGROUPS space-separated "label:/sys/fs/cgroup/path" list +# KNELPERF_SNAP_IFACES space-separated interface names +# Fields (snap.log): ts|l1|l5|l15|run|blocked|memavail_kb|psi_cpu|psi_mem| +# psi_io|| +# psi_* = avg60 "some" percentages. *_usec = cgroup cpu.stat usage_usec +# (cumulative; analyze deltas). +set -u +CONF=/etc/knelperf/perfsnap.conf +[ -r "$CONF" ] && . "$CONF" +LOGDIR=${KNELPERF_PERF_LOGDIR:-/var/log/knelperf-perf} +SNAP_CGROUPS=${KNELPERF_SNAP_CGROUPS:-} +SNAP_IFACES=${KNELPERF_SNAP_IFACES:-} +mkdir -p "$LOGDIR" +ts=$(date +%s) + +read -r l1 l5 l15 _ < /proc/loadavg +run=$(awk '/procs_running/{print $2}' /proc/stat) +blocked=$(awk '/procs_blocked/{print $2}' /proc/stat) +memavail=$(awk '/MemAvailable/{print $2}' /proc/meminfo) + +psi() { awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "/proc/pressure/$1" 2>/dev/null || echo 0; } + +line="$ts|$l1|$l5|$l15|$run|$blocked|$memavail|$(psi cpu)|$(psi memory)|$(psi io)" + +cgusec() { awk '/^usage_usec/{print $2}' "$1/cpu.stat" 2>/dev/null || echo 0; } +for spec in $SNAP_CGROUPS; do + line="$line|$(cgusec "${spec#*:}")" +done + +nd() { awk -v i="$1" '$1==i":":{print $2" "$10}' /proc/net/dev 2>/dev/null || echo "0 0"; } +for ifc in $SNAP_IFACES; do + read -r rx tx <<< "$(nd "$ifc")" + line="$line|$rx|$tx" +done + +echo "$line" >> "$LOGDIR/snap.log" + +ps -eo pcpu,pid,comm --sort=-pcpu --no-headers | head -5 \ + | awk -v t="$ts" '{print t"|"$1"|"$2"|"$3}' >> "$LOGDIR/procs.log" + +# retention: 14 days; rotate anything over 50MB +find "$LOGDIR" -name '*.log' -mtime +14 -delete 2>/dev/null +for f in "$LOGDIR"/*.log; do + [ "$(wc -c <"$f" 2>/dev/null || echo 0)" -gt 52428800 ] && mv "$f" "$f.1" +done +exit 0 diff --git a/scheduler/collectors/knelperf-psi-textfile.sh b/scheduler/collectors/knelperf-psi-textfile.sh new file mode 100755 index 0000000..716c3b3 --- /dev/null +++ b/scheduler/collectors/knelperf-psi-textfile.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector. +# Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations: +# - cgroup glob is env-driven (any slice namespace, not just ukrrs-*) +# - metric prefix is env-driven (default knelperf_) +set -u +OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile} +CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup} +CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice} +PREFIX=${KNELPERF_PREFIX:-knelperf} +mkdir -p "$OUT_DIR" +tmp=$(mktemp "$OUT_DIR/.knelperf.XXXXXX") + +some60() { # file -> prints avg60 value or nothing + awk '$1=="some"{for(i=2;i<=NF;i++) if($i ~ /^avg60=/){sub("avg60=","",$i); print $i; exit}}' "$1" 2>/dev/null +} + +emit() { # name cgroup_path + local res v + for res in cpu memory io; do + [ -r "$2/$res.pressure" ] || continue + v=$(some60 "$2/$res.pressure") + [ -n "${v:-}" ] && echo "${PREFIX}_slice_pressure_some60{slice=\"$1\",res=\"$res\"} $v" + done +} + +echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp" +echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp" +for cg in $CG_GLOB; do + [ -d "$cg" ] && emit "${cg##*/}" "$cg" +done +[ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice" + +echo "# HELP ${PREFIX}_host_pressure_some60 PSI some avg60 (percent) host-wide" >>"$tmp" +echo "# TYPE ${PREFIX}_host_pressure_some60 gauge" >>"$tmp" +for res in cpu memory io; do + [ -r "/proc/pressure/$res" ] || continue + v=$(some60 "/proc/pressure/$res") + [ -n "${v:-}" ] && echo "${PREFIX}_host_pressure_some60{res=\"$res\"} $v" +done + +mv "$tmp" "$OUT_DIR/${PREFIX}_pressure.prom" +chmod 0644 "$OUT_DIR/${PREFIX}_pressure.prom" diff --git a/scheduler/daynight/knelperf-daynight.conf.example b/scheduler/daynight/knelperf-daynight.conf.example new file mode 100644 index 0000000..9496547 --- /dev/null +++ b/scheduler/daynight/knelperf-daynight.conf.example @@ -0,0 +1,28 @@ +# /etc/knelperf/daynight.conf — knelperf-daynight.sh configuration. +# Copy to /etc/knelperf/daynight.conf and edit per host. Every field is +# optional; omit slices/sections you do not manage on this host. +# +# Slice spec format (space-separated list): +# slice[:CPUWeight[:AllowedCPUs[:MemoryHigh]]] +# Empty field = skip that property. Day AND night lists are required for +# any slice you list (reboot-safety invariant: day values == unit-file +# defaults). + +# Example (workstation, 8 vCPU — mirrors the ultix values): +# DAY_SLICES="reachableceo-batch.slice:25:0-5:12G reachableceo-gateway.slice:900::" +# NIGHT_SLICES="reachableceo-batch.slice:400:0-6:30G reachableceo-gateway.slice:500::" + +# Example (PVE host with a batch pool): +# DAY_SLICES="knelperf-batch.slice:25:0-3:8G" +# NIGHT_SLICES="knelperf-batch.slice:400:0-7:24G" + +# Block devices for night readahead boost (partitions auto-included): +# BLOCK_DEVS="/dev/sdb" + +# Defaults shown in the script; override only what differs: +# DATA_RA_DAY=256 +# DATA_RA_NIGHT=2048 +# DIRTY_DAY=1073741824 +# DIRTY_NIGHT=2147483648 +# DIRTY_BG_DAY=268435456 +# DIRTY_BG_NIGHT=536870912 diff --git a/scheduler/daynight/knelperf-daynight.sh b/scheduler/daynight/knelperf-daynight.sh new file mode 100755 index 0000000..1e01c3b --- /dev/null +++ b/scheduler/daynight/knelperf-daynight.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# KNELPerf day/night resource profile engine. +# +# Broken out of ultix staged/ukrrs-daynight.sh (2026-09-06, #826) and +# redesigned: slice names, devices, and all numeric knobs are CONF-driven +# (no host-specific hardcoding); slice targets are a list so any host can +# run it. Runtime-only changes (set-property --runtime + sysctl + readahead): +# a reboot always lands safely in day mode, because slice unit files carry +# the day defaults (unchanged ultix invariant). +# +# Usage: knelperf-daynight.sh [--conf FILE] day|night +set -euo pipefail + +CONF=${KNELPERF_DAYNIGHT_CONF:-/etc/knelperf/daynight.conf} +while [ $# -gt 0 ]; do + case "$1" in + --conf) CONF=$2; shift 2 ;; + day|night) MODE=$1; shift ;; + *) echo "usage: $0 [--conf FILE] day|night" >&2; exit 2 ;; + esac +done +[ "${MODE:-}" ] || { echo "usage: $0 [--conf FILE] day|night" >&2; exit 2; } +[ -r "$CONF" ] && . "$CONF" + +# Targets: "slice:CPUWeight:AllowedCPUs:MemoryHigh" per mode, space-separated. +# Empty fields mean "skip that property". See knelperf-daynight.conf.example. +: "${DAY_SLICES:=}"; : "${NIGHT_SLICES:=}" +# Block devices for readahead tuning (space-separated; partitions auto-appended). +: "${BLOCK_DEVS:=}" +: "${DATA_RA_DAY:=256}"; : "${DATA_RA_NIGHT:=2048}" +: "${DIRTY_DAY:=1073741824}"; : "${DIRTY_NIGHT:=2147483648}" +: "${DIRTY_BG_DAY:=268435456}"; : "${DIRTY_BG_NIGHT:=536870912}" + +log() { echo "[knelperf-daynight] $*"; } +ra=$DATA_RA_DAY; dirty=$DIRTY_DAY; dirtybg=$DIRTY_BG_DAY; slices=$DAY_SLICES +[ "$MODE" = night ] && { ra=$DATA_RA_NIGHT; dirty=$DIRTY_NIGHT; dirtybg=$DIRTY_BG_NIGHT; slices=$NIGHT_SLICES; } + +apply_slice() { # "slice:weight:cpus:memhigh" + local spec=$1 slice weight cpus memhigh + IFS=: read -r slice weight cpus memhigh <<< "$spec" + [ -n "$slice" ] || return 0 + local -a props=() + [ -n "$weight" ] && props+=(CPUWeight="$weight") + [ -n "$cpus" ] && props+=(AllowedCPUs="$cpus") + [ -n "$memhigh" ] && props+=(MemoryHigh="$memhigh") + if [ ${#props[@]} -gt 0 ]; then + systemctl set-property --runtime "$slice" "${props[@]}" \ + || log "WARN: set-property failed for $slice (slice not loaded?)" + fi +} + +for spec in $slices; do apply_slice "$spec"; done + +if [ -n "$BLOCK_DEVS" ]; then + for dev in $BLOCK_DEVS; do + blockdev --setra "$ra" "$dev" 2>/dev/null || log "readahead skip: $dev" + for part in "${dev}"[0-9]*; do + [ -e "$part" ] && blockdev --setra "$ra" "$part" 2>/dev/null || true + done + done +fi +sysctl -q -w vm.dirty_bytes="$dirty" vm.dirty_background_bytes="$dirtybg" +log "profile $MODE applied $(date -Is)" diff --git a/scheduler/daynight/systemd/README.md b/scheduler/daynight/systemd/README.md new file mode 100644 index 0000000..588f42d --- /dev/null +++ b/scheduler/daynight/systemd/README.md @@ -0,0 +1,24 @@ +# systemd units — day/night profile timers + +Install per host (adjust names to your unit namespace if not knelperf-): + + install -m 0644 knelperf-dayprofile.service /etc/systemd/system/ + install -m 0644 knelperf-dayprofile.timer /etc/systemd/system/ + install -m 0644 knelperf-nightprofile.service /etc/systemd/system/ + install -m 0644 knelperf-nightprofile.timer /etc/systemd/system/ + systemctl daemon-reload + systemctl enable --now knelperf-dayprofile.timer knelperf-nightprofile.timer + +Design notes (carried from ultix, keep true here): + +- `Persistent=true` — a missed flip (host powered off) replays at next boot, + so the host can never get stuck in the wrong profile across a reboot cycle. +- Services are `Type=oneshot` calling the profile engine with the SAME conf + file; the conf — not the timer — is the only place values live (fixes the + ultix wart where the day-flip time was sed-edited on the live timer). +- Day schedule 07:00, night 22:00 local time (batch window 22:00–07:00). + Change the OnCalendar here when a host needs a different window; the timers + stay trivial and identical in shape. + +Reboot-safety invariant: slice unit files must carry the DAY values as their +static defaults, so an unattended reboot always lands in day mode. diff --git a/scheduler/daynight/systemd/knelperf-dayprofile.service b/scheduler/daynight/systemd/knelperf-dayprofile.service new file mode 100644 index 0000000..21517f8 --- /dev/null +++ b/scheduler/daynight/systemd/knelperf-dayprofile.service @@ -0,0 +1,9 @@ +[Unit] +Description=KNELPerf day resource profile (interactive-first) +After=multi-user.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/knelperf-daynight.sh day +# Never restart a profile flip; the timer owns scheduling +Restart=no diff --git a/scheduler/daynight/systemd/knelperf-dayprofile.timer b/scheduler/daynight/systemd/knelperf-dayprofile.timer new file mode 100644 index 0000000..f9dd95d --- /dev/null +++ b/scheduler/daynight/systemd/knelperf-dayprofile.timer @@ -0,0 +1,10 @@ +[Unit] +Description=KNELPerf day profile flip (07:00 local) + +[Timer] +OnCalendar=*-*-* 07:00:00 +Persistent=true +Unit=knelperf-dayprofile.service + +[Install] +WantedBy=timers.target diff --git a/scheduler/daynight/systemd/knelperf-nightprofile.service b/scheduler/daynight/systemd/knelperf-nightprofile.service new file mode 100644 index 0000000..f57fe7e --- /dev/null +++ b/scheduler/daynight/systemd/knelperf-nightprofile.service @@ -0,0 +1,8 @@ +[Unit] +Description=KNELPerf night resource profile (batch window 22:00-07:00) +After=multi-user.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/knelperf-daynight.sh night +Restart=no diff --git a/scheduler/daynight/systemd/knelperf-nightprofile.timer b/scheduler/daynight/systemd/knelperf-nightprofile.timer new file mode 100644 index 0000000..4e0bbbd --- /dev/null +++ b/scheduler/daynight/systemd/knelperf-nightprofile.timer @@ -0,0 +1,10 @@ +[Unit] +Description=KNELPerf night profile flip (22:00 local) + +[Timer] +OnCalendar=*-*-* 22:00:00 +Persistent=true +Unit=knelperf-nightprofile.service + +[Install] +WantedBy=timers.target diff --git a/scheduler/proxmox/proxmox-ctl.sh b/scheduler/proxmox/proxmox-ctl.sh new file mode 100755 index 0000000..6501f88 --- /dev/null +++ b/scheduler/proxmox/proxmox-ctl.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# UKRRS Proxmox control wrapper for harness/human use. HARD RULES baked in: +# - vmids listed in PROTECTED are never touched (list THIS VM first). +# - graceful shutdown only; no reset, no hard stop, no bare down (house rule). +# Env: +# PVE_LOCAL=1 run ON the pve host itself (pvesh direct) +# or PVE_HOST + PVE_TOKEN remote, token format: user@realm!tokenid=secret +# PVE_NODE node name (required for vm ops) +# PROTECTED required, comma-separated vmids +# Usage: +# proxmox-ctl.sh vms +# proxmox-ctl.sh shutdown +# proxmox-ctl.sh start +# proxmox-ctl.sh snapshot +# Token setup (on the pve host, see docs/REPORT.md appendix B): +# pveum user add ukrrs-infra@pam +# pveum user token add ukrrs-infra@pam harness -privsep 0 -expire 0 +# pveum acl modify /pool/ -user ukrrs-infra@pam -role PVEVMUser +set -euo pipefail +: "${PROTECTED:?set PROTECTED=vmid1,vmid2,... (this VM must be in the list)}" + +pvesh_() { + if [ "${PVE_LOCAL:-0}" = 1 ]; then + command pvesh "$@" + else + : "${PVE_HOST:?}" "${PVE_TOKEN:?}" + command pvesh --host "$PVE_HOST" --api-token "$PVE_TOKEN" "$@" + fi +} + +guard() { + case ",$PROTECTED," in + *",$1,"*) echo "REFUSED: vmid $1 is PROTECTED" >&2; exit 3 ;; + esac +} + +cmd=${1:-}; shift || true +case "$cmd" in + vms) + pvesh_ get /cluster/resources --type vm ;; + shutdown) + [ $# = 1 ] || { echo "usage: $0 shutdown " >&2; exit 2; } + guard "$1"; : "${PVE_NODE:?}" + pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/shutdown" --timeout 120 ;; + start) + [ $# = 1 ] || { echo "usage: $0 start " >&2; exit 2; } + guard "$1"; : "${PVE_NODE:?}" + pvesh_ create "/nodes/$PVE_NODE/qemu/$1/status/start" ;; + snapshot) + [ $# = 2 ] || { echo "usage: $0 snapshot " >&2; exit 2; } + guard "$1"; : "${PVE_NODE:?}" + pvesh_ create "/nodes/$PVE_NODE/qemu/$1/snapshot" snapname="$2" ;; + *) + echo "usage: $0 vms|shutdown |start |snapshot " >&2 + exit 2 ;; +esac