KNELPerf/docs/ARCHITECTURE.md

KNELPerf v2 — architecture for dynamic, intelligent performance optimization

Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end baseline → audit/benchmark → tweak → re-baseline loop, two-week program.

1. The loop

        ┌──────────────────────────────────────────────────┐
        │                                                  │
   BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
   (capture)     (diff + beszel/       (CR-gated,   (compare vs
                 PSI/rapl context)     idempotent)  baseline; keep
                                                     or roll back)

• baseline: loop/perf-loop.sh baseline <host> stores a bundle (cpu-bench.tsv + bench-run.tsv + PSI snapshot) under data/baselines/<host>/<ts>/. Storage/network legs reuse bench/bench-run.sh (fio + iperf3); the CPU leg (loop/cpu-bench.sh) is new — the repo previously had zero CPU coverage.
• audit: loop/perf-loop.sh audit <host> re-runs the read-only probes and diffs against the latest bundle via loop/baseline-diff.sh (tolerance-based, catches governor flips exactly). This is what the scheduled loop runs.
• tweak: every mutation is a file under tweaks/<id>.sh (idempotent, rollback helper included). Production tweaks ride a GLPI CR; the loop refuses tweak on prod hosts without .crush/active-cr.
• re-baseline: same capture, then compare — the delta table is the evidence artifact that lands on the Redmine ticket.

Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring: the audit itself is low-cost); re-baseline only after a tweak or hardware change. First full-fleet baseline wave is this week's deliverable.

2. Scheduling architecture (Ultix breakout redesign)

See scheduler/README.md for the port map and the lessons-learned list. Core redesign decisions:

1. Single source of truth per host — /etc/knelperf/daynight.conf; no value ever lives in a timer file, unit default, or script constant.
2. Reboot lands in day mode — invariant carried from Ultix.
3. Fleet-generic: slice specs are data (slice:weight:cpus:memhigh), so PVE hosts tune qemu.slice children and host services with the same engine the workstation uses.
4. Deployed by AWX (KNELIAC), not ad-hoc ssh — scripts here are the payload; job templates own the rollout waves.

3. Proxmox resource groups & VM balancing

Current fleet (dmidecode-verified 2026-09-06):

node	platform	cores	RAM	storage character
pfv-tsys1	OptiPlex 9020 (i7-4770)	8	32G	1x 1TB spinner — edge role
pfv-tsys3	Precision 7510 (E3-1535M v5)	8	32G	NVMe (PM961) — fast singleton
pfv-tsys4	Precision T1700 (E3-1246 v3)	8	16G	6 spindles + 3 more USB toasters coming = spindle farm
pfv-tsys5	Precision T7500 (E5620)	8	96G	mixed; NFS server for -02 (NEVER casually rebooted)
pfv-tsys6	PowerEdge R610 (2x E5530)	16	128G	1x 2TB spinner
pfv-tsys7	PowerEdge R620 (2x E5-2630 v2)	24	192G	1x 2TB spinner
pfv-tsys8 (new)	Lenovo, NVMe, 12G	?	12G	fast NVMe, small — scheduler/quorum/edge
pfv-tsys2 (soon)	Precision 7510 clone + NVMe	8	32G	NVMe

Directions:

• Proxmox resource groups / pools: map pools to tenant classes (see §6), so scheduler policy attaches to the pool, not per-VM.
• Balancing across hosts AND spindles: nodes are standalone (no corosync/HA), so "balancing" = placement policy at create/migrate time, not live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6; RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8, tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with one-job-per-spindle placement (fio baselines per spindle will quantify). The audit loop's beszel/PSI data feeds a placement scorecard.
• tsys6 vs tsys7 split: RECOMMEND tsys7 = the big k8s worker (24c/192G — largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.). Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and the core count to soak k8s bins; tsys6's older CPUs are fine for steady-state VMs but it lacks DCMI telemetry on old firmware. Decision needs founder ruling — will be proposed on #826 with a capacity table once tsys8/2 land.

4. Where k8s scheduling fits; Slurm?

• k8s (pfv-k8s, k3s) owns long-running services + tenant classes (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for class 1–2 workloads; node labels/taints pin tenant classes to node pools.
• HFNOC batch (class 3) = interruptible, deadline-loose, large — this is HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics (preemption via priority classes — fits "start and stop easily"); (b) Slurm on a dedicated partition across idle capacity. Recommendation: start with k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm co-located with k8s on shared nodes via cgroup slices is possible (knelperf-batch.slice) but doubles scheduling systems for little gain at this fleet size.
• The knelperf daynight engine gives the batch window (22:00–07:00) its expanded resource envelope on each participating node.

5. Telemetry spine: beszel + power

• Beszel has 100% VM+physical coverage → it is the always-on signal for the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics feed the placement scorecard; PSI textfile collector (scheduler/collectors/) adds pressure (queueing) semantics beszel lacks.
• Power telemetry (see docs/report-amt-power-telemetry.md): per-node watts come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5 needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the un-metered UPS, everything else on the metered one) is compatible: the nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.

6. Workload priority classes (founder-set order)

class	tenants	profile	scheduling
1	RackRental, Side Door Group, Starting Line (paid reservations)	OLTP, latency-sensitive, high margin	guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7)
2	Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center)	bursty, business hours	Burstable QoS; preemptable by class 1
3	HFNOC batch	GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning	night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2

7. Solar + Home Assistant integration

• SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home Assistant exposes real-time production. The loop consumes: surplus watts = green light to start class-3 batches (excess production is the cheapest compute on earth); forecast + reservation calendar (KillBill) → promotion timing advice ("run the Side Door promo the week of X: projected surplus + idle capacity Y").
• Integration path: HA REST/websocket (creds pattern exists: ~/.creds/homeassistant.env, tools/ha-ws-call.py in KNELBMS) → a small exporter writing knelperf_solar_surplus_watts into the textfile collector's directory; the batch launcher gates on it.
• SITER-Solar math review (2026-09-06): found constant drift ($301.08/23952 vs plan's $264.47/22614), a stale Dockerfile entrypoint (deprecated python script), header/ROI inconsistencies, and fixed-cost scenario assumptions — fixes landing in that repo.

8. Two-site power economics (site 2: printer farm / laser / CNC)

The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on MUCH more expensive grid power than the main site. That asymmetry drives scheduling policy as much as raw performance:

• Marginal-price model: every site has an effective $/kWh curve. Main site = cheap grid + (soon) solar with export at $0.04 and self-use at $0.085 (SITER-Solar contract). Site 2 = expensive grid. The scheduler's objective becomes: run flexible load (compute AND manufacturing) where the marginal $/kWh is lowest at that hour.
• Solar surplus is main-site-first: excess production self-consumed at the main site displaces the CHEAPEST power — class-3 batch compute and any deferrable main-site load should chase the surplus curve.
• Site-2 manufacturing scheduling: printer/laser/CNC jobs are batchy and often deferrable (prints especially — hours-long, unattended). Candidate policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows (if its tariff has TOU) or batch them for periods when main-site solar surplus can't absorb more compute. Home Assistant + metered UPS legs give the measured baseline; site 2 needs its own metering (smart plug / CT per machine) before scheduling — queued as a data dependency.
• Never shift OLTP: class-1 paid workloads stay on their reserved capacity regardless of power price — revenue and latency beat energy cost.
• KillBill reservation calendar × solar forecast × site-2 tariff = the promotion/job-timing advisor output: "cheapest week to run X".

9. Hardware plan hooks

• Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins are placement + spindles (tsys4's 3 USB toasters join the spindle farm: baseline each with fio, then one-batch-job-per-spindle).
• tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory row pending (PFVSystemInventory.csv not found in repo — founder to drop in).
• tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair with tsys3 for latency-sensitive class-1 storage.

10. Organization

Performance management org reports to VPTechOps, dotted line VPFacilities — identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI groups). All work filed under #826; CRs for any prod change.

Docs live on Discourse — this repo is the executable source of truth. Perf topic: https://community.turnsys.com/t/328