--- title: The Program visible: true --- # πŸ—οΈ KNELPerf v2 β€” architecture for dynamic, intelligent performance optimization > Status: DRAFT v1 (2026-09-06, > [#826](https://projects.knownelement.com/issues/826)). Founder directive: > end-to-end baseline β†’ audit/benchmark β†’ tweak β†’ re-baseline loop, > two-week program. ## πŸ“‘ Contents 1. [The loop](#1-the-loop) Β· 2. [Scheduling architecture](#2-scheduling-architecture-ultix-breakout-redesign) Β· 3. [Proxmox resource groups & VM balancing](#3-proxmox-resource-groups--vm-balancing) Β· 4. [k8s scheduling & Slurm](#4-where-k8s-scheduling-fits-slurm) Β· 5. [Telemetry spine](#5-telemetry-spine-beszel--power) Β· 6. [Workload priority classes](#6-workload-priority-classes-founder-set-order) Β· 7. [Solar + Home Assistant](#7-solar--home-assistant-integration) Β· 8. [Two-site power economics](#8-two-site-power-economics-site-2-printer-farm--laser--cnc) Β· 9. [Hardware plan hooks](#9-hardware-plan-hooks) Β· 10. [Organization](#10-organization) ## 1. The loop ```mermaid flowchart LR B[πŸ“‹ BASELINE
cpu-bench.tsv + bench-run.tsv
+ PSI snapshot per host] --> A[πŸ” AUDIT
tolerance diff vs bundle
beszel + PSI context] A -->|drift found| T[πŸ”§ TWEAK
tweaks/<id>.sh β€” idempotent,
GLPI CR-gated on prod] T --> R[πŸ“ˆ RE-BASELINE
capture + compare
evidence to Redmine] R --> A A -->|clean| B ``` - **baseline**: `loop/perf-loop.sh baseline ` stores a bundle (cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines///`. Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg (`loop/cpu-bench.sh`) is new β€” the repo previously had zero CPU coverage. - **audit**: `loop/perf-loop.sh audit ` re-runs the read-only probes and diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based, catches governor flips exactly). This is what the scheduled loop runs. - **tweak**: every mutation is a file under `tweaks/.sh` (idempotent, rollback helper included). Production tweaks ride a GLPI CR; the loop refuses `tweak` on prod hosts without `.crush/active-cr`. - **re-baseline**: same capture, then `compare` β€” the delta table is the evidence artifact that lands on the Redmine ticket. Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring: the audit itself is low-cost); re-baseline only after a tweak or hardware change. First full-fleet baseline wave is this week's deliverable. ## 2. πŸ—“οΈ Scheduling architecture (Ultix breakout redesign) See `scheduler/README.md` for the port map and the lessons-learned list. Core redesign decisions: 1. **Single source of truth per host** β€” `/etc/knelperf/daynight.conf`; no value ever lives in a timer file, unit default, or script constant. 2. **Reboot lands in day mode** β€” invariant carried from Ultix. 3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so PVE hosts tune `qemu.slice` children and host services with the same engine the workstation uses. 4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** β€” scripts here are the payload; job templates own the rollout waves. ## 3. πŸ–₯️ Proxmox resource groups & VM balancing Current fleet (dmidecode-verified 2026-09-06): | node | platform | cores | RAM | storage character | |---|---|---|---|---| | pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner β€” edge role | | pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) β€” fast singleton | | pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm | | pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) | | pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner | | pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner | | pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small β€” scheduler/quorum/edge | | pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe | Directions: - **Proxmox resource groups / pools**: map pools to tenant classes (see Β§6), so scheduler policy attaches to the pool, not per-VM. - **Balancing across hosts AND spindles**: nodes are standalone (no corosync/HA), so "balancing" = placement policy at create/migrate time, not live DRRS. Initial heuristic: CPU-bound guests β†’ tsys7 (24c) then tsys6; RAM-hungry β†’ tsys7/6/5; IO-latency-sensitive β†’ NVMe nodes (tsys3, tsys8, tsys2); sequential-throughput batch (HFNOC ETL) β†’ tsys4 spindle farm with one-job-per-spindle placement (fio baselines per spindle will quantify). The audit loop's beszel/PSI data feeds a placement scorecard. - **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G β€” largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.). Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (Β§5) and the core count to soak k8s bins; tsys6's older CPUs are fine for steady-state VMs but it lacks DCMI telemetry on old firmware. Decision needs founder ruling β€” will be proposed on #826 with a capacity table once tsys8/2 land. ## 4. ☸️ Where k8s scheduling fits; Slurm? - **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for class 1–2 workloads; node labels/taints pin tenant classes to node pools. - **HFNOC batch (class 3)** = interruptible, deadline-loose, large β€” this is HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics (preemption via priority classes β€” fits "start and stop easily"); (b) Slurm on a dedicated partition across idle capacity. Recommendation: start with k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm co-located with k8s on shared nodes via cgroup slices is possible (knelperf-batch.slice) but doubles scheduling systems for little gain at this fleet size. - The knelperf daynight engine gives the batch window (22:00–07:00) its expanded resource envelope on each participating node. ## 5. πŸ“‘ Telemetry spine: beszel + power - **Beszel has 100% VM+physical coverage** β†’ it is the always-on signal for the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics feed the placement scorecard; PSI textfile collector (`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks. - **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5 needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the un-metered UPS, everything else on the metered one) is compatible: the nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes. - **Load-side telemetry = per-circuit monitoring (#623, Emporia Vue 3 pending founder pick)**: per-circuit watts via the HA LOCAL (HACS) integration β†’ knelperf textfile exporter β†’ the class-3 batch gate gets a real deferrable-load menu (server-room subpanel vs HVAC vs site-2 equipment). Dedicated CTs on the server-room subpanel + the tsys6/7 UPS legs keep fleet draw separable from house load β€” needed exactly on EEA days, so local (non-cloud) telemetry matters. ## 6. πŸ† Workload priority classes (founder-set order) | class | tenants | profile | scheduling | |---|---|---|---| | 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar β†’ capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (Β§7) | | 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 | | 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 | ## 7. β˜€οΈ Solar + Home Assistant integration - SITER-Solar (4 kW baseline design, PVWatts) β†’ once the plant is live, Home Assistant exposes real-time production. The loop consumes: surplus watts = green light to start class-3 batches (excess production is the cheapest compute on earth); forecast + reservation calendar (KillBill) β†’ promotion timing advice ("run the Side Door promo the week of X: projected surplus + idle capacity Y"). - Integration path: HA REST/websocket (creds pattern exists: `~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) β†’ a small exporter writing `knelperf_solar_surplus_watts` into the textfile collector's directory; the batch launcher gates on it. - SITER-Solar math review (2026-09-06): found constant drift ([redacted β€” LLC-private]/23952 vs plan's [redacted β€” LLC-private]/22614), a stale Dockerfile entrypoint (deprecated python script), header/ROI inconsistencies, and fixed-cost scenario assumptions β€” fixes landing in that repo. ## 8. 🏭 Two-site power economics (site 2: printer farm / laser / CNC) The fleet has a second site with 3D-printer farm, laser cutter, and CNC β€” on MUCH more expensive grid power than the main site. That asymmetry drives scheduling policy as much as raw performance: - **Marginal-price model**: every site has an effective $/kWh curve. Main site = cheap grid + (soon) solar with export at $[redacted β€” LLC-private]and self-use at $[redacted β€” LLC-private] (SITER-Solar contract). Site 2 = expensive grid. The scheduler's objective becomes: run flexible load (compute AND manufacturing) where the marginal $/kWh is lowest at that hour. - **Solar surplus is main-site-first**: excess production self-consumed at the main site displaces the CHEAPEST power β€” class-3 batch compute and any deferrable main-site load should chase the surplus curve. - **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and often deferrable (prints especially β€” hours-long, unattended). Candidate policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows (if its tariff has TOU) or batch them for periods when main-site solar surplus can't absorb more compute. Home Assistant + metered UPS legs give the measured baseline; site 2 needs its own metering (smart plug / CT per machine) before scheduling β€” queued as a data dependency. - **Never shift OLTP**: class-1 paid workloads stay on their reserved capacity regardless of power price β€” revenue and latency beat energy cost. - KillBill reservation calendar Γ— solar forecast Γ— site-2 tariff = the promotion/job-timing advisor output: "cheapest week to run X". ## 9. πŸ”© Hardware plan hooks - Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins are placement + spindles (tsys4's 3 USB toasters join the spindle farm: baseline each with fio, then one-batch-job-per-spindle). - tsys8 (Lenovo NVMe 12G) β†’ scheduler/edge roles + fast NVMe pool; inventory row pending (PFVSystemInventory.csv not found in repo β€” founder to drop in). - tsys2 (Precision 7510 clone + NVMe) β†’ second fast singleton; NVMe pair with tsys3 for latency-sensitive class-1 storage. ## 10. πŸ‘₯ Organization Performance management org reports to VPTechOps, dotted line VPFacilities β€” identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI groups). All work filed under #826; CRs for any prod change. > **Docs live on Discourse β€” this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328 ## 11. πŸ“Š Capacity planning & margin tracking in Dolibarr ERP landscape: 19 Dolibarr instances exist (one per brand); **erp.knownelement.com** is the mothership (v24.0.0, REST API live, perf-org API identity `vpperf` β€” creds `~/.creds/dolibarr-perf.env`). KillBill stays the subscription billing engine; Dolibarr is the financial brain where capacity, cost, and margin meet. ### Capacity ledger (supply) - Fleet capacity is derived from ground truth we already collect: GLPI CMDB + dmidecode survey + baseline bundles β†’ per-node vCPU/RAM/NVMe/spindle inventory (ARCHITECTURE Β§3 table is the seed). - Represented in Dolibarr as **virtual stock**: one product per capacity SKU per node class (e.g. `cap-oltp-4c8g-nvme`, `cap-batch-spindle`, `cap-k8s-worker-core`). Stock quantity = bookable units; GLPI drift report reconciles it (the nightly reconcile already exists for CIs β€” extend to capacity products). ### Reservations (demand) - Paid reservations (class 1) = Dolibarr customer contracts/order lines that consume capacity stock; KillBill invoices the money, Dolibarr holds the commitment. Utilization % = booked/available per SKU per week. - The promotion advisor consumes exactly this: KillBill reservation calendar x projected solar surplus x utilization headroom (Β§7). ### Margin tracking (profitability) - **Revenue** per tenant: KillBill (exports/journal) β†’ Dolibarr customer invoices/analytic accounts. - **Direct costs** per tenant class: power (both sites β€” utility bills #827; solar self-consumption at contract rate), hardware amortization (post 2026-10-11 purchases), WAN/tailnet overheads. - Dolibarr **analytic accounting** (accounts + analytic distribution tables are already present) allocates costs to tenant classes β†’ margin per class per month. PRIVATE dimension: margin numbers are revenue/expense detail β€” LLC-members-only, never published to the public hub (transparency policy). ### Integration plumbing - `loop/` baselines + beszel utilization β†’ utilization updater script β†’ Dolibarr API (product stock / extrafields) β€” read/write via vpperf key. - Hub publishes ONLY utilization % and capacity counts (public-safe); margins stay inside the ERP. API gotchas learned (2026-09-06): table prefix is `dolibarr_` (not `llx_`); API auth needs `DOLAPIKEY` header; a cloned user needs explicit `dolibarr_user_rights` rows copied (admin flag alone does not grant API module rights).