ic-builder
e62c19df43
ci / audit (push) Successful in 23s
https://projects.knownelement.com/issues/826#note-9
185 lines
11 KiB
Markdown
185 lines
11 KiB
Markdown
# KNELPerf v2 — architecture for dynamic, intelligent performance optimization
|
||
|
||
Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end
|
||
baseline → audit/benchmark → tweak → re-baseline loop, two-week program.
|
||
|
||
## 1. The loop
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────┐
|
||
│ │
|
||
BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
|
||
(capture) (diff + beszel/ (CR-gated, (compare vs
|
||
PSI/rapl context) idempotent) baseline; keep
|
||
or roll back)
|
||
```
|
||
|
||
- **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
|
||
(cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines/<host>/<ts>/`.
|
||
Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg
|
||
(`loop/cpu-bench.sh`) is new — the repo previously had zero CPU coverage.
|
||
- **audit**: `loop/perf-loop.sh audit <host>` re-runs the read-only probes and
|
||
diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based,
|
||
catches governor flips exactly). This is what the scheduled loop runs.
|
||
- **tweak**: every mutation is a file under `tweaks/<id>.sh` (idempotent,
|
||
rollback helper included). Production tweaks ride a GLPI CR; the loop
|
||
refuses `tweak` on prod hosts without `.crush/active-cr`.
|
||
- **re-baseline**: same capture, then `compare` — the delta table is the
|
||
evidence artifact that lands on the Redmine ticket.
|
||
|
||
Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
|
||
the audit itself is low-cost); re-baseline only after a tweak or hardware
|
||
change. First full-fleet baseline wave is this week's deliverable.
|
||
|
||
## 2. Scheduling architecture (Ultix breakout redesign)
|
||
|
||
See `scheduler/README.md` for the port map and the lessons-learned list. Core
|
||
redesign decisions:
|
||
|
||
1. **Single source of truth per host** — `/etc/knelperf/daynight.conf`; no
|
||
value ever lives in a timer file, unit default, or script constant.
|
||
2. **Reboot lands in day mode** — invariant carried from Ultix.
|
||
3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so
|
||
PVE hosts tune `qemu.slice` children and host services with the same engine
|
||
the workstation uses.
|
||
4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
|
||
payload; job templates own the rollout waves.
|
||
|
||
## 3. Proxmox resource groups & VM balancing
|
||
|
||
Current fleet (dmidecode-verified 2026-09-06):
|
||
|
||
| node | platform | cores | RAM | storage character |
|
||
|---|---|---|---|---|
|
||
| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role |
|
||
| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton |
|
||
| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm |
|
||
| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) |
|
||
| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner |
|
||
| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner |
|
||
| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge |
|
||
| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe |
|
||
|
||
Directions:
|
||
- **Proxmox resource groups / pools**: map pools to tenant classes (see §6),
|
||
so scheduler policy attaches to the pool, not per-VM.
|
||
- **Balancing across hosts AND spindles**: nodes are standalone (no
|
||
corosync/HA), so "balancing" = placement policy at create/migrate time, not
|
||
live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6;
|
||
RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8,
|
||
tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with
|
||
one-job-per-spindle placement (fio baselines per spindle will quantify).
|
||
The audit loop's beszel/PSI data feeds a placement scorecard.
|
||
- **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G —
|
||
largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.).
|
||
Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and
|
||
the core count to soak k8s bins; tsys6's older CPUs are fine for
|
||
steady-state VMs but it lacks DCMI telemetry on old firmware. Decision
|
||
needs founder ruling — will be proposed on #826 with a capacity table once
|
||
tsys8/2 land.
|
||
|
||
## 4. Where k8s scheduling fits; Slurm?
|
||
|
||
- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
|
||
(ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
|
||
class 1–2 workloads; node labels/taints pin tenant classes to node pools.
|
||
- **HFNOC batch (class 3)** = interruptible, deadline-loose, large — this is
|
||
HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics
|
||
(preemption via priority classes — fits "start and stop easily"); (b) Slurm
|
||
on a dedicated partition across idle capacity. Recommendation: start with
|
||
k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only
|
||
introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm
|
||
co-located with k8s on shared nodes via cgroup slices is possible
|
||
(knelperf-batch.slice) but doubles scheduling systems for little gain at
|
||
this fleet size.
|
||
- The knelperf daynight engine gives the batch window (22:00–07:00) its
|
||
expanded resource envelope on each participating node.
|
||
|
||
## 5. Telemetry spine: beszel + power
|
||
|
||
- **Beszel has 100% VM+physical coverage** → it is the always-on signal for
|
||
the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
|
||
feed the placement scorecard; PSI textfile collector
|
||
(`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks.
|
||
- **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts
|
||
come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5
|
||
needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the
|
||
un-metered UPS, everything else on the metered one) is compatible: the
|
||
nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.
|
||
- **Load-side telemetry = per-circuit monitoring (#623, Emporia Vue 3
|
||
pending founder pick)**: per-circuit watts via the HA LOCAL (HACS)
|
||
integration → knelperf textfile exporter → the class-3 batch gate gets a
|
||
real deferrable-load menu (server-room subpanel vs HVAC vs site-2
|
||
equipment). Dedicated CTs on the server-room subpanel + the tsys6/7 UPS
|
||
legs keep fleet draw separable from house load — needed exactly on EEA
|
||
days, so local (non-cloud) telemetry matters.
|
||
|
||
## 6. Workload priority classes (founder-set order)
|
||
|
||
| class | tenants | profile | scheduling |
|
||
|---|---|---|---|
|
||
| 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7) |
|
||
| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
|
||
| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
|
||
|
||
## 7. Solar + Home Assistant integration
|
||
|
||
- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
|
||
Assistant exposes real-time production. The loop consumes: surplus watts =
|
||
green light to start class-3 batches (excess production is the cheapest
|
||
compute on earth); forecast + reservation calendar (KillBill) → promotion
|
||
timing advice ("run the Side Door promo the week of X: projected surplus +
|
||
idle capacity Y").
|
||
- Integration path: HA REST/websocket (creds pattern exists:
|
||
`~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) → a small
|
||
exporter writing `knelperf_solar_surplus_watts` into the textfile
|
||
collector's directory; the batch launcher gates on it.
|
||
- SITER-Solar math review (2026-09-06): found constant drift
|
||
($301.08/23952 vs plan's $264.47/22614), a stale Dockerfile entrypoint
|
||
(deprecated python script), header/ROI inconsistencies, and fixed-cost
|
||
scenario assumptions — fixes landing in that repo.
|
||
|
||
## 8. Two-site power economics (site 2: printer farm / laser / CNC)
|
||
|
||
The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
|
||
MUCH more expensive grid power than the main site. That asymmetry drives
|
||
scheduling policy as much as raw performance:
|
||
|
||
- **Marginal-price model**: every site has an effective $/kWh curve. Main
|
||
site = cheap grid + (soon) solar with export at $0.04 and self-use at
|
||
$0.085 (SITER-Solar contract). Site 2 = expensive grid. The scheduler's
|
||
objective becomes: run flexible load (compute AND manufacturing) where the
|
||
marginal $/kWh is lowest at that hour.
|
||
- **Solar surplus is main-site-first**: excess production self-consumed at
|
||
the main site displaces the CHEAPEST power — class-3 batch compute and any
|
||
deferrable main-site load should chase the surplus curve.
|
||
- **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and
|
||
often deferrable (prints especially — hours-long, unattended). Candidate
|
||
policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows
|
||
(if its tariff has TOU) or batch them for periods when main-site solar
|
||
surplus can't absorb more compute. Home Assistant + metered UPS legs give
|
||
the measured baseline; site 2 needs its own metering (smart plug / CT per
|
||
machine) before scheduling — queued as a data dependency.
|
||
- **Never shift OLTP**: class-1 paid workloads stay on their reserved
|
||
capacity regardless of power price — revenue and latency beat energy cost.
|
||
- KillBill reservation calendar × solar forecast × site-2 tariff = the
|
||
promotion/job-timing advisor output: "cheapest week to run X".
|
||
|
||
## 9. Hardware plan hooks
|
||
|
||
- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
|
||
are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
|
||
baseline each with fio, then one-batch-job-per-spindle).
|
||
- tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory
|
||
row pending (PFVSystemInventory.csv not found in repo — founder to drop in).
|
||
- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
|
||
with tsys3 for latency-sensitive class-1 storage.
|
||
|
||
## 10. Organization
|
||
|
||
Performance management org reports to VPTechOps, dotted line VPFacilities —
|
||
identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
|
||
groups). All work filed under #826; CRs for any prod change.
|
||
|
||
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
|