diff --git a/README.md b/README.md index 92d4f7a..59b3f4b 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,90 @@ -# perf +# ⚡ KNELPerf — dynamic performance optimization for the PFV fleet + +> **Baseline → Audit → Tweak → Re-baseline, on autopilot** — +> a bash-only performance loop for a nine-node Proxmox fleet, wired to +> solar surplus, ERCOT risk signals, and three classes of paid work. + +![license](https://img.shields.io/badge/license-AGPLv3-blue) +![bash](https://img.shields.io/badge/pure-bash-4EAA25?logo=gnubash&logoColor=white) +![loop](https://img.shields.io/badge/loop-baseline--%3Eaudit--%3Etweak--%3Ere--baseline-9c27b0) +![nodes](https://img.shields.io/badge/PVE%20nodes-7%20%2B%202%20coming-f97316) + +KNELPerf v2 (2026-09-06, two-week program +[#826](https://projects.knownelement.com/issues/826)) breaks the +organically-grown Ultix scheduling subsystem out into a fleet-grade module, +adds the missing CPU benchmark leg, and drives a continuous +baseline/audit/tweak/re-baseline loop across every host. > **Docs live on Discourse — this repo is the executable source of truth.** > **Topic:** https://community.turnsys.com/t/328 -> **Redmine:** https://projects.knownelement.com/issues/709 · **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769) +> **Redmine:** [#826](https://projects.knownelement.com/issues/826) (v2 program) · +> [#709](https://projects.knownelement.com/issues/709) (v1, closed) · +> **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769) -Performance baseline + tuning: fio/iperf3 harness v1.1 (PSI, load, device identity), tuned-profile deploys, audit scripts (probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io). Switch .cmds files now live in KNEL/netinfra. [\#709] [\#732] [\#737] +--- -## Layout -- `scripts/` — rule engine + hooks (see `bash scripts/check-rules.sh --fast`) -- (imported content at repo root, mirroring its PFVCluster path layout) +## 🗺️ The loop -## Provenance -Code imported from KNEL/PFVCluster (041d311); full git history retained in PFVCluster. Enforcement layer copied per ADOPTING.md. IaC consumers: KNEL/KNELIAC references this repo. +```mermaid +flowchart LR + B[📋 BASELINE
cpu-bench + fio/iperf3
+ PSI snapshot] --> A[🔍 AUDIT
tolerance diff vs bundle
beszel + PSI context] + A -->|drift found| T[🔧 TWEAK
idempotent tweaks/<id>.sh
GLPI CR-gated on prod] + T --> R[📈 RE-BASELINE
capture + compare
evidence to Redmine] + R --> A + A -->|clean| B + + subgraph SIGNALS[📡 External signals] + SOL[☀️ solar surplus
HA + SITER-Solar] + ERC[⚡ ERCOT risk
LZ_SOUTH RTM] + RES[📅 reservations
KillBill] + end + SIGNALS -.gates class-3 batch load.-> A +``` + +## 📦 Modules + +| Module | What it does | Entry point | +|---|---|---| +| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline ` | +| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` | +| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` | +| `scripts/` | tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit | `Makefile` verbs | +| `docs/` | architecture + decision reports | index below | + +## 🚀 Quick start + +```bash +# capture a baseline bundle for a host (read-only) +./loop/perf-loop.sh baseline pfv-tsys7 + +# audit vs latest baseline (tolerance diff, catches governor flips) +./loop/perf-loop.sh audit pfv-tsys7 + +# apply a CR-gated tweak, then re-baseline and compare +./loop/perf-loop.sh tweak pfv-tsys7 +./loop/perf-loop.sh rebaseline pfv-tsys7 + +# CPU leg alone (fast, zero-install) +KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh +``` + +## 📚 Docs index + +| Doc | Contents | +|---|---| +| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics | +| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths | +| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) | +| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward | + +Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar) +(solar economics + ONCOR arbitrage + ERCOT risk brief) · +[KNEL/KNELIAC](https://git.knownelement.com/KNEL/KNELIAC) (AWX deployment of these scripts) + +## 🧭 Provenance & enforcement + +Code imported from KNEL/PFVCluster (041d311; full git history retained +there); v2 scheduler module broken out of `~/projects/ultix` 2026-09-06. +CI: `.gitea/workflows/ci.yml` (shellcheck-zero + secret scan + +check-rules). Framework: `scripts/check-rules.sh --fast`; ticket gate per +AGENTS.md. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 106b4e2..f8dc193 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,17 +1,30 @@ -# KNELPerf v2 — architecture for dynamic, intelligent performance optimization +# 🏗️ KNELPerf v2 — architecture for dynamic, intelligent performance optimization -Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end -baseline → audit/benchmark → tweak → re-baseline loop, two-week program. +> Status: DRAFT v1 (2026-09-06, +> [#826](https://projects.knownelement.com/issues/826)). Founder directive: +> end-to-end baseline → audit/benchmark → tweak → re-baseline loop, +> two-week program. + +## 📑 Contents + +1. [The loop](#1-the-loop) · 2. [Scheduling architecture](#2-scheduling-architecture-ultix-breakout-redesign) · +3. [Proxmox resource groups & VM balancing](#3-proxmox-resource-groups--vm-balancing) · +4. [k8s scheduling & Slurm](#4-where-k8s-scheduling-fits-slurm) · +5. [Telemetry spine](#5-telemetry-spine-beszel--power) · +6. [Workload priority classes](#6-workload-priority-classes-founder-set-order) · +7. [Solar + Home Assistant](#7-solar--home-assistant-integration) · +8. [Two-site power economics](#8-two-site-power-economics-site-2-printer-farm--laser--cnc) · +9. [Hardware plan hooks](#9-hardware-plan-hooks) · 10. [Organization](#10-organization) ## 1. The loop -``` - ┌──────────────────────────────────────────────────┐ - │ │ - BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘ - (capture) (diff + beszel/ (CR-gated, (compare vs - PSI/rapl context) idempotent) baseline; keep - or roll back) +```mermaid +flowchart LR + B[📋 BASELINE
cpu-bench.tsv + bench-run.tsv
+ PSI snapshot per host] --> A[🔍 AUDIT
tolerance diff vs bundle
beszel + PSI context] + A -->|drift found| T[🔧 TWEAK
tweaks/<id>.sh — idempotent,
GLPI CR-gated on prod] + T --> R[📈 RE-BASELINE
capture + compare
evidence to Redmine] + R --> A + A -->|clean| B ``` - **baseline**: `loop/perf-loop.sh baseline ` stores a bundle @@ -31,7 +44,7 @@ Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring: the audit itself is low-cost); re-baseline only after a tweak or hardware change. First full-fleet baseline wave is this week's deliverable. -## 2. Scheduling architecture (Ultix breakout redesign) +## 2. 🗓️ Scheduling architecture (Ultix breakout redesign) See `scheduler/README.md` for the port map and the lessons-learned list. Core redesign decisions: @@ -45,7 +58,7 @@ redesign decisions: 4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the payload; job templates own the rollout waves. -## 3. Proxmox resource groups & VM balancing +## 3. 🖥️ Proxmox resource groups & VM balancing Current fleet (dmidecode-verified 2026-09-06): @@ -78,7 +91,7 @@ Directions: needs founder ruling — will be proposed on #826 with a capacity table once tsys8/2 land. -## 4. Where k8s scheduling fits; Slurm? +## 4. ☸️ Where k8s scheduling fits; Slurm? - **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for @@ -95,7 +108,7 @@ Directions: - The knelperf daynight engine gives the batch window (22:00–07:00) its expanded resource envelope on each participating node. -## 5. Telemetry spine: beszel + power +## 5. 📡 Telemetry spine: beszel + power - **Beszel has 100% VM+physical coverage** → it is the always-on signal for the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics @@ -114,7 +127,7 @@ Directions: legs keep fleet draw separable from house load — needed exactly on EEA days, so local (non-cloud) telemetry matters. -## 6. Workload priority classes (founder-set order) +## 6. 🏆 Workload priority classes (founder-set order) | class | tenants | profile | scheduling | |---|---|---|---| @@ -122,7 +135,7 @@ Directions: | 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 | | 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 | -## 7. Solar + Home Assistant integration +## 7. ☀️ Solar + Home Assistant integration - SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home Assistant exposes real-time production. The loop consumes: surplus watts = @@ -139,7 +152,7 @@ Directions: (deprecated python script), header/ROI inconsistencies, and fixed-cost scenario assumptions — fixes landing in that repo. -## 8. Two-site power economics (site 2: printer farm / laser / CNC) +## 8. 🏭 Two-site power economics (site 2: printer farm / laser / CNC) The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on MUCH more expensive grid power than the main site. That asymmetry drives @@ -165,7 +178,7 @@ scheduling policy as much as raw performance: - KillBill reservation calendar × solar forecast × site-2 tariff = the promotion/job-timing advisor output: "cheapest week to run X". -## 9. Hardware plan hooks +## 9. 🔩 Hardware plan hooks - Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins are placement + spindles (tsys4's 3 USB toasters join the spindle farm: @@ -175,7 +188,7 @@ scheduling policy as much as raw performance: - tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair with tsys3 for latency-sensitive class-1 storage. -## 10. Organization +## 10. 👥 Organization Performance management org reports to VPTechOps, dotted line VPFacilities — identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI diff --git a/docs/report-amt-power-telemetry.md b/docs/report-amt-power-telemetry.md index 6a6e6c3..3ef0cd0 100644 --- a/docs/report-amt-power-telemetry.md +++ b/docs/report-amt-power-telemetry.md @@ -1,4 +1,4 @@ -# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7) +# 🔌 Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7) Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included). diff --git a/docs/report-moonlight-desktop.md b/docs/report-moonlight-desktop.md index efbfd0b..0d471da 100644 --- a/docs/report-moonlight-desktop.md +++ b/docs/report-moonlight-desktop.md @@ -1,4 +1,4 @@ -# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s +# 🖼️ Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826. Source research: web-surveyed 2026-09-06 (sources linked at the end). diff --git a/scheduler/README.md b/scheduler/README.md index 90a8e78..241d6c1 100644 --- a/scheduler/README.md +++ b/scheduler/README.md @@ -1,4 +1,4 @@ -# KNELPerf scheduler — breakout from Ultix (#826) +# 🗓️ KNELPerf scheduler — breakout from Ultix (#826) The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`) was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains