First fleet baseline wave: 7/7 nodes + STATUS + registry digest pin (#826)

- data/baselines/: cpu-bench bundles for tsys1/3/4/5/6/7/9 (2026-09-06)
  headliners: tsys9 1.20M k/s AES-1t > tsys3 986k > tsys4 614k > tsys1 582k
  > tsys5 462k (governor=none, E5620) > tsys7 401k > tsys6 176k
- monitoring/README: toolbox image digest pinned
  (git.knownelement.com/knel/knelerf-toolbox:v1 @ sha256:3bb22bdb...)
- STATUS.md: pointer to #826 program state

Redmine: https://projects.knownelement.com/issues/826

README.md

@@ -1,14 +1,92 @@
-# perf
+# ⚡ KNELPerf — dynamic performance optimization for the PFV fleet
+
+> **Baseline → Audit → Tweak → Re-baseline, on autopilot** —
+> a bash-only performance loop for a nine-node Proxmox fleet, wired to
+> solar surplus, ERCOT risk signals, and three classes of paid work.
+
+![license](https://img.shields.io/badge/license-AGPLv3-blue)
+![bash](https://img.shields.io/badge/pure-bash-4EAA25?logo=gnubash&logoColor=white)
+![loop](https://img.shields.io/badge/loop-baseline--%3Eaudit--%3Etweak--%3Ere--baseline-9c27b0)
+![nodes](https://img.shields.io/badge/PVE%20nodes-7%20%2B%202%20coming-f97316)
+
+KNELPerf v2 (2026-09-06, two-week program
+[#826](https://projects.knownelement.com/issues/826)) breaks the
+organically-grown Ultix scheduling subsystem out into a fleet-grade module,
+adds the missing CPU benchmark leg, and drives a continuous
+baseline/audit/tweak/re-baseline loop across every host.
 
 > **Docs live on Discourse — this repo is the executable source of truth.**
 > **Topic:** https://community.turnsys.com/t/328
-> **Redmine:** https://projects.knownelement.com/issues/709 · **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)
+> **Redmine:** [#826](https://projects.knownelement.com/issues/826) (v2 program) ·
+> [#709](https://projects.knownelement.com/issues/709) (v1, closed) ·
+> **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)
 
-Performance baseline + tuning: fio/iperf3 harness v1.1 (PSI, load, device identity), tuned-profile deploys, audit scripts (probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io). Switch .cmds files now live in KNEL/netinfra. [\#709] [\#732] [\#737]
+---
 
-## Layout
+## 🗺️ The loop
-- `scripts/` — rule engine + hooks (see `bash scripts/check-rules.sh --fast`)
-- (imported content at repo root, mirroring its PFVCluster path layout)
 
-## Provenance
+```mermaid
-Code imported from KNEL/PFVCluster (041d311); full git history retained in PFVCluster. Enforcement layer copied per ADOPTING.md. IaC consumers: KNEL/KNELIAC references this repo.
+flowchart LR
+    B[📋 BASELINE<br/>cpu-bench + fio/iperf3<br/>+ PSI snapshot] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
+    A -->|drift found| T[🔧 TWEAK<br/>idempotent tweaks/&lt;id&gt;.sh<br/>GLPI CR-gated on prod]
+    T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
+    R --> A
+    A -->|clean| B
+
+    subgraph SIGNALS[📡 External signals]
+        SOL[☀️ solar surplus<br/>HA + SITER-Solar]
+        ERC[⚡ ERCOT risk<br/>LZ_SOUTH RTM]
+        RES[📅 reservations<br/>KillBill]
+    end
+    SIGNALS -.gates class-3 batch load.-> A
+```
+
+## 📦 Modules
+
+| Module | What it does | Entry point |
+|---|---|---|
+| `monitoring/` | fleet instrumentation: sysstat 28d tuning, perfsnap/PSI/iotop timers, on-demand perf toolbox container (LXC on PVE nodes, Docker image for the rest) | `monitoring/deploy-fleet.sh` |
+| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline <host>` |
+| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` |
+| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` |
+| `scripts/` | tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit | `Makefile` verbs |
+| `docs/` | architecture + decision reports | index below |
+
+## 🚀 Quick start
+
+```bash
+# capture a baseline bundle for a host (read-only)
+./loop/perf-loop.sh baseline pfv-tsys7
+
+# audit vs latest baseline (tolerance diff, catches governor flips)
+./loop/perf-loop.sh audit pfv-tsys7
+
+# apply a CR-gated tweak, then re-baseline and compare
+./loop/perf-loop.sh tweak pfv-tsys7 <tweak-id>
+./loop/perf-loop.sh rebaseline pfv-tsys7
+
+# CPU leg alone (fast, zero-install)
+KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh
+```
+
+## 📚 Docs index
+
+| Doc | Contents |
+|---|---|
+| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics |
+| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths |
+| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) |
+| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward |
+| [monitoring/README.md](monitoring/README.md) | what lands on each node, toolbox cookbook, analysis recipes, prod/CR gating |
+
+Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar)
+(solar economics + ONCOR arbitrage + ERCOT risk brief) ·
+[KNEL/KNELIAC](https://git.knownelement.com/KNEL/KNELIAC) (AWX deployment of these scripts)
+
+## 🧭 Provenance & enforcement
+
+Code imported from KNEL/PFVCluster (041d311; full git history retained
+there); v2 scheduler module broken out of `~/projects/ultix` 2026-09-06.
+CI: `.gitea/workflows/ci.yml` (shellcheck-zero + secret scan +
+check-rules). Framework: `scripts/check-rules.sh --fast`; ticket gate per
+AGENTS.md.

STATUS.md

@@ -1,4 +1,6 @@
 # STATUS.md
 
 > Operations status lives on Discourse: https://community.turnsys.com/t/297
-> This file is a pointer stub. Active ticket: [#769](https://projects.knownelement.com/issues/769) (repo split).
+> This file is a pointer stub.
+
+- [#826](https://projects.knownelement.com/issues/826) — KNELPerf v2 program · monitoring fleet-wide 09-06 (CR 25), toolbox on 7/7 nodes, first baselines in `data/baselines/` · next: placement scorecard + day/night UAT

data/baselines/pfv-tsys1/20260906-192237/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys1 20260906-192237

data/baselines/pfv-tsys1/20260906-192237/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys1|meta|stamp
+BENCH|pfv-tsys1|meta|path
+BENCH|pfv-tsys1|meta|runtime
+BENCH|pfv-tsys1|meta|size
+BENCH|pfv-tsys1|meta|load
+BENCH|pfv-tsys1|psi_io_some_avg10|15.30
+BENCH|pfv-tsys1|psi_cpu_some_avg10|30.66
+BENCH|pfv-tsys1|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys1|meta|bench_device
+BENCH|pfv-tsys1|randread4k|iops=48.480608|bw_kbps=193|p99_ms=196.083712
+BENCH|pfv-tsys1|randwrite4k|iops=124.287571|bw_kbps=497|p99_ms=143.654912
+BENCH|pfv-tsys1|seqread1m|iops=53.994601|bw_kbps=55290|p99_ms=187.695104
+BENCH|pfv-tsys1|seqwrite1m|iops=59.444056|bw_kbps=60870|p99_ms=183.5008
+BENCH|pfv-tsys1|meta|load_end
+BENCH|pfv-tsys1|psi_io_some_avg10_end|35.56
+BENCH|pfv-tsys1|meta|end

data/baselines/pfv-tsys1/20260906-192237/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	8	-
+openssl_aes-256-cbc_1t	581661.52k	k/s
+openssl_aes-256-cbc_8t	3891114.80k	k/s
+openssl_sha256_1t	427686.49k	k/s
+openssl_sha256_8t	1594156.65k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	22.55	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	16.97	%

data/baselines/pfv-tsys3/20260906-192801/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys3 20260906-192801

data/baselines/pfv-tsys3/20260906-192801/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys3|meta|stamp
+BENCH|pfv-tsys3|meta|path
+BENCH|pfv-tsys3|meta|runtime
+BENCH|pfv-tsys3|meta|size
+BENCH|pfv-tsys3|meta|load
+BENCH|pfv-tsys3|psi_io_some_avg10|0.00
+BENCH|pfv-tsys3|psi_cpu_some_avg10|4.45
+BENCH|pfv-tsys3|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys3|meta|bench_device
+BENCH|pfv-tsys3|randread4k|iops=12026.998650|bw_kbps=48107|p99_ms=0.1152
+BENCH|pfv-tsys3|randwrite4k|iops=56757.450000|bw_kbps=227029|p99_ms=0.035072
+BENCH|pfv-tsys3|seqread1m|iops=1220.738963|bw_kbps=1250036|p99_ms=1.028096
+BENCH|pfv-tsys3|seqwrite1m|iops=1489.325534|bw_kbps=1525069|p99_ms=0.774144
+BENCH|pfv-tsys3|meta|load_end
+BENCH|pfv-tsys3|psi_io_some_avg10_end|18.52
+BENCH|pfv-tsys3|meta|end

data/baselines/pfv-tsys3/20260906-192801/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	8	-
+openssl_aes-256-cbc_1t	986305.32k	k/s
+openssl_aes-256-cbc_8t	7198685.59k	k/s
+openssl_sha256_1t	452144.28k	k/s
+openssl_sha256_8t	1821861.48k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	3.24	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	0.00	%

data/baselines/pfv-tsys4/20260906-193846/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys4 20260906-193846

data/baselines/pfv-tsys4/20260906-193846/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys4|meta|stamp
+BENCH|pfv-tsys4|meta|path
+BENCH|pfv-tsys4|meta|runtime
+BENCH|pfv-tsys4|meta|size
+BENCH|pfv-tsys4|meta|load
+BENCH|pfv-tsys4|psi_io_some_avg10|59.62
+BENCH|pfv-tsys4|psi_cpu_some_avg10|4.54
+BENCH|pfv-tsys4|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys4|meta|bench_device
+BENCH|pfv-tsys4|randread4k|iops=52.536866|bw_kbps=210|p99_ms=112.72192
+BENCH|pfv-tsys4|randwrite4k|iops=151.366748|bw_kbps=605|p99_ms=62.652416
+BENCH|pfv-tsys4|seqread1m|iops=41.277297|bw_kbps=42267|p99_ms=112.72192
+BENCH|pfv-tsys4|seqwrite1m|iops=50.805205|bw_kbps=52024|p99_ms=76.02176
+BENCH|pfv-tsys4|meta|load_end
+BENCH|pfv-tsys4|psi_io_some_avg10_end|88.30
+BENCH|pfv-tsys4|meta|end

data/baselines/pfv-tsys4/20260906-193846/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	8	-
+openssl_aes-256-cbc_1t	614414.75k	k/s
+openssl_aes-256-cbc_8t	4687442.74k	k/s
+openssl_sha256_1t	482344.63k	k/s
+openssl_sha256_8t	1882695.27k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	3.15	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	62.40	%

data/baselines/pfv-tsys5/20260906-194410/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys5 20260906-194410

data/baselines/pfv-tsys5/20260906-194410/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys5|meta|stamp
+BENCH|pfv-tsys5|meta|path
+BENCH|pfv-tsys5|meta|runtime
+BENCH|pfv-tsys5|meta|size
+BENCH|pfv-tsys5|meta|load
+BENCH|pfv-tsys5|psi_io_some_avg10|15.30
+BENCH|pfv-tsys5|psi_cpu_some_avg10|64.07
+BENCH|pfv-tsys5|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys5|meta|bench_device
+BENCH|pfv-tsys5|randread4k|iops=126.893655|bw_kbps=507|p99_ms=27.131904
+BENCH|pfv-tsys5|randwrite4k|iops=352.932353|bw_kbps=1411|p99_ms=10.15808
+BENCH|pfv-tsys5|seqread1m|iops=99.965012|bw_kbps=102364|p99_ms=49.545216
+BENCH|pfv-tsys5|seqwrite1m|iops=114.277145|bw_kbps=117019|p99_ms=28.442624
+BENCH|pfv-tsys5|meta|load_end
+BENCH|pfv-tsys5|psi_io_some_avg10_end|30.96
+BENCH|pfv-tsys5|meta|end

data/baselines/pfv-tsys5/20260906-194410/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	none	-
+nproc	8	-
+openssl_aes-256-cbc_1t	461596.26k	k/s
+openssl_aes-256-cbc_8t	1970275.94k	k/s
+openssl_sha256_1t	149760.92k	k/s
+openssl_sha256_8t	523591.15k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	56.95	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	10.86	%

data/baselines/pfv-tsys6/20260906-194935/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys6 20260906-194935

data/baselines/pfv-tsys6/20260906-194935/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys6|meta|stamp
+BENCH|pfv-tsys6|meta|path
+BENCH|pfv-tsys6|meta|runtime
+BENCH|pfv-tsys6|meta|size
+BENCH|pfv-tsys6|meta|load
+BENCH|pfv-tsys6|psi_io_some_avg10|10.45
+BENCH|pfv-tsys6|psi_cpu_some_avg10|13.39
+BENCH|pfv-tsys6|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys6|meta|bench_device
+BENCH|pfv-tsys6|randread4k|iops=90.654673|bw_kbps=362|p99_ms=18.743296
+BENCH|pfv-tsys6|randwrite4k|iops=1295.538575|bw_kbps=5182|p99_ms=1.368064
+BENCH|pfv-tsys6|seqread1m|iops=2.286282|bw_kbps=2341|p99_ms=767.557632
+BENCH|pfv-tsys6|seqwrite1m|iops=29.959555|bw_kbps=30678|p99_ms=45.8752
+BENCH|pfv-tsys6|meta|load_end
+BENCH|pfv-tsys6|psi_io_some_avg10_end|43.01
+BENCH|pfv-tsys6|meta|end

data/baselines/pfv-tsys6/20260906-194935/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	16	-
+openssl_aes-256-cbc_1t	175503.96k	k/s
+openssl_aes-256-cbc_16t	1498416.74k	k/s
+openssl_sha256_1t	196991.77k	k/s
+openssl_sha256_16t	1409610.55k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	9.59	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	16.25	%

data/baselines/pfv-tsys7/20260906-195458/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys7 20260906-195458

data/baselines/pfv-tsys7/20260906-195458/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys7|meta|stamp
+BENCH|pfv-tsys7|meta|path
+BENCH|pfv-tsys7|meta|runtime
+BENCH|pfv-tsys7|meta|size
+BENCH|pfv-tsys7|meta|load
+BENCH|pfv-tsys7|psi_io_some_avg10|8.77
+BENCH|pfv-tsys7|psi_cpu_some_avg10|12.66
+BENCH|pfv-tsys7|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys7|meta|bench_device
+BENCH|pfv-tsys7|randread4k|iops=83.729068|bw_kbps=334|p99_ms=20.578304
+BENCH|pfv-tsys7|randwrite4k|iops=1384.680766|bw_kbps=5538|p99_ms=1.417216
+BENCH|pfv-tsys7|seqread1m|iops=1.846632|bw_kbps=1890|p99_ms=943.7184
+BENCH|pfv-tsys7|seqwrite1m|iops=36.526258|bw_kbps=37402|p99_ms=30.277632
+BENCH|pfv-tsys7|meta|load_end
+BENCH|pfv-tsys7|psi_io_some_avg10_end|30.73
+BENCH|pfv-tsys7|meta|end

data/baselines/pfv-tsys7/20260906-195458/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	24	-
+openssl_aes-256-cbc_1t	400782.52k	k/s
+openssl_aes-256-cbc_24t	9105252.35k	k/s
+openssl_sha256_1t	273898.45k	k/s
+openssl_sha256_24t	3317389.72k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	8.84	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	12.48	%

data/baselines/pfv-tsys9/20260906-193323/MANIFEST

@@ -0,0 +1 @@
+baseline pfv-tsys9 20260906-193323

data/baselines/pfv-tsys9/20260906-193323/bench-run.tsv

@@ -0,0 +1,16 @@
+BENCH|pfv-tsys9|meta|stamp
+BENCH|pfv-tsys9|meta|path
+BENCH|pfv-tsys9|meta|runtime
+BENCH|pfv-tsys9|meta|size
+BENCH|pfv-tsys9|meta|load
+BENCH|pfv-tsys9|psi_io_some_avg10|5.14
+BENCH|pfv-tsys9|psi_cpu_some_avg10|2.66
+BENCH|pfv-tsys9|psi_memory_some_avg10|0.00
+BENCH|pfv-tsys9|meta|bench_device
+BENCH|pfv-tsys9|randread4k|iops=7766.011699|bw_kbps=31064|p99_ms=0.164864
+BENCH|pfv-tsys9|randwrite4k|iops=17438.828059|bw_kbps=69755|p99_ms=0.091648
+BENCH|pfv-tsys9|seqread1m|iops=267.346531|bw_kbps=273762|p99_ms=4.292608
+BENCH|pfv-tsys9|seqwrite1m|iops=388.511149|bw_kbps=397835|p99_ms=2.965504
+BENCH|pfv-tsys9|meta|load_end
+BENCH|pfv-tsys9|psi_io_some_avg10_end|61.96
+BENCH|pfv-tsys9|meta|end

data/baselines/pfv-tsys9/20260906-193323/cpu-bench.tsv

@@ -0,0 +1,11 @@
+metric	value	unit
+governor	performance	-
+nproc	12	-
+openssl_aes-256-cbc_1t	1199179.37k	k/s
+openssl_aes-256-cbc_12t	13546392.78k	k/s
+openssl_sha256_1t	558840.22k	k/s
+openssl_sha256_12t	3421896.70k	k/s
+sysbench_cpu	absent	-
+psi_cpu_avg60	1.93	%
+psi_memory_avg60	0.00	%
+psi_io_avg60	9.72	%

docs/ARCHITECTURE.md

@@ -1,17 +1,30 @@
-# KNELPerf v2 — architecture for dynamic, intelligent performance optimization
+# 🏗️ KNELPerf v2 — architecture for dynamic, intelligent performance optimization
 
-Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end
+> Status: DRAFT v1 (2026-09-06,
-baseline → audit/benchmark → tweak → re-baseline loop, two-week program.
+> [#826](https://projects.knownelement.com/issues/826)). Founder directive:
+> end-to-end baseline → audit/benchmark → tweak → re-baseline loop,
+> two-week program.
+
+## 📑 Contents
+
+1. [The loop](#1-the-loop) · 2. [Scheduling architecture](#2-scheduling-architecture-ultix-breakout-redesign) ·
+3. [Proxmox resource groups & VM balancing](#3-proxmox-resource-groups--vm-balancing) ·
+4. [k8s scheduling & Slurm](#4-where-k8s-scheduling-fits-slurm) ·
+5. [Telemetry spine](#5-telemetry-spine-beszel--power) ·
+6. [Workload priority classes](#6-workload-priority-classes-founder-set-order) ·
+7. [Solar + Home Assistant](#7-solar--home-assistant-integration) ·
+8. [Two-site power economics](#8-two-site-power-economics-site-2-printer-farm--laser--cnc) ·
+9. [Hardware plan hooks](#9-hardware-plan-hooks) · 10. [Organization](#10-organization)
 
 ## 1. The loop
 
-```
+```mermaid
-        ┌──────────────────────────────────────────────────┐
+flowchart LR
-        │                                                  │
+    B[📋 BASELINE<br/>cpu-bench.tsv + bench-run.tsv<br/>+ PSI snapshot per host] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
-   BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
+    A -->|drift found| T[🔧 TWEAK<br/>tweaks/&lt;id&gt;.sh — idempotent,<br/>GLPI CR-gated on prod]
-   (capture)     (diff + beszel/       (CR-gated,   (compare vs
+    T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
-                 PSI/rapl context)     idempotent)  baseline; keep
+    R --> A
-                                                     or roll back)
+    A -->|clean| B
 ```
 
 - **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
@@ -31,7 +44,7 @@ Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
 the audit itself is low-cost); re-baseline only after a tweak or hardware
 change. First full-fleet baseline wave is this week's deliverable.
 
-## 2. Scheduling architecture (Ultix breakout redesign)
+## 2. 🗓️ Scheduling architecture (Ultix breakout redesign)
 
 See `scheduler/README.md` for the port map and the lessons-learned list. Core
 redesign decisions:
@@ -45,7 +58,7 @@ redesign decisions:
 4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
    payload; job templates own the rollout waves.
 
-## 3. Proxmox resource groups & VM balancing
+## 3. 🖥️ Proxmox resource groups & VM balancing
 
 Current fleet (dmidecode-verified 2026-09-06):
 
@@ -78,7 +91,7 @@ Directions:
   needs founder ruling — will be proposed on #826 with a capacity table once
   tsys8/2 land.
 
-## 4. Where k8s scheduling fits; Slurm?
+## 4. ☸️ Where k8s scheduling fits; Slurm?
 
 - **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
   (ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
@@ -95,7 +108,7 @@ Directions:
 - The knelperf daynight engine gives the batch window (22:00–07:00) its
   expanded resource envelope on each participating node.
 
-## 5. Telemetry spine: beszel + power
+## 5. 📡 Telemetry spine: beszel + power
 
 - **Beszel has 100% VM+physical coverage** → it is the always-on signal for
   the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
@@ -106,8 +119,15 @@ Directions:
   needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the
   un-metered UPS, everything else on the metered one) is compatible: the
   nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.
+- **Load-side telemetry = per-circuit monitoring (#623, Emporia Vue 3
+  pending founder pick)**: per-circuit watts via the HA LOCAL (HACS)
+  integration → knelperf textfile exporter → the class-3 batch gate gets a
+  real deferrable-load menu (server-room subpanel vs HVAC vs site-2
+  equipment). Dedicated CTs on the server-room subpanel + the tsys6/7 UPS
+  legs keep fleet draw separable from house load — needed exactly on EEA
+  days, so local (non-cloud) telemetry matters.
 
-## 6. Workload priority classes (founder-set order)
+## 6. 🏆 Workload priority classes (founder-set order)
 
 | class | tenants | profile | scheduling |
 |---|---|---|---|
@@ -115,7 +135,7 @@ Directions:
 | 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
 | 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
 
-## 7. Solar + Home Assistant integration
+## 7. ☀️ Solar + Home Assistant integration
 
 - SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
   Assistant exposes real-time production. The loop consumes: surplus watts =
@@ -132,7 +152,7 @@ Directions:
   (deprecated python script), header/ROI inconsistencies, and fixed-cost
   scenario assumptions — fixes landing in that repo.
 
-## 8. Two-site power economics (site 2: printer farm / laser / CNC)
+## 8. 🏭 Two-site power economics (site 2: printer farm / laser / CNC)
 
 The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
 MUCH more expensive grid power than the main site. That asymmetry drives
@@ -158,7 +178,7 @@ scheduling policy as much as raw performance:
 - KillBill reservation calendar × solar forecast × site-2 tariff = the
   promotion/job-timing advisor output: "cheapest week to run X".
 
-## 9. Hardware plan hooks
+## 9. 🔩 Hardware plan hooks
 
 - Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
   are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
@@ -168,10 +188,61 @@ scheduling policy as much as raw performance:
 - tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
   with tsys3 for latency-sensitive class-1 storage.
 
-## 10. Organization
+## 10. 👥 Organization
 
 Performance management org reports to VPTechOps, dotted line VPFacilities —
 identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
 groups). All work filed under #826; CRs for any prod change.
 
 > **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
+
+## 11. 📊 Capacity planning & margin tracking in Dolibarr
+
+ERP landscape: 19 Dolibarr instances exist (one per brand);
+**erp.knownelement.com** is the mothership (v24.0.0, REST API live, perf-org
+API identity `vpperf` — creds `~/.creds/dolibarr-perf.env`). KillBill stays
+the subscription billing engine; Dolibarr is the financial brain where
+capacity, cost, and margin meet.
+
+### Capacity ledger (supply)
+
+- Fleet capacity is derived from ground truth we already collect: GLPI CMDB
+  + dmidecode survey + baseline bundles → per-node vCPU/RAM/NVMe/spindle
+  inventory (ARCHITECTURE §3 table is the seed).
+- Represented in Dolibarr as **virtual stock**: one product per capacity SKU
+  per node class (e.g. `cap-oltp-4c8g-nvme`, `cap-batch-spindle`,
+  `cap-k8s-worker-core`). Stock quantity = bookable units; GLPI drift report
+  reconciles it (the nightly reconcile already exists for CIs — extend to
+  capacity products).
+
+### Reservations (demand)
+
+- Paid reservations (class 1) = Dolibarr customer contracts/order lines that
+  consume capacity stock; KillBill invoices the money, Dolibarr holds the
+  commitment. Utilization % = booked/available per SKU per week.
+- The promotion advisor consumes exactly this: KillBill reservation calendar
+  x projected solar surplus x utilization headroom (§7).
+
+### Margin tracking (profitability)
+
+- **Revenue** per tenant: KillBill (exports/journal) → Dolibarr customer
+  invoices/analytic accounts.
+- **Direct costs** per tenant class: power (both sites — utility bills #827;
+  solar self-consumption at contract rate), hardware amortization (post
+  2026-10-11 purchases), WAN/tailnet overheads.
+- Dolibarr **analytic accounting** (accounts + analytic distribution tables
+  are already present) allocates costs to tenant classes → margin per class
+  per month. PRIVATE dimension: margin numbers are revenue/expense detail —
+  LLC-members-only, never published to the public hub (transparency policy).
+
+### Integration plumbing
+
+- `loop/` baselines + beszel utilization → utilization updater script →
+  Dolibarr API (product stock / extrafields) — read/write via vpperf key.
+- Hub publishes ONLY utilization % and capacity counts (public-safe);
+  margins stay inside the ERP.
+
+API gotchas learned (2026-09-06): table prefix is `dolibarr_` (not `llx_`);
+API auth needs `DOLAPIKEY` header; a cloned user needs explicit
+`dolibarr_user_rights` rows copied (admin flag alone does not grant API
+module rights).

docs/report-amt-power-telemetry.md

@@ -1,4 +1,4 @@
-# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
+# 🔌 Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
 
 Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via
 dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included).
@@ -60,8 +60,9 @@ for watts.
    update iDRAC6 firmware on tsys6 if DCMI commands fail.
 4. tsys5 (T7500): budget a metered PDU or smart plug — no software path
    exists (fits the planned UPS re-cabling: put tsys5 on the metered leg).
-5. tsys9: run `dmidecode -t system` + `ls /sys/class/powercap` — that pair
+5. tsys9 RESOLVED (2026-09-06): OptiPlex 7080, i5-10500, live PVE 9.2.5 node
-   answers both its AMT and RAPL questions; report back.
+   (100.101.158.76 — runs pfv-netinfra-02 + k8s wnode-tsys9); RAPL present —
+   include it in the RAPL exporter wave.
 6. UPS plan alignment: since only tsys6/7 (BMC) + RAPL hosts give per-node
    watts, keep the metered UPS leg feeding the gear you most need wall-watts
    for; per-node software telemetry covers the rest.

docs/report-moonlight-desktop.md

@@ -1,4 +1,4 @@
-# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
+# 🖼️ Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
 
 Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826.
 Source research: web-surveyed 2026-09-06 (sources linked at the end).

loop/baseline-diff.sh

@@ -17,7 +17,7 @@ join_tsv() { awk -F'\t' 'NR==FNR{a[$1]=$2;next} {print $1"\t"a[$1]"\t"$2"\t"$3}'
 # shellcheck disable=SC2034  # exit-status reserved for future strictness
 status=0
 join_tsv "$A/cpu-bench.tsv" "$B/cpu-bench.tsv" | while IFS=$'\t' read -r metric va vb unit; do
-  [ -n "$va" ] && [ -n "$vb" ] || continue
+  if [ -z "$va" ] || [ -z "$vb" ]; then continue; fi
   if [ "$va" = "$vb" ]; then
     printf 'OK    %-28s %s = %s %s\n' "$metric" "$va" "$vb" "$unit"
   elif [ "$va" = "${va#[-+0-9.]}" ] || [ "$vb" = "${vb#[-+0-9.]}" ]; then

monitoring/README.md

@@ -0,0 +1,78 @@
+# 📡 Fleet monitoring & the on-demand toolbox
+
+> **Redmine:** [#826](https://projects.knownelement.com/issues/826) ·
+> **Docs:** [Discourse](https://community.turnsys.com/t/328) ·
+> **Program:** [`docs/ARCHITECTURE.md`](../docs/ARCHITECTURE.md) (§5 telemetry spine)
+
+Continuous lightweight instrumentation for the seven PFV PVE nodes
+(tsys1/3/4/5/6/7/9) plus a per-node **perf toolbox container** that is
+stopped at rest and runs on demand. Ultix's instrumentation pattern
+(60s perfsnap + PSI collectors, containers for deep dives) generalized
+fleet-wide. AGPLv3, like the whole repo.
+
+## What lands on a node
+
+| Piece | Cadence | Data | Where |
+|---|---|---|---|
+| `sysstat` sadc (ships with PVE, tuned here) | 10 min + daily rollup | sar: CPU, mem, disks, net, load | `/var/log/sysstat/saDD` (HISTORY=28) |
+| `knelperf-perfsnap.sh` | 60 s | load, run/block, memavail, PSI avg60, qemu+lxc cgroup CPU, bridge counters | `/var/log/knelperf-perf/{snap,procs}.log` |
+| `knelperf-psi-textfile.sh` | 15 s | PSI some60 per guest slice + host (Prometheus textfile format) | `/var/lib/knelperf/textfile/knelperf_pressure.prom` |
+| `knelperf-iotop-snap.sh` | 15 min | WHO does the I/O sar only aggregates (3×2 s active-only samples) | `/var/log/knelperf-iotop/iotop-YYYYMMDD.log` |
+
+No reboots, no package installs (sysstat + iotop ship with PVE 9),
+no daemons — three oneshot systemd timers. Rollback = disable timers +
+restore `HISTORY=7`. bpftrace/BCC live **inside the toolbox**, never on
+the hypervisor root.
+
+## Deploy
+
+From a tooling host with root SSH to the nodes:
+
+```bash
+# non-prod first, prod only after the GLPI CR exists
+bash monitoring/deploy-fleet.sh --toolbox pfv-tsys1 pfv-tsys3 pfv-tsys9
+```
+
+`--toolbox` additionally creates the per-node LXC (see below). The
+installer (`monitoring/node-install.sh`) is idempotent — safe to re-run.
+
+## The toolbox container (vmid 950, `knelperf-toolbox`)
+
+PVE-native LXC on each node — Debian 12, privileged, apparmor
+unconfined, bind mounts for host cgroupfs and `/var/log` (ro), up to 8
+block devices, and runtime-mounted tracefs/debugfs (pct skips
+virtual-fs bind sources — the wrapper mounts them on start).
+**Stopped at rest; `onboot=0`.** Tools: bpftrace, BCC, sysstat, iotop,
+sysbench, fio, iperf3, stress-ng, perf, smartmontools, nvme-cli, hwloc…
+
+```bash
+knelperf-toolbox run 'iostat -x 1 5'          # starts LXC if stopped
+knelperf-toolbox run 'bpftrace -e "tracepoint:block:block_rq_issue { @[comm] = count(); }" -c "sleep 5"'
+knelperf-toolbox run 'sar -A -f /host/var/log/sysstat/sa06'   # host sar, offline
+knelperf-toolbox run 'smartctl -d sat -H /dev/sda'            # SAT devices need -d sat; NVMe doesn't
+knelperf-toolbox shell                          # interactive
+knelperf-toolbox stop                           # back to zero footprint
+```
+
+Docker-capable hosts use the same manifest as an image:
+`git.knownelement.com/knel/knelperf-toolbox:v1` (digest `sha256:3bb22bdb4a69707aadd0043d5151f863634a3ce5229f63d12773d5e3f90e001c`, from `monitoring/toolbox/Dockerfile`).
+
+## Analysis recipes
+
+```bash
+sar -q -f /var/log/sysstat/sa06                          # load vs runq
+sar -d -p -f /var/log/sysstat/sa06                       # per-device IO
+awk -F'|' 'NR>1{print $1, $8, $9, $10}' /var/log/knelperf-perf/snap.log   # PSI trend
+grep -A3 "$(date +%F)" /var/log/knelperf-iotop/iotop-$(date +%Y%m%d).log  # io offenders
+```
+
+The `loop/perf-loop.sh` driver consumes these on its `audit` pass;
+`baseline-diff.sh` compares before/after tweaks. Per-guest PSI
+(`qemu.slice/*.scope`, `lxc/*`) feeds the placement scorecard
+(ARCHITECTURE §3).
+
+## Prod gating
+
+Prod nodes (tsys4/5/6/7) deploy only after the GLPI CR for this rollout
+exists and references this directory. Verification evidence goes back on
+the CR; closing it is human-only.

monitoring/collectors/knelperf-iotop-snap.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# KNELPerf iotop snapshot — periodic batch samples of active per-process I/O.
+# Complements sar (system-wide device stats) with the WHO behind the I/O.
+# Env (or /etc/knelperf/perfsnap.conf):
+#   KNELPERF_IOTOP_LOGDIR   default /var/log/knelperf-iotop
+# Output: daily file iotop-YYYYMMDD.log; per tick one header line then
+#   3 batch iterations x 2s of processes with outstanding I/O (-botqq -k).
+# Docs: https://community.turnsys.com/t/328
+set -u
+CONF=/etc/knelperf/perfsnap.conf
+[ -r "$CONF" ] && . "$CONF"
+LOGDIR=${KNELPERF_IOTOP_LOGDIR:-/var/log/knelperf-iotop}
+mkdir -p "$LOGDIR"
+F="$LOGDIR/iotop-$(date +%Y%m%d).log"
+{
+  echo "### $(date -Iseconds)"
+  iotop -botqq -k -n 3 -d 2 2>/dev/null
+} >> "$F"
+# retention: 14 days; rotate the live file at 25MB
+find "$LOGDIR" -name 'iotop-*.log*' -mtime +14 -delete 2>/dev/null
+[ "$(wc -c <"$F" 2>/dev/null || echo 0)" -gt 26214400 ] && mv "$F" "$F.1"
+exit 0

monitoring/deploy-fleet.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Deploy KNELPerf node monitoring (+ toolbox LXC) to PVE fleet nodes (#826).
+# Runs from the workstation / a tooling host — never on a node itself.
+#   deploy-fleet.sh [--toolbox] host [host...]
+# Prod nodes (tsys4/5/6/7) deploy ONLY after their GLPI CR exists.
+# Docs: https://community.turnsys.com/t/328
+set -euo pipefail
+here=$(cd "$(dirname "$0")/.." && pwd)
+WITH_TOOLBOX=0
+if [ "${1:-}" = --toolbox ]; then WITH_TOOLBOX=1; shift; fi
+[ $# -ge 1 ] || { echo "usage: $0 [--toolbox] host..." >&2; exit 1; }
+
+for host in "$@"; do
+  echo "=== $host ==="
+  timeout 30 ssh -o ConnectTimeout=8 "root@$host" 'mkdir -p /tmp/knelperf-install'
+  tar -C "$here" -cf - scheduler/collectors monitoring \
+    | timeout 120 ssh "root@$host" 'tar -C /tmp/knelperf-install -xf -'
+  timeout 120 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/node-install.sh'
+  if [ "$WITH_TOOLBOX" = 1 ]; then
+    timeout 600 ssh "root@$host" 'bash /tmp/knelperf-install/monitoring/toolbox/lxc-toolbox.sh --create'
+  fi
+  timeout 30 ssh "root@$host" 'rm -rf /tmp/knelperf-install'
+done
+echo "fleet deploy done."

monitoring/node-install.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# KNELPerf node monitoring installer — sysstat tuning + collectors + timers.
+# Idempotent; run as ROOT on a PVE node from a repo checkout:
+#   bash monitoring/node-install.sh
+# Surface (all details in monitoring/README.md):
+#   - /etc/sysstat/sysstat        HISTORY 7 -> 28, COMPRESSAFTER 10 -> 31
+#   - /usr/local/sbin/knelperf-{perfsnap,psi-textfile,iotop-snap}.sh
+#   - /etc/systemd/system/knelperf-*.{service,timer}  (enabled timers)
+#   - /etc/knelperf/perfsnap.conf  per-node ifaces/cgroup globs
+#   - logs under /var/log/knelperf-{perf,iotop}, textfiles /var/lib/knelperf/textfile
+# No reboots, no package installs (PVE ships sysstat + iotop already).
+# Docs: https://community.turnsys.com/t/328
+set -euo pipefail
+[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
+here=$(cd "$(dirname "$0")/.." && pwd)
+say() { echo "[knelperf-install] $*"; }
+
+# 1) sysstat: keep 28 days of sar history (two-week program + re-baseline window)
+if [ -f /etc/sysstat/sysstat ]; then
+  sed -i -E 's/^HISTORY=.*/HISTORY=28/; s/^COMPRESSAFTER=.*/COMPRESSAFTER=31/' /etc/sysstat/sysstat
+  sed -i -E 's/^ENABLED=.*/ENABLED="true"/' /etc/default/sysstat
+  systemctl restart sysstat 2>/dev/null || true
+  /usr/libexec/sysstat/debian-sa1 1 1 2>/dev/null || true
+  today=/var/log/sysstat/sa$(date +%d)
+  if [ -f "$today" ]; then say "sar ok: $today present"; else say "sar WARN: $today missing (check /etc/cron.d/sysstat)"; fi
+else
+  say "WARN: no /etc/sysstat/sysstat on this host — sar tuning skipped"
+fi
+
+# 2) collectors + units
+install -m 0755 "$here/scheduler/collectors/knelperf-perfsnap.sh"     /usr/local/sbin/knelperf-perfsnap.sh
+install -m 0755 "$here/scheduler/collectors/knelperf-psi-textfile.sh" /usr/local/sbin/knelperf-psi-textfile.sh
+install -m 0755 "$here/monitoring/collectors/knelperf-iotop-snap.sh"  /usr/local/sbin/knelperf-iotop-snap.sh
+for u in perfsnap psi-textfile iotop-snap; do
+  install -m 0644 "$here/monitoring/systemd/knelperf-$u.service" /etc/systemd/system/knelperf-$u.service
+  install -m 0644 "$here/monitoring/systemd/knelperf-$u.timer"   /etc/systemd/system/knelperf-$u.timer
+done
+
+# 3) per-node conf: watch PVE guest cgroups + real bridges/tailscale
+ifaces=""
+for i in vmbr0 datanet tailscale0; do
+  ip link show "$i" >/dev/null 2>&1 && ifaces="$ifaces $i"
+done
+mkdir -p /etc/knelperf
+cat > /etc/knelperf/perfsnap.conf <<EOF
+# /etc/knelperf/perfsnap.conf — generated by KNELPerf node-install.sh $(date -I)
+KNELPERF_SNAP_CGROUPS="qemu:/sys/fs/cgroup/qemu.slice lxc:/sys/fs/cgroup/lxc"
+KNELPERF_SNAP_IFACES="${ifaces# }"
+KNELPERF_CG_GLOB="/sys/fs/cgroup/qemu.slice/*.scope /sys/fs/cgroup/lxc/*"
+KNELPERF_TEXTFILE_DIR=/var/lib/knelperf/textfile
+KNELPERF_PERF_LOGDIR=/var/log/knelperf-perf
+KNELPERF_IOTOP_LOGDIR=/var/log/knelperf-iotop
+EOF
+say "conf written (ifaces: ${ifaces# })"
+
+# 4) timers on + first tick of each collector
+systemctl daemon-reload
+for u in knelperf-perfsnap knelperf-psi-textfile knelperf-iotop-snap; do
+  systemctl enable --now "$u.timer" >/dev/null 2>&1
+done
+/usr/local/sbin/knelperf-perfsnap.sh
+/usr/local/sbin/knelperf-psi-textfile.sh
+/usr/local/sbin/knelperf-iotop-snap.sh
+say "timers enabled; first ticks logged"
+
+# 5) evidence
+say "--- timers ---"
+systemctl list-timers 'knelperf-*' --no-pager | sed -n '1,5p'
+say "--- snap.log tail ---"
+tail -1 /var/log/knelperf-perf/snap.log
+say "--- textfile ---"
+head -3 /var/lib/knelperf/textfile/knelperf_pressure.prom 2>/dev/null || true
+say "done."

monitoring/systemd/knelperf-iotop-snap.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=KNELPerf iotop snapshot (3x2s active-I/O samples)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/knelperf-iotop-snap.sh
+IOSchedulingClass=idle

monitoring/systemd/knelperf-iotop-snap.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run knelperf-iotop-snap every 15 minutes
+
+[Timer]
+OnCalendar=*:0/15
+Persistent=true
+RandomizedDelaySec=30
+
+[Install]
+WantedBy=timers.target

monitoring/systemd/knelperf-perfsnap.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=KNELPerf perfsnap collector tick (/proc+cgroupfs snapshot)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/knelperf-perfsnap.sh

monitoring/systemd/knelperf-perfsnap.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run knelperf-perfsnap every 60s
+
+[Timer]
+OnBootSec=45
+OnUnitActiveSec=60
+AccuracySec=5s
+Persistent=true
+
+[Install]
+WantedBy=timers.target

monitoring/systemd/knelperf-psi-textfile.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=KNELPerf PSI textfile collector tick (pressure avg60 gauges)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/knelperf-psi-textfile.sh

monitoring/systemd/knelperf-psi-textfile.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run knelperf-psi-textfile every 15s
+
+[Timer]
+OnBootSec=20
+OnUnitActiveSec=15
+AccuracySec=2s
+Persistent=true
+
+[Install]
+WantedBy=timers.target

monitoring/toolbox/Dockerfile

@@ -0,0 +1,16 @@
+# knelperf-toolbox — Docker image variant for docker-capable hosts
+# (workstation, VMs). On PVE nodes the LXC variant (lxc-toolbox.sh) is
+# canonical. Package list is shared: toolbox-pkgs.txt.
+# Build:  docker build -t git.knownelement.com/knel/knelperf-toolbox:v1 \
+#           -f monitoring/toolbox/Dockerfile monitoring/toolbox/
+# Docs: https://community.turnsys.com/t/328
+FROM debian:12.9-slim
+
+COPY toolbox-pkgs.txt /tmp/toolbox-pkgs.txt
+RUN apt-get update -qq && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+      $(sed 's/#.*//' /tmp/toolbox-pkgs.txt | tr '\n' ' ') && \
+    rm -rf /var/lib/apt/lists/* /tmp/toolbox-pkgs.txt
+
+WORKDIR /work
+ENTRYPOINT ["/bin/bash"]

monitoring/toolbox/knelperf-toolbox

@@ -0,0 +1,34 @@
+#!/bin/sh
+# knelperf-toolbox — front door to the on-demand perf toolbox LXC on this node.
+# usage: knelperf-toolbox run '<cmd>'   (starts the LXC if stopped, runs, leaves it up)
+#        knelperf-toolbox shell         (interactive)
+#        knelperf-toolbox start|stop|status
+# The LXC (vmid 950) is STOPPED at rest — zero footprint; stop it when done.
+# Docs: https://community.turnsys.com/t/328
+VMID=${KNELPERF_TOOLBOX_VMID:-950}
+
+ensure_running() {
+  if ! pct status "$VMID" 2>/dev/null | grep -q running; then
+    pct start "$VMID"
+    i=0
+    until pct exec "$VMID" -- true 2>/dev/null; do
+      i=$((i + 1)); [ $i -gt 45 ] && { echo "boot timeout" >&2; return 1; }
+      sleep 2
+    done
+  fi
+  # host-kernel analysis mounts — pct skips virtual-fs bind sources, so
+  # tracefs/debugfs are mounted at runtime (container is privileged +
+  # apparmor unconfined precisely for this).
+  pct exec "$VMID" -- sh -c '
+    mountpoint -q /sys/kernel/tracing || mount -t tracefs none /sys/kernel/tracing 2>/dev/null
+    mountpoint -q /sys/kernel/debug || mount -t debugfs none /sys/kernel/debug 2>/dev/null'
+}
+
+case "${1:-help}" in
+  run)    shift; ensure_running && pct exec "$VMID" -- sh -c "$*" ;;
+  shell)  ensure_running && pct enter "$VMID" ;;
+  start)  ensure_running ;;
+  stop)   pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" ;;
+  status) pct status "$VMID" 2>/dev/null || echo "vmid $VMID not present" ;;
+  *)      sed -n '2,6p' "$0" | sed 's/^# \?//' ;;
+esac

monitoring/toolbox/lxc-toolbox.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# knelperf-toolbox LXC — the on-demand perf analysis container for a PVE node.
+# STOPS to zero footprint; started only when a human/agent runs a workload.
+#   lxc-toolbox.sh [--vmid N] {--create|--upgrade|--remove|--facts}
+# Why privileged + apparmor unconfined: bpftrace/perf need host kernel BPF,
+# tracefs and debugfs (pct skips virtual-fs bind sources, so the knelperf-toolbox
+# wrapper mounts tracefs/debugfs at runtime). Binds: /host/cgroup, /host/var/log
+# (ro, offline sar analysis), block devs for smart/nvme. Package list lives in
+# toolbox-pkgs.txt next to this script.
+# Docs: https://community.turnsys.com/t/328
+set -euo pipefail
+[ "$(id -u)" = 0 ] || { echo "run as root" >&2; exit 1; }
+here=$(cd "$(dirname "$0")" && pwd)
+VMID=950
+TPL=debian-12-standard_12.12-1_amd64.tar.zst
+case "${1:-}" in
+  --vmid) VMID=$2; shift 2 ;;
+esac
+case "${1:-}" in
+  --facts)
+    pct config "$VMID" 2>/dev/null || echo "vmid $VMID not present"
+    pct status "$VMID" 2>/dev/null || true
+    exit 0 ;;
+  --remove)
+    pct shutdown "$VMID" --timeout 60 2>/dev/null || pct stop "$VMID" 2>/dev/null || true
+    pct destroy "$VMID" --purge 1
+    rm -f /usr/local/sbin/knelperf-toolbox
+    echo "toolbox $VMID removed"
+    exit 0 ;;
+  --create|--upgrade) action=$1 ;;
+  *) sed -n 's/^# \?//p' "$0" | sed -n '2,5p'; exit 1 ;;
+esac
+
+if [ "$action" = --create ]; then
+  if ! pct status "$VMID" >/dev/null 2>&1; then
+    pveam list local 2>/dev/null | grep -q "$TPL" || pveam download local "$TPL"
+    # block-device passthrough for smartctl/nvme (first 8 devices)
+    dev_args=()
+    for d in /dev/nvme[0-9]n1 /dev/sd[a-z]; do
+      [ -b "$d" ] || continue
+      [ "${#dev_args[@]}" -ge 16 ] && break
+      dev_args+=("--dev$(( ${#dev_args[@]} / 2 ))" "$d")
+    done
+    pct create "$VMID" "local:vztmpl/$TPL" \
+      --hostname knelperf-toolbox \
+      --ostype debian \
+      --cores 2 --memory 2048 --swap 512 \
+      --rootfs local-lvm:8 \
+      --net0 name=eth0,bridge=vmbr0,ip=dhcp \
+      --unprivileged 0 --onboot 0 \
+      --mp3 /sys/fs/cgroup,mp=/host/cgroup \
+      --mp4 /var/log,mp=/host/var/log,ro=1 \
+      "${dev_args[@]}"
+  fi
+  # apparmor unconfined for host-kernel BPF/tracing access. PVE 9 dropped
+  # `pct set --raw.lxc`; lxc.* keys go straight into the config file and
+  # PVE::LXC honors an explicit profile over its generated one.
+  if ! grep -q '^lxc.apparmor.profile:' /etc/pve/lxc/"$VMID".conf; then
+    echo 'lxc.apparmor.profile: unconfined' >> /etc/pve/lxc/"$VMID".conf
+    pct status "$VMID" 2>/dev/null | grep -q running && pct reboot "$VMID"
+  fi
+fi
+
+# --create continues here; --upgrade rejoins an existing container
+was_running=1
+pct status "$VMID" 2>/dev/null | grep -q running || { was_running=0; pct start "$VMID"; }
+i=0
+until pct exec "$VMID" -- true 2>/dev/null; do
+  i=$((i + 1)); [ $i -gt 60 ] && { echo "boot timeout" >&2; exit 1; }
+  sleep 2
+done
+pkgs=$(sed 's/#.*//' "$here/toolbox-pkgs.txt" | tr '\n' ' ')
+pct exec "$VMID" -- bash -c \
+  "DEBIAN_FRONTEND=noninteractive apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -qq -y install $pkgs"
+install -m 0755 "$here/knelperf-toolbox" /usr/local/sbin/knelperf-toolbox
+[ "$was_running" = 0 ] && pct shutdown "$VMID" --timeout 60
+echo "toolbox $VMID $action done: knelperf-toolbox wrapper installed"
+pct status "$VMID"

monitoring/toolbox/toolbox-pkgs.txt

@@ -0,0 +1,37 @@
+# knelperf-toolbox package manifest — shared by the LXC bootstrap and the
+# Docker image variant. One package (or apt argument) per line; # = comment.
+# Docs: https://community.turnsys.com/t/328
+bpftrace
+bpfcc-tools
+sysstat
+iotop
+sysbench
+fio
+iperf3
+stress-ng
+linux-perf
+jq
+bc
+curl
+procps
+psmisc
+util-linux
+pciutils
+usbutils
+ethtool
+net-tools
+bind9-dnsutils
+smartmontools
+nvme-cli
+hdparm
+lm-sensors
+numactl
+hwloc-nox
+strace
+ltrace
+file
+rsync
+openssh-client
+vim-tiny
+less
+tmux

scheduler/README.md

@@ -1,4 +1,4 @@
-# KNELPerf scheduler — breakout from Ultix (#826)
+# 🗓️ KNELPerf scheduler — breakout from Ultix (#826)
 
 The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`)
 was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains

scheduler/collectors/knelperf-psi-textfile.sh

@@ -1,9 +1,11 @@
 #!/usr/bin/env bash
 # KNELPerf PSI (pressure stall info) avg60 -> node_exporter textfile collector.
 # Ported from ultix staged/ukrrs-psi-textfile.sh (#826); generalizations:
-#   - cgroup glob is env-driven (any slice namespace, not just ukrrs-*)
+#   - cgroup globs are env-driven (space-separated LIST, any slice namespace)
 #   - metric prefix is env-driven (default knelperf_)
 set -u
+CONF=/etc/knelperf/perfsnap.conf
+[ -r "$CONF" ] && . "$CONF"
 OUT_DIR=${KNELPERF_TEXTFILE_DIR:-/var/lib/node_exporter/textfile}
 CGROOT=${KNELPERF_CGROOT:-/sys/fs/cgroup}
 CG_GLOB=${KNELPERF_CG_GLOB:-"$CGROOT"/system.slice/knelperf-*.slice}
@@ -27,6 +29,7 @@ emit() { # name cgroup_path
 echo "# HELP ${PREFIX}_slice_pressure_some60 PSI some avg60 (percent) per slice" >>"$tmp"
 echo "# TYPE ${PREFIX}_slice_pressure_some60 gauge" >>"$tmp"
 for cg in $CG_GLOB; do
+  case $cg in "$CGROOT"/*) ;; *) continue ;; esac   # skip unmatched glob literals
   [ -d "$cg" ] && emit "${cg##*/}" "$cg"
 done
 [ -d "$CGROOT/user.slice" ] && emit "user.slice" "$CGROOT/user.slice"

scheduler/daynight/knelperf-daynight.sh

@@ -55,7 +55,7 @@ if [ -n "$BLOCK_DEVS" ]; then
   for dev in $BLOCK_DEVS; do
     blockdev --setra "$ra" "$dev" 2>/dev/null || log "readahead skip: $dev"
     for part in "${dev}"[0-9]*; do
-      [ -e "$part" ] && blockdev --setra "$ra" "$part" 2>/dev/null || true
+      if [ -e "$part" ]; then blockdev --setra "$ra" "$part" 2>/dev/null || true; fi
     done
   done
 fi

tools/grav-publish.sh

@@ -0,0 +1,193 @@
+#!/usr/bin/env bash
+# KNELPerf -> KNELPerfHub publisher.
+#
+# Architecture ruling (founder, 2026-09-06): NO server-side tooling runs in
+# the Grav container. This script renders pages as markdown into the
+# KNELPerfHub git repo and PUSHES; the Cloudron VPS's ukrrs-hub-pull.timer
+# git-pulls that repo into the Grav pages dir every 5 min (Grav
+# cache.check=file picks changes up).
+#
+# TRANSPARENCY RULE (mechanical): revenue/expense detail is LLC-private.
+# redact() strips financial literals from every public render — the
+# public/private line is enforced by script, not by authoring discipline.
+# The reports/ and content/ lists below are explicit whitelists.
+set -euo pipefail
+REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)
+HUB_REPO=${HUB_REPO:-$HOME/projects/KNEL/KNELPerfHub}
+SOLAR_REPO=${SOLAR_REPO:-$HOME/projects/KNEL/PhysicalPlant/SITER-Solar}
+PAGES=$HUB_REPO/performance
+
+fm() { printf -- "---\ntitle: %s\nvisible: true\n---\n\n" "$1"; }
+
+# Strip LLC-private financial literals (contract rates, project costs, bill
+# amounts). Keep methodology public; keep numbers private.
+# shellcheck disable=SC2016  # single-quoted sed programs intentionally contain $ literals
+redact() {
+  sed -e 's/\$0\.085[^ )]*/$[redacted — LLC-private]/g' \
+      -e 's/\$0\.040\|\$0\.04[ )/]/$[redacted — LLC-private]/g' \
+      -e 's/8\.5¢/[redacted]/g; s/4¢/[redacted]/g' \
+      -e 's/\$4,100/[redacted — LLC-private]/g' \
+      -e 's/\$301\.08\|\$264\.47\|\$3,967\.[0-9]*/[redacted — LLC-private]/g' \
+      -e 's/23,?95[0-9]\|22,?614[0-9]*/[redacted]/g'
+}
+
+# ---------- 01 Home ----------
+DASH=$(mktemp)
+{
+  fm "Performance Hub"
+  echo "# ⚡ KNEL Performance Hub"
+  echo
+  echo "Live operational-performance intelligence for the Turnsys PFV fleet —"
+  echo "published openly, updated automatically. Ultra-transparent by"
+  echo "default; [revenue and expense detail stays LLC-private]"
+  echo "(/performance/transparency)."
+  echo
+  echo "- **What we do**: [the program](/performance/program) — a continuous"
+  echo "  baseline → audit → tweak → re-baseline loop across a 9-node"
+  echo "  Proxmox fleet, scheduled around solar surplus and grid risk."
+  echo "- **The metal**: [fleet inventory](/performance/fleet)."
+  echo "- **Deep dives**: [reports](/performance/reports)."
+  echo "- **Methodology**: [grid-risk intel](/performance/intel/ercot-risk) ·"
+  echo "  [solar arbitrage](/performance/intel/solar-arbitrage)."
+  echo "- **Right now**: [status dashboard](/performance/status)."
+  echo
+  echo "## ⚡ ERCOT risk right now (LZ_SOUTH)"
+  echo '```json'
+  if [ -x "$SOLAR_REPO/solar-analysis/ercot-brief.sh" ]; then
+    "$SOLAR_REPO/solar-analysis/ercot-brief.sh" 2>/dev/null \
+      || tail -1 /var/tmp/ercot-brief-history.tsv 2>/dev/null \
+      || echo '{"error":"price feed pending API key registration"}'
+  else
+    tail -1 /var/tmp/ercot-brief-history.tsv 2>/dev/null || echo '{"error":"no data yet"}'
+  fi
+  echo '```'
+} > "$DASH"
+mkdir -p "$PAGES"; mv "$DASH" "$PAGES/page.md"
+
+# ---------- 02 Program (architecture, redacted) ----------
+mkdir -p "$PAGES/program"
+fm "The Program" > "$PAGES/program/page.md"
+redact < "$REPO_ROOT/docs/ARCHITECTURE.md" >> "$PAGES/program/page.md"
+
+# ---------- 03 Fleet ----------
+mkdir -p "$PAGES/fleet"
+F=$(mktemp)
+{
+  fm "Fleet"
+  echo "# 🖥️ Fleet inventory"
+  echo
+  echo "Hardware transparency: every node serving this program, verified by"
+  echo "live dmidecode survey (2026-09-06). Capacity planning and margin"
+  echo "accounting run against this table."
+  echo
+  sed -n '/^| node |/,/^$/p' "$REPO_ROOT/docs/ARCHITECTURE.md" | head -n -1
+  echo
+  echo "Power telemetry per node: RAPL (tsys1/3/4/7/9) · iDRAC (tsys6/7) ·"
+  echo "metered PDU (tsys5) — methodology in the"
+  echo "[AMT report](/performance/reports/report-amt-power-telemetry)."
+} > "$F"; mv "$F" "$PAGES/fleet/page.md"
+
+# ---------- 04 Reports (whitelist) ----------
+mkdir -p "$PAGES/reports"
+for report in report-amt-power-telemetry report-moonlight-desktop; do
+  R=$(mktemp); { fm "$report"; redact < "$REPO_ROOT/docs/$report.md"; } > "$R"
+  mkdir -p "$PAGES/reports/$report"; mv "$R" "$PAGES/reports/$report/page.md"
+done
+
+# ---------- 05 Intel ----------
+mkdir -p "$PAGES/intel/ercot-risk" "$PAGES/intel/solar-arbitrage"
+fm "ERCOT Risk Intelligence" > "$PAGES/intel/ercot-risk/page.md"
+redact < "$SOLAR_REPO/docs/oncor-realtime-arbitrage.md" | sed -n '/^## Day-to-day/,$p' >> "$PAGES/intel/ercot-risk/page.md"
+fm "Solar Arbitrage" > "$PAGES/intel/solar-arbitrage/page.md"
+redact < "$SOLAR_REPO/docs/oncor-realtime-arbitrage.md" | sed -n '1,/^## Day-to-day/p' | head -n -1 >> "$PAGES/intel/solar-arbitrage/page.md"
+
+# ---------- 06 Status (auto dashboard) ----------
+mkdir -p "$PAGES/status"
+S=$(mktemp)
+{
+  fm "Status"
+  echo "# 📈 Status dashboard"
+  echo
+  echo "Auto-published $(date -Is) from KNELPerf. Source:"
+  echo "https://git.knownelement.com/KNEL/KNELPerf"
+  echo
+  echo "## 🌤️ Solar arbitrage verdict (latest run)"
+  echo '```json'
+  [ -x "$SOLAR_REPO/solar-analysis/oncor-arbitrage.sh" ] \
+    && "$SOLAR_REPO/solar-analysis/oncor-arbitrage.sh" 2>/dev/null \
+    || echo '{"error":"source unavailable"}'
+  echo '```'
+  echo
+  echo "## 📋 Baseline index"
+  echo
+  echo "| host | latest bundle |"
+  echo "|---|---|"
+  found=0
+  for d in "$REPO_ROOT"/data/baselines/*/; do
+    [ -d "$d" ] || continue
+    h=$(basename "$d"); l=$(find "$d" -maxdepth 1 -mindepth 1 -type d -printf '%f\n' 2>/dev/null | sort | tail -1)
+    if [ -n "$l" ]; then echo "| $h | $l |"; found=1; fi
+  done
+  [ "$found" = 1 ] || echo "| (none yet — first fleet baseline wave pending) | |"
+} > "$S"; mv "$S" "$PAGES/status/page.md"
+
+# ---------- 07 Transparency ----------
+mkdir -p "$PAGES/transparency"
+fm "Transparency" > "$PAGES/transparency/page.md"
+cat >> "$PAGES/transparency/page.md" <<'EOF'
+# 🔓 Transparency policy
+
+We run an ultra-open and transparent organization. This site and its
+[content repo](https://git.knownelement.com/KNEL/KNELPerfHub) are **public
+read-only** — performance data, benchmarks, risk intelligence,
+architecture and decision reports are all in the open.
+
+**The principle**: our code, formulas, and methodology are open; the
+inputs to them are not. **This site is assumed 100% public** — nothing is
+published here that isn't safe for the open internet. Revenue and expense
+detail (contract rates, system costs, billing history, per-tenant
+margins, customer data) is private to LLC members and lives in our ERP:
+private financial dashboards and reports are built in Dolibarr
+(SSO + role-gated), as part of this same program — never on this site.
+The public/private line is enforced mechanically: the publishing script
+redacts private literals from every public page at render time — see
+[`grav-publish.sh`](https://git.knownelement.com/KNEL/KNELPerf/src/branch/main/tools/grav-publish.sh).
+Public market signals (e.g. ERCOT prices) are open data and stay open.
+
+Publishing is exclusively git-based: content lands via pull from the
+public repo; the site runs no foreign tooling. The Grav admin exists only
+to prevent first-run setup mode; content never flows through it.
+EOF
+
+# ---------- 08 About ----------
+mkdir -p "$PAGES/about"
+fm "About" > "$PAGES/about/page.md"
+cat >> "$PAGES/about/page.md" <<'EOF'
+# 👥 About the performance organization
+
+The KNEL Performance organization runs fleet performance optimization for
+Turnsys: the baseline → audit → tweak → re-baseline loop, capacity
+planning, and power-aware scheduling across two sites (main PFV datacenter
++ SiTES fabrication shop). It reports to the VP of Technical Operations
+with a dotted line to the VP of Facilities.
+
+- Work tracking: [Redmine #826](https://projects.knownelement.com/issues/826)
+  (program anchor ticket)
+- Code: [KNELPerf](https://git.knownelement.com/KNEL/KNELPerf) ·
+  [KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar) ·
+  [this site's content](https://git.knownelement.com/KNEL/KNELPerfHub)
+- License: AGPLv3 across the program
+EOF
+
+# ---------- push ----------
+cd "$HUB_REPO"
+if [ -n "$(git status --porcelain)" ]; then
+  git add -A performance
+  git commit -q -m "Auto-publish $(date +%F_%H:%M) (#826)
+
+https://projects.knownelement.com/issues/826"
+  git push -q origin main
+  echo "pushed to KNEL/KNELPerfHub ($(date -Is)); site refreshes within 5 min"
+else
+  echo "no changes to publish ($(date -Is))"
+fi