# ⚡ KNELPerf — dynamic performance optimization for the PFV fleet

> **Baseline → Audit → Tweak → Re-baseline, on autopilot** —
> a bash-only performance loop for a nine-node Proxmox fleet, wired to
> solar surplus, ERCOT risk signals, and three classes of paid work.

![license](https://img.shields.io/badge/license-AGPLv3-blue)
![bash](https://img.shields.io/badge/pure-bash-4EAA25?logo=gnubash&logoColor=white)
![loop](https://img.shields.io/badge/loop-baseline--%3Eaudit--%3Etweak--%3Ere--baseline-9c27b0)
![nodes](https://img.shields.io/badge/PVE%20nodes-7%20%2B%202%20coming-f97316)

KNELPerf v2 (2026-09-06, two-week program
[#826](https://projects.knownelement.com/issues/826)) breaks the
organically-grown Ultix scheduling subsystem out into a fleet-grade module,
adds the missing CPU benchmark leg, and drives a continuous
baseline/audit/tweak/re-baseline loop across every host.

> **Docs live on Discourse — this repo is the executable source of truth.**
> **Topic:** https://community.turnsys.com/t/328
> **Redmine:** [#826](https://projects.knownelement.com/issues/826) (v2 program) ·
> [#709](https://projects.knownelement.com/issues/709) (v1, closed) ·
> **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)

---

## 🗺️ The loop

```mermaid
flowchart LR
    B[📋 BASELINE<br/>cpu-bench + fio/iperf3<br/>+ PSI snapshot] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
    A -->|drift found| T[🔧 TWEAK<br/>idempotent tweaks/&lt;id&gt;.sh<br/>GLPI CR-gated on prod]
    T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
    R --> A
    A -->|clean| B

    subgraph SIGNALS[📡 External signals]
        SOL[☀️ solar surplus<br/>HA + SITER-Solar]
        ERC[⚡ ERCOT risk<br/>LZ_SOUTH RTM]
        RES[📅 reservations<br/>KillBill]
    end
    SIGNALS -.gates class-3 batch load.-> A
```

## 📦 Modules

| Module | What it does | Entry point |
|---|---|---|
| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline <host>` |
| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` |
| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` |
| `scripts/` | tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit | `Makefile` verbs |
| `docs/` | architecture + decision reports | index below |

## 🚀 Quick start

```bash
# capture a baseline bundle for a host (read-only)
./loop/perf-loop.sh baseline pfv-tsys7

# audit vs latest baseline (tolerance diff, catches governor flips)
./loop/perf-loop.sh audit pfv-tsys7

# apply a CR-gated tweak, then re-baseline and compare
./loop/perf-loop.sh tweak pfv-tsys7 <tweak-id>
./loop/perf-loop.sh rebaseline pfv-tsys7

# CPU leg alone (fast, zero-install)
KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh
```

## 📚 Docs index

| Doc | Contents |
|---|---|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics |
| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths |
| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) |
| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward |

Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar)
(solar economics + ONCOR arbitrage + ERCOT risk brief) ·
[KNEL/KNELIAC](https://git.knownelement.com/KNEL/KNELIAC) (AWX deployment of these scripts)

## 🧭 Provenance & enforcement

Code imported from KNEL/PFVCluster (041d311; full git history retained
there); v2 scheduler module broken out of `~/projects/ultix` 2026-09-06.
CI: `.gitea/workflows/ci.yml` (shellcheck-zero + secret scan +
check-rules). Framework: `scripts/check-rules.sh --fast`; ticket gate per
AGENTS.md.