ic-builder
3680f1dba1
ci / audit (push) Successful in 26s
- monitoring/node-install.sh: idempotent per-node installer (sar HISTORY=28, three oneshot timers, per-node conf with qemu.slice/lxc cgroup globs) - monitoring/collectors + systemd units: iotop-snap 15min, perfsnap 60s, psi-textfile 15s (now sources /etc/knelperf/perfsnap.conf; multi-glob) - monitoring/toolbox: pct LXC (vmid 950, privileged, apparmor unconfined, stopped at rest) + knelperf-toolbox wrapper + Dockerfile variant - monitoring/deploy-fleet.sh: workstation-side rollout driver - piloted on pfv-tsys1: per-VM PSI gauges verified live Redmine: https://projects.knownelement.com/issues/826
93 lines
4.6 KiB
Markdown
93 lines
4.6 KiB
Markdown
# ⚡ KNELPerf — dynamic performance optimization for the PFV fleet
|
|
|
|
> **Baseline → Audit → Tweak → Re-baseline, on autopilot** —
|
|
> a bash-only performance loop for a nine-node Proxmox fleet, wired to
|
|
> solar surplus, ERCOT risk signals, and three classes of paid work.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
KNELPerf v2 (2026-09-06, two-week program
|
|
[#826](https://projects.knownelement.com/issues/826)) breaks the
|
|
organically-grown Ultix scheduling subsystem out into a fleet-grade module,
|
|
adds the missing CPU benchmark leg, and drives a continuous
|
|
baseline/audit/tweak/re-baseline loop across every host.
|
|
|
|
> **Docs live on Discourse — this repo is the executable source of truth.**
|
|
> **Topic:** https://community.turnsys.com/t/328
|
|
> **Redmine:** [#826](https://projects.knownelement.com/issues/826) (v2 program) ·
|
|
> [#709](https://projects.knownelement.com/issues/709) (v1, closed) ·
|
|
> **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)
|
|
|
|
---
|
|
|
|
## 🗺️ The loop
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
B[📋 BASELINE<br/>cpu-bench + fio/iperf3<br/>+ PSI snapshot] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
|
|
A -->|drift found| T[🔧 TWEAK<br/>idempotent tweaks/<id>.sh<br/>GLPI CR-gated on prod]
|
|
T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
|
|
R --> A
|
|
A -->|clean| B
|
|
|
|
subgraph SIGNALS[📡 External signals]
|
|
SOL[☀️ solar surplus<br/>HA + SITER-Solar]
|
|
ERC[⚡ ERCOT risk<br/>LZ_SOUTH RTM]
|
|
RES[📅 reservations<br/>KillBill]
|
|
end
|
|
SIGNALS -.gates class-3 batch load.-> A
|
|
```
|
|
|
|
## 📦 Modules
|
|
|
|
| Module | What it does | Entry point |
|
|
|---|---|---|
|
|
| `monitoring/` | fleet instrumentation: sysstat 28d tuning, perfsnap/PSI/iotop timers, on-demand perf toolbox container (LXC on PVE nodes, Docker image for the rest) | `monitoring/deploy-fleet.sh` |
|
|
| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline <host>` |
|
|
| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` |
|
|
| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` |
|
|
| `scripts/` | tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit | `Makefile` verbs |
|
|
| `docs/` | architecture + decision reports | index below |
|
|
|
|
## 🚀 Quick start
|
|
|
|
```bash
|
|
# capture a baseline bundle for a host (read-only)
|
|
./loop/perf-loop.sh baseline pfv-tsys7
|
|
|
|
# audit vs latest baseline (tolerance diff, catches governor flips)
|
|
./loop/perf-loop.sh audit pfv-tsys7
|
|
|
|
# apply a CR-gated tweak, then re-baseline and compare
|
|
./loop/perf-loop.sh tweak pfv-tsys7 <tweak-id>
|
|
./loop/perf-loop.sh rebaseline pfv-tsys7
|
|
|
|
# CPU leg alone (fast, zero-install)
|
|
KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh
|
|
```
|
|
|
|
## 📚 Docs index
|
|
|
|
| Doc | Contents |
|
|
|---|---|
|
|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics |
|
|
| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths |
|
|
| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) |
|
|
| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward |
|
|
| [monitoring/README.md](monitoring/README.md) | what lands on each node, toolbox cookbook, analysis recipes, prod/CR gating |
|
|
|
|
Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar)
|
|
(solar economics + ONCOR arbitrage + ERCOT risk brief) ·
|
|
[KNEL/KNELIAC](https://git.knownelement.com/KNEL/KNELIAC) (AWX deployment of these scripts)
|
|
|
|
## 🧭 Provenance & enforcement
|
|
|
|
Code imported from KNEL/PFVCluster (041d311; full git history retained
|
|
there); v2 scheduler module broken out of `~/projects/ultix` 2026-09-06.
|
|
CI: `.gitea/workflows/ci.yml` (shellcheck-zero + secret scan +
|
|
check-rules). Framework: `scripts/check-rules.sh --fast`; ticket gate per
|
|
AGENTS.md.
|