Files

T

ci / audit (push) Successful in 22s

Details

Docs beautify pass: README + ARCHITECTURE to PhysicalPlant standard (#826 )

Emoji headers, shields, mermaid loop diagram, contents TOC, module table,
quick start, docs index, cross-links. Pattern: pfv-bms README.

https://projects.knownelement.com/issues/826#note-10

2026-09-06 15:05:30 -05:00

4.3 KiB

Raw Permalink Blame History

⚡ KNELPerf — dynamic performance optimization for the PFV fleet

Baseline → Audit → Tweak → Re-baseline, on autopilot — a bash-only performance loop for a nine-node Proxmox fleet, wired to solar surplus, ERCOT risk signals, and three classes of paid work.

KNELPerf v2 (2026-09-06, two-week program #826) breaks the organically-grown Ultix scheduling subsystem out into a fleet-grade module, adds the missing CPU benchmark leg, and drives a continuous baseline/audit/tweak/re-baseline loop across every host.

Docs live on Discourse — this repo is the executable source of truth. Topic: https://community.turnsys.com/t/328 Redmine: #826 (v2 program) · #709 (v1, closed) · Split from KNEL/PFVCluster@041d311 per #769

🗺️ The loop

flowchart LR
    B[📋 BASELINE<br/>cpu-bench + fio/iperf3<br/>+ PSI snapshot] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
    A -->|drift found| T[🔧 TWEAK<br/>idempotent tweaks/&lt;id&gt;.sh<br/>GLPI CR-gated on prod]
    T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
    R --> A
    A -->|clean| B

    subgraph SIGNALS[📡 External signals]
        SOL[☀️ solar surplus<br/>HA + SITER-Solar]
        ERC[⚡ ERCOT risk<br/>LZ_SOUTH RTM]
        RES[📅 reservations<br/>KillBill]
    end
    SIGNALS -.gates class-3 batch load.-> A

📦 Modules

Module	What it does	Entry point
`loop/`	the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff	`loop/perf-loop.sh baseline <host>`
`scheduler/`	Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates	`scheduler/README.md`
`bench/`	fleet storage+network harness v1.1 (fio JSON, iperf3)	`bench/bench-run.sh`
`scripts/`	tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit	`Makefile` verbs
`docs/`	architecture + decision reports	index below

🚀 Quick start

# capture a baseline bundle for a host (read-only)
./loop/perf-loop.sh baseline pfv-tsys7

# audit vs latest baseline (tolerance diff, catches governor flips)
./loop/perf-loop.sh audit pfv-tsys7

# apply a CR-gated tweak, then re-baseline and compare
./loop/perf-loop.sh tweak pfv-tsys7 <tweak-id>
./loop/perf-loop.sh rebaseline pfv-tsys7

# CPU leg alone (fast, zero-install)
KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh

📚 Docs index

Doc	Contents
docs/ARCHITECTURE.md	the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics
docs/report-amt-power-telemetry.md	Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths
docs/report-moonlight-desktop.md	KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node)
scheduler/README.md	Ultix breakout map, redesign deltas, 8 lessons-learned carried forward

Related repos: KNEL/KNELSiterSolar (solar economics + ONCOR arbitrage + ERCOT risk brief) · KNEL/KNELIAC (AWX deployment of these scripts)

🧭 Provenance & enforcement

Code imported from KNEL/PFVCluster (041d311; full git history retained there); v2 scheduler module broken out of ~/projects/ultix 2026-09-06. CI: .gitea/workflows/ci.yml (shellcheck-zero + secret scan + check-rules). Framework: scripts/check-rules.sh --fast; ticket gate per AGENTS.md.

4.3 KiB Raw Permalink Blame History