Docs beautify pass: README + ARCHITECTURE to PhysicalPlant standard (#826)
ci / audit (push) Successful in 22s
ci / audit (push) Successful in 22s
Emoji headers, shields, mermaid loop diagram, contents TOC, module table, quick start, docs index, cross-links. Pattern: pfv-bms README. https://projects.knownelement.com/issues/826#note-10
This commit is contained in:
@@ -1,14 +1,90 @@
|
||||
# perf
|
||||
# ⚡ KNELPerf — dynamic performance optimization for the PFV fleet
|
||||
|
||||
> **Baseline → Audit → Tweak → Re-baseline, on autopilot** —
|
||||
> a bash-only performance loop for a nine-node Proxmox fleet, wired to
|
||||
> solar surplus, ERCOT risk signals, and three classes of paid work.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
KNELPerf v2 (2026-09-06, two-week program
|
||||
[#826](https://projects.knownelement.com/issues/826)) breaks the
|
||||
organically-grown Ultix scheduling subsystem out into a fleet-grade module,
|
||||
adds the missing CPU benchmark leg, and drives a continuous
|
||||
baseline/audit/tweak/re-baseline loop across every host.
|
||||
|
||||
> **Docs live on Discourse — this repo is the executable source of truth.**
|
||||
> **Topic:** https://community.turnsys.com/t/328
|
||||
> **Redmine:** https://projects.knownelement.com/issues/709 · **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)
|
||||
> **Redmine:** [#826](https://projects.knownelement.com/issues/826) (v2 program) ·
|
||||
> [#709](https://projects.knownelement.com/issues/709) (v1, closed) ·
|
||||
> **Split from** KNEL/PFVCluster@041d311 per [#769](https://projects.knownelement.com/issues/769)
|
||||
|
||||
Performance baseline + tuning: fio/iperf3 harness v1.1 (PSI, load, device identity), tuned-profile deploys, audit scripts (probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io). Switch .cmds files now live in KNEL/netinfra. [\#709] [\#732] [\#737]
|
||||
---
|
||||
|
||||
## Layout
|
||||
- `scripts/` — rule engine + hooks (see `bash scripts/check-rules.sh --fast`)
|
||||
- (imported content at repo root, mirroring its PFVCluster path layout)
|
||||
## 🗺️ The loop
|
||||
|
||||
## Provenance
|
||||
Code imported from KNEL/PFVCluster (041d311); full git history retained in PFVCluster. Enforcement layer copied per ADOPTING.md. IaC consumers: KNEL/KNELIAC references this repo.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
B[📋 BASELINE<br/>cpu-bench + fio/iperf3<br/>+ PSI snapshot] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
|
||||
A -->|drift found| T[🔧 TWEAK<br/>idempotent tweaks/<id>.sh<br/>GLPI CR-gated on prod]
|
||||
T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
|
||||
R --> A
|
||||
A -->|clean| B
|
||||
|
||||
subgraph SIGNALS[📡 External signals]
|
||||
SOL[☀️ solar surplus<br/>HA + SITER-Solar]
|
||||
ERC[⚡ ERCOT risk<br/>LZ_SOUTH RTM]
|
||||
RES[📅 reservations<br/>KillBill]
|
||||
end
|
||||
SIGNALS -.gates class-3 batch load.-> A
|
||||
```
|
||||
|
||||
## 📦 Modules
|
||||
|
||||
| Module | What it does | Entry point |
|
||||
|---|---|---|
|
||||
| `loop/` | the baseline→audit→tweak→re-baseline driver, CPU bench (openssl-based), tolerance diff | `loop/perf-loop.sh baseline <host>` |
|
||||
| `scheduler/` | Ultix breakout: day/night profile engine (conf-driven redesign), PSI/perfsnap collectors, proxmox-ctl, systemd timer templates | `scheduler/README.md` |
|
||||
| `bench/` | fleet storage+network harness v1.1 (fio JSON, iperf3) | `bench/bench-run.sh` |
|
||||
| `scripts/` | tuning deploys, probes (network/storage/drift/VM disks/guest IO), LACP diagnostics, switch audit | `Makefile` verbs |
|
||||
| `docs/` | architecture + decision reports | index below |
|
||||
|
||||
## 🚀 Quick start
|
||||
|
||||
```bash
|
||||
# capture a baseline bundle for a host (read-only)
|
||||
./loop/perf-loop.sh baseline pfv-tsys7
|
||||
|
||||
# audit vs latest baseline (tolerance diff, catches governor flips)
|
||||
./loop/perf-loop.sh audit pfv-tsys7
|
||||
|
||||
# apply a CR-gated tweak, then re-baseline and compare
|
||||
./loop/perf-loop.sh tweak pfv-tsys7 <tweak-id>
|
||||
./loop/perf-loop.sh rebaseline pfv-tsys7
|
||||
|
||||
# CPU leg alone (fast, zero-install)
|
||||
KNELPERF_CPUBENCH_SECS=5 ./loop/cpu-bench.sh
|
||||
```
|
||||
|
||||
## 📚 Docs index
|
||||
|
||||
| Doc | Contents |
|
||||
|---|---|
|
||||
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | the full program: loop design, fleet hardware table, resource groups & placement, k8s-vs-Slurm, telemetry spine, workload classes, solar + two-site power economics |
|
||||
| [docs/report-amt-power-telemetry.md](docs/report-amt-power-telemetry.md) | Intel AMT/vPro per-system feasibility — spoiler: no AMT version ever exposed watts; RAPL/iDRAC are the real paths |
|
||||
| [docs/report-moonlight-desktop.md](docs/report-moonlight-desktop.md) | KDE/xrdp VM → containerized desktop from k8s (webtop/KasmVNC now, Selkies on the future GPU node) |
|
||||
| [scheduler/README.md](scheduler/README.md) | Ultix breakout map, redesign deltas, 8 lessons-learned carried forward |
|
||||
|
||||
Related repos: [KNEL/KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar)
|
||||
(solar economics + ONCOR arbitrage + ERCOT risk brief) ·
|
||||
[KNEL/KNELIAC](https://git.knownelement.com/KNEL/KNELIAC) (AWX deployment of these scripts)
|
||||
|
||||
## 🧭 Provenance & enforcement
|
||||
|
||||
Code imported from KNEL/PFVCluster (041d311; full git history retained
|
||||
there); v2 scheduler module broken out of `~/projects/ultix` 2026-09-06.
|
||||
CI: `.gitea/workflows/ci.yml` (shellcheck-zero + secret scan +
|
||||
check-rules). Framework: `scripts/check-rules.sh --fast`; ticket gate per
|
||||
AGENTS.md.
|
||||
|
||||
+32
-19
@@ -1,17 +1,30 @@
|
||||
# KNELPerf v2 — architecture for dynamic, intelligent performance optimization
|
||||
# 🏗️ KNELPerf v2 — architecture for dynamic, intelligent performance optimization
|
||||
|
||||
Status: DRAFT v1 (2026-09-06, #826). Founder directive: end-to-end
|
||||
baseline → audit/benchmark → tweak → re-baseline loop, two-week program.
|
||||
> Status: DRAFT v1 (2026-09-06,
|
||||
> [#826](https://projects.knownelement.com/issues/826)). Founder directive:
|
||||
> end-to-end baseline → audit/benchmark → tweak → re-baseline loop,
|
||||
> two-week program.
|
||||
|
||||
## 📑 Contents
|
||||
|
||||
1. [The loop](#1-the-loop) · 2. [Scheduling architecture](#2-scheduling-architecture-ultix-breakout-redesign) ·
|
||||
3. [Proxmox resource groups & VM balancing](#3-proxmox-resource-groups--vm-balancing) ·
|
||||
4. [k8s scheduling & Slurm](#4-where-k8s-scheduling-fits-slurm) ·
|
||||
5. [Telemetry spine](#5-telemetry-spine-beszel--power) ·
|
||||
6. [Workload priority classes](#6-workload-priority-classes-founder-set-order) ·
|
||||
7. [Solar + Home Assistant](#7-solar--home-assistant-integration) ·
|
||||
8. [Two-site power economics](#8-two-site-power-economics-site-2-printer-farm--laser--cnc) ·
|
||||
9. [Hardware plan hooks](#9-hardware-plan-hooks) · 10. [Organization](#10-organization)
|
||||
|
||||
## 1. The loop
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ │
|
||||
BASELINE ──► AUDIT/BENCHMARK ──► TWEAK ──► RE-BASELINE ─┘
|
||||
(capture) (diff + beszel/ (CR-gated, (compare vs
|
||||
PSI/rapl context) idempotent) baseline; keep
|
||||
or roll back)
|
||||
```mermaid
|
||||
flowchart LR
|
||||
B[📋 BASELINE<br/>cpu-bench.tsv + bench-run.tsv<br/>+ PSI snapshot per host] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
|
||||
A -->|drift found| T[🔧 TWEAK<br/>tweaks/<id>.sh — idempotent,<br/>GLPI CR-gated on prod]
|
||||
T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
|
||||
R --> A
|
||||
A -->|clean| B
|
||||
```
|
||||
|
||||
- **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
|
||||
@@ -31,7 +44,7 @@ Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
|
||||
the audit itself is low-cost); re-baseline only after a tweak or hardware
|
||||
change. First full-fleet baseline wave is this week's deliverable.
|
||||
|
||||
## 2. Scheduling architecture (Ultix breakout redesign)
|
||||
## 2. 🗓️ Scheduling architecture (Ultix breakout redesign)
|
||||
|
||||
See `scheduler/README.md` for the port map and the lessons-learned list. Core
|
||||
redesign decisions:
|
||||
@@ -45,7 +58,7 @@ redesign decisions:
|
||||
4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
|
||||
payload; job templates own the rollout waves.
|
||||
|
||||
## 3. Proxmox resource groups & VM balancing
|
||||
## 3. 🖥️ Proxmox resource groups & VM balancing
|
||||
|
||||
Current fleet (dmidecode-verified 2026-09-06):
|
||||
|
||||
@@ -78,7 +91,7 @@ Directions:
|
||||
needs founder ruling — will be proposed on #826 with a capacity table once
|
||||
tsys8/2 land.
|
||||
|
||||
## 4. Where k8s scheduling fits; Slurm?
|
||||
## 4. ☸️ Where k8s scheduling fits; Slurm?
|
||||
|
||||
- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
|
||||
(ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
|
||||
@@ -95,7 +108,7 @@ Directions:
|
||||
- The knelperf daynight engine gives the batch window (22:00–07:00) its
|
||||
expanded resource envelope on each participating node.
|
||||
|
||||
## 5. Telemetry spine: beszel + power
|
||||
## 5. 📡 Telemetry spine: beszel + power
|
||||
|
||||
- **Beszel has 100% VM+physical coverage** → it is the always-on signal for
|
||||
the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
|
||||
@@ -114,7 +127,7 @@ Directions:
|
||||
legs keep fleet draw separable from house load — needed exactly on EEA
|
||||
days, so local (non-cloud) telemetry matters.
|
||||
|
||||
## 6. Workload priority classes (founder-set order)
|
||||
## 6. 🏆 Workload priority classes (founder-set order)
|
||||
|
||||
| class | tenants | profile | scheduling |
|
||||
|---|---|---|---|
|
||||
@@ -122,7 +135,7 @@ Directions:
|
||||
| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
|
||||
| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
|
||||
|
||||
## 7. Solar + Home Assistant integration
|
||||
## 7. ☀️ Solar + Home Assistant integration
|
||||
|
||||
- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
|
||||
Assistant exposes real-time production. The loop consumes: surplus watts =
|
||||
@@ -139,7 +152,7 @@ Directions:
|
||||
(deprecated python script), header/ROI inconsistencies, and fixed-cost
|
||||
scenario assumptions — fixes landing in that repo.
|
||||
|
||||
## 8. Two-site power economics (site 2: printer farm / laser / CNC)
|
||||
## 8. 🏭 Two-site power economics (site 2: printer farm / laser / CNC)
|
||||
|
||||
The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
|
||||
MUCH more expensive grid power than the main site. That asymmetry drives
|
||||
@@ -165,7 +178,7 @@ scheduling policy as much as raw performance:
|
||||
- KillBill reservation calendar × solar forecast × site-2 tariff = the
|
||||
promotion/job-timing advisor output: "cheapest week to run X".
|
||||
|
||||
## 9. Hardware plan hooks
|
||||
## 9. 🔩 Hardware plan hooks
|
||||
|
||||
- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
|
||||
are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
|
||||
@@ -175,7 +188,7 @@ scheduling policy as much as raw performance:
|
||||
- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
|
||||
with tsys3 for latency-sensitive class-1 storage.
|
||||
|
||||
## 10. Organization
|
||||
## 10. 👥 Organization
|
||||
|
||||
Performance management org reports to VPTechOps, dotted line VPFacilities —
|
||||
identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
|
||||
# 🔌 Report: Intel AMT / vPro per-system power-telemetry feasibility (tsys1,3,4,5,9 + 6,7)
|
||||
|
||||
Status: PROPOSAL (v1, 2026-09-06). Ticket: #826. Hardware verified live via
|
||||
dmidecode 2026-09-06 (tsys9 not yet reachable — identification step included).
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
|
||||
# 🖼️ Report: replacing the KDE/xrdp VM with a containerized desktop served from k8s
|
||||
|
||||
Status: PROPOSAL (v1, 2026-09-06) — awaits founder ruling. Ticket: #826.
|
||||
Source research: web-surveyed 2026-09-06 (sources linked at the end).
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# KNELPerf scheduler — breakout from Ultix (#826)
|
||||
# 🗓️ KNELPerf scheduler — breakout from Ultix (#826)
|
||||
|
||||
The scheduling subsystem of the Ultix workstation repo (`~/projects/ultix`)
|
||||
was broken out here on 2026-09-06 and redesigned for fleet use. Ultix remains
|
||||
|
||||
Reference in New Issue
Block a user