Auto-publish 2026-09-06_15:52 (#826)
https://projects.knownelement.com/issues/826
This commit is contained in:
@@ -0,0 +1,19 @@
|
|||||||
|
---
|
||||||
|
title: About
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# 👥 About the performance organization
|
||||||
|
|
||||||
|
The KNEL Performance organization runs fleet performance optimization for
|
||||||
|
Turnsys: the baseline → audit → tweak → re-baseline loop, capacity
|
||||||
|
planning, and power-aware scheduling across two sites (main PFV datacenter
|
||||||
|
+ SiTES fabrication shop). It reports to the VP of Technical Operations
|
||||||
|
with a dotted line to the VP of Facilities.
|
||||||
|
|
||||||
|
- Work tracking: [Redmine #826](https://projects.knownelement.com/issues/826)
|
||||||
|
(program anchor ticket)
|
||||||
|
- Code: [KNELPerf](https://git.knownelement.com/KNEL/KNELPerf) ·
|
||||||
|
[KNELSiterSolar](https://git.knownelement.com/KNEL/KNELSiterSolar) ·
|
||||||
|
[this site's content](https://git.knownelement.com/KNEL/KNELPerfHub)
|
||||||
|
- License: AGPLv3 across the program
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
title: Fleet
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🖥️ Fleet inventory
|
||||||
|
|
||||||
|
Hardware transparency: every node serving this program, verified by
|
||||||
|
live dmidecode survey (2026-09-06). Capacity planning and margin
|
||||||
|
accounting run against this table.
|
||||||
|
|
||||||
|
| node | platform | cores | RAM | storage character |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role |
|
||||||
|
| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton |
|
||||||
|
| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm |
|
||||||
|
| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) |
|
||||||
|
| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner |
|
||||||
|
| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner |
|
||||||
|
| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge |
|
||||||
|
| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe |
|
||||||
|
|
||||||
|
Power telemetry per node: RAPL (tsys1/3/4/7/9) · iDRAC (tsys6/7) ·
|
||||||
|
metered PDU (tsys5) — methodology in the
|
||||||
|
[AMT report](/performance/reports/report-amt-power-telemetry).
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
---
|
||||||
|
title: ERCOT Risk Intelligence
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
## Day-to-day ISO risk intelligence (ercot-brief.sh)
|
||||||
|
|
||||||
|
Founder ruling 2026-09-06: Base Power must remain the retail provider (no
|
||||||
|
direct ERCOT interconnection). So the ISO's signals are RISK INTELLIGENCE,
|
||||||
|
not an export market:
|
||||||
|
|
||||||
|
- `ercot-brief.sh` produces a daily risk level from LZ_SOUTH RTM (now +
|
||||||
|
24h peak): GREEN (<$150/MWh) / WATCH (>=150) / ELEVATED (>=500) /
|
||||||
|
CRITICAL (>=2000, adders/EEA territory).
|
||||||
|
- Actions it emits: precharge battery (WATCH+), defer class-3 batch load
|
||||||
|
(ELEVATED+), rtw_counterfactual_flag (RTM > $85).
|
||||||
|
- On EEA-watch days expect Base to dispatch the fleet battery and expect
|
||||||
|
RTW-plan counterfactual spikes — history TSV accumulates for trend stats.
|
||||||
|
- Same source chain as the arbitrage tool (GRIDSTATUS_API_KEY /
|
||||||
|
ERCOT_API_KEY / override — key registration still pending, see #826/#827).
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
---
|
||||||
|
title: Solar Arbitrage
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# ONCOR real-time arbitrage: sell to Base Power vs serve load
|
||||||
|
|
||||||
|
Status: v1 (2026-09-06, #826). Location: Pflugerville, Travis County, TX —
|
||||||
|
ONCOR wires, ERCOT South load zone (`LZ_SOUTH`). Tool:
|
||||||
|
`solar-analysis/oncor-arbitrage.sh`.
|
||||||
|
|
||||||
|
## The plan facts that shape everything
|
||||||
|
|
||||||
|
Base Power (verified 2026-09-06):
|
||||||
|
- **Import: $[redacted — LLC-private] flat** (contract through Oct 2028)
|
||||||
|
- **Export: $[redacted — LLC-private]/kWh flat** solar credit — Base does NOT pay real-time
|
||||||
|
wholesale for exports
|
||||||
|
|
||||||
|
So on Base Power, **both sides of your meter are flat rates**. ERCOT's
|
||||||
|
real-time market price (RTM, 15-min settlement point prices) does not change
|
||||||
|
what you pay or earn today — it is a *counterfactual* and a *planning
|
||||||
|
signal*:
|
||||||
|
|
||||||
|
## Decision rules (what the tool emits)
|
||||||
|
|
||||||
|
For each kWh of solar surplus, best action = max of:
|
||||||
|
| option | value | when it wins |
|
||||||
|
|---|---|---|
|
||||||
|
| serve own/deferrable load | $[redacted — LLC-private] avoided | almost always ([redacted] > [redacted]) |
|
||||||
|
| export to Base | $[redacted — LLC-private] flat | only when NO deferrable load exists |
|
||||||
|
| export on an RTW plan* | RTM $/kWh | only when RTM > $[redacted — LLC-private] (rare spikes) |
|
||||||
|
|
||||||
|
*counterfactual — would require switching export credit to a real-time-
|
||||||
|
wholesale retailer plan (e.g. Octopus-style RTW). Texas RTW export averages
|
||||||
|
~2¢/kWh with rare spikes >$5/kWh — as a steady diet it LOSES to Base's flat
|
||||||
|
[redacted]; it only wins if you can time exports into spikes (needs battery).
|
||||||
|
|
||||||
|
**Practical rules:**
|
||||||
|
1. **Every surplus kWh has a home: load first.** HFNOC batch compute, site-2
|
||||||
|
deferrable manufacturing, battery charging — all displace [redacted] power.
|
||||||
|
This is why the KNELPerf class-3 batch gate keys on solar surplus.
|
||||||
|
2. **Only truly surplus-after-load gets exported at [redacted].**
|
||||||
|
3. **RTM spike alerts** (`rtw_would_beat_self_consume: true`, RTM >
|
||||||
|
$85/MWh): under a future RTW plan you'd earn more exporting than
|
||||||
|
self-consuming. Track how often this happens; if frequent and clustered,
|
||||||
|
an RTW plan + battery beats Base's flat buyback for exports. Until then,
|
||||||
|
staying on Base and self-consuming is strictly better.
|
||||||
|
4. **Battery note**: Base's free battery — verify who controls dispatch
|
||||||
|
(Base dispatches it for grid services; if they own the dispatch, do not
|
||||||
|
count it in household arbitrage). Open question logged on #826.
|
||||||
|
|
||||||
|
## Price sources
|
||||||
|
|
||||||
|
Official API (api.ercot.com public API) and gridstatus.io both require a
|
||||||
|
free API key; ercot.com's dashboard JSON is behind an Imperva bot-wall
|
||||||
|
(tested 2026-09-06). Tool source precedence: `RTM_OVERRIDE` →
|
||||||
|
`GRIDSTATUS_API_KEY` → `ERCOT_API_KEY` → cache file. Key registration is a
|
||||||
|
founder action ( NeedsInput on #826).
|
||||||
|
|
||||||
|
## KNELPerf integration
|
||||||
|
|
||||||
|
`oncor-arbitrage.sh` emits `start_deferrable_load` + `rtw_would_beat_self_consume`
|
||||||
|
as JSON — the batch launcher consumes this plus the HA solar-surplus sensor
|
||||||
|
once the plant is live. Docs: KNELPerf docs/ARCHITECTURE.md §7-8.
|
||||||
|
|
||||||
|
> **Docs live on Discourse — this repo is the executable source of truth.**
|
||||||
|
|
||||||
+15
-20
@@ -5,30 +5,25 @@ visibility: private
|
|||||||
|
|
||||||
# ⚡ KNEL Performance Hub
|
# ⚡ KNEL Performance Hub
|
||||||
|
|
||||||
Auto-published 2026-09-06T15:28:59-05:00 from KNELPerf (#826). Repo:
|
Live operational-performance intelligence for the Turnsys PFV fleet —
|
||||||
https://git.knownelement.com/KNEL/KNELPerf
|
published openly, updated automatically. Ultra-transparent by
|
||||||
|
default; [revenue and expense detail stays LLC-private]
|
||||||
|
(/performance/transparency).
|
||||||
|
|
||||||
## ⚡ ERCOT risk (LZ_SOUTH)
|
- **What we do**: [the program](/performance/program) — a continuous
|
||||||
|
baseline → audit → tweak → re-baseline loop across a 9-node
|
||||||
|
Proxmox fleet, scheduled around solar surplus and grid risk.
|
||||||
|
- **The metal**: [fleet inventory](/performance/fleet).
|
||||||
|
- **Deep dives**: [reports](/performance/reports).
|
||||||
|
- **Methodology**: [grid-risk intel](/performance/intel/ercot-risk) ·
|
||||||
|
[solar arbitrage](/performance/intel/solar-arbitrage).
|
||||||
|
- **Right now**: [status dashboard](/performance/status).
|
||||||
|
|
||||||
|
## ⚡ ERCOT risk right now (LZ_SOUTH)
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"ok": false,
|
"ok": false,
|
||||||
"error": "no price source (set GRIDSTATUS_API_KEY or ERCOT_API_KEY; override with RTM_OVERRIDE)"
|
"error": "no price source (set GRIDSTATUS_API_KEY or ERCOT_API_KEY; override with RTM_OVERRIDE)"
|
||||||
}
|
}
|
||||||
{"error":"no source yet (API key pending)"}
|
{"error":"price feed pending API key registration"}
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🌤️ Solar arbitrage verdict (latest run)
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"ts": "2026-09-06T20:28:59Z",
|
|
||||||
"ok": false,
|
|
||||||
"error": "no price source available (set ERCOT_API_KEY or GRIDSTATUS_API_KEY; see docs/oncor-realtime-arbitrage.md)"
|
|
||||||
}
|
|
||||||
{"error":"source unavailable"}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📋 Baseline index
|
|
||||||
|
|
||||||
| host | latest bundle |
|
|
||||||
|---|---|
|
|
||||||
| (none yet — first fleet baseline wave pending) | |
|
|
||||||
|
|||||||
@@ -0,0 +1,253 @@
|
|||||||
|
---
|
||||||
|
title: The Program
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🏗️ KNELPerf v2 — architecture for dynamic, intelligent performance optimization
|
||||||
|
|
||||||
|
> Status: DRAFT v1 (2026-09-06,
|
||||||
|
> [#826](https://projects.knownelement.com/issues/826)). Founder directive:
|
||||||
|
> end-to-end baseline → audit/benchmark → tweak → re-baseline loop,
|
||||||
|
> two-week program.
|
||||||
|
|
||||||
|
## 📑 Contents
|
||||||
|
|
||||||
|
1. [The loop](#1-the-loop) · 2. [Scheduling architecture](#2-scheduling-architecture-ultix-breakout-redesign) ·
|
||||||
|
3. [Proxmox resource groups & VM balancing](#3-proxmox-resource-groups--vm-balancing) ·
|
||||||
|
4. [k8s scheduling & Slurm](#4-where-k8s-scheduling-fits-slurm) ·
|
||||||
|
5. [Telemetry spine](#5-telemetry-spine-beszel--power) ·
|
||||||
|
6. [Workload priority classes](#6-workload-priority-classes-founder-set-order) ·
|
||||||
|
7. [Solar + Home Assistant](#7-solar--home-assistant-integration) ·
|
||||||
|
8. [Two-site power economics](#8-two-site-power-economics-site-2-printer-farm--laser--cnc) ·
|
||||||
|
9. [Hardware plan hooks](#9-hardware-plan-hooks) · 10. [Organization](#10-organization)
|
||||||
|
|
||||||
|
## 1. The loop
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
B[📋 BASELINE<br/>cpu-bench.tsv + bench-run.tsv<br/>+ PSI snapshot per host] --> A[🔍 AUDIT<br/>tolerance diff vs bundle<br/>beszel + PSI context]
|
||||||
|
A -->|drift found| T[🔧 TWEAK<br/>tweaks/<id>.sh — idempotent,<br/>GLPI CR-gated on prod]
|
||||||
|
T --> R[📈 RE-BASELINE<br/>capture + compare<br/>evidence to Redmine]
|
||||||
|
R --> A
|
||||||
|
A -->|clean| B
|
||||||
|
```
|
||||||
|
|
||||||
|
- **baseline**: `loop/perf-loop.sh baseline <host>` stores a bundle
|
||||||
|
(cpu-bench.tsv + bench-run.tsv + PSI snapshot) under `data/baselines/<host>/<ts>/`.
|
||||||
|
Storage/network legs reuse `bench/bench-run.sh` (fio + iperf3); the CPU leg
|
||||||
|
(`loop/cpu-bench.sh`) is new — the repo previously had zero CPU coverage.
|
||||||
|
- **audit**: `loop/perf-loop.sh audit <host>` re-runs the read-only probes and
|
||||||
|
diffs against the latest bundle via `loop/baseline-diff.sh` (tolerance-based,
|
||||||
|
catches governor flips exactly). This is what the scheduled loop runs.
|
||||||
|
- **tweak**: every mutation is a file under `tweaks/<id>.sh` (idempotent,
|
||||||
|
rollback helper included). Production tweaks ride a GLPI CR; the loop
|
||||||
|
refuses `tweak` on prod hosts without `.crush/active-cr`.
|
||||||
|
- **re-baseline**: same capture, then `compare` — the delta table is the
|
||||||
|
evidence artifact that lands on the Redmine ticket.
|
||||||
|
|
||||||
|
Cadence: nightly audit during the 22:00–07:00 batch window (self-measuring:
|
||||||
|
the audit itself is low-cost); re-baseline only after a tweak or hardware
|
||||||
|
change. First full-fleet baseline wave is this week's deliverable.
|
||||||
|
|
||||||
|
## 2. 🗓️ Scheduling architecture (Ultix breakout redesign)
|
||||||
|
|
||||||
|
See `scheduler/README.md` for the port map and the lessons-learned list. Core
|
||||||
|
redesign decisions:
|
||||||
|
|
||||||
|
1. **Single source of truth per host** — `/etc/knelperf/daynight.conf`; no
|
||||||
|
value ever lives in a timer file, unit default, or script constant.
|
||||||
|
2. **Reboot lands in day mode** — invariant carried from Ultix.
|
||||||
|
3. **Fleet-generic**: slice specs are data (`slice:weight:cpus:memhigh`), so
|
||||||
|
PVE hosts tune `qemu.slice` children and host services with the same engine
|
||||||
|
the workstation uses.
|
||||||
|
4. **Deployed by AWX (KNELIAC), not ad-hoc ssh** — scripts here are the
|
||||||
|
payload; job templates own the rollout waves.
|
||||||
|
|
||||||
|
## 3. 🖥️ Proxmox resource groups & VM balancing
|
||||||
|
|
||||||
|
Current fleet (dmidecode-verified 2026-09-06):
|
||||||
|
|
||||||
|
| node | platform | cores | RAM | storage character |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| pfv-tsys1 | OptiPlex 9020 (i7-4770) | 8 | 32G | 1x 1TB spinner — edge role |
|
||||||
|
| pfv-tsys3 | Precision 7510 (E3-1535M v5) | 8 | 32G | NVMe (PM961) — fast singleton |
|
||||||
|
| pfv-tsys4 | Precision T1700 (E3-1246 v3) | 8 | 16G | 6 spindles + 3 more USB toasters coming = spindle farm |
|
||||||
|
| pfv-tsys5 | Precision T7500 (E5620) | 8 | 96G | mixed; NFS server for -02 (NEVER casually rebooted) |
|
||||||
|
| pfv-tsys6 | PowerEdge R610 (2x E5530) | 16 | 128G | 1x 2TB spinner |
|
||||||
|
| pfv-tsys7 | PowerEdge R620 (2x E5-2630 v2) | 24 | 192G | 1x 2TB spinner |
|
||||||
|
| pfv-tsys8 (new) | Lenovo, NVMe, 12G | ? | 12G | fast NVMe, small — scheduler/quorum/edge |
|
||||||
|
| pfv-tsys2 (soon) | Precision 7510 clone + NVMe | 8 | 32G | NVMe |
|
||||||
|
|
||||||
|
Directions:
|
||||||
|
- **Proxmox resource groups / pools**: map pools to tenant classes (see §6),
|
||||||
|
so scheduler policy attaches to the pool, not per-VM.
|
||||||
|
- **Balancing across hosts AND spindles**: nodes are standalone (no
|
||||||
|
corosync/HA), so "balancing" = placement policy at create/migrate time, not
|
||||||
|
live DRRS. Initial heuristic: CPU-bound guests → tsys7 (24c) then tsys6;
|
||||||
|
RAM-hungry → tsys7/6/5; IO-latency-sensitive → NVMe nodes (tsys3, tsys8,
|
||||||
|
tsys2); sequential-throughput batch (HFNOC ETL) → tsys4 spindle farm with
|
||||||
|
one-job-per-spindle placement (fio baselines per spindle will quantify).
|
||||||
|
The audit loop's beszel/PSI data feeds a placement scorecard.
|
||||||
|
- **tsys6 vs tsys7 split**: RECOMMEND tsys7 = the big k8s worker (24c/192G —
|
||||||
|
largest), tsys6 = non-k8s VM farm (RackRental ContainerLabOLTP etc.).
|
||||||
|
Rationale: tsys7's Ivy Bridge has RAPL + iDRAC7 power telemetry (§5) and
|
||||||
|
the core count to soak k8s bins; tsys6's older CPUs are fine for
|
||||||
|
steady-state VMs but it lacks DCMI telemetry on old firmware. Decision
|
||||||
|
needs founder ruling — will be proposed on #826 with a capacity table once
|
||||||
|
tsys8/2 land.
|
||||||
|
|
||||||
|
## 4. ☸️ Where k8s scheduling fits; Slurm?
|
||||||
|
|
||||||
|
- **k8s (pfv-k8s, k3s)** owns long-running services + tenant classes
|
||||||
|
(ADR-0001 buckets). Resource requests/limits ARE the scheduling policy for
|
||||||
|
class 1–2 workloads; node labels/taints pin tenant classes to node pools.
|
||||||
|
- **HFNOC batch (class 3)** = interruptible, deadline-loose, large — this is
|
||||||
|
HPC-batch shaped. Options: (a) Kubernetes Jobs on spot-ish semantics
|
||||||
|
(preemption via priority classes — fits "start and stop easily"); (b) Slurm
|
||||||
|
on a dedicated partition across idle capacity. Recommendation: start with
|
||||||
|
k8s Jobs + PriorityClasses (no new control plane, flux-managed), and only
|
||||||
|
introduce Slurm if multi-node tightly-coupled MPI-style jobs appear. Slurm
|
||||||
|
co-located with k8s on shared nodes via cgroup slices is possible
|
||||||
|
(knelperf-batch.slice) but doubles scheduling systems for little gain at
|
||||||
|
this fleet size.
|
||||||
|
- The knelperf daynight engine gives the batch window (22:00–07:00) its
|
||||||
|
expanded resource envelope on each participating node.
|
||||||
|
|
||||||
|
## 5. 📡 Telemetry spine: beszel + power
|
||||||
|
|
||||||
|
- **Beszel has 100% VM+physical coverage** → it is the always-on signal for
|
||||||
|
the audit phase (CPU/mem/disk/net per host). Plan: beszel agent metrics
|
||||||
|
feed the placement scorecard; PSI textfile collector
|
||||||
|
(`scheduler/collectors/`) adds pressure (queueing) semantics beszel lacks.
|
||||||
|
- **Power telemetry** (see docs/report-amt-power-telemetry.md): per-node watts
|
||||||
|
come from RAPL (tsys1/3/4/7) + iDRAC (tsys6/7) + metered UPS legs; tsys5
|
||||||
|
needs a metered PDU. UPS re-cabling plan (switches+tsys6/7 on the
|
||||||
|
un-metered UPS, everything else on the metered one) is compatible: the
|
||||||
|
nodes losing wall-meter visibility are exactly the RAPL/iDRAC nodes.
|
||||||
|
- **Load-side telemetry = per-circuit monitoring (#623, Emporia Vue 3
|
||||||
|
pending founder pick)**: per-circuit watts via the HA LOCAL (HACS)
|
||||||
|
integration → knelperf textfile exporter → the class-3 batch gate gets a
|
||||||
|
real deferrable-load menu (server-room subpanel vs HVAC vs site-2
|
||||||
|
equipment). Dedicated CTs on the server-room subpanel + the tsys6/7 UPS
|
||||||
|
legs keep fleet draw separable from house load — needed exactly on EEA
|
||||||
|
days, so local (non-cloud) telemetry matters.
|
||||||
|
|
||||||
|
## 6. 🏆 Workload priority classes (founder-set order)
|
||||||
|
|
||||||
|
| class | tenants | profile | scheduling |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | RackRental, Side Door Group, Starting Line (paid reservations) | OLTP, latency-sensitive, high margin | guaranteed reservations (k8s Guaranteed QoS / PVE cpuunits high); KillBill integration gives reservation calendar → capacity pre-reservation; promotions advised when solar surplus + spare capacity projected (§7) |
|
||||||
|
| 2 | Rogue Technologies (staff engagements), Suborbital Systems (R&D cost center) | bursty, business hours | Burstable QoS; preemptable by class 1 |
|
||||||
|
| 3 | HFNOC batch | GIS ETL, weather, time-series OLAP, ERP MRP/capacity-planning | night window (22:00–07:00) or solar-surplus hours; checkpointable, preemptable by 1–2 |
|
||||||
|
|
||||||
|
## 7. ☀️ Solar + Home Assistant integration
|
||||||
|
|
||||||
|
- SITER-Solar (4 kW baseline design, PVWatts) → once the plant is live, Home
|
||||||
|
Assistant exposes real-time production. The loop consumes: surplus watts =
|
||||||
|
green light to start class-3 batches (excess production is the cheapest
|
||||||
|
compute on earth); forecast + reservation calendar (KillBill) → promotion
|
||||||
|
timing advice ("run the Side Door promo the week of X: projected surplus +
|
||||||
|
idle capacity Y").
|
||||||
|
- Integration path: HA REST/websocket (creds pattern exists:
|
||||||
|
`~/.creds/homeassistant.env`, tools/ha-ws-call.py in KNELBMS) → a small
|
||||||
|
exporter writing `knelperf_solar_surplus_watts` into the textfile
|
||||||
|
collector's directory; the batch launcher gates on it.
|
||||||
|
- SITER-Solar math review (2026-09-06): found constant drift
|
||||||
|
([redacted — LLC-private]/23952 vs plan's [redacted — LLC-private]/22614), a stale Dockerfile entrypoint
|
||||||
|
(deprecated python script), header/ROI inconsistencies, and fixed-cost
|
||||||
|
scenario assumptions — fixes landing in that repo.
|
||||||
|
|
||||||
|
## 8. 🏭 Two-site power economics (site 2: printer farm / laser / CNC)
|
||||||
|
|
||||||
|
The fleet has a second site with 3D-printer farm, laser cutter, and CNC — on
|
||||||
|
MUCH more expensive grid power than the main site. That asymmetry drives
|
||||||
|
scheduling policy as much as raw performance:
|
||||||
|
|
||||||
|
- **Marginal-price model**: every site has an effective $/kWh curve. Main
|
||||||
|
site = cheap grid + (soon) solar with export at $[redacted — LLC-private]and self-use at
|
||||||
|
$[redacted — LLC-private] (SITER-Solar contract). Site 2 = expensive grid. The scheduler's
|
||||||
|
objective becomes: run flexible load (compute AND manufacturing) where the
|
||||||
|
marginal $/kWh is lowest at that hour.
|
||||||
|
- **Solar surplus is main-site-first**: excess production self-consumed at
|
||||||
|
the main site displaces the CHEAPEST power — class-3 batch compute and any
|
||||||
|
deferrable main-site load should chase the surplus curve.
|
||||||
|
- **Site-2 manufacturing scheduling**: printer/laser/CNC jobs are batchy and
|
||||||
|
often deferrable (prints especially — hours-long, unattended). Candidate
|
||||||
|
policy: schedule site-2 energy-heavy jobs into site-2 off-peak windows
|
||||||
|
(if its tariff has TOU) or batch them for periods when main-site solar
|
||||||
|
surplus can't absorb more compute. Home Assistant + metered UPS legs give
|
||||||
|
the measured baseline; site 2 needs its own metering (smart plug / CT per
|
||||||
|
machine) before scheduling — queued as a data dependency.
|
||||||
|
- **Never shift OLTP**: class-1 paid workloads stay on their reserved
|
||||||
|
capacity regardless of power price — revenue and latency beat energy cost.
|
||||||
|
- KillBill reservation calendar × solar forecast × site-2 tariff = the
|
||||||
|
promotion/job-timing advisor output: "cheapest week to run X".
|
||||||
|
|
||||||
|
## 9. 🔩 Hardware plan hooks
|
||||||
|
|
||||||
|
- Purchases (RAM/CPU) deferred until after 2026-10-11; until then the wins
|
||||||
|
are placement + spindles (tsys4's 3 USB toasters join the spindle farm:
|
||||||
|
baseline each with fio, then one-batch-job-per-spindle).
|
||||||
|
- tsys8 (Lenovo NVMe 12G) → scheduler/edge roles + fast NVMe pool; inventory
|
||||||
|
row pending (PFVSystemInventory.csv not found in repo — founder to drop in).
|
||||||
|
- tsys2 (Precision 7510 clone + NVMe) → second fast singleton; NVMe pair
|
||||||
|
with tsys3 for latency-sensitive class-1 storage.
|
||||||
|
|
||||||
|
## 10. 👥 Organization
|
||||||
|
|
||||||
|
Performance management org reports to VPTechOps, dotted line VPFacilities —
|
||||||
|
identities per the TSGCOO pattern (Cloudron + Bitwarden + Redmine/Gitea/GLPI
|
||||||
|
groups). All work filed under #826; CRs for any prod change.
|
||||||
|
|
||||||
|
> **Docs live on Discourse — this repo is the executable source of truth.** Perf topic: https://community.turnsys.com/t/328
|
||||||
|
|
||||||
|
## 11. 📊 Capacity planning & margin tracking in Dolibarr
|
||||||
|
|
||||||
|
ERP landscape: 19 Dolibarr instances exist (one per brand);
|
||||||
|
**erp.knownelement.com** is the mothership (v24.0.0, REST API live, perf-org
|
||||||
|
API identity `vpperf` — creds `~/.creds/dolibarr-perf.env`). KillBill stays
|
||||||
|
the subscription billing engine; Dolibarr is the financial brain where
|
||||||
|
capacity, cost, and margin meet.
|
||||||
|
|
||||||
|
### Capacity ledger (supply)
|
||||||
|
|
||||||
|
- Fleet capacity is derived from ground truth we already collect: GLPI CMDB
|
||||||
|
+ dmidecode survey + baseline bundles → per-node vCPU/RAM/NVMe/spindle
|
||||||
|
inventory (ARCHITECTURE §3 table is the seed).
|
||||||
|
- Represented in Dolibarr as **virtual stock**: one product per capacity SKU
|
||||||
|
per node class (e.g. `cap-oltp-4c8g-nvme`, `cap-batch-spindle`,
|
||||||
|
`cap-k8s-worker-core`). Stock quantity = bookable units; GLPI drift report
|
||||||
|
reconciles it (the nightly reconcile already exists for CIs — extend to
|
||||||
|
capacity products).
|
||||||
|
|
||||||
|
### Reservations (demand)
|
||||||
|
|
||||||
|
- Paid reservations (class 1) = Dolibarr customer contracts/order lines that
|
||||||
|
consume capacity stock; KillBill invoices the money, Dolibarr holds the
|
||||||
|
commitment. Utilization % = booked/available per SKU per week.
|
||||||
|
- The promotion advisor consumes exactly this: KillBill reservation calendar
|
||||||
|
x projected solar surplus x utilization headroom (§7).
|
||||||
|
|
||||||
|
### Margin tracking (profitability)
|
||||||
|
|
||||||
|
- **Revenue** per tenant: KillBill (exports/journal) → Dolibarr customer
|
||||||
|
invoices/analytic accounts.
|
||||||
|
- **Direct costs** per tenant class: power (both sites — utility bills #827;
|
||||||
|
solar self-consumption at contract rate), hardware amortization (post
|
||||||
|
2026-10-11 purchases), WAN/tailnet overheads.
|
||||||
|
- Dolibarr **analytic accounting** (accounts + analytic distribution tables
|
||||||
|
are already present) allocates costs to tenant classes → margin per class
|
||||||
|
per month. PRIVATE dimension: margin numbers are revenue/expense detail —
|
||||||
|
LLC-members-only, never published to the public hub (transparency policy).
|
||||||
|
|
||||||
|
### Integration plumbing
|
||||||
|
|
||||||
|
- `loop/` baselines + beszel utilization → utilization updater script →
|
||||||
|
Dolibarr API (product stock / extrafields) — read/write via vpperf key.
|
||||||
|
- Hub publishes ONLY utilization % and capacity counts (public-safe);
|
||||||
|
margins stay inside the ERP.
|
||||||
|
|
||||||
|
API gotchas learned (2026-09-06): table prefix is `dolibarr_` (not `llx_`);
|
||||||
|
API auth needs `DOLAPIKEY` header; a cloned user needs explicit
|
||||||
|
`dolibarr_user_rights` rows copied (admin flag alone does not grant API
|
||||||
|
module rights).
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
title: Status
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# 📈 Status dashboard
|
||||||
|
|
||||||
|
Auto-published 2026-09-06T15:52:58-05:00 from KNELPerf. Source:
|
||||||
|
https://git.knownelement.com/KNEL/KNELPerf
|
||||||
|
|
||||||
|
## 🌤️ Solar arbitrage verdict (latest run)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"ts": "2026-09-06T20:52:58Z",
|
||||||
|
"ok": false,
|
||||||
|
"error": "no price source available (set ERCOT_API_KEY or GRIDSTATUS_API_KEY; see docs/oncor-realtime-arbitrage.md)"
|
||||||
|
}
|
||||||
|
{"error":"source unavailable"}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📋 Baseline index
|
||||||
|
|
||||||
|
| host | latest bundle |
|
||||||
|
|---|---|
|
||||||
|
| (none yet — first fleet baseline wave pending) | |
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
---
|
||||||
|
title: Transparency
|
||||||
|
visibility: private
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🔓 Transparency policy
|
||||||
|
|
||||||
|
We run an ultra-open and transparent organization. This site and its
|
||||||
|
[content repo](https://git.knownelement.com/KNEL/KNELPerfHub) are **public
|
||||||
|
read-only** — performance data, benchmarks, risk intelligence,
|
||||||
|
architecture and decision reports are all in the open.
|
||||||
|
|
||||||
|
**The one carve-out**: revenue and expense detail is private to LLC
|
||||||
|
members and is never published here (system costs, billing history,
|
||||||
|
contract rates, per-tenant margins). This line is enforced mechanically:
|
||||||
|
the publishing script redacts financial literals from every public page
|
||||||
|
at render time — see
|
||||||
|
[`grav-publish.sh`](https://git.knownelement.com/KNEL/KNELPerf/src/branch/main/tools/grav-publish.sh).
|
||||||
|
|
||||||
|
Publishing is exclusively git-based: content lands via pull from the
|
||||||
|
public repo; the site runs no foreign tooling. The Grav admin exists only
|
||||||
|
to prevent first-run setup mode; content never flows through it.
|
||||||
Reference in New Issue
Block a user