docs(audit): RAM/CPU placement audit + codify capacity goals

Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit logs, and fixed an empty-array abort in deploy-check.sh. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-07-29 05:32:12 -05:00
parent d064754c3e
commit bfb2d44183
20 changed files with 17317 additions and 19981 deletions
@@ -3,7 +3,12 @@
 > **Human read-only. Agents maintain this file automatically after each work
 > session.** Do not edit by hand — the next agent run will overwrite it.

-> **Last updated:** 2026-07-28 by Crush (GLM-5.2)
+> **Last updated:** 2026-07-29 by Crush (GLM-5.2)
+>
+> **Capacity targets are now codified** in
+> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
+> steady-state, ~50% idle CPU. The current placement analysis lives in
+> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md).

 ## Current State: STABLE

@@ -30,9 +35,12 @@ infrastructure, Proxmox cluster ops, and k8s control plane.

 ### Proxmox Cluster Ops
 - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit complete (PROJECT.md has ground truth)
+- [x] Fleet audit refreshed 2026-07-29 (live data, all 7 hosts)
+- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 72%
+- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
+- [ ] CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware)
 - [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
+- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
 - [ ] tsys2: pending rebuild from Win10 to Proxmox

 ### Kubernetes Control Plane (k3s HA — LIVE)
@@ -78,22 +86,32 @@ infrastructure, Proxmox cluster ops, and k8s control plane.

 | Issue | Impact | Status |
 |-------|--------|--------|
-| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode) |
-| **tsys1/5/9 RAM-overcommitted** | 118%/135%/131% respectively | Friday: migrate NFS-backed VMs to tsys6/tsys7 (77+132 GB free) |
+| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
+| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 72%/37%/60% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §8 |
+| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
+| **cnode4/5 stale VMs running** | Not joined to the 3-node cluster; waste 8 GB + 8 vCPU | Stop + delete (AUDIT-2026-07-29 §10) |
+| **Fleet CPU 0-6% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
 | Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
 | tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |

 ## Pending (next session priorities)

-1. **Wnode storage rebalancing** (2 PDM storage migrations):
-   - wnode-tsys6 (100): D5 → S4 (tsys4 → tsys5)
-   - wnode-tsys7 (701): D5 → S1 (tsys4 → tsys5)
-   - Gives every wnode its own physical disk, balances 3/3 across servers
-2. **Friday maintenance:** physical hardware + VM migrations (see
-   [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5)
-3. **k8s workers:** Join wnodes to the k3s cluster (after rebalancing)
-4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
-5. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
+> **RAM/CPU placement actions** are itemized in
+> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §12.
+> Storage/disk actions are deferred to post-Friday (see below).
+
+1. **VM migrations (RAM/CPU relief, zero hardware):**
+   - tsys1 → tsys7: librenms(104), pdm(105), workbench(117), tailscale-router(109), kali-tsys(114)
+   - tsys9 → tsys6: tsys-siem(901)
+2. **Grow wnode-tsys9** to 6 vCPU / 16 GB (after siem move) — cross-compile node
+3. **Stop + delete stale cnode4 (601) / cnode5 (706)** — not in the 3-node cluster
+4. **Join wnodes to the k3s cluster** (after migrations free resources)
+5. **Friday maintenance:** physical hardware + storage work (see
+   [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5
+   and [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
+6. **Wnode storage rebalancing** (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1
+7. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
+8. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
   solar-aware scale-out

 > **See [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md)
@@ -3,7 +3,7 @@
 > **Index of all documentation in this repo.** Agents must update this file
 > whenever a doc is added, removed, or substantively changed.

-> **Last updated:** 2026-07-28
+> **Last updated:** 2026-07-29

 ## Kubernetes Architecture ([`k8s/`](k8s/))

@@ -23,7 +23,9 @@ Fleet operations, hardware, performance tuning, storage architecture.
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
-| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Fresh fleet audit with current VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
+| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
+| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU placement audit vs the capacity goals; live data from all 7 hosts | 2026-07-29 |
+| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
 | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
 | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |

@@ -148,7 +148,11 @@ while [ "${#remaining[@]}" -gt 0 ]; do
            new_remaining+=("$host")
        fi
    done
-    remaining=("${new_remaining[@]:-}")
+    if [ "${#new_remaining[@]}" -gt 0 ]; then
+        remaining=("${new_remaining[@]}")
+    else
+        remaining=()
+    fi
    if [ "${#remaining[@]}" -gt 0 ]; then
        log "still running: ${remaining[*]} (${#remaining[@]} hosts)"
    fi
@@ -1,5 +1,10 @@
 # Fresh Fleet Audit — 2026-07-28 (revised)

+> **A newer audit exists:** [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (RAM/CPU
+> placement focus, with codified capacity goals). Placement numbers below are
+> superseded where the two conflict; this doc remains authoritative for the
+> **Friday storage/hardware plan (§5)** and storage-redundancy analysis.
+
 > **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**

 **Audit time:** 2026-07-28 20:30 CDT
@@ -0,0 +1,388 @@
+# Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
+
+> **Scope:** End-to-end analysis of **memory and CPU placement** against the
+> operator's capacity goals. Storage/disk optimization is intentionally
+> **out of scope** for this pass and is deferred to post-Friday work (NVMe on
+> tsys5 + k8s StorageClass design). See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)
+> for the standing targets.
+
+**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
+**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
+capacity probe (`/proc/meminfo`, `qm config`, 5s CPU sample, `kubectl top`)
+collected via [`tests/remote.sh`](../../tests/remote.sh).
+**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
+**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days)
+**Supersedes:** placement numbers in [`PROJECT.md`](PROJECT.md) and
+[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) where they conflict.
+
+---
+
+## 1. Executive summary — fleet vs the two goals
+
+| Goal | Status | Detail |
+|------|--------|--------|
+| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 72%. Fleet is well within budget. |
+| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-6% busy**. Fleet is heavily CPU-underutilized. |
+
+**Headline:** The cluster has **abundant RAM and CPU headroom everywhere**. The
+memory goal is met with margin to spare; the CPU-utilization goal is not
+remotely met because there simply isn't enough sustained workload yet. The right
+response is *not* to add hardware — it is to (a) consolidate the always-on
+baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and
+ready for RackRental/compute bursts, and (c) let utilization rise naturally as
+RackRental + ETL demand comes online.
+
+| Finding | Severity | § |
+|---------|----------|---|
+| **tsys1/5/9 are over-allocated (>115%)** though actual usage is fine | Medium | 4 |
+| **tsys6/7 idle at 6%/9% with 117/171 GB free** — perfect, underused RackRental capacity | Info | 6 |
+| **tsys5 has only ONE CPU** (CPU2 socket unpopulated) — prior "dual E5620" record was wrong | Medium | 9 |
+| **cnode4 + cnode5 are stale running VMs not in the k3s cluster** (waste 8 GB + 8 vCPU) | Low | 10 |
+| **Fleet CPU ~0-6% idle** — no path to 50% without more workloads | Info | 5 |
+| **DDR3 sticks: compatibility-blocked from the high-RAM hosts** (they need RDIMM) | Medium | 7 |
+
+---
+
+## 2. Host capacity (live, 2026-07-29)
+
+CPU "threads" = logical processors (`nproc`). "vCPU" = sum of running-VM cores
+× sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling).
+"Actual%" = `(MemTotal - MemAvailable) / MemTotal` (steady-state truth).
+
+| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | vCPU | CPU busy | Load |
+|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|------|----------|------|
+| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **72%** | 0 | 26 | 3% | 1.3 |
+| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 8 | 0% | 0.2 |
+| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **13%** | 832 MB | 2 | 0% | 3.0 |
+| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 72 | 6% | 5.1 |
+| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 3 | 48 GB | 38% | **6%** | 0 | 12 | 0% | 0.2 |
+| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 6 | 56 GB | 30% | **9%** | 0 | 22 | 1% | 1.2 |
+| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **60%** | 0 | 16 | 1% | 0.2 |
+
+**Notes on the table:**
+
+- **tsys4 load 3.0 with 0% CPU busy** = NFS/disk I/O wait (it is the storage
+  server). The 832 MB of swap used is the only swap activity in the fleet —
+  minor, but worth watching; it predates this audit.
+- **tsys5 "single CPU":** dmidecode reports `CPU2: Status Unpopulated`. The T7500
+  is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads).
+  See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and
+  useless until a 2nd CPU is seated.
+- **tsys5 alloc 134%** looks alarming but actual usage is only **37%** — the 17
+  VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual
+  low). Still, it is the most over-committed host by allocation.
+
+---
+
+## 3. Running VM inventory (RAM/CPU only)
+
+### tsys1 — Infrastructure (11 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB |
+| 101 | tsys-ca | 2 | 2 GB |
+| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB |
+| 103 | pfv-netinfra-01 | 2 | 2 GB |
+| 104 | tsys-librenms | 2 | 2 GB |
+| 105 | tsys-proxmox-datacenter | 2 | 2 GB |
+| 106 | pfv-k8s-cnode3 | 2 | 4 GB |
+| 108 | tsys-ucs-01 | 4 | 8 GB |
+| 109 | tailscale-router | 2 | 2 GB |
+| 114 | kali-tsys | 2 | 2 GB |
+| 117 | tsys-secure-workbench | 2 | 4 GB |
+
+### tsys3 — Compute (1 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB |
+
+### tsys4 — Storage (1 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 400 | pfv-proxmox-backup-server | 2 | 2 GB |
+
+### tsys5 — Storage + sandboxes (17 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB |
+| 5101 | sectestbed-siem | 4 | 10 GB |
+| 5111 | ultix-streaming | 4 | 9 GB |
+| 53100 | tsys-preprod-awx | 4 | 9 GB |
+| 53101 | tsys-preprod-siem | 4 | 12 GB |
+| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB |
+| 5112 | ultix-offstage | 4 | 6 GB |
+| 51010-51013,5105-5109,6000 | sectestbed suite (×10) | 4 ea | 4 GB ea |
+
+### tsys6 — RackRental-ready (3 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB |
+| 600 | tsys-awx | 4 | 12 GB |
+| 601 | pfv-k8s-cnode4 **(stale — not in cluster)** | 4 | 4 GB |
+
+### tsys7 — RackRental-ready (6 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB |
+| 702 | hfnoc-uisp | 4 | 8 GB |
+| 704 | TCTC | 4 | 6 GB |
+| 705 | pfv-k8s-cnode2 | 4 | 4 GB |
+| 706 | pfv-k8s-cnode5 **(stale — not in cluster)** | 4 | 4 GB |
+| 703 | rr-middleware | 2 | 2 GB |
+
+### tsys9 — Infra + control plane (6 running)
+
+| VMID | Name | vCPU | RAM |
+|------|------|------|-----|
+| 901 | tsys-siem | 2 | 8 GB |
+| 902 | tsys-ucs-02 | 4 | 8 GB |
+| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB |
+| 904 | pfv-netinfra-02 | 2 | 4 GB |
+| 906 | pfv-k8s-cnode1 | 2 | 4 GB |
+| 903 | kali-rd | 2 | 2 GB |
+
+### k3s control plane (ground truth via `kubectl`)
+
+Only **3 cnodes** are actually joined to the pfv-k8s cluster:
+
+| Node | Host | CPU | Mem | Role |
+|------|------|-----|-----|------|
+| cnode1 (906) | tsys9 | 2% | 1181 Mi (30%) | control-plane,etcd |
+| cnode2 (705) | tsys7 | 2% | 1206 Mi (30%) | control-plane,etcd |
+| cnode3 (106) | tsys1 | 2% | 1124 Mi (28%) | control-plane,etcd |
+
+cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are **running as VMs but not
+joined** — leftovers from the pre-rebuild 5-node cluster. See §10. **No wnodes
+are joined yet** (all workers run standalone, pending the join step).
+
+---
+
+## 4. Memory analysis vs the 80% goal
+
+**Verdict: every host passes actual usage.** The 80% steady-state rule is met
+fleet-wide with margin. The hosts closest to the line:
+
+| Host | Actual% | Alloc% | Risk | Action |
+|------|---------|--------|------|--------|
+| **tsys1** | **72%** | 115% | Closest to 80%; alloc > 100% | Shed 5 lightweight VMs → tsys7 (§8) |
+| **tsys9** | 60% | 129% | Alloc > 100% but actual comfortable | Move tsys-siem (8 GB) → tsys6 (§8) |
+| **tsys5** | 37% | 134% | Alloc highest, but mostly idle sandboxes | No action now; monitor if sandboxes wake |
+| tsys3 | 10% | 90% | None | — |
+| tsys6 | 6% | 38% | None (most empty) | Absorb migrated VMs |
+| tsys7 | 9% | 30% | None | Absorb migrated VMs |
+
+**Key insight:** actual usage (72% max) is far below allocation (134% max)
+because of KSM/balloon. The memory goal is about *actual*, and the fleet is
+healthy. The over-allocation on tsys1/5/9 is a *planning* concern, resolved by
+the migrations in §8 — **no RAM purchase required**.
+
+## 5. CPU analysis vs the 50%-idle goal
+
+**Verdict: the fleet misses the target badly — in the *safe* direction.** Every
+host is 0-6% busy at idle. This is not a capacity problem; it is an
+*under-utilization* problem.
+
+You cannot manufacture 50% utilization without workloads to run. The realistic
+strategy, in priority order:
+
+1. **Consolidate the always-on baseline onto the efficient hosts.** tsys9
+   (Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the
+   control plane + infra. Their baseline will rise toward the target as compute
+   workloads land.
+2. **Keep tsys6/7 warm for bursts, not baseline.** They are the RackRental and
+   parallel-batch hosts. Their idle utilization will climb to ~50% *naturally*
+   as RackRental customers spin up labs — that is the designed use, not waste.
+3. **Solar-gate the inefficient idle hosts when truly idle.** tsys5 (single
+   Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running
+   RackRental/batch, they are candidates for power-down / solar-aware scheduling
+   (aligns with the existing solar-aware scale-out concept). Running a Nehalem
+   box at 0% busy 24/7 is pure electricity cost.
+4. **Grow into it.** As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7)
+   demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend
+   to chase the number.
+
+---
+
+## 6. Workload placement model
+
+| Workload family | Needs | Best hosts | Why |
+|-----------------|-------|-----------|-----|
+| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + only local SSD/NVMe in fleet |
+| **RackRental.net (network labs)** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
+| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
+
+CPU quality ranking (newest → oldest IPC): **tsys9** (Comet Lake 2020) >
+**tsys3** (Skylake 2015) > **tsys7** (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell
+2013) > **tsys5/6** (Westmere/Nehalem 2009-2010).
+
+---
+
+## 7. Answers to the operator's questions
+
+### 7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"
+
+**tsys9 — YES.** It has the newest CPU in the fleet (i5-10500, 12 threads) and a
+local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural
+**cross-compile champion**. To grow it, first free RAM by moving tsys-siem
+(8 GB) to tsys6 (§8), then bump wnode-tsys9 to **6 vCPU / 16 GB**. Actual usage
+on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.
+
+**tsys1 — NO (premise correction).** The i7-4770 is **Haswell 2013** — the
+second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most
+RAM-loaded host (72% actual, 115% alloc). It should **shed** VMs, not grow its
+wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are
+tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.
+
+### 7.2 "Are tsys6/7 better for RackRental.net workloads?"
+
+**Yes — exactly right.** RackRental (renting ephemeral network labs) is
+RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls.
+tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM.
+Their older CPUs are irrelevant because labs sit idle between interactions.
+
+**They do NOT need more RAM.** They idle at 9%/6% actual with 171/117 GB free.
+They are over-provisioned in the right direction. Their real bottleneck is
+**disk** (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls),
+which is the deferred Friday + StorageClass work, not a RAM/CPU issue.
+
+### 7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)
+
+**First, identify the exact DIMM type.** Compatibility is decisive and the
+fleet's DDR3 hosts are **not interchangeable**:
+
+| Host | Platform | Accepts | Current | Empty slots | Notes |
+|------|----------|---------|---------|-------------|-------|
+| **tsys6** (R610) | PowerEdge server | **RDIMM ECC only** | 126 GB RDIMM 1333 | many (B-branch) | Consumer UDIMM → **no POST** |
+| **tsys7** (R620) | PowerEdge server | **RDIMM ECC only** | 189 GB RDIMM 1600 | many (A7/A8 + B) | Consumer UDIMM → **no POST** |
+| **tsys5** (T7500) | Workstation | **RDIMM ECC** | 94 GB RDIMM 1600 | 6 (CPU2 branch) | Useless without a 2nd CPU (§9) |
+| **tsys4** (T1700) | Xeon E3 workstation | **ECC UDIMM** | 16 GB (4×4) | 0 (must swap) | Max 32 GB (4×8). 16GB sticks unsupported |
+| **tsys1** (OptiPlex 9020) | Consumer | non-ECC UDIMM | 32 GB (4×8) | 0 | **Maxed.** Haswell caps at 8 GB/slot |
+
+**Decision tree (run this before buying/installing anything):**
+
+1. **Read the SPD** on each stick (labels or `decode-dimms` / `dmidecode -t
+   memory` once installed in *any* testable board). The binary question is:
+   **Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.**
+2. **If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):**
+   - They fit **nowhere useful.** tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is
+     maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable
+     there). tsys4 requires ECC and caps at 8 GB/slot.
+   - The *only* possible target is **tsys4** — and only the 8 GB sticks, and only
+     if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4
+     16→32 GB. Marginal.
+3. **If the sticks are RDIMM ECC (unlikely for Patriot, possible for some
+   Kingston server lines):**
+   - Deploy 16 GB RDIMM into **tsys7/tsys6** empty B-branch slots to balance
+     memory across both CPUs (currently CPU-B branches are partly empty —
+     single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
+   - But **value is low**: those hosts don't need more RAM (§7.2).
+
+**Bottom line on DDR3:** Do not spend effort/money chasing RAM into tsys6/7 —
+they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9
+over-allocation) is solved for free by **VM migration** (§8), not by sticks. If
+the sticks are free/cheap UDIMM, the realistic win is just **tsys4 16→32 GB**
+(if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7
+will stop them from POSTing.
+
+---
+
+## 8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)
+
+These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge
+free RAM. **NFS-backed VMs migrate cleanly via PDM** (storage stays on the same
+NFS export; only the compute host changes).
+
+**From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):**
+
+| VMID | Name | RAM freed on tsys1 |
+|------|------|--------------------|
+| 104 | tsys-librenms | 2 GB |
+| 105 | tsys-proxmox-datacenter | 2 GB |
+| 117 | tsys-secure-workbench | 4 GB |
+| 109 | tailscale-router | 2 GB |
+| 114 | kali-tsys | 2 GB |
+| | **Total** | **12 GB** |
+
+Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB
+trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.
+
+**From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):**
+
+| VMID | Name | RAM freed on tsys9 |
+|------|------|--------------------|
+| 901 | tsys-siem | 8 GB |
+
+Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).
+
+> **Note:** the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are
+> hardware-locked and only move with a physical dongle relocation — defer to the
+> onsite window, not part of this RAM/CPU pass.
+
+---
+
+## 9. Hardware corrections discovered
+
+### tsys5 is single-CPU, not dual
+
+dmidecode: `CPU2: Status Unpopulated`. The Precision T7500 has **one E5620**
+(4 cores / 8 threads), not two. `PROJECT.md` and prior audits said "2× E5620" —
+**that was wrong.** Implications:
+
+- CPU capacity is half what was documented. tsys5 is the weakest single-socket
+  CPU in the fleet (tied era with tsys6's Nehalem).
+- The 6 empty DIMM slots are on the **CPU2 branch** — they are useless until a
+  2nd CPU is seated. Adding RAM there today does nothing.
+- If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU
+  capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth.
+  This is a higher-value hardware move than adding RAM sticks.
+
+### tsys4 is using 832 MB of swap
+
+Only host in the fleet with active swap. Predates this audit and is minor, but a
+16 GB storage server running PBS should not be swapping. After the planned
+16→32/64 GB upgrade, monitor that swap returns to zero.
+
+---
+
+## 10. Cleanup items
+
+| Item | Impact | Fix |
+|------|--------|-----|
+| **cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster** | 8 GB + 8 vCPU wasted on stale pre-rebuild VMs | Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node. |
+| **No wnodes joined to k3s** | Workers run standalone; cluster has no schedulable capacity | Join wnodes (post-migration) — the §8 moves free the resources to size them well |
+| **rr-middleware (703) on tsys7** | RackRental middleware lives on a RackRental host — fine | Intentional; just confirming placement is consistent |
+
+---
+
+## 11. Deferred to post-Friday (storage — intentionally out of scope here)
+
+Per operator direction, disk/storage optimization happens after the Friday
+maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s
+StorageClass design. Tracked in [`TODO.md`](TODO.md) and
+[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) §5. The one RAM/CPU-adjacent note:
+**tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck**, not their CPU/RAM.
+
+---
+
+## 12. Summary of recommended actions (this pass)
+
+| # | Action | Cost | Effect |
+|---|--------|------|--------|
+| 1 | Migrate 5 lightweight VMs tsys1→tsys7 | Free (PDM) | tsys1 alloc 115%→77% |
+| 2 | Migrate tsys-siem tsys9→tsys6 | Free (PDM) | tsys9 alloc 129%→95%; room to grow wnode |
+| 3 | Stop + delete stale cnode4/cnode5 | Free | Reclaim 8 GB + 8 vCPU |
+| 4 | Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2) | Free | Cross-compile node ready |
+| 5 | **Do not** grow wnode-tsys1; keep it small | Free | Corrects the CPU-quality premise |
+| 6 | Identify DDR3 stick type before any install | Free | Avoids no-POST on tsys6/7 |
+| 7 | Consider seating a 2nd CPU in tsys5 (if spare LGA771) | Cheap | Doubles tsys5 CPU; activates RAM branch |
+| 8 | Solar-gate tsys5/6 when idle (no RackRental load) | Free | Stops wasting power on 0%-busy old cores |
+
+> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
+> hardware** — and by consolidating/solar-gating so idle silicon isn't burning
+> power for nothing.
@@ -0,0 +1,130 @@
+# Capacity & Sizing Goals
+
+**Date:** 2026-07-29
+**Status:** Authoritative. These are the operator's standing targets. Every
+fleet audit and VM-placement decision is evaluated against them.
+
+> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
+> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).
+
+---
+
+## 1. Memory target: 80% steady-state
+
+**Rule:** Keep each node at or below **80% physical RAM utilization at steady
+state.** Occasional bursts above 80% are acceptable (that is what swap is for);
+the *sustained* idle-to-average load must sit at ~80% or below.
+
+**Two metrics must both be tracked:**
+
+| Metric | Definition | Target |
+|--------|-----------|--------|
+| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
+| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |
+
+**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
+often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
+72% actual). Actual usage is the steady-state truth; allocation is the
+worst-case ceiling if every VM suddenly fills its reservation. A host is
+"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
+balloon-deflate storm could OOM it.
+
+**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
+(>~1 GB on a steady basis) means the host is over-committed and VMs should
+migrate off.
+
+## 2. CPU target: ~50% utilization at idle
+
+**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**
+
+This is a *utilization* goal, not a headroom goal. The fleet is
+**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
+silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
+The intent is to **pack workloads** so the baseline load consumes roughly half
+the compute, reserving the other half for bursts.
+
+**Practical implications:**
+
+- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
+  doing useful work at ~50% or be candidates to power down / solar-gate when
+  idle. Running them at 0-6% wastes power per core.
+- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
+  idle and are the natural home for always-on control-plane + infra.
+- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
+  utilization, not vCPU ratios.
+
+## 3. Workload placement model
+
+The fleet serves three distinct workload families. Placement should follow CPU
+quality + storage speed + RAM size, in that order of relevance per family.
+
+### 3.1 ETL / HPC / cross-compile (CPU + storage bound)
+
+Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
+**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
+(8-32 GB per job).
+
+| Host | CPU | Storage | Fit |
+|------|-----|---------|-----|
+| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
+| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
+| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |
+
+These three hosts have the newest CPUs and the only fast local storage. Their
+wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
+labeled `workload=compute`.
+
+### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)
+
+Business idea: rent out ephemeral network labs (containerlab topologies) on
+demand. Many containers / network namespaces, short-lived, mostly idle once up.
+**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
+(labs are idle between keystrokes).
+
+| Host | CPU | RAM | Fit |
+|------|-----|-----|-----|
+| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
+| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |
+
+**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
+irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
+labeled `workload=rackrental`.
+
+**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
+is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
+problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
+ready today.
+
+### 3.3 Infrastructure + k8s control plane (always-on, low resource)
+
+netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
+must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
+etcd diversity).
+
+---
+
+## 4. Host role summary (target)
+
+| Host | CPU era | RAM | Primary role | Wnode class |
+|------|---------|-----|--------------|-------------|
+| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
+| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
+| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
+| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
+| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
+| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
+| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
+| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |
+
+## 5. DDR3 memory policy
+
+**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
+117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
+not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
+over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
+free RAM, not by installing sticks.
+
+Compatibility is decisive and is documented per-host in
+[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
+**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
+(Patriot / most Kingston) fits essentially nowhere useful here.
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys1.log 951792 16094
+ok pfv-tsys1.log 991975 16254
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys3.log 750211 11660
+ok pfv-tsys3.log 756980 10665
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys4.log 617476 11115
+ok pfv-tsys4.log 632523 11207
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys5.log 1064227 18745
+ok pfv-tsys5.log 1064870 17635
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys6.log 907424 16997
+ok pfv-tsys6.log 923060 16577
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys7.log 1114764 18714
+ok pfv-tsys7.log 1120464 18358
@@ -1,2 +1,2 @@
 running
-ok pfv-tsys9.log 793781 14230
+ok pfv-tsys9.log 876496 13648