KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): RAM/CPU placement audit + codify capacity goals

Browse Source 
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 05:32:12 -05:00
parent 428e9ecb1d
commit 575d79f611
20 changed files with 17317 additions and 19981 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-07-29.md
+388
View File
@@ -0,0 +1,388 @@
# Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
> **Scope:** End-to-end analysis of **memory and CPU placement** against the
> operator's capacity goals. Storage/disk optimization is intentionally
> **out of scope** for this pass and is deferred to post-Friday work (NVMe on
> tsys5 + k8s StorageClass design). See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)
> for the standing targets.
**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
capacity probe (`/proc/meminfo`, `qm config`, 5s CPU sample, `kubectl top`)
collected via [`tests/remote.sh`](../../tests/remote.sh).
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days)
**Supersedes:** placement numbers in [`PROJECT.md`](PROJECT.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) where they conflict.
---
## 1. Executive summary — fleet vs the two goals
| Goal | Status | Detail |
|------|--------|--------|
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 72%. Fleet is well within budget. |
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-6% busy**. Fleet is heavily CPU-underutilized. |
**Headline:** The cluster has **abundant RAM and CPU headroom everywhere**. The
memory goal is met with margin to spare; the CPU-utilization goal is not
remotely met because there simply isn't enough sustained workload yet. The right
response is *not* to add hardware — it is to (a) consolidate the always-on
baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and
ready for RackRental/compute bursts, and (c) let utilization rise naturally as
RackRental + ETL demand comes online.
| Finding | Severity | § |
|---------|----------|---|
| **tsys1/5/9 are over-allocated (>115%)** though actual usage is fine | Medium | 4 |
| **tsys6/7 idle at 6%/9% with 117/171 GB free** — perfect, underused RackRental capacity | Info | 6 |
| **tsys5 has only ONE CPU** (CPU2 socket unpopulated) — prior "dual E5620" record was wrong | Medium | 9 |
| **cnode4 + cnode5 are stale running VMs not in the k3s cluster** (waste 8 GB + 8 vCPU) | Low | 10 |
| **Fleet CPU ~0-6% idle** — no path to 50% without more workloads | Info | 5 |
| **DDR3 sticks: compatibility-blocked from the high-RAM hosts** (they need RDIMM) | Medium | 7 |
---
## 2. Host capacity (live, 2026-07-29)
CPU "threads" = logical processors (`nproc`). "vCPU" = sum of running-VM cores
× sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling).
"Actual%" = `(MemTotal - MemAvailable) / MemTotal` (steady-state truth).
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | vCPU | CPU busy | Load |
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|------|----------|------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **72%** | 0 | 26 | 3% | 1.3 |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 8 | 0% | 0.2 |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **13%** | 832 MB | 2 | 0% | 3.0 |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 72 | 6% | 5.1 |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 3 | 48 GB | 38% | **6%** | 0 | 12 | 0% | 0.2 |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 6 | 56 GB | 30% | **9%** | 0 | 22 | 1% | 1.2 |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **60%** | 0 | 16 | 1% | 0.2 |
**Notes on the table:**
- **tsys4 load 3.0 with 0% CPU busy** = NFS/disk I/O wait (it is the storage
server). The 832 MB of swap used is the only swap activity in the fleet —
minor, but worth watching; it predates this audit.
- **tsys5 "single CPU":** dmidecode reports `CPU2: Status Unpopulated`. The T7500
is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads).
See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and
useless until a 2nd CPU is seated.
- **tsys5 alloc 134%** looks alarming but actual usage is only **37%** — the 17
VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual
low). Still, it is the most over-committed host by allocation.
---
## 3. Running VM inventory (RAM/CPU only)
### tsys1 — Infrastructure (11 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB |
| 101 | tsys-ca | 2 | 2 GB |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB |
| 103 | pfv-netinfra-01 | 2 | 2 GB |
| 104 | tsys-librenms | 2 | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB |
| 108 | tsys-ucs-01 | 4 | 8 GB |
| 109 | tailscale-router | 2 | 2 GB |
| 114 | kali-tsys | 2 | 2 GB |
| 117 | tsys-secure-workbench | 2 | 4 GB |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB |
| 5101 | sectestbed-siem | 4 | 10 GB |
| 5111 | ultix-streaming | 4 | 9 GB |
| 53100 | tsys-preprod-awx | 4 | 9 GB |
| 53101 | tsys-preprod-siem | 4 | 12 GB |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB |
| 5112 | ultix-offstage | 4 | 6 GB |
| 51010-51013,5105-5109,6000 | sectestbed suite (×10) | 4 ea | 4 GB ea |
### tsys6 — RackRental-ready (3 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB |
| 600 | tsys-awx | 4 | 12 GB |
| 601 | pfv-k8s-cnode4 **(stale — not in cluster)** | 4 | 4 GB |
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB |
| 702 | hfnoc-uisp | 4 | 8 GB |
| 704 | TCTC | 4 | 6 GB |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB |
| 706 | pfv-k8s-cnode5 **(stale — not in cluster)** | 4 | 4 GB |
| 703 | rr-middleware | 2 | 2 GB |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 901 | tsys-siem | 2 | 8 GB |
| 902 | tsys-ucs-02 | 4 | 8 GB |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB |
| 904 | pfv-netinfra-02 | 2 | 4 GB |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB |
| 903 | kali-rd | 2 | 2 GB |
### k3s control plane (ground truth via `kubectl`)
Only **3 cnodes** are actually joined to the pfv-k8s cluster:
| Node | Host | CPU | Mem | Role |
|------|------|-----|-----|------|
| cnode1 (906) | tsys9 | 2% | 1181 Mi (30%) | control-plane,etcd |
| cnode2 (705) | tsys7 | 2% | 1206 Mi (30%) | control-plane,etcd |
| cnode3 (106) | tsys1 | 2% | 1124 Mi (28%) | control-plane,etcd |
cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are **running as VMs but not
joined** — leftovers from the pre-rebuild 5-node cluster. See §10. **No wnodes
are joined yet** (all workers run standalone, pending the join step).
---
## 4. Memory analysis vs the 80% goal
**Verdict: every host passes actual usage.** The 80% steady-state rule is met
fleet-wide with margin. The hosts closest to the line:
| Host | Actual% | Alloc% | Risk | Action |
|------|---------|--------|------|--------|
| **tsys1** | **72%** | 115% | Closest to 80%; alloc > 100% | Shed 5 lightweight VMs → tsys7 (§8) |
| **tsys9** | 60% | 129% | Alloc > 100% but actual comfortable | Move tsys-siem (8 GB) → tsys6 (§8) |
| **tsys5** | 37% | 134% | Alloc highest, but mostly idle sandboxes | No action now; monitor if sandboxes wake |
| tsys3 | 10% | 90% | None | — |
| tsys6 | 6% | 38% | None (most empty) | Absorb migrated VMs |
| tsys7 | 9% | 30% | None | Absorb migrated VMs |
**Key insight:** actual usage (72% max) is far below allocation (134% max)
because of KSM/balloon. The memory goal is about *actual*, and the fleet is
healthy. The over-allocation on tsys1/5/9 is a *planning* concern, resolved by
the migrations in §8 — **no RAM purchase required**.
## 5. CPU analysis vs the 50%-idle goal
**Verdict: the fleet misses the target badly — in the *safe* direction.** Every
host is 0-6% busy at idle. This is not a capacity problem; it is an
*under-utilization* problem.
You cannot manufacture 50% utilization without workloads to run. The realistic
strategy, in priority order:
1. **Consolidate the always-on baseline onto the efficient hosts.** tsys9
(Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the
control plane + infra. Their baseline will rise toward the target as compute
workloads land.
2. **Keep tsys6/7 warm for bursts, not baseline.** They are the RackRental and
parallel-batch hosts. Their idle utilization will climb to ~50% *naturally*
as RackRental customers spin up labs — that is the designed use, not waste.
3. **Solar-gate the inefficient idle hosts when truly idle.** tsys5 (single
Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running
RackRental/batch, they are candidates for power-down / solar-aware scheduling
(aligns with the existing solar-aware scale-out concept). Running a Nehalem
box at 0% busy 24/7 is pure electricity cost.
4. **Grow into it.** As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7)
demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend
to chase the number.
---
## 6. Workload placement model
| Workload family | Needs | Best hosts | Why |
|-----------------|-------|-----------|-----|
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + only local SSD/NVMe in fleet |
| **RackRental.net (network labs)** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
CPU quality ranking (newest → oldest IPC): **tsys9** (Comet Lake 2020) >
**tsys3** (Skylake 2015) > **tsys7** (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell
2013) > **tsys5/6** (Westmere/Nehalem 2009-2010).
---
## 7. Answers to the operator's questions
### 7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"
**tsys9 — YES.** It has the newest CPU in the fleet (i5-10500, 12 threads) and a
local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural
**cross-compile champion**. To grow it, first free RAM by moving tsys-siem
(8 GB) to tsys6 (§8), then bump wnode-tsys9 to **6 vCPU / 16 GB**. Actual usage
on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.
**tsys1 — NO (premise correction).** The i7-4770 is **Haswell 2013** — the
second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most
RAM-loaded host (72% actual, 115% alloc). It should **shed** VMs, not grow its
wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are
tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.
### 7.2 "Are tsys6/7 better for RackRental.net workloads?"
**Yes — exactly right.** RackRental (renting ephemeral network labs) is
RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls.
tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM.
Their older CPUs are irrelevant because labs sit idle between interactions.
**They do NOT need more RAM.** They idle at 9%/6% actual with 171/117 GB free.
They are over-provisioned in the right direction. Their real bottleneck is
**disk** (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls),
which is the deferred Friday + StorageClass work, not a RAM/CPU issue.
### 7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)
**First, identify the exact DIMM type.** Compatibility is decisive and the
fleet's DDR3 hosts are **not interchangeable**:
| Host | Platform | Accepts | Current | Empty slots | Notes |
|------|----------|---------|---------|-------------|-------|
| **tsys6** (R610) | PowerEdge server | **RDIMM ECC only** | 126 GB RDIMM 1333 | many (B-branch) | Consumer UDIMM → **no POST** |
| **tsys7** (R620) | PowerEdge server | **RDIMM ECC only** | 189 GB RDIMM 1600 | many (A7/A8 + B) | Consumer UDIMM → **no POST** |
| **tsys5** (T7500) | Workstation | **RDIMM ECC** | 94 GB RDIMM 1600 | 6 (CPU2 branch) | Useless without a 2nd CPU (§9) |
| **tsys4** (T1700) | Xeon E3 workstation | **ECC UDIMM** | 16 GB (4×4) | 0 (must swap) | Max 32 GB (4×8). 16GB sticks unsupported |
| **tsys1** (OptiPlex 9020) | Consumer | non-ECC UDIMM | 32 GB (4×8) | 0 | **Maxed.** Haswell caps at 8 GB/slot |
**Decision tree (run this before buying/installing anything):**
1. **Read the SPD** on each stick (labels or `decode-dimms` / `dmidecode -t
memory` once installed in *any* testable board). The binary question is:
**Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.**
2. **If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):**
- They fit **nowhere useful.** tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is
maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable
there). tsys4 requires ECC and caps at 8 GB/slot.
- The *only* possible target is **tsys4** — and only the 8 GB sticks, and only
if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4
16→32 GB. Marginal.
3. **If the sticks are RDIMM ECC (unlikely for Patriot, possible for some
Kingston server lines):**
- Deploy 16 GB RDIMM into **tsys7/tsys6** empty B-branch slots to balance
memory across both CPUs (currently CPU-B branches are partly empty —
single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
- But **value is low**: those hosts don't need more RAM (§7.2).
**Bottom line on DDR3:** Do not spend effort/money chasing RAM into tsys6/7 —
they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9
over-allocation) is solved for free by **VM migration** (§8), not by sticks. If
the sticks are free/cheap UDIMM, the realistic win is just **tsys4 16→32 GB**
(if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7
will stop them from POSTing.
---
## 8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)
These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge
free RAM. **NFS-backed VMs migrate cleanly via PDM** (storage stays on the same
NFS export; only the compute host changes).
**From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):**
| VMID | Name | RAM freed on tsys1 |
|------|------|--------------------|
| 104 | tsys-librenms | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 GB |
| 117 | tsys-secure-workbench | 4 GB |
| 109 | tailscale-router | 2 GB |
| 114 | kali-tsys | 2 GB |
| | **Total** | **12 GB** |
Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB
trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.
**From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):**
| VMID | Name | RAM freed on tsys9 |
|------|------|--------------------|
| 901 | tsys-siem | 8 GB |
Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).
> **Note:** the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are
> hardware-locked and only move with a physical dongle relocation — defer to the
> onsite window, not part of this RAM/CPU pass.
---
## 9. Hardware corrections discovered
### tsys5 is single-CPU, not dual
dmidecode: `CPU2: Status Unpopulated`. The Precision T7500 has **one E5620**
(4 cores / 8 threads), not two. `PROJECT.md` and prior audits said "2× E5620" —
**that was wrong.** Implications:
- CPU capacity is half what was documented. tsys5 is the weakest single-socket
CPU in the fleet (tied era with tsys6's Nehalem).
- The 6 empty DIMM slots are on the **CPU2 branch** — they are useless until a
2nd CPU is seated. Adding RAM there today does nothing.
- If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU
capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth.
This is a higher-value hardware move than adding RAM sticks.
### tsys4 is using 832 MB of swap
Only host in the fleet with active swap. Predates this audit and is minor, but a
16 GB storage server running PBS should not be swapping. After the planned
16→32/64 GB upgrade, monitor that swap returns to zero.
---
## 10. Cleanup items
| Item | Impact | Fix |
|------|--------|-----|
| **cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster** | 8 GB + 8 vCPU wasted on stale pre-rebuild VMs | Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node. |
| **No wnodes joined to k3s** | Workers run standalone; cluster has no schedulable capacity | Join wnodes (post-migration) — the §8 moves free the resources to size them well |
| **rr-middleware (703) on tsys7** | RackRental middleware lives on a RackRental host — fine | Intentional; just confirming placement is consistent |
---
## 11. Deferred to post-Friday (storage — intentionally out of scope here)
Per operator direction, disk/storage optimization happens after the Friday
maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s
StorageClass design. Tracked in [`TODO.md`](TODO.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) §5. The one RAM/CPU-adjacent note:
**tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck**, not their CPU/RAM.
---
## 12. Summary of recommended actions (this pass)
| # | Action | Cost | Effect |
|---|--------|------|--------|
| 1 | Migrate 5 lightweight VMs tsys1→tsys7 | Free (PDM) | tsys1 alloc 115%→77% |
| 2 | Migrate tsys-siem tsys9→tsys6 | Free (PDM) | tsys9 alloc 129%→95%; room to grow wnode |
| 3 | Stop + delete stale cnode4/cnode5 | Free | Reclaim 8 GB + 8 vCPU |
| 4 | Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2) | Free | Cross-compile node ready |
| 5 | **Do not** grow wnode-tsys1; keep it small | Free | Corrects the CPU-quality premise |
| 6 | Identify DDR3 stick type before any install | Free | Avoids no-POST on tsys6/7 |
| 7 | Consider seating a 2nd CPU in tsys5 (if spare LGA771) | Cheap | Doubles tsys5 CPU; activates RAM branch |
| 8 | Solar-gate tsys5/6 when idle (no RackRental load) | Free | Stops wasting power on 0%-busy old cores |
> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
> hardware** — and by consolidating/solar-gating so idle silicon isn't burning
> power for nothing.