PFVCluster/proxmox/docs/AUDIT-2026-07-29.md

Fleet Audit — 2026-07-29 (RAM/CPU placement focus)

Scope: End-to-end analysis of memory and CPU placement against the operator's capacity goals. Storage/disk optimization is intentionally out of scope for this pass and is deferred to post-Friday work (NVMe on tsys5 + k8s StorageClass design). See CAPACITY-GOALS.md for the standing targets.

Audit time: 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts) Method: perf/deploy-check.sh (full hardware dump per host) + a live capacity probe (/proc/meminfo, qm config, 5s CPU sample, kubectl top) collected via tests/remote.sh. Hosts online: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 Hosts offline: tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days) Supersedes: placement numbers in PROJECT.md and AUDIT-2026-07-28.md where they conflict.

---

1. Executive summary — fleet vs the two goals

Goal	Status	Detail
Memory ≤ 80% actual steady-state	PASS (all 7 hosts)	Highest is tsys1 at 72%. Fleet is well within budget.
CPU ~50% at idle	MISS (all 7 hosts)	Every host idles at 0-6% busy. Fleet is heavily CPU-underutilized.

Headline: The cluster has abundant RAM and CPU headroom everywhere. The memory goal is met with margin to spare; the CPU-utilization goal is not remotely met because there simply isn't enough sustained workload yet. The right response is not to add hardware — it is to (a) consolidate the always-on baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and ready for RackRental/compute bursts, and (c) let utilization rise naturally as RackRental + ETL demand comes online.

Finding	Severity	§
tsys1/5/9 are over-allocated (>115%) though actual usage is fine	Medium	4
tsys6/7 idle at 6%/9% with 117/171 GB free — perfect, underused RackRental capacity	Info	6
tsys5 has only ONE CPU (CPU2 socket unpopulated) — prior "dual E5620" record was wrong	Medium	9
cnode4 + cnode5 are stale running VMs not in the k3s cluster (waste 8 GB + 8 vCPU)	Low	10
Fleet CPU ~0-6% idle — no path to 50% without more workloads	Info	5
DDR3 sticks: compatibility-blocked from the high-RAM hosts (they need RDIMM)	Medium	7

---

2. Host capacity (live, 2026-07-29)

CPU "threads" = logical processors (nproc). "vCPU" = sum of running-VM cores × sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling). "Actual%" = (MemTotal - MemAvailable) / MemTotal (steady-state truth).

Host	CPU (era)	Threads	RAM	Running VMs	Alloc RAM	Alloc%	Actual%	Swap used	vCPU	CPU busy	Load
tsys1	i7-4770 Haswell (2013)	8	31 GB	11	36 GB	115%	72%	0	26	3%	1.3
tsys3	Xeon E3-1535M v5 Skylake (2015)	8	31 GB	1	28 GB	90%	10%	0	8	0%	0.2
tsys4	Xeon E3-1246 v3 Haswell (2013)	8	16 GB	1	2 GB	13%	13%	832 MB	2	0%	3.0
tsys5	Xeon E5620 Westmere (2010) single CPU	8	94 GB	17	126 GB	134%	37%	0	72	6%	5.1
tsys6	2× Xeon E5530 Nehalem (2009)	16	126 GB	3	48 GB	38%	6%	0	12	0%	0.2
tsys7	2× Xeon E5-2630 v2 Ivy Bridge (2013)	24	189 GB	6	56 GB	30%	9%	0	22	1%	1.2
tsys9	i5-10500 Comet Lake (2020)	12	23 GB	6	30 GB	129%	60%	0	16	1%	0.2

Notes on the table:

• tsys4 load 3.0 with 0% CPU busy = NFS/disk I/O wait (it is the storage server). The 832 MB of swap used is the only swap activity in the fleet — minor, but worth watching; it predates this audit.
• tsys5 "single CPU": dmidecode reports CPU2: Status Unpopulated. The T7500 is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads). See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and useless until a 2nd CPU is seated.
• tsys5 alloc 134% looks alarming but actual usage is only 37% — the 17 VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual low). Still, it is the most over-committed host by allocation.

---

3. Running VM inventory (RAM/CPU only)

tsys1 — Infrastructure (11 running)

VMID	Name	vCPU	RAM
100	pfv-bms (HomeAssistant)	2	4 GB
101	tsys-ca	2	2 GB
102	pfv-k8s-wnode-tsys1	4	4 GB
103	pfv-netinfra-01	2	2 GB
104	tsys-librenms	2	2 GB
105	tsys-proxmox-datacenter	2	2 GB
106	pfv-k8s-cnode3	2	4 GB
108	tsys-ucs-01	4	8 GB
109	tailscale-router	2	2 GB
114	kali-tsys	2	2 GB
117	tsys-secure-workbench	2	4 GB

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM
313	pfv-k8s-wnode-tsys3	8	28 GB

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM
400	pfv-proxmox-backup-server	2	2 GB

tsys5 — Storage + sandboxes (17 running)

VMID	Name	vCPU	RAM
509	pfv-k8s-wnode-tsys5	8	32 GB
5101	sectestbed-siem	4	10 GB
5111	ultix-streaming	4	9 GB
53100	tsys-preprod-awx	4	9 GB
53101	tsys-preprod-siem	4	12 GB
53102	tsys-preprod-rancherplatform	4	8 GB
5112	ultix-offstage	4	6 GB
51010-51013,5105-5109,6000	sectestbed suite (×10)	4 ea	4 GB ea

tsys6 — RackRental-ready (3 running)

VMID	Name	vCPU	RAM
100	pfv-k8s-wnode-tsys6	4	32 GB
600	tsys-awx	4	12 GB
601	pfv-k8s-cnode4 (stale — not in cluster)	4	4 GB

tsys7 — RackRental-ready (6 running)

VMID	Name	vCPU	RAM
701	pfv-k8s-wnode-tsys7	4	32 GB
702	hfnoc-uisp	4	8 GB
704	TCTC	4	6 GB
705	pfv-k8s-cnode2	4	4 GB
706	pfv-k8s-cnode5 (stale — not in cluster)	4	4 GB
703	rr-middleware	2	2 GB

tsys9 — Infra + control plane (6 running)

VMID	Name	vCPU	RAM
901	tsys-siem	2	8 GB
902	tsys-ucs-02	4	8 GB
905	pfv-k8s-wnode-tsys9	4	4 GB
904	pfv-netinfra-02	2	4 GB
906	pfv-k8s-cnode1	2	4 GB
903	kali-rd	2	2 GB

k3s control plane (ground truth via kubectl)

Only 3 cnodes are actually joined to the pfv-k8s cluster:

Node	Host	CPU	Mem	Role
cnode1 (906)	tsys9	2%	1181 Mi (30%)	control-plane,etcd
cnode2 (705)	tsys7	2%	1206 Mi (30%)	control-plane,etcd
cnode3 (106)	tsys1	2%	1124 Mi (28%)	control-plane,etcd

cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are running as VMs but not joined — leftovers from the pre-rebuild 5-node cluster. See §10. No wnodes are joined yet (all workers run standalone, pending the join step).

---

4. Memory analysis vs the 80% goal

Verdict: every host passes actual usage. The 80% steady-state rule is met fleet-wide with margin. The hosts closest to the line:

Host	Actual%	Alloc%	Risk	Action
tsys1	72%	115%	Closest to 80%; alloc > 100%	Shed 5 lightweight VMs → tsys7 (§8)
tsys9	60%	129%	Alloc > 100% but actual comfortable	Move tsys-siem (8 GB) → tsys6 (§8)
tsys5	37%	134%	Alloc highest, but mostly idle sandboxes	No action now; monitor if sandboxes wake
tsys3	10%	90%	None	—
tsys6	6%	38%	None (most empty)	Absorb migrated VMs
tsys7	9%	30%	None	Absorb migrated VMs

Key insight: actual usage (72% max) is far below allocation (134% max) because of KSM/balloon. The memory goal is about actual, and the fleet is healthy. The over-allocation on tsys1/5/9 is a planning concern, resolved by the migrations in §8 — no RAM purchase required.

5. CPU analysis vs the 50%-idle goal

Verdict: the fleet misses the target badly — in the safe direction. Every host is 0-6% busy at idle. This is not a capacity problem; it is an under-utilization problem.

You cannot manufacture 50% utilization without workloads to run. The realistic strategy, in priority order:

1. Consolidate the always-on baseline onto the efficient hosts. tsys9 (Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the control plane + infra. Their baseline will rise toward the target as compute workloads land.
2. Keep tsys6/7 warm for bursts, not baseline. They are the RackRental and parallel-batch hosts. Their idle utilization will climb to ~50% naturally as RackRental customers spin up labs — that is the designed use, not waste.
3. Solar-gate the inefficient idle hosts when truly idle. tsys5 (single Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running RackRental/batch, they are candidates for power-down / solar-aware scheduling (aligns with the existing solar-aware scale-out concept). Running a Nehalem box at 0% busy 24/7 is pure electricity cost.
4. Grow into it. As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7) demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend to chase the number.

---

6. Workload placement model

Workload family	Needs	Best hosts	Why
ETL / HPC / cross-compile	High IPC + fast storage	tsys9, tsys3, tsys2(in)	Newest CPUs + only local SSD/NVMe in fleet
RackRental.net (network labs)	Massive RAM, CPU-light	tsys7, tsys6	189/126 GB RAM; weak CPU fine for idle labs
Infra + k8s control plane	Always-on, low resource	tsys1, tsys9	Efficient enough; cnodes spread for etcd diversity

CPU quality ranking (newest → oldest IPC): tsys9 (Comet Lake 2020) > tsys3 (Skylake 2015) > tsys7 (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell 2013) > tsys5/6 (Westmere/Nehalem 2009-2010).

---

7. Answers to the operator's questions

7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"

tsys9 — YES. It has the newest CPU in the fleet (i5-10500, 12 threads) and a local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural cross-compile champion. To grow it, first free RAM by moving tsys-siem (8 GB) to tsys6 (§8), then bump wnode-tsys9 to 6 vCPU / 16 GB. Actual usage on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.

tsys1 — NO (premise correction). The i7-4770 is Haswell 2013 — the second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most RAM-loaded host (72% actual, 115% alloc). It should shed VMs, not grow its wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.

7.2 "Are tsys6/7 better for RackRental.net workloads?"

Yes — exactly right. RackRental (renting ephemeral network labs) is RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls. tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM. Their older CPUs are irrelevant because labs sit idle between interactions.

They do NOT need more RAM. They idle at 9%/6% actual with 171/117 GB free. They are over-provisioned in the right direction. Their real bottleneck is disk (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls), which is the deferred Friday + StorageClass work, not a RAM/CPU issue.

7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)

First, identify the exact DIMM type. Compatibility is decisive and the fleet's DDR3 hosts are not interchangeable:

Host	Platform	Accepts	Current	Empty slots	Notes
tsys6 (R610)	PowerEdge server	RDIMM ECC only	126 GB RDIMM 1333	many (B-branch)	Consumer UDIMM → no POST
tsys7 (R620)	PowerEdge server	RDIMM ECC only	189 GB RDIMM 1600	many (A7/A8 + B)	Consumer UDIMM → no POST
tsys5 (T7500)	Workstation	RDIMM ECC	94 GB RDIMM 1600	6 (CPU2 branch)	Useless without a 2nd CPU (§9)
tsys4 (T1700)	Xeon E3 workstation	ECC UDIMM	16 GB (4×4)	0 (must swap)	Max 32 GB (4×8). 16GB sticks unsupported
tsys1 (OptiPlex 9020)	Consumer	non-ECC UDIMM	32 GB (4×8)	0	Maxed. Haswell caps at 8 GB/slot

Decision tree (run this before buying/installing anything):

1. Read the SPD on each stick (labels or decode-dimms / dmidecode -t memory once installed in any testable board). The binary question is: Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.
2. If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):
   • They fit nowhere useful. tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable there). tsys4 requires ECC and caps at 8 GB/slot.
   • The only possible target is tsys4 — and only the 8 GB sticks, and only if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4 16→32 GB. Marginal.
3. If the sticks are RDIMM ECC (unlikely for Patriot, possible for some Kingston server lines):
   • Deploy 16 GB RDIMM into tsys7/tsys6 empty B-branch slots to balance memory across both CPUs (currently CPU-B branches are partly empty — single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
   • But value is low: those hosts don't need more RAM (§7.2).

Bottom line on DDR3: Do not spend effort/money chasing RAM into tsys6/7 — they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9 over-allocation) is solved for free by VM migration (§8), not by sticks. If the sticks are free/cheap UDIMM, the realistic win is just tsys4 16→32 GB (if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7 will stop them from POSTing.

---

8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)

These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge free RAM. NFS-backed VMs migrate cleanly via PDM (storage stays on the same NFS export; only the compute host changes).

From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):

VMID	Name	RAM freed on tsys1
104	tsys-librenms	2 GB
105	tsys-proxmox-datacenter	2 GB
117	tsys-secure-workbench	4 GB
109	tailscale-router	2 GB
114	kali-tsys	2 GB
	Total	12 GB

Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.

From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):

VMID	Name	RAM freed on tsys9
901	tsys-siem	8 GB

Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).

Note: the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are hardware-locked and only move with a physical dongle relocation — defer to the onsite window, not part of this RAM/CPU pass.

---

9. Hardware corrections discovered

tsys5 is single-CPU, not dual

dmidecode: CPU2: Status Unpopulated. The Precision T7500 has one E5620 (4 cores / 8 threads), not two. PROJECT.md and prior audits said "2× E5620" — that was wrong. Implications:

• CPU capacity is half what was documented. tsys5 is the weakest single-socket CPU in the fleet (tied era with tsys6's Nehalem).
• The 6 empty DIMM slots are on the CPU2 branch — they are useless until a 2nd CPU is seated. Adding RAM there today does nothing.
• If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth. This is a higher-value hardware move than adding RAM sticks.

tsys4 is using 832 MB of swap

Only host in the fleet with active swap. Predates this audit and is minor, but a 16 GB storage server running PBS should not be swapping. After the planned 16→32/64 GB upgrade, monitor that swap returns to zero.

---

10. Cleanup items

Item	Impact	Fix
cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster	8 GB + 8 vCPU wasted on stale pre-rebuild VMs	Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node.
No wnodes joined to k3s	Workers run standalone; cluster has no schedulable capacity	Join wnodes (post-migration) — the §8 moves free the resources to size them well
rr-middleware (703) on tsys7	RackRental middleware lives on a RackRental host — fine	Intentional; just confirming placement is consistent

---

11. Deferred to post-Friday (storage — intentionally out of scope here)

Per operator direction, disk/storage optimization happens after the Friday maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s StorageClass design. Tracked in TODO.md and AUDIT-2026-07-28.md §5. The one RAM/CPU-adjacent note: tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck, not their CPU/RAM.

---

12. Summary of recommended actions (this pass)

#	Action	Cost	Effect
1	Migrate 5 lightweight VMs tsys1→tsys7	Free (PDM)	tsys1 alloc 115%→77%
2	Migrate tsys-siem tsys9→tsys6	Free (PDM)	tsys9 alloc 129%→95%; room to grow wnode
3	Stop + delete stale cnode4/cnode5	Free	Reclaim 8 GB + 8 vCPU
4	Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2)	Free	Cross-compile node ready
5	Do not grow wnode-tsys1; keep it small	Free	Corrects the CPU-quality premise
6	Identify DDR3 stick type before any install	Free	Avoids no-POST on tsys6/7
7	Consider seating a 2nd CPU in tsys5 (if spare LGA771)	Cheap	Doubles tsys5 CPU; activates RAM branch
8	Solar-gate tsys5/6 when idle (no RackRental load)	Free	Stops wasting power on 0%-busy old cores

Memory goal: already met. CPU goal: chase it by adding workloads, not hardware — and by consolidating/solar-gating so idle silicon isn't burning power for nothing.