PFVCluster/proxmox/docs/AUDIT-2026-07-29.md

Fleet Audit — 2026-07-29 (RAM/CPU placement focus)

Scope: End-to-end analysis of memory and CPU placement against the operator's capacity goals, plus HDD disk placement for IOPS balancing (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally deferred to post-Friday work (NVMe on tsys5). See CAPACITY-GOALS.md for the standing targets.

Audit time: 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts) Method: perf/deploy-check.sh (full hardware dump per host) + a live capacity probe (/proc/meminfo, qm config, 5s CPU sample, kubectl top) collected via tests/remote.sh. Hosts online: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 Hosts offline: tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days) Supersedes: placement numbers in PROJECT.md and AUDIT-2026-07-28.md where they conflict.

---

1. Executive summary — fleet vs the two goals

Goal	Status	Detail
Memory ≤ 80% actual steady-state	PASS (all 7 hosts)	Highest is tsys1 at 72%. Fleet is well within budget.
CPU ~50% at idle	MISS (all 7 hosts)	Every host idles at 0-6% busy. Fleet is heavily CPU-underutilized.

Headline: The cluster has abundant RAM and CPU headroom everywhere. The memory goal is met with margin to spare; the CPU-utilization goal is not remotely met because there simply isn't enough sustained workload yet. The right response is not to add hardware — it is to (a) consolidate the always-on baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and ready for RackRental/compute bursts, and (c) let utilization rise naturally as RackRental + ETL demand comes online.

Finding	Severity	§
tsys1/5/9 are over-allocated (>115%) though actual usage is fine	Medium	4
tsys6/7 idle at 6%/9% with 117/171 GB free — perfect, underused RackRental capacity	Info	6
tsys5 has only ONE CPU (CPU2 socket unpopulated) — prior "dual E5620" record was wrong	Medium	9
cnode4 + cnode5 are stale running VMs not in the k3s cluster (waste 8 GB + 8 vCPU)	Low	10
Fleet CPU ~0-6% idle — no path to 50% without more workloads	Info	5
DDR3 sticks: compatibility-blocked from the high-RAM hosts (they need RDIMM)	Medium	7

---

2. Host capacity (live, 2026-07-29)

CPU "threads" = logical processors (nproc). "vCPU" = sum of running-VM cores × sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling). "Actual%" = (MemTotal - MemAvailable) / MemTotal (steady-state truth).

Host	CPU (era)	Threads	RAM	Running VMs	Alloc RAM	Alloc%	Actual%	Swap used	vCPU	CPU busy	Load
tsys1	i7-4770 Haswell (2013)	8	31 GB	11	36 GB	115%	72%	0	26	3%	1.3
tsys3	Xeon E3-1535M v5 Skylake (2015)	8	31 GB	1	28 GB	90%	10%	0	8	0%	0.2
tsys4	Xeon E3-1246 v3 Haswell (2013)	8	16 GB	1	2 GB	13%	13%	832 MB	2	0%	3.0
tsys5	Xeon E5620 Westmere (2010) single CPU	8	94 GB	17	126 GB	134%	37%	0	72	6%	5.1
tsys6	2× Xeon E5530 Nehalem (2009)	16	126 GB	3	48 GB	38%	6%	0	12	0%	0.2
tsys7	2× Xeon E5-2630 v2 Ivy Bridge (2013)	24	189 GB	6	56 GB	30%	9%	0	22	1%	1.2
tsys9	i5-10500 Comet Lake (2020)	12	23 GB	6	30 GB	129%	60%	0	16	1%	0.2

Notes on the table:

• tsys4 load 3.0 with 0% CPU busy = NFS/disk I/O wait (it is the storage server). The 832 MB of swap used is the only swap activity in the fleet — minor, but worth watching; it predates this audit.
• tsys5 "single CPU": dmidecode reports CPU2: Status Unpopulated. The T7500 is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads). See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and useless until a 2nd CPU is seated.
• tsys5 alloc 134% looks alarming but actual usage is only 37% — the 17 VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual low). Still, it is the most over-committed host by allocation.

---

3. Running VM inventory (RAM/CPU only)

tsys1 — Infrastructure (11 running)

VMID	Name	vCPU	RAM
100	pfv-bms (HomeAssistant)	2	4 GB
101	tsys-ca	2	2 GB
102	pfv-k8s-wnode-tsys1	4	4 GB
103	pfv-netinfra-01	2	2 GB
104	tsys-librenms	2	2 GB
105	tsys-proxmox-datacenter	2	2 GB
106	pfv-k8s-cnode3	2	4 GB
108	tsys-ucs-01	4	8 GB
109	tailscale-router	2	2 GB
114	kali-tsys	2	2 GB
117	tsys-secure-workbench	2	4 GB

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM
313	pfv-k8s-wnode-tsys3	8	28 GB

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM
400	pfv-proxmox-backup-server	2	2 GB

tsys5 — Storage + sandboxes (17 running)

VMID	Name	vCPU	RAM
509	pfv-k8s-wnode-tsys5	8	32 GB
5101	sectestbed-siem	4	10 GB
5111	ultix-streaming	4	9 GB
53100	tsys-preprod-awx	4	9 GB
53101	tsys-preprod-siem	4	12 GB
53102	tsys-preprod-rancherplatform	4	8 GB
5112	ultix-offstage	4	6 GB
51010-51013,5105-5109,6000	sectestbed suite (×10)	4 ea	4 GB ea

tsys6 — RackRental-ready (3 running)

VMID	Name	vCPU	RAM
100	pfv-k8s-wnode-tsys6	4	32 GB
600	tsys-awx	4	12 GB
601	pfv-k8s-cnode4 (stale — not in cluster)	4	4 GB

tsys7 — RackRental-ready (6 running)

VMID	Name	vCPU	RAM
701	pfv-k8s-wnode-tsys7	4	32 GB
702	hfnoc-uisp	4	8 GB
704	TCTC	4	6 GB
705	pfv-k8s-cnode2	4	4 GB
706	pfv-k8s-cnode5 (stale — not in cluster)	4	4 GB
703	rr-middleware	2	2 GB

tsys9 — Infra + control plane (6 running)

VMID	Name	vCPU	RAM
901	tsys-siem	2	8 GB
902	tsys-ucs-02	4	8 GB
905	pfv-k8s-wnode-tsys9	4	4 GB
904	pfv-netinfra-02	2	4 GB
906	pfv-k8s-cnode1	2	4 GB
903	kali-rd	2	2 GB

k3s control plane (ground truth via kubectl)

Only 3 cnodes are actually joined to the pfv-k8s cluster:

Node	Host	CPU	Mem	Role
cnode1 (906)	tsys9	2%	1181 Mi (30%)	control-plane,etcd
cnode2 (705)	tsys7	2%	1206 Mi (30%)	control-plane,etcd
cnode3 (106)	tsys1	2%	1124 Mi (28%)	control-plane,etcd

cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are running as VMs but not joined — leftovers from the pre-rebuild 5-node cluster. See §10. No wnodes are joined yet (all workers run standalone, pending the join step).

---

4. Memory analysis vs the 80% goal

Verdict: every host passes actual usage. The 80% steady-state rule is met fleet-wide with margin. The hosts closest to the line:

Host	Actual%	Alloc%	Risk	Action
tsys1	72%	115%	Closest to 80%; alloc > 100%	Shed 5 lightweight VMs → tsys7 (§8)
tsys9	60%	129%	Alloc > 100% but actual comfortable	Move tsys-siem (8 GB) → tsys6 (§8)
tsys5	37%	134%	Alloc highest, but mostly idle sandboxes	No action now; monitor if sandboxes wake
tsys3	10%	90%	None	—
tsys6	6%	38%	None (most empty)	Absorb migrated VMs
tsys7	9%	30%	None	Absorb migrated VMs

Key insight: actual usage (72% max) is far below allocation (134% max) because of KSM/balloon. The memory goal is about actual, and the fleet is healthy. The over-allocation on tsys1/5/9 is a planning concern, resolved by the migrations in §8 — no RAM purchase required.

5. CPU analysis vs the 50%-idle goal

Verdict: the fleet misses the target badly — in the safe direction. Every host is 0-6% busy at idle. This is not a capacity problem; it is an under-utilization problem.

You cannot manufacture 50% utilization without workloads to run. The realistic strategy, in priority order:

1. Consolidate the always-on baseline onto the efficient hosts. tsys9 (Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the control plane + infra. Their baseline will rise toward the target as compute workloads land.
2. Keep tsys6/7 warm for bursts, not baseline. They are the RackRental and parallel-batch hosts. Their idle utilization will climb to ~50% naturally as RackRental customers spin up labs — that is the designed use, not waste.
3. Solar-gate the inefficient idle hosts when truly idle. tsys5 (single Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running RackRental/batch, they are candidates for power-down / solar-aware scheduling (aligns with the existing solar-aware scale-out concept). Running a Nehalem box at 0% busy 24/7 is pure electricity cost.
4. Grow into it. As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7) demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend to chase the number.

---

6. Workload placement model

Workload family	Needs	Best hosts	Why
ETL / HPC / cross-compile	High IPC + fast storage	tsys9, tsys3, tsys2(in)	Newest CPUs + only local SSD/NVMe in fleet
RackRental.net (network labs)	Massive RAM, CPU-light	tsys7, tsys6	189/126 GB RAM; weak CPU fine for idle labs
Infra + k8s control plane	Always-on, low resource	tsys1, tsys9	Efficient enough; cnodes spread for etcd diversity

CPU quality ranking (newest → oldest IPC): tsys9 (Comet Lake 2020) > tsys3 (Skylake 2015) > tsys7 (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell 2013) > tsys5/6 (Westmere/Nehalem 2009-2010).

---

7. Answers to the operator's questions

7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"

tsys9 — YES. It has the newest CPU in the fleet (i5-10500, 12 threads) and a local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural cross-compile champion. To grow it, first free RAM by moving tsys-siem (8 GB) to tsys6 (§8), then bump wnode-tsys9 to 6 vCPU / 16 GB. Actual usage on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.

tsys1 — NO (premise correction). The i7-4770 is Haswell 2013 — the second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most RAM-loaded host (72% actual, 115% alloc). It should shed VMs, not grow its wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.

7.2 "Are tsys6/7 better for RackRental.net workloads?"

Yes — exactly right. RackRental (renting ephemeral network labs) is RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls. tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM. Their older CPUs are irrelevant because labs sit idle between interactions.

They do NOT need more RAM. They idle at 9%/6% actual with 171/117 GB free. They are over-provisioned in the right direction. Their real bottleneck is disk (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls), which is the deferred Friday + StorageClass work, not a RAM/CPU issue.

7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)

First, identify the exact DIMM type. Compatibility is decisive and the fleet's DDR3 hosts are not interchangeable:

Host	Platform	Accepts	Current	Empty slots	Notes
tsys6 (R610)	PowerEdge server	RDIMM ECC only	126 GB RDIMM 1333	many (B-branch)	Consumer UDIMM → no POST
tsys7 (R620)	PowerEdge server	RDIMM ECC only	189 GB RDIMM 1600	many (A7/A8 + B)	Consumer UDIMM → no POST
tsys5 (T7500)	Workstation	RDIMM ECC	94 GB RDIMM 1600	6 (CPU2 branch)	Useless without a 2nd CPU (§9)
tsys4 (T1700)	Xeon E3 workstation	ECC UDIMM	16 GB (4×4)	0 (must swap)	Max 32 GB (4×8). 16GB sticks unsupported
tsys1 (OptiPlex 9020)	Consumer	non-ECC UDIMM	32 GB (4×8)	0	Maxed. Haswell caps at 8 GB/slot

Decision tree (run this before buying/installing anything):

1. Read the SPD on each stick (labels or decode-dimms / dmidecode -t memory once installed in any testable board). The binary question is: Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.
2. If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):
   • They fit nowhere useful. tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable there). tsys4 requires ECC and caps at 8 GB/slot.
   • The only possible target is tsys4 — and only the 8 GB sticks, and only if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4 16→32 GB. Marginal.
3. If the sticks are RDIMM ECC (unlikely for Patriot, possible for some Kingston server lines):
   • Deploy 16 GB RDIMM into tsys7/tsys6 empty B-branch slots to balance memory across both CPUs (currently CPU-B branches are partly empty — single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
   • But value is low: those hosts don't need more RAM (§7.2).

Bottom line on DDR3: Do not spend effort/money chasing RAM into tsys6/7 — they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9 over-allocation) is solved for free by VM migration (§8), not by sticks. If the sticks are free/cheap UDIMM, the realistic win is just tsys4 16→32 GB (if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7 will stop them from POSTing.

---

8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)

These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge free RAM. NFS-backed VMs migrate cleanly via PDM (storage stays on the same NFS export; only the compute host changes).

From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):

VMID	Name	RAM freed on tsys1
104	tsys-librenms	2 GB
105	tsys-proxmox-datacenter	2 GB
117	tsys-secure-workbench	4 GB
109	tailscale-router	2 GB
114	kali-tsys	2 GB
	Total	12 GB

Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.

From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):

VMID	Name	RAM freed on tsys9
901	tsys-siem	8 GB

Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).

Note: the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are hardware-locked and only move with a physical dongle relocation — defer to the onsite window, not part of this RAM/CPU pass.

---

9. Hardware corrections discovered

tsys5 is single-CPU, not dual

dmidecode: CPU2: Status Unpopulated. The Precision T7500 has one E5620 (4 cores / 8 threads), not two. PROJECT.md and prior audits said "2× E5620" — that was wrong. Implications:

• CPU capacity is half what was documented. tsys5 is the weakest single-socket CPU in the fleet (tied era with tsys6's Nehalem).
• The 6 empty DIMM slots are on the CPU2 branch — they are useless until a 2nd CPU is seated. Adding RAM there today does nothing.
• If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth. This is a higher-value hardware move than adding RAM sticks.

tsys4 is using 832 MB of swap

Only host in the fleet with active swap. Predates this audit and is minor, but a 16 GB storage server running PBS should not be swapping. After the planned 16→32/64 GB upgrade, monitor that swap returns to zero.

---

10. Cleanup items

Item	Impact	Fix
cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster	8 GB + 8 vCPU wasted on stale pre-rebuild VMs	Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node.
No wnodes joined to k3s	Workers run standalone; cluster has no schedulable capacity	Join wnodes (post-migration) — the §8 moves free the resources to size them well
rr-middleware (703) on tsys7	RackRental middleware lives on a RackRental host — fine	Intentional; just confirming placement is consistent

---

11. Deferred to post-Friday (storage — intentionally out of scope here)

Per operator direction, disk/storage optimization happens after the Friday maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s StorageClass design. Tracked in TODO.md and AUDIT-2026-07-28.md §5. The one RAM/CPU-adjacent note: tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck, not their CPU/RAM.

---

12. Summary of recommended actions (this pass)

#	Action	Cost	Effect
1	Migrate 5 lightweight VMs tsys1→tsys7	Free (PDM)	tsys1 alloc 115%→77%
2	Migrate tsys-siem tsys9→tsys6	Free (PDM)	tsys9 alloc 129%→95%; room to grow wnode
3	Stop + delete stale cnode4/cnode5	Free	Reclaim 8 GB + 8 vCPU
4	Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2)	Free	Cross-compile node ready
5	Do not grow wnode-tsys1; keep it small	Free	Corrects the CPU-quality premise
6	Identify DDR3 stick type before any install	Free	Avoids no-POST on tsys6/7
7	Consider seating a 2nd CPU in tsys5 (if spare LGA771)	Cheap	Doubles tsys5 CPU; activates RAM branch
8	Solar-gate tsys5/6 when idle (no RackRental load)	Free	Stops wasting power on 0%-busy old cores

Memory goal: already met. CPU goal: chase it by adding workloads, not hardware — and by consolidating/solar-gating so idle silicon isn't burning power for nothing.

---

13. Current VM inventory (host + disk, as of 2026-07-29)

Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate, DellOpenManage, sectestbed stopped clones) are omitted — they consume no RAM/CPU. Disk = the NFS export holding the primary qcow2; the server in parens is which physical box serves it (tsys4 or tsys5).

tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual)

VMID	Name	vCPU	RAM	Disk (server)
100	pfv-bms (HomeAssistant)	2	4 GB	D2 (tsys4)
101	tsys-ca	2	2 GB	D2 (tsys4)
102	pfv-k8s-wnode-tsys1	4	4 GB	S3 (tsys5)
103	pfv-netinfra-01	2	2 GB	S2 (tsys5)
104	tsys-librenms	2	2 GB	D2 (tsys4)
105	tsys-proxmox-datacenter	2	2 GB	D2 (tsys4)
106	pfv-k8s-cnode3	2	4 GB	D5 (tsys4)
108	tsys-ucs-01	4	8 GB	D2 (tsys4)
109	tailscale-router	2	2 GB	D2 (tsys4)
114	kali-tsys	2	2 GB	D2 (tsys4)
117	tsys-secure-workbench	2	4 GB	D2 (tsys4)

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM	Disk (server)
313	pfv-k8s-wnode-tsys3	8	28 GB	D5 (tsys4)

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM	Disk (server)
400	pfv-proxmox-backup-server	2	2 GB	local-lvm (LOCAL)

tsys5 — Storage + sandboxes (17 running)

VMID	Name	vCPU	RAM	Disk (server)
509	pfv-k8s-wnode-tsys5	8	32 GB	D2 (tsys4)
5111	ultix-streaming	4	9 GB	local (LOCAL)
5112	ultix-offstage	4	6 GB	local-lvm (LOCAL)
5101	sectestbed-siem	4	10 GB	local-nonprod (LOCAL)
53100	tsys-preprod-awx	4	9 GB	local-nonprod (LOCAL)
53101	tsys-preprod-siem	4	12 GB	local-nonprod (LOCAL)
53102	tsys-preprod-rancherplatform	4	8 GB	local-nonprod (LOCAL)
×9	sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000)	4 ea	4 GB ea	local-nonprod (LOCAL)

The 16 LOCAL-tagged VMs are locked to tsys5 (disks on local-nonprod / local-lvm — cannot migrate). They are idle sandboxes; leave them.

tsys6 — RackRental-ready (3 running)

VMID	Name	vCPU	RAM	Disk (server)
100	pfv-k8s-wnode-tsys6	4	32 GB	S4 (tsys5)
600	tsys-awx	4	12 GB	D2 (tsys4)
601	pfv-k8s-cnode4 (stale)	4	4 GB	D2 (tsys4)

tsys7 — RackRental-ready (6 running)

VMID	Name	vCPU	RAM	Disk (server)
701	pfv-k8s-wnode-tsys7	4	32 GB	S3 (tsys5)
702	hfnoc-uisp	4	8 GB	D2 (tsys4)
703	rr-middleware	2	2 GB	D2 (tsys4)
704	TCTC	4	6 GB	D2 (tsys4)
705	pfv-k8s-cnode2	4	4 GB	D2 (tsys4)
706	pfv-k8s-cnode5 (stale)	4	4 GB	S2 (tsys5)

tsys9 — Infra + control plane (6 running)

VMID	Name	vCPU	RAM	Disk (server)
901	tsys-siem	2	8 GB	D2 (tsys4)
902	tsys-ucs-02	4	8 GB	S2 (tsys5)
903	kali-rd	2	2 GB	D5 (tsys4)
904	pfv-netinfra-02	2	4 GB	D2 (tsys4)
905	pfv-k8s-wnode-tsys9	4	4 GB	S2 (tsys5)
906	pfv-k8s-cnode1	2	4 GB	S3 (tsys5)

Current per-spindle IOPS load (HDD exports only)

Spindle	Server	# VMs	Status
D2 (WDC Red 3TB)	tsys4	17	CRITICAL — 65% of all NFS VMs contend here
D5 (Hitachi 2TB)	tsys4	3	OK
S1 (Seagate 916GB)	tsys5	0	EMPTY — wasted spindle
S2 (Seagate 916GB)	tsys5	4	OK
S3 (Seagate 916GB)	tsys5	3	OK
S4 (Toshiba 458GB)	tsys5	1	Nearly empty

The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.

---

14. Target end-state inventory (RAM/CPU + IOPS-balanced)

All disks stay on HDD (no SSD/NVMe moves — deferred to post-Friday). The goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers.

tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)

VMID	Name	vCPU	RAM	Disk (server)	Change
100	pfv-bms	2	4 GB	D2 (tsys4)	none
101	tsys-ca	2	2 GB	D2 (tsys4)	none
102	pfv-k8s-wnode-tsys1	4	4 GB	S3 (tsys5)	none
103	pfv-netinfra-01	2	2 GB	S2 (tsys5)	none
106	pfv-k8s-cnode3	2	4 GB	D5 (tsys4)	none
108	tsys-ucs-01	4	8 GB	D2 (tsys4)	none

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM	Disk (server)	Change
313	pfv-k8s-wnode-tsys3	8	28 GB	D5 (tsys4)	none

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM	Disk (server)	Change
400	pfv-proxmox-backup-server	2	2 GB	local-lvm	none

tsys5 — Storage + sandboxes (17 running)

VMID	Name	vCPU	RAM	Disk (server)	Change
509	pfv-k8s-wnode-tsys5	8	32 GB	S4 (tsys5)	disk D2→S4
—	(16 locked sandbox VMs)	—	—	local-nonprod	none

tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)

VMID	Name	vCPU	RAM	Disk (server)	Change
100	pfv-k8s-wnode-tsys6	4	32 GB	S4 (tsys5)	none
600	tsys-awx	4	12 GB	D5 (tsys4)	disk D2→D5
901	tsys-siem	2	8 GB	S1 (tsys5)	host tsys9→tsys6, disk D2→S1
601	cnode4 (stale)	—	—	—	DELETE

tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)

VMID	Name	vCPU	RAM	Disk (server)	Change
701	pfv-k8s-wnode-tsys7	4	32 GB	S3 (tsys5)	none
702	hfnoc-uisp	4	8 GB	S1 (tsys5)	disk D2→S1
703	rr-middleware	2	2 GB	S4 (tsys5)	disk D2→S4
704	TCTC	4	6 GB	S1 (tsys5)	disk D2→S1
705	pfv-k8s-cnode2	4	4 GB	S2 (tsys5)	disk D2→S2
104	tsys-librenms	2	2 GB	S1 (tsys5)	host tsys1→tsys7, disk D2→S1
105	tsys-proxmox-datacenter	2	2 GB	S1 (tsys5)	host tsys1→tsys7, disk D2→S1
109	tailscale-router	2	2 GB	D2 (tsys4)	host tsys1→tsys7
114	kali-tsys	2	2 GB	S3 (tsys5)	host tsys1→tsys7, disk D2→S3
117	tsys-secure-workbench	2	4 GB	S4 (tsys5)	host tsys1→tsys7, disk D2→S4
706	cnode5 (stale)	—	—	—	DELETE

tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*)

VMID	Name	vCPU	RAM	Disk (server)	Change
902	tsys-ucs-02	4	8 GB	S2 (tsys5)	none
903	kali-rd	2	2 GB	D5 (tsys4)	none
904	pfv-netinfra-02	2	4 GB	D2 (tsys4)	none
905	pfv-k8s-wnode-tsys9	6	8 GB	S2 (tsys5)	grow 4c/4G→6c/8G
906	pfv-k8s-cnode1	2	4 GB	S3 (tsys5)	none
901	tsys-siem	—	—	—	move→tsys6

*tsys9 stays over 100% by allocation, but actual usage stays <80% via KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to 8 GB / 6 cores — enough to be useful for cross-compile. The full 16 GB cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and tsys2 (post-Friday).

End-state per-spindle IOPS load

Spindle	Server	# VMs	VMs on it
D2 (WDC Red 3TB)	tsys4	5	bms, ca, ucs01, tailscale, netinfra02
D5 (Hitachi 2TB)	tsys4	4	cnode3, wnode3, awx, kali-rd
S1 (Seagate 916GB)	tsys5	5	librenms, pdm, siem, uisp, TCTC
S2 (Seagate 916GB)	tsys5	4	netinfra01, cnode2, ucs02, wnode9
S3 (Seagate 916GB)	tsys5	4	wnode1, wnode7, cnode1, kali
S4 (Toshiba 458GB)	tsys5	4	wnode5, wnode6, rr-middleware, workbench

D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a hotspot. HA pairs stay split (netinfra: S2/D2; ucs: D2/S2).

---

15. Action checklist (do in this order)

Two operation types: host migrate (PDM "Migrate" — moves compute, disk stays put) and disk migrate (Proxmox UI: VM → Hardware → disk → Move Storage, or qm move-disk). For NFS VMs both can be done live with brief I/O pauses. Recommended: do disk moves first, then host moves.

Phase 0 — cleanup (frees resources first)

• Stop + destroy cnode4 (VMID 601) on tsys6 — not joined to the 3-node cluster; stale.
• Stop + destroy cnode5 (VMID 706) on tsys7 — same. Frees 8 GB + 8 vCPU total.

Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves)

Order: move the VMs whose HOST is also changing last (Phase 2), so you do one disk move per still-on-original-host VM first. All targets are HDD exports.

• wnode-tsys5 (509): move disk D2 → S4 (host stays tsys5)
• tsys-awx (600): move disk D2 → D5 (host stays tsys6)
• hfnoc-uisp (702): move disk D2 → S1 (host stays tsys7)
• rr-middleware (703): move disk D2 → S4 (host stays tsys7)
• TCTC (704): move disk D2 → S1 (host stays tsys7)
• cnode2 (705): move disk D2 → S2 (host stays tsys7)

Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs)

• tsys-librenms (104): host tsys1 → tsys7, disk D2 → S1
• tsys-proxmox-datacenter (105): host tsys1 → tsys7, disk D2 → S1
• tailscale-router (109): host tsys1 → tsys7 (disk stays D2)
• kali-tsys (114): host tsys1 → tsys7, disk D2 → S3
• tsys-secure-workbench (117): host tsys1 → tsys7, disk D2 → S4
• tsys-siem (901): host tsys9 → tsys6, disk D2 → S1

Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs.

Phase 3 — grow the compute wnode

• wnode-tsys9 (905): resize 4c/4GB → 6c/8GB (requires stop/start; done after siem leaves tsys9)

Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9)

• tsys-ucs-02 (902): host tsys9 → tsys6 (disk stays S2) — frees 8 GB on tsys9
• Then grow wnode-tsys9 (905): 8 GB → 16 GB (tsys9 alloc then ~108% but actual still <80%)

Phase 5 — bring the workers into the cluster (after placement settles)

• Join the wnodes (wnode-tsys1/3/5/6/7/9) to the k3s cluster via k8s/join-servers.sh
• Label them by role: workload=compute (tsys3/9/2), workload=rackrental (tsys6/7), workload=light (tsys1/5)

Verification after each phase

• qm list on each host — confirm VM count + status
• pveperf / cat /proc/meminfo — confirm actual RAM ≤ 80%
• kubectl top nodes — confirm cnodes healthy
• After Phase 1/2: ssh root@pfv-tsys4 'df -h /mnt/*' and ssh root@pfv-tsys5 'df -h /mnt/*' — confirm disks rebalanced (D2 frees up, S1 fills)