PFVCluster/proxmox/docs/AUDIT-2026-07-29.md

Fleet Audit — 2026-07-29 (consolidated, storage-corrected)

The single authoritative audit. Supersedes all prior audits and the placement data in PROJECT.md §4-§8 where they conflict. Capacity targets are codified in CAPACITY-GOALS.md.

Audit time: 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware dumps + ~05:40 CDT capacity probe). Storage configs read directly from /etc/pve/storage.cfg, /etc/exports, /etc/fstab on every host. Hosts online: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 Hosts offline: tsys2 (Win10, pending rebuild), tsys8 (offline)

---

1. Executive summary

Goal	Status	Detail
Memory ≤ 80% actual steady-state	PASS (all 7 hosts)	Highest is tsys1 at 73%. Fleet is healthy.
CPU ~50% at idle	MISS (all 7 hosts)	Every host idles at 0-5% busy. Under-utilized, not over-loaded.

Finding	Severity	§
D2 carries 17 VMs — the worst IOPS hotspot in the fleet	High	5
S1 is NOT empty — it carries 16 sectestbed VMs via local-nonprod (previous audit was wrong)	High	4, 5
tsys6 S1 NFS export typo (S21) — FIXED by operator 2026-07-29; all 8 stores active	Resolved	6
tsys9 missing S1 in storage.cfg — FIXED by operator 2026-07-29; S1 active	Resolved	6
local-nonprod on tsys5 = the S1 physical disk (dir storage, not a separate disk)	Topology	4
tsys5 is single-CPU (CPU2 socket unpopulated; prior "dual E5620" was wrong)	Medium	9
cnode4 (601) stopped, cnode5 (706) deleted (user action this session)	Resolved	10

---

2. Host capacity (live, 2026-07-29 ~05:40 CDT)

Host	CPU (era)	Threads	RAM	Running VMs	Alloc RAM	Alloc%	Actual%	Swap used	CPU busy	Load
tsys1	i7-4770 Haswell (2013)	8	31 GB	11	36 GB	115%	73%	0	4%	1.1
tsys3	Xeon E3-1535M v5 Skylake (2015)	8	31 GB	1	28 GB	90%	10%	0	0%	0.3
tsys4	Xeon E3-1246 v3 Haswell (2013)	8	16 GB	1	2 GB	13%	14%	1.1 GB	0%	1.3
tsys5	Xeon E5620 Westmere (2010) single CPU	8	94 GB	17	126 GB	134%	37%	0	5%	4.0
tsys6	2× Xeon E5530 Nehalem (2009)	16	126 GB	2	44 GB	35%	4%	0	0%	0.6
tsys7	2× Xeon E5-2630 v2 Ivy Bridge (2013)	24	189 GB	5	52 GB	28%	7%	0	1%	0.9
tsys9	i5-10500 Comet Lake (2020)	12	23 GB	6	30 GB	129%	62%	0	1%	0.3

Changes since first collection (~05:09): cnode4 (601) stopped on tsys6, cnode5 (706) deleted from tsys7. No host migrations performed yet.

---

3. Storage topology (CORRECTED — read from live configs)

storage.cfg is per-host, not cluster-wide. Each Proxmox node defines its own storage entries. The same physical disk can appear under different storage IDs on different hosts. This was the source of the prior audit's errors.

3.1 Physical disk → Proxmox storage ID mapping

Physical disk	Server	NFS export	NFS ID (clients)	Local ID (on tsys5)	Type
WDC Red 3TB	tsys4	/mnt/tsys4/D2	D2	—	HDD
SK hynix SC300 512GB	tsys4	/mnt/tsys4/D3	D3	—	SSD (USB)
Hitachi 2TB	tsys4	/mnt/tsys4/D5	D5	—	HDD
(disk1)	tsys4	/mnt/tsys4/D1	(not in most storage.cfg)	—	unknown
(disk7)	tsys4	/mnt/tsys4/D7	(not in most storage.cfg)	—	unknown
Seagate 916GB	tsys5	/mnt/pfv-tsys5/S1	S1	local-nonprod	HDD
Seagate 916GB	tsys5	/mnt/pfv-tsys5/S2	S2	S2 (NFS client)	HDD
Seagate 916GB	tsys5	/mnt/pfv-tsys5/S3	S3	S3 (NFS client)	HDD
Toshiba 458GB	tsys5	/mnt/pfv-tsys5/S4	S4	S4 (NFS client)	HDD
Samsung 860 PRO 234GB	tsys5	/mnt/pfv-tsys5/ssd	T5-SSD	SSD (dir, local-only)	SSD

3.2 Critical correction: local-nonprod = S1

On tsys5, the storage entry is:

dir: local-nonprod
    path /mnt/pfv-tsys5/S1

local-nonprod is NOT a separate disk. It is the S1 Seagate 916GB HDD, accessed locally as a directory storage instead of via NFS. The 16 sectestbed and preprod VMs "on local-nonprod" are physically on S1. They appear locked to tsys5 because Proxmox treats local-nonprod (dir type) as a different storage than S1 (NFS type), even though they point to the same directory.

Implication: S1 has 16 VMs of IOPS load, not 0. The prior audit's plan to move production VMs onto S1 would have stacked them on top of 16 sandboxes.

3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday)

Tier	Spindles	Speed	Current use
NFS-HDD (tsys4)	D2, D5	80-120 MB/s	Production VMs
NFS-HDD (tsys5)	S1, S2, S3, S4	80-120 MB/s	Production + sectestbed VMs
Local HDD (tsys5)	local-nonprod (=S1)	80-120 MB/s	Sectestbed suite (locked to tsys5)
SSD (tsys4 USB)	D3	unreliable (USB 2.0)	Empty — do not use until Friday
SSD (tsys5 local)	SSD dir (=Samsung 860 PRO)	200-400 MB/s	ultix-streaming only

---

4. Per-spindle IOPS load (CORRECTED)

Spindle	Server	NFS-visible VMs	Local-only VMs	Total physical load
D2 (3TB)	tsys4	17	0	17 — worst hotspot
D5 (2TB)	tsys4	3	0	3
S1 (916GB)	tsys5	0	16 (via local-nonprod)	16 — second worst
S2 (916GB)	tsys5	4	0	4
S3 (916GB)	tsys5	3	0	3
S4 (458GB)	tsys5	1	0	1 — most spare capacity

D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.

The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI pipeline, test suite), S1 becomes a bottleneck. Do not add production VMs to S1. Target S2, S3, S4, and D5 for IOPS spreading.

---

5. Storage bugs and config gaps found

5.1 BUG: tsys6 S1 export path typo (S21) — RESOLVED ✓

Fixed by operator 2026-07-29. The export path was corrected from S21 to S1. Verified: pvesm status on tsys6 shows all 8 NFS stores active, including S1 (96 GB, 10.3% used by the sectestbed VMs visible via NFS).

5.2 GAP: tsys9 missing S1 in storage.cfg — RESOLVED ✓

Fixed by operator 2026-07-29. S1 entry added to tsys9's storage.cfg. Verified: all 8 NFS stores active on tsys9, S1 listing identical to tsys6.

5.3 ORPHANED: D1 and D7 exports on tsys4

tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg.

• D1 holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped templates only, no running VMs.
• D7 holds a disk for DellOpenManageEnterprise (500) — stopped.

These aren't causing problems but are underdocumented. Not actionable now.

---

6. Current VM inventory (host + disk, post-user-changes)

cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from start of session. Disk = NFS export or local storage holding the primary qcow2.

tsys1 — Infrastructure (11 running)

VMID	Name	vCPU	RAM	Disk	Server
100	pfv-bms (HomeAssistant)	2	4 GB	D2	tsys4
101	tsys-ca	2	2 GB	D2	tsys4
102	pfv-k8s-wnode-tsys1	4	4 GB	S3	tsys5
103	pfv-netinfra-01	2	2 GB	S2	tsys5
104	tsys-librenms	2	2 GB	D2	tsys4
105	tsys-proxmox-datacenter	2	2 GB	D2	tsys4
106	pfv-k8s-cnode3	2	4 GB	D5	tsys4
108	tsys-ucs-01	4	8 GB	D2	tsys4
109	tailscale-router	2	2 GB	D2	tsys4
114	kali-tsys	2	2 GB	D2	tsys4
117	tsys-secure-workbench	2	4 GB	D2	tsys4

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM	Disk	Server
313	pfv-k8s-wnode-tsys3	8	28 GB	D5	tsys4

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM	Disk	Server
400	pfv-proxmox-backup-server	2	2 GB	local-lvm	LOCAL

tsys5 — Storage + sandboxes (17 running)

VMID	Name	vCPU	RAM	Disk	Server	Migratable?
509	pfv-k8s-wnode-tsys5	8	32 GB	D2	tsys4	Yes (NFS)
5111	ultix-streaming	4	9 GB	SSD (local dir)	LOCAL	No (local)
5112	ultix-offstage	4	6 GB	local-lvm	LOCAL	No (local)
5101	sectestbed-siem	4	10 GB	local-nonprod (=S1)	LOCAL*	No (dir storage)
53100	tsys-preprod-awx	4	9 GB	local-nonprod (=S1)	LOCAL*	No
53101	tsys-preprod-siem	4	12 GB	local-nonprod (=S1)	LOCAL*	No
53102	tsys-preprod-rancherplatform	4	8 GB	local-nonprod (=S1)	LOCAL*	No
×10	sectestbed suite	4 ea	4 GB ea	local-nonprod (=S1)	LOCAL*	No

*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk but accessed via dir storage, not NFS — Proxmox will not migrate them. To unlock, move disk from local-nonprod → S1 (same physical disk, changes the storage ID to the NFS-shared one). Low priority — these are idle sandboxes.

tsys6 — RackRental-ready (2 running, 1 stopped)

VMID	Name	vCPU	RAM	Disk	Server	Status
100	pfv-k8s-wnode-tsys6	4	32 GB	S4	tsys5	running
600	tsys-awx	4	12 GB	D2	tsys4	running
601	pfv-k8s-cnode4	4	4 GB	D2	tsys4	STOPPED

tsys7 — RackRental-ready (5 running)

VMID	Name	vCPU	RAM	Disk	Server
701	pfv-k8s-wnode-tsys7	4	32 GB	S3	tsys5
702	hfnoc-uisp	4	8 GB	D2	tsys4
703	rr-middleware	2	2 GB	D2	tsys4
704	TCTC	4	6 GB	D2	tsys4
705	pfv-k8s-cnode2	4	4 GB	D2	tsys4

tsys9 — Infra + control plane (6 running)

VMID	Name	vCPU	RAM	Disk	Server
901	tsys-siem	2	8 GB	D2	tsys4
902	tsys-ucs-02	4	8 GB	S2	tsys5
903	kali-rd	2	2 GB	D5	tsys4
904	pfv-netinfra-02	2	4 GB	D2	tsys4
905	pfv-k8s-wnode-tsys9	4	4 GB	S2	tsys5
906	pfv-k8s-cnode1	2	4 GB	S3	tsys5

k3s control plane (ground truth via kubectl)

3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1). All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. No wnodes joined yet.

---

7. Workload placement model

Workload family	Needs	Best hosts	Why
ETL / HPC / cross-compile	High IPC + fast storage	tsys9, tsys3, tsys2(in)	Newest CPUs + local SSD/NVMe
RackRental.net	Massive RAM, CPU-light	tsys7, tsys6	189/126 GB RAM; weak CPU fine for idle labs
Infra + k8s control plane	Always-on, low resource	tsys1, tsys9	Efficient enough; cnodes spread for etcd diversity

---

8. Answers to operator's questions

8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?"

tsys9 — YES. Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB. After moving tsys-siem (8 GB) to tsys6, grow to 6c/8-16 GB.

tsys1 — NO. The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode.

8.2 "Are tsys6/7 better for RackRental.net?"

Yes. RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s) — deferred to Friday + StorageClass work.

8.3 "How should I deploy the DDR3?"

Verify type first. tsys5/6/7 require RDIMM ECC (consumer UDIMM won't POST). tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB). Most likely only useful for tsys4 16→32 GB. The real RAM lever is free VM migration, not sticks. See CAPACITY-GOALS.md §5.

---

9. Hardware corrections

tsys5 is single-CPU. dmidecode: CPU2: Status Unpopulated. The T7500 has one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch — useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity and activate the CPU2 DIMM branch.

tsys4 is using ~1.1 GB of swap. Only host with active swap. Minor, but monitor after the planned RAM upgrade.

---

10. Target end-state inventory (RAM/CPU + HDD IOPS balanced)

All disks stay on HDD (SSD/NVMe deferred to Friday). Goals: (a) relieve tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4 (currently 4/3/1), (d) avoid S1 (already has 16 local VMs), (e) keep HA pairs split across storage servers.

tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)

VMID	Name	vCPU	RAM	Disk	Server	Change
100	pfv-bms	2	4 GB	D2	tsys4	none
101	tsys-ca	2	2 GB	D2	tsys4	none
102	pfv-k8s-wnode-tsys1	4	4 GB	S3	tsys5	none
103	pfv-netinfra-01	2	2 GB	S2	tsys5	none
106	pfv-k8s-cnode3	2	4 GB	D5	tsys4	none
108	tsys-ucs-01	4	8 GB	D2	tsys4	none

tsys3 — Compute (1 running)

VMID	Name	vCPU	RAM	Disk	Server	Change
313	pfv-k8s-wnode-tsys3	8	28 GB	D5	tsys4	none

tsys4 — Storage (1 running)

VMID	Name	vCPU	RAM	Disk	Change
400	pfv-proxmox-backup-server	2	2 GB	local-lvm	none

tsys5 — Storage + sandboxes (17 running, unchanged)

All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay locked (idle sandboxes, low priority).

tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)

VMID	Name	vCPU	RAM	Disk	Server	Change
100	pfv-k8s-wnode-tsys6	4	32 GB	S4	tsys5	none
600	tsys-awx	4	12 GB	D5	tsys4	disk D2→D5
901	tsys-siem	2	8 GB	S2	tsys5	host tsys9→tsys6, disk D2→S2
601	cnode4	—	—	—	—	DELETE (stopped, stale)

tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)

VMID	Name	vCPU	RAM	Disk	Server	Change
701	pfv-k8s-wnode-tsys7	4	32 GB	S3	tsys5	none
702	hfnoc-uisp	4	8 GB	S4	tsys5	disk D2→S4
703	rr-middleware	2	2 GB	S4	tsys5	disk D2→S4
704	TCTC	4	6 GB	S3	tsys5	disk D2→S3
705	pfv-k8s-cnode2	4	4 GB	S2	tsys5	disk D2→S2
104	tsys-librenms	2	2 GB	S3	tsys5	host tsys1→tsys7, disk D2→S3
105	tsys-proxmox-datacenter	2	2 GB	S4	tsys5	host tsys1→tsys7, disk D2→S4
109	tailscale-router	2	2 GB	D2	tsys4	host tsys1→tsys7 (disk stays D2)
114	kali-tsys	2	2 GB	S2	tsys5	host tsys1→tsys7, disk D2→S2
117	tsys-secure-workbench	2	4 GB	S3	tsys5	host tsys1→tsys7, disk D2→S3

tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%)

VMID	Name	vCPU	RAM	Disk	Server	Change
902	tsys-ucs-02	4	8 GB	S2	tsys5	none
903	kali-rd	2	2 GB	D5	tsys4	none
904	pfv-netinfra-02	2	4 GB	D2	tsys4	none
905	pfv-k8s-wnode-tsys9	6	8 GB	S2	tsys5	grow 4c/4G→6c/8G
906	pfv-k8s-cnode1	2	4 GB	S3	tsys5	none
901	tsys-siem	—	—	—	—	move→tsys6

End-state per-spindle IOPS load

Spindle	Server	# VMs	VMs on it
D2 (3TB)	tsys4	5	bms, ca, ucs01, tailscale, netinfra02
D5 (2TB)	tsys4	4	cnode3, wnode3, awx, kali-rd
S1 (916GB)	tsys5	16 (unchanged)	sectestbed suite (local-nonprod, idle)
S2 (916GB)	tsys5	7	netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm*
S3 (916GB)	tsys5	7	wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench
S4 (458GB)	tsys5	5	wnode5, wnode6, rr-middleware, uisp*...

S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16 sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7 production VMs each is fine (they're 916 GB disks with light per-VM I/O).

D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.

HA pairs stay split:

• netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓
• ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓

---

11. Action checklist (do in this order)

Phase 0 — fix storage config bugs first — DONE ✓

• Fix tsys6 S1 typo — operator fixed 2026-07-29 (verified: 8/8 stores active)
• Add S1 to tsys9 storage.cfg — operator fixed 2026-07-29 (verified)

Phase 1 — cleanup

• Delete cnode4 (VMID 601) on tsys6 — stopped, stale, not in cluster
• Confirm cnode5 (706) gone — already deleted by operator ✓

Phase 2 — disk migrations off D2 (IOPS rebalancing)

Do disk moves on VMs that are staying on their current host first.

• wnode-tsys5 (509): disk D2 → S4 (host stays tsys5)
• tsys-awx (600): disk D2 → D5 (host stays tsys6)
• hfnoc-uisp (702): disk D2 → S4 (host stays tsys7)
• rr-middleware (703): disk D2 → S4 (host stays tsys7)
• TCTC (704): disk D2 → S3 (host stays tsys7)
• cnode2 (705): disk D2 → S2 (host stays tsys7)

Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs)

• tsys-librenms (104): host tsys1 → tsys7, disk D2 → S3
• tsys-proxmox-datacenter (105): host tsys1 → tsys7, disk D2 → S4
• tailscale-router (109): host tsys1 → tsys7 (disk stays D2)
• kali-tsys (114): host tsys1 → tsys7, disk D2 → S2
• tsys-secure-workbench (117): host tsys1 → tsys7, disk D2 → S3
• tsys-siem (901): host tsys9 → tsys6, disk D2 → S2

Phase 4 — grow the compute wnode

• wnode-tsys9 (905): resize 4c/4GB → 6c/8GB (stop/start required)

Phase 5 — join workers to k3s

• Join wnodes via k8s/join-servers.sh
• Label: workload=compute (tsys3/9), workload=rackrental (tsys6/7), workload=light (tsys1/5)

Verification after each phase

• qm list per host — confirm VM count + status
• cat /proc/meminfo — confirm actual RAM ≤ 80%
• kubectl top nodes — confirm cnodes healthy
• ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2' — confirm D2 frees up
• ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4' — confirm fills

---

12. Future: local storage as ephemeral scratch

Codified as standing policy in CAPACITY-GOALS.md §6.

The fleet has substantial unused local storage IOPS on several hosts. OS drives stay on NFS (no per-node disk redundancy — see policy), but local storage is ideal for ephemeral scratch (container image cache, build artifacts, emptyDir volumes) where data loss on node failure is acceptable.

Per-host opportunity:

Host	Local	Available	Scratch-suitable?
tsys3	NVMe (PM961)	349 GB	Yes — best existing
tsys9	SSD (PNY CS900)	136 GB	Yes
tsys2	NVMe + SSD	512 GB + 1 TB	Yes (incoming)
tsys5	HDD (Hitachi)	~1.7 TB	Yes (bulk)
tsys6	USB 2.0 HDD	1.7 TB	NO (30 MB/s)
tsys7	USB 2.0 HDD	1.7 TB	NO (30 MB/s)

The tsys6/7 problem: they are the RackRental hosts (most in need of fast image pulls) but their local storage is USB 2.0 — slower than NFS. The real fix is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5 empty SATA ports). Deferred to a physical-hardware session.

Implementation: post-Friday, once SSD/NVMe tiers are installed and the k8s StorageClass design is finalized. Wnodes with local storage get a /scratch mount for ephemeral data; persistent data stays on NFS. See CAPACITY-GOALS.md §6.4 for the StorageClass mapping.