PFVCluster/proxmox/docs/CAPACITY-GOALS.md

Capacity & Sizing Goals

Date: 2026-07-29 Status: Authoritative. These are the operator's standing targets. Every fleet audit and VM-placement decision is evaluated against them.

Companion to AUDIT-2026-07-29.md (the current placement analysis) and PROJECT.md (the fleet hardware report).

---

1. Memory target: 80% steady-state

Rule: Keep each node at or below 80% physical RAM utilization at steady state. Occasional bursts above 80% are acceptable (that is what swap is for); the sustained idle-to-average load must sit at ~80% or below.

Two metrics must both be tracked:

Metric	Definition	Target
Actual usage	MemTotal - MemAvailable from /proc/meminfo	≤ 80% steady state (the real rule)
Allocation commitment	sum of running-VM memory: vs host RAM	planning ceiling — keep < 100% where possible

Why both: KSM (same-page merging) and virtio-balloon mean actual usage is often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses 72% actual). Actual usage is the steady-state truth; allocation is the worst-case ceiling if every VM suddenly fills its reservation. A host is "healthy" when actual ≤ 80% and allocation isn't so far over 100% that a balloon-deflate storm could OOM it.

Tolerance: brief actual spikes to 90-100% are fine. Persistent swap use (>~1 GB on a steady basis) means the host is over-committed and VMs should migrate off.

2. CPU target: ~50% utilization at idle

Rule: Ideally, nodes sit around 50% CPU utilization at idle.

This is a utilization goal, not a headroom goal. The fleet is CPU-underutilized (most hosts idle at 0-6%). On a shoestring budget, idle silicon is wasted investment and (for the PowerEdge hosts) wasted electricity. The intent is to pack workloads so the baseline load consumes roughly half the compute, reserving the other half for bursts.

Practical implications:

• The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be doing useful work at ~50% or be candidates to power down / solar-gate when idle. Running them at 0-6% wastes power per core.
• The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run idle and are the natural home for always-on control-plane + infra.
• vCPU overcommit is fine for idle workloads; the 50% target is about actual utilization, not vCPU ratios.

3. Workload placement model

The fleet serves three distinct workload families. Placement should follow CPU quality + storage speed + RAM size, in that order of relevance per family.

3.1 ETL / HPC / cross-compile (CPU + storage bound)

Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines. Needs high IPC (modern cores) + fast storage (SSD/NVMe). RAM is moderate (8-32 GB per job).

Host	CPU	Storage	Fit
tsys9	i5-10500 Comet Lake (2020) — best IPC in fleet	local SSD (PNY CS900)	Primary compute node
tsys3	Xeon E3-1535M v5 Skylake (2015)	local NVMe (Samsung PM961)	Fast compute node
tsys2 (incoming)	i7-7820HQ Kaby Lake (2017)	NVMe 960 PRO + SSD 850 EVO	Fast compute node (post-rebuild)

These three hosts have the newest CPUs and the only fast local storage. Their wnodes should be the compute-tier workers, sized larger (more cores/RAM) and labeled workload=compute.

3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)

Business idea: rent out ephemeral network labs (containerlab topologies) on demand. Many containers / network namespaces, short-lived, mostly idle once up. Needs massive RAM + fast container-image pulls. CPU quality is secondary (labs are idle between keystrokes).

Host	CPU	RAM	Fit
tsys7	2× E5-2630 v2 Ivy Bridge (2013), 24 threads	189 GB	Primary RackRental host
tsys6	2× E5530 Nehalem (2009), 16 threads	126 GB	Secondary RackRental host

Verdict: tsys6/7 are the correct RackRental hosts. Their weak CPUs are irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be labeled workload=rackrental.

Caveat (deferred): their local-lvm is USB 2.0 portable HDD (~30 MB/s), which is terrible for container image pulls. This is a disk problem, not a RAM/CPU problem, and is deferred to the post-Friday storage-class work. RAM-wise they are ready today.

3.3 Infrastructure + k8s control plane (always-on, low resource)

netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight, must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for etcd diversity).

---

4. Host role summary (target)

Host	CPU era	RAM	Primary role	Wnode class
tsys1	Haswell 2013	32 GB (maxed)	Infra + control plane	light (4c/4G)
tsys2	Kaby Lake 2017	32 GB	Compute (incoming)	compute
tsys3	Skylake 2015	32 GB	Compute	compute
tsys4	Haswell 2013	16 GB	Storage (NFS + PBS)	—
tsys5	Westmere 2010	96 GB	Storage + fast-tier hub	compute (NVMe, post-Friday)
tsys6	Nehalem 2009	126 GB	RackRental	rackrental
tsys7	Ivy Bridge 2013	189 GB	RackRental	rackrental
tsys9	Comet Lake 2020	24 GB	Infra + control plane + compute	compute (growable)

5. DDR3 memory policy

Do not blindly add RAM to tsys6/7. They idle at 6%/9% actual usage with 117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned, not starved). The fleet's RAM pressure is on the infra hosts (tsys1/5/9 over-allocated), and that is solved by migrating VMs to tsys6/7's existing free RAM, not by installing sticks.

Compatibility is decisive and is documented per-host in AUDIT-2026-07-29.md §5. Short version: tsys5/6/7 require Registered ECC (RDIMM); only RDIMM sticks will POST. Consumer UDIMM (Patriot / most Kingston) fits essentially nowhere useful here.

---

6. Local storage policy: ephemeral scratch, not OS drives

Principle: Local disk storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral workloads where data loss on node failure is acceptable by definition.

6.1 Why OS drives stay on NFS

Individual Proxmox nodes have no local disk redundancy (single disk, no RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone. Redundancy is instead achieved at the storage-server + application layer:

• -01/-02 HA pairs (netinfra, UCS): each half on a different NFS server
• k8s spreading: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no single disk or server failure takes down the whole cluster
• PBS backups: all VMs are backed up to Proxmox Backup Server regardless of disk tier

This is deliberate. Moving OS drives to local storage would trade NFS network latency for single-disk-failure risk. Not worth it for persistent workloads.

6.2 What local storage IS good for: ephemeral scratch

Workloads that produce or consume temporary data that can be discarded on node failure. These are exactly the workloads where NFS latency hurts most and local IOPS help most:

Use case	Why local scratch helps	Examples
Container image pulls	Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow	RackRental labs, any pod startup
Build artifacts	Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD	ETL, cross-compile, CI pipelines
Temporary processing	Intermediate data that doesn't need to survive a reboot	Weather/GIS transform scratch, log rotation
kubelet ephemeral storage	emptyDir volumes, container writable layers	All k8s pods

6.3 Per-host local storage inventory and suitability

Host	Local storage	Type	Available	Scratch-suitable?	Notes
tsys2	NVMe 960 PRO + SSD 850 EVO	NVMe + SSD	512 GB + 1 TB	Yes — best in fleet	Incoming (post-rebuild). No NFS needed at all
tsys3	local-lvm (Samsung PM961)	NVMe	349 GB	Yes — excellent	Fastest existing local storage. 0% used
tsys9	local-lvm (PNY CS900)	SSD	136 GB	Yes — excellent	Low latency. 0% used
tsys5	local-lvm (Hitachi)	HDD	~1.7 TB	Yes (bulk)	Large capacity, HDD speed. Storage host — scratch is secondary
tsys1	local-lvm	HDD	~90 GB	Marginal (small)	Barely worth it; tsys1 is infra-loaded anyway
tsys4	local-lvm	HDD	~62 GB	No	PBS host; minimal free space
tsys6	local-lvm (WD My Passport)	USB 2.0 HDD	1.7 TB	NO — 30 MB/s	Slower than NFS. Do not use for VMs
tsys7	local-lvm (WD portable)	USB 2.0 HDD	1.7 TB	NO — 30 MB/s	Slower than NFS. Do not use for VMs

tsys6/7 are the hard problem. They are the RackRental hosts — exactly the workload that benefits most from local scratch (fast container image pulls) — but their local storage is USB 2.0 portable HDD, which is slower than NFS-over-HDD. Options for the future:

1. Install internal drives (tsys6 has a disabled SAS controller; tsys7 has 5 empty SATA ports). This is the real fix — gives both RackRental hosts proper local scratch. Deferred to a physical-hardware session.
2. Accept NFS for image pulls and use local-lvm only for swap/temp that tolerates 30 MB/s. Functional but sluggish for lab spin-up.

6.4 How this maps to k8s StorageClasses (future, post-Friday)

The scratch tier maps to two k8s concepts:

StorageClass: local-fast
  provisioner: local-path
  where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
  use: emptyDir replacement, build artifacts, container runtime cache
  redundancy: none — ephemeral, dies with the pod

StorageClass: nfs-hdd  (existing concept)
  provisioner: nfs-subdir-external
  where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
  use: persistent bulk data, VM OS drives, anything that must survive restarts
  redundancy: cross-server spreading + PBS backup

A wnode with local scratch gets a second disk (or a partition on local-lvm) mounted as /scratch. ETL jobs write intermediate data to /scratch (local-fast) and final output to /data (nfs-hdd). Pods that need ephemeral fast storage request local-fast PVCs; pods that need persistence request nfs-hdd.

This is forward-looking. The concrete implementation (formatting local-lvm, adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes) happens after the Friday maintenance window when the SSD/NVMe tiers are physically installed and the storage architecture is finalized.