PFVCluster/proxmox/docs/CAPACITY-GOALS.md

Capacity & Sizing Goals

Date: 2026-07-29 Status: Authoritative. These are the operator's standing targets. Every fleet audit and VM-placement decision is evaluated against them.

Companion to AUDIT-2026-07-29.md (the current placement analysis) and PROJECT.md (the fleet hardware report).

---

1. Memory target: 80% steady-state

Rule: Keep each node at or below 80% physical RAM utilization at steady state. Occasional bursts above 80% are acceptable (that is what swap is for); the sustained idle-to-average load must sit at ~80% or below.

Two metrics must both be tracked:

Metric	Definition	Target
Actual usage	MemTotal - MemAvailable from /proc/meminfo	≤ 80% steady state (the real rule)
Allocation commitment	sum of running-VM memory: vs host RAM	planning ceiling — keep < 100% where possible

Why both: KSM (same-page merging) and virtio-balloon mean actual usage is often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses 72% actual). Actual usage is the steady-state truth; allocation is the worst-case ceiling if every VM suddenly fills its reservation. A host is "healthy" when actual ≤ 80% and allocation isn't so far over 100% that a balloon-deflate storm could OOM it.

Tolerance: brief actual spikes to 90-100% are fine. Persistent swap use (>~1 GB on a steady basis) means the host is over-committed and VMs should migrate off.

2. CPU target: ~50% utilization at idle

Rule: Ideally, nodes sit around 50% CPU utilization at idle.

This is a utilization goal, not a headroom goal. The fleet is CPU-underutilized (most hosts idle at 0-6%). On a shoestring budget, idle silicon is wasted investment and (for the PowerEdge hosts) wasted electricity. The intent is to pack workloads so the baseline load consumes roughly half the compute, reserving the other half for bursts.

Practical implications:

• The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be doing useful work at ~50% or be candidates to power down / solar-gate when idle. Running them at 0-6% wastes power per core.
• The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run idle and are the natural home for always-on control-plane + infra.
• vCPU overcommit is fine for idle workloads; the 50% target is about actual utilization, not vCPU ratios.

3. Workload placement model

The fleet serves three distinct workload families. Placement should follow CPU quality + storage speed + RAM size, in that order of relevance per family.

3.1 ETL / HPC / cross-compile (CPU + storage bound)

Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines. Needs high IPC (modern cores) + fast storage (SSD/NVMe). RAM is moderate (8-32 GB per job).

Host	CPU	Storage	Fit
tsys9	i5-10500 Comet Lake (2020) — best IPC in fleet	local SSD (PNY CS900)	Primary compute node
tsys3	Xeon E3-1535M v5 Skylake (2015)	local NVMe (Samsung PM961)	Fast compute node
tsys2 (incoming)	i7-7820HQ Kaby Lake (2017)	NVMe 960 PRO + SSD 850 EVO	Fast compute node (post-rebuild)

These three hosts have the newest CPUs and the only fast local storage. Their wnodes should be the compute-tier workers, sized larger (more cores/RAM) and labeled workload=compute.

3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)

Business idea: rent out ephemeral network labs (containerlab topologies) on demand. Many containers / network namespaces, short-lived, mostly idle once up. Needs massive RAM + fast container-image pulls. CPU quality is secondary (labs are idle between keystrokes).

Host	CPU	RAM	Fit
tsys7	2× E5-2630 v2 Ivy Bridge (2013), 24 threads	189 GB	Primary RackRental host
tsys6	2× E5530 Nehalem (2009), 16 threads	126 GB	Secondary RackRental host

Verdict: tsys6/7 are the correct RackRental hosts. Their weak CPUs are irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be labeled workload=rackrental.

Caveat (deferred): their local-lvm is USB 2.0 portable HDD (~30 MB/s), which is terrible for container image pulls. This is a disk problem, not a RAM/CPU problem, and is deferred to the post-Friday storage-class work. RAM-wise they are ready today.

3.3 Infrastructure + k8s control plane (always-on, low resource)

netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight, must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for etcd diversity).

---

4. Host role summary (target)

Host	CPU era	RAM	Primary role	Wnode class
tsys1	Haswell 2013	32 GB (maxed)	Infra + control plane	light (4c/4G)
tsys2	Kaby Lake 2017	32 GB	Compute (incoming)	compute
tsys3	Skylake 2015	32 GB	Compute	compute
tsys4	Haswell 2013	16 GB	Storage (NFS + PBS)	—
tsys5	Westmere 2010	96 GB	Storage + fast-tier hub	compute (NVMe, post-Friday)
tsys6	Nehalem 2009	126 GB	RackRental	rackrental
tsys7	Ivy Bridge 2013	189 GB	RackRental	rackrental
tsys9	Comet Lake 2020	24 GB	Infra + control plane + compute	compute (growable)

5. DDR3 memory policy

Do not blindly add RAM to tsys6/7. They idle at 6%/9% actual usage with 117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned, not starved). The fleet's RAM pressure is on the infra hosts (tsys1/5/9 over-allocated), and that is solved by migrating VMs to tsys6/7's existing free RAM, not by installing sticks.

Compatibility is decisive and is documented per-host in AUDIT-2026-07-29.md §5. Short version: tsys5/6/7 require Registered ECC (RDIMM); only RDIMM sticks will POST. Consumer UDIMM (Patriot / most Kingston) fits essentially nowhere useful here.