# Capacity & Sizing Goals **Date:** 2026-07-29 **Status:** Authoritative. These are the operator's standing targets. Every fleet audit and VM-placement decision is evaluated against them. > Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current > placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report). --- ## 1. Memory target: 80% steady-state **Rule:** Keep each node at or below **80% physical RAM utilization at steady state.** Occasional bursts above 80% are acceptable (that is what swap is for); the *sustained* idle-to-average load must sit at ~80% or below. **Two metrics must both be tracked:** | Metric | Definition | Target | |--------|-----------|--------| | **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) | | **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible | **Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses 72% actual). Actual usage is the steady-state truth; allocation is the worst-case ceiling if every VM suddenly fills its reservation. A host is "healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a balloon-deflate storm could OOM it. **Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use (>~1 GB on a steady basis) means the host is over-committed and VMs should migrate off. ## 2. CPU target: ~50% utilization at idle **Rule:** Ideally, nodes sit around **50% CPU utilization at idle.** This is a *utilization* goal, not a headroom goal. The fleet is **CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle silicon is wasted investment and (for the PowerEdge hosts) wasted electricity. The intent is to **pack workloads** so the baseline load consumes roughly half the compute, reserving the other half for bursts. **Practical implications:** - The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be doing useful work at ~50% or be candidates to power down / solar-gate when idle. Running them at 0-6% wastes power per core. - The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run idle and are the natural home for always-on control-plane + infra. - vCPU overcommit is fine for idle workloads; the 50% target is about *actual* utilization, not vCPU ratios. ## 3. Workload placement model The fleet serves three distinct workload families. Placement should follow CPU quality + storage speed + RAM size, in that order of relevance per family. ### 3.1 ETL / HPC / cross-compile (CPU + storage bound) Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines. **Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate (8-32 GB per job). | Host | CPU | Storage | Fit | |------|-----|---------|-----| | **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** | | **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** | | tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) | These three hosts have the newest CPUs and the only fast local storage. Their wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and labeled `workload=compute`. ### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light) Business idea: rent out ephemeral network labs (containerlab topologies) on demand. Many containers / network namespaces, short-lived, mostly idle once up. **Needs massive RAM + fast container-image pulls. CPU quality is secondary** (labs are idle between keystrokes). | Host | CPU | RAM | Fit | |------|-----|-----|-----| | **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** | | **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** | **Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be labeled `workload=rackrental`. **Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU problem, and is deferred to the post-Friday storage-class work. RAM-wise they are ready today. ### 3.3 Infrastructure + k8s control plane (always-on, low resource) netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight, must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for etcd diversity). --- ## 4. Host role summary (target) | Host | CPU era | RAM | Primary role | Wnode class | |------|---------|-----|--------------|-------------| | tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) | | tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute | | tsys3 | Skylake 2015 | 32 GB | Compute | compute | | tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — | | tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) | | tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental | | tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental | | tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) | ## 5. DDR3 memory policy **Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with 117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned, not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9 over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing free RAM, not by installing sticks. Compatibility is decisive and is documented per-host in [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require **Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM (Patriot / most Kingston) fits essentially nowhere useful here.