# Capacity & Sizing Goals

**Date:** 2026-07-29
**Status:** Authoritative. These are the operator's standing targets. Every
fleet audit and VM-placement decision is evaluated against them.

> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).

---

## 1. Memory target: 80% steady-state

**Rule:** Keep each node at or below **80% physical RAM utilization at steady
state.** Occasional bursts above 80% are acceptable (that is what swap is for);
the *sustained* idle-to-average load must sit at ~80% or below.

**Two metrics must both be tracked:**

| Metric | Definition | Target |
|--------|-----------|--------|
| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |

**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
72% actual). Actual usage is the steady-state truth; allocation is the
worst-case ceiling if every VM suddenly fills its reservation. A host is
"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
balloon-deflate storm could OOM it.

**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
(>~1 GB on a steady basis) means the host is over-committed and VMs should
migrate off.

## 2. CPU target: ~50% utilization at idle

**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**

This is a *utilization* goal, not a headroom goal. The fleet is
**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
The intent is to **pack workloads** so the baseline load consumes roughly half
the compute, reserving the other half for bursts.

**Practical implications:**

- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
  doing useful work at ~50% or be candidates to power down / solar-gate when
  idle. Running them at 0-6% wastes power per core.
- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
  idle and are the natural home for always-on control-plane + infra.
- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
  utilization, not vCPU ratios.

## 3. Workload placement model

The fleet serves three distinct workload families. Placement should follow CPU
quality + storage speed + RAM size, in that order of relevance per family.

### 3.1 ETL / HPC / cross-compile (CPU + storage bound)

Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
(8-32 GB per job).

| Host | CPU | Storage | Fit |
|------|-----|---------|-----|
| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |

These three hosts have the newest CPUs and the only fast local storage. Their
wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
labeled `workload=compute`.

### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)

Business idea: rent out ephemeral network labs (containerlab topologies) on
demand. Many containers / network namespaces, short-lived, mostly idle once up.
**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
(labs are idle between keystrokes).

| Host | CPU | RAM | Fit |
|------|-----|-----|-----|
| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |

**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
labeled `workload=rackrental`.

**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
ready today.

### 3.3 Infrastructure + k8s control plane (always-on, low resource)

netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
etcd diversity).

---

## 4. Host role summary (target)

| Host | CPU era | RAM | Primary role | Wnode class |
|------|---------|-----|--------------|-------------|
| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |

## 5. DDR3 memory policy

**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
free RAM, not by installing sticks.

Compatibility is decisive and is documented per-host in
[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
(Patriot / most Kingston) fits essentially nowhere useful here.

---

## 6. Local storage policy: ephemeral scratch, not OS drives

**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
workloads where data loss on node failure is acceptable by definition.

### 6.1 Why OS drives stay on NFS

Individual Proxmox nodes have **no local disk redundancy** (single disk, no
RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
Redundancy is instead achieved at the **storage-server + application layer**:

- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
  single disk or server failure takes down the whole cluster
- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
  disk tier

This is deliberate. Moving OS drives to local storage would trade NFS network
latency for single-disk-failure risk. Not worth it for persistent workloads.

### 6.2 What local storage IS good for: ephemeral scratch

Workloads that produce or consume temporary data that can be discarded on node
failure. These are exactly the workloads where NFS latency hurts most and local
IOPS help most:

| Use case | Why local scratch helps | Examples |
|----------|------------------------|---------|
| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |

### 6.3 Per-host local storage inventory and suitability

| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
|------|--------------|------|-----------|-------------------|-------|
| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |

**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
workload that benefits most from local scratch (fast container image pulls) —
but their local storage is USB 2.0 portable HDD, which is slower than
NFS-over-HDD. Options for the future:

1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
   5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
   local scratch. Deferred to a physical-hardware session.
2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
   tolerates 30 MB/s. Functional but sluggish for lab spin-up.

### 6.4 How this maps to k8s StorageClasses (future, post-Friday)

The scratch tier maps to two k8s concepts:

```
StorageClass: local-fast
  provisioner: local-path
  where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
  use: emptyDir replacement, build artifacts, container runtime cache
  redundancy: none — ephemeral, dies with the pod

StorageClass: nfs-hdd  (existing concept)
  provisioner: nfs-subdir-external
  where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
  use: persistent bulk data, VM OS drives, anything that must survive restarts
  redundancy: cross-server spreading + PBS backup
```

A wnode with local scratch gets a second disk (or a partition on local-lvm)
mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
fast storage request `local-fast` PVCs; pods that need persistence request
`nfs-hdd`.

**This is forward-looking.** The concrete implementation (formatting local-lvm,
adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
happens after the Friday maintenance window when the SSD/NVMe tiers are
physically installed and the storage architecture is finalized.
