mrcharles
9c47d93a2f
Add CAPACITY-GOALS.md §6: local storage is a scratch tier for ephemeral workloads
(container image cache, build artifacts, emptyDir), NOT for VM OS drives (which
stay on NFS where redundancy is handled via 01/02 pairs + cross-disk spreading).
Documents per-host local storage suitability: tsys3/9/2 excellent (NVMe/SSD),
tsys5 usable (HDD bulk), tsys6/7 unusable (USB 2.0 at 30 MB/s). Includes the
future k8s StorageClass mapping (local-fast for scratch, nfs-hdd for persistent).
Adds forward-looking §12 to the audit referencing this policy.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
221 lines
11 KiB
Markdown
221 lines
11 KiB
Markdown
# Capacity & Sizing Goals
|
||
|
||
**Date:** 2026-07-29
|
||
**Status:** Authoritative. These are the operator's standing targets. Every
|
||
fleet audit and VM-placement decision is evaluated against them.
|
||
|
||
> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
|
||
> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).
|
||
|
||
---
|
||
|
||
## 1. Memory target: 80% steady-state
|
||
|
||
**Rule:** Keep each node at or below **80% physical RAM utilization at steady
|
||
state.** Occasional bursts above 80% are acceptable (that is what swap is for);
|
||
the *sustained* idle-to-average load must sit at ~80% or below.
|
||
|
||
**Two metrics must both be tracked:**
|
||
|
||
| Metric | Definition | Target |
|
||
|--------|-----------|--------|
|
||
| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
|
||
| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |
|
||
|
||
**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
|
||
often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
|
||
72% actual). Actual usage is the steady-state truth; allocation is the
|
||
worst-case ceiling if every VM suddenly fills its reservation. A host is
|
||
"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
|
||
balloon-deflate storm could OOM it.
|
||
|
||
**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
|
||
(>~1 GB on a steady basis) means the host is over-committed and VMs should
|
||
migrate off.
|
||
|
||
## 2. CPU target: ~50% utilization at idle
|
||
|
||
**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**
|
||
|
||
This is a *utilization* goal, not a headroom goal. The fleet is
|
||
**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
|
||
silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
|
||
The intent is to **pack workloads** so the baseline load consumes roughly half
|
||
the compute, reserving the other half for bursts.
|
||
|
||
**Practical implications:**
|
||
|
||
- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
|
||
doing useful work at ~50% or be candidates to power down / solar-gate when
|
||
idle. Running them at 0-6% wastes power per core.
|
||
- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
|
||
idle and are the natural home for always-on control-plane + infra.
|
||
- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
|
||
utilization, not vCPU ratios.
|
||
|
||
## 3. Workload placement model
|
||
|
||
The fleet serves three distinct workload families. Placement should follow CPU
|
||
quality + storage speed + RAM size, in that order of relevance per family.
|
||
|
||
### 3.1 ETL / HPC / cross-compile (CPU + storage bound)
|
||
|
||
Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
|
||
**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
|
||
(8-32 GB per job).
|
||
|
||
| Host | CPU | Storage | Fit |
|
||
|------|-----|---------|-----|
|
||
| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
|
||
| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
|
||
| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |
|
||
|
||
These three hosts have the newest CPUs and the only fast local storage. Their
|
||
wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
|
||
labeled `workload=compute`.
|
||
|
||
### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)
|
||
|
||
Business idea: rent out ephemeral network labs (containerlab topologies) on
|
||
demand. Many containers / network namespaces, short-lived, mostly idle once up.
|
||
**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
|
||
(labs are idle between keystrokes).
|
||
|
||
| Host | CPU | RAM | Fit |
|
||
|------|-----|-----|-----|
|
||
| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
|
||
| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |
|
||
|
||
**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
|
||
irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
|
||
labeled `workload=rackrental`.
|
||
|
||
**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
|
||
is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
|
||
problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
|
||
ready today.
|
||
|
||
### 3.3 Infrastructure + k8s control plane (always-on, low resource)
|
||
|
||
netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
|
||
must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
|
||
etcd diversity).
|
||
|
||
---
|
||
|
||
## 4. Host role summary (target)
|
||
|
||
| Host | CPU era | RAM | Primary role | Wnode class |
|
||
|------|---------|-----|--------------|-------------|
|
||
| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
|
||
| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
|
||
| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
|
||
| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
|
||
| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
|
||
| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
|
||
| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
|
||
| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |
|
||
|
||
## 5. DDR3 memory policy
|
||
|
||
**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
|
||
117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
|
||
not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
|
||
over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
|
||
free RAM, not by installing sticks.
|
||
|
||
Compatibility is decisive and is documented per-host in
|
||
[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
|
||
**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
|
||
(Patriot / most Kingston) fits essentially nowhere useful here.
|
||
|
||
---
|
||
|
||
## 6. Local storage policy: ephemeral scratch, not OS drives
|
||
|
||
**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
|
||
VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
|
||
across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
|
||
workloads where data loss on node failure is acceptable by definition.
|
||
|
||
### 6.1 Why OS drives stay on NFS
|
||
|
||
Individual Proxmox nodes have **no local disk redundancy** (single disk, no
|
||
RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
|
||
Redundancy is instead achieved at the **storage-server + application layer**:
|
||
|
||
- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
|
||
- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
|
||
single disk or server failure takes down the whole cluster
|
||
- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
|
||
disk tier
|
||
|
||
This is deliberate. Moving OS drives to local storage would trade NFS network
|
||
latency for single-disk-failure risk. Not worth it for persistent workloads.
|
||
|
||
### 6.2 What local storage IS good for: ephemeral scratch
|
||
|
||
Workloads that produce or consume temporary data that can be discarded on node
|
||
failure. These are exactly the workloads where NFS latency hurts most and local
|
||
IOPS help most:
|
||
|
||
| Use case | Why local scratch helps | Examples |
|
||
|----------|------------------------|---------|
|
||
| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
|
||
| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
|
||
| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
|
||
| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |
|
||
|
||
### 6.3 Per-host local storage inventory and suitability
|
||
|
||
| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
|
||
|------|--------------|------|-----------|-------------------|-------|
|
||
| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
|
||
| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
|
||
| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
|
||
| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
|
||
| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
|
||
| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
|
||
| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
|
||
| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
|
||
|
||
**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
|
||
workload that benefits most from local scratch (fast container image pulls) —
|
||
but their local storage is USB 2.0 portable HDD, which is slower than
|
||
NFS-over-HDD. Options for the future:
|
||
|
||
1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
|
||
5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
|
||
local scratch. Deferred to a physical-hardware session.
|
||
2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
|
||
tolerates 30 MB/s. Functional but sluggish for lab spin-up.
|
||
|
||
### 6.4 How this maps to k8s StorageClasses (future, post-Friday)
|
||
|
||
The scratch tier maps to two k8s concepts:
|
||
|
||
```
|
||
StorageClass: local-fast
|
||
provisioner: local-path
|
||
where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
|
||
use: emptyDir replacement, build artifacts, container runtime cache
|
||
redundancy: none — ephemeral, dies with the pod
|
||
|
||
StorageClass: nfs-hdd (existing concept)
|
||
provisioner: nfs-subdir-external
|
||
where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
|
||
use: persistent bulk data, VM OS drives, anything that must survive restarts
|
||
redundancy: cross-server spreading + PBS backup
|
||
```
|
||
|
||
A wnode with local scratch gets a second disk (or a partition on local-lvm)
|
||
mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
|
||
(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
|
||
fast storage request `local-fast` PVCs; pods that need persistence request
|
||
`nfs-hdd`.
|
||
|
||
**This is forward-looking.** The concrete implementation (formatting local-lvm,
|
||
adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
|
||
happens after the Friday maintenance window when the SSD/NVMe tiers are
|
||
physically installed and the storage architecture is finalized.
|