mrcharles
bfb2d44183
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
131 lines
6.0 KiB
Markdown
131 lines
6.0 KiB
Markdown
# Capacity & Sizing Goals
|
||
|
||
**Date:** 2026-07-29
|
||
**Status:** Authoritative. These are the operator's standing targets. Every
|
||
fleet audit and VM-placement decision is evaluated against them.
|
||
|
||
> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
|
||
> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).
|
||
|
||
---
|
||
|
||
## 1. Memory target: 80% steady-state
|
||
|
||
**Rule:** Keep each node at or below **80% physical RAM utilization at steady
|
||
state.** Occasional bursts above 80% are acceptable (that is what swap is for);
|
||
the *sustained* idle-to-average load must sit at ~80% or below.
|
||
|
||
**Two metrics must both be tracked:**
|
||
|
||
| Metric | Definition | Target |
|
||
|--------|-----------|--------|
|
||
| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
|
||
| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |
|
||
|
||
**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
|
||
often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
|
||
72% actual). Actual usage is the steady-state truth; allocation is the
|
||
worst-case ceiling if every VM suddenly fills its reservation. A host is
|
||
"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
|
||
balloon-deflate storm could OOM it.
|
||
|
||
**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
|
||
(>~1 GB on a steady basis) means the host is over-committed and VMs should
|
||
migrate off.
|
||
|
||
## 2. CPU target: ~50% utilization at idle
|
||
|
||
**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**
|
||
|
||
This is a *utilization* goal, not a headroom goal. The fleet is
|
||
**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
|
||
silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
|
||
The intent is to **pack workloads** so the baseline load consumes roughly half
|
||
the compute, reserving the other half for bursts.
|
||
|
||
**Practical implications:**
|
||
|
||
- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
|
||
doing useful work at ~50% or be candidates to power down / solar-gate when
|
||
idle. Running them at 0-6% wastes power per core.
|
||
- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
|
||
idle and are the natural home for always-on control-plane + infra.
|
||
- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
|
||
utilization, not vCPU ratios.
|
||
|
||
## 3. Workload placement model
|
||
|
||
The fleet serves three distinct workload families. Placement should follow CPU
|
||
quality + storage speed + RAM size, in that order of relevance per family.
|
||
|
||
### 3.1 ETL / HPC / cross-compile (CPU + storage bound)
|
||
|
||
Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
|
||
**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
|
||
(8-32 GB per job).
|
||
|
||
| Host | CPU | Storage | Fit |
|
||
|------|-----|---------|-----|
|
||
| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
|
||
| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
|
||
| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |
|
||
|
||
These three hosts have the newest CPUs and the only fast local storage. Their
|
||
wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
|
||
labeled `workload=compute`.
|
||
|
||
### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)
|
||
|
||
Business idea: rent out ephemeral network labs (containerlab topologies) on
|
||
demand. Many containers / network namespaces, short-lived, mostly idle once up.
|
||
**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
|
||
(labs are idle between keystrokes).
|
||
|
||
| Host | CPU | RAM | Fit |
|
||
|------|-----|-----|-----|
|
||
| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
|
||
| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |
|
||
|
||
**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
|
||
irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
|
||
labeled `workload=rackrental`.
|
||
|
||
**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
|
||
is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
|
||
problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
|
||
ready today.
|
||
|
||
### 3.3 Infrastructure + k8s control plane (always-on, low resource)
|
||
|
||
netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
|
||
must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
|
||
etcd diversity).
|
||
|
||
---
|
||
|
||
## 4. Host role summary (target)
|
||
|
||
| Host | CPU era | RAM | Primary role | Wnode class |
|
||
|------|---------|-----|--------------|-------------|
|
||
| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
|
||
| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
|
||
| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
|
||
| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
|
||
| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
|
||
| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
|
||
| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
|
||
| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |
|
||
|
||
## 5. DDR3 memory policy
|
||
|
||
**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
|
||
117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
|
||
not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
|
||
over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
|
||
free RAM, not by installing sticks.
|
||
|
||
Compatibility is decisive and is documented per-host in
|
||
[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
|
||
**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
|
||
(Patriot / most Kingston) fits essentially nowhere useful here.
|