docs: migrate all documentation to Discourse wiki topics
All knowledge docs (architecture, runbooks, references, audits, policies) have been migrated to community.turnsys.com as wiki topics in the VP TechOps category. Discourse is now the canonical source of truth for documentation; git edit history no longer serves as the doc changelog. 37 .md files converted to short pointers linking to their Discourse topics. AGENTS.md updated with new documentation workflow policy. Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
@@ -1,220 +1,10 @@
|
||||
# Capacity & Sizing Goals
|
||||
# proxmox/docs/CAPACITY-GOALS.md
|
||||
|
||||
**Date:** 2026-07-29
|
||||
**Status:** Authoritative. These are the operator's standing targets. Every
|
||||
fleet audit and VM-placement decision is evaluated against them.
|
||||
|
||||
> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
|
||||
> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).
|
||||
|
||||
---
|
||||
|
||||
## 1. Memory target: 80% steady-state
|
||||
|
||||
**Rule:** Keep each node at or below **80% physical RAM utilization at steady
|
||||
state.** Occasional bursts above 80% are acceptable (that is what swap is for);
|
||||
the *sustained* idle-to-average load must sit at ~80% or below.
|
||||
|
||||
**Two metrics must both be tracked:**
|
||||
|
||||
| Metric | Definition | Target |
|
||||
|--------|-----------|--------|
|
||||
| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
|
||||
| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |
|
||||
|
||||
**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
|
||||
often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
|
||||
72% actual). Actual usage is the steady-state truth; allocation is the
|
||||
worst-case ceiling if every VM suddenly fills its reservation. A host is
|
||||
"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
|
||||
balloon-deflate storm could OOM it.
|
||||
|
||||
**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
|
||||
(>~1 GB on a steady basis) means the host is over-committed and VMs should
|
||||
migrate off.
|
||||
|
||||
## 2. CPU target: ~50% utilization at idle
|
||||
|
||||
**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**
|
||||
|
||||
This is a *utilization* goal, not a headroom goal. The fleet is
|
||||
**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
|
||||
silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
|
||||
The intent is to **pack workloads** so the baseline load consumes roughly half
|
||||
the compute, reserving the other half for bursts.
|
||||
|
||||
**Practical implications:**
|
||||
|
||||
- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
|
||||
doing useful work at ~50% or be candidates to power down / solar-gate when
|
||||
idle. Running them at 0-6% wastes power per core.
|
||||
- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
|
||||
idle and are the natural home for always-on control-plane + infra.
|
||||
- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
|
||||
utilization, not vCPU ratios.
|
||||
|
||||
## 3. Workload placement model
|
||||
|
||||
The fleet serves three distinct workload families. Placement should follow CPU
|
||||
quality + storage speed + RAM size, in that order of relevance per family.
|
||||
|
||||
### 3.1 ETL / HPC / cross-compile (CPU + storage bound)
|
||||
|
||||
Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
|
||||
**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
|
||||
(8-32 GB per job).
|
||||
|
||||
| Host | CPU | Storage | Fit |
|
||||
|------|-----|---------|-----|
|
||||
| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
|
||||
| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
|
||||
| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |
|
||||
|
||||
These three hosts have the newest CPUs and the only fast local storage. Their
|
||||
wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
|
||||
labeled `workload=compute`.
|
||||
|
||||
### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)
|
||||
|
||||
Business idea: rent out ephemeral network labs (containerlab topologies) on
|
||||
demand. Many containers / network namespaces, short-lived, mostly idle once up.
|
||||
**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
|
||||
(labs are idle between keystrokes).
|
||||
|
||||
| Host | CPU | RAM | Fit |
|
||||
|------|-----|-----|-----|
|
||||
| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
|
||||
| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |
|
||||
|
||||
**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
|
||||
irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
|
||||
labeled `workload=rackrental`.
|
||||
|
||||
**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
|
||||
is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
|
||||
problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
|
||||
ready today.
|
||||
|
||||
### 3.3 Infrastructure + k8s control plane (always-on, low resource)
|
||||
|
||||
netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
|
||||
must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
|
||||
etcd diversity).
|
||||
|
||||
---
|
||||
|
||||
## 4. Host role summary (target)
|
||||
|
||||
| Host | CPU era | RAM | Primary role | Wnode class |
|
||||
|------|---------|-----|--------------|-------------|
|
||||
| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
|
||||
| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
|
||||
| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
|
||||
| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
|
||||
| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
|
||||
| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
|
||||
| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
|
||||
| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |
|
||||
|
||||
## 5. DDR3 memory policy
|
||||
|
||||
**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
|
||||
117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
|
||||
not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
|
||||
over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
|
||||
free RAM, not by installing sticks.
|
||||
|
||||
Compatibility is decisive and is documented per-host in
|
||||
[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
|
||||
**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
|
||||
(Patriot / most Kingston) fits essentially nowhere useful here.
|
||||
|
||||
---
|
||||
|
||||
## 6. Local storage policy: ephemeral scratch, not OS drives
|
||||
|
||||
**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
|
||||
VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
|
||||
across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
|
||||
workloads where data loss on node failure is acceptable by definition.
|
||||
|
||||
### 6.1 Why OS drives stay on NFS
|
||||
|
||||
Individual Proxmox nodes have **no local disk redundancy** (single disk, no
|
||||
RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
|
||||
Redundancy is instead achieved at the **storage-server + application layer**:
|
||||
|
||||
- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
|
||||
- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
|
||||
single disk or server failure takes down the whole cluster
|
||||
- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
|
||||
disk tier
|
||||
|
||||
This is deliberate. Moving OS drives to local storage would trade NFS network
|
||||
latency for single-disk-failure risk. Not worth it for persistent workloads.
|
||||
|
||||
### 6.2 What local storage IS good for: ephemeral scratch
|
||||
|
||||
Workloads that produce or consume temporary data that can be discarded on node
|
||||
failure. These are exactly the workloads where NFS latency hurts most and local
|
||||
IOPS help most:
|
||||
|
||||
| Use case | Why local scratch helps | Examples |
|
||||
|----------|------------------------|---------|
|
||||
| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
|
||||
| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
|
||||
| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
|
||||
| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |
|
||||
|
||||
### 6.3 Per-host local storage inventory and suitability
|
||||
|
||||
| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
|
||||
|------|--------------|------|-----------|-------------------|-------|
|
||||
| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
|
||||
| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
|
||||
| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
|
||||
| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
|
||||
| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
|
||||
| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
|
||||
| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
|
||||
| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
|
||||
|
||||
**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
|
||||
workload that benefits most from local scratch (fast container image pulls) —
|
||||
but their local storage is USB 2.0 portable HDD, which is slower than
|
||||
NFS-over-HDD. Options for the future:
|
||||
|
||||
1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
|
||||
5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
|
||||
local scratch. Deferred to a physical-hardware session.
|
||||
2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
|
||||
tolerates 30 MB/s. Functional but sluggish for lab spin-up.
|
||||
|
||||
### 6.4 How this maps to k8s StorageClasses (future, post-Friday)
|
||||
|
||||
The scratch tier maps to two k8s concepts:
|
||||
|
||||
```
|
||||
StorageClass: local-fast
|
||||
provisioner: local-path
|
||||
where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
|
||||
use: emptyDir replacement, build artifacts, container runtime cache
|
||||
redundancy: none — ephemeral, dies with the pod
|
||||
|
||||
StorageClass: nfs-hdd (existing concept)
|
||||
provisioner: nfs-subdir-external
|
||||
where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
|
||||
use: persistent bulk data, VM OS drives, anything that must survive restarts
|
||||
redundancy: cross-server spreading + PBS backup
|
||||
```
|
||||
|
||||
A wnode with local scratch gets a second disk (or a partition on local-lvm)
|
||||
mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
|
||||
(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
|
||||
fast storage request `local-fast` PVCs; pods that need persistence request
|
||||
`nfs-hdd`.
|
||||
|
||||
**This is forward-looking.** The concrete implementation (formatting local-lvm,
|
||||
adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
|
||||
happens after the Friday maintenance window when the SSD/NVMe tiers are
|
||||
physically installed and the storage architecture is finalized.
|
||||
> **Documentation moved to Discourse — the canonical source of truth.**
|
||||
>
|
||||
> **Capacity targets + workload placement model**
|
||||
>
|
||||
> **Read it here:** https://community.turnsys.com/t/304
|
||||
>
|
||||
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
|
||||
> Do not update content here — edit the Discourse wiki topic instead.*
|
||||
|
||||
Reference in New Issue
Block a user