From 9c47d93a2fd7af67b20ab370eaab61e01bb879c2 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Wed, 29 Jul 2026 16:33:56 -0500 Subject: [PATCH] =?UTF-8?q?docs(goals):=20codify=20local=20storage=20polic?= =?UTF-8?q?y=20=E2=80=94=20ephemeral=20scratch,=20not=20OS=20drives?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add CAPACITY-GOALS.md ยง6: local storage is a scratch tier for ephemeral workloads (container image cache, build artifacts, emptyDir), NOT for VM OS drives (which stay on NFS where redundancy is handled via 01/02 pairs + cross-disk spreading). Documents per-host local storage suitability: tsys3/9/2 excellent (NVMe/SSD), tsys5 usable (HDD bulk), tsys6/7 unusable (USB 2.0 at 30 MB/s). Includes the future k8s StorageClass mapping (local-fast for scratch, nfs-hdd for persistent). Adds forward-looking ยง12 to the audit referencing this policy. ๐Ÿ’˜ Generated with Crush Assisted-by: Crush:glm-5.2 --- proxmox/docs/AUDIT-2026-07-29.md | 32 ++++++++++++ proxmox/docs/CAPACITY-GOALS.md | 90 ++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) diff --git a/proxmox/docs/AUDIT-2026-07-29.md b/proxmox/docs/AUDIT-2026-07-29.md index 9f63776..0f619c3 100644 --- a/proxmox/docs/AUDIT-2026-07-29.md +++ b/proxmox/docs/AUDIT-2026-07-29.md @@ -418,3 +418,35 @@ HA pairs stay split: - `kubectl top nodes` โ€” confirm cnodes healthy - `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` โ€” confirm D2 frees up - `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` โ€” confirm fills + +--- + +## 12. Future: local storage as ephemeral scratch + +> Codified as standing policy in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) ยง6. + +The fleet has substantial **unused local storage IOPS** on several hosts. OS +drives stay on NFS (no per-node disk redundancy โ€” see policy), but local storage +is ideal for **ephemeral scratch** (container image cache, build artifacts, +`emptyDir` volumes) where data loss on node failure is acceptable. + +**Per-host opportunity:** + +| Host | Local | Available | Scratch-suitable? | +|------|-------|-----------|-------------------| +| tsys3 | NVMe (PM961) | 349 GB | **Yes โ€” best existing** | +| tsys9 | SSD (PNY CS900) | 136 GB | **Yes** | +| tsys2 | NVMe + SSD | 512 GB + 1 TB | **Yes (incoming)** | +| tsys5 | HDD (Hitachi) | ~1.7 TB | Yes (bulk) | +| tsys6 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** | +| tsys7 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** | + +**The tsys6/7 problem:** they are the RackRental hosts (most in need of fast +image pulls) but their local storage is USB 2.0 โ€” slower than NFS. The real fix +is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5 +empty SATA ports). Deferred to a physical-hardware session. + +**Implementation:** post-Friday, once SSD/NVMe tiers are installed and the k8s +StorageClass design is finalized. Wnodes with local storage get a `/scratch` +mount for ephemeral data; persistent data stays on NFS. See +[`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) ยง6.4 for the StorageClass mapping. diff --git a/proxmox/docs/CAPACITY-GOALS.md b/proxmox/docs/CAPACITY-GOALS.md index 6dbcfc6..a8649ce 100644 --- a/proxmox/docs/CAPACITY-GOALS.md +++ b/proxmox/docs/CAPACITY-GOALS.md @@ -128,3 +128,93 @@ Compatibility is decisive and is documented per-host in [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) ยง5. Short version: tsys5/6/7 require **Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM (Patriot / most Kingston) fits essentially nowhere useful here. + +--- + +## 6. Local storage policy: ephemeral scratch, not OS drives + +**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier. +VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading +across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral +workloads where data loss on node failure is acceptable by definition. + +### 6.1 Why OS drives stay on NFS + +Individual Proxmox nodes have **no local disk redundancy** (single disk, no +RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone. +Redundancy is instead achieved at the **storage-server + application layer**: + +- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server +- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no + single disk or server failure takes down the whole cluster +- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of + disk tier + +This is deliberate. Moving OS drives to local storage would trade NFS network +latency for single-disk-failure risk. Not worth it for persistent workloads. + +### 6.2 What local storage IS good for: ephemeral scratch + +Workloads that produce or consume temporary data that can be discarded on node +failure. These are exactly the workloads where NFS latency hurts most and local +IOPS help most: + +| Use case | Why local scratch helps | Examples | +|----------|------------------------|---------| +| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup | +| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines | +| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation | +| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods | + +### 6.3 Per-host local storage inventory and suitability + +| Host | Local storage | Type | Available | Scratch-suitable? | Notes | +|------|--------------|------|-----------|-------------------|-------| +| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes โ€” best in fleet** | Incoming (post-rebuild). No NFS needed at all | +| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes โ€” excellent** | Fastest existing local storage. 0% used | +| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes โ€” excellent** | Low latency. 0% used | +| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host โ€” scratch is secondary | +| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway | +| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space | +| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO โ€” 30 MB/s** | Slower than NFS. Do not use for VMs | +| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO โ€” 30 MB/s** | Slower than NFS. Do not use for VMs | + +**tsys6/7 are the hard problem.** They are the RackRental hosts โ€” exactly the +workload that benefits most from local scratch (fast container image pulls) โ€” +but their local storage is USB 2.0 portable HDD, which is slower than +NFS-over-HDD. Options for the future: + +1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has + 5 empty SATA ports). This is the real fix โ€” gives both RackRental hosts proper + local scratch. Deferred to a physical-hardware session. +2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that + tolerates 30 MB/s. Functional but sluggish for lab spin-up. + +### 6.4 How this maps to k8s StorageClasses (future, post-Friday) + +The scratch tier maps to two k8s concepts: + +``` +StorageClass: local-fast + provisioner: local-path + where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD) + use: emptyDir replacement, build artifacts, container runtime cache + redundancy: none โ€” ephemeral, dies with the pod + +StorageClass: nfs-hdd (existing concept) + provisioner: nfs-subdir-external + where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5) + use: persistent bulk data, VM OS drives, anything that must survive restarts + redundancy: cross-server spreading + PBS backup +``` + +A wnode with local scratch gets a second disk (or a partition on local-lvm) +mounted as `/scratch`. ETL jobs write intermediate data to `/scratch` +(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral +fast storage request `local-fast` PVCs; pods that need persistence request +`nfs-hdd`. + +**This is forward-looking.** The concrete implementation (formatting local-lvm, +adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes) +happens after the Friday maintenance window when the SSD/NVMe tiers are +physically installed and the storage architecture is finalized.