docs(goals): codify local storage policy — ephemeral scratch, not OS drives

Add CAPACITY-GOALS.md §6: local storage is a scratch tier for ephemeral workloads
(container image cache, build artifacts, emptyDir), NOT for VM OS drives (which
stay on NFS where redundancy is handled via 01/02 pairs + cross-disk spreading).
Documents per-host local storage suitability: tsys3/9/2 excellent (NVMe/SSD),
tsys5 usable (HDD bulk), tsys6/7 unusable (USB 2.0 at 30 MB/s). Includes the
future k8s StorageClass mapping (local-fast for scratch, nfs-hdd for persistent).
Adds forward-looking §12 to the audit referencing this policy.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

proxmox/docs/CAPACITY-GOALS.md

@@ -128,3 +128,93 @@ Compatibility is decisive and is documented per-host in
 [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
 **Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
 (Patriot / most Kingston) fits essentially nowhere useful here.
+
+---
+
+## 6. Local storage policy: ephemeral scratch, not OS drives
+
+**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
+VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
+across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
+workloads where data loss on node failure is acceptable by definition.
+
+### 6.1 Why OS drives stay on NFS
+
+Individual Proxmox nodes have **no local disk redundancy** (single disk, no
+RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
+Redundancy is instead achieved at the **storage-server + application layer**:
+
+- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
+- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
+  single disk or server failure takes down the whole cluster
+- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
+  disk tier
+
+This is deliberate. Moving OS drives to local storage would trade NFS network
+latency for single-disk-failure risk. Not worth it for persistent workloads.
+
+### 6.2 What local storage IS good for: ephemeral scratch
+
+Workloads that produce or consume temporary data that can be discarded on node
+failure. These are exactly the workloads where NFS latency hurts most and local
+IOPS help most:
+
+| Use case | Why local scratch helps | Examples |
+|----------|------------------------|---------|
+| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
+| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
+| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
+| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |
+
+### 6.3 Per-host local storage inventory and suitability
+
+| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
+|------|--------------|------|-----------|-------------------|-------|
+| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
+| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
+| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
+| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
+| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
+| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
+| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
+| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
+
+**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
+workload that benefits most from local scratch (fast container image pulls) —
+but their local storage is USB 2.0 portable HDD, which is slower than
+NFS-over-HDD. Options for the future:
+
+1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
+   5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
+   local scratch. Deferred to a physical-hardware session.
+2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
+   tolerates 30 MB/s. Functional but sluggish for lab spin-up.
+
+### 6.4 How this maps to k8s StorageClasses (future, post-Friday)
+
+The scratch tier maps to two k8s concepts:
+
+```
+StorageClass: local-fast
+  provisioner: local-path
+  where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
+  use: emptyDir replacement, build artifacts, container runtime cache
+  redundancy: none — ephemeral, dies with the pod
+
+StorageClass: nfs-hdd  (existing concept)
+  provisioner: nfs-subdir-external
+  where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
+  use: persistent bulk data, VM OS drives, anything that must survive restarts
+  redundancy: cross-server spreading + PBS backup
+```
+
+A wnode with local scratch gets a second disk (or a partition on local-lvm)
+mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
+(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
+fast storage request `local-fast` PVCs; pods that need persistence request
+`nfs-hdd`.
+
+**This is forward-looking.** The concrete implementation (formatting local-lvm,
+adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
+happens after the Friday maintenance window when the SSD/NVMe tiers are
+physically installed and the storage architecture is finalized.