KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(goals): codify local storage policy — ephemeral scratch, not OS drives

Browse Source 
Add CAPACITY-GOALS.md §6: local storage is a scratch tier for ephemeral workloads
(container image cache, build artifacts, emptyDir), NOT for VM OS drives (which
stay on NFS where redundancy is handled via 01/02 pairs + cross-disk spreading).
Documents per-host local storage suitability: tsys3/9/2 excellent (NVMe/SSD),
tsys5 usable (HDD bulk), tsys6/7 unusable (USB 2.0 at 30 MB/s). Includes the
future k8s StorageClass mapping (local-fast for scratch, nfs-hdd for persistent).
Adds forward-looking §12 to the audit referencing this policy.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 16:33:56 -05:00
parent edb34a75ec
commit 9c47d93a2f
2 changed files with 122 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-07-29.md
+32
View File
@@ -418,3 +418,35 @@ HA pairs stay split:
- `kubectl top nodes` — confirm cnodes healthy
- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up
- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills
---
## 12. Future: local storage as ephemeral scratch
> Codified as standing policy in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.
The fleet has substantial **unused local storage IOPS** on several hosts. OS
drives stay on NFS (no per-node disk redundancy — see policy), but local storage
is ideal for **ephemeral scratch** (container image cache, build artifacts,
`emptyDir` volumes) where data loss on node failure is acceptable.
**Per-host opportunity:**
| Host | Local | Available | Scratch-suitable? |
|------|-------|-----------|-------------------|
| tsys3 | NVMe (PM961) | 349 GB | **Yes — best existing** |
| tsys9 | SSD (PNY CS900) | 136 GB | **Yes** |
| tsys2 | NVMe + SSD | 512 GB + 1 TB | **Yes (incoming)** |
| tsys5 | HDD (Hitachi) | ~1.7 TB | Yes (bulk) |
| tsys6 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
| tsys7 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
**The tsys6/7 problem:** they are the RackRental hosts (most in need of fast
image pulls) but their local storage is USB 2.0 — slower than NFS. The real fix
is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5
empty SATA ports). Deferred to a physical-hardware session.
**Implementation:** post-Friday, once SSD/NVMe tiers are installed and the k8s
StorageClass design is finalized. Wnodes with local storage get a `/scratch`
mount for ephemeral data; persistent data stays on NFS. See
[`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.4 for the StorageClass mapping.
proxmox/docs/CAPACITY-GOALS.md
+90
View File
@@ -128,3 +128,93 @@ Compatibility is decisive and is documented per-host in
[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
(Patriot / most Kingston) fits essentially nowhere useful here.
---
## 6. Local storage policy: ephemeral scratch, not OS drives
**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
workloads where data loss on node failure is acceptable by definition.
### 6.1 Why OS drives stay on NFS
Individual Proxmox nodes have **no local disk redundancy** (single disk, no
RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
Redundancy is instead achieved at the **storage-server + application layer**:
- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
single disk or server failure takes down the whole cluster
- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
disk tier
This is deliberate. Moving OS drives to local storage would trade NFS network
latency for single-disk-failure risk. Not worth it for persistent workloads.
### 6.2 What local storage IS good for: ephemeral scratch
Workloads that produce or consume temporary data that can be discarded on node
failure. These are exactly the workloads where NFS latency hurts most and local
IOPS help most:
| Use case | Why local scratch helps | Examples |
|----------|------------------------|---------|
| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |
### 6.3 Per-host local storage inventory and suitability
| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
|------|--------------|------|-----------|-------------------|-------|
| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
workload that benefits most from local scratch (fast container image pulls) —
but their local storage is USB 2.0 portable HDD, which is slower than
NFS-over-HDD. Options for the future:
1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
local scratch. Deferred to a physical-hardware session.
2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
tolerates 30 MB/s. Functional but sluggish for lab spin-up.
### 6.4 How this maps to k8s StorageClasses (future, post-Friday)
The scratch tier maps to two k8s concepts:
```
StorageClass: local-fast
provisioner: local-path
where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
use: emptyDir replacement, build artifacts, container runtime cache
redundancy: none — ephemeral, dies with the pod
StorageClass: nfs-hdd (existing concept)
provisioner: nfs-subdir-external
where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
use: persistent bulk data, VM OS drives, anything that must survive restarts
redundancy: cross-server spreading + PBS backup
```
A wnode with local scratch gets a second disk (or a partition on local-lvm)
mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
fast storage request `local-fast` PVCs; pods that need persistence request
`nfs-hdd`.
**This is forward-looking.** The concrete implementation (formatting local-lvm,
adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
happens after the Friday maintenance window when the SSD/NVMe tiers are
physically installed and the storage architecture is finalized.