KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: correct storage bus types after deep-dive on tsys6/7

Browse Source 
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.

Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.

Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.

Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-27 11:45:06 -05:00
parent db575080c9
commit ea2048dc87
3 changed files with 83 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT.md
+54 -24
View File
@@ -126,33 +126,63 @@ storage tier in the fleet. See section 7.4 for placement recommendation.
### 3.3 Local storage tiers (per host, with utilization)
| Host | Storage ID | Disk type | Total | Used | Free | Used by VMs? |
|------|-----------|-----------|-------|------|------|-------------|
| tsys1 | local-lvm | HDD | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
| tsys3 | local-lvm | HDD | 349 GB | 0 GB | **349 GB** | No (all VMs on NFS) |
| tsys4 | local-lvm | HDD | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | 1.7 TB | 40 MB | **1.7 TB** | No |
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
| tsys6 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
| tsys7 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming — fastest boot tier after tsys5 NVMe** |
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **1 TB** | (Win10) | **1 TB** | **Incoming — bulk data on SSD, not rust** |
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|------|-----------|-----------|-----|-------|------|------|-------------|
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
**Critical observation: every k8s host has 0% used local-lvm.** tsys3 (349 GB),
tsys6 (1.7 TB), tsys7 (1.7 TB) all have massive unused local storage. All their
VMs boot from NFS instead. Moving wnode boot disks to local-lvm would eliminate
NFS dependency and network latency for every k8s worker -- at zero cost.
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
for VM storage.
Additionally, both servers have completely empty internal drive bays:
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
drives.
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
DVD-ROM on port 5). No internal drives.
This is a reliability risk beyond performance: the entire host OS boots
from a consumer-grade portable USB drive not designed for 24/7 server use.
**tsys3 correction:** Previously documented as HDD. Actually boots from a
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
storage.
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
NFS-over-HDD and unsuitable for wnode boot disks.
### 3.4 Disk speed tiers summary
| Tier | Where | Speed class | Best for |
|------|-------|------------|----------|
| **NVMe** | tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot |
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd, bulk SSD data |
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
| **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs |
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs |
| **Local HDD** | tsys3/6/7 local-lvm | 100-150 MB/s | wnode boot disk (better than NFS-HDD due to no network hop) |
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
### 3.5 All exports are single-disk with no redundancy
@@ -603,10 +633,10 @@ and NFS-HDD for bulk data.**
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|-------|------|--------------------------|---------------------|-----------|
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
| wnode-tsys3 | tsys3 | **local-lvm (349 GB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 349 GB local is plenty for boot. |
| wnode-tsys6 | tsys6 | **local-lvm (1.7 TB HDD)** | D2 or S2 (NFS) | Move from NFS to local. 1.7 TB local is massive. Restart (currently stopped). |
| wnode-tsys7 | tsys7 | **local-lvm (1.7 TB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 1.7 TB local is massive. |
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 or D2 (NFS) | Move from NFS to local SSD. **Fastest boot disk for any wnode.** |
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |