KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: correct storage bus types after deep-dive on tsys6/7

Browse Source 
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.

Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.

Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.

Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-27 11:45:06 -05:00
parent db575080c9
commit ea2048dc87
3 changed files with 83 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
K8S.md
+25 -20
View File
@@ -95,10 +95,10 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
| wnode-tsys3 | worker | tsys3 | **local-lvm (HDD)** | S3 (NFS) | LOCAL-HDD |
| wnode-tsys5 | worker | tsys5 | **NVMe (local)** | local-nonprod (HDD) | **NVMe** |
| wnode-tsys6 | worker | tsys6 | **local-lvm (HDD)** | D2 (NFS) | LOCAL-HDD |
| wnode-tsys7 | worker | tsys7 | **local-lvm (HDD)** | D5 (NFS) | LOCAL-HDD |
| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
### 3.3 Storage server distribution after changes
@@ -110,7 +110,9 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
**Note: wnode-tsys2 needs no NFS** -- it has 1.5 TB local SSD (NVMe + SATA SSD).
**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
than NFS-HDD, and the user has chosen not to install internal drives.
**No single disk or server is a quorum-losing failure point.**
@@ -218,8 +220,8 @@ responsive. **This is the single highest-impact change for k8s performance.**
| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
| tsys6 | wnode-tsys6 | **local-lvm (1.7 TB)** | D2 (NFS) | 128 GB | 64-96 GB | **Heavy worker, max RAM** |
| tsys7 | wnode-tsys7 | **local-lvm (1.7 TB)** | D5 (NFS) | 192 GB | 96-128 GB | **Heavy worker, max RAM** |
| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
### 5.2 Why boot disks on local-lvm
@@ -239,8 +241,8 @@ With local-lvm boot disks:
| Host | Recommended wnode config | Rationale |
|------|------------------------|-----------|
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, local-lvm boot + NFS data | Largest host -- run the heaviest ETL/HPC jobs here |
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, local-lvm boot + NFS data | Second-largest -- parallel heavy jobs |
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
@@ -341,8 +343,8 @@ disks across both servers:
|-------|-------------|-----------------|------------|
| wnode-tsys3 | local-lvm | S3 | tsys5 |
| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
| wnode-tsys6 | local-lvm | D2 | tsys4 |
| wnode-tsys7 | local-lvm | D5 | tsys4 |
| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
@@ -392,12 +394,15 @@ For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
it on local-lvm of the heavy hosts:
```
tsys6 local-lvm (1.7 TB): /data/cache/weather/ -- pre-populated datasets
tsys7 local-lvm (1.7 TB): /data/cache/gis/ -- pre-populated datasets
tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier
tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier
```
This avoids re-reading the same data from NFS on every job. The 1.7 TB of
local HDD on each heavy host is perfect for this -- it is currently 100% empty.
**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
This avoids re-reading the same data from the same NFS export on every job
if the data is already cached in the page cache.
---
@@ -408,8 +413,8 @@ local HDD on each heavy host is perfect for this -- it is currently 100% empty.
After tsys5 cable + NVMe and tsys4 NIC + RAM:
1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
2. **Restart wnode-tsys6** on local-lvm (not NFS). Recreate VM 100 with disk
on `local-lvm` instead of D5.
2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
do not use for VM storage. Recreate on D2 or D5 NFS.
3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
### 8.2 Phase 2: Cnode rebalance (maintenance window)
@@ -426,10 +431,10 @@ Plan for a maintenance window with the k8s cluster briefly down.
### 8.3 Phase 3: Wnode local storage migration (maintenance window)
1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
2. Recreate wnode-tsys6 with boot disk on local-lvm (1.7 TB).
3. Recreate wnode-tsys7 with boot disk on local-lvm (1.7 TB).
2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
3. wnode-tsys7 stays on NFS (same reason).
4. Recreate wnode-tsys5 with boot disk on NVMe.
5. Add data disks (NFS) as second SCSI devices.
5. Add data disks (NFS) as second SCSI devices where applicable.
### 8.4 Phase 4: tsys2 integration (when rebuilt)