fix: correct storage bus types after deep-dive on tsys6/7
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -59,11 +59,14 @@ on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
|
||||
if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
|
||||
(tsys4 SSD), ucs-02 to tsys9 local SSD.
|
||||
3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
|
||||
unused local SSD; all k8s hosts (tsys3/6/7) have 0% used local-lvm.
|
||||
unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
|
||||
4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
|
||||
NFS export. Should host ucs-01 (LDAP/AD latency benefit).
|
||||
5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
|
||||
used. VMs should be rebalanced to reduce tsys4 blast radius.
|
||||
6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
|
||||
Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
|
||||
hosts boots from USB 2.0 -- user accepts this risk.
|
||||
|
||||
## Storage network IPs (VLAN1000, 10.100.100.0/24)
|
||||
|
||||
|
||||
@@ -95,10 +95,10 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
|
||||
| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
|
||||
| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
|
||||
| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
|
||||
| wnode-tsys3 | worker | tsys3 | **local-lvm (HDD)** | S3 (NFS) | LOCAL-HDD |
|
||||
| wnode-tsys5 | worker | tsys5 | **NVMe (local)** | local-nonprod (HDD) | **NVMe** |
|
||||
| wnode-tsys6 | worker | tsys6 | **local-lvm (HDD)** | D2 (NFS) | LOCAL-HDD |
|
||||
| wnode-tsys7 | worker | tsys7 | **local-lvm (HDD)** | D5 (NFS) | LOCAL-HDD |
|
||||
| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
|
||||
| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
|
||||
| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
|
||||
|
||||
### 3.3 Storage server distribution after changes
|
||||
@@ -110,7 +110,9 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
|
||||
| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
|
||||
| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
|
||||
|
||||
**Note: wnode-tsys2 needs no NFS** -- it has 1.5 TB local SSD (NVMe + SATA SSD).
|
||||
**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
|
||||
NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
|
||||
than NFS-HDD, and the user has chosen not to install internal drives.
|
||||
|
||||
**No single disk or server is a quorum-losing failure point.**
|
||||
|
||||
@@ -218,8 +220,8 @@ responsive. **This is the single highest-impact change for k8s performance.**
|
||||
| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
|
||||
| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
|
||||
| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
|
||||
| tsys6 | wnode-tsys6 | **local-lvm (1.7 TB)** | D2 (NFS) | 128 GB | 64-96 GB | **Heavy worker, max RAM** |
|
||||
| tsys7 | wnode-tsys7 | **local-lvm (1.7 TB)** | D5 (NFS) | 192 GB | 96-128 GB | **Heavy worker, max RAM** |
|
||||
| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
|
||||
|
||||
### 5.2 Why boot disks on local-lvm
|
||||
@@ -239,8 +241,8 @@ With local-lvm boot disks:
|
||||
|
||||
| Host | Recommended wnode config | Rationale |
|
||||
|------|------------------------|-----------|
|
||||
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, local-lvm boot + NFS data | Largest host -- run the heaviest ETL/HPC jobs here |
|
||||
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, local-lvm boot + NFS data | Second-largest -- parallel heavy jobs |
|
||||
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
|
||||
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
|
||||
| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
|
||||
| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
|
||||
| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
|
||||
@@ -341,8 +343,8 @@ disks across both servers:
|
||||
|-------|-------------|-----------------|------------|
|
||||
| wnode-tsys3 | local-lvm | S3 | tsys5 |
|
||||
| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
|
||||
| wnode-tsys6 | local-lvm | D2 | tsys4 |
|
||||
| wnode-tsys7 | local-lvm | D5 | tsys4 |
|
||||
| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
|
||||
|
||||
This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
|
||||
@@ -392,12 +394,15 @@ For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
|
||||
it on local-lvm of the heavy hosts:
|
||||
|
||||
```
|
||||
tsys6 local-lvm (1.7 TB): /data/cache/weather/ -- pre-populated datasets
|
||||
tsys7 local-lvm (1.7 TB): /data/cache/gis/ -- pre-populated datasets
|
||||
tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier
|
||||
tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier
|
||||
```
|
||||
|
||||
This avoids re-reading the same data from NFS on every job. The 1.7 TB of
|
||||
local HDD on each heavy host is perfect for this -- it is currently 100% empty.
|
||||
**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
|
||||
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
|
||||
|
||||
This avoids re-reading the same data from the same NFS export on every job
|
||||
if the data is already cached in the page cache.
|
||||
|
||||
---
|
||||
|
||||
@@ -408,8 +413,8 @@ local HDD on each heavy host is perfect for this -- it is currently 100% empty.
|
||||
After tsys5 cable + NVMe and tsys4 NIC + RAM:
|
||||
|
||||
1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
|
||||
2. **Restart wnode-tsys6** on local-lvm (not NFS). Recreate VM 100 with disk
|
||||
on `local-lvm` instead of D5.
|
||||
2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
|
||||
do not use for VM storage. Recreate on D2 or D5 NFS.
|
||||
3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
|
||||
|
||||
### 8.2 Phase 2: Cnode rebalance (maintenance window)
|
||||
@@ -426,10 +431,10 @@ Plan for a maintenance window with the k8s cluster briefly down.
|
||||
### 8.3 Phase 3: Wnode local storage migration (maintenance window)
|
||||
|
||||
1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
|
||||
2. Recreate wnode-tsys6 with boot disk on local-lvm (1.7 TB).
|
||||
3. Recreate wnode-tsys7 with boot disk on local-lvm (1.7 TB).
|
||||
2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
|
||||
3. wnode-tsys7 stays on NFS (same reason).
|
||||
4. Recreate wnode-tsys5 with boot disk on NVMe.
|
||||
5. Add data disks (NFS) as second SCSI devices.
|
||||
5. Add data disks (NFS) as second SCSI devices where applicable.
|
||||
|
||||
### 8.4 Phase 4: tsys2 integration (when rebuilt)
|
||||
|
||||
|
||||
+54
-24
@@ -126,33 +126,63 @@ storage tier in the fleet. See section 7.4 for placement recommendation.
|
||||
|
||||
### 3.3 Local storage tiers (per host, with utilization)
|
||||
|
||||
| Host | Storage ID | Disk type | Total | Used | Free | Used by VMs? |
|
||||
|------|-----------|-----------|-------|------|------|-------------|
|
||||
| tsys1 | local-lvm | HDD | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
|
||||
| tsys3 | local-lvm | HDD | 349 GB | 0 GB | **349 GB** | No (all VMs on NFS) |
|
||||
| tsys4 | local-lvm | HDD | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
|
||||
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | 1.7 TB | 40 MB | **1.7 TB** | No |
|
||||
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
|
||||
| tsys6 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
|
||||
| tsys7 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
|
||||
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
|
||||
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming — fastest boot tier after tsys5 NVMe** |
|
||||
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **1 TB** | (Win10) | **1 TB** | **Incoming — bulk data on SSD, not rust** |
|
||||
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|
||||
|------|-----------|-----------|-----|-------|------|------|-------------|
|
||||
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
|
||||
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
|
||||
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
|
||||
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
|
||||
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
|
||||
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
|
||||
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
|
||||
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
|
||||
|
||||
**Critical observation: every k8s host has 0% used local-lvm.** tsys3 (349 GB),
|
||||
tsys6 (1.7 TB), tsys7 (1.7 TB) all have massive unused local storage. All their
|
||||
VMs boot from NFS instead. Moving wnode boot disks to local-lvm would eliminate
|
||||
NFS dependency and network latency for every k8s worker -- at zero cost.
|
||||
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
|
||||
|
||||
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
|
||||
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
|
||||
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
|
||||
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
|
||||
|
||||
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
|
||||
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
|
||||
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
|
||||
for VM storage.
|
||||
|
||||
Additionally, both servers have completely empty internal drive bays:
|
||||
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
|
||||
drives.
|
||||
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
|
||||
DVD-ROM on port 5). No internal drives.
|
||||
|
||||
This is a reliability risk beyond performance: the entire host OS boots
|
||||
from a consumer-grade portable USB drive not designed for 24/7 server use.
|
||||
|
||||
**tsys3 correction:** Previously documented as HDD. Actually boots from a
|
||||
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
|
||||
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
|
||||
|
||||
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
|
||||
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
|
||||
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
|
||||
storage.
|
||||
|
||||
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
|
||||
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
|
||||
NFS-over-HDD and unsuitable for wnode boot disks.
|
||||
|
||||
### 3.4 Disk speed tiers summary
|
||||
|
||||
| Tier | Where | Speed class | Best for |
|
||||
|------|-------|------------|----------|
|
||||
| **NVMe** | tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot |
|
||||
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd, bulk SSD data |
|
||||
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
|
||||
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
|
||||
| **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs |
|
||||
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs |
|
||||
| **Local HDD** | tsys3/6/7 local-lvm | 100-150 MB/s | wnode boot disk (better than NFS-HDD due to no network hop) |
|
||||
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
|
||||
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
|
||||
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
|
||||
|
||||
### 3.5 All exports are single-disk with no redundancy
|
||||
|
||||
@@ -603,10 +633,10 @@ and NFS-HDD for bulk data.**
|
||||
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|
||||
|-------|------|--------------------------|---------------------|-----------|
|
||||
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
|
||||
| wnode-tsys3 | tsys3 | **local-lvm (349 GB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 349 GB local is plenty for boot. |
|
||||
| wnode-tsys6 | tsys6 | **local-lvm (1.7 TB HDD)** | D2 or S2 (NFS) | Move from NFS to local. 1.7 TB local is massive. Restart (currently stopped). |
|
||||
| wnode-tsys7 | tsys7 | **local-lvm (1.7 TB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 1.7 TB local is massive. |
|
||||
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 or D2 (NFS) | Move from NFS to local SSD. **Fastest boot disk for any wnode.** |
|
||||
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
|
||||
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
|
||||
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
|
||||
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user