diff --git a/AGENTS.md b/AGENTS.md index 9e4a3b4..a402e04 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,11 +59,14 @@ on a VPS in Reston VA (Cloudron) -- this cluster is R&D only. if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3 (tsys4 SSD), ucs-02 to tsys9 local SSD. 3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB - unused local SSD; all k8s hosts (tsys3/6/7) have 0% used local-lvm. + unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961). 4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD NFS export. Should host ucs-01 (LDAP/AD latency benefit). 5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely used. VMs should be rebalanced to reduce tsys4 blast radius. +6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s). + Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these + hosts boots from USB 2.0 -- user accepts this risk. ## Storage network IPs (VLAN1000, 10.100.100.0/24) diff --git a/K8S.md b/K8S.md index b84f659..584114f 100644 --- a/K8S.md +++ b/K8S.md @@ -95,10 +95,10 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both. | cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD | | wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) | | wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** | -| wnode-tsys3 | worker | tsys3 | **local-lvm (HDD)** | S3 (NFS) | LOCAL-HDD | -| wnode-tsys5 | worker | tsys5 | **NVMe (local)** | local-nonprod (HDD) | **NVMe** | -| wnode-tsys6 | worker | tsys6 | **local-lvm (HDD)** | D2 (NFS) | LOCAL-HDD | -| wnode-tsys7 | worker | tsys7 | **local-lvm (HDD)** | D5 (NFS) | LOCAL-HDD | +| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** | +| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** | +| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) | +| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) | | wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** | ### 3.3 Storage server distribution after changes @@ -110,7 +110,9 @@ SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both. | tsys5 (S2) | cnode3 | wnode-tsys9 | -- | | tsys5 (S3) | cnode5 | -- | wnode-tsys3 | -**Note: wnode-tsys2 needs no NFS** -- it has 1.5 TB local SSD (NVMe + SATA SSD). +**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on +NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower +than NFS-HDD, and the user has chosen not to install internal drives. **No single disk or server is a quorum-losing failure point.** @@ -218,8 +220,8 @@ responsive. **This is the single highest-impact change for k8s performance.** | tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** | | tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker | | tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** | -| tsys6 | wnode-tsys6 | **local-lvm (1.7 TB)** | D2 (NFS) | 128 GB | 64-96 GB | **Heavy worker, max RAM** | -| tsys7 | wnode-tsys7 | **local-lvm (1.7 TB)** | D5 (NFS) | 192 GB | 96-128 GB | **Heavy worker, max RAM** | +| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS | +| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS | | tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot | ### 5.2 Why boot disks on local-lvm @@ -239,8 +241,8 @@ With local-lvm boot disks: | Host | Recommended wnode config | Rationale | |------|------------------------|-----------| -| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, local-lvm boot + NFS data | Largest host -- run the heaviest ETL/HPC jobs here | -| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, local-lvm boot + NFS data | Second-largest -- parallel heavy jobs | +| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 | +| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 | | tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC | | tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker | | tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds | @@ -341,8 +343,8 @@ disks across both servers: |-------|-------------|-----------------|------------| | wnode-tsys3 | local-lvm | S3 | tsys5 | | wnode-tsys5 | NVMe | local-nonprod | local (no NFS) | -| wnode-tsys6 | local-lvm | D2 | tsys4 | -| wnode-tsys7 | local-lvm | D5 | tsys4 | +| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 | +| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 | | wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 | This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5. @@ -392,12 +394,15 @@ For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache it on local-lvm of the heavy hosts: ``` -tsys6 local-lvm (1.7 TB): /data/cache/weather/ -- pre-populated datasets -tsys7 local-lvm (1.7 TB): /data/cache/gis/ -- pre-populated datasets +tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier +tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier ``` -This avoids re-reading the same data from NFS on every job. The 1.7 TB of -local HDD on each heavy host is perfect for this -- it is currently 100% empty. +**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot +be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead. + +This avoids re-reading the same data from the same NFS export on every job +if the data is already cached in the page cache. --- @@ -408,8 +413,8 @@ local HDD on each heavy host is perfect for this -- it is currently 100% empty. After tsys5 cable + NVMe and tsys4 NIC + RAM: 1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`) -2. **Restart wnode-tsys6** on local-lvm (not NFS). Recreate VM 100 with disk - on `local-lvm` instead of D5. +2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 -- + do not use for VM storage. Recreate on D2 or D5 NFS. 3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD). ### 8.2 Phase 2: Cnode rebalance (maintenance window) @@ -426,10 +431,10 @@ Plan for a maintenance window with the k8s cluster briefly down. ### 8.3 Phase 3: Wnode local storage migration (maintenance window) 1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB). -2. Recreate wnode-tsys6 with boot disk on local-lvm (1.7 TB). -3. Recreate wnode-tsys7 with boot disk on local-lvm (1.7 TB). +2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable). +3. wnode-tsys7 stays on NFS (same reason). 4. Recreate wnode-tsys5 with boot disk on NVMe. -5. Add data disks (NFS) as second SCSI devices. +5. Add data disks (NFS) as second SCSI devices where applicable. ### 8.4 Phase 4: tsys2 integration (when rebuilt) diff --git a/PROJECT.md b/PROJECT.md index 226ab22..493f7de 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -126,33 +126,63 @@ storage tier in the fleet. See section 7.4 for placement recommendation. ### 3.3 Local storage tiers (per host, with utilization) -| Host | Storage ID | Disk type | Total | Used | Free | Used by VMs? | -|------|-----------|-----------|-------|------|------|-------------| -| tsys1 | local-lvm | HDD | ~90 GB | low | ~90 GB | No (all VMs on NFS) | -| tsys3 | local-lvm | HDD | 349 GB | 0 GB | **349 GB** | No (all VMs on NFS) | -| tsys4 | local-lvm | HDD | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) | -| tsys5 | local-lvm | HDD (Hitachi 1.8T) | 1.7 TB | 40 MB | **1.7 TB** | No | -| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) | -| tsys6 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) | -| tsys7 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) | -| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** | -| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming — fastest boot tier after tsys5 NVMe** | -| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **1 TB** | (Win10) | **1 TB** | **Incoming — bulk data on SSD, not rust** | +| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? | +|------|-----------|-----------|-----|-------|------|------|-------------| +| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) | +| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** | +| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) | +| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No | +| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) | +| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** | +| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** | +| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** | +| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** | +| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** | -**Critical observation: every k8s host has 0% used local-lvm.** tsys3 (349 GB), -tsys6 (1.7 TB), tsys7 (1.7 TB) all have massive unused local storage. All their -VMs boot from NFS instead. Moving wnode boot disks to local-lvm would eliminate -NFS dependency and network latency for every k8s worker -- at zero cost. +**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.** + +The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a +single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport +260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI +(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host. + +**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than +NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these +hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used +for VM storage. + +Additionally, both servers have completely empty internal drive bays: +- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal +drives. +- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only +DVD-ROM on port 5). No internal drives. + +This is a reliability risk beyond performance: the entire host OS boots +from a consumer-grade portable USB drive not designed for 24/7 server use. + +**tsys3 correction:** Previously documented as HDD. Actually boots from a +**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the +fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use. + +**Critical observation: every k8s host has 0% used local-lvm.** tsys3 +(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD, +1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local +storage. + +**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and +tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than +NFS-over-HDD and unsuitable for wnode boot disks. ### 3.4 Disk speed tiers summary | Tier | Where | Speed class | Best for | |------|-------|------------|----------| -| **NVMe** | tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot | -| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd, bulk SSD data | +| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd | +| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd | | **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs | -| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs | -| **Local HDD** | tsys3/6/7 local-lvm | 100-150 MB/s | wnode boot disk (better than NFS-HDD due to no network hop) | +| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** | +| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only | +| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** | ### 3.5 All exports are single-disk with no redundancy @@ -603,10 +633,10 @@ and NFS-HDD for bulk data.** | wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale | |-------|------|--------------------------|---------------------|-----------| | wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. | -| wnode-tsys3 | tsys3 | **local-lvm (349 GB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 349 GB local is plenty for boot. | -| wnode-tsys6 | tsys6 | **local-lvm (1.7 TB HDD)** | D2 or S2 (NFS) | Move from NFS to local. 1.7 TB local is massive. Restart (currently stopped). | -| wnode-tsys7 | tsys7 | **local-lvm (1.7 TB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 1.7 TB local is massive. | -| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 or D2 (NFS) | Move from NFS to local SSD. **Fastest boot disk for any wnode.** | +| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. | +| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. | +| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. | +| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. | | wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. | | wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |