From 019ab468bff9bbd7e4a195bbf729f8b065989dcb Mon Sep 17 00:00:00 2001 From: reachableceo Date: Mon, 27 Jul 2026 11:50:16 -0500 Subject: [PATCH] feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep-dive on tsys4/5 storage controllers revealed: - tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0 - tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0 Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB bottleneck and consolidates all fast storage on tsys5: NVMe (local) + D3 SSD + T5-SSD + 4 HDD exports. Updated storage architecture, speed tiers, critical VM placement, failure survival matrix, StorageClass design, and Friday plan. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- AGENTS.md | 19 +++++++++++---- K8S.md | 17 ++++++++----- PROJECT.md | 70 +++++++++++++++++++++++++++++++----------------------- TODO.md | 19 +++++++++++---- 4 files changed, 79 insertions(+), 46 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a402e04..8cc97dd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,17 +76,26 @@ tsys1=.1 tsys3=.3 tsys4=.4 tsys5=.5 tsys6=.6 tsys7=.7 tsys9=.9 ## NFS export topology -**tsys4 exports (primary, 68% of VMs depend on it):** +**tsys4 exports (primary, overloaded):** - D2 = WDC Red 3TB HDD — most VMs live here -- D3 = SK hynix 512GB SSD (USB-attached on tsys4) - D5 = Hitachi 2TB HDD +- (sda Hitachi 1.8T at /mnt/albert — not NFS shared) +- (sdd WDC 1T — idle, unmounted, removable) +- (sdf WDC 4.5T SMR at /mnt/backup — PBS target) -**tsys5 exports (secondary, barely used by k8s):** +**tsys5 exports (fast-tier hub -- consolidated Friday):** - S1/S2/S3 = Seagate 1TB HDD each - S4 = Toshiba 500GB HDD -- T5-SSD = Samsung 860 PRO 256GB SSD +- T5-SSD = Samsung 860 PRO 256GB SSD (existing) +- **D3 = SK hynix SC300 512GB SSD** (moving from tsys4 USB to tsys5 SAS Friday) +- **NVMe (local, new Friday)** — local-only, wnode-tsys5 boot + HPC scratch -**New Friday:** tsys5 NVMe (recommend local-only, not NFS-exported). +**tsys5 hardware:** LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port, +2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe. + +**Friday change:** D3 export repoints from pfv-tsys4-nfs-stor to +pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently +none of significance — it's 99% empty) keep working after remount. ## SSH access diff --git a/K8S.md b/K8S.md index 584114f..2a8033e 100644 --- a/K8S.md +++ b/K8S.md @@ -326,13 +326,18 @@ spec: storage: 50Gi ``` -### 6.3 NFS-SSD tier (D3 and T5-SSD) +### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday) -The SSD NFS exports are a middle tier. Use cases: -- **D3 (tsys4, 445 GB free):** CI/CD artifacts, container image cache for - sectestbed nodes. Currently 99% empty. -- **T5-SSD (tsys5, 140 GB free):** SIEM databases, monitoring data. Currently - 35% used by ultix-streaming. +The SSD NFS exports are a middle tier. **Both consolidated on tsys5** after +the Friday D3 relocation: +- **D3 (tsys5 SAS, 445 GB free):** CI/CD artifacts, container image cache, + UCS/LDAP. Currently 99% empty. Fastest NFS tier after the move off USB. +- **T5-SSD (tsys5 SAS, 140 GB free):** SIEM databases, monitoring data. + Currently 35% used by ultix-streaming. + +**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host. +This simplifies the StorageClass design -- latency-sensitive PVCs target +tsys5 exports, bulk PVCs target either server. ### 6.4 NFS data distribution across storage servers diff --git a/PROJECT.md b/PROJECT.md index 493f7de..c369603 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -86,43 +86,51 @@ running Cloudron -- not in this cluster. | Export | Disk model | Type | Bus | Total | Used | Free | Use% | |--------|-----------|------|-----|-------|------|------|------| | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% | -| D3 | SK hynix SC300 | **SSD** | **USB** | 469 GB | 2 MB | **445 GB** | **0%** | +| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** | | D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% | Non-exported disks on tsys4: - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free -- sdd (WDC 1T) -- not mounted, idle +- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use) - sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free -**Key finding: D3 (SSD, 445 GB) is 99% empty.** Nobody is using the only SSD -export on the primary storage server. This is the fastest available NFS tier -from tsys4 and should be used for latency-sensitive VMs (UCS/LDAP, etcd). +**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on +tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port, +eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb +/sdc/sdd), so tsys5 is the better target. See section 3.2. -**Warning: D3 is USB-attached.** The SK hynix SSD is connected via a USB adapter -(`usb-SSD_ThinkPlus`). It achieves SSD speeds but carries the same USB -reliability risk as the cdc_ncm NIC dongles. After tsys4 gets its PCIe NIC -(Friday), consider adding a proper SATA/SAS SSD internally to replace the -USB-SSD for D3. - -### 3.2 NFS exports from tsys5 (secondary storage server) +### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub) | Export | Disk model | Type | Bus | Total | Used | Free | Use% | |--------|-----------|------|-----|-------|------|------|------| -| S1 | Seagate ST1000VN | HDD | SATA | 916 GB | 53 GB | 817 GB | 7% | -| S2 | Seagate ST1000VN | HDD | SATA | 916 GB | 2 MB | **870 GB** | **0%** | -| S3 | Seagate ST1000VN | HDD | SATA | 916 GB | 3 MB | **870 GB** | **0%** | -| S4 | Toshiba DT01ACA050 | HDD | SATA | 458 GB | 2 MB | **435 GB** | **0%** | -| T5-SSD | Samsung 860 PRO | **SSD** | SATA | 234 GB | 83 GB | **140 GB** | 35% | +| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% | +| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** | +| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** | +| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** | +| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% | +| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** | + +**tsys5 storage controllers (plenty of free ports):** +- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate), + **5 free** +- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free** +- Intel ICH10 SATA #2 (2-port): **status unknown, likely free** +- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7) **Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server -has 2.2 TB of unused HDD capacity plus 140 GB of unused SSD. This is where -VMs should be relocated to reduce tsys4's blast radius. +has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to +reduce tsys4's blast radius. -T5-SSD (Samsung 860 PRO) is 35% used by ultix-streaming (VM 5111, 288 GB disk). -It has 140 GB free for additional SSD-tier VMs. +**Friday additions:** +1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates + USB 2.0 bottleneck. Becomes the second SSD-tier NFS export. +2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5 + boot disk and HPC scratch. The fastest tier in the fleet. -**New Friday:** PCI NVMe drive being added to tsys5. This will be the fastest -storage tier in the fleet. See section 7.4 for placement recommendation. +After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD +NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the +natural home for latency-sensitive workloads and the k8s StorageClass design +center. ### 3.3 Local storage tiers (per host, with utilization) @@ -179,7 +187,7 @@ NFS-over-HDD and unsuitable for wnode boot disks. |------|-------|------------|----------| | **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd | | **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd | -| **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs | +| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs | | **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** | | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only | | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** | @@ -570,21 +578,23 @@ any single-point failure (host, storage server, or disk). |----|------|---------|------|-----------|-----------| | **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. | | **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. | -| **ucs-01** (108) | tsys1 | **D3 (tsys4 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 is 99% empty. | +| **ucs-01** (108) | tsys1 | **D3 (tsys5 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 moves to tsys5 Friday. | | **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. | **Failure survival matrix (all single-point failures):** | Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result | |---------|-------------|-------------|--------|--------|--------| -| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 down) | **alive** (local) | 3 of 4 alive | -| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D3) | **alive** (local) | 3 of 4 alive | +| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 on tsys5) | **alive** (local) | **3 of 4 alive** | +| tsys5 dies | **alive** (D5) | dies (S3) | dies (D3) | **alive** (local) | 2 of 4 alive (one pair survives) | | tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive | | tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive | -| D2 disk fails | **alive** (D5) | **alive** (S3) | **alive** (D3) | **alive** (local) | **all 4 alive** | +| Any single disk | **all 4 on different disks** | **all 4 alive** | | | | -Note: ucs-01 on D3 goes down if tsys4 dies (D3 is on tsys4), but ucs-02 on -tsys9 local SSD survives. That is the HA pair working as designed. +Note: if tsys5 dies, ucs-01 (D3) goes down too. But ucs-02 on tsys9 local SSD +survives -- that is the HA pair working. ucs-01 could alternatively go to D2 +on tsys4, but then a tsys4 failure takes down both ucs-01 AND netinfra-01. +The D3-on-tsys5 placement is the best balance. **Why this works:** Every row has at least one netinfra and one UCS alive. The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark. diff --git a/TODO.md b/TODO.md index b3bb66f..f591708 100644 --- a/TODO.md +++ b/TODO.md @@ -138,7 +138,7 @@ wmic path win32_videocontroller get name,adapterram,driverversion 5. Reboot to activate new NIC and NFS nconnect. 6. Run `validate-fixes.sh pfv-tsys4` to confirm. -### tsys5 — plug storage cable + install NVMe +### tsys5 — plug storage cable + install NVMe + relocate D3 SSD 1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC. 2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for @@ -147,10 +147,19 @@ wmic path win32_videocontroller get name,adapterram,driverversion ```bash echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy ``` -4. Install the **PCI NVMe drive**. -5. Format NVMe as local directory storage (see TODO section 3 below). -6. Reboot tsys5 to activate NFS nconnect. -7. Run `validate-fixes.sh pfv-tsys5` to confirm. +4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port: + - Power down tsys4 + - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4 + - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E) + - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports` + - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from + `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path + - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially + empty, so minimal migration) +5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port). +6. Format NVMe as local directory storage (see TODO section 3 below). +7. Reboot tsys5 to activate NFS nconnect. +8. Run `validate-fixes.sh pfv-tsys5` to confirm. ---