KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday

Browse Source 
Deep-dive on tsys4/5 storage controllers revealed:
- tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0
- tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0

Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB
bottleneck and consolidates all fast storage on tsys5: NVMe (local) +
D3 SSD + T5-SSD + 4 HDD exports.

Updated storage architecture, speed tiers, critical VM placement,
failure survival matrix, StorageClass design, and Friday plan.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-27 11:50:16 -05:00
parent ea2048dc87
commit 019ab468bf
4 changed files with 79 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+14 -5
View File
@@ -76,17 +76,26 @@ tsys1=.1 tsys3=.3 tsys4=.4 tsys5=.5 tsys6=.6 tsys7=.7 tsys9=.9
## NFS export topology ## NFS export topology
**tsys4 exports (primary, 68% of VMs depend on it):** **tsys4 exports (primary, overloaded):**
- D2 = WDC Red 3TB HDD — most VMs live here - D2 = WDC Red 3TB HDD — most VMs live here
- D3 = SK hynix 512GB SSD (USB-attached on tsys4)
- D5 = Hitachi 2TB HDD - D5 = Hitachi 2TB HDD
- (sda Hitachi 1.8T at /mnt/albert — not NFS shared)
- (sdd WDC 1T — idle, unmounted, removable)
- (sdf WDC 4.5T SMR at /mnt/backup — PBS target)
**tsys5 exports (secondary, barely used by k8s):** **tsys5 exports (fast-tier hub -- consolidated Friday):**
- S1/S2/S3 = Seagate 1TB HDD each - S1/S2/S3 = Seagate 1TB HDD each
- S4 = Toshiba 500GB HDD - S4 = Toshiba 500GB HDD
- T5-SSD = Samsung 860 PRO 256GB SSD - T5-SSD = Samsung 860 PRO 256GB SSD (existing)
- **D3 = SK hynix SC300 512GB SSD** (moving from tsys4 USB to tsys5 SAS Friday)
- **NVMe (local, new Friday)** — local-only, wnode-tsys5 boot + HPC scratch
**New Friday:** tsys5 NVMe (recommend local-only, not NFS-exported). **tsys5 hardware:** LSI SAS1068E (8-port, 5 free) + Intel ICH10 SATA (6-port,
2 free) + 2x Renesas USB 3.0 xHCI. Plenty of room for the SSD + NVMe.
**Friday change:** D3 export repoints from pfv-tsys4-nfs-stor to
pfv-tsys5-nfs-stor. Update storage.cfg cluster-wide. VMs on D3 (currently
none of significance — it's 99% empty) keep working after remount.
## SSH access ## SSH access
K8S.md
+11 -6
View File
@@ -326,13 +326,18 @@ spec:
storage: 50Gi storage: 50Gi
``` ```
### 6.3 NFS-SSD tier (D3 and T5-SSD) ### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
The SSD NFS exports are a middle tier. Use cases: The SSD NFS exports are a middle tier. **Both consolidated on tsys5** after
- **D3 (tsys4, 445 GB free):** CI/CD artifacts, container image cache for the Friday D3 relocation:
sectestbed nodes. Currently 99% empty. - **D3 (tsys5 SAS, 445 GB free):** CI/CD artifacts, container image cache,
- **T5-SSD (tsys5, 140 GB free):** SIEM databases, monitoring data. Currently UCS/LDAP. Currently 99% empty. Fastest NFS tier after the move off USB.
35% used by ultix-streaming. - **T5-SSD (tsys5 SAS, 140 GB free):** SIEM databases, monitoring data.
Currently 35% used by ultix-streaming.
**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
This simplifies the StorageClass design -- latency-sensitive PVCs target
tsys5 exports, bulk PVCs target either server.
### 6.4 NFS data distribution across storage servers ### 6.4 NFS data distribution across storage servers
PROJECT.md
+40 -30
View File
@@ -86,43 +86,51 @@ running Cloudron -- not in this cluster.
| Export | Disk model | Type | Bus | Total | Used | Free | Use% | | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------| |--------|-----------|------|-----|-------|------|------|------|
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% | | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
| D3 | SK hynix SC300 | **SSD** | **USB** | 469 GB | 2 MB | **445 GB** | **0%** | | ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% | | D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
Non-exported disks on tsys4: Non-exported disks on tsys4:
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
- sdd (WDC 1T) -- not mounted, idle - sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free - sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
**Key finding: D3 (SSD, 445 GB) is 99% empty.** Nobody is using the only SSD **D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
export on the primary storage server. This is the fastest available NFS tier tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
from tsys4 and should be used for latency-sensitive VMs (UCS/LDAP, etcd). eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
/sdc/sdd), so tsys5 is the better target. See section 3.2.
**Warning: D3 is USB-attached.** The SK hynix SSD is connected via a USB adapter ### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
(`usb-SSD_ThinkPlus`). It achieves SSD speeds but carries the same USB
reliability risk as the cdc_ncm NIC dongles. After tsys4 gets its PCIe NIC
(Friday), consider adding a proper SATA/SAS SSD internally to replace the
USB-SSD for D3.
### 3.2 NFS exports from tsys5 (secondary storage server)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% | | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------| |--------|-----------|------|-----|-------|------|------|------|
| S1 | Seagate ST1000VN | HDD | SATA | 916 GB | 53 GB | 817 GB | 7% | | S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SATA | 916 GB | 2 MB | **870 GB** | **0%** | | S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** |
| S3 | Seagate ST1000VN | HDD | SATA | 916 GB | 3 MB | **870 GB** | **0%** | | S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** |
| S4 | Toshiba DT01ACA050 | HDD | SATA | 458 GB | 2 MB | **435 GB** | **0%** | | S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
| T5-SSD | Samsung 860 PRO | **SSD** | SATA | 234 GB | 83 GB | **140 GB** | 35% | | T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% |
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
**tsys5 storage controllers (plenty of free ports):**
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
**5 free**
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server **Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
has 2.2 TB of unused HDD capacity plus 140 GB of unused SSD. This is where has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to
VMs should be relocated to reduce tsys4's blast radius. reduce tsys4's blast radius.
T5-SSD (Samsung 860 PRO) is 35% used by ultix-streaming (VM 5111, 288 GB disk). **Friday additions:**
It has 140 GB free for additional SSD-tier VMs. 1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
boot disk and HPC scratch. The fastest tier in the fleet.
**New Friday:** PCI NVMe drive being added to tsys5. This will be the fastest After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
storage tier in the fleet. See section 7.4 for placement recommendation. NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
natural home for latency-sensitive workloads and the k8s StorageClass design
center.
### 3.3 Local storage tiers (per host, with utilization) ### 3.3 Local storage tiers (per host, with utilization)
@@ -179,7 +187,7 @@ NFS-over-HDD and unsuitable for wnode boot disks.
|------|-------|------------|----------| |------|-------|------------|----------|
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd | | **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd | | **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
| **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs | | **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs |
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** | | **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only | | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** | | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
@@ -570,21 +578,23 @@ any single-point failure (host, storage server, or disk).
|----|------|---------|------|-----------|-----------| |----|------|---------|------|-----------|-----------|
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. | | **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. |
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. | | **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
| **ucs-01** (108) | tsys1 | **D3 (tsys4 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 is 99% empty. | | **ucs-01** (108) | tsys1 | **D3 (tsys5 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 moves to tsys5 Friday. |
| **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. | | **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. |
**Failure survival matrix (all single-point failures):** **Failure survival matrix (all single-point failures):**
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result | | Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|---------|-------------|-------------|--------|--------|--------| |---------|-------------|-------------|--------|--------|--------|
| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 down) | **alive** (local) | 3 of 4 alive | | tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 on tsys5) | **alive** (local) | **3 of 4 alive** |
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D3) | **alive** (local) | 3 of 4 alive | | tsys5 dies | **alive** (D5) | dies (S3) | dies (D3) | **alive** (local) | 2 of 4 alive (one pair survives) |
| tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive | | tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive |
| tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive | | tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive |
| D2 disk fails | **alive** (D5) | **alive** (S3) | **alive** (D3) | **alive** (local) | **all 4 alive** | | Any single disk | **all 4 on different disks** | **all 4 alive** | | | |
Note: ucs-01 on D3 goes down if tsys4 dies (D3 is on tsys4), but ucs-02 on Note: if tsys5 dies, ucs-01 (D3) goes down too. But ucs-02 on tsys9 local SSD
tsys9 local SSD survives. That is the HA pair working as designed. survives -- that is the HA pair working. ucs-01 could alternatively go to D2
on tsys4, but then a tsys4 failure takes down both ucs-01 AND netinfra-01.
The D3-on-tsys5 placement is the best balance.
**Why this works:** Every row has at least one netinfra and one UCS alive. **Why this works:** Every row has at least one netinfra and one UCS alive.
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark. The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
TODO.md
+14 -5
View File
@@ -138,7 +138,7 @@ wmic path win32_videocontroller get name,adapterram,driverversion
5. Reboot to activate new NIC and NFS nconnect. 5. Reboot to activate new NIC and NFS nconnect.
6. Run `validate-fixes.sh pfv-tsys4` to confirm. 6. Run `validate-fixes.sh pfv-tsys4` to confirm.
### tsys5 — plug storage cable + install NVMe ### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC. 1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for 2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
@@ -147,10 +147,19 @@ wmic path win32_videocontroller get name,adapterram,driverversion
```bash ```bash
echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
``` ```
4. Install the **PCI NVMe drive**. 4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
5. Format NVMe as local directory storage (see TODO section 3 below). - Power down tsys4
6. Reboot tsys5 to activate NFS nconnect. - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
7. Run `validate-fixes.sh pfv-tsys5` to confirm. - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
- On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
- Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
`pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
- Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
empty, so minimal migration)
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
6. Format NVMe as local directory storage (see TODO section 3 below).
7. Reboot tsys5 to activate NFS nconnect.
8. Run `validate-fixes.sh pfv-tsys5` to confirm.
--- ---