KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: keep UCS on spinning disk, repurpose D3 SSD for k8s/CI workloads

Browse Source 
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
  redundancy only, not for speed

D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.

Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-27 11:51:30 -05:00
parent 019ab468bf
commit 6814374bac
2 changed files with 51 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT.md
+26 -14
View File
@@ -192,7 +192,19 @@ NFS-over-HDD and unsuitable for wnode boot disks.
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
### 3.5 All exports are single-disk with no redundancy
### 3.5 Storage tier characterization per host
| Host | Storage profile | Detail |
|------|----------------|--------|
| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
### 3.6 All exports are single-disk with no redundancy
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
@@ -576,29 +588,29 @@ any single-point failure (host, storage server, or disk).
| VM | Host | Storage | Tier | Free space | Rationale |
|----|------|---------|------|-----------|-----------|
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. |
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
| **ucs-01** (108) | tsys1 | **D3 (tsys5 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 moves to tsys5 Friday. |
| **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. |
| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
**Failure survival matrix (all single-point failures):**
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|---------|-------------|-------------|--------|--------|--------|
| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 on tsys5) | **alive** (local) | **3 of 4 alive** |
| tsys5 dies | **alive** (D5) | dies (S3) | dies (D3) | **alive** (local) | 2 of 4 alive (one pair survives) |
| tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive |
| tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive |
| Any single disk | **all 4 on different disks** | **all 4 alive** | | | |
Note: if tsys5 dies, ucs-01 (D3) goes down too. But ucs-02 on tsys9 local SSD
survives -- that is the HA pair working. ucs-01 could alternatively go to D2
on tsys4, but then a tsys4 failure takes down both ucs-01 AND netinfra-01.
The D3-on-tsys5 placement is the best balance.
| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
**Why this works:** Every row has at least one netinfra and one UCS alive.
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
Friday) is freed for latency-sensitive workloads that actually benefit from
SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
registry mirror. Not infrastructure VMs.
### 10.2 CRITICAL: Cnode (control plane) storage split
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
TODO.md
+25 -1
View File
@@ -206,7 +206,31 @@ After the NVMe is physically installed and visible in Proxmox:
---
## 5. Open questions for next session
## 5. UCS storage migration to spinning disk (do today)
UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
UCS VMs should stay on spinning disk (HDD) and be split across storage
servers for redundancy.
| VM | Current | Target | Action |
|----|---------|--------|--------|
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
**To migrate ucs-02 to S2:**
1. Shut down VM 902 on tsys9
2. Copy disk: `cp /mnt/pve/D5/902/vm-902-disk-0.qcow2 /mnt/pve/S2/902/`
3. Update VM 902 config: change `scsi0` from `D5:902/...` to `S2:902/...`
4. Start VM 902, verify it boots and LDAP/AD services are healthy
5. Remove old disk from D5
Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
ucs-02 needs to move -- it should be on a different storage server than
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
---
## 6. Open questions for next session
- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
Determines whether live migration is available.