diff --git a/PROJECT.md b/PROJECT.md index c369603..f5cc888 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -192,7 +192,19 @@ NFS-over-HDD and unsuitable for wnode boot disks. | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only | | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** | -### 3.5 All exports are single-disk with no redundancy +### 3.5 Storage tier characterization per host + +| Host | Storage profile | Detail | +|------|----------------|--------| +| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. | +| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. | +| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD | +| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. | +| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client | +| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client | +| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. | + +### 3.6 All exports are single-disk with no redundancy Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS mirror, no mdraid. A single disk failure takes down every VM whose disk lives @@ -576,29 +588,29 @@ any single-point failure (host, storage server, or disk). | VM | Host | Storage | Tier | Free space | Rationale | |----|------|---------|------|-----------|-----------| -| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. | +| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. | | **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. | -| **ucs-01** (108) | tsys1 | **D3 (tsys5 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 moves to tsys5 Friday. | -| **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. | +| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. | +| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. | **Failure survival matrix (all single-point failures):** | Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result | |---------|-------------|-------------|--------|--------|--------| -| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 on tsys5) | **alive** (local) | **3 of 4 alive** | -| tsys5 dies | **alive** (D5) | dies (S3) | dies (D3) | **alive** (local) | 2 of 4 alive (one pair survives) | -| tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive | -| tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive | -| Any single disk | **all 4 on different disks** | **all 4 alive** | | | | - -Note: if tsys5 dies, ucs-01 (D3) goes down too. But ucs-02 on tsys9 local SSD -survives -- that is the HA pair working. ucs-01 could alternatively go to D2 -on tsys4, but then a tsys4 failure takes down both ucs-01 AND netinfra-01. -The D3-on-tsys5 placement is the best balance. +| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive | +| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive | +| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives | +| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives | +| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | | **Why this works:** Every row has at least one netinfra and one UCS alive. The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark. +**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5 +Friday) is freed for latency-sensitive workloads that actually benefit from +SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image +registry mirror. Not infrastructure VMs. + ### 10.2 CRITICAL: Cnode (control plane) storage split **Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd diff --git a/TODO.md b/TODO.md index f591708..2ad1ca0 100644 --- a/TODO.md +++ b/TODO.md @@ -206,7 +206,31 @@ After the NVMe is physically installed and visible in Proxmox: --- -## 5. Open questions for next session +## 5. UCS storage migration to spinning disk (do today) + +UCS (Univention Corporate Server / open-source AD) does not need SSD. Both +UCS VMs should stay on spinning disk (HDD) and be split across storage +servers for redundancy. + +| VM | Current | Target | Action | +|----|---------|--------|--------| +| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct | +| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy | + +**To migrate ucs-02 to S2:** +1. Shut down VM 902 on tsys9 +2. Copy disk: `cp /mnt/pve/D5/902/vm-902-disk-0.qcow2 /mnt/pve/S2/902/` +3. Update VM 902 config: change `scsi0` from `D5:902/...` to `S2:902/...` +4. Start VM 902, verify it boots and LDAP/AD services are healthy +5. Remove old disk from D5 + +Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only +ucs-02 needs to move -- it should be on a different storage server than +ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair. + +--- + +## 6. Open questions for next session - Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs? Determines whether live migration is available.