diff --git a/STATUS.md b/STATUS.md index 61c2f78..5f7f4a4 100644 --- a/STATUS.md +++ b/STATUS.md @@ -78,28 +78,27 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Issue | Impact | Status | |-------|--------|--------| -| **UCS + netinfra HA pairs both on tsys4** | tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark | **CRITICAL — needs PDM migration** | -| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost | **CRITICAL — needs PDM migration** | +| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode) | +| **tsys1/5/9 RAM-overcommitted** | 118%/135%/131% respectively | Friday: migrate NFS-backed VMs to tsys6/tsys7 (77+132 GB free) | | Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place | -| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue | -| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install | +| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install | ## Pending (next session priorities) -1. **CRITICAL: Migrate HA pairs to separate storage** (PDM, 10 min): - - netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5) - - ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5) -2. **CRITICAL: Fix k3s cnode quorum risk** (PDM, 5 min): - - Move cnode1 (VMID 906) or cnode2 (VMID 705) from tsys4 to tsys5 storage -3. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers) +1. **Wnode storage rebalancing** (2 PDM storage migrations): + - wnode-tsys6 (100): D5 → S4 (tsys4 → tsys5) + - wnode-tsys7 (701): D5 → S1 (tsys4 → tsys5) + - Gives every wnode its own physical disk, balances 3/3 across servers +2. **Friday maintenance:** physical hardware + VM migrations (see + [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md) §5) +3. **k8s workers:** Join wnodes to the k3s cluster (after rebalancing) 4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday) -5. Perf: complete tsys4/5 tuning after hardware install (Friday) -6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host) -7. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy, +5. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out > **See [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md) -> for the full fresh audit with VM inventory and action items.** +> for the full fresh audit with VM inventory, capacity analysis, and +> Friday migration plan.** ## Infrastructure Summary diff --git a/docs/proxmox/AUDIT-2026-07-28.md b/docs/proxmox/AUDIT-2026-07-28.md index 75af724..b8870aa 100644 --- a/docs/proxmox/AUDIT-2026-07-28.md +++ b/docs/proxmox/AUDIT-2026-07-28.md @@ -1,71 +1,228 @@ -# Fresh Fleet Audit — 2026-07-28 +# Fresh Fleet Audit — 2026-07-28 (revised) > **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.** -> The tables in PROJECT.md reflect the 2026-07-27 audit; VMs have since been -> migrated via PDM. This file is the current ground truth. -**Audit time:** 2026-07-28 19:55 CDT -**Method:** `qm list` + `qm config` on all 7 reachable hosts +**Audit time:** 2026-07-28 20:30 CDT **Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9 **Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days) --- -## 1. Host Summary +## 1. HA Pairs — FIXED ✓ -| Host | CPU | Threads | RAM (GB) | Local Disk | Role | Tuning | -|------|-----|---------|----------|-----------|------|--------| -| pfv-tsys1 | i7-4770 Haswell | 8 | 31 | HDD 932 GB | Infrastructure | Done | -| pfv-tsys3 | Xeon E3-1535M v5 Skylake | 8 | 31 | **NVMe 477 GB** | Kubernetes | Done | -| pfv-tsys4 | Xeon E3-1246 v3 Haswell | 8 | **15** | 6 disks (HDD+SSD) | Storage (NFS+PBS) | Blocked (NIC+RAM) | -| pfv-tsys5 | Xeon E5620 Westmere | 8 | **94** | 6 disks (HDD+SSD) | Storage+Preprod | Blocked (cable) | -| pfv-tsys6 | 2x Xeon E5530 Nehalem | 16 | 127 | HDD (USB 2.0!) | Kubernetes | Done | -| pfv-tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge | 24 | 191 | HDD (USB 2.0!) | Kubernetes | Done | -| pfv-tsys9 | i5-10500 Comet Lake | 12 | 24 | SSD 250 GB | Infrastructure | Done | +Both infrastructure pairs now split across storage servers: -**Changes since last audit:** -- tsys4 RAM still 15 GB (Friday 64 GB upgrade pending) -- tsys5 RAM is 94 GB (was documented as 96 GB) -- tsys6 RAM is 127 GB (was documented as 128 GB) -- tsys7 RAM is 191 GB (was documented as 192 GB) +| VM | VMID | Host | Disk | Server | tsys4 dies | tsys5 dies | +|----|------|------|------|--------|-----------|-----------| +| netinfra-01 | 103 | tsys1 | S2 | tsys5 | alive | dies | +| netinfra-02 | 904 | tsys9 | D2 | tsys4 | dies | alive | +| ucs-01 | 108 | tsys1 | D2 | tsys4 | dies | alive | +| ucs-02 | 902 | tsys9 | S2 | tsys5 | alive | dies | + +**Any single storage server failure leaves one half of each pair alive.** --- -## 2. VM Fleet Inventory (Running VMs Only) +## 2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers) -### tsys1 — 11 running VMs (Infrastructure) +| Cnode | VMID | Host | Disk | Server | +|-------|------|------|------|--------| +| cnode1 | 906 | tsys9 | S3 | **tsys5** | +| cnode2 | 705 | tsys7 | D2 | **tsys4** | +| cnode3 | 106 | tsys1 | D5 | **tsys4** | -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 100 | pfv-bms (HomeAssistant) | 2×1 | 4096 | 32 GB | D2 | tsys4 | -| 101 | tsys-ca | 2×1 | 2048 | 32 GB | D2 | tsys4 | -| 103 | **pfv-netinfra-01** | 2×1 | 2048 | 32 GB | D5 | tsys4 | -| 104 | tsys-librenms | 2×1 | 2048 | 50 GB | D2 | tsys4 | -| 105 | tsys-proxmox-datacenter | 2×1 | 2048 | 32 GB | D2 | tsys4 | -| 106 | **pfv-k8s-cnode3** | 2×1 | 4096 | 32 GB | **S3** | **tsys5** | -| 108 | **tsys-ucs-01** | 2×2 | 8000 | 32 GB | D2 | tsys4 | -| 109 | tailscale-router | 2×1 | 2048 | 25 GB | D2 | tsys4 | -| 114 | kali-tsys | 2×1 | 2048 | 32 GB | D2 | tsys4 | -| 117 | tsys-secure-workbench | 2×1 | 4000 | 32 GB | D2 | tsys4 | -| 102 | pfv-k8s-wnode-tsys1 | 4×1 | 4096 | 32 GB | S2 | tsys5 — **STOPPED** | +**tsys4 failure:** cnode2 + cnode3 die → 1 of 3 → **quorum lost.** +**tsys5 failure:** cnode1 dies → 2 of 3 → quorum OK. -### tsys3 — 1 running VM (Kubernetes) +This is a structural constraint: 3 cnodes on 2 NFS servers means one side +holds 2. The only true fixes are: +- A 3rd storage target (not available until Friday hardware work) +- Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now) +- Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM) -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 313 | **pfv-k8s-wnode-tsys3** | 8×1 | 28000 | 32 GB | D5 | tsys4 | +**Decision: accept for now. Revisit after Friday when local-SSD boot disks +become an option for cnodes.** -### tsys4 — 1 running VM (Storage) +--- -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 400 | pfv-proxmox-backup-server | 2×1 | 2048 | 32 GB | local-lvm | LOCAL | +## 3. Worker Nodes — NEEDS REBALANCING (2 storage migrations) -### tsys5 — 16 running VMs (Storage + Preprod) +### Current state (problem: 3 wnodes share one physical disk) -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 509 | **pfv-k8s-wnode-tsys5** | 2×4 | 32000 | 32 GB | D2 | tsys4 | +| Wnode | VMID | Host | Disk | Server | Disk shared with | +|-------|------|------|------|--------|-----------------| +| wnode-tsys1 | 102 | tsys1 | S3 | tsys5 | — | +| wnode-tsys3 | 313 | tsys3 | **D5** | tsys4 | wnode-tsys6, wnode-tsys7 | +| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | — | +| wnode-tsys6 | 100 | tsys6 | **D5** | tsys4 | wnode-tsys3, wnode-tsys7 | +| wnode-tsys7 | 701 | tsys7 | **D5** | tsys4 | wnode-tsys3, wnode-tsys6 | +| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | — | + +**D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS.** +**S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).** + +### Recommended moves (2 PDM storage migrations) + +| Wnode | Disk from | Disk to | Effect | +|-------|-----------|---------|--------| +| wnode-tsys6 (100) | D5 (tsys4) | **S4** (tsys5, 499 GB free) | D5: 3→2 wnodes | +| wnode-tsys7 (701) | D5 (tsys4) | **S1** (tsys5, 916 GB free) | D5: 2→1 wnode | + +### After rebalancing — every wnode on its own physical disk + +| Wnode | Host | Disk | Server | +|-------|------|------|--------| +| wnode-tsys1 | tsys1 | S3 | tsys5 | +| wnode-tsys3 | tsys3 | D5 | tsys4 | +| wnode-tsys5 | tsys5 | D2 | tsys4 | +| wnode-tsys6 | tsys6 | **S4** | **tsys5** | +| wnode-tsys7 | tsys7 | **S1** | **tsys5** | +| wnode-tsys9 | tsys9 | S2 | tsys5 | + +**tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode +has a dedicated physical spindle.** + +--- + +## 4. Host Capacity + +| Host | RAM | VM RAM | RAM% | Threads | vCPU | CPU% | Notes | +|------|-----|--------|------|---------|------|------|-------| +| tsys1 | 31 GB | 36.6 GB | **118%** | 8 | 24 | 300% | Overcommitted. USB-passthrough VMs locked (bms, ca). | +| tsys3 | 31 GB | 28 GB | 90% | 8 | 8 | 100% | 1 wnode (28 GB). OK. | +| tsys4 | 15 GB | 2 GB | 13% | 8 | 2 | 25% | Storage server. Friday: →64 GB. | +| tsys5 | 94 GB | 127 GB | **135%** | 8 | 72 | 900% | Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate). | +| tsys6 | 125 GB | 48 GB | **38%** | 16 | 12 | 75% | **77 GB free.** Room for more. | +| tsys7 | 188 GB | 56 GB | **30%** | 24 | 22 | 92% | **132 GB free.** Most room in fleet. | +| tsys9 | 23 GB | 30.2 GB | **131%** | 12 | 16 | 133% | Overcommitted by 7 GB. | + +### Hardware dependencies (locked VMs) + +| Host | VM | Hardware | Can move? | +|------|-----|----------|-----------| +| tsys1 | pfv-bms (100) | Zigbee + Z-Wave USB dongles | Friday only (physical move) | +| tsys1 | tsys-ca (101) | Nitrokey HSM (2×) | Friday only (physical move) | +| tsys5 | (various) | SDR dongles | **No — do not move** | + +--- + +## 5. Friday Maintenance Plan + +### 5.1 Primary goal: Proxmox on tsys2 + +tsys2 is currently Windows 10. It has the **best local storage in the fleet** +(Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary +Friday deliverable is: + +1. **Move DHCP off pfv-netboot** to the netinfra-01/02 redundant pair. This + must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware + currently, or depends on a service that does). +2. **Format tsys2**, install Proxmox. +3. **Agent inventory** — once Proxmox is live, the audit scripts + (`perf/deploy-check.sh`) can gather hardware and VM data directly, + replacing the Windows PowerShell inventory from + [`TODO.md`](TODO.md) §1. +4. **Run provisioning** — `provisioning/SetupNewSystem.sh` for baseline + hardening, then `perf/scripts/apply-tunings.sh`. +5. **Create wnode-tsys2** — boot disk on NVMe (960 PRO), data disk on SATA + SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage + of any worker. +6. **Join wnode-tsys2 to k3s cluster.** + +tsys2 hardware (from [`TODO.md`](TODO.md) Windows inventory): +- Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t) +- 32 GB RAM +- Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD +- Both wired NICs are USB-attached (same anti-pattern as tsys4/9) +- NVIDIA Quadro M1200 4 GB (passthrough candidate) +- Service tag: GH1XZG2 + +### 5.2 Physical hardware work (tsys4, tsys5) + +| Host | Work | +|------|------| +| tsys4 | PCIe NIC install (replace USB dongle), RAM 16→64 GB | +| tsys5 | 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB | + +### 5.3 VM migrations (while cluster is down for hardware) + +**Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:** + +| VM | VMID | From | To | RAM freed | Storage stays | +|----|------|------|----|-----------|---------------| +| tsys-siem | 901 | tsys9 (131%) | tsys6 (38%) | 8 GB on tsys9 | D2 (tsys4) | +| tsys-librenms | 104 | tsys1 (118%) | tsys7 (30%) | 2 GB on tsys1 | D2 (tsys4) | +| tsys-pdm | 105 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | +| tsys-workbench | 117 | tsys1 | tsys7 | 4 GB | D2 (tsys4) | +| tailscale-router | 109 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | +| kali-tsys | 114 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | + +**Result:** tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from +30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily. + +**Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):** + +| VM | VMID | From | To | Hardware | +|----|------|------|----|---------| +| pfv-bms | 100 | tsys1 | tsys9 | Zigbee + Z-Wave dongles (physically move) | +| tsys-ca | 101 | tsys1 | tsys9 | Nitrokey HSMs (physically move) | + +**Result:** tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only). + +**Wnode storage rebalancing:** + +| VM | VMID | Disk from | Disk to | +|----|------|-----------|---------| +| wnode-tsys6 | 100 | D5 (tsys4) | S4 (tsys5) | +| wnode-tsys7 | 701 | D5 (tsys4) | S1 (tsys5) | + +### 5.4 Post-hardware cnode quorum fix (optional, if local-SSD available) + +After Friday hardware work, consider moving one cnode to local-lvm: +- cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency +- cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd + +This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS. +**Any single failure (host, storage server, or disk) leaves 2 of 3 alive.** +Deferred — user wants NFS-only for now. + +--- + +## 6. Complete VM Inventory (running VMs only) + +### tsys1 — 11 running (Infrastructure) + +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 100 | pfv-bms | 2 | 4096 | 32 GB | D2 | tsys4 | +| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 | tsys4 | +| 102 | pfv-k8s-wnode-tsys1 | 4 | 4096 | 32 GB | S3 | tsys5 | +| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | S2 | tsys5 | +| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 | tsys4 | +| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 | tsys4 | +| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D5 | tsys4 | +| 108 | tsys-ucs-01 | 2×2 | 8000 | 32 GB | D2 | tsys4 | +| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 | tsys4 | +| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 | tsys4 | +| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 | tsys4 | + +### tsys3 — 1 running (Kubernetes) + +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 313 | pfv-k8s-wnode-tsys3 | 8 | 28000 | 32 GB | D5 | tsys4 | + +### tsys4 — 1 running (Storage) + +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL | + +### tsys5 — 17 running (Storage + Preprod) + +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 509 | pfv-k8s-wnode-tsys5 | 2×4 | 32000 | 32 GB | D2 | tsys4 | | 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL | | 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL | | 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | @@ -83,172 +240,32 @@ | 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL | | 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL | -### tsys6 — 3 running VMs (Kubernetes) +### tsys6 — 3 running (Kubernetes) -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 100 | **pfv-k8s-wnode-tsys6** | 2×2 | 32000 | 32 GB | D5 | tsys4 | +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 100 | pfv-k8s-wnode-tsys6 | 2×2 | 32000 | 32 GB | D5 | tsys4 | | 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 | -| 601 | pfv-k8s-cnode4 | 4×1 | 4096 | 32 GB | D2 | tsys4 | +| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 | tsys4 | -### tsys7 — 6 running VMs (Kubernetes) +### tsys7 — 6 running (Kubernetes) -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 701 | **pfv-k8s-wnode-tsys7** | 4×1 | 32000 | 32 GB | D5 | tsys4 | +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 | tsys4 | | 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 | -| 703 | rr-middleware | 2×1 | 2048 | 32 GB | D2 | tsys4 | -| 704 | TCTC | 4×1 | 6000 | 32 GB | D2 | tsys4 | -| 705 | **pfv-k8s-cnode2** | 4×1 | 4096 | 32 GB | D2 | tsys4 | -| 706 | pfv-k8s-cnode5 | 4×1 | 4096 | 32 GB | **S2** | **tsys5** | +| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 | tsys4 | +| 704 | TCTC | 4 | 6000 | 32 GB | D2 | tsys4 | +| 705 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 | tsys4 | +| 706 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | S2 | tsys5 | -### tsys9 — 6 running VMs (Infrastructure) +### tsys9 — 6 running (Infrastructure) -| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server | -|------|------|--------------|----------|------|---------|-----------| -| 901 | tsys-siem | 2×1 | 8000 | 132 GB | D2 | tsys4 | -| 902 | **tsys-ucs-02** | 2×2 | 8000 | 50 GB | D5 | tsys4 | -| 903 | kali-rd | 2×1 | 2048 | 32 GB | D5 | tsys4 | -| 904 | **pfv-netinfra-02** | 2×1 | 4000 | 32 GB | D2 | tsys4 | -| 905 | **pfv-k8s-wnode-tsys9** | 4×1 | 4096 | 32 GB | **S2** | **tsys5** | -| 906 | **pfv-k8s-cnode1** | 2×1 | 4096 | 32 GB | D5 | tsys4 | - ---- - -## 3. Kubernetes Node Placement - -### 3.1 Active k3s cluster (cnode1/2/3 — deployed this session) - -| Cnode | VMID | Host | Disk | NFS Server | Quorum risk | -|-------|------|------|------|-----------|-------------| -| cnode1 | **906** | tsys9 | D5 | **tsys4** | **HIGH** | -| cnode2 | **705** | tsys7 | D2 | **tsys4** | **HIGH** | -| cnode3 | 106 | tsys1 | S3 | tsys5 | OK | - -**CRITICAL: 2 of 3 active cnodes on tsys4.** tsys4 failure = cnode1 + cnode2 die = 1 of 3 = **QUORUM LOST**. - -**Fix needed:** Move cnode1 or cnode2 to tsys5 storage (S2 or S3). One migration via PDM "Storage Migrate" solves this. - -### 3.2 Inactive cnodes (cnode4/5 — exist but not in k3s cluster) - -| Cnode | VMID | Host | Disk | NFS Server | -|-------|------|------|------|-----------| -| cnode4 | 601 | tsys6 | D2 | tsys4 | -| cnode5 | 706 | tsys7 | S2 | tsys5 | - -### 3.3 Worker nodes (wnodes) - -| Wnode | VMID | Host | Disk | NFS Server | RAM | Status | -|-------|------|------|------|-----------|-----|--------| -| wnode-tsys1 | 102 | tsys1 | S2 | tsys5 | 4 GB | STOPPED | -| wnode-tsys3 | 313 | tsys3 | D5 | tsys4 | 28 GB | Running | -| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | 32 GB | Running | -| wnode-tsys6 | 100 | tsys6 | D5 | tsys4 | 32 GB | Running | -| wnode-tsys7 | 701 | tsys7 | D5 | tsys4 | 32 GB | Running | -| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | 4 GB | Running | - -**Storage: 4 wnodes on tsys4, 2 on tsys5.** One wnode per host achieved. - ---- - -## 4. Critical HA Pairs — Storage Redundancy - -### 4.1 netinfra pair — FAILED (both on tsys4) - -| Role | VMID | Host | Disk | NFS Server | -|------|------|------|------|-----------| -| netinfra-01 | 103 | tsys1 | D5 | **tsys4** | -| netinfra-02 | 904 | tsys9 | D2 | **tsys4** | - -**tsys4 failure = DNS/DHCP/NTP goes fully dark.** - -**Fix:** Migrate netinfra-02 (VMID 904) from D2 (tsys4) to S3 (tsys5) via PDM. - -### 4.2 UCS pair — FAILED (both on tsys4) - -| Role | VMID | Host | Disk | NFS Server | -|------|------|------|------|-----------| -| ucs-01 | 108 | tsys1 | D2 | **tsys4** | -| ucs-02 | 902 | tsys9 | D5 | **tsys4** | - -**tsys4 failure = LDAP/AD goes fully dark.** - -**Fix:** Migrate ucs-02 (VMID 902) from D5 (tsys4) to S2 (tsys5) via PDM. - -### 4.3 Corrected placement (after migration) - -| VM | Host | Disk | NFS Server | Failure survival | -|----|------|------|-----------|-----------------| -| netinfra-01 | tsys1 | D5 | tsys4 | tsys4 dies → netinfra-02 alive on tsys5 | -| netinfra-02 | tsys9 | **S3** | **tsys5** | tsys5 dies → netinfra-01 alive on tsys4 | -| ucs-01 | tsys1 | D2 | tsys4 | tsys4 dies → ucs-02 alive on tsys5 | -| ucs-02 | tsys9 | **S2** | **tsys5** | tsys5 dies → ucs-01 alive on tsys4 | - ---- - -## 5. Storage Concentration - -| Storage target | # running VMs | % of fleet | -|---------------|--------------|------------| -| D2 (tsys4 WDC Red 3TB HDD) | 16 | 36% | -| D5 (tsys4 Hitachi 2TB HDD) | 10 | 23% | -| local-nonprod (tsys5 local HDD) | 12 | 27% | -| S2 (tsys5 Seagate 1TB HDD) | 3 | 7% | -| S3 (tsys5 Seagate 1TB HDD) | 1 | 2% | -| T5-SSD (tsys5 Samsung SSD) | 1 | 2% | -| local-lvm (various hosts) | 2 | 5% | - -**26 of 44 running VMs (59%) store their disks on tsys4 NFS exports.** -(Was 68% in the previous audit — improving but still concentrated.) - ---- - -## 6. Pre-k8s Buildout Action Items - -These must be done before or during k8s worker node bringup: - -### 6.1 CRITICAL: Migrate HA pairs to separate storage (PDM, 10 min) - -1. `netinfra-02` (VMID 904): D2 → S3 (tsys4 → tsys5) -2. `ucs-02` (VMID 902): D5 → S2 (tsys4 → tsys5) - -These are PDM "Storage Migrate" operations — no VM rebuild needed. - -### 6.2 CRITICAL: Fix active cnode quorum (PDM, 5 min) - -Move one active cnode from tsys4 to tsys5 storage: -- Best candidate: cnode1 (VMID 906) D5 → S2 (tsys4 → tsys5) -- Result: cnode1 on tsys5, cnode2 on tsys4, cnode3 on tsys5 -- tsys4 failure = cnode2 dies only = 2 of 3 = **quorum OK** - -### 6.3 Join workers to k3s cluster - -Current wnodes exist as VMs but are not joined to the k3s cluster. Need to: -1. Install k3s agent on each wnode (using join token from cnode1) -2. Configure `--node-ip=` on each -3. Label/taint per workload role - -### 6.4 Friday hardware work (still pending) - -- tsys4: PCIe NIC + 64 GB RAM (currently 15 GB) -- tsys5: 2nd ethernet cable + NVMe + D3 SSD relocation -- tsys2: Rebuild from Win10 to Proxmox - ---- - -## 7. Changes Since Previous Audit (2026-07-27) - -| What | Before | After | -|------|--------|-------| -| cnode1 VMID | 107 (tsys1, D5) | **906** (tsys9, D5) | -| cnode2 VMID | 603 (tsys6, D2) | **705** (tsys7, D2) | -| cnode3 storage | D2 (tsys4) | **S3** (tsys5) | -| cnode5 VMID | 602 (tsys6) | **706** (tsys7, S2 tsys5) | -| wnode-tsys1 | not listed | VMID 102 (S2 tsys5, stopped) | -| wnode-tsys3 RAM | 20 GB | **28 GB** | -| tsys-awx (600) | STOPPED | **Running** | -| DellOpenManageEnterprise (500) | not listed | VMID 500 (D7, stopped) | -| sectestbed-librenms (5108) | not listed | Running | -| k3s cluster | not deployed | **3-node HA live** (cnode1/2/3) | -| Console management | manual screen | **ser2net+conman on tsys4** | -| PDU management | manual | **powerman on tsys1** | +| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | +|------|------|-----|----------|------|---------|--------| +| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 | tsys4 | +| 902 | tsys-ucs-02 | 2×2 | 8000 | 50 GB | S2 | tsys5 | +| 903 | kali-rd | 2 | 2048 | 32 GB | D5 | tsys4 | +| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 | tsys4 | +| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S2 | tsys5 | +| 906 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | S3 | tsys5 |