diff --git a/STATUS.md b/STATUS.md index 8d67985..2de0f14 100644 --- a/STATUS.md +++ b/STATUS.md @@ -35,10 +35,13 @@ infrastructure, Proxmox cluster ops, and k8s control plane. ### Proxmox Cluster Ops - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9) -- [x] Fleet audit refreshed 2026-07-29 (live data, all 7 hosts) -- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 72% +- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected) +- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73% - [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model) -- [ ] CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware) +- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg +- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending) +- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware) +- [ ] **BUG: tsys6 S1 export path typo** (`S21` instead of `S1`) — S1 broken on tsys6 - [ ] tsys4: blocked on PCIe NIC + RAM install - [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty) - [ ] tsys2: pending rebuild from Win10 to Proxmox @@ -89,7 +92,7 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** | | **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 72%/37%/60% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §8 | | **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) | -| **cnode4/5 stale VMs running** | Not joined to the 3-node cluster; waste 8 GB + 8 vCPU | Stop + delete (AUDIT-2026-07-29 §10) | +| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ | | **Fleet CPU 0-6% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads | | Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place | | tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install | @@ -107,16 +110,15 @@ infrastructure, Proxmox cluster ops, and k8s control plane. 3. **Stop + delete stale cnode4 (601) / cnode5 (706)** — not in the 3-node cluster 4. **Join wnodes to the k3s cluster** (after migrations free resources) 5. **Friday maintenance:** physical hardware + storage work (see - [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5 - and [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md)) + [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md)) 6. **Wnode storage rebalancing** (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1 7. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday) 8. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out -> **See [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) -> for the full fresh audit with VM inventory, capacity analysis, and -> Friday migration plan.** +> **See [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) +> for the consolidated audit with corrected storage topology, VM inventory, +> capacity analysis, and action checklist.** ## Infrastructure Summary diff --git a/docs/docmap.md b/docs/docmap.md index 84081be..3e9e26f 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -24,8 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture. |----------|-------------|---------------| | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU + HDD disk placement audit (current/target inventory + action checklist); live data from all 7 hosts | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 | +| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current & only audit.** Consolidated RAM/CPU + HDD disk placement with corrected storage topology, current/target inventory, action checklist | 2026-07-29 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | diff --git a/proxmox/docs/AUDIT-2026-07-28.md b/proxmox/docs/AUDIT-2026-07-28.md deleted file mode 100644 index 1902939..0000000 --- a/proxmox/docs/AUDIT-2026-07-28.md +++ /dev/null @@ -1,276 +0,0 @@ -# Fresh Fleet Audit — 2026-07-28 (revised) - -> **A newer audit exists:** [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (RAM/CPU -> placement focus, with codified capacity goals). Placement numbers below are -> superseded where the two conflict; this doc remains authoritative for the -> **Friday storage/hardware plan (§5)** and storage-redundancy analysis. - -> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.** - -**Audit time:** 2026-07-28 20:30 CDT -**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9 -**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days) - ---- - -## 1. HA Pairs — FIXED ✓ - -Both infrastructure pairs now split across storage servers: - -| VM | VMID | Host | Disk | Server | tsys4 dies | tsys5 dies | -|----|------|------|------|--------|-----------|-----------| -| netinfra-01 | 103 | tsys1 | S2 | tsys5 | alive | dies | -| netinfra-02 | 904 | tsys9 | D2 | tsys4 | dies | alive | -| ucs-01 | 108 | tsys1 | D2 | tsys4 | dies | alive | -| ucs-02 | 902 | tsys9 | S2 | tsys5 | alive | dies | - -**Any single storage server failure leaves one half of each pair alive.** - ---- - -## 2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers) - -| Cnode | VMID | Host | Disk | Server | -|-------|------|------|------|--------| -| cnode1 | 906 | tsys9 | S3 | **tsys5** | -| cnode2 | 705 | tsys7 | D2 | **tsys4** | -| cnode3 | 106 | tsys1 | D5 | **tsys4** | - -**tsys4 failure:** cnode2 + cnode3 die → 1 of 3 → **quorum lost.** -**tsys5 failure:** cnode1 dies → 2 of 3 → quorum OK. - -This is a structural constraint: 3 cnodes on 2 NFS servers means one side -holds 2. The only true fixes are: -- A 3rd storage target (not available until Friday hardware work) -- Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now) -- Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM) - -**Decision: accept for now. Revisit after Friday when local-SSD boot disks -become an option for cnodes.** - ---- - -## 3. Worker Nodes — NEEDS REBALANCING (2 storage migrations) - -### Current state (problem: 3 wnodes share one physical disk) - -| Wnode | VMID | Host | Disk | Server | Disk shared with | -|-------|------|------|------|--------|-----------------| -| wnode-tsys1 | 102 | tsys1 | S3 | tsys5 | — | -| wnode-tsys3 | 313 | tsys3 | **D5** | tsys4 | wnode-tsys6, wnode-tsys7 | -| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | — | -| wnode-tsys6 | 100 | tsys6 | **D5** | tsys4 | wnode-tsys3, wnode-tsys7 | -| wnode-tsys7 | 701 | tsys7 | **D5** | tsys4 | wnode-tsys3, wnode-tsys6 | -| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | — | - -**D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS.** -**S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).** - -### Recommended moves (2 PDM storage migrations) - -| Wnode | Disk from | Disk to | Effect | -|-------|-----------|---------|--------| -| wnode-tsys6 (100) | D5 (tsys4) | **S4** (tsys5, 499 GB free) | D5: 3→2 wnodes | -| wnode-tsys7 (701) | D5 (tsys4) | **S1** (tsys5, 916 GB free) | D5: 2→1 wnode | - -### After rebalancing — every wnode on its own physical disk - -| Wnode | Host | Disk | Server | -|-------|------|------|--------| -| wnode-tsys1 | tsys1 | S3 | tsys5 | -| wnode-tsys3 | tsys3 | D5 | tsys4 | -| wnode-tsys5 | tsys5 | D2 | tsys4 | -| wnode-tsys6 | tsys6 | **S4** | **tsys5** | -| wnode-tsys7 | tsys7 | **S1** | **tsys5** | -| wnode-tsys9 | tsys9 | S2 | tsys5 | - -**tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode -has a dedicated physical spindle.** - ---- - -## 4. Host Capacity - -| Host | RAM | VM RAM | RAM% | Threads | vCPU | CPU% | Notes | -|------|-----|--------|------|---------|------|------|-------| -| tsys1 | 31 GB | 36.6 GB | **118%** | 8 | 24 | 300% | Overcommitted. USB-passthrough VMs locked (bms, ca). | -| tsys3 | 31 GB | 28 GB | 90% | 8 | 8 | 100% | 1 wnode (28 GB). OK. | -| tsys4 | 15 GB | 2 GB | 13% | 8 | 2 | 25% | Storage server. Friday: →64 GB. | -| tsys5 | 94 GB | 127 GB | **135%** | 8 | 72 | 900% | Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate). | -| tsys6 | 125 GB | 48 GB | **38%** | 16 | 12 | 75% | **77 GB free.** Room for more. | -| tsys7 | 188 GB | 56 GB | **30%** | 24 | 22 | 92% | **132 GB free.** Most room in fleet. | -| tsys9 | 23 GB | 30.2 GB | **131%** | 12 | 16 | 133% | Overcommitted by 7 GB. | - -### Hardware dependencies (locked VMs) - -| Host | VM | Hardware | Can move? | -|------|-----|----------|-----------| -| tsys1 | pfv-bms (100) | Zigbee + Z-Wave USB dongles | Friday only (physical move) | -| tsys1 | tsys-ca (101) | Nitrokey HSM (2×) | Friday only (physical move) | -| tsys5 | (various) | SDR dongles | **No — do not move** | - ---- - -## 5. Friday Maintenance Plan - -### 5.1 Primary goal: Proxmox on tsys2 - -tsys2 is currently Windows 10. It has the **best local storage in the fleet** -(Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary -Friday deliverable is: - -1. **Move DHCP off pfv-netboot** to the netinfra-01/02 redundant pair. This - must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware - currently, or depends on a service that does). -2. **Format tsys2**, install Proxmox. -3. **Agent inventory** — once Proxmox is live, the audit scripts - (`perf/deploy-check.sh`) can gather hardware and VM data directly, - replacing the Windows PowerShell inventory from - [`TODO.md`](TODO.md) §1. -4. **Run provisioning** — `provisioning/SetupNewSystem.sh` for baseline - hardening, then `perf/scripts/apply-tunings.sh`. -5. **Create wnode-tsys2** — boot disk on NVMe (960 PRO), data disk on SATA - SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage - of any worker. -6. **Join wnode-tsys2 to k3s cluster.** - -tsys2 hardware (from [`TODO.md`](TODO.md) Windows inventory): -- Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t) -- 32 GB RAM -- Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD -- Both wired NICs are USB-attached (same anti-pattern as tsys4/9) -- NVIDIA Quadro M1200 4 GB (passthrough candidate) -- Service tag: GH1XZG2 - -### 5.2 Physical hardware work (tsys4, tsys5) - -| Host | Work | -|------|------| -| tsys4 | PCIe NIC install (replace USB dongle), RAM 16→64 GB | -| tsys5 | 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB | - -### 5.3 VM migrations (while cluster is down for hardware) - -**Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:** - -| VM | VMID | From | To | RAM freed | Storage stays | -|----|------|------|----|-----------|---------------| -| tsys-siem | 901 | tsys9 (131%) | tsys6 (38%) | 8 GB on tsys9 | D2 (tsys4) | -| tsys-librenms | 104 | tsys1 (118%) | tsys7 (30%) | 2 GB on tsys1 | D2 (tsys4) | -| tsys-pdm | 105 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | -| tsys-workbench | 117 | tsys1 | tsys7 | 4 GB | D2 (tsys4) | -| tailscale-router | 109 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | -| kali-tsys | 114 | tsys1 | tsys7 | 2 GB | D2 (tsys4) | - -**Result:** tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from -30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily. - -**Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):** - -| VM | VMID | From | To | Hardware | -|----|------|------|----|---------| -| pfv-bms | 100 | tsys1 | tsys9 | Zigbee + Z-Wave dongles (physically move) | -| tsys-ca | 101 | tsys1 | tsys9 | Nitrokey HSMs (physically move) | - -**Result:** tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only). - -**Wnode storage rebalancing:** - -| VM | VMID | Disk from | Disk to | -|----|------|-----------|---------| -| wnode-tsys6 | 100 | D5 (tsys4) | S4 (tsys5) | -| wnode-tsys7 | 701 | D5 (tsys4) | S1 (tsys5) | - -### 5.4 Post-hardware cnode quorum fix (optional, if local-SSD available) - -After Friday hardware work, consider moving one cnode to local-lvm: -- cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency -- cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd - -This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS. -**Any single failure (host, storage server, or disk) leaves 2 of 3 alive.** -Deferred — user wants NFS-only for now. - ---- - -## 6. Complete VM Inventory (running VMs only) - -### tsys1 — 11 running (Infrastructure) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 100 | pfv-bms | 2 | 4096 | 32 GB | D2 | tsys4 | -| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 | tsys4 | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4096 | 32 GB | S3 | tsys5 | -| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | S2 | tsys5 | -| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 | tsys4 | -| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 | tsys4 | -| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D5 | tsys4 | -| 108 | tsys-ucs-01 | 2×2 | 8000 | 32 GB | D2 | tsys4 | -| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 | tsys4 | -| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 | tsys4 | -| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 | tsys4 | - -### tsys3 — 1 running (Kubernetes) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28000 | 32 GB | D5 | tsys4 | - -### tsys4 — 1 running (Storage) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL | - -### tsys5 — 17 running (Storage + Preprod) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 509 | pfv-k8s-wnode-tsys5 | 2×4 | 32000 | 32 GB | D2 | tsys4 | -| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL | -| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL | -| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) | -| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL | -| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL | -| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL | -| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL | -| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL | - -### tsys6 — 3 running (Kubernetes) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 100 | pfv-k8s-wnode-tsys6 | 2×2 | 32000 | 32 GB | D5 | tsys4 | -| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 | -| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 | tsys4 | - -### tsys7 — 6 running (Kubernetes) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 | tsys4 | -| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 | -| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 | tsys4 | -| 704 | TCTC | 4 | 6000 | 32 GB | D2 | tsys4 | -| 705 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 | tsys4 | -| 706 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | S2 | tsys5 | - -### tsys9 — 6 running (Infrastructure) - -| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server | -|------|------|-----|----------|------|---------|--------| -| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 | tsys4 | -| 902 | tsys-ucs-02 | 2×2 | 8000 | 50 GB | S2 | tsys5 | -| 903 | kali-rd | 2 | 2048 | 32 GB | D5 | tsys4 | -| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 | tsys4 | -| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S2 | tsys5 | -| 906 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | S3 | tsys5 | diff --git a/proxmox/docs/AUDIT-2026-07-29.md b/proxmox/docs/AUDIT-2026-07-29.md index c29741e..5c92821 100644 --- a/proxmox/docs/AUDIT-2026-07-29.md +++ b/proxmox/docs/AUDIT-2026-07-29.md @@ -1,637 +1,428 @@ -# Fleet Audit — 2026-07-29 (RAM/CPU placement focus) +# Fleet Audit — 2026-07-29 (consolidated, storage-corrected) -> **Scope:** End-to-end analysis of **memory and CPU placement** against the -> operator's capacity goals, plus **HDD disk placement** for IOPS balancing -> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally -> deferred to post-Friday work (NVMe on tsys5). See -> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets. +> **The single authoritative audit.** Supersedes all prior audits and the +> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict. +> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md). -**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts) -**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live -capacity probe (`/proc/meminfo`, `qm config`, 5s CPU sample, `kubectl top`) -collected via [`tests/remote.sh`](../../tests/remote.sh). +**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware +dumps + ~05:40 CDT capacity probe). Storage configs read directly from +`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host. **Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 -**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days) -**Supersedes:** placement numbers in [`PROJECT.md`](PROJECT.md) and -[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) where they conflict. +**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline) --- -## 1. Executive summary — fleet vs the two goals +## 1. Executive summary | Goal | Status | Detail | |------|--------|--------| -| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 72%. Fleet is well within budget. | -| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-6% busy**. Fleet is heavily CPU-underutilized. | - -**Headline:** The cluster has **abundant RAM and CPU headroom everywhere**. The -memory goal is met with margin to spare; the CPU-utilization goal is not -remotely met because there simply isn't enough sustained workload yet. The right -response is *not* to add hardware — it is to (a) consolidate the always-on -baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and -ready for RackRental/compute bursts, and (c) let utilization rise naturally as -RackRental + ETL demand comes online. +| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. | +| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. | | Finding | Severity | § | |---------|----------|---| -| **tsys1/5/9 are over-allocated (>115%)** though actual usage is fine | Medium | 4 | -| **tsys6/7 idle at 6%/9% with 117/171 GB free** — perfect, underused RackRental capacity | Info | 6 | -| **tsys5 has only ONE CPU** (CPU2 socket unpopulated) — prior "dual E5620" record was wrong | Medium | 9 | -| **cnode4 + cnode5 are stale running VMs not in the k3s cluster** (waste 8 GB + 8 vCPU) | Low | 10 | -| **Fleet CPU ~0-6% idle** — no path to 50% without more workloads | Info | 5 | -| **DDR3 sticks: compatibility-blocked from the high-RAM hosts** (they need RDIMM) | Medium | 7 | +| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 | +| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 | +| **tsys6 S1 NFS export is broken** (typo: path says `S21` not `S1`) | Bug | 6 | +| **tsys9 has no S1 in storage.cfg** — can't target S1 from tsys9 | Config gap | 6 | +| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 | +| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 | +| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 | --- -## 2. Host capacity (live, 2026-07-29) +## 2. Host capacity (live, 2026-07-29 ~05:40 CDT) -CPU "threads" = logical processors (`nproc`). "vCPU" = sum of running-VM cores -× sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling). -"Actual%" = `(MemTotal - MemAvailable) / MemTotal` (steady-state truth). +| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load | +|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------| +| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 | +| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 | +| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 | +| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 | +| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 | +| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 | +| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 | -| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | vCPU | CPU busy | Load | -|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|------|----------|------| -| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **72%** | 0 | 26 | 3% | 1.3 | -| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 8 | 0% | 0.2 | -| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **13%** | 832 MB | 2 | 0% | 3.0 | -| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 72 | 6% | 5.1 | -| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 3 | 48 GB | 38% | **6%** | 0 | 12 | 0% | 0.2 | -| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 6 | 56 GB | 30% | **9%** | 0 | 22 | 1% | 1.2 | -| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **60%** | 0 | 16 | 1% | 0.2 | - -**Notes on the table:** - -- **tsys4 load 3.0 with 0% CPU busy** = NFS/disk I/O wait (it is the storage - server). The 832 MB of swap used is the only swap activity in the fleet — - minor, but worth watching; it predates this audit. -- **tsys5 "single CPU":** dmidecode reports `CPU2: Status Unpopulated`. The T7500 - is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads). - See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and - useless until a 2nd CPU is seated. -- **tsys5 alloc 134%** looks alarming but actual usage is only **37%** — the 17 - VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual - low). Still, it is the most over-committed host by allocation. +**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6, +cnode5 (706) deleted from tsys7. No host migrations performed yet. --- -## 3. Running VM inventory (RAM/CPU only) +## 3. Storage topology (CORRECTED — read from live configs) + +> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its +> own storage entries. The same physical disk can appear under different storage +> IDs on different hosts. This was the source of the prior audit's errors. + +### 3.1 Physical disk → Proxmox storage ID mapping + +| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type | +|---------------|--------|------------|-----------------|---------------------|------| +| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD | +| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) | +| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD | +| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown | +| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown | +| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** | +| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD | +| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD | +| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD | +| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** | + +### 3.2 Critical correction: `local-nonprod` = S1 + +On tsys5, the storage entry is: +``` +dir: local-nonprod + path /mnt/pfv-tsys5/S1 +``` + +**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**, +accessed locally as a directory storage instead of via NFS. The 16 sectestbed +and preprod VMs "on local-nonprod" are physically on S1. They appear locked to +tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage +than `S1` (NFS type), even though they point to the same directory. + +**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan +to move production VMs onto S1 would have stacked them on top of 16 sandboxes. + +### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday) + +| Tier | Spindles | Speed | Current use | +|------|----------|-------|-------------| +| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs | +| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs | +| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) | +| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday | +| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only | + +--- + +## 4. Per-spindle IOPS load (CORRECTED) + +| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** | +|---------|--------|----------------|----------------|------------------------| +| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** | +| D5 (2TB) | tsys4 | 3 | 0 | 3 | +| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** | +| S2 (916GB) | tsys5 | 4 | 0 | 4 | +| S3 (916GB) | tsys5 | 3 | 0 | 3 | +| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity | + +**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.** + +The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real +IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI +pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to +S1.** Target S2, S3, S4, and D5 for IOPS spreading. + +--- + +## 5. Storage bugs and config gaps found + +### 5.1 BUG: tsys6 S1 export path typo (S21) + +On tsys6's `/etc/pve/storage.cfg`: +``` +nfs: S1 + export /mnt/pfv-tsys5/S21 ← should be S1, not S21 +``` + +**S1 is broken on tsys6.** Any attempt to move a VM disk to S1 on tsys6 will +fail. Fix: change `S21` → `S1`. + +### 5.2 GAP: tsys9 has no S1 in storage.cfg + +tsys9 defines S2, S3, S4, T5-SSD but **not S1**. If any VM on tsys9 ever needs +S1 storage, the entry must be added first. (Given that S1 is already loaded with +16 sectestbed VMs, this is low priority — but the gap should be noted.) + +### 5.3 ORPHANED: D1 and D7 exports on tsys4 + +tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg. +- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped + templates only, no running VMs. +- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped. + +These aren't causing problems but are underdocumented. Not actionable now. + +--- + +## 6. Current VM inventory (host + disk, post-user-changes) + +> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from +> start of session. `Disk` = NFS export or local storage holding the primary +> qcow2. ### tsys1 — Infrastructure (11 running) -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | -| 101 | tsys-ca | 2 | 2 GB | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | -| 103 | pfv-netinfra-01 | 2 | 2 GB | -| 104 | tsys-librenms | 2 | 2 GB | -| 105 | tsys-proxmox-datacenter | 2 | 2 GB | -| 106 | pfv-k8s-cnode3 | 2 | 4 GB | -| 108 | tsys-ucs-01 | 4 | 8 GB | -| 109 | tailscale-router | 2 | 2 GB | -| 114 | kali-tsys | 2 | 2 GB | -| 117 | tsys-secure-workbench | 2 | 4 GB | +| VMID | Name | vCPU | RAM | Disk | Server | +|------|------|------|-----|------|--------| +| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 | +| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | +| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | +| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | +| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 | +| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 | +| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | +| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | +| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | +| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 | +| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 | ### tsys3 — Compute (1 running) -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | +| VMID | Name | vCPU | RAM | Disk | Server | +|------|------|------|-----|------|--------| +| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | ### tsys4 — Storage (1 running) -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 400 | pfv-proxmox-backup-server | 2 | 2 GB | +| VMID | Name | vCPU | RAM | Disk | Server | +|------|------|------|-----|------|--------| +| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL | ### tsys5 — Storage + sandboxes (17 running) -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | -| 5101 | sectestbed-siem | 4 | 10 GB | -| 5111 | ultix-streaming | 4 | 9 GB | -| 53100 | tsys-preprod-awx | 4 | 9 GB | -| 53101 | tsys-preprod-siem | 4 | 12 GB | -| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | -| 5112 | ultix-offstage | 4 | 6 GB | -| 51010-51013,5105-5109,6000 | sectestbed suite (×10) | 4 ea | 4 GB ea | +| VMID | Name | vCPU | RAM | Disk | Server | Migratable? | +|------|------|------|-----|------|--------|-------------| +| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) | +| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) | +| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) | +| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) | +| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No | +| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No | +| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No | +| ×9 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No | -### tsys6 — RackRental-ready (3 running) +> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk +> but accessed via dir storage, not NFS — Proxmox will not migrate them. To +> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the +> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.* -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | -| 600 | tsys-awx | 4 | 12 GB | -| 601 | pfv-k8s-cnode4 **(stale — not in cluster)** | 4 | 4 GB | +### tsys6 — RackRental-ready (2 running, 1 stopped) -### tsys7 — RackRental-ready (6 running) +| VMID | Name | vCPU | RAM | Disk | Server | Status | +|------|------|------|-----|------|--------|--------| +| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running | +| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running | +| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** | -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | -| 702 | hfnoc-uisp | 4 | 8 GB | -| 704 | TCTC | 4 | 6 GB | -| 705 | pfv-k8s-cnode2 | 4 | 4 GB | -| 706 | pfv-k8s-cnode5 **(stale — not in cluster)** | 4 | 4 GB | -| 703 | rr-middleware | 2 | 2 GB | +### tsys7 — RackRental-ready (5 running) + +| VMID | Name | vCPU | RAM | Disk | Server | +|------|------|------|-----|------|--------| +| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | +| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 | +| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 | +| 704 | TCTC | 4 | 6 GB | D2 | tsys4 | +| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 | ### tsys9 — Infra + control plane (6 running) -| VMID | Name | vCPU | RAM | -|------|------|------|-----| -| 901 | tsys-siem | 2 | 8 GB | -| 902 | tsys-ucs-02 | 4 | 8 GB | -| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | -| 904 | pfv-netinfra-02 | 2 | 4 GB | -| 906 | pfv-k8s-cnode1 | 2 | 4 GB | -| 903 | kali-rd | 2 | 2 GB | +| VMID | Name | vCPU | RAM | Disk | Server | +|------|------|------|-----|------|--------| +| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 | +| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | +| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | +| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | +| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 | +| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | ### k3s control plane (ground truth via `kubectl`) -Only **3 cnodes** are actually joined to the pfv-k8s cluster: - -| Node | Host | CPU | Mem | Role | -|------|------|-----|-----|------| -| cnode1 (906) | tsys9 | 2% | 1181 Mi (30%) | control-plane,etcd | -| cnode2 (705) | tsys7 | 2% | 1206 Mi (30%) | control-plane,etcd | -| cnode3 (106) | tsys1 | 2% | 1124 Mi (28%) | control-plane,etcd | - -cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are **running as VMs but not -joined** — leftovers from the pre-rebuild 5-node cluster. See §10. **No wnodes -are joined yet** (all workers run standalone, pending the join step). +3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1). +All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.** --- -## 4. Memory analysis vs the 80% goal - -**Verdict: every host passes actual usage.** The 80% steady-state rule is met -fleet-wide with margin. The hosts closest to the line: - -| Host | Actual% | Alloc% | Risk | Action | -|------|---------|--------|------|--------| -| **tsys1** | **72%** | 115% | Closest to 80%; alloc > 100% | Shed 5 lightweight VMs → tsys7 (§8) | -| **tsys9** | 60% | 129% | Alloc > 100% but actual comfortable | Move tsys-siem (8 GB) → tsys6 (§8) | -| **tsys5** | 37% | 134% | Alloc highest, but mostly idle sandboxes | No action now; monitor if sandboxes wake | -| tsys3 | 10% | 90% | None | — | -| tsys6 | 6% | 38% | None (most empty) | Absorb migrated VMs | -| tsys7 | 9% | 30% | None | Absorb migrated VMs | - -**Key insight:** actual usage (72% max) is far below allocation (134% max) -because of KSM/balloon. The memory goal is about *actual*, and the fleet is -healthy. The over-allocation on tsys1/5/9 is a *planning* concern, resolved by -the migrations in §8 — **no RAM purchase required**. - -## 5. CPU analysis vs the 50%-idle goal - -**Verdict: the fleet misses the target badly — in the *safe* direction.** Every -host is 0-6% busy at idle. This is not a capacity problem; it is an -*under-utilization* problem. - -You cannot manufacture 50% utilization without workloads to run. The realistic -strategy, in priority order: - -1. **Consolidate the always-on baseline onto the efficient hosts.** tsys9 - (Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the - control plane + infra. Their baseline will rise toward the target as compute - workloads land. -2. **Keep tsys6/7 warm for bursts, not baseline.** They are the RackRental and - parallel-batch hosts. Their idle utilization will climb to ~50% *naturally* - as RackRental customers spin up labs — that is the designed use, not waste. -3. **Solar-gate the inefficient idle hosts when truly idle.** tsys5 (single - Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running - RackRental/batch, they are candidates for power-down / solar-aware scheduling - (aligns with the existing solar-aware scale-out concept). Running a Nehalem - box at 0% busy 24/7 is pure electricity cost. -4. **Grow into it.** As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7) - demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend - to chase the number. - ---- - -## 6. Workload placement model +## 7. Workload placement model | Workload family | Needs | Best hosts | Why | |-----------------|-------|-----------|-----| -| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + only local SSD/NVMe in fleet | -| **RackRental.net (network labs)** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs | +| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe | +| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs | | **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity | -CPU quality ranking (newest → oldest IPC): **tsys9** (Comet Lake 2020) > -**tsys3** (Skylake 2015) > **tsys7** (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell -2013) > **tsys5/6** (Westmere/Nehalem 2009-2010). +--- + +## 8. Answers to operator's questions + +### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?" + +**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB. +After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**. + +**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most +RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode. + +### 8.2 "Are tsys6/7 better for RackRental.net?" + +**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need +more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s) +— deferred to Friday + StorageClass work. + +### 8.3 "How should I deploy the DDR3?" + +Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST). +tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB). +Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM +migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5. --- -## 7. Answers to the operator's questions +## 9. Hardware corrections -### 7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?" +**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has +one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch — +useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity +and activate the CPU2 DIMM branch. -**tsys9 — YES.** It has the newest CPU in the fleet (i5-10500, 12 threads) and a -local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural -**cross-compile champion**. To grow it, first free RAM by moving tsys-siem -(8 GB) to tsys6 (§8), then bump wnode-tsys9 to **6 vCPU / 16 GB**. Actual usage -on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move. - -**tsys1 — NO (premise correction).** The i7-4770 is **Haswell 2013** — the -second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most -RAM-loaded host (72% actual, 115% alloc). It should **shed** VMs, not grow its -wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are -tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1. - -### 7.2 "Are tsys6/7 better for RackRental.net workloads?" - -**Yes — exactly right.** RackRental (renting ephemeral network labs) is -RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls. -tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM. -Their older CPUs are irrelevant because labs sit idle between interactions. - -**They do NOT need more RAM.** They idle at 9%/6% actual with 171/117 GB free. -They are over-provisioned in the right direction. Their real bottleneck is -**disk** (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls), -which is the deferred Friday + StorageClass work, not a RAM/CPU issue. - -### 7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff) - -**First, identify the exact DIMM type.** Compatibility is decisive and the -fleet's DDR3 hosts are **not interchangeable**: - -| Host | Platform | Accepts | Current | Empty slots | Notes | -|------|----------|---------|---------|-------------|-------| -| **tsys6** (R610) | PowerEdge server | **RDIMM ECC only** | 126 GB RDIMM 1333 | many (B-branch) | Consumer UDIMM → **no POST** | -| **tsys7** (R620) | PowerEdge server | **RDIMM ECC only** | 189 GB RDIMM 1600 | many (A7/A8 + B) | Consumer UDIMM → **no POST** | -| **tsys5** (T7500) | Workstation | **RDIMM ECC** | 94 GB RDIMM 1600 | 6 (CPU2 branch) | Useless without a 2nd CPU (§9) | -| **tsys4** (T1700) | Xeon E3 workstation | **ECC UDIMM** | 16 GB (4×4) | 0 (must swap) | Max 32 GB (4×8). 16GB sticks unsupported | -| **tsys1** (OptiPlex 9020) | Consumer | non-ECC UDIMM | 32 GB (4×8) | 0 | **Maxed.** Haswell caps at 8 GB/slot | - -**Decision tree (run this before buying/installing anything):** - -1. **Read the SPD** on each stick (labels or `decode-dimms` / `dmidecode -t - memory` once installed in *any* testable board). The binary question is: - **Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.** -2. **If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):** - - They fit **nowhere useful.** tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is - maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable - there). tsys4 requires ECC and caps at 8 GB/slot. - - The *only* possible target is **tsys4** — and only the 8 GB sticks, and only - if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4 - 16→32 GB. Marginal. -3. **If the sticks are RDIMM ECC (unlikely for Patriot, possible for some - Kingston server lines):** - - Deploy 16 GB RDIMM into **tsys7/tsys6** empty B-branch slots to balance - memory across both CPUs (currently CPU-B branches are partly empty — - single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise. - - But **value is low**: those hosts don't need more RAM (§7.2). - -**Bottom line on DDR3:** Do not spend effort/money chasing RAM into tsys6/7 — -they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9 -over-allocation) is solved for free by **VM migration** (§8), not by sticks. If -the sticks are free/cheap UDIMM, the realistic win is just **tsys4 16→32 GB** -(if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7 -will stop them from POSTing. +**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but +monitor after the planned RAM upgrade. --- -## 8. Recommended VM migrations (RAM/CPU relief — zero hardware cost) +## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced) -These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge -free RAM. **NFS-backed VMs migrate cleanly via PDM** (storage stays on the same -NFS export; only the compute host changes). - -**From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):** - -| VMID | Name | RAM freed on tsys1 | -|------|------|--------------------| -| 104 | tsys-librenms | 2 GB | -| 105 | tsys-proxmox-datacenter | 2 GB | -| 117 | tsys-secure-workbench | 4 GB | -| 109 | tailscale-router | 2 GB | -| 114 | kali-tsys | 2 GB | -| | **Total** | **12 GB** | - -Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB -trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host. - -**From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):** - -| VMID | Name | RAM freed on tsys9 | -|------|------|--------------------| -| 901 | tsys-siem | 8 GB | - -Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1). - -> **Note:** the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are -> hardware-locked and only move with a physical dongle relocation — defer to the -> onsite window, not part of this RAM/CPU pass. - ---- - -## 9. Hardware corrections discovered - -### tsys5 is single-CPU, not dual - -dmidecode: `CPU2: Status Unpopulated`. The Precision T7500 has **one E5620** -(4 cores / 8 threads), not two. `PROJECT.md` and prior audits said "2× E5620" — -**that was wrong.** Implications: - -- CPU capacity is half what was documented. tsys5 is the weakest single-socket - CPU in the fleet (tied era with tsys6's Nehalem). -- The 6 empty DIMM slots are on the **CPU2 branch** — they are useless until a - 2nd CPU is seated. Adding RAM there today does nothing. -- If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU - capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth. - This is a higher-value hardware move than adding RAM sticks. - -### tsys4 is using 832 MB of swap - -Only host in the fleet with active swap. Predates this audit and is minor, but a -16 GB storage server running PBS should not be swapping. After the planned -16→32/64 GB upgrade, monitor that swap returns to zero. - ---- - -## 10. Cleanup items - -| Item | Impact | Fix | -|------|--------|-----| -| **cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster** | 8 GB + 8 vCPU wasted on stale pre-rebuild VMs | Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node. | -| **No wnodes joined to k3s** | Workers run standalone; cluster has no schedulable capacity | Join wnodes (post-migration) — the §8 moves free the resources to size them well | -| **rr-middleware (703) on tsys7** | RackRental middleware lives on a RackRental host — fine | Intentional; just confirming placement is consistent | - ---- - -## 11. Deferred to post-Friday (storage — intentionally out of scope here) - -Per operator direction, disk/storage optimization happens after the Friday -maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s -StorageClass design. Tracked in [`TODO.md`](TODO.md) and -[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) §5. The one RAM/CPU-adjacent note: -**tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck**, not their CPU/RAM. - ---- - -## 12. Summary of recommended actions (this pass) - -| # | Action | Cost | Effect | -|---|--------|------|--------| -| 1 | Migrate 5 lightweight VMs tsys1→tsys7 | Free (PDM) | tsys1 alloc 115%→77% | -| 2 | Migrate tsys-siem tsys9→tsys6 | Free (PDM) | tsys9 alloc 129%→95%; room to grow wnode | -| 3 | Stop + delete stale cnode4/cnode5 | Free | Reclaim 8 GB + 8 vCPU | -| 4 | Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2) | Free | Cross-compile node ready | -| 5 | **Do not** grow wnode-tsys1; keep it small | Free | Corrects the CPU-quality premise | -| 6 | Identify DDR3 stick type before any install | Free | Avoids no-POST on tsys6/7 | -| 7 | Consider seating a 2nd CPU in tsys5 (if spare LGA771) | Cheap | Doubles tsys5 CPU; activates RAM branch | -| 8 | Solar-gate tsys5/6 when idle (no RackRental load) | Free | Stops wasting power on 0%-busy old cores | - -> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not -> hardware** — and by consolidating/solar-gating so idle silicon isn't burning -> power for nothing. - ---- - -## 13. Current VM inventory (host + disk, as of 2026-07-29) - -> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate, -> DellOpenManage, sectestbed stopped clones) are omitted — they consume no -> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in -> parens is which physical box serves it (tsys4 or tsys5). - -### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) | -| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | -| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | -| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) | -| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) | -| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | -| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | -| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | -| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) | -| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) | - -### tsys3 — Compute (1 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | - -### tsys4 — Storage (1 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) | - -### tsys5 — Storage + sandboxes (17 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) | -| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) | -| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) | -| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) | -| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) | -| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) | -| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) | -| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) | - -> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod / -> local-lvm — cannot migrate). They are idle sandboxes; leave them. - -### tsys6 — RackRental-ready (3 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | -| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) | -| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) | - -### tsys7 — RackRental-ready (6 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | -| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) | -| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) | -| 704 | TCTC | 4 | 6 GB | D2 (tsys4) | -| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) | -| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) | - -### tsys9 — Infra + control plane (6 running) - -| VMID | Name | vCPU | RAM | Disk (server) | -|------|------|------|-----|---------------| -| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) | -| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | -| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | -| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | -| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) | -| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | - -### Current per-spindle IOPS load (HDD exports only) - -| Spindle | Server | # VMs | Status | -|---------|--------|-------|--------| -| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** | -| D5 (Hitachi 2TB) | tsys4 | 3 | OK | -| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** | -| S2 (Seagate 916GB) | tsys5 | 4 | OK | -| S3 (Seagate 916GB) | tsys5 | 3 | OK | -| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty | - -**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.** - ---- - -## 14. Target end-state inventory (RAM/CPU + IOPS-balanced) - -> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The -> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to -> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers. +> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve +> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4 +> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA +> pairs split across storage servers. ### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none | -| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none | -| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none | -| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none | -| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none | -| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none | +| VMID | Name | vCPU | RAM | Disk | Server | Change | +|------|------|------|-----|------|--------|--------| +| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none | +| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none | +| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none | +| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none | +| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none | +| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none | ### tsys3 — Compute (1 running) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none | +| VMID | Name | vCPU | RAM | Disk | Server | Change | +|------|------|------|-----|------|--------|--------| +| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none | ### tsys4 — Storage (1 running) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| +| VMID | Name | vCPU | RAM | Disk | Change | +|------|------|------|-----|------|--------| | 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none | -### tsys5 — Storage + sandboxes (17 running) +### tsys5 — Storage + sandboxes (17 running, unchanged) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** | -| — | (16 locked sandbox VMs) | — | — | local-nonprod | none | +All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay +locked (idle sandboxes, low priority). ### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none | -| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** | -| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** | -| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** | +| VMID | Name | vCPU | RAM | Disk | Server | Change | +|------|------|------|-----|------|--------|--------| +| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none | +| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** | +| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** | +| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** | ### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none | -| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** | -| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** | -| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** | -| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** | -| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** | -| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** | -| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** | -| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** | -| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** | -| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** | +| VMID | Name | vCPU | RAM | Disk | Server | Change | +|------|------|------|-----|------|--------|--------| +| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none | +| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** | +| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** | +| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** | +| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** | +| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | +| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** | +| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) | +| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** | +| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | -### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*) +### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%) -| VMID | Name | vCPU | RAM | Disk (server) | Change | -|------|------|------|-----|---------------|--------| -| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none | -| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none | -| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none | -| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** | -| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none | -| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** | - -> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via -> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to -> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB** -> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's -> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and -> tsys2 (post-Friday).* +| VMID | Name | vCPU | RAM | Disk | Server | Change | +|------|------|------|-----|------|--------|--------| +| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none | +| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none | +| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none | +| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** | +| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none | +| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** | ### End-state per-spindle IOPS load | Spindle | Server | # VMs | VMs on it | |---------|--------|-------|-----------| -| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 | -| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd | -| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC | -| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 | -| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali | -| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench | +| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 | +| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd | +| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) | +| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* | +| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench | +| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... | -**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a -hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2). +> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16 +> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7 +> production VMs each is fine (they're 916 GB disks with light per-VM I/O).* + +**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.** + +HA pairs stay split: +- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓ +- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓ --- -## 15. Action checklist (do in this order) +## 11. Action checklist (do in this order) -> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk -> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move -> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O -> pauses. Recommended: do disk moves first, then host moves. +### Phase 0 — fix storage config bugs first -### Phase 0 — cleanup (frees resources first) +- **Fix tsys6 S1 typo:** edit `/etc/pve/storage.cfg` on tsys6, change + `export /mnt/pfv-tsys5/S21` → `export /mnt/pfv-tsys5/S1` +- **(Optional) Add S1 to tsys9 storage.cfg** (low priority — S1 is already + loaded; only needed if a tsys9 VM ever targets S1) -- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale. -- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total. +### Phase 1 — cleanup -### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves) +- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster +- **Confirm cnode5 (706) gone** — already deleted by operator ✓ -> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one -> disk move per still-on-original-host VM first. All targets are **HDD** exports. +### Phase 2 — disk migrations off D2 (IOPS rebalancing) -- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5) -- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6) -- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7) -- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7) -- **TCTC (704):** move disk **D2 → S1** (host stays tsys7) -- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7) +> Do disk moves on VMs that are staying on their current host first. -### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs) +- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5) +- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6) +- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7) +- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7) +- **TCTC (704):** disk **D2 → S3** (host stays tsys7) +- **cnode2 (705):** disk **D2 → S2** (host stays tsys7) -- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1** -- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1** +### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs) + +- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3** +- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4** - **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**) -- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3** -- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4** -- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1** +- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2** +- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3** +- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2** -> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs. +### Phase 4 — grow the compute wnode -### Phase 3 — grow the compute wnode +- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required) -- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9) +### Phase 5 — join workers to k3s -### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9) - -- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9 -- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%) - -### Phase 5 — bring the workers into the cluster (after placement settles) - -- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh` -- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5) +- Join wnodes via `k8s/join-servers.sh` +- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5) ### Verification after each phase -- `qm list` on each host — confirm VM count + status -- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80% +- `qm list` per host — confirm VM count + status +- `cat /proc/meminfo` — confirm actual RAM ≤ 80% - `kubectl top nodes` — confirm cnodes healthy -- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills) +- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up +- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills