diff --git a/docs/docmap.md b/docs/docmap.md index df59f23..84081be 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -24,7 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture. |----------|-------------|---------------| | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU placement audit vs the capacity goals; live data from all 7 hosts | 2026-07-29 | +| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU + HDD disk placement audit (current/target inventory + action checklist); live data from all 7 hosts | 2026-07-29 | | [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | diff --git a/proxmox/docs/AUDIT-2026-07-29.md b/proxmox/docs/AUDIT-2026-07-29.md index 13cd299..c29741e 100644 --- a/proxmox/docs/AUDIT-2026-07-29.md +++ b/proxmox/docs/AUDIT-2026-07-29.md @@ -1,10 +1,10 @@ # Fleet Audit — 2026-07-29 (RAM/CPU placement focus) > **Scope:** End-to-end analysis of **memory and CPU placement** against the -> operator's capacity goals. Storage/disk optimization is intentionally -> **out of scope** for this pass and is deferred to post-Friday work (NVMe on -> tsys5 + k8s StorageClass design). See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) -> for the standing targets. +> operator's capacity goals, plus **HDD disk placement** for IOPS balancing +> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally +> deferred to post-Friday work (NVMe on tsys5). See +> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets. **Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts) **Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live @@ -386,3 +386,252 @@ StorageClass design. Tracked in [`TODO.md`](TODO.md) and > Memory goal: **already met.** CPU goal: **chase it by adding workloads, not > hardware** — and by consolidating/solar-gating so idle silicon isn't burning > power for nothing. + +--- + +## 13. Current VM inventory (host + disk, as of 2026-07-29) + +> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate, +> DellOpenManage, sectestbed stopped clones) are omitted — they consume no +> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in +> parens is which physical box serves it (tsys4 or tsys5). + +### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) | +| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | +| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | +| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | +| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) | +| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) | +| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | +| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | +| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | +| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) | +| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) | + +### tsys3 — Compute (1 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | + +### tsys4 — Storage (1 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) | + +### tsys5 — Storage + sandboxes (17 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) | +| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) | +| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) | +| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) | +| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) | +| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) | +| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) | +| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) | + +> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod / +> local-lvm — cannot migrate). They are idle sandboxes; leave them. + +### tsys6 — RackRental-ready (3 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | +| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) | +| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) | + +### tsys7 — RackRental-ready (6 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | +| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) | +| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) | +| 704 | TCTC | 4 | 6 GB | D2 (tsys4) | +| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) | +| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) | + +### tsys9 — Infra + control plane (6 running) + +| VMID | Name | vCPU | RAM | Disk (server) | +|------|------|------|-----|---------------| +| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) | +| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | +| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | +| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | +| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) | +| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | + +### Current per-spindle IOPS load (HDD exports only) + +| Spindle | Server | # VMs | Status | +|---------|--------|-------|--------| +| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** | +| D5 (Hitachi 2TB) | tsys4 | 3 | OK | +| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** | +| S2 (Seagate 916GB) | tsys5 | 4 | OK | +| S3 (Seagate 916GB) | tsys5 | 3 | OK | +| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty | + +**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.** + +--- + +## 14. Target end-state inventory (RAM/CPU + IOPS-balanced) + +> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The +> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to +> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers. + +### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none | +| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none | +| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none | +| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none | +| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none | +| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none | + +### tsys3 — Compute (1 running) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none | + +### tsys4 — Storage (1 running) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none | + +### tsys5 — Storage + sandboxes (17 running) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** | +| — | (16 locked sandbox VMs) | — | — | local-nonprod | none | + +### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none | +| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** | +| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** | +| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** | + +### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none | +| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** | +| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** | +| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** | +| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** | +| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** | +| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** | +| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** | +| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** | +| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** | +| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** | + +### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*) + +| VMID | Name | vCPU | RAM | Disk (server) | Change | +|------|------|------|-----|---------------|--------| +| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none | +| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none | +| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none | +| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** | +| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none | +| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** | + +> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via +> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to +> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB** +> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's +> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and +> tsys2 (post-Friday).* + +### End-state per-spindle IOPS load + +| Spindle | Server | # VMs | VMs on it | +|---------|--------|-------|-----------| +| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 | +| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd | +| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC | +| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 | +| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali | +| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench | + +**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a +hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2). + +--- + +## 15. Action checklist (do in this order) + +> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk +> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move +> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O +> pauses. Recommended: do disk moves first, then host moves. + +### Phase 0 — cleanup (frees resources first) + +- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale. +- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total. + +### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves) + +> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one +> disk move per still-on-original-host VM first. All targets are **HDD** exports. + +- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5) +- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6) +- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7) +- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7) +- **TCTC (704):** move disk **D2 → S1** (host stays tsys7) +- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7) + +### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs) + +- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1** +- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1** +- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**) +- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3** +- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4** +- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1** + +> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs. + +### Phase 3 — grow the compute wnode + +- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9) + +### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9) + +- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9 +- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%) + +### Phase 5 — bring the workers into the cluster (after placement settles) + +- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh` +- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5) + +### Verification after each phase + +- `qm list` on each host — confirm VM count + status +- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80% +- `kubectl top nodes` — confirm cnodes healthy +- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills)