docs(audit): add VM placement plan (current/target inventory + action checklist)

Add §13-§15 to AUDIT-2026-07-29: current per-VM host+disk inventory, target
end-state inventory (RAM/CPU relief + HDD IOPS rebalancing), and a phased action
checklist. Headline disk finding: D2 carries 17 of 26 NFS VMs while S1 sits
empty — the plan rebalances to 4-5 VMs per spindle (D2 17->5, S1 0->5) using HDD
exports only. Host moves relieve tsys1 (115%->77% alloc) and tsys9. All disks
stay on HDD (SSD/NVMe tiering deferred to post-Friday).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

docs/docmap.md

@@ -24,7 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
 |----------|-------------|---------------|
 | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
 | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
-| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU placement audit vs the capacity goals; live data from all 7 hosts | 2026-07-29 |
+| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU + HDD disk placement audit (current/target inventory + action checklist); live data from all 7 hosts | 2026-07-29 |
 | [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
 | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
 | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |

proxmox/docs/AUDIT-2026-07-29.md

@@ -1,10 +1,10 @@
 # Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
 
 > **Scope:** End-to-end analysis of **memory and CPU placement** against the
-> operator's capacity goals. Storage/disk optimization is intentionally
+> operator's capacity goals, plus **HDD disk placement** for IOPS balancing
-> **out of scope** for this pass and is deferred to post-Friday work (NVMe on
+> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally
-> tsys5 + k8s StorageClass design). See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)
+> deferred to post-Friday work (NVMe on tsys5). See
-> for the standing targets.
+> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets.
 
 **Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
 **Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
@@ -386,3 +386,252 @@ StorageClass design. Tracked in [`TODO.md`](TODO.md) and
 > Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
 > hardware** — and by consolidating/solar-gating so idle silicon isn't burning
 > power for nothing.
+
+---
+
+## 13. Current VM inventory (host + disk, as of 2026-07-29)
+
+> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate,
+> DellOpenManage, sectestbed stopped clones) are omitted — they consume no
+> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in
+> parens is which physical box serves it (tsys4 or tsys5).
+
+### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) |
+| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) |
+| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) |
+| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) |
+| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) |
+| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) |
+| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) |
+| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) |
+| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) |
+| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) |
+| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) |
+
+### tsys3 — Compute (1 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) |
+
+### tsys4 — Storage (1 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) |
+
+### tsys5 — Storage + sandboxes (17 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) |
+| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) |
+| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) |
+| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) |
+| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) |
+| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) |
+| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) |
+| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) |
+
+> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod /
+> local-lvm — cannot migrate). They are idle sandboxes; leave them.
+
+### tsys6 — RackRental-ready (3 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) |
+| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) |
+| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) |
+
+### tsys7 — RackRental-ready (6 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) |
+| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) |
+| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) |
+| 704 | TCTC | 4 | 6 GB | D2 (tsys4) |
+| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) |
+| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) |
+
+### tsys9 — Infra + control plane (6 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) |
+|------|------|------|-----|---------------|
+| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) |
+| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) |
+| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) |
+| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) |
+| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) |
+| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) |
+
+### Current per-spindle IOPS load (HDD exports only)
+
+| Spindle | Server | # VMs | Status |
+|---------|--------|-------|--------|
+| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** |
+| D5 (Hitachi 2TB) | tsys4 | 3 | OK |
+| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** |
+| S2 (Seagate 916GB) | tsys5 | 4 | OK |
+| S3 (Seagate 916GB) | tsys5 | 3 | OK |
+| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty |
+
+**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.**
+
+---
+
+## 14. Target end-state inventory (RAM/CPU + IOPS-balanced)
+
+> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The
+> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to
+> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers.
+
+### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none |
+| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none |
+| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none |
+| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none |
+| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none |
+| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none |
+
+### tsys3 — Compute (1 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none |
+
+### tsys4 — Storage (1 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
+
+### tsys5 — Storage + sandboxes (17 running)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** |
+| — | (16 locked sandbox VMs) | — | — | local-nonprod | none |
+
+### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none |
+| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** |
+| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** |
+| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** |
+
+### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none |
+| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** |
+| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** |
+| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** |
+| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** |
+| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
+| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
+| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** |
+| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** |
+| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** |
+| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** |
+
+### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*)
+
+| VMID | Name | vCPU | RAM | Disk (server) | Change |
+|------|------|------|-----|---------------|--------|
+| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none |
+| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none |
+| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none |
+| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** |
+| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none |
+| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** |
+
+> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via
+> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to
+> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB**
+> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's
+> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and
+> tsys2 (post-Friday).*
+
+### End-state per-spindle IOPS load
+
+| Spindle | Server | # VMs | VMs on it |
+|---------|--------|-------|-----------|
+| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
+| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
+| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC |
+| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 |
+| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali |
+| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench |
+
+**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a
+hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2).
+
+---
+
+## 15. Action checklist (do in this order)
+
+> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk
+> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move
+> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O
+> pauses. Recommended: do disk moves first, then host moves.
+
+### Phase 0 — cleanup (frees resources first)
+
+- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale.
+- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total.
+
+### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves)
+
+> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one
+> disk move per still-on-original-host VM first. All targets are **HDD** exports.
+
+- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5)
+- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6)
+- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7)
+- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7)
+- **TCTC (704):** move disk **D2 → S1** (host stays tsys7)
+- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7)
+
+### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs)
+
+- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1**
+- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1**
+- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
+- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3**
+- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4**
+- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1**
+
+> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs.
+
+### Phase 3 — grow the compute wnode
+
+- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9)
+
+### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9)
+
+- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9
+- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%)
+
+### Phase 5 — bring the workers into the cluster (after placement settles)
+
+- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh`
+- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
+
+### Verification after each phase
+
+- `qm list` on each host — confirm VM count + status
+- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80%
+- `kubectl top nodes` — confirm cnodes healthy
+- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills)