KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): add VM placement plan (current/target inventory + action checklist)

Browse Source 
Add §13-§15 to AUDIT-2026-07-29: current per-VM host+disk inventory, target
end-state inventory (RAM/CPU relief + HDD IOPS rebalancing), and a phased action
checklist. Headline disk finding: D2 carries 17 of 26 NFS VMs while S1 sits
empty — the plan rebalances to 4-5 VMs per spindle (D2 17->5, S1 0->5) using HDD
exports only. Host moves relieve tsys1 (115%->77% alloc) and tsys9. All disks
stay on HDD (SSD/NVMe tiering deferred to post-Friday).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 06:57:51 -05:00
parent 575d79f611
commit 6199368710
2 changed files with 254 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docmap.md
+1 -1
View File
@@ -24,7 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
|----------|-------------|---------------|
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU placement audit vs the capacity goals; live data from all 7 hosts | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU + HDD disk placement audit (current/target inventory + action checklist); live data from all 7 hosts | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
proxmox/docs/AUDIT-2026-07-29.md
+253 -4
View File
@@ -1,10 +1,10 @@
# Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
> **Scope:** End-to-end analysis of **memory and CPU placement** against the
> operator's capacity goals. Storage/disk optimization is intentionally
> **out of scope** for this pass and is deferred to post-Friday work (NVMe on
> tsys5 + k8s StorageClass design). See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)
> for the standing targets.
> operator's capacity goals, plus **HDD disk placement** for IOPS balancing
> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally
> deferred to post-Friday work (NVMe on tsys5). See
> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets.
**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
@@ -386,3 +386,252 @@ StorageClass design. Tracked in [`TODO.md`](TODO.md) and
> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
> hardware** — and by consolidating/solar-gating so idle silicon isn't burning
> power for nothing.
---
## 13. Current VM inventory (host + disk, as of 2026-07-29)
> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate,
> DellOpenManage, sectestbed stopped clones) are omitted — they consume no
> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in
> parens is which physical box serves it (tsys4 or tsys5).
### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) |
| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) |
| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) |
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) |
| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) |
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) |
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) |
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) |
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) |
| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) |
> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod /
> local-lvm — cannot migrate). They are idle sandboxes; leave them.
### tsys6 — RackRental-ready (3 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) |
| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) |
| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) |
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) |
| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) |
| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) |
| 704 | TCTC | 4 | 6 GB | D2 (tsys4) |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) |
| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) |
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) |
### Current per-spindle IOPS load (HDD exports only)
| Spindle | Server | # VMs | Status |
|---------|--------|-------|--------|
| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** |
| D5 (Hitachi 2TB) | tsys4 | 3 | OK |
| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** |
| S2 (Seagate 916GB) | tsys5 | 4 | OK |
| S3 (Seagate 916GB) | tsys5 | 3 | OK |
| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty |
**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.**
---
## 14. Target end-state inventory (RAM/CPU + IOPS-balanced)
> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The
> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to
> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers.
### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** |
| — | (16 locked sandbox VMs) | — | — | local-nonprod | none |
### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none |
| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** |
| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** |
| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** |
### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none |
| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** |
| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** |
| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** |
| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** |
| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** |
| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** |
| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** |
### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none |
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none |
| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** |
> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via
> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to
> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB**
> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's
> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and
> tsys2 (post-Friday).*
### End-state per-spindle IOPS load
| Spindle | Server | # VMs | VMs on it |
|---------|--------|-------|-----------|
| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC |
| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 |
| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali |
| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench |
**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a
hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2).
---
## 15. Action checklist (do in this order)
> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk
> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move
> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O
> pauses. Recommended: do disk moves first, then host moves.
### Phase 0 — cleanup (frees resources first)
- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale.
- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total.
### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves)
> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one
> disk move per still-on-original-host VM first. All targets are **HDD** exports.
- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5)
- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6)
- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7)
- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7)
- **TCTC (704):** move disk **D2 → S1** (host stays tsys7)
- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7)
### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs)
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1**
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1**
- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4**
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1**
> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs.
### Phase 3 — grow the compute wnode
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9)
### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9)
- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9
- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%)
### Phase 5 — bring the workers into the cluster (after placement settles)
- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh`
- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
### Verification after each phase
- `qm list` on each host — confirm VM count + status
- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80%
- `kubectl top nodes` — confirm cnodes healthy
- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills)