mrcharles
575d79f611
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
12 KiB
Fresh Fleet Audit — 2026-07-28 (revised)
A newer audit exists:
AUDIT-2026-07-29.md(RAM/CPU placement focus, with codified capacity goals). Placement numbers below are superseded where the two conflict; this doc remains authoritative for the Friday storage/hardware plan (§5) and storage-redundancy analysis.
Supersedes placement data in
PROJECT.md§4-§8.
Audit time: 2026-07-28 20:30 CDT Hosts audited: pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9 Hosts offline: pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
1. HA Pairs — FIXED ✓
Both infrastructure pairs now split across storage servers:
| VM | VMID | Host | Disk | Server | tsys4 dies | tsys5 dies |
|---|---|---|---|---|---|---|
| netinfra-01 | 103 | tsys1 | S2 | tsys5 | alive | dies |
| netinfra-02 | 904 | tsys9 | D2 | tsys4 | dies | alive |
| ucs-01 | 108 | tsys1 | D2 | tsys4 | dies | alive |
| ucs-02 | 902 | tsys9 | S2 | tsys5 | alive | dies |
Any single storage server failure leaves one half of each pair alive.
2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers)
| Cnode | VMID | Host | Disk | Server |
|---|---|---|---|---|
| cnode1 | 906 | tsys9 | S3 | tsys5 |
| cnode2 | 705 | tsys7 | D2 | tsys4 |
| cnode3 | 106 | tsys1 | D5 | tsys4 |
tsys4 failure: cnode2 + cnode3 die → 1 of 3 → quorum lost. tsys5 failure: cnode1 dies → 2 of 3 → quorum OK.
This is a structural constraint: 3 cnodes on 2 NFS servers means one side holds 2. The only true fixes are:
- A 3rd storage target (not available until Friday hardware work)
- Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now)
- Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM)
Decision: accept for now. Revisit after Friday when local-SSD boot disks become an option for cnodes.
3. Worker Nodes — NEEDS REBALANCING (2 storage migrations)
Current state (problem: 3 wnodes share one physical disk)
| Wnode | VMID | Host | Disk | Server | Disk shared with |
|---|---|---|---|---|---|
| wnode-tsys1 | 102 | tsys1 | S3 | tsys5 | — |
| wnode-tsys3 | 313 | tsys3 | D5 | tsys4 | wnode-tsys6, wnode-tsys7 |
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | — |
| wnode-tsys6 | 100 | tsys6 | D5 | tsys4 | wnode-tsys3, wnode-tsys7 |
| wnode-tsys7 | 701 | tsys7 | D5 | tsys4 | wnode-tsys3, wnode-tsys6 |
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | — |
D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS. S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).
Recommended moves (2 PDM storage migrations)
| Wnode | Disk from | Disk to | Effect |
|---|---|---|---|
| wnode-tsys6 (100) | D5 (tsys4) | S4 (tsys5, 499 GB free) | D5: 3→2 wnodes |
| wnode-tsys7 (701) | D5 (tsys4) | S1 (tsys5, 916 GB free) | D5: 2→1 wnode |
After rebalancing — every wnode on its own physical disk
| Wnode | Host | Disk | Server |
|---|---|---|---|
| wnode-tsys1 | tsys1 | S3 | tsys5 |
| wnode-tsys3 | tsys3 | D5 | tsys4 |
| wnode-tsys5 | tsys5 | D2 | tsys4 |
| wnode-tsys6 | tsys6 | S4 | tsys5 |
| wnode-tsys7 | tsys7 | S1 | tsys5 |
| wnode-tsys9 | tsys9 | S2 | tsys5 |
tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode has a dedicated physical spindle.
4. Host Capacity
| Host | RAM | VM RAM | RAM% | Threads | vCPU | CPU% | Notes |
|---|---|---|---|---|---|---|---|
| tsys1 | 31 GB | 36.6 GB | 118% | 8 | 24 | 300% | Overcommitted. USB-passthrough VMs locked (bms, ca). |
| tsys3 | 31 GB | 28 GB | 90% | 8 | 8 | 100% | 1 wnode (28 GB). OK. |
| tsys4 | 15 GB | 2 GB | 13% | 8 | 2 | 25% | Storage server. Friday: →64 GB. |
| tsys5 | 94 GB | 127 GB | 135% | 8 | 72 | 900% | Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate). |
| tsys6 | 125 GB | 48 GB | 38% | 16 | 12 | 75% | 77 GB free. Room for more. |
| tsys7 | 188 GB | 56 GB | 30% | 24 | 22 | 92% | 132 GB free. Most room in fleet. |
| tsys9 | 23 GB | 30.2 GB | 131% | 12 | 16 | 133% | Overcommitted by 7 GB. |
Hardware dependencies (locked VMs)
| Host | VM | Hardware | Can move? |
|---|---|---|---|
| tsys1 | pfv-bms (100) | Zigbee + Z-Wave USB dongles | Friday only (physical move) |
| tsys1 | tsys-ca (101) | Nitrokey HSM (2×) | Friday only (physical move) |
| tsys5 | (various) | SDR dongles | No — do not move |
5. Friday Maintenance Plan
5.1 Primary goal: Proxmox on tsys2
tsys2 is currently Windows 10. It has the best local storage in the fleet (Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary Friday deliverable is:
- Move DHCP off pfv-netboot to the netinfra-01/02 redundant pair. This must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware currently, or depends on a service that does).
- Format tsys2, install Proxmox.
- Agent inventory — once Proxmox is live, the audit scripts
(
perf/deploy-check.sh) can gather hardware and VM data directly, replacing the Windows PowerShell inventory fromTODO.md§1. - Run provisioning —
provisioning/SetupNewSystem.shfor baseline hardening, thenperf/scripts/apply-tunings.sh. - Create wnode-tsys2 — boot disk on NVMe (960 PRO), data disk on SATA SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage of any worker.
- Join wnode-tsys2 to k3s cluster.
tsys2 hardware (from TODO.md Windows inventory):
- Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t)
- 32 GB RAM
- Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD
- Both wired NICs are USB-attached (same anti-pattern as tsys4/9)
- NVIDIA Quadro M1200 4 GB (passthrough candidate)
- Service tag: GH1XZG2
5.2 Physical hardware work (tsys4, tsys5)
| Host | Work |
|---|---|
| tsys4 | PCIe NIC install (replace USB dongle), RAM 16→64 GB |
| tsys5 | 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB |
5.3 VM migrations (while cluster is down for hardware)
Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:
| VM | VMID | From | To | RAM freed | Storage stays |
|---|---|---|---|---|---|
| tsys-siem | 901 | tsys9 (131%) | tsys6 (38%) | 8 GB on tsys9 | D2 (tsys4) |
| tsys-librenms | 104 | tsys1 (118%) | tsys7 (30%) | 2 GB on tsys1 | D2 (tsys4) |
| tsys-pdm | 105 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
| tsys-workbench | 117 | tsys1 | tsys7 | 4 GB | D2 (tsys4) |
| tailscale-router | 109 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
| kali-tsys | 114 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
Result: tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from 30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily.
Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):
| VM | VMID | From | To | Hardware |
|---|---|---|---|---|
| pfv-bms | 100 | tsys1 | tsys9 | Zigbee + Z-Wave dongles (physically move) |
| tsys-ca | 101 | tsys1 | tsys9 | Nitrokey HSMs (physically move) |
Result: tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only).
Wnode storage rebalancing:
| VM | VMID | Disk from | Disk to |
|---|---|---|---|
| wnode-tsys6 | 100 | D5 (tsys4) | S4 (tsys5) |
| wnode-tsys7 | 701 | D5 (tsys4) | S1 (tsys5) |
5.4 Post-hardware cnode quorum fix (optional, if local-SSD available)
After Friday hardware work, consider moving one cnode to local-lvm:
- cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency
- cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd
This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS. Any single failure (host, storage server, or disk) leaves 2 of 3 alive. Deferred — user wants NFS-only for now.
6. Complete VM Inventory (running VMs only)
tsys1 — 11 running (Infrastructure)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 100 | pfv-bms | 2 | 4096 | 32 GB | D2 | tsys4 |
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4096 | 32 GB | S3 | tsys5 |
| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | S2 | tsys5 |
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 | tsys4 |
| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D5 | tsys4 |
| 108 | tsys-ucs-01 | 2×2 | 8000 | 32 GB | D2 | tsys4 |
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 | tsys4 |
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 | tsys4 |
tsys3 — 1 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28000 | 32 GB | D5 | tsys4 |
tsys4 — 1 running (Storage)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
tsys5 — 17 running (Storage + Preprod)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 509 | pfv-k8s-wnode-tsys5 | 2×4 | 32000 | 32 GB | D2 | tsys4 |
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
tsys6 — 3 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 100 | pfv-k8s-wnode-tsys6 | 2×2 | 32000 | 32 GB | D5 | tsys4 |
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 | tsys4 |
tsys7 — 6 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 | tsys4 |
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 | tsys4 |
| 704 | TCTC | 4 | 6000 | 32 GB | D2 | tsys4 |
| 705 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 | tsys4 |
| 706 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | S2 | tsys5 |
tsys9 — 6 running (Infrastructure)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|---|---|---|---|---|---|---|
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 | tsys4 |
| 902 | tsys-ucs-02 | 2×2 | 8000 | 50 GB | S2 | tsys5 |
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 | tsys4 |
| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 | tsys4 |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S2 | tsys5 |
| 906 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | S3 | tsys5 |