KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d064754c3edb8efba961864c709ceae488580dc8
PFVCluster/proxmox/docs/AUDIT-2026-07-28.md
T
mrcharles d064754c3e refactor(docs): co-locate k8s/proxmox docs with their code + fix broken perf paths 
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-29 05:28:07 -05:00

12 KiB
Raw Blame History

Fresh Fleet Audit — 2026-07-28 (revised)

Supersedes placement data in PROJECT.md §4-§8.

Audit time: 2026-07-28 20:30 CDT Hosts audited: pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9 Hosts offline: pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)

1. HA Pairs — FIXED ✓

Both infrastructure pairs now split across storage servers:

VM VMID Host Disk Server tsys4 dies tsys5 dies
netinfra-01 103 tsys1 S2 tsys5 alive dies
netinfra-02 904 tsys9 D2 tsys4 dies alive
ucs-01 108 tsys1 D2 tsys4 dies alive
ucs-02 902 tsys9 S2 tsys5 alive dies

Any single storage server failure leaves one half of each pair alive.

2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers)

Cnode VMID Host Disk Server
cnode1 906 tsys9 S3 tsys5
cnode2 705 tsys7 D2 tsys4
cnode3 106 tsys1 D5 tsys4

tsys4 failure: cnode2 + cnode3 die → 1 of 3 → quorum lost. tsys5 failure: cnode1 dies → 2 of 3 → quorum OK.

This is a structural constraint: 3 cnodes on 2 NFS servers means one side holds 2. The only true fixes are:

  • A 3rd storage target (not available until Friday hardware work)
  • Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now)
  • Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM)

Decision: accept for now. Revisit after Friday when local-SSD boot disks become an option for cnodes.

3. Worker Nodes — NEEDS REBALANCING (2 storage migrations)

Current state (problem: 3 wnodes share one physical disk)

Wnode VMID Host Disk Server Disk shared with
wnode-tsys1 102 tsys1 S3 tsys5 —
wnode-tsys3 313 tsys3 D5 tsys4 wnode-tsys6, wnode-tsys7
wnode-tsys5 509 tsys5 D2 tsys4 —
wnode-tsys6 100 tsys6 D5 tsys4 wnode-tsys3, wnode-tsys7
wnode-tsys7 701 tsys7 D5 tsys4 wnode-tsys3, wnode-tsys6
wnode-tsys9 905 tsys9 S2 tsys5 —

D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS. S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).

Recommended moves (2 PDM storage migrations)

Wnode Disk from Disk to Effect
wnode-tsys6 (100) D5 (tsys4) S4 (tsys5, 499 GB free) D5: 3→2 wnodes
wnode-tsys7 (701) D5 (tsys4) S1 (tsys5, 916 GB free) D5: 2→1 wnode

After rebalancing — every wnode on its own physical disk

Wnode Host Disk Server
wnode-tsys1 tsys1 S3 tsys5
wnode-tsys3 tsys3 D5 tsys4
wnode-tsys5 tsys5 D2 tsys4
wnode-tsys6 tsys6 S4 tsys5
wnode-tsys7 tsys7 S1 tsys5
wnode-tsys9 tsys9 S2 tsys5

tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode has a dedicated physical spindle.

4. Host Capacity

Host RAM VM RAM RAM% Threads vCPU CPU% Notes
tsys1 31 GB 36.6 GB 118% 8 24 300% Overcommitted. USB-passthrough VMs locked (bms, ca).
tsys3 31 GB 28 GB 90% 8 8 100% 1 wnode (28 GB). OK.
tsys4 15 GB 2 GB 13% 8 2 25% Storage server. Friday: →64 GB.
tsys5 94 GB 127 GB 135% 8 72 900% Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate).
tsys6 125 GB 48 GB 38% 16 12 75% 77 GB free. Room for more.
tsys7 188 GB 56 GB 30% 24 22 92% 132 GB free. Most room in fleet.
tsys9 23 GB 30.2 GB 131% 12 16 133% Overcommitted by 7 GB.

Hardware dependencies (locked VMs)

Host VM Hardware Can move?
tsys1 pfv-bms (100) Zigbee + Z-Wave USB dongles Friday only (physical move)
tsys1 tsys-ca (101) Nitrokey HSM (2×) Friday only (physical move)
tsys5 (various) SDR dongles No — do not move

5. Friday Maintenance Plan

5.1 Primary goal: Proxmox on tsys2

tsys2 is currently Windows 10. It has the best local storage in the fleet (Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary Friday deliverable is:

  1. Move DHCP off pfv-netboot to the netinfra-01/02 redundant pair. This must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware currently, or depends on a service that does).
  2. Format tsys2, install Proxmox.
  3. Agent inventory — once Proxmox is live, the audit scripts (perf/deploy-check.sh) can gather hardware and VM data directly, replacing the Windows PowerShell inventory from TODO.md §1.
  4. Run provisioning — provisioning/SetupNewSystem.sh for baseline hardening, then perf/scripts/apply-tunings.sh.
  5. Create wnode-tsys2 — boot disk on NVMe (960 PRO), data disk on SATA SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage of any worker.
  6. Join wnode-tsys2 to k3s cluster.

tsys2 hardware (from TODO.md Windows inventory):

  • Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t)
  • 32 GB RAM
  • Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD
  • Both wired NICs are USB-attached (same anti-pattern as tsys4/9)
  • NVIDIA Quadro M1200 4 GB (passthrough candidate)
  • Service tag: GH1XZG2

5.2 Physical hardware work (tsys4, tsys5)

Host Work
tsys4 PCIe NIC install (replace USB dongle), RAM 16→64 GB
tsys5 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB

5.3 VM migrations (while cluster is down for hardware)

Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:

VM VMID From To RAM freed Storage stays
tsys-siem 901 tsys9 (131%) tsys6 (38%) 8 GB on tsys9 D2 (tsys4)
tsys-librenms 104 tsys1 (118%) tsys7 (30%) 2 GB on tsys1 D2 (tsys4)
tsys-pdm 105 tsys1 tsys7 2 GB D2 (tsys4)
tsys-workbench 117 tsys1 tsys7 4 GB D2 (tsys4)
tailscale-router 109 tsys1 tsys7 2 GB D2 (tsys4)
kali-tsys 114 tsys1 tsys7 2 GB D2 (tsys4)

Result: tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from 30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily.

Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):

VM VMID From To Hardware
pfv-bms 100 tsys1 tsys9 Zigbee + Z-Wave dongles (physically move)
tsys-ca 101 tsys1 tsys9 Nitrokey HSMs (physically move)

Result: tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only).

Wnode storage rebalancing:

VM VMID Disk from Disk to
wnode-tsys6 100 D5 (tsys4) S4 (tsys5)
wnode-tsys7 701 D5 (tsys4) S1 (tsys5)

5.4 Post-hardware cnode quorum fix (optional, if local-SSD available)

After Friday hardware work, consider moving one cnode to local-lvm:

  • cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency
  • cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd

This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS. Any single failure (host, storage server, or disk) leaves 2 of 3 alive. Deferred — user wants NFS-only for now.

6. Complete VM Inventory (running VMs only)

tsys1 — 11 running (Infrastructure)

VMID Name CPU RAM (MB) Disk Storage Server
100 pfv-bms 2 4096 32 GB D2 tsys4
101 tsys-ca 2 2048 32 GB D2 tsys4
102 pfv-k8s-wnode-tsys1 4 4096 32 GB S3 tsys5
103 pfv-netinfra-01 2 2048 32 GB S2 tsys5
104 tsys-librenms 2 2048 50 GB D2 tsys4
105 tsys-proxmox-datacenter 2 2048 32 GB D2 tsys4
106 pfv-k8s-cnode3 2 4096 32 GB D5 tsys4
108 tsys-ucs-01 2×2 8000 32 GB D2 tsys4
109 tailscale-router 2 2048 25 GB D2 tsys4
114 kali-tsys 2 2048 32 GB D2 tsys4
117 tsys-secure-workbench 2 4000 32 GB D2 tsys4

tsys3 — 1 running (Kubernetes)

VMID Name CPU RAM (MB) Disk Storage Server
313 pfv-k8s-wnode-tsys3 8 28000 32 GB D5 tsys4

tsys4 — 1 running (Storage)

VMID Name CPU RAM (MB) Disk Storage Server
400 pfv-proxmox-backup-server 2 2048 32 GB local-lvm LOCAL

tsys5 — 17 running (Storage + Preprod)

VMID Name CPU RAM (MB) Disk Storage Server
509 pfv-k8s-wnode-tsys5 2×4 32000 32 GB D2 tsys4
5101 sectestbed-siem 2×2 10000 132 GB local-nonprod LOCAL
5105 sectestbed-awx 2×2 4096 288 GB local-nonprod LOCAL
5106 sectestbed-k8s-cnode 2×2 4096 32 GB local-nonprod LOCAL
5107 sectestbed-k8s-wnode 2×2 4096 32 GB local-nonprod LOCAL
5108 sectestbed-librenms 2×2 4096 32 GB local-nonprod LOCAL
5109 sectestbed-netinfra 2×2 4096 32 GB local-nonprod LOCAL
5111 ultix-streaming 2×2 9000 288 GB T5-SSD tsys5 (SSD)
5112 ultix-offstage 2×2 6000 288 GB local-lvm LOCAL
6000 sectestbed-sandbox 2×2 4096 32 GB local-nonprod LOCAL
51010 sectestbed-tctc 2×2 4096 32 GB local-nonprod LOCAL
51011 sectestbed-cloudron 2×2 4096 32 GB local-nonprod LOCAL
51012 sectestbed-hfnoc 2×2 4096 32 GB local-nonprod LOCAL
51013 sectestbed-rancherplatform 2×2 4096 32 GB local-nonprod LOCAL
53100 tsys-preprod-awx 2×2 9000 160 GB local-nonprod LOCAL
53101 tsys-preprod-siem 2×2 12000 32 GB local-nonprod LOCAL
53102 tsys-preprod-rancherplatform 2×2 8000 32 GB local-nonprod LOCAL

tsys6 — 3 running (Kubernetes)

VMID Name CPU RAM (MB) Disk Storage Server
100 pfv-k8s-wnode-tsys6 2×2 32000 32 GB D5 tsys4
600 tsys-awx 2×2 12000 32 GB D2 tsys4
601 pfv-k8s-cnode4 4 4096 32 GB D2 tsys4

tsys7 — 6 running (Kubernetes)

VMID Name CPU RAM (MB) Disk Storage Server
701 pfv-k8s-wnode-tsys7 4 32000 32 GB D5 tsys4
702 hfnoc-uisp 2×2 8000 100 GB D2 tsys4
703 rr-middleware 2 2048 32 GB D2 tsys4
704 TCTC 4 6000 32 GB D2 tsys4
705 pfv-k8s-cnode2 4 4096 32 GB D2 tsys4
706 pfv-k8s-cnode5 4 4096 32 GB S2 tsys5

tsys9 — 6 running (Infrastructure)

VMID Name CPU RAM (MB) Disk Storage Server
901 tsys-siem 2 8000 132 GB D2 tsys4
902 tsys-ucs-02 2×2 8000 50 GB S2 tsys5
903 kali-rd 2 2048 32 GB D5 tsys4
904 pfv-netinfra-02 2 4000 32 GB D2 tsys4
905 pfv-k8s-wnode-tsys9 4 4096 32 GB S2 tsys5
906 pfv-k8s-cnode1 2 4096 32 GB S3 tsys5
Reference in New Issue View Git Blame Copy Permalink