KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
aeb2e80ab15b52e9f9b927a32da38b24bc0aa15f
PFVCluster/docs/proxmox/AUDIT-2026-07-28.md
T
mrcharles 21adb89d4e docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts 
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-28 20:07:09 -05:00

11 KiB
Raw Blame History

Fresh Fleet Audit — 2026-07-28

Supersedes placement data in PROJECT.md §4-§8. The tables in PROJECT.md reflect the 2026-07-27 audit; VMs have since been migrated via PDM. This file is the current ground truth.

Audit time: 2026-07-28 19:55 CDT Method: qm list + qm config on all 7 reachable hosts Hosts audited: pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9 Hosts offline: pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)

1. Host Summary

Host CPU Threads RAM (GB) Local Disk Role Tuning
pfv-tsys1 i7-4770 Haswell 8 31 HDD 932 GB Infrastructure Done
pfv-tsys3 Xeon E3-1535M v5 Skylake 8 31 NVMe 477 GB Kubernetes Done
pfv-tsys4 Xeon E3-1246 v3 Haswell 8 15 6 disks (HDD+SSD) Storage (NFS+PBS) Blocked (NIC+RAM)
pfv-tsys5 Xeon E5620 Westmere 8 94 6 disks (HDD+SSD) Storage+Preprod Blocked (cable)
pfv-tsys6 2x Xeon E5530 Nehalem 16 127 HDD (USB 2.0!) Kubernetes Done
pfv-tsys7 2x Xeon E5-2630 v2 Ivy Bridge 24 191 HDD (USB 2.0!) Kubernetes Done
pfv-tsys9 i5-10500 Comet Lake 12 24 SSD 250 GB Infrastructure Done

Changes since last audit:

  • tsys4 RAM still 15 GB (Friday 64 GB upgrade pending)
  • tsys5 RAM is 94 GB (was documented as 96 GB)
  • tsys6 RAM is 127 GB (was documented as 128 GB)
  • tsys7 RAM is 191 GB (was documented as 192 GB)

2. VM Fleet Inventory (Running VMs Only)

tsys1 — 11 running VMs (Infrastructure)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
100 pfv-bms (HomeAssistant) 2×1 4096 32 GB D2 tsys4
101 tsys-ca 2×1 2048 32 GB D2 tsys4
103 pfv-netinfra-01 2×1 2048 32 GB D5 tsys4
104 tsys-librenms 2×1 2048 50 GB D2 tsys4
105 tsys-proxmox-datacenter 2×1 2048 32 GB D2 tsys4
106 pfv-k8s-cnode3 2×1 4096 32 GB S3 tsys5
108 tsys-ucs-01 2×2 8000 32 GB D2 tsys4
109 tailscale-router 2×1 2048 25 GB D2 tsys4
114 kali-tsys 2×1 2048 32 GB D2 tsys4
117 tsys-secure-workbench 2×1 4000 32 GB D2 tsys4
102 pfv-k8s-wnode-tsys1 4×1 4096 32 GB S2 tsys5 — STOPPED

tsys3 — 1 running VM (Kubernetes)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
313 pfv-k8s-wnode-tsys3 8×1 28000 32 GB D5 tsys4

tsys4 — 1 running VM (Storage)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
400 pfv-proxmox-backup-server 2×1 2048 32 GB local-lvm LOCAL

tsys5 — 16 running VMs (Storage + Preprod)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
509 pfv-k8s-wnode-tsys5 2×4 32000 32 GB D2 tsys4
5101 sectestbed-siem 2×2 10000 132 GB local-nonprod LOCAL
5105 sectestbed-awx 2×2 4096 288 GB local-nonprod LOCAL
5106 sectestbed-k8s-cnode 2×2 4096 32 GB local-nonprod LOCAL
5107 sectestbed-k8s-wnode 2×2 4096 32 GB local-nonprod LOCAL
5108 sectestbed-librenms 2×2 4096 32 GB local-nonprod LOCAL
5109 sectestbed-netinfra 2×2 4096 32 GB local-nonprod LOCAL
5111 ultix-streaming 2×2 9000 288 GB T5-SSD tsys5 (SSD)
5112 ultix-offstage 2×2 6000 288 GB local-lvm LOCAL
6000 sectestbed-sandbox 2×2 4096 32 GB local-nonprod LOCAL
51010 sectestbed-tctc 2×2 4096 32 GB local-nonprod LOCAL
51011 sectestbed-cloudron 2×2 4096 32 GB local-nonprod LOCAL
51012 sectestbed-hfnoc 2×2 4096 32 GB local-nonprod LOCAL
51013 sectestbed-rancherplatform 2×2 4096 32 GB local-nonprod LOCAL
53100 tsys-preprod-awx 2×2 9000 160 GB local-nonprod LOCAL
53101 tsys-preprod-siem 2×2 12000 32 GB local-nonprod LOCAL
53102 tsys-preprod-rancherplatform 2×2 8000 32 GB local-nonprod LOCAL

tsys6 — 3 running VMs (Kubernetes)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
100 pfv-k8s-wnode-tsys6 2×2 32000 32 GB D5 tsys4
600 tsys-awx 2×2 12000 32 GB D2 tsys4
601 pfv-k8s-cnode4 4×1 4096 32 GB D2 tsys4

tsys7 — 6 running VMs (Kubernetes)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
701 pfv-k8s-wnode-tsys7 4×1 32000 32 GB D5 tsys4
702 hfnoc-uisp 2×2 8000 100 GB D2 tsys4
703 rr-middleware 2×1 2048 32 GB D2 tsys4
704 TCTC 4×1 6000 32 GB D2 tsys4
705 pfv-k8s-cnode2 4×1 4096 32 GB D2 tsys4
706 pfv-k8s-cnode5 4×1 4096 32 GB S2 tsys5

tsys9 — 6 running VMs (Infrastructure)

VMID Name Cores×Sockets RAM (MB) Disk Storage NFS Server
901 tsys-siem 2×1 8000 132 GB D2 tsys4
902 tsys-ucs-02 2×2 8000 50 GB D5 tsys4
903 kali-rd 2×1 2048 32 GB D5 tsys4
904 pfv-netinfra-02 2×1 4000 32 GB D2 tsys4
905 pfv-k8s-wnode-tsys9 4×1 4096 32 GB S2 tsys5
906 pfv-k8s-cnode1 2×1 4096 32 GB D5 tsys4

3. Kubernetes Node Placement

3.1 Active k3s cluster (cnode1/2/3 — deployed this session)

Cnode VMID Host Disk NFS Server Quorum risk
cnode1 906 tsys9 D5 tsys4 HIGH
cnode2 705 tsys7 D2 tsys4 HIGH
cnode3 106 tsys1 S3 tsys5 OK

CRITICAL: 2 of 3 active cnodes on tsys4. tsys4 failure = cnode1 + cnode2 die = 1 of 3 = QUORUM LOST.

Fix needed: Move cnode1 or cnode2 to tsys5 storage (S2 or S3). One migration via PDM "Storage Migrate" solves this.

3.2 Inactive cnodes (cnode4/5 — exist but not in k3s cluster)

Cnode VMID Host Disk NFS Server
cnode4 601 tsys6 D2 tsys4
cnode5 706 tsys7 S2 tsys5

3.3 Worker nodes (wnodes)

Wnode VMID Host Disk NFS Server RAM Status
wnode-tsys1 102 tsys1 S2 tsys5 4 GB STOPPED
wnode-tsys3 313 tsys3 D5 tsys4 28 GB Running
wnode-tsys5 509 tsys5 D2 tsys4 32 GB Running
wnode-tsys6 100 tsys6 D5 tsys4 32 GB Running
wnode-tsys7 701 tsys7 D5 tsys4 32 GB Running
wnode-tsys9 905 tsys9 S2 tsys5 4 GB Running

Storage: 4 wnodes on tsys4, 2 on tsys5. One wnode per host achieved.

4. Critical HA Pairs — Storage Redundancy

4.1 netinfra pair — FAILED (both on tsys4)

Role VMID Host Disk NFS Server
netinfra-01 103 tsys1 D5 tsys4
netinfra-02 904 tsys9 D2 tsys4

tsys4 failure = DNS/DHCP/NTP goes fully dark.

Fix: Migrate netinfra-02 (VMID 904) from D2 (tsys4) to S3 (tsys5) via PDM.

4.2 UCS pair — FAILED (both on tsys4)

Role VMID Host Disk NFS Server
ucs-01 108 tsys1 D2 tsys4
ucs-02 902 tsys9 D5 tsys4

tsys4 failure = LDAP/AD goes fully dark.

Fix: Migrate ucs-02 (VMID 902) from D5 (tsys4) to S2 (tsys5) via PDM.

4.3 Corrected placement (after migration)

VM Host Disk NFS Server Failure survival
netinfra-01 tsys1 D5 tsys4 tsys4 dies → netinfra-02 alive on tsys5
netinfra-02 tsys9 S3 tsys5 tsys5 dies → netinfra-01 alive on tsys4
ucs-01 tsys1 D2 tsys4 tsys4 dies → ucs-02 alive on tsys5
ucs-02 tsys9 S2 tsys5 tsys5 dies → ucs-01 alive on tsys4

5. Storage Concentration

Storage target # running VMs % of fleet
D2 (tsys4 WDC Red 3TB HDD) 16 36%
D5 (tsys4 Hitachi 2TB HDD) 10 23%
local-nonprod (tsys5 local HDD) 12 27%
S2 (tsys5 Seagate 1TB HDD) 3 7%
S3 (tsys5 Seagate 1TB HDD) 1 2%
T5-SSD (tsys5 Samsung SSD) 1 2%
local-lvm (various hosts) 2 5%

26 of 44 running VMs (59%) store their disks on tsys4 NFS exports. (Was 68% in the previous audit — improving but still concentrated.)

6. Pre-k8s Buildout Action Items

These must be done before or during k8s worker node bringup:

6.1 CRITICAL: Migrate HA pairs to separate storage (PDM, 10 min)

  1. netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5)
  2. ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5)

These are PDM "Storage Migrate" operations — no VM rebuild needed.

6.2 CRITICAL: Fix active cnode quorum (PDM, 5 min)

Move one active cnode from tsys4 to tsys5 storage:

  • Best candidate: cnode1 (VMID 906) D5 → S2 (tsys4 → tsys5)
  • Result: cnode1 on tsys5, cnode2 on tsys4, cnode3 on tsys5
  • tsys4 failure = cnode2 dies only = 2 of 3 = quorum OK

6.3 Join workers to k3s cluster

Current wnodes exist as VMs but are not joined to the k3s cluster. Need to:

  1. Install k3s agent on each wnode (using join token from cnode1)
  2. Configure --node-ip=<tailscale-ip> on each
  3. Label/taint per workload role

6.4 Friday hardware work (still pending)

  • tsys4: PCIe NIC + 64 GB RAM (currently 15 GB)
  • tsys5: 2nd ethernet cable + NVMe + D3 SSD relocation
  • tsys2: Rebuild from Win10 to Proxmox

7. Changes Since Previous Audit (2026-07-27)

What Before After
cnode1 VMID 107 (tsys1, D5) 906 (tsys9, D5)
cnode2 VMID 603 (tsys6, D2) 705 (tsys7, D2)
cnode3 storage D2 (tsys4) S3 (tsys5)
cnode5 VMID 602 (tsys6) 706 (tsys7, S2 tsys5)
wnode-tsys1 not listed VMID 102 (S2 tsys5, stopped)
wnode-tsys3 RAM 20 GB 28 GB
tsys-awx (600) STOPPED Running
DellOpenManageEnterprise (500) not listed VMID 500 (D7, stopped)
sectestbed-librenms (5108) not listed Running
k3s cluster not deployed 3-node HA live (cnode1/2/3)
Console management manual screen ser2net+conman on tsys4
PDU management manual powerman on tsys1
Reference in New Issue View Git Blame Copy Permalink