KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts

Browse Source 
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-28 20:07:09 -05:00
parent 0600321968
commit 28e0b0c7a6
13 changed files with 281 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/proxmox/AUDIT-2026-07-28.md
+254
View File
@@ -0,0 +1,254 @@
# Fresh Fleet Audit — 2026-07-28
> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**
> The tables in PROJECT.md reflect the 2026-07-27 audit; VMs have since been
> migrated via PDM. This file is the current ground truth.
**Audit time:** 2026-07-28 19:55 CDT
**Method:** `qm list` + `qm config` on all 7 reachable hosts
**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9
**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
---
## 1. Host Summary
| Host | CPU | Threads | RAM (GB) | Local Disk | Role | Tuning |
|------|-----|---------|----------|-----------|------|--------|
| pfv-tsys1 | i7-4770 Haswell | 8 | 31 | HDD 932 GB | Infrastructure | Done |
| pfv-tsys3 | Xeon E3-1535M v5 Skylake | 8 | 31 | **NVMe 477 GB** | Kubernetes | Done |
| pfv-tsys4 | Xeon E3-1246 v3 Haswell | 8 | **15** | 6 disks (HDD+SSD) | Storage (NFS+PBS) | Blocked (NIC+RAM) |
| pfv-tsys5 | Xeon E5620 Westmere | 8 | **94** | 6 disks (HDD+SSD) | Storage+Preprod | Blocked (cable) |
| pfv-tsys6 | 2x Xeon E5530 Nehalem | 16 | 127 | HDD (USB 2.0!) | Kubernetes | Done |
| pfv-tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge | 24 | 191 | HDD (USB 2.0!) | Kubernetes | Done |
| pfv-tsys9 | i5-10500 Comet Lake | 12 | 24 | SSD 250 GB | Infrastructure | Done |
**Changes since last audit:**
- tsys4 RAM still 15 GB (Friday 64 GB upgrade pending)
- tsys5 RAM is 94 GB (was documented as 96 GB)
- tsys6 RAM is 127 GB (was documented as 128 GB)
- tsys7 RAM is 191 GB (was documented as 192 GB)
---
## 2. VM Fleet Inventory (Running VMs Only)
### tsys1 — 11 running VMs (Infrastructure)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 100 | pfv-bms (HomeAssistant) | 2×1 | 4096 | 32 GB | D2 | tsys4 |
| 101 | tsys-ca | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 103 | **pfv-netinfra-01** | 2×1 | 2048 | 32 GB | D5 | tsys4 |
| 104 | tsys-librenms | 2×1 | 2048 | 50 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 106 | **pfv-k8s-cnode3** | 2×1 | 4096 | 32 GB | **S3** | **tsys5** |
| 108 | **tsys-ucs-01** | 2×2 | 8000 | 32 GB | D2 | tsys4 |
| 109 | tailscale-router | 2×1 | 2048 | 25 GB | D2 | tsys4 |
| 114 | kali-tsys | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2×1 | 4000 | 32 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4×1 | 4096 | 32 GB | S2 | tsys5 — **STOPPED** |
### tsys3 — 1 running VM (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 313 | **pfv-k8s-wnode-tsys3** | 8×1 | 28000 | 32 GB | D5 | tsys4 |
### tsys4 — 1 running VM (Storage)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 400 | pfv-proxmox-backup-server | 2×1 | 2048 | 32 GB | local-lvm | LOCAL |
### tsys5 — 16 running VMs (Storage + Preprod)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 509 | **pfv-k8s-wnode-tsys5** | 2×4 | 32000 | 32 GB | D2 | tsys4 |
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
### tsys6 — 3 running VMs (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 100 | **pfv-k8s-wnode-tsys6** | 2×2 | 32000 | 32 GB | D5 | tsys4 |
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
| 601 | pfv-k8s-cnode4 | 4×1 | 4096 | 32 GB | D2 | tsys4 |
### tsys7 — 6 running VMs (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 701 | **pfv-k8s-wnode-tsys7** | 4×1 | 32000 | 32 GB | D5 | tsys4 |
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
| 703 | rr-middleware | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 704 | TCTC | 4×1 | 6000 | 32 GB | D2 | tsys4 |
| 705 | **pfv-k8s-cnode2** | 4×1 | 4096 | 32 GB | D2 | tsys4 |
| 706 | pfv-k8s-cnode5 | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
### tsys9 — 6 running VMs (Infrastructure)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 901 | tsys-siem | 2×1 | 8000 | 132 GB | D2 | tsys4 |
| 902 | **tsys-ucs-02** | 2×2 | 8000 | 50 GB | D5 | tsys4 |
| 903 | kali-rd | 2×1 | 2048 | 32 GB | D5 | tsys4 |
| 904 | **pfv-netinfra-02** | 2×1 | 4000 | 32 GB | D2 | tsys4 |
| 905 | **pfv-k8s-wnode-tsys9** | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
| 906 | **pfv-k8s-cnode1** | 2×1 | 4096 | 32 GB | D5 | tsys4 |
---
## 3. Kubernetes Node Placement
### 3.1 Active k3s cluster (cnode1/2/3 — deployed this session)
| Cnode | VMID | Host | Disk | NFS Server | Quorum risk |
|-------|------|------|------|-----------|-------------|
| cnode1 | **906** | tsys9 | D5 | **tsys4** | **HIGH** |
| cnode2 | **705** | tsys7 | D2 | **tsys4** | **HIGH** |
| cnode3 | 106 | tsys1 | S3 | tsys5 | OK |
**CRITICAL: 2 of 3 active cnodes on tsys4.** tsys4 failure = cnode1 + cnode2 die = 1 of 3 = **QUORUM LOST**.
**Fix needed:** Move cnode1 or cnode2 to tsys5 storage (S2 or S3). One migration via PDM "Storage Migrate" solves this.
### 3.2 Inactive cnodes (cnode4/5 — exist but not in k3s cluster)
| Cnode | VMID | Host | Disk | NFS Server |
|-------|------|------|------|-----------|
| cnode4 | 601 | tsys6 | D2 | tsys4 |
| cnode5 | 706 | tsys7 | S2 | tsys5 |
### 3.3 Worker nodes (wnodes)
| Wnode | VMID | Host | Disk | NFS Server | RAM | Status |
|-------|------|------|------|-----------|-----|--------|
| wnode-tsys1 | 102 | tsys1 | S2 | tsys5 | 4 GB | STOPPED |
| wnode-tsys3 | 313 | tsys3 | D5 | tsys4 | 28 GB | Running |
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | 32 GB | Running |
| wnode-tsys6 | 100 | tsys6 | D5 | tsys4 | 32 GB | Running |
| wnode-tsys7 | 701 | tsys7 | D5 | tsys4 | 32 GB | Running |
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | 4 GB | Running |
**Storage: 4 wnodes on tsys4, 2 on tsys5.** One wnode per host achieved.
---
## 4. Critical HA Pairs — Storage Redundancy
### 4.1 netinfra pair — FAILED (both on tsys4)
| Role | VMID | Host | Disk | NFS Server |
|------|------|------|------|-----------|
| netinfra-01 | 103 | tsys1 | D5 | **tsys4** |
| netinfra-02 | 904 | tsys9 | D2 | **tsys4** |
**tsys4 failure = DNS/DHCP/NTP goes fully dark.**
**Fix:** Migrate netinfra-02 (VMID 904) from D2 (tsys4) to S3 (tsys5) via PDM.
### 4.2 UCS pair — FAILED (both on tsys4)
| Role | VMID | Host | Disk | NFS Server |
|------|------|------|------|-----------|
| ucs-01 | 108 | tsys1 | D2 | **tsys4** |
| ucs-02 | 902 | tsys9 | D5 | **tsys4** |
**tsys4 failure = LDAP/AD goes fully dark.**
**Fix:** Migrate ucs-02 (VMID 902) from D5 (tsys4) to S2 (tsys5) via PDM.
### 4.3 Corrected placement (after migration)
| VM | Host | Disk | NFS Server | Failure survival |
|----|------|------|-----------|-----------------|
| netinfra-01 | tsys1 | D5 | tsys4 | tsys4 dies → netinfra-02 alive on tsys5 |
| netinfra-02 | tsys9 | **S3** | **tsys5** | tsys5 dies → netinfra-01 alive on tsys4 |
| ucs-01 | tsys1 | D2 | tsys4 | tsys4 dies → ucs-02 alive on tsys5 |
| ucs-02 | tsys9 | **S2** | **tsys5** | tsys5 dies → ucs-01 alive on tsys4 |
---
## 5. Storage Concentration
| Storage target | # running VMs | % of fleet |
|---------------|--------------|------------|
| D2 (tsys4 WDC Red 3TB HDD) | 16 | 36% |
| D5 (tsys4 Hitachi 2TB HDD) | 10 | 23% |
| local-nonprod (tsys5 local HDD) | 12 | 27% |
| S2 (tsys5 Seagate 1TB HDD) | 3 | 7% |
| S3 (tsys5 Seagate 1TB HDD) | 1 | 2% |
| T5-SSD (tsys5 Samsung SSD) | 1 | 2% |
| local-lvm (various hosts) | 2 | 5% |
**26 of 44 running VMs (59%) store their disks on tsys4 NFS exports.**
(Was 68% in the previous audit — improving but still concentrated.)
---
## 6. Pre-k8s Buildout Action Items
These must be done before or during k8s worker node bringup:
### 6.1 CRITICAL: Migrate HA pairs to separate storage (PDM, 10 min)
1. `netinfra-02` (VMID 904): D2 → S3 (tsys4 → tsys5)
2. `ucs-02` (VMID 902): D5 → S2 (tsys4 → tsys5)
These are PDM "Storage Migrate" operations — no VM rebuild needed.
### 6.2 CRITICAL: Fix active cnode quorum (PDM, 5 min)
Move one active cnode from tsys4 to tsys5 storage:
- Best candidate: cnode1 (VMID 906) D5 → S2 (tsys4 → tsys5)
- Result: cnode1 on tsys5, cnode2 on tsys4, cnode3 on tsys5
- tsys4 failure = cnode2 dies only = 2 of 3 = **quorum OK**
### 6.3 Join workers to k3s cluster
Current wnodes exist as VMs but are not joined to the k3s cluster. Need to:
1. Install k3s agent on each wnode (using join token from cnode1)
2. Configure `--node-ip=<tailscale-ip>` on each
3. Label/taint per workload role
### 6.4 Friday hardware work (still pending)
- tsys4: PCIe NIC + 64 GB RAM (currently 15 GB)
- tsys5: 2nd ethernet cable + NVMe + D3 SSD relocation
- tsys2: Rebuild from Win10 to Proxmox
---
## 7. Changes Since Previous Audit (2026-07-27)
| What | Before | After |
|------|--------|-------|
| cnode1 VMID | 107 (tsys1, D5) | **906** (tsys9, D5) |
| cnode2 VMID | 603 (tsys6, D2) | **705** (tsys7, D2) |
| cnode3 storage | D2 (tsys4) | **S3** (tsys5) |
| cnode5 VMID | 602 (tsys6) | **706** (tsys7, S2 tsys5) |
| wnode-tsys1 | not listed | VMID 102 (S2 tsys5, stopped) |
| wnode-tsys3 RAM | 20 GB | **28 GB** |
| tsys-awx (600) | STOPPED | **Running** |
| DellOpenManageEnterprise (500) | not listed | VMID 500 (D7, stopped) |
| sectestbed-librenms (5108) | not listed | Running |
| k3s cluster | not deployed | **3-node HA live** (cnode1/2/3) |
| Console management | manual screen | **ser2net+conman on tsys4** |
| PDU management | manual | **powerman on tsys1** |