KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
aeb2e80ab15b52e9f9b927a32da38b24bc0aa15f
PFVCluster/docs/proxmox/AUDIT-2026-07-28.md
T
mrcharles 21adb89d4e docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts 
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-28 20:07:09 -05:00

255 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Fresh Fleet Audit — 2026-07-28
> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**
> The tables in PROJECT.md reflect the 2026-07-27 audit; VMs have since been
> migrated via PDM. This file is the current ground truth.
**Audit time:** 2026-07-28 19:55 CDT
**Method:** `qm list` + `qm config` on all 7 reachable hosts
**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9
**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
---
## 1. Host Summary
| Host | CPU | Threads | RAM (GB) | Local Disk | Role | Tuning |
|------|-----|---------|----------|-----------|------|--------|
| pfv-tsys1 | i7-4770 Haswell | 8 | 31 | HDD 932 GB | Infrastructure | Done |
| pfv-tsys3 | Xeon E3-1535M v5 Skylake | 8 | 31 | **NVMe 477 GB** | Kubernetes | Done |
| pfv-tsys4 | Xeon E3-1246 v3 Haswell | 8 | **15** | 6 disks (HDD+SSD) | Storage (NFS+PBS) | Blocked (NIC+RAM) |
| pfv-tsys5 | Xeon E5620 Westmere | 8 | **94** | 6 disks (HDD+SSD) | Storage+Preprod | Blocked (cable) |
| pfv-tsys6 | 2x Xeon E5530 Nehalem | 16 | 127 | HDD (USB 2.0!) | Kubernetes | Done |
| pfv-tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge | 24 | 191 | HDD (USB 2.0!) | Kubernetes | Done |
| pfv-tsys9 | i5-10500 Comet Lake | 12 | 24 | SSD 250 GB | Infrastructure | Done |
**Changes since last audit:**
- tsys4 RAM still 15 GB (Friday 64 GB upgrade pending)
- tsys5 RAM is 94 GB (was documented as 96 GB)
- tsys6 RAM is 127 GB (was documented as 128 GB)
- tsys7 RAM is 191 GB (was documented as 192 GB)
---
## 2. VM Fleet Inventory (Running VMs Only)
### tsys1 — 11 running VMs (Infrastructure)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 100 | pfv-bms (HomeAssistant) | 2×1 | 4096 | 32 GB | D2 | tsys4 |
| 101 | tsys-ca | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 103 | **pfv-netinfra-01** | 2×1 | 2048 | 32 GB | D5 | tsys4 |
| 104 | tsys-librenms | 2×1 | 2048 | 50 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 106 | **pfv-k8s-cnode3** | 2×1 | 4096 | 32 GB | **S3** | **tsys5** |
| 108 | **tsys-ucs-01** | 2×2 | 8000 | 32 GB | D2 | tsys4 |
| 109 | tailscale-router | 2×1 | 2048 | 25 GB | D2 | tsys4 |
| 114 | kali-tsys | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2×1 | 4000 | 32 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4×1 | 4096 | 32 GB | S2 | tsys5 — **STOPPED** |
### tsys3 — 1 running VM (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 313 | **pfv-k8s-wnode-tsys3** | 8×1 | 28000 | 32 GB | D5 | tsys4 |
### tsys4 — 1 running VM (Storage)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 400 | pfv-proxmox-backup-server | 2×1 | 2048 | 32 GB | local-lvm | LOCAL |
### tsys5 — 16 running VMs (Storage + Preprod)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 509 | **pfv-k8s-wnode-tsys5** | 2×4 | 32000 | 32 GB | D2 | tsys4 |
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
### tsys6 — 3 running VMs (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 100 | **pfv-k8s-wnode-tsys6** | 2×2 | 32000 | 32 GB | D5 | tsys4 |
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
| 601 | pfv-k8s-cnode4 | 4×1 | 4096 | 32 GB | D2 | tsys4 |
### tsys7 — 6 running VMs (Kubernetes)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 701 | **pfv-k8s-wnode-tsys7** | 4×1 | 32000 | 32 GB | D5 | tsys4 |
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
| 703 | rr-middleware | 2×1 | 2048 | 32 GB | D2 | tsys4 |
| 704 | TCTC | 4×1 | 6000 | 32 GB | D2 | tsys4 |
| 705 | **pfv-k8s-cnode2** | 4×1 | 4096 | 32 GB | D2 | tsys4 |
| 706 | pfv-k8s-cnode5 | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
### tsys9 — 6 running VMs (Infrastructure)
| VMID | Name | Cores×Sockets | RAM (MB) | Disk | Storage | NFS Server |
|------|------|--------------|----------|------|---------|-----------|
| 901 | tsys-siem | 2×1 | 8000 | 132 GB | D2 | tsys4 |
| 902 | **tsys-ucs-02** | 2×2 | 8000 | 50 GB | D5 | tsys4 |
| 903 | kali-rd | 2×1 | 2048 | 32 GB | D5 | tsys4 |
| 904 | **pfv-netinfra-02** | 2×1 | 4000 | 32 GB | D2 | tsys4 |
| 905 | **pfv-k8s-wnode-tsys9** | 4×1 | 4096 | 32 GB | **S2** | **tsys5** |
| 906 | **pfv-k8s-cnode1** | 2×1 | 4096 | 32 GB | D5 | tsys4 |
---
## 3. Kubernetes Node Placement
### 3.1 Active k3s cluster (cnode1/2/3 — deployed this session)
| Cnode | VMID | Host | Disk | NFS Server | Quorum risk |
|-------|------|------|------|-----------|-------------|
| cnode1 | **906** | tsys9 | D5 | **tsys4** | **HIGH** |
| cnode2 | **705** | tsys7 | D2 | **tsys4** | **HIGH** |
| cnode3 | 106 | tsys1 | S3 | tsys5 | OK |
**CRITICAL: 2 of 3 active cnodes on tsys4.** tsys4 failure = cnode1 + cnode2 die = 1 of 3 = **QUORUM LOST**.
**Fix needed:** Move cnode1 or cnode2 to tsys5 storage (S2 or S3). One migration via PDM "Storage Migrate" solves this.
### 3.2 Inactive cnodes (cnode4/5 — exist but not in k3s cluster)
| Cnode | VMID | Host | Disk | NFS Server |
|-------|------|------|------|-----------|
| cnode4 | 601 | tsys6 | D2 | tsys4 |
| cnode5 | 706 | tsys7 | S2 | tsys5 |
### 3.3 Worker nodes (wnodes)
| Wnode | VMID | Host | Disk | NFS Server | RAM | Status |
|-------|------|------|------|-----------|-----|--------|
| wnode-tsys1 | 102 | tsys1 | S2 | tsys5 | 4 GB | STOPPED |
| wnode-tsys3 | 313 | tsys3 | D5 | tsys4 | 28 GB | Running |
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | 32 GB | Running |
| wnode-tsys6 | 100 | tsys6 | D5 | tsys4 | 32 GB | Running |
| wnode-tsys7 | 701 | tsys7 | D5 | tsys4 | 32 GB | Running |
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | 4 GB | Running |
**Storage: 4 wnodes on tsys4, 2 on tsys5.** One wnode per host achieved.
---
## 4. Critical HA Pairs — Storage Redundancy
### 4.1 netinfra pair — FAILED (both on tsys4)
| Role | VMID | Host | Disk | NFS Server |
|------|------|------|------|-----------|
| netinfra-01 | 103 | tsys1 | D5 | **tsys4** |
| netinfra-02 | 904 | tsys9 | D2 | **tsys4** |
**tsys4 failure = DNS/DHCP/NTP goes fully dark.**
**Fix:** Migrate netinfra-02 (VMID 904) from D2 (tsys4) to S3 (tsys5) via PDM.
### 4.2 UCS pair — FAILED (both on tsys4)
| Role | VMID | Host | Disk | NFS Server |
|------|------|------|------|-----------|
| ucs-01 | 108 | tsys1 | D2 | **tsys4** |
| ucs-02 | 902 | tsys9 | D5 | **tsys4** |
**tsys4 failure = LDAP/AD goes fully dark.**
**Fix:** Migrate ucs-02 (VMID 902) from D5 (tsys4) to S2 (tsys5) via PDM.
### 4.3 Corrected placement (after migration)
| VM | Host | Disk | NFS Server | Failure survival |
|----|------|------|-----------|-----------------|
| netinfra-01 | tsys1 | D5 | tsys4 | tsys4 dies → netinfra-02 alive on tsys5 |
| netinfra-02 | tsys9 | **S3** | **tsys5** | tsys5 dies → netinfra-01 alive on tsys4 |
| ucs-01 | tsys1 | D2 | tsys4 | tsys4 dies → ucs-02 alive on tsys5 |
| ucs-02 | tsys9 | **S2** | **tsys5** | tsys5 dies → ucs-01 alive on tsys4 |
---
## 5. Storage Concentration
| Storage target | # running VMs | % of fleet |
|---------------|--------------|------------|
| D2 (tsys4 WDC Red 3TB HDD) | 16 | 36% |
| D5 (tsys4 Hitachi 2TB HDD) | 10 | 23% |
| local-nonprod (tsys5 local HDD) | 12 | 27% |
| S2 (tsys5 Seagate 1TB HDD) | 3 | 7% |
| S3 (tsys5 Seagate 1TB HDD) | 1 | 2% |
| T5-SSD (tsys5 Samsung SSD) | 1 | 2% |
| local-lvm (various hosts) | 2 | 5% |
**26 of 44 running VMs (59%) store their disks on tsys4 NFS exports.**
(Was 68% in the previous audit — improving but still concentrated.)
---
## 6. Pre-k8s Buildout Action Items
These must be done before or during k8s worker node bringup:
### 6.1 CRITICAL: Migrate HA pairs to separate storage (PDM, 10 min)
1. `netinfra-02` (VMID 904): D2 → S3 (tsys4 → tsys5)
2. `ucs-02` (VMID 902): D5 → S2 (tsys4 → tsys5)
These are PDM "Storage Migrate" operations — no VM rebuild needed.
### 6.2 CRITICAL: Fix active cnode quorum (PDM, 5 min)
Move one active cnode from tsys4 to tsys5 storage:
- Best candidate: cnode1 (VMID 906) D5 → S2 (tsys4 → tsys5)
- Result: cnode1 on tsys5, cnode2 on tsys4, cnode3 on tsys5
- tsys4 failure = cnode2 dies only = 2 of 3 = **quorum OK**
### 6.3 Join workers to k3s cluster
Current wnodes exist as VMs but are not joined to the k3s cluster. Need to:
1. Install k3s agent on each wnode (using join token from cnode1)
2. Configure `--node-ip=<tailscale-ip>` on each
3. Label/taint per workload role
### 6.4 Friday hardware work (still pending)
- tsys4: PCIe NIC + 64 GB RAM (currently 15 GB)
- tsys5: 2nd ethernet cable + NVMe + D3 SSD relocation
- tsys2: Rebuild from Win10 to Proxmox
---
## 7. Changes Since Previous Audit (2026-07-27)
| What | Before | After |
|------|--------|-------|
| cnode1 VMID | 107 (tsys1, D5) | **906** (tsys9, D5) |
| cnode2 VMID | 603 (tsys6, D2) | **705** (tsys7, D2) |
| cnode3 storage | D2 (tsys4) | **S3** (tsys5) |
| cnode5 VMID | 602 (tsys6) | **706** (tsys7, S2 tsys5) |
| wnode-tsys1 | not listed | VMID 102 (S2 tsys5, stopped) |
| wnode-tsys3 RAM | 20 GB | **28 GB** |
| tsys-awx (600) | STOPPED | **Running** |
| DellOpenManageEnterprise (500) | not listed | VMID 500 (D7, stopped) |
| sectestbed-librenms (5108) | not listed | Running |
| k3s cluster | not deployed | **3-node HA live** (cnode1/2/3) |
| Console management | manual screen | **ser2net+conman on tsys4** |
| PDU management | manual | **powerman on tsys1** |
Reference in New Issue View Git Blame Copy Permalink