KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): full ground-truth audit — network, storage, compute, switches

Browse Source 
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.

Key findings documented in AUDIT-2026-07-30.md:

Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
  host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
  Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).

Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
  installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.

Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).

Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-30 23:01:43 -05:00
parent d9b32e4eef
commit 26de312c42
6 changed files with 497 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+50 -43
View File
@@ -3,12 +3,14 @@
> **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — HA NUT integration live
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute)
>
> **Capacity targets are now codified** in
> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md)
> — network topology, switch LAG configs, corrected storage topology, VM config
> audit, fleet consistency, cross-rack NFS root-cause analysis.
> Capacity targets are codified in
> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
> steady-state, ~50% idle CPU. The current placement analysis lives in
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md).
> steady-state, ~50% idle CPU.
## Current State: STABLE
@@ -36,15 +38,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73%
- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg
- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
- [x] **tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator** (verified: 8/8 NFS stores active on both)
- [x] **Full ground-truth audit 2026-07-30** (network + storage + compute + switches)
- [x] **Cross-rack NFS root cause identified:** core-sw01 LAG hash=layer-2-3 (no L4),
g16/g17 cable mismatch (3 of 4 links active), static mode (no LACP)
- [x] **tsys4/tsys5 storage changes documented:** new NVMe (CT500P1 500GB), relocated
SK hynix SSD, D3 changed to USB HDD, /mnt/albert + /mnt/backup added
- [x] **Console symlink bug fixed:** udev rule singular→plural mismatch broke all 7
console ports after reboot
- [x] **Portable audit tooling built:** probe-storage.sh, probe-network.sh,
conman-console.py, snmp-switch-audit.py (AGPLv3-friendly, config-driven)
- [ ] **Memory goal (≤80% actual): PASS fleet-wide** — highest is tsys9 at 69%
- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
- [ ] tsys4: PVE 9.1.5 (behind fleet 9.2.5), needs upgrade
- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Kubernetes Control Plane (k3s HA — LIVE)
@@ -131,50 +137,51 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| Issue | Impact | Status |
|-------|--------|--------|
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 73%/37%/62% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §11 |
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ |
| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
| **D2 IOPS hotspot** | 17 of 26 NFS-backed VMs on one spindle; S1/S4 underused | Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11) |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
| **core-sw01 LAG hash = layer-2-3** | NFS cross-rack capped at 1 Gbps per host pair | Fix: change to layer-2-3-4 (switch change, pending approval) |
| **core-sw01 ch1 g16/g17 mismatch** | Only 3 of 4 cross-rack links active | Fix: change member list g(13-15,17)→g(13-16) |
| **Cross-rack trunk uses `mode on`** | No LACP failure detection | Fix: change to `mode active` (both switches) |
| **tsys5 bond0 degraded** (1 of 2 slaves) | Single link, no redundancy | Physical: 2nd ethernet cable needed |
| **tsys5 NVMe + SSD not configured** | Best local storage unused | Config: mount + add to Proxmox storage.cfg |
| **D2 carries 13 running VMs** | IOPS hotspot persists | Migrate to S2/S3/new disks |
| **tsys4 PVE 9.1.5** | Behind fleet (9.2.5) | Upgrade needed |
| **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Offload to new SK hynix SSD |
| **Port errors on core-sw01** | g9:2902, g38:590, g26:73 inErr | Investigate CRC/cable |
| **tsys5 is single-CPU** | CPU2 socket unpopulated | Consider seating spare LGA771 Xeon |
| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; grow via workloads |
## Pending (next session priorities)
> **Full phased action checklist with current/target inventory is in**
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §11.
> Summary below. SSD/NVMe tiering is deferred to post-Friday.
> **Full audit with findings + recommendations is in**
> [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md).
**Phase 0 — Storage config fixes — DONE ✓** (tsys6 S1 typo, tsys9 S1 missing)
**Awaiting operator review of AUDIT-2026-07-30 before executing any fixes.**
**Phase 1 — Cleanup:**
- Delete cnode4 (601) on tsys6 (stopped, stale)
- cnode5 (706) already deleted ✓
**Switch changes (need approval):**
- Fix core-sw01 ch1 member: g(13-15,17) → g(13-16) [restores 4th link]
- Fix core-sw01 LAG hash: layer-2-3 → layer-2-3-4 [unlocks multi-link NFS]
- Convert trunk to LACP: mode on → mode active [failure detection]
- Remove stale tsys8 config from tor3-stor (ch4/g13/g14)
**Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):**
- wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4
- rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2
**Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):**
- tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117)
- tsys9→tsys6: tsys-siem(901)
**Phase 4 — Grow wnode-tsys9:** 4c/4GB → 6c/8GB (after siem leaves tsys9)
**Phase 5 — Join wnodes to k3s** (after migrations free resources)
**Host-side changes (no switch access needed):**
- Mount + configure tsys5 NVMe (CT500P1 500GB) as LVM-thin
- Mount + configure tsys5 SK hynix SSD (512GB, relocated from tsys4)
- Clean stale S3/S4 exports + storage.cfg on tsys5
- Upgrade tsys4 to PVE 9.2.5 / kernel 7.0.14
- Apply apply-tunings.sh to tsys4/5 (nconnect, bond hash, rmem/wmem)
- Audit VM cache modes (writethrough → writeback for NFS)
**Other:**
- **Friday maintenance:** tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
- **PDU:** Rename outlets in powerman.conf (Friday onsite)
- **k8s deferred:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out
- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. HA NUT integration live (battery charge, status sensors). No ups.load over USB HID (firmware limitation, confirmed via apcupsd test). Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port).
- Investigate core-sw01 port errors (g9/g38/g26)
- k8s: join wnodes, label by workload class
- k8s deferred: ETL tooling, HPC scheduler, vcluster policy
## Infrastructure Summary
| Component | Details |
|-----------|---------|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) |
| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + new NVMe unconfigured) |
| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3), + 1 unidentified (192.168.0.8) |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| DNS production | tailscale-router (read-only source of truth) |