Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Kubernetes Control Plane (k3s HA — LIVE)
@@ -131,50 +137,51 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| Issue | Impact | Status |
|-------|--------|--------|
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 73%/37%/62% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §11 |
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. HA NUT integration live (battery charge, status sensors). No ups.load over USB HID (firmware limitation, confirmed via apcupsd test). Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port).
| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |
---
## 6. Console infrastructure status (fixed during this audit)
**Bug found + fixed:**`console/generate-config.sh` wrote udev rules with
`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
console ports failed until `setup.sh`'s manual fallback re-created symlinks.
**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
consoles now working (verified via conman-console.py).
**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
conmand systemd service from starting.
---
## 7. Findings summary (prioritized)
### Critical (cross-rack performance)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |
### High (storage configuration)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
! Read-only show commands for LAG/port-channel/VLAN audit
! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down)
terminal datadump
show version
show interfaces status
show interfaces port-channel
show port-channel summary
show lacp
show vlan
show running-config interface port-channel 1
show running-config interface port-channel 3
enable
show running-config
show interfaces configuration
show interfaces description
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
