mrcharles
26de312c42
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
PFVCluster
Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
→ Current Status · → Documentation Index · → Agent Guidelines
Directory Structure
| Directory | Description |
|---|---|
provisioning/ |
Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) |
tests/ |
Test suite + VM validation harness |
dns-cluster-setup/ |
Technitium DNS cluster replication scripts |
k8s/ |
k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs |
powerman/ |
Cyclades PM10i PDU management via powerman on pfv-tsys1 |
console/ |
Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
ups/ |
UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale |
perf/ |
Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
proxmox/ |
Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
netinfra/ |
pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |
switches/ |
Switch configuration captures |
awx/ |
Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) |
docs/ |
Server-build docs, documentation index (docmap), and archive |
vendor/ |
Vendored KNELShellFramework |
Quick Start
Provision a new server
sudo bash provisioning/SetupNewSystem.sh
Validate on the sandbox VM
VM_ID=6000 ./tests/vm-validation.sh all
Deploy DNS cluster
cd dns-cluster-setup/ && ./setup.sh all
Deploy perf tunings
cd perf/ && ./deploy-tuning.sh
Architecture
- Proxmox hosts: 7 standalone PVE installs managed via PDM
- DNS: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
- NTP: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
- Production: Cloudron VPS in Reston VA (this cluster is R&D only)
- Backups: Proxmox Backup Server (PBS)