Fleet-wide changes (live, documented for audit/AWX):
Persistent tuning via systemd oneshot (all 7 hosts):
- New pfv-nfs-tuning.service runs AFTER tuned.service to override
16MB TCP buffer caps with 128MB for high-BDP NFS
- New 99-pfv-nfs.conf sysctl fragment (128MB rmem/wmem, 250K backlog)
- Deployed to: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
- Survives reboot — no more tuned clobbering our sysctl values
tsys5 fixes:
- VM 509 stopped, NFS D2/D3/D5 remounted with nconnect=4, VM 509
restarted (nconnect now live on all tsys5 NFS client mounts)
- SK hynix SSD (sdb1) persisted in fstab at /mnt/ssd2 (UUID-based)
- ssd2 added to Proxmox storage.cfg as dir storage
- Stale S3 export removed from /etc/exports (disk not mounted)
- Broken S4 dir entry removed from storage.cfg
- Duplicate noatime,noatime in fstab fixed
- version=4.2 bug in storage.cfg options fixed (invalid mount option)
tsys4 fixes:
- SSH keys re-standardized to 2 approved (upgrade regenerated host key)
- Tuning persistence working (128MB buffers confirmed live + persistent)
apply-tunings.sh fix:
- Removed version=4.2 from NFS options string (not a valid mount.nfs
option — causes mount failures; Proxmox auto-negotiates NFSv4.2)
Config files saved to provisioning/ConfigFiles/ for AWX/Ansible consumption:
- 99-pfv-nfs.conf
- pfv-nfs-tuning.service
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Console fix: generate-config.sh wrote SYMLINK+="console/$name" (singular)
but ser2net.yaml opens /dev/consoles/$name (plural). They never matched,
so after every reboot the console ports failed until setup.sh's manual
fallback re-created the symlinks. Fixed the udev rule to use "consoles/"
to match ser2net and the README.
New portable read-only audit tools (AGPLv3-friendly, config-driven):
- perf/scripts/probe-storage.sh: disk/mount/export/SMART/storage.cfg probe
- perf/scripts/probe-network.sh: NIC/bond/LLDP/NFS/nconnect probe
- perf/scripts/conman-console.py: PTY-based conman console driver (replaces
the old sw-capture.py that conflicted with ser2net)
- perf/scripts/snmp-switch-audit.py: SNMP-based switch inventory (interfaces,
LLDP, LAG, VLANs) via pysnmp or net-snmp
Removed stale pre-conman switch tooling (sw-capture-remote.sh, sw-capture.py,
sw-probe.sh, sw-conman-probe.sh) and old .cmds files. Added fresh .cmds
files for the two cross-rack trunk endpoint switches.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-tsys8 is permanently retired. Strike its only doc reference and
correct the stale comment in perf/deploy-check.sh that wrongly listed
pfv-tsys9 as "off the air" (tsys9 is online and in the host array;
tsys8 is the retired host).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.
Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
migrations were the #1 recommendation from the previous audit and
have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)
Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles