Commit Graph
2 Commits
Author SHA1 Message Date
mrcharles 24b08aa592 docs(audit): document observability stack + Saturday OAM day plan
Operator-defined monitoring architecture (no tool duplication):
- Beszel (Cloudron VPS): CPU/RAM/disk metrics on all Proxmox hosts via Tailscale
- Uptime Kuma (Cloudron VPS): up/down alerting via Pushover
- LibreNMS (VMID 104, on-prem): SNMP polling for switches (non-Tailsale gear)
- Home Assistant (VMID 100, on-prem): power/temp instrumentation + alerting

Saturday 2026-08-01 is designated OAM Day: goal is 100% instrumentation
coverage across all four tools + centralized syslog.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:02:11 -05:00
mrcharles 26de312c42 docs(audit): full ground-truth audit — network, storage, compute, switches
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.

Key findings documented in AUDIT-2026-07-30.md:

Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
  host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
  Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).

Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
  installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.

Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).

Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:01:43 -05:00