From 24b08aa592d5e8ceb587086324d58ca0b495e9d3 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Thu, 30 Jul 2026 23:02:11 -0500 Subject: [PATCH] docs(audit): document observability stack + Saturday OAM day plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator-defined monitoring architecture (no tool duplication): - Beszel (Cloudron VPS): CPU/RAM/disk metrics on all Proxmox hosts via Tailscale - Uptime Kuma (Cloudron VPS): up/down alerting via Pushover - LibreNMS (VMID 104, on-prem): SNMP polling for switches (non-Tailsale gear) - Home Assistant (VMID 100, on-prem): power/temp instrumentation + alerting Saturday 2026-08-01 is designated OAM Day: goal is 100% instrumentation coverage across all four tools + centralized syslog. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- STATUS.md | 4 +++ proxmox/docs/AUDIT-2026-07-30.md | 45 ++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/STATUS.md b/STATUS.md index add566a..e6fe8f8 100644 --- a/STATUS.md +++ b/STATUS.md @@ -171,6 +171,10 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - Audit VM cache modes (writethrough → writeback for NFS) **Other:** +- **Saturday OAM Day (2026-08-01):** 100% instrumentation coverage — Beszel + (metrics) on all hosts, Uptime Kuma (up/down) → Pushover alerts, LibreNMS + (SNMP) for switches, Home Assistant (power/temp) for physical infra, + centralized syslog (tool TBD) - Investigate core-sw01 port errors (g9/g38/g26) - k8s: join wnodes, label by workload class - k8s deferred: ETL tooling, HPC scheduler, vcluster policy diff --git a/proxmox/docs/AUDIT-2026-07-30.md b/proxmox/docs/AUDIT-2026-07-30.md index b033070..187440f 100644 --- a/proxmox/docs/AUDIT-2026-07-30.md +++ b/proxmox/docs/AUDIT-2026-07-30.md @@ -383,6 +383,51 @@ conmand systemd service from starting. --- +## 9. Observability stack (operator direction) + +The operator has defined a clear monitoring architecture with three tools, +each serving a distinct role. Do NOT duplicate functionality across tools. + +### 9.1 Current architecture + +| Tool | Where it runs | Scope | Role | +|------|---------------|-------|------| +| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts | +| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover | +| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent | +| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) | + +### 9.2 Division of responsibility (no duplication) + +| Metric class | Tool | Notes | +|-------------|------|-------| +| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) | +| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications | +| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) | +| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations | + +### 9.3 Saturday "OAM Day" plan (2026-08-01) + +The operator has designated Saturday as a dedicated operations/admin/management +day with the goal of achieving **100% instrumentation coverage**: + +- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover) +- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog + relay, or Loki/Grafana stack) +- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale + devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12) +- [ ] Beszel agent coverage on all 7 Proxmox hosts +- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU + via powerman, any temp sensors) +- [ ] Verify alerting paths end-to-end (test each alert fires correctly) + +> Much of the groundwork is already done: NUT is live on tsys1, powerman is +> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live +> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring +> continuous monitoring + alerting on top of these. + +--- + ## 8. Recommendations (for operator review — NO changes applied) ### 8.1 Switch changes (require maintenance window + operator approval)