KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): document observability stack + Saturday OAM day plan

Browse Source 
Operator-defined monitoring architecture (no tool duplication):
- Beszel (Cloudron VPS): CPU/RAM/disk metrics on all Proxmox hosts via Tailscale
- Uptime Kuma (Cloudron VPS): up/down alerting via Pushover
- LibreNMS (VMID 104, on-prem): SNMP polling for switches (non-Tailsale gear)
- Home Assistant (VMID 100, on-prem): power/temp instrumentation + alerting

Saturday 2026-08-01 is designated OAM Day: goal is 100% instrumentation
coverage across all four tools + centralized syslog.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-30 23:02:11 -05:00
parent 26de312c42
commit 24b08aa592
2 changed files with 49 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-07-30.md
+45
View File
@@ -383,6 +383,51 @@ conmand systemd service from starting.
---
## 9. Observability stack (operator direction)
The operator has defined a clear monitoring architecture with three tools,
each serving a distinct role. Do NOT duplicate functionality across tools.
### 9.1 Current architecture
| Tool | Where it runs | Scope | Role |
|------|---------------|-------|------|
| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts |
| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover |
| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent |
| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) |
### 9.2 Division of responsibility (no duplication)
| Metric class | Tool | Notes |
|-------------|------|-------|
| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) |
| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications |
| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) |
| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations |
### 9.3 Saturday "OAM Day" plan (2026-08-01)
The operator has designated Saturday as a dedicated operations/admin/management
day with the goal of achieving **100% instrumentation coverage**:
- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover)
- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog
relay, or Loki/Grafana stack)
- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale
devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12)
- [ ] Beszel agent coverage on all 7 Proxmox hosts
- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU
via powerman, any temp sensors)
- [ ] Verify alerting paths end-to-end (test each alert fires correctly)
> Much of the groundwork is already done: NUT is live on tsys1, powerman is
> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live
> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring
> continuous monitoring + alerting on top of these.
---
## 8. Recommendations (for operator review — NO changes applied)
### 8.1 Switch changes (require maintenance window + operator approval)