diff --git a/aug11.md b/aug11.md index d15a48e..fedb97b 100644 --- a/aug11.md +++ b/aug11.md @@ -48,35 +48,102 @@ Access: `VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm ''` - UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron) - BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user - Then build Grafana dashboard for UniFi +- UNPoller v3.4.1 exports Prometheus metrics at http://tsys-librenms.knel.net:8084/metrics ### 2. LibreNMS alert transports (#338) - Configure Pushover as alert delivery in LibreNMS - BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER) -- Same Pushover config needed for Beszel (#428) and Uptime Kuma updates +- **Pushover is the STANDARD alert transport for ALL OAM tools** (user directive): + - Uptime Kuma: already using Pushover + - LibreNMS: #338 (this ticket) + - Beszel: #428 + - Grafana: when wired via #430 -### 3. TEMPer USB graphing (#341) -- Deploy on pfv-tsys1 + stlpc-artroom +### 3. SNMP trap strategy (#429) +- Decide: Dell OME owns server/iDRAC traps, LibreNMS owns network traps (recommended split) +- Awaiting Dell OME deployment (#339) before implementing +- Switches may need trap destination configured + +### 4. TEMPer USB graphing (#341) +- Deploy on pfv-tsys1 (Server Room) + stlpc-artroom (Art Room) - USB probe driver + RRD/LibreNMS integration +- stlpc-garage (EE Lab) may also have a probe — verify with user -### 4. Monitoring coverage matrix (#343) +### 5. Monitoring coverage matrix (#343) - Governance doc mapping every tool to its domain (no overlap) - Write to Discourse, update ticket +- Tool domains: Uptime Kuma=HTTP/service availability, Beszel=host metrics, + LibreNMS=SNMP/network, Dell OME=Dell hardware, Smokeping=latency, + NetDisco=L2 topology, Oxidized=config backup, phpIPAM=IP inventory, + UNPoller=UniFi metrics -### 5. Uptime Kuma API + comprehensive monitors (#426/#435) +### 6. Uptime Kuma API + comprehensive monitors (#426/#435) - Enable API write access -- Add monitors for all PFV/SITER + Cloudron services +- Add monitors for all PFV/SITER + Cloudron services (HTTPS, DNS, DB, Docker, k8s) -### 6. Dell OpenManage Enterprise (#339) -- User deploying manually (VM creation + disk import) -- Agent role: post-deployment integration only +### 7. Dell OpenManage Enterprise (#339) +- USER DEPLOYING MANUALLY — has KVM image downloaded, will do qm import + VM creation +- Agent role: post-deployment integration only (SNMP traps #429, coverage matrix #343) +- Fleet is 100% Dell (documented in Discourse #304) ## Blocked Items (waiting on user or hardware) - #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436 - #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track -- #430: Prometheus/Grafana creds needed -- #338: Pushover API key needed -- #426: Uptime Kuma API access needed +- #430: Prometheus/Grafana creds needed (~/.creds/prometheus.env + grafana.env EMPTY) +- #338: Pushover API key needed (~/.creds/pushover.env NOT CREATED YET) +- #426: Uptime Kuma API access needed (read works, write not enabled) +- #425: phpIPAM scan agent — user may need to create from web UI and trigger first scan + +## User Action Items (things only the user can do) + +1. **Populate ~/.creds/pushover.env** — PUSHOVER_TOKEN + PUSHOVER_USER (unblocks #338, #428) +2. **Populate ~/.creds/prometheus.env + grafana.env** — Cloudron API keys (unblocks #430) +3. **Deploy Dell OME manually** — VM creation + qm import + OOB config (#339) +4. **Replace Cisco router fan** — physical onsite work (#431, blocks RackRental track) +5. **Create phpIPAM scan agent** — from web UI, trigger first discovery scan (#425) +6. **Enable Uptime Kuma API write access** — from web UI (#426) +7. **Run interactive netdisco-deploy** — DONE (user already did this) + +## Important Technical Notes + +### snmpd is NOT fleet-wide (user directive) +snmpd will NOT be rolled out to Linux hosts. Beszel covers host metrics. +SNMP (community: kn3lmgmt) is ONLY for network devices (switches, routers, PDUs). +Do NOT add snmpd deployment to any ticket scope. + +### Cloudron Tailscale routing +Cloudron is on Tailscale. pfv-netinfra-01/02 are Tailscale subnet routers advertising +192.168.0.0/22. This means Cloudron-hosted services (phpIPAM, Prometheus, Grafana) +CAN reach the LAN switches via Tailscale for SNMP discovery/scraping. + +### NetDisco required interactive deploy +The `netdisco-deploy` script requires a TTY (interactive yes/no prompts). +The agent cannot run this through remote.sh. The user ran it manually. +If NetDisco needs redeployment, the user must run this step. + +### Weathermap PHP fix +Weathermap plugin required `register_argc_argv = On` in /etc/php/8.4/fpm/php.ini +on tsys-librenms. Already applied. If PHP gets updated, verify this setting persists. + +### RackRental split-network design (Q6 answer in questions-v1.md) +tsys6/7 have 4-6 ethernet ports. User wants: +- 2 NICs for storage bond (keep existing bond0) +- 1 NIC for mgmt (non-RackRental side) +- 1 NIC trunked to RackRental VLAN 2 +- RackRental router prevents network loops + +### Location taxonomy +- PFV/Server-Room/R{n} for server room gear by rack +- PFV/EE-Lab for stlpc-garage +- PFV/Art-Room for stlpc-artroom +- Applied to all 4 switches via SNMP. Cisco pending fan fix (#431). +- phpIPAM sections match this taxonomy. + +### Ticket gate discussion +User wants to discuss relaxing the ticket gate for handoff/continuation sessions. +For now, use a "dummy" ticket (#430 as placeholder) but don't bypass the rules. +This is an open discussion item — see questions-v1.md for future Q&A. ## Credential Files (in ~/.creds/) @@ -150,6 +217,18 @@ All Dell PowerConnect switches: `kn3lmgmt` (v2c) | 436 | Cisco router → Oxidized (after fan) | New | | 437 | Switch location standardization | 90% | +## Tickets Updated (not created) This Session + +| # | What we did | +|---|-------------| +| 314 | Parent umbrella for OAM alerts — updated children list | +| 337 | LibreNMS integration modules — 90% done (Oxidized/NetDisco/UNPoller deployed) | +| 339 | Dell OME — confirmed 100% Dell fleet, user deploying manually | +| 375 | Monitoring expansion — smokeping done, weathermap done, snmpd removed from scope | +| 406 | Beszel API — cross-referenced with OAM platform | +| 425 | phpIPAM — revised from "deploy" to "configure existing Cloudron instance" | +| 370 | Vault umbrella — clarified parent-child relationships | + ## Tomorrow's Goal: IAC User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408)