From 44e9687c75be767de8c37b922d0ffca4ec203698 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Tue, 11 Aug 2026 20:17:58 -0500 Subject: [PATCH] docs: add aug11.md session handoff file [#430] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete context for a new Crush session to pick up OAM work without re-reading this conversation. Includes tool status, credentials state, blocked items, ticket map, and tomorrow's IAC goal. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- aug11.md | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 aug11.md diff --git a/aug11.md b/aug11.md new file mode 100644 index 0000000..d15a48e --- /dev/null +++ b/aug11.md @@ -0,0 +1,157 @@ + +# Handoff: 2026-08-11 OAM Session + +Point a new Crush session at this file: `cat aug11.md` (or read it). It contains +everything needed to pick up where this session left off. + +## Quick Start (for the new session) + +1. Follow normal session-start gate (AGENTS.md steps 1-4) +2. Set active ticket: `echo '#430' > .crush/active-ticket` +3. Read this file for full context +4. Check Redmine queue for latest status + +## Session Summary + +Deployed the entire OAM monitoring stack on tsys-librenms (VM 104 on pfv-tsys1). +All tools running as Docker containers. 4 of 5 tools are 100% operational. + +## OAM Stack Status (tsys-librenms) + +| Tool | Port | Container | Status | Ticket | +|------|------|-----------|--------|--------| +| Smokeping | 8081 | oam-smokeping | DONE | #375 (90%) | +| NetDisco | 8082 | oam-netdisco-{web,backend,db} | DONE | #337 (90%) | +| Oxidized | 8083 | oam-oxidized | DONE (4/5 switches) | #436 (Cisco pending) | +| UNPoller | 8084 | oam-unpoller | POLLING but needs Prometheus/Grafana | #430 | +| Weathermap | LibreNMS | (plugin) | DONE | #375 (90%) | +| phpIPAM | Cloudron | ipam.knownelement.com | 90% configured | #425 | + +Access: `VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm ''` + +## What's Done (don't re-do) + +- Smokeping: 16 targets, FPing every 5 min, fully operational +- NetDisco: 4 Dell switches discovered, LLDP neighbors working (18 entries), web UI at :8082 +- Oxidized: oxidized user created on 4 switches, configs backing up to KNEL/network-device-configs on Gitea, auto-push via SSH key +- UNPoller: authenticated to UniFi controller (pfv-bms:8443, user: unpoller), Prometheus metrics on :8084 +- Weathermap: LibreNMS plugin enabled, generating maps via cron (network.conf) +- phpIPAM: PFV section with Server Room/EE Lab/Art Room sub-sections, 9 devices, 64 DHCP reservations imported, SNMP on 4 switches, scan agent configured +- Switch locations standardized: PFV/Server-Room/R{n} format on all 4 switches +- Switch DNS names fixed in LibreNMS (#424 closed) +- LibreNMS poller cron fix verified (#423 closed) +- Rules updated: remote.sh mandatory, DNS names only, command timeouts, questions-v(N) versioning + +## What's Left (in priority order) + +### 1. UNPoller → Prometheus/Grafana wiring (#430) +- UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron) +- BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user +- Then build Grafana dashboard for UniFi + +### 2. LibreNMS alert transports (#338) +- Configure Pushover as alert delivery in LibreNMS +- BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER) +- Same Pushover config needed for Beszel (#428) and Uptime Kuma updates + +### 3. TEMPer USB graphing (#341) +- Deploy on pfv-tsys1 + stlpc-artroom +- USB probe driver + RRD/LibreNMS integration + +### 4. Monitoring coverage matrix (#343) +- Governance doc mapping every tool to its domain (no overlap) +- Write to Discourse, update ticket + +### 5. Uptime Kuma API + comprehensive monitors (#426/#435) +- Enable API write access +- Add monitors for all PFV/SITER + Cloudron services + +### 6. Dell OpenManage Enterprise (#339) +- User deploying manually (VM creation + disk import) +- Agent role: post-deployment integration only + +## Blocked Items (waiting on user or hardware) + +- #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436 +- #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track +- #430: Prometheus/Grafana creds needed +- #338: Pushover API key needed +- #426: Uptime Kuma API access needed + +## Credential Files (in ~/.creds/) + +| File | Status | +|------|--------| +| redmine.env | OK | +| discourse.env | OK | +| phpipam.env | OK (app_code auth working) | +| unifi-creds.env | OK (user: unpoller, polling working) | +| oxidized.env | OK (password stored) | +| switch-creds.env | Created, user to populate if needed | +| pushover.env | NOT CREATED — needs PUSHOVER_TOKEN + PUSHOVER_USER | +| prometheus.env | Created, EMPTY — user to populate | +| grafana.env | Created, EMPTY — user to populate | + +## Rules Changed This Session + +All committed and pushed to both PFVCluster and TSYSGroupAIOS (meta): + +1. **remote.sh mandatory**: ALL SSH through chokepoint scripts, never direct ssh/scp +2. **DNS names only**: NEVER use IP literals in any command/script/config +3. **Command timeouts**: 30s reads, 120s standard, 300s deployments — every remote command wrapped in `timeout` +4. **questions-v(N) versioning**: increment filename each time answers land (v1→v2→v3) + +## Key Commits This Session + +- `1f634a6` — questions versioning rule +- `2806eab` — command timeout rule +- `eea5495` — remote.sh + DNS-names-only rule +- `1590923` — Oxidized fully operational +- `acf880a` — Weathermap deployed +- `7216eb5` — OAM stack initial deployment +- `29b3989` — UNPoller image fix + +## Key Gitea Repos + +- PFVCluster: `ssh://git@git.knownelement.com:29418/KNEL/PFVCluster.git` (main) +- TSYSGroupAIOS (meta): `ssh://git@git.knownelement.com:29418/TSYSGroupCorporate/TSYSGroupAIOS.git` (master) +- network-device-configs: `ssh://git@git.knownelement.com:29418/KNEL/network-device-configs.git` (master) — Oxidized backup target + +## SNMP Community + +All Dell PowerConnect switches: `kn3lmgmt` (v2c) + +## Switch Inventory + +| Switch | Model | Location | IP | Oxidized User | +|--------|-------|----------|----|---------------| +| pfv-r5-core-01 | Dell PowerConnect 5448 | PFV/Server-Room/R5 | 192.168.0.12 | oxidized (priv 1) | +| pfv-r3-tor-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.7 | oxidized (priv 1) | +| pfv-r3-tor-stor-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.9 | oxidized (priv 1) | +| pfv-r6-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R6 | 192.168.0.8 | oxidized (priv 1) | +| pfv-rrinfra-rtr | Cisco router | PFV/Server-Room/R2 (pending) | unknown | BLOCKED (#431 fan) | + +## Tickets Created This Session + +| # | Subject | Status | +|---|---------|--------| +| 424 | LibreNMS switch DNS rename | CLOSED | +| 425 | phpIPAM config | 90% | +| 426 | Uptime Kuma API | New | +| 427 | Credential management + IAC | New | +| 428 | Beszel alert delivery (Pushover) | New | +| 429 | SNMP trap strategy (LibreNMS vs Dell OME) | New | +| 430 | UNPoller → Prometheus/Grafana | New | +| 431 | Cisco router fan replacement | New | +| 432 | RackRental VLAN 2 | New | +| 433 | Trunk ports tsys6/7 for RackRental | New | +| 434 | Cyclades PDU IP confirm | New | +| 435 | Uptime Kuma comprehensive monitors | New | +| 436 | Cisco router → Oxidized (after fan) | New | +| 437 | Switch location standardization | 90% | + +## Tomorrow's Goal: IAC + +User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408) +unblocks everything: Ansible Vault → AWX playbooks → full IAC rebuild path for +all OAM tools deployed this session.