KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
44e9687c75be767de8c37b922d0ffca4ec203698
PFVCluster/aug11.md
T
mrcharles 44e9687c75 docs: add aug11.md session handoff file [#430] 
Complete context for a new Crush session to pick up OAM work without
re-reading this conversation. Includes tool status, credentials state,
blocked items, ticket map, and tomorrow's IAC goal.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-11 20:17:58 -05:00

6.7 KiB
Raw Blame History

Handoff: 2026-08-11 OAM Session

Point a new Crush session at this file: cat aug11.md (or read it). It contains everything needed to pick up where this session left off.

Quick Start (for the new session)

  1. Follow normal session-start gate (AGENTS.md steps 1-4)
  2. Set active ticket: echo '#430' > .crush/active-ticket
  3. Read this file for full context
  4. Check Redmine queue for latest status

Session Summary

Deployed the entire OAM monitoring stack on tsys-librenms (VM 104 on pfv-tsys1). All tools running as Docker containers. 4 of 5 tools are 100% operational.

OAM Stack Status (tsys-librenms)

Tool Port Container Status Ticket
Smokeping 8081 oam-smokeping DONE #375 (90%)
NetDisco 8082 oam-netdisco-{web,backend,db} DONE #337 (90%)
Oxidized 8083 oam-oxidized DONE (4/5 switches) #436 (Cisco pending)
UNPoller 8084 oam-unpoller POLLING but needs Prometheus/Grafana #430
Weathermap LibreNMS (plugin) DONE #375 (90%)
phpIPAM Cloudron ipam.knownelement.com 90% configured #425

Access: VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm '<cmd>'

What's Done (don't re-do)

  • Smokeping: 16 targets, FPing every 5 min, fully operational
  • NetDisco: 4 Dell switches discovered, LLDP neighbors working (18 entries), web UI at :8082
  • Oxidized: oxidized user created on 4 switches, configs backing up to KNEL/network-device-configs on Gitea, auto-push via SSH key
  • UNPoller: authenticated to UniFi controller (pfv-bms:8443, user: unpoller), Prometheus metrics on :8084
  • Weathermap: LibreNMS plugin enabled, generating maps via cron (network.conf)
  • phpIPAM: PFV section with Server Room/EE Lab/Art Room sub-sections, 9 devices, 64 DHCP reservations imported, SNMP on 4 switches, scan agent configured
  • Switch locations standardized: PFV/Server-Room/R{n} format on all 4 switches
  • Switch DNS names fixed in LibreNMS (#424 closed)
  • LibreNMS poller cron fix verified (#423 closed)
  • Rules updated: remote.sh mandatory, DNS names only, command timeouts, questions-v(N) versioning

What's Left (in priority order)

1. UNPoller → Prometheus/Grafana wiring (#430)

  • UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron)
  • BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user
  • Then build Grafana dashboard for UniFi

2. LibreNMS alert transports (#338)

  • Configure Pushover as alert delivery in LibreNMS
  • BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER)
  • Same Pushover config needed for Beszel (#428) and Uptime Kuma updates

3. TEMPer USB graphing (#341)

  • Deploy on pfv-tsys1 + stlpc-artroom
  • USB probe driver + RRD/LibreNMS integration

4. Monitoring coverage matrix (#343)

  • Governance doc mapping every tool to its domain (no overlap)
  • Write to Discourse, update ticket

5. Uptime Kuma API + comprehensive monitors (#426/#435)

  • Enable API write access
  • Add monitors for all PFV/SITER + Cloudron services

6. Dell OpenManage Enterprise (#339)

  • User deploying manually (VM creation + disk import)
  • Agent role: post-deployment integration only

Blocked Items (waiting on user or hardware)

  • #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436
  • #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track
  • #430: Prometheus/Grafana creds needed
  • #338: Pushover API key needed
  • #426: Uptime Kuma API access needed

Credential Files (in ~/.creds/)

File Status
redmine.env OK
discourse.env OK
phpipam.env OK (app_code auth working)
unifi-creds.env OK (user: unpoller, polling working)
oxidized.env OK (password stored)
switch-creds.env Created, user to populate if needed
pushover.env NOT CREATED — needs PUSHOVER_TOKEN + PUSHOVER_USER
prometheus.env Created, EMPTY — user to populate
grafana.env Created, EMPTY — user to populate

Rules Changed This Session

All committed and pushed to both PFVCluster and TSYSGroupAIOS (meta):

  1. remote.sh mandatory: ALL SSH through chokepoint scripts, never direct ssh/scp
  2. DNS names only: NEVER use IP literals in any command/script/config
  3. Command timeouts: 30s reads, 120s standard, 300s deployments — every remote command wrapped in timeout
  4. questions-v(N) versioning: increment filename each time answers land (v1→v2→v3)

Key Commits This Session

  • 1f634a6 — questions versioning rule
  • 2806eab — command timeout rule
  • eea5495 — remote.sh + DNS-names-only rule
  • 1590923 — Oxidized fully operational
  • acf880a — Weathermap deployed
  • 7216eb5 — OAM stack initial deployment
  • 29b3989 — UNPoller image fix

Key Gitea Repos

  • PFVCluster: ssh://git@git.knownelement.com:29418/KNEL/PFVCluster.git (main)
  • TSYSGroupAIOS (meta): ssh://git@git.knownelement.com:29418/TSYSGroupCorporate/TSYSGroupAIOS.git (master)
  • network-device-configs: ssh://git@git.knownelement.com:29418/KNEL/network-device-configs.git (master) — Oxidized backup target

SNMP Community

All Dell PowerConnect switches: kn3lmgmt (v2c)

Switch Inventory

Switch Model Location IP Oxidized User
pfv-r5-core-01 Dell PowerConnect 5448 PFV/Server-Room/R5 192.168.0.12 oxidized (priv 1)
pfv-r3-tor-mgmt-01 Dell PowerConnect 5324 PFV/Server-Room/R3 192.168.0.7 oxidized (priv 1)
pfv-r3-tor-stor-01 Dell PowerConnect 5324 PFV/Server-Room/R3 192.168.0.9 oxidized (priv 1)
pfv-r6-mgmt-01 Dell PowerConnect 5324 PFV/Server-Room/R6 192.168.0.8 oxidized (priv 1)
pfv-rrinfra-rtr Cisco router PFV/Server-Room/R2 (pending) unknown BLOCKED (#431 fan)

Tickets Created This Session

# Subject Status
424 LibreNMS switch DNS rename CLOSED
425 phpIPAM config 90%
426 Uptime Kuma API New
427 Credential management + IAC New
428 Beszel alert delivery (Pushover) New
429 SNMP trap strategy (LibreNMS vs Dell OME) New
430 UNPoller → Prometheus/Grafana New
431 Cisco router fan replacement New
432 RackRental VLAN 2 New
433 Trunk ports tsys6/7 for RackRental New
434 Cyclades PDU IP confirm New
435 Uptime Kuma comprehensive monitors New
436 Cisco router → Oxidized (after fan) New
437 Switch location standardization 90%

Tomorrow's Goal: IAC

User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408) unblocks everything: Ansible Vault → AWX playbooks → full IAC rebuild path for all OAM tools deployed this session.

Reference in New Issue View Git Blame Copy Permalink