PFVCluster/aug11.md

Handoff: 2026-08-11 OAM Session

Point a new Crush session at this file: cat aug11.md (or read it). It contains everything needed to pick up where this session left off.

Quick Start (for the new session)

1. Follow normal session-start gate (AGENTS.md steps 1-4)
2. Set active ticket: echo '#430' > .crush/active-ticket
3. Read this file for full context
4. Check Redmine queue for latest status

Session Summary

Deployed the entire OAM monitoring stack on tsys-librenms (VM 104 on pfv-tsys1). All tools running as Docker containers. 4 of 5 tools are 100% operational.

OAM Stack Status (tsys-librenms)

Tool	Port	Container	Status	Ticket
Smokeping	8081	oam-smokeping	DONE	#375 (90%)
NetDisco	8082	oam-netdisco-{web,backend,db}	DONE	#337 (90%)
Oxidized	8083	oam-oxidized	DONE (4/5 switches)	#436 (Cisco pending)
UNPoller	8084	oam-unpoller	POLLING but needs Prometheus/Grafana	#430
Weathermap	LibreNMS	(plugin)	DONE	#375 (90%)
phpIPAM	Cloudron	ipam.knownelement.com	90% configured	#425

Access: VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm '<cmd>'

What's Done (don't re-do)

• Smokeping: 16 targets, FPing every 5 min, fully operational
• NetDisco: 4 Dell switches discovered, LLDP neighbors working (18 entries), web UI at :8082
• Oxidized: oxidized user created on 4 switches, configs backing up to KNEL/network-device-configs on Gitea, auto-push via SSH key
• UNPoller: authenticated to UniFi controller (pfv-bms:8443, user: unpoller), Prometheus metrics on :8084
• Weathermap: LibreNMS plugin enabled, generating maps via cron (network.conf)
• phpIPAM: PFV section with Server Room/EE Lab/Art Room sub-sections, 9 devices, 64 DHCP reservations imported, SNMP on 4 switches, scan agent configured
• Switch locations standardized: PFV/Server-Room/R{n} format on all 4 switches
• Switch DNS names fixed in LibreNMS (#424 closed)
• LibreNMS poller cron fix verified (#423 closed)
• Rules updated: remote.sh mandatory, DNS names only, command timeouts, questions-v(N) versioning

What's Left (in priority order)

1. UNPoller → Prometheus/Grafana wiring (#430)

• UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron)
• BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user
• Then build Grafana dashboard for UniFi
• UNPoller v3.4.1 exports Prometheus metrics at http://tsys-librenms.knel.net:8084/metrics

2. LibreNMS alert transports (#338)

• Configure Pushover as alert delivery in LibreNMS
• BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER)
• Pushover is the STANDARD alert transport for ALL OAM tools (user directive):
  • Uptime Kuma: already using Pushover
  • LibreNMS: #338 (this ticket)
  • Beszel: #428
  • Grafana: when wired via #430

3. SNMP trap strategy (#429)

• Decide: Dell OME owns server/iDRAC traps, LibreNMS owns network traps (recommended split)
• Awaiting Dell OME deployment (#339) before implementing
• Switches may need trap destination configured

4. TEMPer USB graphing (#341)

• Deploy on pfv-tsys1 (Server Room) + stlpc-artroom (Art Room)
• USB probe driver + RRD/LibreNMS integration
• stlpc-garage (EE Lab) may also have a probe — verify with user

5. Monitoring coverage matrix (#343)

• Governance doc mapping every tool to its domain (no overlap)
• Write to Discourse, update ticket
• Tool domains: Uptime Kuma=HTTP/service availability, Beszel=host metrics, LibreNMS=SNMP/network, Dell OME=Dell hardware, Smokeping=latency, NetDisco=L2 topology, Oxidized=config backup, phpIPAM=IP inventory, UNPoller=UniFi metrics

6. Uptime Kuma API + comprehensive monitors (#426/#435)

• Enable API write access
• Add monitors for all PFV/SITER + Cloudron services (HTTPS, DNS, DB, Docker, k8s)

7. Dell OpenManage Enterprise (#339)

• USER DEPLOYING MANUALLY — has KVM image downloaded, will do qm import + VM creation
• Agent role: post-deployment integration only (SNMP traps #429, coverage matrix #343)
• Fleet is 100% Dell (documented in Discourse #304)

Blocked Items (waiting on user or hardware)

• #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436
• #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track
• #430: Prometheus/Grafana creds needed (~/.creds/prometheus.env + grafana.env EMPTY)
• #338: Pushover API key needed (~/.creds/pushover.env NOT CREATED YET)
• #426: Uptime Kuma API access needed (read works, write not enabled)
• #425: phpIPAM scan agent — user may need to create from web UI and trigger first scan

User Action Items (things only the user can do)

1. Populate ~/.creds/pushover.env — PUSHOVER_TOKEN + PUSHOVER_USER (unblocks #338, #428)
2. Populate ~/.creds/prometheus.env + grafana.env — Cloudron API keys (unblocks #430)
3. Deploy Dell OME manually — VM creation + qm import + OOB config (#339)
4. Replace Cisco router fan — physical onsite work (#431, blocks RackRental track)
5. Create phpIPAM scan agent — from web UI, trigger first discovery scan (#425)
6. Enable Uptime Kuma API write access — from web UI (#426)
7. Run interactive netdisco-deploy — DONE (user already did this)

Important Technical Notes

snmpd is NOT fleet-wide (user directive)

snmpd will NOT be rolled out to Linux hosts. Beszel covers host metrics. SNMP (community: kn3lmgmt) is ONLY for network devices (switches, routers, PDUs). Do NOT add snmpd deployment to any ticket scope.

Cloudron Tailscale routing

Cloudron is on Tailscale. pfv-netinfra-01/02 are Tailscale subnet routers advertising 192.168.0.0/22. This means Cloudron-hosted services (phpIPAM, Prometheus, Grafana) CAN reach the LAN switches via Tailscale for SNMP discovery/scraping.

NetDisco required interactive deploy

The netdisco-deploy script requires a TTY (interactive yes/no prompts). The agent cannot run this through remote.sh. The user ran it manually. If NetDisco needs redeployment, the user must run this step.

Weathermap PHP fix

Weathermap plugin required register_argc_argv = On in /etc/php/8.4/fpm/php.ini on tsys-librenms. Already applied. If PHP gets updated, verify this setting persists.

RackRental split-network design (Q6 answer in questions-v1.md)

tsys6/7 have 4-6 ethernet ports. User wants:

• 2 NICs for storage bond (keep existing bond0)
• 1 NIC for mgmt (non-RackRental side)
• 1 NIC trunked to RackRental VLAN 2
• RackRental router prevents network loops

Location taxonomy

• PFV/Server-Room/R{n} for server room gear by rack
• PFV/EE-Lab for stlpc-garage
• PFV/Art-Room for stlpc-artroom
• Applied to all 4 switches via SNMP. Cisco pending fan fix (#431).
• phpIPAM sections match this taxonomy.

Ticket gate discussion

User wants to discuss relaxing the ticket gate for handoff/continuation sessions. For now, use a "dummy" ticket (#430 as placeholder) but don't bypass the rules. This is an open discussion item — see questions-v1.md for future Q&A.

Credential Files (in ~/.creds/)

File	Status
redmine.env	OK
discourse.env	OK
phpipam.env	OK (app_code auth working)
unifi-creds.env	OK (user: unpoller, polling working)
oxidized.env	OK (password stored)
switch-creds.env	Created, user to populate if needed
pushover.env	NOT CREATED — needs PUSHOVER_TOKEN + PUSHOVER_USER
prometheus.env	Created, EMPTY — user to populate
grafana.env	Created, EMPTY — user to populate

Rules Changed This Session

All committed and pushed to both PFVCluster and TSYSGroupAIOS (meta):

1. remote.sh mandatory: ALL SSH through chokepoint scripts, never direct ssh/scp
2. DNS names only: NEVER use IP literals in any command/script/config
3. Command timeouts: 30s reads, 120s standard, 300s deployments — every remote command wrapped in timeout
4. questions-v(N) versioning: increment filename each time answers land (v1→v2→v3)

Key Commits This Session

• 1f634a6 — questions versioning rule
• 2806eab — command timeout rule
• eea5495 — remote.sh + DNS-names-only rule
• 1590923 — Oxidized fully operational
• acf880a — Weathermap deployed
• 7216eb5 — OAM stack initial deployment
• 29b3989 — UNPoller image fix

Key Gitea Repos

• PFVCluster: ssh://git@git.knownelement.com:29418/KNEL/PFVCluster.git (main)
• TSYSGroupAIOS (meta): ssh://git@git.knownelement.com:29418/TSYSGroupCorporate/TSYSGroupAIOS.git (master)
• network-device-configs: ssh://git@git.knownelement.com:29418/KNEL/network-device-configs.git (master) — Oxidized backup target

SNMP Community

All Dell PowerConnect switches: kn3lmgmt (v2c)

Switch Inventory

Switch	Model	Location	IP	Oxidized User
pfv-r5-core-01	Dell PowerConnect 5448	PFV/Server-Room/R5	192.168.0.12	oxidized (priv 1)
pfv-r3-tor-mgmt-01	Dell PowerConnect 5324	PFV/Server-Room/R3	192.168.0.7	oxidized (priv 1)
pfv-r3-tor-stor-01	Dell PowerConnect 5324	PFV/Server-Room/R3	192.168.0.9	oxidized (priv 1)
pfv-r6-mgmt-01	Dell PowerConnect 5324	PFV/Server-Room/R6	192.168.0.8	oxidized (priv 1)
pfv-rrinfra-rtr	Cisco router	PFV/Server-Room/R2 (pending)	unknown	BLOCKED (#431 fan)

Tickets Created This Session

#	Subject	Status
424	LibreNMS switch DNS rename	CLOSED
425	phpIPAM config	90%
426	Uptime Kuma API	New
427	Credential management + IAC	New
428	Beszel alert delivery (Pushover)	New
429	SNMP trap strategy (LibreNMS vs Dell OME)	New
430	UNPoller → Prometheus/Grafana	New
431	Cisco router fan replacement	New
432	RackRental VLAN 2	New
433	Trunk ports tsys6/7 for RackRental	New
434	Cyclades PDU IP confirm	New
435	Uptime Kuma comprehensive monitors	New
436	Cisco router → Oxidized (after fan)	New
437	Switch location standardization	90%

Tickets Updated (not created) This Session

#	What we did
314	Parent umbrella for OAM alerts — updated children list
337	LibreNMS integration modules — 90% done (Oxidized/NetDisco/UNPoller deployed)
339	Dell OME — confirmed 100% Dell fleet, user deploying manually
375	Monitoring expansion — smokeping done, weathermap done, snmpd removed from scope
406	Beszel API — cross-referenced with OAM platform
425	phpIPAM — revised from "deploy" to "configure existing Cloudron instance"
370	Vault umbrella — clarified parent-child relationships

Tomorrow's Goal: IAC

User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408) unblocks everything: Ansible Vault → AWX playbooks → full IAC rebuild path for all OAM tools deployed this session.