<!-- Discourse: https://community.turnsys.com/t/298 -->
# Handoff: 2026-08-11 OAM Session

Point a new Crush session at this file: `cat aug11.md` (or read it). It contains
everything needed to pick up where this session left off.

## Quick Start (for the new session)

1. Follow normal session-start gate (AGENTS.md steps 1-4)
2. Set active ticket: `echo '#430' > .crush/active-ticket`
3. Read this file for full context
4. Check Redmine queue for latest status

## Session Summary

Deployed the entire OAM monitoring stack on tsys-librenms (VM 104 on pfv-tsys1).
All tools running as Docker containers. 4 of 5 tools are 100% operational.

## OAM Stack Status (tsys-librenms)

| Tool | Port | Container | Status | Ticket |
|------|------|-----------|--------|--------|
| Smokeping | 8081 | oam-smokeping | DONE | #375 (90%) |
| NetDisco | 8082 | oam-netdisco-{web,backend,db} | DONE | #337 (90%) |
| Oxidized | 8083 | oam-oxidized | DONE (4/5 switches) | #436 (Cisco pending) |
| UNPoller | 8084 | oam-unpoller | POLLING but needs Prometheus/Grafana | #430 |
| Weathermap | LibreNMS | (plugin) | DONE | #375 (90%) |
| phpIPAM | Cloudron | ipam.knownelement.com | 90% configured | #425 |

Access: `VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm '<cmd>'`

## What's Done (don't re-do)

- Smokeping: 16 targets, FPing every 5 min, fully operational
- NetDisco: 4 Dell switches discovered, LLDP neighbors working (18 entries), web UI at :8082
- Oxidized: oxidized user created on 4 switches, configs backing up to KNEL/network-device-configs on Gitea, auto-push via SSH key
- UNPoller: authenticated to UniFi controller (pfv-bms:8443, user: unpoller), Prometheus metrics on :8084
- Weathermap: LibreNMS plugin enabled, generating maps via cron (network.conf)
- phpIPAM: PFV section with Server Room/EE Lab/Art Room sub-sections, 9 devices, 64 DHCP reservations imported, SNMP on 4 switches, scan agent configured
- Switch locations standardized: PFV/Server-Room/R{n} format on all 4 switches
- Switch DNS names fixed in LibreNMS (#424 closed)
- LibreNMS poller cron fix verified (#423 closed)
- Rules updated: remote.sh mandatory, DNS names only, command timeouts, questions-v(N) versioning

## What's Left (in priority order)

### 1. UNPoller → Prometheus/Grafana wiring (#430)
- UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron)
- BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user
- Then build Grafana dashboard for UniFi

### 2. LibreNMS alert transports (#338)
- Configure Pushover as alert delivery in LibreNMS
- BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER)
- Same Pushover config needed for Beszel (#428) and Uptime Kuma updates

### 3. TEMPer USB graphing (#341)
- Deploy on pfv-tsys1 + stlpc-artroom
- USB probe driver + RRD/LibreNMS integration

### 4. Monitoring coverage matrix (#343)
- Governance doc mapping every tool to its domain (no overlap)
- Write to Discourse, update ticket

### 5. Uptime Kuma API + comprehensive monitors (#426/#435)
- Enable API write access
- Add monitors for all PFV/SITER + Cloudron services

### 6. Dell OpenManage Enterprise (#339)
- User deploying manually (VM creation + disk import)
- Agent role: post-deployment integration only

## Blocked Items (waiting on user or hardware)

- #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436
- #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track
- #430: Prometheus/Grafana creds needed
- #338: Pushover API key needed
- #426: Uptime Kuma API access needed

## Credential Files (in ~/.creds/)

| File | Status |
|------|--------|
| redmine.env | OK |
| discourse.env | OK |
| phpipam.env | OK (app_code auth working) |
| unifi-creds.env | OK (user: unpoller, polling working) |
| oxidized.env | OK (password stored) |
| switch-creds.env | Created, user to populate if needed |
| pushover.env | NOT CREATED — needs PUSHOVER_TOKEN + PUSHOVER_USER |
| prometheus.env | Created, EMPTY — user to populate |
| grafana.env | Created, EMPTY — user to populate |

## Rules Changed This Session

All committed and pushed to both PFVCluster and TSYSGroupAIOS (meta):

1. **remote.sh mandatory**: ALL SSH through chokepoint scripts, never direct ssh/scp
2. **DNS names only**: NEVER use IP literals in any command/script/config
3. **Command timeouts**: 30s reads, 120s standard, 300s deployments — every remote command wrapped in `timeout`
4. **questions-v(N) versioning**: increment filename each time answers land (v1→v2→v3)

## Key Commits This Session

- `1f634a6` — questions versioning rule
- `2806eab` — command timeout rule
- `eea5495` — remote.sh + DNS-names-only rule
- `1590923` — Oxidized fully operational
- `acf880a` — Weathermap deployed
- `7216eb5` — OAM stack initial deployment
- `29b3989` — UNPoller image fix

## Key Gitea Repos

- PFVCluster: `ssh://git@git.knownelement.com:29418/KNEL/PFVCluster.git` (main)
- TSYSGroupAIOS (meta): `ssh://git@git.knownelement.com:29418/TSYSGroupCorporate/TSYSGroupAIOS.git` (master)
- network-device-configs: `ssh://git@git.knownelement.com:29418/KNEL/network-device-configs.git` (master) — Oxidized backup target

## SNMP Community

All Dell PowerConnect switches: `kn3lmgmt` (v2c)

## Switch Inventory

| Switch | Model | Location | IP | Oxidized User |
|--------|-------|----------|----|---------------|
| pfv-r5-core-01 | Dell PowerConnect 5448 | PFV/Server-Room/R5 | 192.168.0.12 | oxidized (priv 1) |
| pfv-r3-tor-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.7 | oxidized (priv 1) |
| pfv-r3-tor-stor-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.9 | oxidized (priv 1) |
| pfv-r6-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R6 | 192.168.0.8 | oxidized (priv 1) |
| pfv-rrinfra-rtr | Cisco router | PFV/Server-Room/R2 (pending) | unknown | BLOCKED (#431 fan) |

## Tickets Created This Session

| # | Subject | Status |
|---|---------|--------|
| 424 | LibreNMS switch DNS rename | CLOSED |
| 425 | phpIPAM config | 90% |
| 426 | Uptime Kuma API | New |
| 427 | Credential management + IAC | New |
| 428 | Beszel alert delivery (Pushover) | New |
| 429 | SNMP trap strategy (LibreNMS vs Dell OME) | New |
| 430 | UNPoller → Prometheus/Grafana | New |
| 431 | Cisco router fan replacement | New |
| 432 | RackRental VLAN 2 | New |
| 433 | Trunk ports tsys6/7 for RackRental | New |
| 434 | Cyclades PDU IP confirm | New |
| 435 | Uptime Kuma comprehensive monitors | New |
| 436 | Cisco router → Oxidized (after fan) | New |
| 437 | Switch location standardization | 90% |

## Tomorrow's Goal: IAC

User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408)
unblocks everything: Ansible Vault → AWX playbooks → full IAC rebuild path for
all OAM tools deployed this session.