diff --git a/aug11.md b/aug11.md index fedb97b..cad11df 100644 --- a/aug11.md +++ b/aug11.md @@ -1,236 +1,9 @@ -# Handoff: 2026-08-11 OAM Session + -Point a new Crush session at this file: `cat aug11.md` (or read it). It contains -everything needed to pick up where this session left off. +# Session Handoff: 2026-08-11/12 OAM -## Quick Start (for the new session) - -1. Follow normal session-start gate (AGENTS.md steps 1-4) -2. Set active ticket: `echo '#430' > .crush/active-ticket` -3. Read this file for full context -4. Check Redmine queue for latest status - -## Session Summary - -Deployed the entire OAM monitoring stack on tsys-librenms (VM 104 on pfv-tsys1). -All tools running as Docker containers. 4 of 5 tools are 100% operational. - -## OAM Stack Status (tsys-librenms) - -| Tool | Port | Container | Status | Ticket | -|------|------|-----------|--------|--------| -| Smokeping | 8081 | oam-smokeping | DONE | #375 (90%) | -| NetDisco | 8082 | oam-netdisco-{web,backend,db} | DONE | #337 (90%) | -| Oxidized | 8083 | oam-oxidized | DONE (4/5 switches) | #436 (Cisco pending) | -| UNPoller | 8084 | oam-unpoller | POLLING but needs Prometheus/Grafana | #430 | -| Weathermap | LibreNMS | (plugin) | DONE | #375 (90%) | -| phpIPAM | Cloudron | ipam.knownelement.com | 90% configured | #425 | - -Access: `VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm ''` - -## What's Done (don't re-do) - -- Smokeping: 16 targets, FPing every 5 min, fully operational -- NetDisco: 4 Dell switches discovered, LLDP neighbors working (18 entries), web UI at :8082 -- Oxidized: oxidized user created on 4 switches, configs backing up to KNEL/network-device-configs on Gitea, auto-push via SSH key -- UNPoller: authenticated to UniFi controller (pfv-bms:8443, user: unpoller), Prometheus metrics on :8084 -- Weathermap: LibreNMS plugin enabled, generating maps via cron (network.conf) -- phpIPAM: PFV section with Server Room/EE Lab/Art Room sub-sections, 9 devices, 64 DHCP reservations imported, SNMP on 4 switches, scan agent configured -- Switch locations standardized: PFV/Server-Room/R{n} format on all 4 switches -- Switch DNS names fixed in LibreNMS (#424 closed) -- LibreNMS poller cron fix verified (#423 closed) -- Rules updated: remote.sh mandatory, DNS names only, command timeouts, questions-v(N) versioning - -## What's Left (in priority order) - -### 1. UNPoller → Prometheus/Grafana wiring (#430) -- UNPoller is running and polling. Need to add scrape config to Prometheus (Cloudron) -- BLOCKED: needs ~/.creds/prometheus.env and ~/.creds/grafana.env populated by user -- Then build Grafana dashboard for UniFi -- UNPoller v3.4.1 exports Prometheus metrics at http://tsys-librenms.knel.net:8084/metrics - -### 2. LibreNMS alert transports (#338) -- Configure Pushover as alert delivery in LibreNMS -- BLOCKED: needs ~/.creds/pushover.env (PUSHOVER_TOKEN + PUSHOVER_USER) -- **Pushover is the STANDARD alert transport for ALL OAM tools** (user directive): - - Uptime Kuma: already using Pushover - - LibreNMS: #338 (this ticket) - - Beszel: #428 - - Grafana: when wired via #430 - -### 3. SNMP trap strategy (#429) -- Decide: Dell OME owns server/iDRAC traps, LibreNMS owns network traps (recommended split) -- Awaiting Dell OME deployment (#339) before implementing -- Switches may need trap destination configured - -### 4. TEMPer USB graphing (#341) -- Deploy on pfv-tsys1 (Server Room) + stlpc-artroom (Art Room) -- USB probe driver + RRD/LibreNMS integration -- stlpc-garage (EE Lab) may also have a probe — verify with user - -### 5. Monitoring coverage matrix (#343) -- Governance doc mapping every tool to its domain (no overlap) -- Write to Discourse, update ticket -- Tool domains: Uptime Kuma=HTTP/service availability, Beszel=host metrics, - LibreNMS=SNMP/network, Dell OME=Dell hardware, Smokeping=latency, - NetDisco=L2 topology, Oxidized=config backup, phpIPAM=IP inventory, - UNPoller=UniFi metrics - -### 6. Uptime Kuma API + comprehensive monitors (#426/#435) -- Enable API write access -- Add monitors for all PFV/SITER + Cloudron services (HTTPS, DNS, DB, Docker, k8s) - -### 7. Dell OpenManage Enterprise (#339) -- USER DEPLOYING MANUALLY — has KVM image downloaded, will do qm import + VM creation -- Agent role: post-deployment integration only (SNMP traps #429, coverage matrix #343) -- Fleet is 100% Dell (documented in Discourse #304) - -## Blocked Items (waiting on user or hardware) - -- #431: Cisco router fan replacement (physical, onsite) → blocks #432, #433, #434, #436 -- #427: Credential management (Vault/RADIUS/Ansible Vault) → tomorrow's IAC track -- #430: Prometheus/Grafana creds needed (~/.creds/prometheus.env + grafana.env EMPTY) -- #338: Pushover API key needed (~/.creds/pushover.env NOT CREATED YET) -- #426: Uptime Kuma API access needed (read works, write not enabled) -- #425: phpIPAM scan agent — user may need to create from web UI and trigger first scan - -## User Action Items (things only the user can do) - -1. **Populate ~/.creds/pushover.env** — PUSHOVER_TOKEN + PUSHOVER_USER (unblocks #338, #428) -2. **Populate ~/.creds/prometheus.env + grafana.env** — Cloudron API keys (unblocks #430) -3. **Deploy Dell OME manually** — VM creation + qm import + OOB config (#339) -4. **Replace Cisco router fan** — physical onsite work (#431, blocks RackRental track) -5. **Create phpIPAM scan agent** — from web UI, trigger first discovery scan (#425) -6. **Enable Uptime Kuma API write access** — from web UI (#426) -7. **Run interactive netdisco-deploy** — DONE (user already did this) - -## Important Technical Notes - -### snmpd is NOT fleet-wide (user directive) -snmpd will NOT be rolled out to Linux hosts. Beszel covers host metrics. -SNMP (community: kn3lmgmt) is ONLY for network devices (switches, routers, PDUs). -Do NOT add snmpd deployment to any ticket scope. - -### Cloudron Tailscale routing -Cloudron is on Tailscale. pfv-netinfra-01/02 are Tailscale subnet routers advertising -192.168.0.0/22. This means Cloudron-hosted services (phpIPAM, Prometheus, Grafana) -CAN reach the LAN switches via Tailscale for SNMP discovery/scraping. - -### NetDisco required interactive deploy -The `netdisco-deploy` script requires a TTY (interactive yes/no prompts). -The agent cannot run this through remote.sh. The user ran it manually. -If NetDisco needs redeployment, the user must run this step. - -### Weathermap PHP fix -Weathermap plugin required `register_argc_argv = On` in /etc/php/8.4/fpm/php.ini -on tsys-librenms. Already applied. If PHP gets updated, verify this setting persists. - -### RackRental split-network design (Q6 answer in questions-v1.md) -tsys6/7 have 4-6 ethernet ports. User wants: -- 2 NICs for storage bond (keep existing bond0) -- 1 NIC for mgmt (non-RackRental side) -- 1 NIC trunked to RackRental VLAN 2 -- RackRental router prevents network loops - -### Location taxonomy -- PFV/Server-Room/R{n} for server room gear by rack -- PFV/EE-Lab for stlpc-garage -- PFV/Art-Room for stlpc-artroom -- Applied to all 4 switches via SNMP. Cisco pending fan fix (#431). -- phpIPAM sections match this taxonomy. - -### Ticket gate discussion -User wants to discuss relaxing the ticket gate for handoff/continuation sessions. -For now, use a "dummy" ticket (#430 as placeholder) but don't bypass the rules. -This is an open discussion item — see questions-v1.md for future Q&A. - -## Credential Files (in ~/.creds/) - -| File | Status | -|------|--------| -| redmine.env | OK | -| discourse.env | OK | -| phpipam.env | OK (app_code auth working) | -| unifi-creds.env | OK (user: unpoller, polling working) | -| oxidized.env | OK (password stored) | -| switch-creds.env | Created, user to populate if needed | -| pushover.env | NOT CREATED — needs PUSHOVER_TOKEN + PUSHOVER_USER | -| prometheus.env | Created, EMPTY — user to populate | -| grafana.env | Created, EMPTY — user to populate | - -## Rules Changed This Session - -All committed and pushed to both PFVCluster and TSYSGroupAIOS (meta): - -1. **remote.sh mandatory**: ALL SSH through chokepoint scripts, never direct ssh/scp -2. **DNS names only**: NEVER use IP literals in any command/script/config -3. **Command timeouts**: 30s reads, 120s standard, 300s deployments — every remote command wrapped in `timeout` -4. **questions-v(N) versioning**: increment filename each time answers land (v1→v2→v3) - -## Key Commits This Session - -- `1f634a6` — questions versioning rule -- `2806eab` — command timeout rule -- `eea5495` — remote.sh + DNS-names-only rule -- `1590923` — Oxidized fully operational -- `acf880a` — Weathermap deployed -- `7216eb5` — OAM stack initial deployment -- `29b3989` — UNPoller image fix - -## Key Gitea Repos - -- PFVCluster: `ssh://git@git.knownelement.com:29418/KNEL/PFVCluster.git` (main) -- TSYSGroupAIOS (meta): `ssh://git@git.knownelement.com:29418/TSYSGroupCorporate/TSYSGroupAIOS.git` (master) -- network-device-configs: `ssh://git@git.knownelement.com:29418/KNEL/network-device-configs.git` (master) — Oxidized backup target - -## SNMP Community - -All Dell PowerConnect switches: `kn3lmgmt` (v2c) - -## Switch Inventory - -| Switch | Model | Location | IP | Oxidized User | -|--------|-------|----------|----|---------------| -| pfv-r5-core-01 | Dell PowerConnect 5448 | PFV/Server-Room/R5 | 192.168.0.12 | oxidized (priv 1) | -| pfv-r3-tor-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.7 | oxidized (priv 1) | -| pfv-r3-tor-stor-01 | Dell PowerConnect 5324 | PFV/Server-Room/R3 | 192.168.0.9 | oxidized (priv 1) | -| pfv-r6-mgmt-01 | Dell PowerConnect 5324 | PFV/Server-Room/R6 | 192.168.0.8 | oxidized (priv 1) | -| pfv-rrinfra-rtr | Cisco router | PFV/Server-Room/R2 (pending) | unknown | BLOCKED (#431 fan) | - -## Tickets Created This Session - -| # | Subject | Status | -|---|---------|--------| -| 424 | LibreNMS switch DNS rename | CLOSED | -| 425 | phpIPAM config | 90% | -| 426 | Uptime Kuma API | New | -| 427 | Credential management + IAC | New | -| 428 | Beszel alert delivery (Pushover) | New | -| 429 | SNMP trap strategy (LibreNMS vs Dell OME) | New | -| 430 | UNPoller → Prometheus/Grafana | New | -| 431 | Cisco router fan replacement | New | -| 432 | RackRental VLAN 2 | New | -| 433 | Trunk ports tsys6/7 for RackRental | New | -| 434 | Cyclades PDU IP confirm | New | -| 435 | Uptime Kuma comprehensive monitors | New | -| 436 | Cisco router → Oxidized (after fan) | New | -| 437 | Switch location standardization | 90% | - -## Tickets Updated (not created) This Session - -| # | What we did | -|---|-------------| -| 314 | Parent umbrella for OAM alerts — updated children list | -| 337 | LibreNMS integration modules — 90% done (Oxidized/NetDisco/UNPoller deployed) | -| 339 | Dell OME — confirmed 100% Dell fleet, user deploying manually | -| 375 | Monitoring expansion — smokeping done, weathermap done, snmpd removed from scope | -| 406 | Beszel API — cross-referenced with OAM platform | -| 425 | phpIPAM — revised from "deploy" to "configure existing Cloudron instance" | -| 370 | Vault umbrella — clarified parent-child relationships | - -## Tomorrow's Goal: IAC - -User wants to move to IAC (AWX/Ansible) tomorrow/Thursday night. Vault (#408) -unblocks everything: Ansible Vault → AWX playbooks → full IAC rebuild path for -all OAM tools deployed this session. +**Full handoff lives in Redmine #314** (session summary note, 2026-08-12). +**OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309). +**Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303). +**Open questions:** `questions-v2.md` in this repo. diff --git a/questions-v1.md b/questions-v1.md index 2b1d5f8..9238b49 100644 --- a/questions-v1.md +++ b/questions-v1.md @@ -56,6 +56,6 @@ No more JOURNAL.md . Redmine is the system of record. JOURNAL.md was a hack I wa ### Q6. RackRental split-network design on pfv-tsys6/7 [#433] - **Context:** TSYS6 and TSYS7 need trunk ports for RackRental. Servers will run a "split network configuration — half on the non-RackRental switch/router side, half on the RackRental switch/router side." RackRental router prevents network loops. - **Question:** Is the split per-NIC (one physical NIC to normal network, one to RackRental) or per-VM (some VMs bridge to RackRental, some to normal)? Both TSYS6/7 currently have bond0 with 2 slaves for storage network. How should the bonds be reconfigured? -- **Answer:** _(human)_ +- **Answer:** Keep the storage bonds. We need that bandwidth. I believe tsys6,7 have six ethernet ports each? Two on a riser card, and a four port add-on? Or maybe its four total? If four total, the 2 for storage, one for mgmt non rackrental, one trunked to rackrental would be how I want to go. - **Decision:** _(human/agent)_ - **Synthesized to:** — diff --git a/questions-v2.md b/questions-v2.md new file mode 100644 index 0000000..9284e39 --- /dev/null +++ b/questions-v2.md @@ -0,0 +1,53 @@ +# questions-v2.md + +> Git-tracked question log. The agent writes; the human reviews/edits inline. +> Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine. +> See BASELINE-PROMPT.md §9. + +## Resolved (from v1) + +### Q1. Git remote for meta? — RESOLVED +- **Decision:** New Gitea repo under TSYSGroupCorporate org: `TSYSGroupAIOS`. Template repo. Created and pushed. +- **Synthesized to:** Meta repo created, `tea` + `docker login` noted as available on workstations. + +### Q2. The bin/ wrapper gap — RESOLVED +- **Decision:** No bin/ wrappers. Invoke real containers with `docker run --env-file`. +- **Synthesized to:** PFVCluster AGENTS.md §CLI access; meta AGENTS.md §CLI invocation. + +### Q3. Discourse pointer-header pattern — RESOLVED +- **Decision:** Mandatory for ALL projects. No exceptions. Discourse/Redmine required. +- **Synthesized to:** Meta AGENTS.md, check-rules.sh enforces. + +### Q4. Sub-agent nudge hook — RESOLVED +- **Decision:** No harness-specific hooks. Move to portable git hooks + AGENTS.md protocols. User shifting away from Crush to OpenWebUI/Hermes. +- **Synthesized to:** All hooks ported to git hooks. No Crush-specific dependencies. + +### Q5. JOURNAL.md vs Discourse — RESOLVED +- **Decision:** No more JOURNAL.md. Redmine is SoR for work. Discourse for synthesized knowledge. +- **Synthesized to:** Meta AGENTS.md, PFVCluster AGENTS.md. + +### Q6. RackRental split-network design — RESOLVED +- **Decision:** Keep storage bonds. If 4 NICs total: 2 storage, 1 mgmt non-RackRental, 1 trunked to RackRental VLAN 2. +- **Synthesized to:** #433 (trunk ports), #432 (VLAN config). + +## Open questions + +### Q7. Close #167 (Small UPS Units Re-build)? +- **Context:** #167 is an old (Dec 2024) vague ticket about rebuilding garage UPS units. It's been superseded by #372 (Tripp Lite physical reseat) and #439 (UPS data integration). Dates are stale (Aug 2025). +- **Question:** Should we close #167 as superseded? +- **Answer:** _(human)_ +- **Decision:** _(human/agent)_ +- **Synthesized to:** — + +### Q8. Friday onsite work order? +- **Context:** Friday 2026-08-14 onsite batch. Tickets: #341 (TEMPer deploy), #372 (Tripp Lite UPS reseat), #431 (Cisco fan replacement), #414 (tsys5 nic2 cable swap), #374 (PDU outlet renaming), #439 (UPS integration). +- **Question:** What order do you want to tackle these? Cisco fan (#431) blocks #432/#433/#434/#436 — should it go first? TEMPer is the "fun" project — save for last? +- **Answer:** _(human)_ +- **Decision:** _(human/agent)_ +- **Synthesized to:** — + +### Q9. Session-start gate: relax for continuation sessions? +- **Context:** You mentioned last session wanting to discuss relaxing the ticket gate for handoff/continuation sessions. The scope-alignment gate now handles this (presents handoff priorities, waits for user to name target). Is this sufficient, or do you want further changes? +- **Answer:** _(human)_ +- **Decision:** _(human/agent)_ +- **Synthesized to:** —