docs: session cleanup — coverage matrix, questions v2, handoff pointer [#343 ]

- OAM coverage matrix published on Discourse #309 - questions-v2.md: Q1-Q6 resolved, Q7-Q9 added - aug11.md: replaced with pointer to Redmine #314 + Discourse #309 - Redmine cleanup: 16 tracker fixes, cross-refs, stale ticket reviews - UNPoller InfluxDB noise silenced (config mount fix + disable=true) 💘 Generated with Crush Assisted-by: Crush:glm-5.2
docs(agents): add no-direct-DB-access rule — API-first always
2026-08-12 11:44:58 -05:00 · 2026-08-11 22:50:27 -05:00 · 2026-08-11 21:41:47 -05:00 · 2026-08-11 20:34:38 -05:00 · 2026-08-11 20:32:32 -05:00 · 2026-08-11 20:20:45 -05:00
49 changed files with 2219 additions and 138 deletions
@@ -0,0 +1,3 @@
 *
 !crush.json
 !.gitignore
@@ -0,0 +1,11 @@
 {
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "^(bash|edit|write|multiedit|lsp_replace_symbol|lsp_rename)$",
        "command": "./hooks/ticket-gate.sh",
        "timeout": 5
      }
    ]
  }
 }
@@ -1,5 +1,9 @@
-# Crush internal state
+# Crush internal state (track crush.json config, ignore session data)
-.crush/
+.crush/crush.db
 .crush/crush.db-*
 .crush/logs/
 .crush/memory/
 .crush/active-ticket
 # OS/editor
 .DS_Store
@@ -1,15 +1,145 @@
 # Agent Guidelines
 ## Agent Authority (NON-NEGOTIABLE)
 **No work is permissible on any system without an approved Redmine ticket.
 There are no exceptions to this rule.**
 ### Scope of authority
 1. **Tickets govern all work.** The agent performs ONLY the work described in
   the approved Redmine ticket. Anything outside that scope — no matter how
   small, helpful, or "obvious" — is prohibited.
 2. **No autonomous system changes.** The agent does not modify, configure,
   create, delete, or grant anything on a production system unless it is
   explicitly directed by an approved ticket. "Production system" means
   every system in the fleet — there is no "test" exception unless the ticket
   says so.
 3. **Propose, never implement.** If the agent discovers additional work that
   should be done — a bug, a misconfiguration, a missing dependency, an
   enhancement — it does NOT implement it. Instead, it creates a Redmine
   sub-ticket (status Feedback) describing the finding and surfaces it to
   the user for approval.
 4. **Security and access changes require extra scrutiny.** Changes to sudoers,
   SSH keys, user accounts, firewall rules, authentication policy, file
   permissions, or any privilege-related configuration are treated as
   policy decisions, not implementation details. The agent may suggest
   them but NEVER implements them without explicit user direction in the
   ticket or a sub-ticket the user has approved.
 5. **The user makes policy. The agent implements policy.** The agent does
   not decide who gets sudo, what keys go where, what services run, or what
   the access model is. The agent executes the user's decisions, exactly
   as specified.
 6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
   require something not explicitly authorized, or if the agent is unsure
   whether an action is in scope — STOP and ask the user via the ticket
   or directly. Asking is always acceptable. Overstepping is never
   acceptable.
 7. **NEVER close a ticket without explicit user permission.** You may
   SUGGEST a close when the result is clearly scoped and delivered. If
   it's ambiguous whether the work is truly complete, don't suggest a
   close — leave that decision to the user. This applies to ALL tickets,
   no exceptions.
 8. **User acceptance testing is MANDATORY before declaring work done.**
   The agent performs implementation and technical validation (services
   running, configs correct, APIs responding). The user performs UAT —
   visually confirming dashboards render data, alerts deliver, tools are
   usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
   closing, and MUST NOT move to the next ticket until the user explicitly
   accepts the work. "Technically wired but blank dashboard" is NOT done.
 9. **NEVER access a database directly if an API exists.** APIs are the
   stable contract; databases change schemas without warning. If a tool
   has an API, use it — exhaust all API endpoints, check the docs, try
   alternative methods. Only fall back to direct DB access as a last
   resort AND with explicit user approval for that specific instance.
   Soon all DB access will route through a proxy under zero trust;
   building API-first habits now ensures that transition is clean.
 ### Access-channel policy: SSH only (NON-NEGOTIABLE)
 **The qemu guest-agent is NEVER an access, execution, or key-delivery
 channel.** SSH (`sshd`) is the only approved remote access path. Every
 command must flow through sshd so it is captured by the standard
 auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
 environment — there is no back-door exception, ever.
 - **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
  over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
  mode). This includes using guest-agent to *deliver* an SSH key, even if
  the resulting SSH login is itself audited — the delivery bypassed audit.
 - **Allowed:** installing or checking qemu-guest-agent for its intended
  purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
  `agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
  never execution.
 - **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
  any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
  no guest-exec mode.
 - **If a system is locked out** (no SSH key, no guest-exec path): surface
  it to the user. Do NOT improvise an alternate back-channel. The user
  authorizes the unblock method (console login, credential, etc.).
 ### Ticket-closing policy (NON-NEGOTIABLE)
 **NEVER close a Redmine ticket without explicit user permission.** You
 may SUGGEST a close when the result is clearly scoped and delivered. If
 it's ambiguous whether the work is truly complete, don't suggest a close
 — leave that decision to the user. This applies to ALL tickets, no
 exceptions.
 ### What this means in practice
 - Discovered a typo in a config during approved work? **Finish the approved
  work. Create a sub-ticket for the typo. Do not fix it inline.**
 - Think a system should also have localuser sudo configured? **Do not add
  it. Propose it in a sub-ticket.**
 - Need to install a package the ticket didn't mention? **Ask first.**
 - Found a security issue? **Create a ticket immediately with full details.
  Do not remediate without approval.**
 This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
 traceable to an approved ticket. There is no "I thought it would help."
 ## Quick Start
 **You are an AI agent working on this project. Your first actions, in order:**
 > **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
 > check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
 > the scope-alignment gate. You MUST NOT begin any task work until the user
 > (a) confirms your rule summary AND (b) names the ticket to work on. This runs
 > **every session, automatically** — the user should never have to ask for it.
 1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
 2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
-3. **Read [questions-v1.md](questions-v1.md)** — open questions awaiting human input.
+3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
-4. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
+4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
-5. **Check current state:** `git log --oneline -10`.
+   back to the user before doing any work: Agent Authority (ticket-governed,
-6. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
+   propose-never-implement), Access-Channel SSH-only policy, Remote access
   (remote.sh chokepoint mandatory, DNS names only — never IP literals),
   Questions policy (no harness question-tools), Documentation policy
   (Discourse is SoR), Redmine tracking, Git policy (always commit+push,
   shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
   routing chokepoints (`tests/remote.sh`), the mechanically enforced
   `check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
   and wait for the user to confirm. This checkpoint guarantees every
   session starts aligned.
 5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
 6. **Check current state:** `git log --oneline -10`.
 7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
 8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
   priority list (or the Redmine queue if no handoff exists). Do NOT scan the
   full ticket queue and pick work on your own — **the user directs what gets
   worked on, always.** Note any new or urgent items from the Redmine check,
   then ask which ticket to work on tonight. **STOP and wait.** Do not set
   `.crush/active-ticket`, do not read systems, do not run diagnostics, until
   the user names the target. This gate prevents the agent from burning
   context on work the user didn't ask for.
 ## Enforcement Model
@@ -23,10 +153,57 @@ for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
 ## Task Tracking
 - **Redmine is the system of record for all work.**
 - **NEVER close a ticket without explicit user permission.** Suggest a
  close when clearly scoped/delivered; if ambiguous, don't suggest.
 - **Ticket-first enforcement (mechanically enforced).** Before starting
  any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
  The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
  until this file exists. If no ticket exists, CREATE ONE FIRST via
  redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
 - **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
  session. The pre-commit hook blocks commits while any task remains unchecked.
 - Clear WORKING.md before responding to the user.
 ## Rolling HUD (session-scoped, NOT persisted)
 The rolling HUD is a **live status display** the agent maintains throughout
 the session to help the human follow along. It is NOT a system of record —
 Redmine, Discourse, and git are the durable systems. The HUD exists purely
 for the human's situational awareness during the session.
 - **Format:** a compact block shown at the end of each significant response
  (after completing a step, hitting a blocker, or pivoting). Example:
  ```
  ┌─ SESSION HUD ────────────────────────────────────────
  │ Active: #343 (Monitoring coverage matrix)
  │ Done:   ✓ scope-alignment gate added to both AGENTS.md
  │         ✓ committed + pushed (6a2550b)
  │ Now:    drafting coverage matrix on Discourse
  │ Next:   → #341 TEMPer USB (deploy on pfv-tsys1)
  │         → #338 LibreNMS alerts (BLOCKED: pushover.env)
  ├─ USER ACTION ITEMS ──────────────────────────────────
  │ • Populate ~/.creds/pushover.env (unblocks #338, #428)
  │ • Populate ~/.creds/prometheus.env + grafana.env (#430)
  └──────────────────────────────────────────────────────
  ```
 - **Placement:** may be written to `.crush/hud.md` on disk to keep context
  window smaller (re-read and update rather than hold in memory). Never
  committed to git. Wiped at session end.
 - **Sections:**
  - **Active:** current ticket number + one-line description
  - **Done:** ✓ items completed this session (append as work progresses)
  - **Now:** what the agent is actively doing
  - **Next:** the queued items (per handoff priority or user direction)
  - **User action items:** things ONLY the user can do (populate creds,
    physical work, manual deploys) with the tickets they unblock
 - **When to show it:** after each logical unit of work, at blockers, and
    when pivoting between tickets. Not every trivial response — use judgment.
 - **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
    work completes, update the durable systems (ticket notes, Discourse wiki,
    commits). The HUD just tracks the live narrative for the human.
 ## Working Style
 - **Stop over-thinking.** Get to code and output faster. Explore with code;
@@ -37,6 +214,26 @@ for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
  inside it.
 - **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
  Never read 10+ files sequentially; batch into agent calls.
 - **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
  system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
  (status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
  If a command hits the timeout, STOP and investigate root cause — never
  blindly retry. A hung command is a failed command. Detect failure fast,
  diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
  This applies to ALL tools — bash, docker, CLIs, sub-agents.
 ## Questions (NON-NEGOTIABLE)
 **NEVER use a harness "question"/"ask user" tool** (structured prompts,
 modal forms, tabbed questions). Banned across every project, every harness.
 They are not portable, not version-controlled, and bypass the git record.
 **All questions go in the current `questions-v(N).md` file** — write the
 question; the human edits the answer inline in the same file. **Version up
 the filename each time answers land** (v1 → v2 → v3...): create
 `questions-v2.md` with resolved Q&A marked, new questions appended. This
 preserves the history of each Q&A round. Synthesize resolved Q&A into
 Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
 §10.
 ## Documentation policy (IMPORTANT)
@@ -88,9 +285,27 @@ vendor/             Vendored KNELShellFramework
  `/home/reachableceo/projects/KNELIAC`.
 - **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
  extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **SSH access**: Direct ssh/scp may be blocked by some AI harnesses. Use
+- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
-  [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
+  through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
-  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
+  (Proxmox hosts + all VMs) or
  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh)
  (DNS infra hosts: netinfra-01/02, tsrouter, netboot). NEVER call
  `ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
  rejects it. There are no exceptions.
 - **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
  (neither LAN nor Tailscale IPs) in any command, script, or config.
  ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
  `VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
  DNS name does not resolve, fix it in DNS (Technitium) or consult the
  [system inventory — Discourse #307](https://community.turnsys.com/t/307).
  Do NOT fall back to IP literals. This rule eliminates the per-session
  discovery tax of finding the right IP for each host.
 - **How to access a production VM:**
  1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
  2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
  3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
     and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
     NOT execution) to find the Tailscale DNS name, then access via that.
 ## Git Policy
@@ -131,7 +346,11 @@ checklists, or TODOs in repo files. Use Redmine tickets instead.
 2. **Reference tickets in docs and commits** using `[#NNN]` notation.
   Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
 3. **When work completes**, update the ticket: set done ratio to 100%,
-   add a note describing what was done and where the code lives, then close.
+   add a note describing what was done and where the code lives.
   **NEVER close a ticket without explicit user permission.** You may
   SUGGEST a close when the result is clearly scoped and delivered. If
   it's ambiguous whether the work is truly complete, don't suggest a
   close — leave that decision to the user.
 4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
 5. **Link code to tickets** — ticket descriptions and notes should reference
   the relevant file paths in this repo (e.g., `dcinfra/ups/`).
@@ -253,8 +472,8 @@ live in the centralized store at `~/.creds/discourse.env`.
 Switch configs are pulled via serial console through a conman + ser2net
 stack on pfv-tsys4.
- **conmand server:** pfv-tsys4 (Tailscale `100.70.77.93`, port 7890)
+- **conmand server:** pfv-tsys4 (port 7890)
- **ser2net:** pfv-tsys4, TCP ports 2001-2007 on Tailscale IP
+- **ser2net:** pfv-tsys4, TCP ports 2001-2006
  (do NOT connect to ser2net directly — it conflicts with conman's
  persistent sessions; always use the conman client)
 - **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
@@ -266,18 +485,17 @@ stack on pfv-tsys4.
 ```bash
 # Pull a switch config (example):
 CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
-  --console pfv-core-sw01 --cmds netinfra/switches/pfv-core-sw01.cmds
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
 ```
 | Console name | TCP port | Device |
 |--------------|----------|--------|
-| pfv-core-sw01 | 2001 | Dell PowerConnect 5448 (core, rack 5) |
+| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
-| pfv-tor3-mgmt | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
+| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
-| pfv-tor3-stor | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
+| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
 | pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
-| pfv-r2-tor-top | 2005 | Rack 2 top-of-rack switch |
+| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
-| subodev-torsw | 2006 | Suborbital device TOR switch |
+| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
 | pfv-r2-sw | 2007 | Rack 2 old Dell switch |
 ## Mandatory: Document ALL Infrastructure Changes
@@ -324,16 +542,19 @@ docker run --rm --env-file ~/.creds/redmine.env \
 docker run --rm --env-file ~/.creds/discourse.env \
    git.knownelement.com/reachableceo/discourse-cli:latest whoami
-# Proxmox host access:
+# Proxmox host access (DNS names only — never IPs):
 PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
 PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
 # Production VM access (DNS names only — never IPs):
 VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
 # DNS infra access:
 bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
-# Switch console (read-only config pull):
+# Switch console (read-only config pull — DNS names only):
-CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
+CONMAN_SERVER=pfv-tsys4:7890 python3 proxmox/perf/scripts/conman-console.py \
-  --console pfv-core-sw01 --cmds netinfra/switches/pfv-core-sw01.cmds
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
 ```
 ## Credential Management
@@ -1,9 +1,11 @@
 # WORKING.md — Active Session Tracker
 Agent work only. The human decides when it's done.
 The todos tool is banned; this is the only task tracker.
 A commit is blocked while any task below remains unchecked.
 ## Current Tasks
-(none — all agent work is committed)
+(all done — session complete)
 - [x] Add ticket-gate Crush hook (blocks work without active ticket)
 - [x] Document active-ticket workflow in AGENTS.md + meta template
@@ -0,0 +1,67 @@
 #!/usr/bin/bash
 # access-matrix.sh — definitive access verification across all Linux Tailscale nodes.
 # Uses the correct SSH user(s) per system type, checks sudo where applicable.
 # Routes through remote.sh (the only allowed ssh path).
 set -u
 cd /home/reachableceo/projects/PFVCluster || exit 1
 # Policy-excluded systems (never attempt access)
 EXCLUDE=':tsys-cloudron:pfv-bms:tsys-umbrel:tsys-ucs-01:tsys-ucs-02:stlpc-bizoffice:ultix-highside:'
 # Determine the SSH user(s) for a given hostname and whether sudo is expected.
 # Returns "user1:user2:...:sudoflag" where sudoflag is "yes" or "no".
 users_for() {
  local name="$1"
  case "$name" in
    pfv-tsys[0-9])                   echo "root:no" ;;
    *-proxmox-datacenter)            echo "root:no" ;;
    *-proxmox-pve)                   echo "root:no" ;;
    *-proxmox-pbs)                   echo "root:no" ;;
    *-proxmox-mailgw*)               echo "root:no" ;;
    *-proxmox-backup*)               echo "root:no" ;;
    tsys-ucs-*)                      echo "root:no" ;;
    ultix-streaming)                 echo "root:no" ;;
    stlpc-*)                         echo "root:labuser:no" ;;
    ultix-field)                     echo "ultixfield:yes" ;;
    subopi*)                         echo "subodev:yes" ;;
    *)                               echo "localuser:yes" ;;
  esac
 }
 check_user() {
  local ip="$1" user="$2" expect_sudo="$3"
  local out sudo
  out=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'echo SSHOK; id -un' </dev/null 2>&1 | grep -oE 'SSHOK|keyboard-interactive|Connection refused' | head -1)
  case "$out" in
    SSHOK)
      if [ "$expect_sudo" = "yes" ]; then
        sudo=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'sudo -n true 2>/dev/null && echo SUDOOK || echo SUDONO' </dev/null 2>&1 | grep -oE 'SUDOOK|SUDONO' | head -1)
        printf '%s(%s)' "$user" "${sudo:-?}"
      else
        printf '%s(ok)' "$user"
      fi
      ;;
    keyboard-interactive) printf '%s(2FA)' "$user" ;;
    'Connection refused')  printf '%s(NOSSH)' "$user" ;;
    *)                      printf '%s(NOKEY)' "$user" ;;
  esac
 }
 printf '%-32s %-16s %s\n' "NAME" "TS-IP" "ACCESS"
 printf '%-32s %-16s %s\n' "----" "-----" "------"
 tailscale status 2>/dev/null | awk '$4=="linux" {print $2, $1}' | sort | while read -r name ip; do
  [ -n "$name" ] || continue
  case "$EXCLUDE" in *":$name:"*) printf '%-32s %-16s %s\n' "$name" "$ip" "EXCLUDED"; continue;; esac
  map=$(users_for "$name")
  expect_sudo="${map##*:}"
  users="${map%:*}"
  result=""
  IFS=':' read -ra user_list <<< "$users"
  for u in "${user_list[@]}"; do
    r=$(check_user "$ip" "$u" "$expect_sudo")
    [ -z "$result" ] && result="$r" || result="$result $r"
  done
  printf '%-32s %-16s %s\n' "$name" "$ip" "$result"
 done
@@ -0,0 +1,60 @@
 #!/bin/sh
 # agent-bootstrap.sh
 #
 # Run INSIDE a guest (via SSH, console, or guest-agent) to bring the
 # system fully under agent management in one shot:
 #   1. install + enable qemu-guest-agent  (VMs only, skipped on bare metal)
 #   2. push the agent SSH key to root + AGENT_USER (+ labuser if present)
 #   3. grant AGENT_USER passwordless sudo
 #
 # AGENT_USER defaults to "localuser". Override for systems with a different
 # unprivileged agent user:
 #   AGENT_USER=subodev bash agent-bootstrap.sh
 #
 # After this runs once, the agent has SSH+sudo immediately.
 #
 # Usage (from a root shell in the guest):
 #   bash agent-bootstrap.sh
 #   AGENT_USER=subodev bash agent-bootstrap.sh
 set -eu
 KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIWms/uCXnjjo4KyxHBcYI2TDHe8OZ2wle6W/0hSRQLu reachableceo@ultix-streaming'
 AGENT_USER="${AGENT_USER:-localuser}"
 # 1. guest-agent (skip on bare metal — no virtio-serial device)
 if command -v systemd-detect-virt >/dev/null 2>&1 && \
   [ "$(systemd-detect-virt --vm 2>/dev/null || echo none)" != "none" ]; then
  if ! command -v qemu-ga >/dev/null 2>&1; then
    if command -v apt-get >/dev/null 2>&1; then
      apt-get update
      DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
    elif command -v dnf >/dev/null 2>&1; then
      dnf install -y qemu-guest-agent
    elif command -v yum >/dev/null 2>&1; then
      yum install -y qemu-guest-agent
    else
      echo "WARN: no supported package manager; skipping agent install" >&2
    fi
  fi
  systemctl enable --now qemu-guest-agent 2>/dev/null || \
    systemctl enable --now qemu-ga 2>/dev/null || true
 fi
 # 2. SSH key for root + AGENT_USER + labuser (if present)
 for u in root "$AGENT_USER" labuser; do
  if ! getent passwd "$u" >/dev/null 2>&1; then continue; fi
  H=$(getent passwd "$u" | cut -d: -f6)
  mkdir -p "$H/.ssh"; chmod 700 "$H/.ssh"
  AK="$H/.ssh/authorized_keys"; touch "$AK"; chmod 600 "$AK"
  grep -qF "$KEY" "$AK" || echo "$KEY" >> "$AK"
  chown -R "$u": "$H/.ssh"
 done
 # 3. passwordless sudo for AGENT_USER only
 if getent passwd "$AGENT_USER" >/dev/null 2>&1 && [ -d /etc/sudoers.d ]; then
  echo "${AGENT_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/010-agent
  chmod 440 /etc/sudoers.d/010-agent
 fi
 echo BOOTSTRAP-DONE
@@ -0,0 +1,9 @@
 <!-- Discourse: https://community.turnsys.com/t/298 -->
 <!-- Redmine: https://projects.knownelement.com/issues/314 -->
 # Session Handoff: 2026-08-11/12 OAM
 **Full handoff lives in Redmine #314** (session summary note, 2026-08-12).
 **OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309).
 **Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303).
 **Open questions:** `questions-v2.md` in this repo.
@@ -0,0 +1,65 @@
 #!/usr/bin/bash
 # bootstrap-all.sh — push agent SSH key + passwordless sudo to remaining systems.
 #
 # SSH is the ONLY approved access channel (see AGENTS.md "Access-channel
 # policy: SSH only"). This script reaches systems that still allow password
 # auth over sshd. Systems that reject password auth (publickey-only) cannot
 # be reached this way — see the CONSOLE-ONLY section printed at the end.
 #
 # Two escalation methods:
 #   sudo → Ubuntu-style systems (no root pw; localuser has sudo)
 #   su   → Debian-style systems (root has a password)
 #
 # Passes AGENT_USER so agent-bootstrap.sh targets the correct unprivileged
 # user. You enter passwords interactively. Idempotent: safe to re-run.
 #
 # Verified state (access-matrix.sh, 2026-08-10): 68/70 non-excluded systems
 # at intended access state. Only tsys-siem remains below.
 set -u
 cd "$(dirname "$0")" || exit 1
 SCRIPT=agent-bootstrap.sh
 SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
 run_with_sudo() {
  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
  echo "========================================"
  echo "  $name ($ip) — $user (sudo, agent=${agent_user})"
  echo "========================================"
  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "sudo AGENT_USER=${agent_user} bash /tmp/$SCRIPT" \
    && echo "  -> $name DONE" \
    || echo "  -> $name FAILED"
  echo
 }
 run_with_su() {
  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
  echo "========================================"
  echo "  $name ($ip) — $user (su, agent=${agent_user})"
  echo "========================================"
  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "su -c 'AGENT_USER=${agent_user} bash /tmp/$SCRIPT'" \
    && echo "  -> $name DONE" \
    || echo "  -> $name FAILED"
  echo
 }
 # All password-auth-reachable systems have been bootstrapped.
 # Verified state (access-matrix.sh, 2026-08-10): 69/70 non-excluded systems
 # at intended access state. The only remaining NOKEY (stlp-3dscanner) is
 # deferred to [#417] and requires a rename + bring-online first — out of
 # scope here. Run `access-matrix.sh` to re-verify at any time.
 echo "All password-auth-reachable systems are bootstrapped."
 echo "Remaining gap: stlp-3dscanner (deferred to [#417])."
 echo "Run access-matrix.sh to re-verify."
 echo "Deferred (separate ticket):"
 echo "  stlp-3dscanner — rename + bring online first [#417]"
 echo "========================================"
 echo "By design (leave alone):"
 echo "  sectestbed-sandbox — 2FA enforced"
 echo "========================================"
 echo "Excluded by policy (no SSH access):"
 echo "  pfv-bms (API), tsys-cloudron (prod revenue),"
 echo "  tsys-ucs-01/02 (API-managed), tsys-umbrel (treasury)"
 echo "========================================"
@@ -6,7 +6,7 @@
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
-#      to a named symlink like /dev/consoles/pfv-core-sw01
+#      to a named symlink like /dev/consoles/pfv-r5-core-01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing
@@ -17,13 +17,13 @@
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
-2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
+2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
-2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
+2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
-2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
+2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
-2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
+2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
-2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
+2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
-2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
+# 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
@@ -5,7 +5,7 @@
 #
 # Usage:
 #   bash console/query-remote.sh                    # list consoles
-#   bash console/query-remote.sh pfv-core-sw01      # connect to a console
+#   bash console/query-remote.sh pfv-r5-core-01      # connect to a console
 #
 set -euo pipefail
@@ -52,7 +52,7 @@ if [ -z "$CONSOLE" ]; then
  conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
  echo ""
  echo "To connect: bash $0 <console-name>"
-  echo "  e.g: bash $0 pfv-core-sw01"
+  echo "  e.g: bash $0 pfv-r5-core-01"
 else
  echo ""
  echo "--- Connecting to: $CONSOLE ---"
@@ -202,7 +202,7 @@ echo "    ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
 echo "    conman connects via telnet for logging + multiplexing"
 echo ""
 echo "  Connect from any Tailscale workstation:"
-echo "    conman -d pfv-tsys4:7890 -f pfv-core-sw01"
+echo "    conman -d pfv-tsys4:7890 -f pfv-r5-core-01"
 echo "    conman -d pfv-tsys4:7890 -q              # list consoles"
 echo ""
 echo "  Direct telnet (emergency, conflicts with conman):"
@@ -45,7 +45,7 @@ echo "--- 3. Trigger log capture: connect to each console briefly ---"
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
-for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
+for name in pfv-r5-core-01 pfv-r3-tor-mgmt-01 pfv-r3-tor-stor-01 pfv-rrinfra-rtr pfv-r2-tor-01 pfv-r6-mgmt-01; do
  logfile="$LOGDIR/${name}.log"
  if [ -f "$logfile" ]; then
    SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
@@ -0,0 +1,41 @@
 #!/usr/bin/bash
 # powerman/identify-outlets.sh — flash each PDU outlet sequentially for physical tracing
 #
 # Run this from the workstation. It flashes each outlet one at a time so you
 # can walk the rack and see which device's LED blinks. Write down the mapping,
 # then run rename-outlets.sh with that mapping.
 #
 # Usage:
 #   bash dcinfra/powerman/identify-outlets.sh
 #
 # On Friday: run this, walk the rack, note which outlet → which device.
 set -uo pipefail
 PROX_HOST="${PROX_HOST:-pfv-tsys1}"
 REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
 echo "PDU Outlet Identification — Flash Sequence"
 echo "============================================"
 echo "Each outlet will flash for 5 seconds. Walk the rack and note the device."
 echo "Press Enter to start..."
 read -r
 for i in $(seq 1 10); do
  echo "--- Outlet $i: FLASHING (5s) ---"
  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -f outlet-$i" </dev/null 2>/dev/null
  sleep 5
  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -u outlet-$i" </dev/null 2>/dev/null
  echo "    Outlet $i → ? (write it down)"
  echo ""
  [ "$i" -lt 10 ] && { echo "Press Enter for next outlet..."; read -r; }
 done
 echo "============================================"
 echo "Done. Now create your mapping file and run:"
 echo "  bash dcinfra/powerman/rename-outlets.sh"
 echo ""
 echo "Format: outlet-number:new-name (one per line)"
 echo "Example:"
 echo "  1:pfv-tsys1"
 echo "  2:pfv-tsys3"
 echo "  ..."
 echo "============================================"
@@ -0,0 +1,52 @@
 #!/usr/bin/bash
 # powerman/rename-outlets.sh — rename PDU outlets in powerman.conf
 #
 # Takes a mapping file (outlet-number:new-name, one per line) and rewrites
 # the node entries in /etc/powerman/powerman.conf on pfv-tsys1, then
 # restarts powermand.
 #
 # Usage:
 #   bash dcinfra/powerman/rename-outlets.sh <mapping-file>
 #
 # Example mapping file:
 #   1:pfv-tsys1
 #   2:pfv-tsys3
 #   3:pfv-tsys4
 #   ...
 set -euo pipefail
 PROX_HOST="${PROX_HOST:-pfv-tsys1}"
 REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
 MAP_FILE="${1:-}"
 if [ -z "$MAP_FILE" ] || [ ! -f "$MAP_FILE" ]; then
  echo "Usage: $0 <mapping-file>"
  echo "  Format: outlet-number:new-name (one per line)"
  echo "  Run identify-outlets.sh first to get the mapping."
  exit 1
 fi
 # Build the new node lines
 NODE_LINES=""
 while IFS=: read -r num name; do
  [ -z "$num" ] && continue
  NODE_LINES+="node \"$name\" \"cyclades-pm10\" \"$num\""$'\n'
 done < "$MAP_FILE"
 # Send to tsys1: backup conf, write new node section, restart powermand
 PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox-file - <<REMOTE_SCRIPT
 set -euo pipefail
 cp /etc/powerman/powerman.conf /etc/powerman/powerman.conf.bak.\$(date +%Y%m%d-%H%M%S)
 # Strip existing node lines and append new ones
 grep -v '^node "outlet-' /etc/powerman/powerman.conf > /tmp/powerman.conf.new
 cat >> /tmp/powerman.conf.new <<'NODES'
 $(echo -n "$NODE_LINES")
 NODES
 mv /tmp/powerman.conf.new /etc/powerman/powerman.conf
 systemctl restart powerman
 sleep 1
 powerman -l
 REMOTE_SCRIPT
 echo "PDU outlets renamed. Verify with: PROX_HOST=$PROX_HOST bash $REMOTE_SH prox 'powerman -q'"
@@ -0,0 +1,78 @@
 #!/usr/bin/env bash
 # hooks/ticket-gate.sh — enforce ticket-first work policy
 #
 # Blocks modifying operations until an active ticket is established.
 # The agent sets the active ticket via:  echo '#NNN' > .crush/active-ticket
 # And clears it when done:               > .crush/active-ticket
 #
 # Exempts read-only and management commands (so you can create tickets,
 # run audits, check status, etc.).
 set -euo pipefail
 TICKET_FILE="${CRUSH_PROJECT_DIR}/.crush/active-ticket"
 TOOL="${CRUSH_TOOL_NAME:-}"
 CMD="${CRUSH_TOOL_INPUT_COMMAND:-}"
 # Read-only tools — always allowed
 case "$TOOL" in
  view|ls|grep|glob|agent|sourcegraph|fetch|agentic_fetch|download|lsp_diagnostics|lsp_symbols|lsp_definition|lsp_references|lsp_call_hierarchy|crush_info|crush_logs|question|todos)
    exit 0
    ;;
 esac
 # For bash tool: exempt read-only and management commands
 if [ "$TOOL" = "bash" ]; then
  # Ticket/doc/dns management — always allowed
  case "$CMD" in
    *"redmine-cli"*|*"discourse-cli"*|*"dns-cli"*|*"technitium"*) exit 0 ;;
  esac
  # Read-only git
  case "$CMD" in
    *"git status"*|*"git log"*|*"git diff"*|*"git show"*|*"git branch"*) exit 0 ;;
  esac
  # Repo hygiene scripts
  case "$CMD" in
    *"check-rules"*|*"setup-hooks"*|*"shellcheck"*|*"run-tests"*) exit 0 ;;
  esac
  # Monitoring/probe commands
  case "$CMD" in
    *"tailscale status"*|*"access-matrix"*) exit 0 ;;
  esac
  # Setting/clearing the active ticket
  case "$CMD" in
    *active-ticket*) exit 0 ;;
  esac
 fi
 # For edit/write: exempt policy/hook files (these ARE the policy)
 FILE_PATH="${CRUSH_TOOL_INPUT_FILE_PATH:-}"
 case "$FILE_PATH" in
  */AGENTS.md|*/check-rules.sh|*/crush.json|*/hooks/*)
    if [ "$TOOL" = "write" ] || [ "$TOOL" = "edit" ] || [ "$TOOL" = "multiedit" ]; then
      exit 0
    fi
    ;;
 esac
 # Check for active ticket
 if [ -f "$TICKET_FILE" ] && [ -s "$TICKET_FILE" ]; then
  TICKET=$(cat "$TICKET_FILE")
  printf '{"context":"Active ticket: %s"}\n' "$TICKET"
  exit 0
 fi
 # No active ticket — block
 cat >&2 <<'MSG'
 TICKET GATE: No active ticket set.
 This project requires ticket-governed work (AGENTS.md Agent Authority).
 Before modifying systems or code, set the active ticket:
  echo '#NNN' > .crush/active-ticket
 If no ticket exists yet, create one first (redmine-cli create), then set it.
 Clear the ticket when work is complete:
  > .crush/active-ticket
 MSG
 exit 2
@@ -45,11 +45,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
  # ---- host reservations (fixed-address; not subject to failover pool) ----
-  host pfv-r3-mgmt {
+  host pfv-r3-tor-mgmt-01 {
    hardware ethernet 00:14:22:69:1c:37;
    fixed-address 192.168.0.7;
  }
-  host pfv-r3-stor {
+  host pfv-r3-tor-stor-01 {
    hardware ethernet 00:13:72:46:95:e4;
    fixed-address 192.168.0.9;
  }
@@ -57,11 +57,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
    hardware ethernet 40:9f:38:b0:b5:2f;
    fixed-address 192.168.1.84;
  }
-  host pfv-r2-tor1 {
+  host pfv-r2-tor-01 {
    hardware ethernet 00:0d:56:41:7a:4d;
    fixed-address 192.168.0.10;
  }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
    hardware ethernet a4:ba:db:6f:ce:28;
    fixed-address 192.168.0.12;
  }
@@ -177,11 +177,66 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
    hardware ethernet a4:bb:6d:e3:56:86;
    fixed-address 192.168.3.58;
  }
 	# umbrel
 	host tsys-umbrel {
 		hardware ethernet 02:2E:FF:8E:A2:D2;
 		fixed-address 192.168.1.97;
 		}
 	# ultix-streaming
 	host ultix-streaming {
 		hardware ethernet bc:24:11:1a:8f:6f;
 		fixed-address 192.168.3.78;
 		}
 	# ultix-offstage
 	host ultix-offstge {
 		hardware ethernet bc:24:11:1f:9d:83;
 		fixed-address 192.168.3.79;
 		}
 	# ultix-highside
 	host ultix-highside {
 		hardware ethernet a0:4a:5e:ca:46:f3;
 		fixed-address 192.168.3.32;
 		}
 	# pfv-k8s-cnode1
 	host pfv-k8s-cnode1 {
 		hardware ethernet bc:24:11:cb:97:10;
 		fixed-address 192.168.1.91;
 		}
 	# pfv-k8s-cnode2
 	host pfv-k8s-cnode2 {
 		hardware ethernet bc:24:11:40:25:f8;
 		fixed-address 192.168.3.113;
 		}
 	# pfv-k8s-cnode3
 	host pfv-k8s-cnode3 {
 		hardware ethernet bc:24:11:38:c0:58;
 		fixed-address 192.168.1.228;
 		}
 	# devbox-cloudron
 	host devbox-cloudron {
 		hardware ethernet bc:24:11:f7:b1:07;
 		fixed-address 192.168.1.6;
 		}
 	# hfnoc-uisp
 	host hfnoc-uisp {
 		hardware ethernet bc:24:11:a3:87:61;
 		fixed-address 192.168.3.193;
 		}
 	# kali-rd
 	host kali-rd {
 		hardware ethernet bc:24:11:9e:1c:e9;
 		fixed-address 192.168.2.37;
 		}
 	# kali-tsys
 	host kali-tsys {
 		hardware ethernet bc:24:11:16:22:d4;
 		fixed-address 192.168.1.114;
 		}
 }
 # ---- host declarations outside subnet (global scope, same as netboot) ----
-host subodev-torsw01 {
+host pfv-r6-mgmt-01 {
  hardware ethernet 00:14:22:69:18:a7;
  fixed-address 192.168.0.8;
 }
@@ -189,7 +244,80 @@ host pfv-r1-tor-top {
  hardware ethernet 00:23:ae:c1:ad:e8;
  fixed-address 192.168.0.11;
 }
-host tailscale-router {
+
-  hardware ethernet bc:24:11:8a:69:04;
+# --- VM DHCP reservations (generated 2026-08-11, ticket #420) ---
-  fixed-address 192.168.3.16;
+# All pinned to current ARP-observed IPs. No forward DNS needed
 # (forward records point to Tailscale 100.x addresses).
  host tsys-ca {
    hardware ethernet bc:24:11:32:d0:36;
    fixed-address 192.168.1.181;
  }
  host pfv-netinfra-01 {
    hardware ethernet bc:24:11:65:b2:ac;
    fixed-address 192.168.3.252;
  }
  host pfv-netinfra-02 {
    hardware ethernet bc:24:11:e4:37:53;
    fixed-address 192.168.3.253;
  }
  host tsys-librenms {
    hardware ethernet bc:24:11:5c:96:1e;
    fixed-address 192.168.3.176;
  }
  host tsys-proxmox-datacenter {
    hardware ethernet bc:24:11:e6:03:2d;
    fixed-address 192.168.2.44;
  }
  host pfv-k8s-wnode-tsys3 {
    hardware ethernet bc:24:11:ee:7e:7b;
    fixed-address 192.168.1.98;
  }
  host pfv-proxmox-backup-server {
    hardware ethernet bc:24:11:6e:12:69;
    fixed-address 192.168.2.193;
  }
  host pfv-k8s-wnode-tsys5 {
    hardware ethernet bc:24:11:c7:a8:6c;
    fixed-address 192.168.1.5;
  }
  host preprod-hfnoc-uisp {
    hardware ethernet bc:24:11:74:d6:8a;
    fixed-address 192.168.3.192;
  }
  host tsys-awx {
    hardware ethernet bc:24:11:80:0d:16;
    fixed-address 192.168.3.115;
  }
  host pfv-rr-middleware-02 {
    hardware ethernet bc:24:11:96:0e:ee;
    fixed-address 192.168.1.117;
  }
  host tsys-proxmox-mailgw-01 {
    hardware ethernet bc:24:11:56:61:18;
    fixed-address 192.168.1.11;
  }
  host pfv-k8s-wnode-tsys7 {
    hardware ethernet bc:24:11:30:b8:07;
    fixed-address 192.168.1.109;
  }
  host pfv-rr-middleware-01 {
    hardware ethernet bc:24:11:1e:61:cf;
    fixed-address 192.168.1.110;
  }
  host tsys-voip {
    hardware ethernet bc:24:11:23:ce:04;
    fixed-address 192.168.1.70;
  }
  host tsys-proxmox-mailgw-02 {
    hardware ethernet bc:24:11:5f:e5:2c;
    fixed-address 192.168.1.10;
  }
  host pfv-k8s-wnode-tsys6 {
    hardware ethernet bc:24:11:fa:6e:b5;
    fixed-address 192.168.1.111;
  }
  host tsys-siem-new {
    hardware ethernet bc:24:11:ee:67:e2;
    fixed-address 192.168.1.223;
  }
@@ -59,7 +59,7 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
    hardware ethernet 00:0d:56:41:7a:4d;
    fixed-address 192.168.0.10;
  }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
    hardware ethernet a4:ba:db:6f:ce:28;
    fixed-address 192.168.0.12;
  }
@@ -0,0 +1,25 @@
 ! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
 ! Switch: core-sw01 (Dell PowerConnect 5448, rack 5)
 ! WHEN: Friday maintenance window — BOTH switches must change together
 ! RISK: Brief storage-net outage during transition (seconds)
 !
 ! Current: g13-g16 in ch1, mode=on (static, no failure detection)
 ! Target:  g13-g16 in ch1, mode=auto (LACP active partner negotiation)
 !
 ! NOTE: core-sw01 hash stays layer-2-3 (hardware limit — best available)
 !       tor3-stor hash stays layer-2-3-4 (already set)
 !
 ! IMPORTANT: Run this SIMULTANEOUSLY with tor3-stor change.
 !   If one side is LACP and other is static, trunk goes down until
 !   both sides match. Plan for ~30s storage-net outage.
 !
 enable
 configure
 interface range ethernet g13-g16
  no channel-group
  channel-group 1 mode auto
 exit
 exit
 show interfaces status port-channel 1
 show lacp port-channel 1
 copy running-config startup-config
@@ -1,4 +1,4 @@
-! pfv-tor3-stor — MAC table + port details (Dell PowerConnect 5324 commands)
+! pfv-r3-tor-stor-01 — MAC table + port details (Dell PowerConnect 5324 commands)
 enable
 show bridge addressing-table address
 show interfaces description
@@ -1,4 +1,4 @@
-! pfv-tor3-stor — Neyland 24T (Radlan-based, rack 3 storage TOR)
+! pfv-r3-tor-stor-01 — Neyland 24T (Radlan-based, rack 3 storage TOR)
 ! Radlan CLI uses different keywords than DNOS
 enable
 show system
@@ -1,4 +1,4 @@
-! pfv-core-sw01 — MAC address table + LLDP neighbors
+! pfv-r5-core-01 — MAC address table + LLDP neighbors
 terminal datadump
 enable
 show mac-address-table
@@ -1,4 +1,4 @@
-! pfv-core-sw01 — Dell PowerConnect 5448 (core switch, rack 5)
+! pfv-r5-core-01 — Dell PowerConnect 5448 (core switch, rack 5)
 ! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down)
 terminal datadump
 enable
@@ -0,0 +1,24 @@
 ! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
 ! Switch: pfv-r3-tor-stor (Dell PowerConnect 5324 / "Neyland 24T", rack 3)
 ! WHEN: Friday maintenance window — BOTH switches must change together
 ! RISK: Brief storage-net outage during transition (seconds)
 !
 ! Current: g20-g23 in ch1, mode=on (static, no failure detection)
 ! Target:  g20-g23 in ch1, LACP active
 !
 ! Radlan CLI syntax (different from DNOS on core-sw01)
 ! Verify exact keywords on this switch first — 'show running-config'
 ! will show the current ch1 config to confirm syntax.
 !
 ! IMPORTANT: Run this SIMULTANEOUSLY with core-sw01 change.
 !
 enable
 configure
 interface range ethernet g20-g23
  no channel-group
  channel-group 1 mode active
 exit
 exit
 show interfaces status port-channel 1
 show lacp port-channel 1
 copy running-config startup-config
@@ -0,0 +1,41 @@
 # NetDisco deployment configuration [#337]
 # SNMP community: kn3lmgmt (v2c)
 # Database connection (matched to docker-compose services)
 database:
  host: netdisco-postgresql
  dbname: netdisco
  user: netdisco
  pass: netdisco
 # SNMP authentication
 snmp_auth:
  - tag: default_v2c
    community: kn3lmgmt
    read: true
    write: false
    version: 2
  - tag: default_v1
    community: kn3lmgmt
    read: true
    write: false
    version: 1
 # Job scheduling
 schedule:
  discoverall:
    first: 60
    every: 3600
  macwalk:
    first: 120
    every: 900
  arpwalk:
    first: 150
    every: 900
  nbtwalk:
    first: 180
    every: 900
 # Node freshness (keep history)
 node_freshness: 0
@@ -0,0 +1,44 @@
 services:
  netdisco-postgresql:
    image: netdisco/netdisco:2.102001-postgresql
    container_name: oam-netdisco-db
    environment:
      - POSTGRES_USER=netdisco
      - POSTGRES_PASSWORD=netdisco
      - POSTGRES_DB=netdisco
    volumes:
      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
    restart: unless-stopped
  netdisco-backend:
    image: netdisco/netdisco:2.102001-backend
    container_name: oam-netdisco-backend
    depends_on:
      - netdisco-postgresql
    environment:
      - NETDISCO_DB_HOST=netdisco-postgresql
      - NETDISCO_DB_NAME=netdisco
      - NETDISCO_DB_USER=netdisco
      - NETDISCO_DB_PASS=netdisco
      - NETDISCO_SNMP_COMMUNITY=kn3lmgmt
    volumes:
      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
    restart: unless-stopped
  netdisco-web:
    image: netdisco/netdisco:2.102001-web
    container_name: oam-netdisco-web
    depends_on:
      - netdisco-postgresql
      - netdisco-backend
    environment:
      - NETDISCO_DB_HOST=netdisco-postgresql
      - NETDISCO_DB_NAME=netdisco
      - NETDISCO_DB_USER=netdisco
      - NETDISCO_DB_PASS=netdisco
    ports:
      - "8082:5000"
    volumes:
      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
    restart: unless-stopped
@@ -0,0 +1,119 @@
 #!/usr/bin/env bash
 set -euo pipefail
 # Deploy NetDisco on tsys-librenms
 # Part of OAM platform [#337]
 # L2 network discovery and mapping via SNMP
 echo "=== Setting up NetDisco ==="
 mkdir -p /opt/oam/netdisco/{config,data,pgdata}
 # Write deployment config
 cat > /opt/oam/netdisco/config/deployment.yml <<'YMLEOF'
 # NetDisco deployment configuration [#337]
 # SNMP community: kn3lmgmt (v2c)
 database:
  host: netdisco-postgresql
  dbname: netdisco
  user: netdisco
  pass: netdisco
 snmp_auth:
  - tag: default_v2c
    community: kn3lmgmt
    read: true
    write: false
    version: 2
  - tag: default_v1
    community: kn3lmgmt
    read: true
    write: false
    version: 1
 schedule:
  discoverall:
    first: 60
    every: 3600
  macwalk:
    first: 120
    every: 900
  arpwalk:
    first: 150
    every: 900
  nbtwalk:
    first: 180
    every: 900
 node_freshness: 0
 YMLEOF
 # Write docker-compose
 cat > /opt/oam/netdisco/docker-compose.yml <<'DCEOF'
 services:
  netdisco-postgresql:
    image: netdisco/netdisco:2.102001-postgresql
    container_name: oam-netdisco-db
    environment:
      - POSTGRES_USER=netdisco
      - POSTGRES_PASSWORD=netdisco
      - POSTGRES_DB=netdisco
    volumes:
      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
    restart: unless-stopped
  netdisco-backend:
    image: netdisco/netdisco:2.102001-backend
    container_name: oam-netdisco-backend
    depends_on:
      - netdisco-postgresql
    environment:
      - NETDISCO_DB_HOST=netdisco-postgresql
      - NETDISCO_DB_NAME=netdisco
      - NETDISCO_DB_USER=netdisco
      - NETDISCO_DB_PASS=netdisco
    volumes:
      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
    restart: unless-stopped
  netdisco-web:
    image: netdisco/netdisco:2.102001-web
    container_name: oam-netdisco-web
    depends_on:
      - netdisco-postgresql
      - netdisco-backend
    environment:
      - NETDISCO_DB_HOST=netdisco-postgresql
      - NETDISCO_DB_NAME=netdisco
      - NETDISCO_DB_USER=netdisco
      - NETDISCO_DB_PASS=netdisco
    ports:
      - "8082:5000"
    volumes:
      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
    restart: unless-stopped
 DCEOF
 echo "=== Pulling NetDisco images ==="
 cd /opt/oam/netdisco && docker compose pull 2>&1
 echo "=== Starting NetDisco stack ==="
 cd /opt/oam/netdisco && docker compose up -d 2>&1
 echo "=== Waiting for PostgreSQL to initialize (30s) ==="
 sleep 30
 echo "=== Container status ==="
 docker ps --filter name=oam-netdisco --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
 echo "=== Adding devices to NetDisco ==="
 for dev in pfv-r5-core-01.knel.net pfv-r3-tor-mgmt-01.knel.net pfv-r3-tor-stor-01.knel.net pfv-r6-mgmt-01.knel.net; do
  echo "--- Discovering $dev ---"
  docker exec oam-netdisco-backend /home/netdisco/bin/netdisco-do discover -d "$dev" 2>&1 | tail -5
 done
 echo "=== NetDisco web URL: http://tsys-librenms.knel.net:8082 ==="
 echo "=== Done ==="
@@ -0,0 +1,38 @@
 ---
 username: 'oxidized'
 password: 'see ~/.creds/oxidized.env'
 model: powerconnect
 interval: 3600
 use_syslog: false
 debug: false
 threads: 30
 timeout: 20
 retries: 3
 prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
 rest: 0.0.0.0:8888
 next_adds_job: false
 models: {}
 pid: /home/oxidized/.config/oxidized/pid
 log: /home/oxidized/.config/oxidized/logs/log
 snmp: false
 groups: {}
 models: {}
 resolve_dns: true
 interval: 3600
 output:
  default: git
  git:
    user: oxidized
    email: oxidized@tsys-librenms.knel.net
    repo: /home/oxidized/.config/oxidized/git-repos
 source:
  default: csv
  csv:
    file: /home/oxidized/.config/oxidized/router.db
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
    gpg: false
@@ -0,0 +1,12 @@
 services:
  oxidized:
    image: oxidized/oxidized:0.30.1
    container_name: oam-oxidized
    environment:
      - OXIDIZED_CONFIG=/home/oxidized/.config/oxidized/config
      - TZ=America/Chicago
    volumes:
      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
    ports:
      - "8083:8888"
    restart: unless-stopped
@@ -0,0 +1,7 @@
 #!/usr/bin/env bash
 # Oxidized post-push hook: auto-push to Gitea after each config change
 # Mounted into container at /home/oxidized/.config/oxidized/hooks/checkout/git-push.sh
 set -euo pipefail
 cd "$1"
 export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new -i /home/oxidized/.config/oxidized/.ssh/id_ed25519"
 git push origin master 2>&1 || true
@@ -0,0 +1,4 @@
 pfv-r5-core-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
 pfv-r3-tor-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
 pfv-r3-tor-stor-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
 pfv-r6-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
@@ -0,0 +1,90 @@
 #!/usr/bin/env bash
 set -euo pipefail
 # Deploy Oxidized on tsys-librenms
 # Part of OAM platform [#337]
 # Network device config backup/versioning
 echo "=== Setting up Oxidized ==="
 mkdir -p /opt/oam/oxidized/{config,git-repos,logs}
 # Config file
 cat > /opt/oam/oxidized/config/config <<'CFGEOF'
 ---
 username: 'CHANGE_ME'
 password: 'CHANGE_ME'
 model: powerconnect
 interval: 3600
 use_syslog: false
 debug: false
 threads: 30
 timeout: 20
 retries: 3
 prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
 rest: 0.0.0.0:8888
 next_adds_job: false
 pid: /home/oxidized/.config/oxidized/pid
 log: /home/oxidized/.config/oxidized/logs/log
 snmp: false
 resolve_dns: true
 interval: 3600
 output:
  default: git
  git:
    user: oxidized
    email: oxidized@tsys-librenms.knel.net
    repo: /home/oxidized/.config/oxidized/git-repos
 source:
  default: csv
  csv:
    file: /home/oxidized/.config/oxidized/router.db
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
    gpg: false
 CFGEOF
 # Router DB (device list)
 cat > /opt/oam/oxidized/config/router.db <<'DBEOF'
 pfv-r5-core-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
 pfv-r3-tor-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
 pfv-r3-tor-stor-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
 pfv-r6-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
 DBEOF
 # Docker compose
 cat > /opt/oam/oxidized/docker-compose.yml <<'DCEOF'
 services:
  oxidized:
    image: oxidized/oxidized:0.30.1
    container_name: oam-oxidized
    environment:
      - TZ=America/Chicago
    volumes:
      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
    ports:
      - "8083:8888"
    restart: unless-stopped
 DCEOF
 echo "=== Pulling Oxidized image ==="
 cd /opt/oam/oxidized && timeout 120 docker compose pull 2>&1
 echo "=== Starting Oxidized ==="
 cd /opt/oam/oxidized && docker compose up -d 2>&1
 echo "=== Status ==="
 docker ps --filter name=oam-oxidized --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
 echo ""
 echo "=== IMPORTANT ==="
 echo "Oxidized deployed with placeholder credentials (CHANGE_ME)."
 echo "Update /opt/oam/oxidized/config/router.db with real switch credentials"
 echo "(username:password per device) to enable config backups."
 echo "Then restart: cd /opt/oam/oxidized && docker compose restart"
 echo ""
 echo "=== Oxidized REST API: http://tsys-librenms.knel.net:8083 ==="
@@ -0,0 +1,84 @@
 *** General ***
 owner = Charles N Wyble
 contact = reachableceo@knownelement.com
 mailhost = localhost
 cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
 syslogfacility = local0
 # each probe run every 5 minutes, 300 samples = 25 hours at 5min steps
 pagedir = /data/pages
 piddir  = /tmp
 imgcache = /data/cache
 imgurl  = /smokeping/images
 datadir = /data/_data
 dyndir = /data/dyn
 *** Database ***
 step     = 300
 pings    = 20
 # consfn mrhb steps total
 AVERAGE  0.5   1  100800
 AVERAGE  0.5  12  43200
    MIN  0.5  12  43200
    MAX  0.5  12  43200
 AVERAGE  0.5 144   7200
    MAX  0.5 144   7200
    MIN  0.5 144   7200
 *** Presentation ***
 template = /etc/smokeping/basepage.html
 + charts
 + detail
 width = 600
 height = 200
 nradius = 5
 *** Probes ***
 + FPing
 binary = /usr/local/bin/fping
 hostinterval = 1.5
 mininterval = 0.001
 offset = 0%
 *** Alerts ***
 to = reachableceo@knownelement.com
 from = smokeping-alert@tsys-librenms.knel.net
 +bigloss
 type = loss
 # in percent
 pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
 comment = suddenly there is packet loss
 +somedoss
 type = loss
 # in percent
 pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
 comment = incoming packet loss
 +startloss
 type = loss
 # in percent
 pattern = >0%,>0%,>0%
 comment = loss at startup
 +rttdetect
 type = rtt
 # in milliseconds
 pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
 comment = routing change?
 +rttbadd
 type = rtt
 # in milliseconds
 pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
 comment = reroute?
@@ -0,0 +1,111 @@
 *** Targets ***
 probe = FPing
 menu = Top
 title = Network Latency Monitor
 + Infrastructure
 menu = Infrastructure
 title = Core Infrastructure
 ++ pfv-r5-core-01
 menu = pfv-r5-core-01 (Core Switch)
 title = Dell PowerConnect 5448 - Rack 5 Core
 host = pfv-r5-core-01.knel.net
 ++ pfv-r3-tor-mgmt-01
 menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
 title = Neyland 24T - Rack 3 Management TOR
 host = pfv-r3-tor-mgmt-01.knel.net
 ++ pfv-r3-tor-stor-01
 menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
 title = Neyland 24T - Rack 3 Storage TOR
 host = pfv-r3-tor-stor-01.knel.net
 ++ pfv-r6-mgmt-01
 menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
 title = Neyland 24T - Rack 6 Management Switch
 host = pfv-r6-mgmt-01.knel.net
 + Servers
 menu = Servers
 title = Server Infrastructure
 ++ pfv-stor1
 menu = pfv-stor1
 title = Storage Server 1
 host = pfv-stor1.knel.net
 ++ pfv-bms
 menu = pfv-bms (HomeAssistant)
 title = Home Assistant / UniFi Controller
 host = pfv-bms.knel.net
 ++ tsys-librenms
 menu = tsys-librenms
 title = LibreNMS Monitoring Server
 host = tsys-librenms.knel.net
 + Wireless
 menu = Wireless
 title = Wireless Access Points
 ++ ap-tablemount
 menu = ap-tablemount
 title = UniFi AP-AC-Lite (Table Mount)
 host = ap-tablemount.knel.net
 ++ ap-wallmount
 menu = ap-wallmount
 title = UniFi AP-AC-LR (Wall Mount)
 host = ap-wallmount.knel.net
 + ProxmoxHosts
 menu = Proxmox Hosts
 title = Proxmox Hypervisor Nodes
 ++ pfv-tsys1
 menu = pfv-tsys1
 title = Proxmox Host - Infrastructure
 host = pfv-tsys1.knel.net
 ++ pfv-tsys3
 menu = pfv-tsys3
 title = Proxmox Host - Compute
 host = pfv-tsys3.knel.net
 ++ pfv-tsys4
 menu = pfv-tsys4
 title = Proxmox Host - Storage + PBS
 host = pfv-tsys4.knel.net
 ++ pfv-tsys5
 menu = pfv-tsys5
 title = Proxmox Host - Storage + Sandbox
 host = pfv-tsys5.knel.net
 ++ pfv-tsys6
 menu = pfv-tsys6
 title = Proxmox Host - RackRental
 host = pfv-tsys6.knel.net
 ++ pfv-tsys7
 menu = pfv-tsys7
 title = Proxmox Host - RackRental
 host = pfv-tsys7.knel.net
 ++ pfv-tsys9
 menu = pfv-tsys9
 title = Proxmox Host - Infra + Compute
 host = pfv-tsys9.knel.net
 + Power
 menu = Power
 title = Power Devices
 ++ pfv-garage-pdu-01
 menu = pfv-garage-pdu-1
 title = APC PDU (Garage)
 host = pfv-garage-pdu-1.knel.net
@@ -0,0 +1,14 @@
 services:
  smokeping:
    image: linuxserver/smokeping:2.9.0
    container_name: oam-smokeping
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - /opt/oam/smokeping/config:/config
      - /opt/oam/smokeping/data:/data
    ports:
      - "8081:80"
    restart: unless-stopped
@@ -0,0 +1,237 @@
 #!/usr/bin/env bash
 set -euo pipefail
 # Deploy Smokeping on tsys-librenms
 # Part of OAM platform [#337]
 echo "=== Setting up Smokeping ==="
 # Ensure directories
 mkdir -p /opt/oam/smokeping/{config,data}
 # General config
 cat > /opt/oam/smokeping/config/General <<'GENEOF'
 *** General ***
 owner = Charles N Wyble
 contact = reachableceo@knownelement.com
 mailhost = localhost
 cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
 syslogfacility = local0
 pagedir = /data/pages
 piddir  = /tmp
 imgcache = /data/cache
 imgurl  = /smokeping/images
 datadir = /data/_data
 dyndir = /data/dyn
 *** Database ***
 step     = 300
 pings    = 20
 AVERAGE  0.5   1  100800
 AVERAGE  0.5  12  43200
    MIN  0.5  12  43200
    MAX  0.5  12  43200
 AVERAGE  0.5 144   7200
    MAX  0.5 144   7200
    MIN  0.5 144   7200
 *** Presentation ***
 template = /etc/smokeping/basepage.html
 + charts
 + detail
 width = 600
 height = 200
 nradius = 5
 *** Probes ***
 + FPing
 binary = /usr/local/bin/fping
 hostinterval = 1.5
 mininterval = 0.001
 offset = 0%
 *** Alerts ***
 to = reachableceo@knownelement.com
 from = smokeping-alert@tsys-librenms.knel.net
 +bigloss
 type = loss
 pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
 comment = suddenly there is packet loss
 +somedoss
 type = loss
 pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
 comment = incoming packet loss
 +startloss
 type = loss
 pattern = >0%,>0%,>0%
 comment = loss at startup
 +rttdetect
 type = rtt
 pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
 comment = routing change?
 +rttbadd
 type = rtt
 pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
 comment = reroute?
 GENEOF
 # Targets config
 cat > /opt/oam/smokeping/config/Targets <<'TGTEOF'
 *** Targets ***
 probe = FPing
 menu = Top
 title = Network Latency Monitor
 + Infrastructure
 menu = Infrastructure
 title = Core Infrastructure
 ++ pfv-r5-core-01
 menu = pfv-r5-core-01 (Core Switch)
 title = Dell PowerConnect 5448 - Rack 5 Core
 host = pfv-r5-core-01.knel.net
 ++ pfv-r3-tor-mgmt-01
 menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
 title = Neyland 24T - Rack 3 Management TOR
 host = pfv-r3-tor-mgmt-01.knel.net
 ++ pfv-r3-tor-stor-01
 menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
 title = Neyland 24T - Rack 3 Storage TOR
 host = pfv-r3-tor-stor-01.knel.net
 ++ pfv-r6-mgmt-01
 menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
 title = Neyland 24T - Rack 6 Management Switch
 host = pfv-r6-mgmt-01.knel.net
 + Servers
 menu = Servers
 title = Server Infrastructure
 ++ pfv-stor1
 menu = pfv-stor1
 title = Storage Server 1
 host = pfv-stor1.knel.net
 ++ pfv-bms
 menu = pfv-bms (HomeAssistant)
 title = Home Assistant / UniFi Controller
 host = pfv-bms.knel.net
 ++ tsys-librenms
 menu = tsys-librenms
 title = LibreNMS Monitoring Server
 host = tsys-librenms.knel.net
 + Wireless
 menu = Wireless
 title = Wireless Access Points
 ++ ap-tablemount
 menu = ap-tablemount
 title = UniFi AP-AC-Lite (Table Mount)
 host = ap-tablemount.knel.net
 ++ ap-wallmount
 menu = ap-wallmount
 title = UniFi AP-AC-LR (Wall Mount)
 host = ap-wallmount.knel.net
 + ProxmoxHosts
 menu = Proxmox Hosts
 title = Proxmox Hypervisor Nodes
 ++ pfv-tsys1
 menu = pfv-tsys1
 title = Proxmox Host - Infrastructure
 host = pfv-tsys1.knel.net
 ++ pfv-tsys3
 menu = pfv-tsys3
 title = Proxmox Host - Compute
 host = pfv-tsys3.knel.net
 ++ pfv-tsys4
 menu = pfv-tsys4
 title = Proxmox Host - Storage + PBS
 host = pfv-tsys4.knel.net
 ++ pfv-tsys5
 menu = pfv-tsys5
 title = Proxmox Host - Storage + Sandbox
 host = pfv-tsys5.knel.net
 ++ pfv-tsys6
 menu = pfv-tsys6
 title = Proxmox Host - RackRental
 host = pfv-tsys6.knel.net
 ++ pfv-tsys7
 menu = pfv-tsys7
 title = Proxmox Host - RackRental
 host = pfv-tsys7.knel.net
 ++ pfv-tsys9
 menu = pfv-tsys9
 title = Proxmox Host - Infra + Compute
 host = pfv-tsys9.knel.net
 + Power
 menu = Power
 title = Power Devices
 ++ pfv-garage-pdu-01
 menu = pfv-garage-pdu-1
 title = APC PDU (Garage)
 host = pfv-garage-pdu-1.knel.net
 TGTEOF
 # Docker compose
 cat > /opt/oam/smokeping/docker-compose.yml <<'DCEOF'
 services:
  smokeping:
    image: linuxserver/smokeping:2.9.0
    container_name: oam-smokeping
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - /opt/oam/smokeping/config:/config
      - /opt/oam/smokeping/data:/data
    ports:
      - "8081:80"
    restart: unless-stopped
 DCEOF
 echo "=== Starting Smokeping container ==="
 cd /opt/oam/smokeping && docker compose up -d 2>&1
 echo "=== Waiting for startup ==="
 sleep 5
 echo "=== Container status ==="
 docker ps --filter name=oam-smokeping --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
 echo "=== Smokeping logs (last 20) ==="
 docker logs oam-smokeping --tail 20 2>&1
 echo "=== Done ==="
@@ -0,0 +1,83 @@
 #!/usr/bin/env bash
 set -euo pipefail
 # Deploy UNPoller on tsys-librenms
 # Part of OAM platform [#337]
 # UniFi controller polling for metrics export
 echo "=== Setting up UNPoller ==="
 mkdir -p /opt/oam/unpoller
 # Write config
 cat > /opt/oam/unpoller/up.conf <<'CONFEOF'
 [poller]
 debug = false
 quiet = false
 plugins = []
 [unifi.defaults]
 url = "https://pfv-bms.knel.net:8443"
 user = "CHANGE_ME"
 pass = "CHANGE_ME"
 sites = ["all"]
 ssl_verify = false
 [unifi.defaults.save_dpi]
 enabled = true
 [influxdb]
 enable = false
 url = "http://tsys-librenms.knel.net:8086"
 db = "unifi"
 [datadog]
 enable = false
 [prometheus]
 enable = true
 http_listen = ":9130"
 report_errors = true
 [loki]
 enable = false
 CONFEOF
 # Docker compose
 cat > /opt/oam/unpoller/docker-compose.yml <<'DCEOF'
 services:
  unpoller:
    image: ghcr.io/unpoller/unpoller:latest
    container_name: oam-unpoller
    environment:
      - UP_UNIFI_DEFAULT_URL=https://pfv-bms.knel.net:8443
      - UP_UNIFI_DEFAULT_USER=CHANGE_ME
      - UP_UNIFI_DEFAULT_PASS=CHANGE_ME
      - UP_UNIFI_DEFAULT_VERIFY_SSL=false
      - UP_PROMETHEUS_ENABLE=true
      - UP_PROMETHEUS_HTTP_LISTEN=:9130
      - TZ=America/Chicago
    ports:
      - "8084:9130"
    volumes:
      - /opt/oam/unpoller/up.conf:/config/unifi-poller.conf:ro
    restart: unless-stopped
 DCEOF
 echo "=== Pulling UNPoller image ==="
 cd /opt/oam/unpoller && timeout 120 docker compose pull 2>&1
 echo "=== Starting UNPoller ==="
 cd /opt/oam/unpoller && docker compose up -d 2>&1
 echo "=== Status ==="
 docker ps --filter name=oam-unpoller --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
 echo ""
 echo "=== IMPORTANT ==="
 echo "UNPoller deployed with placeholder UniFi credentials (CHANGE_ME)."
 echo "Update UP_UNIFI_DEFAULT_USER and UP_UNINI_DEFAULT_PASS env vars"
 echo "in /opt/oam/unpoller/docker-compose.yml with real UniFi controller"
 echo "credentials, then restart: cd /opt/oam/unpoller && docker compose up -d"
 echo ""
 echo "=== UNPoller Prometheus metrics: http://tsys-librenms.knel.net:8084 ==="
@@ -0,0 +1,57 @@
 WIDTH 800
 HEIGHT 600
 HTMLOUTPUTFILE output/network.html
 IMAGEOUTPUTFILE output/network.png
 BACKGROUNDCOLOR 255 255 255
 TITLE "PFV Cluster Network Map"
 TIMEPOS 0 0
 TITLECOLOR 0 0 0
 TIMECOLOR 0 0 0
 HTMLSTYLE overlib
 KEYPOS DEFAULT 400 580
 KEYTEXTCOLOR 0 0 0
 KEYOUTLINECOLOR 0 0 0
 KEYBGCOLOR 255 255 255
 SCALE DEFAULT 0 10 0 255 0
 SCALE DEFAULT 10 50 255 255 0
 SCALE DEFAULT 50 100 255 128 0
 SCALE DEFAULT 100 1000 255 0 0
 NODE core_sw01
  LABEL pfv-r5-core-01
  POSITION 400 200
 NODE tor_mgmt
  LABEL pfv-r3-tor-mgmt-01
  POSITION 200 350
 NODE tor_stor
  LABEL pfv-r3-tor-stor-01
  POSITION 400 350
 NODE r6_mgmt
  LABEL pfv-r6-mgmt-01
  POSITION 600 350
 NODE pfv_tsys1
  LABEL pfv-tsys1
  POSITION 200 100
 NODE pfv_tsys9
  LABEL pfv-tsys9
  POSITION 600 100
 NODE pfv_bms
  LABEL pfv-bms
  POSITION 100 200
 LINK core_to_mgmt
  NODES core_sw01 tor_mgmt
  BANDWIDTH 1000M
 LINK core_to_stor
  NODES core_sw01 tor_stor
  BANDWIDTH 1000M
 LINK core_to_r6
  NODES core_sw01 r6_mgmt
  BANDWIDTH 1000M
 LINK tsys1_to_core
  NODES pfv_tsys1 core_sw01
  BANDWIDTH 1000M
 LINK tsys9_to_core
  NODES pfv_tsys9 core_sw01
  BANDWIDTH 1000M
 LINK bms_to_core
  NODES pfv_bms core_sw01
  BANDWIDTH 1000M
@@ -288,6 +288,6 @@ echo "===== CONTEXT ====="
 echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
 echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
 echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
-echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
+echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor-01 → pfv-r5-core-01)"
 echo ""
 echo "All logs in: $LOG_DIR/"
@@ -22,7 +22,7 @@ any network with a conman server:
 Usage:
    CONMAN_SERVER=console-host:7890 \\
-    python3 conman-console.py --console pfv-core-sw01 --cmds switches/pfv-core-sw01.cmds
+    python3 conman-console.py --console pfv-r5-core-01 --cmds switches/pfv-r5-core-01.cmds
 Lines starting with '!' or '#' in the cmds file are comments (skipped).
 Blank lines are skipped. The conman escape sequence (&.) is sent automatically
@@ -51,7 +51,7 @@ def main():
    ap = argparse.ArgumentParser(
        description="Drive a conman console session read-only via PTY")
    ap.add_argument("--console", required=True,
-                    help="console name (e.g. pfv-core-sw01)")
+                    help="console name (e.g. pfv-r5-core-01)")
    ap.add_argument("--cmds", required=True,
                    help="command file (one command per line; !/# = comment)")
    ap.add_argument("--server",
@@ -1,32 +1,36 @@
 #!/usr/bin/bash
-# Deploy tuned inside VMs via qm guest exec (no SSH needed)
+# Deploy tuned inside VMs over SSH (sshd is the only approved access channel —
-# Uses Proxmox qemu-guest-agent for out-of-band command execution
+# see AGENTS.md "Access-channel policy: SSH only"). Formerly used the
 # qemu guest-agent channel; converted to SSH now that all VMs have key + sudo.
 set -uo pipefail
 # Profile mapping: most VMs get throughput-performance, k8s/network-sensitive get network-latency
 PROFILE="${1:-throughput-performance}"
 REMOTE_SH="${REMOTE_SH:-$(cd "$(dirname "$0")/../../.." && pwd)/tests/remote.sh}"
 VM_USER="${VM_USER:-localuser}"
 deploy_vm() {
  local host="$1" vmid="$2" name="$3"
  # name is the Tailscale hostname (SSH target); vmid retained for reference.
  echo -n "  VMID $vmid ($name) on $host: "
-  PROX_HOST=$host bash tests/remote.sh prox "
+  local result
-    # Check if guest agent is available
+  result=$(VM_IP="$name" VM_USER="$VM_USER" bash "$REMOTE_SH" vmroot \
-    if ! qm guest cmd \$vmid ping >/dev/null 2>&1; then
+    "DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; \
-      echo 'NO GUEST AGENT - skip'
+     DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; \
-      exit 0
+     tuned-adm profile $PROFILE 2>/dev/null; \
-    fi
+     systemctl enable --now tuned 2>/dev/null; \
-    # Install tuned
+     tuned-adm active 2>/dev/null" </dev/null 2>&1)
-    result=\$(qm guest exec \$vmid --timeout 120 -- /bin/sh -c 'DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; tuned-adm profile $PROFILE 2>/dev/null; systemctl enable tuned 2>/dev/null; systemctl restart tuned 2>/dev/null; tuned-adm active 2>/dev/null' 2>/dev/null)
+  if echo "$result" | grep -q 'Current active'; then
-    if echo \"\$result\" | grep -q 'Current active'; then
+    echo "$result" | grep -o 'Current active.*' | head -1
-      echo \"\$result\" | grep -o 'Current active.*' | head -1
+  elif echo "$result" | grep -qi 'permission denied\|no route\|timed out'; then
    echo "SSH FAILED (no key/no sudo) — run bootstrap-all.sh first"
  else
-      echo 'INSTALL FAILED (apt issue or no network)'
+    echo "INSTALL FAILED (apt issue or no network)"
  fi
  " 2>&1
 }
 echo "============================================"
-echo "  Deploying tuned ($PROFILE) to VMs"
+echo "  Deploying tuned ($PROFILE) to VMs via SSH"
 echo "============================================"
 echo ""
@@ -55,7 +59,7 @@ deploy_vm pfv-tsys6 604 tsys-proxmox-mailgw-01
 echo "--- pfv-tsys7 ---"
 deploy_vm pfv-tsys7 701 pfv-k8s-wnode-tsys7
 deploy_vm pfv-tsys7 702 hfnoc-uisp
-deploy_vm pfv-tsys7 703 rr-middleware-01
+deploy_vm pfv-tsys7 703 pfv-rr-middleware-01
 deploy_vm pfv-tsys7 705 pfv-k8s-cnode2
 deploy_vm pfv-tsys7 706 kali-rd
 deploy_vm pfv-tsys7 707 tsys-siem
@@ -0,0 +1,79 @@
 #!/bin/bash
 ###############################################################################
 # fix-e1000e-offload.sh — Disable offload on e1000e NICs to prevent hangs
 #
 # The Intel e1000e driver has a known bug where TSO/GSO/GRO offload causes
 # "hardware unit hang" resets on certain Intel NICs (I217-LM, I219-LM, 82571EB).
 # This script disables offload features on all e1000e interfaces and installs
 # a systemd service to persist across reboots.
 #
 # References:
 #   https://forum.proxmox.com/threads/e1000-driver-hang.58284/
 #   https://serverfault.com/questions/616485
 #
 # Run on each Proxmox host:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file proxmox/perf/scripts/fix-e1000e-offload.sh
 ###############################################################################
 set -euo pipefail
 echo "=== e1000e Offload Fix on $(hostname) ==="
 # Find all e1000e physical NICs (skip bridges, bonds, virtual interfaces)
 AFFECTED_NICS=()
 for nic_path in /sys/class/net/*; do
    nic=$(basename "$nic_path")
    [ "$nic" = "lo" ] && continue
    # Skip bridges, bonds, virtual interfaces
    [ -d "${nic_path}/bridge" ] && continue
    [ -d "${nic_path}/bonding" ] && continue
    case "$nic" in
        tap*|veth*|fwpr*|fwln*|vmbr*|datanet*|storagenet*|tailscale*) continue ;;
    esac
    driver=$(ethtool -i "$nic" 2>/dev/null | awk '/^driver:/{print $2}')
    if [ "$driver" = "e1000e" ]; then
        AFFECTED_NICS+=("$nic")
        echo "  Found e1000e NIC: $nic"
    fi
 done
 if [ "${#AFFECTED_NICS[@]}" -eq 0 ]; then
    echo "  No e1000e NICs found. Nothing to do."
    exit 0
 fi
 # Apply fix live
 echo ""
 echo "--- Disabling offload features ---"
 for nic in "${AFFECTED_NICS[@]}"; do
    echo "  $nic:"
    ethtool -K "$nic" tso off gro off gso off tx off rx off 2>&1 | sed 's/^/    /' || true
    tso_state=$(ethtool -k "$nic" 2>/dev/null | awk '/tcp-segmentation-offload/{print $2}' | head -1)
    echo "    tso=$tso_state"
 done
 # Install systemd service for persistence
 echo ""
 echo "--- Installing systemd service ---"
 {
    echo "[Unit]"
    echo "Description=Disable offload on e1000e NICs (prevent hardware unit hang)"
    echo "After=network.target"
    echo "Wants=network.target"
    echo ""
    echo "[Service]"
    echo "Type=oneshot"
    echo "RemainAfterExit=yes"
    for nic in "${AFFECTED_NICS[@]}"; do
        echo "ExecStart=/sbin/ethtool -K $nic tso off gro off gso off tx off rx off"
    done
    echo ""
    echo "[Install]"
    echo "WantedBy=multi-user.target"
 } > /etc/systemd/system/fix-e1000e-offload.service
 systemctl daemon-reload
 systemctl enable fix-e1000e-offload.service
 echo "  Service installed and enabled (fix-e1000e-offload.service)"
 echo ""
 echo "=== Done. Affected NICs: ${AFFECTED_NICS[*]} ==="
@@ -0,0 +1,242 @@
 #!/bin/bash
 ###############################################################################
 # perf-matrix.sh — Any-to-any performance testing across the PFVCluster
 #
 # Tests three network planes:
 #   A. Datanet (VLAN 1000): hypervisor-to-hypervisor over storage network
 #   B. Guest-to-guest: k8s/ultix VMs over datanet (10.100.100.x)
 #   C. Storage I/O: dd read/write to NFS mounts
 #
 # Prerequisites:
 #   - iperf3 installed on all hosts (systemd service: iperf3-server)
 #   - iperf3 installed inside guest VMs
 #   - SSH key + passwordless sudo on all guest VMs (remote.sh; sshd is the
 #     only approved access channel — see AGENTS.md)
 #
 # Usage:
 #   bash perf-matrix.sh                  # run all tests
 #   bash perf-matrix.sh datanet          # host-to-host datanet only
 #   bash perf-matrix.sh guests           # guest-to-guest datanet only
 #   bash perf-matrix.sh storage          # NFS I/O only
 #
 # Environment:
 #   REMOTE_SH   path to tests/remote.sh (auto-detected)
 ###############################################################################
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REMOTE_SH="${REMOTE_SH:-$(cd "$SCRIPT_DIR/../../.." && pwd)/tests/remote.sh}"
 LOG_DIR="$(cd "$SCRIPT_DIR/.." && pwd)/returned-logs"
 mkdir -p "$LOG_DIR"
 TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
 # --- Host datanet IPs (VLAN 1000) ---
 declare -A DATANET_IP
 DATANET_IP[pfv-tsys1]="10.100.100.1"
 DATANET_IP[pfv-tsys3]="10.100.100.3"
 DATANET_IP[pfv-tsys4]="10.100.100.4"
 DATANET_IP[pfv-tsys5]="10.100.100.5"
 DATANET_IP[pfv-tsys6]="10.100.100.6"
 DATANET_IP[pfv-tsys7]="10.100.100.7"
 DATANET_IP[pfv-tsys9]="10.100.100.9"
 HOSTS="pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9"
 # --- k8s/ultix VM targets (SSH for control; datanet_ip for iperf traffic) ---
 # Format: prox_host:vmid:datanet_ip:ssh_host:label
 GUEST_TARGETS="
 pfv-tsys1:102:10.100.100.10:pfv-k8s-cnode1:cnode1
 pfv-tsys7:705:10.100.100.11:pfv-k8s-cnode2:cnode2
 pfv-tsys6:603:10.100.100.12:pfv-k8s-cnode3:cnode3
 pfv-tsys3:313:10.100.100.13:pfv-k8s-wnode-tsys3:wnode-tsys3
 pfv-tsys5:500:10.100.100.14:pfv-k8s-wnode-tsys5:wnode-tsys5
 pfv-tsys6:601:10.100.100.15:pfv-k8s-wnode-tsys6:wnode-tsys6
 pfv-tsys7:701:10.100.100.16:pfv-k8s-wnode-tsys7:wnode-tsys7
 pfv-tsys9:905:10.100.100.17:pfv-k8s-wnode-tsys9:wnode-tsys9
 pfv-tsys5:5111:10.100.100.18:ultix-streaming:ultix-streaming
 pfv-tsys5:5112:10.100.100.19:ultix-offstage:ultix-offstage
 "
 DURATION="${DURATION:-3}"    # seconds per iperf3 test
 STREAMS="${STREAMS:-4}"      # parallel streams
 # ============================================================================
 # Helpers
 # ============================================================================
 start_iperf_servers() {
    echo "--- Starting iperf3 servers on all hosts ---"
    for h in $HOSTS; do
        PROX_HOST="$h" bash "$REMOTE_SH" prox \
            'systemctl start iperf3-server 2>/dev/null || iperf3 -s -D; echo ok' \
            >/dev/null 2>&1 &
    done
    wait
    echo "    All servers started."
 }
 stop_iperf_servers() {
    echo "--- Stopping iperf3 servers on all hosts ---"
    for h in $HOSTS; do
        PROX_HOST="$h" bash "$REMOTE_SH" prox \
            'systemctl stop iperf3-server 2>/dev/null; pkill iperf3 2>/dev/null; true' \
            >/dev/null 2>&1 &
    done
    wait
    echo "    All servers stopped."
 }
 # ============================================================================
 # A. Host-to-host datanet matrix
 # ============================================================================
 test_datanet() {
    echo ""
    echo "==================================================================="
    echo "  DATANET (VLAN 1000) — Host-to-Host Bandwidth Matrix"
    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
    echo "==================================================================="
    start_iperf_servers
    local outfile="$LOG_DIR/datanet-host-${TIMESTAMP}.csv"
    echo "host_from,host_to,mbps" > "$outfile"
    for client in $HOSTS; do
        for server in $HOSTS; do
            [ "$client" = "$server" ] && continue
            local sip="${DATANET_IP[$server]}"
            local result
            result=$(PROX_HOST="$client" bash "$REMOTE_SH" prox \
                "iperf3 -c $sip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
                | awk '/SUM.*receiver/{printf "%.0f", $6}')
            if [ -n "$result" ]; then
                printf "  %-14s → %-14s : %s Mbps\n" "$client" "$server" "$result"
                echo "$client,$server,$result" >> "$outfile"
            else
                printf "  %-14s → %-14s : FAIL\n" "$client" "$server"
                echo "$client,$server,FAIL" >> "$outfile"
            fi
        done
    done
    stop_iperf_servers
    echo ""
    echo "  Results saved: $outfile"
 }
 # ============================================================================
 # B. Guest-to-guest datanet
 # ============================================================================
 test_guests() {
    echo ""
    echo "==================================================================="
    echo "  DATANET (VLAN 1000) — Guest-to-Guest (k8s + ultix VMs)"
    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
    echo "==================================================================="
    # Start iperf3 server on the first guest (cnode1)
    local server_entry
    server_entry=$(echo "$GUEST_TARGETS" | head -2 | tail -1)
    local s_ip s_ssh s_label
    s_ip=$(echo "$server_entry" | cut -d: -f3)
    s_ssh=$(echo "$server_entry" | cut -d: -f4)
    s_label=$(echo "$server_entry" | cut -d: -f5)
    echo "  Starting iperf3 server on $s_label ($s_ip)..."
    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
        'pkill iperf3 2>/dev/null; iperf3 -s -D' >/dev/null 2>&1
    sleep 1
    local outfile="$LOG_DIR/datanet-guest-${TIMESTAMP}.csv"
    echo "guest_from,guest_to,mbps" > "$outfile"
    while read -r entry; do
        [ -z "$entry" ] && continue
        local c_ip c_ssh c_label
        c_ip=$(echo "$entry" | cut -d: -f3)
        c_ssh=$(echo "$entry" | cut -d: -f4)
        c_label=$(echo "$entry" | cut -d: -f5)
        [ "$c_ip" = "$s_ip" ] && continue
        local result
        result=$(VM_IP="$c_ssh" bash "$REMOTE_SH" vmroot \
            "iperf3 -c $s_ip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
            | awk '/SUM.*receiver/{printf "%.0f", $6}')
        if [ -n "$result" ]; then
            printf "  %-18s → %-18s : %s Mbps\n" "$c_label" "$s_label" "$result"
            echo "$c_label,$s_label,$result" >> "$outfile"
        else
            printf "  %-18s → %-18s : FAIL\n" "$c_label" "$s_label"
            echo "$c_label,$s_label,FAIL" >> "$outfile"
        fi
    done <<< "$GUEST_TARGETS"
    # Cleanup
    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
        'pkill iperf3' >/dev/null 2>&1
    echo ""
    echo "  Results saved: $outfile"
 }
 # ============================================================================
 # C. Storage I/O (NFS read/write)
 # ============================================================================
 test_storage() {
    echo ""
    echo "==================================================================="
    echo "  STORAGE I/O — NFS Read/Write (100MB dd)"
    echo "==================================================================="
    local outfile="$LOG_DIR/storage-io-${TIMESTAMP}.csv"
    echo "host,mount,write_mbps,read_mbps" > "$outfile"
    local ddscript="/tmp/perf-dd-$$.sh"
    cat > "$ddscript" <<'DDSCRIPT'
 #!/bin/bash
 nfs=$(mount | awk '/type nfs/{print $3}' | grep -v proc)
 for m in $nfs; do
    tf="$m/.perf-$$"
    w=$(dd if=/dev/zero of="$tf" bs=1M count=100 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
    r=$(dd if="$tf" of=/dev/null bs=1M 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
    rm -f "$tf" 2>/dev/null
    echo "$m write=${w:-FAIL}MB/s read=${r:-N/A}MB/s"
 done
 DDSCRIPT
    for h in $HOSTS; do
        echo ""
        echo "  --- $h ---"
        PROX_HOST="$h" bash "$REMOTE_SH" prox-file "$ddscript" 2>&1 | while read -r line; do
            [ -n "$line" ] && echo "    $line"
        done
    done
    rm -f "$ddscript"
    echo ""
    echo "  Results saved: $outfile"
 }
 # ============================================================================
 # Main
 # ============================================================================
 MODE="${1:-all}"
 case "$MODE" in
    datanet|a)  test_datanet ;;
    guests|b)   test_guests ;;
    storage|c)  test_storage ;;
    all|"")     test_datanet; test_guests; test_storage ;;
    *)
        echo "Usage: $0 [datanet|guests|storage|all]"
        echo ""
        echo "  datanet  — host-to-host bandwidth matrix over VLAN 1000"
        echo "  guests   — guest-to-guest (k8s/ultix VMs over VLAN 1000)"
        echo "  storage  — NFS read/write I/O"
        echo "  all      — run all three (default)"
        exit 1
        ;;
 esac
 echo ""
 echo "==================================================================="
 echo "  Perf testing complete. Logs in: $LOG_DIR/"
 echo "==================================================================="
@@ -52,3 +52,10 @@ Preseving tokens/quota burn is a HUGE priority. It lets me and you do far more w
 - **Synthesized to:** —
 No more JOURNAL.md . Redmine is the system of record. JOURNAL.md was a hack I was using until redmine integration was in place. And, yes, discourse can also be used as well. Its a bit of a tricky decision, what should go to redmine vs discourse. I usually keep working notes/evolving status etc in Redmine and then synthesize to Discourse. But thats me as a lowly human :) You figure it out as you go and per project.
 ### Q6. RackRental split-network design on pfv-tsys6/7 [#433]
 - **Context:** TSYS6 and TSYS7 need trunk ports for RackRental. Servers will run a "split network configuration — half on the non-RackRental switch/router side, half on the RackRental switch/router side." RackRental router prevents network loops.
 - **Question:** Is the split per-NIC (one physical NIC to normal network, one to RackRental) or per-VM (some VMs bridge to RackRental, some to normal)? Both TSYS6/7 currently have bond0 with 2 slaves for storage network. How should the bonds be reconfigured?
 - **Answer:** Keep the storage bonds. We need that bandwidth. I believe tsys6,7 have six ethernet ports each? Two on a riser card, and a four port add-on? Or maybe its four total? If four total, the 2 for storage, one for mgmt non rackrental, one trunked to rackrental would be how I want to go.
 - **Decision:** _(human/agent)_
 - **Synthesized to:** —
@@ -0,0 +1,53 @@
 # questions-v2.md
 > Git-tracked question log. The agent writes; the human reviews/edits inline.
 > Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine.
 > See BASELINE-PROMPT.md §9.
 ## Resolved (from v1)
 ### Q1. Git remote for meta? — RESOLVED
 - **Decision:** New Gitea repo under TSYSGroupCorporate org: `TSYSGroupAIOS`. Template repo. Created and pushed.
 - **Synthesized to:** Meta repo created, `tea` + `docker login` noted as available on workstations.
 ### Q2. The bin/ wrapper gap — RESOLVED
 - **Decision:** No bin/ wrappers. Invoke real containers with `docker run --env-file`.
 - **Synthesized to:** PFVCluster AGENTS.md §CLI access; meta AGENTS.md §CLI invocation.
 ### Q3. Discourse pointer-header pattern — RESOLVED
 - **Decision:** Mandatory for ALL projects. No exceptions. Discourse/Redmine required.
 - **Synthesized to:** Meta AGENTS.md, check-rules.sh enforces.
 ### Q4. Sub-agent nudge hook — RESOLVED
 - **Decision:** No harness-specific hooks. Move to portable git hooks + AGENTS.md protocols. User shifting away from Crush to OpenWebUI/Hermes.
 - **Synthesized to:** All hooks ported to git hooks. No Crush-specific dependencies.
 ### Q5. JOURNAL.md vs Discourse — RESOLVED
 - **Decision:** No more JOURNAL.md. Redmine is SoR for work. Discourse for synthesized knowledge.
 - **Synthesized to:** Meta AGENTS.md, PFVCluster AGENTS.md.
 ### Q6. RackRental split-network design — RESOLVED
 - **Decision:** Keep storage bonds. If 4 NICs total: 2 storage, 1 mgmt non-RackRental, 1 trunked to RackRental VLAN 2.
 - **Synthesized to:** #433 (trunk ports), #432 (VLAN config).
 ## Open questions
 ### Q7. Close #167 (Small UPS Units Re-build)?
 - **Context:** #167 is an old (Dec 2024) vague ticket about rebuilding garage UPS units. It's been superseded by #372 (Tripp Lite physical reseat) and #439 (UPS data integration). Dates are stale (Aug 2025).
 - **Question:** Should we close #167 as superseded?
 - **Answer:** _(human)_
 - **Decision:** _(human/agent)_
 - **Synthesized to:** —
 ### Q8. Friday onsite work order?
 - **Context:** Friday 2026-08-14 onsite batch. Tickets: #341 (TEMPer deploy), #372 (Tripp Lite UPS reseat), #431 (Cisco fan replacement), #414 (tsys5 nic2 cable swap), #374 (PDU outlet renaming), #439 (UPS integration).
 - **Question:** What order do you want to tackle these? Cisco fan (#431) blocks #432/#433/#434/#436 — should it go first? TEMPer is the "fun" project — save for last?
 - **Answer:** _(human)_
 - **Decision:** _(human/agent)_
 - **Synthesized to:** —
 ### Q9. Session-start gate: relax for continuation sessions?
 - **Context:** You mentioned last session wanting to discuss relaxing the ticket gate for handoff/continuation sessions. The scope-alignment gate now handles this (presents handoff priorities, waits for user to name target). Is this sufficient, or do you want further changes?
 - **Answer:** _(human)_
 - **Decision:** _(human/agent)_
 - **Synthesized to:** —
@@ -157,7 +157,7 @@ while IFS= read -r -d '' f; do
        POINTER_MISSING=$((POINTER_MISSING + 1))
        $RULE_VERBOSE && printf '    %s\n' "$f"
    fi
-done < <(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -name '*.md' -print0 2>/dev/null)
+done < <(find . -path ./.git -prune -o -path ./.crush -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -name '*.md' -print0 2>/dev/null)
 if [ "$POINTER_MISSING" -eq 0 ]; then
    check "All non-exempt .md cite Discourse ($DISCOURSE_HOST)" "pass"
 else
@@ -242,4 +242,25 @@ if [ "$RULE_FAST" = false ] && [ -x scripts/test.sh ]; then
    fi
 fi
 # ----------------------------------------------------------------------------
 # 11. Access-channel policy — qemu guest-agent must NEVER be used as an
 #     execution or key-delivery channel. SSH (sshd) is the ONLY approved
 #     remote access path; all commands must be logged through standard
 #     auth/audit infrastructure. ITAR/CMMC environment — non-negotiable.
 #     Allowed: installing/checking qemu-guest-agent for Proxmox state
 #     visibility. Forbidden: `qm guest exec` + any `vm-guest` wrapper.
 #     Scans CODE only (not .md) so docs may describe the ban.
 # ----------------------------------------------------------------------------
 $RULE_VERBOSE && log_step "Access-channel policy (no guest-agent exec)"
 GA_HITS="$(grep -rnE 'qm guest exec|vm-guest|_vm_guest' \
    --include='*.sh' --include='*.bash' --include='*.py' \
    . 2>/dev/null | grep -vE 'scripts/check-rules\.sh|/archive/' || true)"
 if [ -z "$GA_HITS" ]; then
    check "No guest-agent exec / vm-guest access patterns in code" "pass"
 else
    GA_COUNT="$(printf '%s\n' "$GA_HITS" | grep -c . || true)"
    $RULE_VERBOSE && printf '%s\n' "$GA_HITS" | sed 's/^/    /'
    check "${GA_COUNT} guest-agent exec / vm-guest reference(s) — SSH-only access policy (AGENTS.md)" "fail"
 fi
 print_summary_and_exit
@@ -30,8 +30,6 @@ PROX_HOST="${PROX_HOST:-pfv-tsys5}"
 PROX_USER="${PROX_USER:-root}"
 VM_IP="${VM_IP:-192.168.3.50}"
 VM_USER="${VM_USER:-localuser}"
 VM_ID="${VM_ID:-}"
 GUEST_TIMEOUT="${GUEST_TIMEOUT:-900}"
 SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
@@ -55,33 +53,6 @@ _copy() {
  fi
 }
 # Out-of-band VM access via the Proxmox qemu-guest-agent. This runs commands
 # as root inside the VM and does NOT depend on SSH, so it works even after
 # secharden-ssh replaces authorized_keys and secharden-2fa enforces
 # publickey+keyboard-interactive (which blocks non-interactive SSH).
 GUEST_PARSER="/root/.knel-guest-parse.py"
 GUEST_PARSER_SRC="import sys, json
 try:
    d = json.load(sys.stdin)
 except Exception:
    sys.exit(3)
 sys.stdout.write(d.get('out-data', '') or '')
 sys.stderr.write(d.get('err-data', '') or '')
 ec = d.get('exitcode', 1)
 sys.exit(ec if ec is not None else 1)"
 _ensure_guest_parser() {
  if _prox "test -f '$GUEST_PARSER'" >/dev/null 2>&1; then return 0; fi
  printf '%s\n' "$GUEST_PARSER_SRC" | _prox "cat > '$GUEST_PARSER'" >/dev/null 2>&1
 }
 _vm_guest() {
  [ -n "$VM_ID" ] || die "vm-guest requires VM_ID"
  _ensure_guest_parser
  local cmdb64; cmdb64="$(printf '%s' "$*" | base64 -w0)"
  _prox "qm guest exec $VM_ID --timeout ${GUEST_TIMEOUT} -- /bin/sh -c 'echo $cmdb64 | base64 -d | /bin/sh' 2>/dev/null | python3 '$GUEST_PARSER'"
 }
 mode="${1:-}"; shift || true
 case "$mode" in
  prox)         [ "$#" -ge 0 ] || die "need command"; _prox "$*" ;;
@@ -91,7 +62,6 @@ case "$mode" in
  vm-file)      [ -f "${1:-}" ] || die "need local script file"; _vm "bash -s" < "$1" ;;
  vm-copy)      [ -f "${1:-}" ] || die "need local file"; _copy "${VM_USER}@${VM_IP}" "$1" "${2:-}" ;;
  prox-copy)    [ -f "${1:-}" ] || die "need local file"; _copy "${PROX_USER}@${PROX_HOST}" "$1" "${2:-}" ;;
  vm-guest)     [ "$#" -ge 1 ] || die "need command"; _vm_guest "$*" ;;
  ""|-h|--help|help) sed -n '2,40p' "${BASH_SOURCE[0]}" >&2; exit 0 ;;
  *) die "unknown mode '$mode'. Run '$0 help'." ;;
 esac
@@ -54,10 +54,9 @@ VM_ID="${VM_ID:-}"
 REPO_URL="${REPO_URL:-https://git.knownelement.com/KNEL/KNELServerBuild.git}"
 REMOTE_REPO="${REMOTE_REPO:-KNELServerBuild}"
 SNAP_PREFIX="${SNAP_PREFIX:-pre-knel-deploy}"
-ACCESS_PUBKEY="${ACCESS_PUBKEY:-$HOME/.ssh/id_ed25519.pub}"
+# SSH-only access policy: if SSH access is lost after a deploy, the agent does
-# Re-inject the validation pubkey after each deploy (secharden-ssh replaces
+# NOT re-inject keys via a back-channel. Set RESTORE_ACCESS=1 to have the
-# authorized_keys with the managed production key set, locking out the
+# script check and warn (it cannot self-restore).
 # bootstrap/dev key). Set RESTORE_ACCESS=0 to disable.
 RESTORE_ACCESS="${RESTORE_ACCESS:-1}"
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -80,7 +79,6 @@ die()  { log "ERROR: $*"; exit 1; }
 vm()      { bash "$REMOTE" vm "$@"; }           # as $VM_USER (SSH)
 vmroot()  { bash "$REMOTE" vmroot "$@"; }       # as root via sudo (SSH)
 vmfile()  { bash "$REMOTE" vm-file "$@"; }      # run local script on VM (SSH)
 vmguest() { bash "$REMOTE" vm-guest "$@"; }     # as root via guest agent (no SSH/2FA)
 prox()    { bash "$REMOTE" prox "$@"; }         # as $PROX_USER on Proxmox
 require_vm_id() {
@@ -104,45 +102,21 @@ wait_for_vm_ssh() {
 resolve_remote_repo() {
  local p
  p="$(vm "cd ~/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
  [[ -n "$p" ]] || p="$(vmguest "cd ~${VM_USER}/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
  printf '%s' "$p"
 }
-# Re-inject the validation pubkey into ~$VM_USER/.ssh/authorized_keys OUT OF
+# SSH-only access policy (AGENTS.md): if SSH access is lost (e.g. secharden-ssh
-# BAND via the Proxmox guest agent (qm guest exec runs as root inside the VM
+# replaced authorized_keys), the agent does NOT re-inject keys via guest-agent
-# and does not depend on SSH). This is necessary because secharden-ssh replaces
+# or any other back-channel. Surface the loss for manual restore.
 # authorized_keys with the managed production key set, which would otherwise
 # lock out the bootstrap key used to drive validation. No-op if SSH still works.
 restore_vm_access() {
  [[ "$RESTORE_ACCESS" = "1" ]] || { log "RESTORE_ACCESS=0; skipping access restore."; return 0; }
  [[ -f "$ACCESS_PUBKEY" ]] || { log "WARN: ACCESS_PUBKEY not found ($ACCESS_PUBKEY); cannot restore access."; return 0; }
  if vm 'true' >/dev/null 2>&1; then
-    log "SSH access already works; no need to restore."
+    log "SSH access works; no restore needed."
    return 0
  fi
-  log "SSH access lost (expected after secharden-ssh). Restoring via Proxmox guest agent..."
+  log "WARN: SSH access lost (e.g. secharden-ssh replaced authorized_keys)."
-  local payload_b64
+  log "      Per SSH-only access policy (AGENTS.md), no back-channel re-inject."
-  # Leading newline guards against the managed authorized_keys lacking a
+  log "      Restore manually: console login + re-add the bootstrap key, then re-run."
  # trailing newline (which would otherwise concatenate two keys into one).
  payload_b64="$(printf '\n%s' "$(cat "$ACCESS_PUBKEY")" | base64 -w0)"
  prox "qm guest exec $VM_ID -- /bin/sh -c 'echo $payload_b64 | base64 -d >> /home/${VM_USER}/.ssh/authorized_keys'" \
    >/dev/null 2>&1 || { log "WARN: guest-agent key append failed."; return 0; }
  prox "qm guest exec $VM_ID -- /bin/sh -c 'chown ${VM_USER}:${VM_USER} /home/${VM_USER}/.ssh/authorized_keys; chmod 600 /home/${VM_USER}/.ssh/authorized_keys'" \
    >/dev/null 2>&1 || true
  if vm 'true' >/dev/null 2>&1; then
    log "Access restored."
    return 0
  fi
  # If SSH still fails after re-injecting the key, 2FA is almost certainly the
  # cause (secharden-2fa enforces publickey+keyboard-interactive, which no
  # non-interactive SSH client can satisfy). That is expected and not fatal:
  # the guest agent still gives us full out-of-band access for log fetch and
  # the validation suite.
  if vmguest 'grep -q "^AuthenticationMethods" /etc/ssh/sshd_config' >/dev/null 2>&1; then
    log "SSH requires 2FA (expected after secharden-2fa); using guest agent for further access."
  else
    log "WARN: access still not working after restore and 2FA not detected. Check sshd_config."
  fi
 }
 # ---------------------------------------------------------------------------