docs: session cleanup — coverage matrix, questions v2, handoff pointer [#343]

- OAM coverage matrix published on Discourse #309
- questions-v2.md: Q1-Q6 resolved, Q7-Q9 added
- aug11.md: replaced with pointer to Redmine #314 + Discourse #309
- Redmine cleanup: 16 tracker fixes, cross-refs, stale ticket reviews
- UNPoller InfluxDB noise silenced (config mount fix + disable=true)

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

.crush/.gitignore

@@ -0,0 +1,3 @@
+*
+!crush.json
+!.gitignore

.crush/crush.json

@@ -0,0 +1,11 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "^(bash|edit|write|multiedit|lsp_replace_symbol|lsp_rename)$",
+        "command": "./hooks/ticket-gate.sh",
+        "timeout": 5
+      }
+    ]
+  }
+}

.env.example

@@ -0,0 +1,7 @@
+# PFVCluster environment variables
+# Copy to .env and fill in values for local development/testing.
+
+# Pi-hole (netinfra/pihole/docker-compose.yml)
+PIHOLE_WEB_PASSWORD=changeme
+
+# Shellcheck wrapper (tests/shellcheck.sh) — no config needed, uses Docker.

.gitignore

@@ -1,5 +1,9 @@
-# Crush internal state
-.crush/
+# Crush internal state (track crush.json config, ignore session data)
+.crush/crush.db
+.crush/crush.db-*
+.crush/logs/
+.crush/memory/
+.crush/active-ticket
 
 # OS/editor
 .DS_Store

AGENTS.md

@@ -1,5 +1,240 @@
 # Agent Guidelines
 
+## Agent Authority (NON-NEGOTIABLE)
+
+**No work is permissible on any system without an approved Redmine ticket.
+There are no exceptions to this rule.**
+
+### Scope of authority
+
+1. **Tickets govern all work.** The agent performs ONLY the work described in
+   the approved Redmine ticket. Anything outside that scope — no matter how
+   small, helpful, or "obvious" — is prohibited.
+
+2. **No autonomous system changes.** The agent does not modify, configure,
+   create, delete, or grant anything on a production system unless it is
+   explicitly directed by an approved ticket. "Production system" means
+   every system in the fleet — there is no "test" exception unless the ticket
+   says so.
+
+3. **Propose, never implement.** If the agent discovers additional work that
+   should be done — a bug, a misconfiguration, a missing dependency, an
+   enhancement — it does NOT implement it. Instead, it creates a Redmine
+   sub-ticket (status Feedback) describing the finding and surfaces it to
+   the user for approval.
+
+4. **Security and access changes require extra scrutiny.** Changes to sudoers,
+   SSH keys, user accounts, firewall rules, authentication policy, file
+   permissions, or any privilege-related configuration are treated as
+   policy decisions, not implementation details. The agent may suggest
+   them but NEVER implements them without explicit user direction in the
+   ticket or a sub-ticket the user has approved.
+
+5. **The user makes policy. The agent implements policy.** The agent does
+   not decide who gets sudo, what keys go where, what services run, or what
+   the access model is. The agent executes the user's decisions, exactly
+   as specified.
+
+6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
+   require something not explicitly authorized, or if the agent is unsure
+   whether an action is in scope — STOP and ask the user via the ticket
+   or directly. Asking is always acceptable. Overstepping is never
+   acceptable.
+
+7. **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user. This applies to ALL tickets,
+   no exceptions.
+8. **User acceptance testing is MANDATORY before declaring work done.**
+   The agent performs implementation and technical validation (services
+   running, configs correct, APIs responding). The user performs UAT —
+   visually confirming dashboards render data, alerts deliver, tools are
+   usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
+   closing, and MUST NOT move to the next ticket until the user explicitly
+   accepts the work. "Technically wired but blank dashboard" is NOT done.
+9. **NEVER access a database directly if an API exists.** APIs are the
+   stable contract; databases change schemas without warning. If a tool
+   has an API, use it — exhaust all API endpoints, check the docs, try
+   alternative methods. Only fall back to direct DB access as a last
+   resort AND with explicit user approval for that specific instance.
+   Soon all DB access will route through a proxy under zero trust;
+   building API-first habits now ensures that transition is clean.
+
+### Access-channel policy: SSH only (NON-NEGOTIABLE)
+
+**The qemu guest-agent is NEVER an access, execution, or key-delivery
+channel.** SSH (`sshd`) is the only approved remote access path. Every
+command must flow through sshd so it is captured by the standard
+auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
+environment — there is no back-door exception, ever.
+
+- **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
+  over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
+  mode). This includes using guest-agent to *deliver* an SSH key, even if
+  the resulting SSH login is itself audited — the delivery bypassed audit.
+- **Allowed:** installing or checking qemu-guest-agent for its intended
+  purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
+  `agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
+  never execution.
+- **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
+  any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
+  no guest-exec mode.
+- **If a system is locked out** (no SSH key, no guest-exec path): surface
+  it to the user. Do NOT improvise an alternate back-channel. The user
+  authorizes the unblock method (console login, credential, etc.).
+
+### Ticket-closing policy (NON-NEGOTIABLE)
+
+**NEVER close a Redmine ticket without explicit user permission.** You
+may SUGGEST a close when the result is clearly scoped and delivered. If
+it's ambiguous whether the work is truly complete, don't suggest a close
+— leave that decision to the user. This applies to ALL tickets, no
+exceptions.
+
+### What this means in practice
+
+- Discovered a typo in a config during approved work? **Finish the approved
+  work. Create a sub-ticket for the typo. Do not fix it inline.**
+- Think a system should also have localuser sudo configured? **Do not add
+  it. Propose it in a sub-ticket.**
+- Need to install a package the ticket didn't mention? **Ask first.**
+- Found a security issue? **Create a ticket immediately with full details.
+  Do not remediate without approval.**
+
+This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
+traceable to an approved ticket. There is no "I thought it would help."
+
+## Quick Start
+
+**You are an AI agent working on this project. Your first actions, in order:**
+
+> **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
+> check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
+> the scope-alignment gate. You MUST NOT begin any task work until the user
+> (a) confirms your rule summary AND (b) names the ticket to work on. This runs
+> **every session, automatically** — the user should never have to ask for it.
+
+1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
+2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
+3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
+4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
+   back to the user before doing any work: Agent Authority (ticket-governed,
+   propose-never-implement), Access-Channel SSH-only policy, Remote access
+   (remote.sh chokepoint mandatory, DNS names only — never IP literals),
+   Questions policy (no harness question-tools), Documentation policy
+   (Discourse is SoR), Redmine tracking, Git policy (always commit+push,
+   shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
+   routing chokepoints (`tests/remote.sh`), the mechanically enforced
+   `check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
+   and wait for the user to confirm. This checkpoint guarantees every
+   session starts aligned.
+5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
+6. **Check current state:** `git log --oneline -10`.
+7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
+8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
+   priority list (or the Redmine queue if no handoff exists). Do NOT scan the
+   full ticket queue and pick work on your own — **the user directs what gets
+   worked on, always.** Note any new or urgent items from the Redmine check,
+   then ask which ticket to work on tonight. **STOP and wait.** Do not set
+   `.crush/active-ticket`, do not read systems, do not run diagnostics, until
+   the user names the target. This gate prevents the agent from burning
+   context on work the user didn't ask for.
+
+## Enforcement Model
+
+Git hooks (`scripts/pre-commit`, `scripts/pre-push`) enforce the rules defined in
+`scripts/check-rules.sh`. The rules engine checks: shellcheck (zero warnings
+including info-level), Docker image pinning (no `:latest`), container naming,
+required files, Discourse pointer headers, and more. Run `bash scripts/check-rules.sh`
+for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
+(emergencies only).
+
+## Task Tracking
+
+- **Redmine is the system of record for all work.**
+- **NEVER close a ticket without explicit user permission.** Suggest a
+  close when clearly scoped/delivered; if ambiguous, don't suggest.
+- **Ticket-first enforcement (mechanically enforced).** Before starting
+  any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
+  The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
+  until this file exists. If no ticket exists, CREATE ONE FIRST via
+  redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
+- **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
+  session. The pre-commit hook blocks commits while any task remains unchecked.
+- Clear WORKING.md before responding to the user.
+
+## Rolling HUD (session-scoped, NOT persisted)
+
+The rolling HUD is a **live status display** the agent maintains throughout
+the session to help the human follow along. It is NOT a system of record —
+Redmine, Discourse, and git are the durable systems. The HUD exists purely
+for the human's situational awareness during the session.
+
+- **Format:** a compact block shown at the end of each significant response
+  (after completing a step, hitting a blocker, or pivoting). Example:
+
+  ```
+  ┌─ SESSION HUD ────────────────────────────────────────
+  │ Active: #343 (Monitoring coverage matrix)
+  │ Done:   ✓ scope-alignment gate added to both AGENTS.md
+  │         ✓ committed + pushed (6a2550b)
+  │ Now:    drafting coverage matrix on Discourse
+  │ Next:   → #341 TEMPer USB (deploy on pfv-tsys1)
+  │         → #338 LibreNMS alerts (BLOCKED: pushover.env)
+  ├─ USER ACTION ITEMS ──────────────────────────────────
+  │ • Populate ~/.creds/pushover.env (unblocks #338, #428)
+  │ • Populate ~/.creds/prometheus.env + grafana.env (#430)
+  └──────────────────────────────────────────────────────
+  ```
+
+- **Placement:** may be written to `.crush/hud.md` on disk to keep context
+  window smaller (re-read and update rather than hold in memory). Never
+  committed to git. Wiped at session end.
+- **Sections:**
+  - **Active:** current ticket number + one-line description
+  - **Done:** ✓ items completed this session (append as work progresses)
+  - **Now:** what the agent is actively doing
+  - **Next:** the queued items (per handoff priority or user direction)
+  - **User action items:** things ONLY the user can do (populate creds,
+    physical work, manual deploys) with the tickets they unblock
+- **When to show it:** after each logical unit of work, at blockers, and
+    when pivoting between tickets. Not every trivial response — use judgment.
+- **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
+    work completes, update the durable systems (ticket notes, Discourse wiki,
+    commits). The HUD just tracks the live narrative for the human.
+
+## Working Style
+
+- **Stop over-thinking.** Get to code and output faster. Explore with code;
+  gather ground truth. Do not burn tokens reasoning about things a quick command
+  answers.
+- **Farm work out to deterministic tooling:** linters, LSPs, formatters, test
+  runners. If an LSP is wired up, use it; otherwise pull a Docker image and lint
+  inside it.
+- **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
+  Never read 10+ files sequentially; batch into agent calls.
+- **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
+  system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
+  (status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
+  If a command hits the timeout, STOP and investigate root cause — never
+  blindly retry. A hung command is a failed command. Detect failure fast,
+  diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
+  This applies to ALL tools — bash, docker, CLIs, sub-agents.
+
+## Questions (NON-NEGOTIABLE)
+
+**NEVER use a harness "question"/"ask user" tool** (structured prompts,
+modal forms, tabbed questions). Banned across every project, every harness.
+They are not portable, not version-controlled, and bypass the git record.
+**All questions go in the current `questions-v(N).md` file** — write the
+question; the human edits the answer inline in the same file. **Version up
+the filename each time answers land** (v1 → v2 → v3...): create
+`questions-v2.md` with resolved Q&A marked, new questions appended. This
+preserves the history of each Q&A round. Synthesize resolved Q&A into
+Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
+§10.
+
 ## Documentation policy (IMPORTANT)
 
 **Discourse is the canonical source of truth for all knowledge documentation.**
@@ -38,6 +273,7 @@ k8s/                k3s cluster setup scripts (HA control plane over Tailscale)
 proxmox/            Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
 awx/                Ansible AWX deployment (k3s + AWX Operator)
 tests/              Test suite + VM validation harness + remote.sh SSH chokepoint
+scripts/            Framework: git hooks, rule engine (check-rules.sh), shared lib
 docs/               Server-build docs, docmap index, and archive
 archive/            Historical/superseded code (provisioning -> replaced by KNELIAC project)
 vendor/             Vendored KNELShellFramework
@@ -49,9 +285,27 @@ vendor/             Vendored KNELShellFramework
   `/home/reachableceo/projects/KNELIAC`.
 - **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
   extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
-- **SSH in Crush**: Direct ssh/scp is blocked. Use
-  [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
-  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
+- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
+  through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
+  (Proxmox hosts + all VMs) or
+  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh)
+  (DNS infra hosts: netinfra-01/02, tsrouter, netboot). NEVER call
+  `ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
+  rejects it. There are no exceptions.
+- **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
+  (neither LAN nor Tailscale IPs) in any command, script, or config.
+  ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
+  `VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
+  DNS name does not resolve, fix it in DNS (Technitium) or consult the
+  [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  Do NOT fall back to IP literals. This rule eliminates the per-session
+  discovery tax of finding the right IP for each host.
+- **How to access a production VM:**
+  1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
+  3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
+     and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
+     NOT execution) to find the Tailscale DNS name, then access via that.
 
 ## Git Policy
 
@@ -92,30 +346,41 @@ checklists, or TODOs in repo files. Use Redmine tickets instead.
 2. **Reference tickets in docs and commits** using `[#NNN]` notation.
    Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
 3. **When work completes**, update the ticket: set done ratio to 100%,
-   add a note describing what was done and where the code lives, then close.
+   add a note describing what was done and where the code lives.
+   **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user.
 4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
 5. **Link code to tickets** — ticket descriptions and notes should reference
    the relevant file paths in this repo (e.g., `dcinfra/ups/`).
 
 ### CLI access (read + write)
 
-Tickets are managed via the `redmine` CLI wrapper in the `~/daytoday/redmine`
-workspace (external to this repo). Full command reference, patterns, and
-the subtask escape hatch live in `~/daytoday/redmine/AGENTS.md` — read it for
-anything beyond the basics.
+Tickets are managed via the `redmine-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the subtask escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/redmine/AGENTS.md`) — read it
+for anything beyond the basics.
 
 ```bash
 # Connection sanity check (run first in any session):
-~/daytoday/redmine/bin/redmine whoami
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
 
 # Your queue (project 55):
-~/daytoday/redmine/bin/redmine list --assigned-to-me -p 55
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55
 
 # Show / create / update / close:
-~/daytoday/redmine/bin/redmine show 367
-~/daytoday/redmine/bin/redmine create -p 55 -s "Subject" -d "desc"
-~/daytoday/redmine/bin/redmine update 367 -n "Done: committed in abc123" --done-ratio 100
-~/daytoday/redmine/bin/redmine close 367
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest show 367
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest create -p 55 -s "Subject" -d "desc"
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest update 367 -n "Done: committed in abc123" --done-ratio 100
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest close 367
 ```
 
 Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
@@ -123,10 +388,10 @@ Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
 Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
 **Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
 `python-redmine` escape hatch inside the container (see
-`~/daytoday/redmine/AGENTS.md`). Always `show` a ticket before updating it.
+`tooling-cli/redmine/AGENTS.md`). Always `show` a ticket before updating it.
 
-The `.env` holding `REDMINE_URL`/`REDMINE_API_KEY` lives (gitignored) at
-`/home/reachableceo/projects/KNEL-AIMiddleware/redmine-cli/.env`.
+Credentials (`REDMINE_URL`/`REDMINE_API_KEY`) live in the centralized store
+at `~/.creds/redmine.env`.
 
 ## Documentation Workflow
 
@@ -149,42 +414,49 @@ session, an agent MUST:
 
 ### CLI access (read + write)
 
-Wiki topics are managed via the `discourse` CLI wrapper in the
-`~/daytoday/discourse` workspace (external to this repo). Full command
-reference, patterns, and the raw-API escape hatch live in
-`~/daytoday/discourse/AGENTS.md` — read it for anything beyond the basics.
+Wiki topics are managed via the `discourse-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the raw-API escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/discourse/AGENTS.md`) — read it
+for anything beyond the basics.
 
 ```bash
 # Connection sanity check (run first in any session):
-~/daytoday/discourse/bin/discourse whoami
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
 
 # List VP TechOps topics:
-~/daytoday/discourse/bin/discourse ls -c vp-techops
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest ls -c vp-techops
 
 # Show a topic / edit a wiki post (find post id via `show`):
-~/daytoday/discourse/bin/discourse show 297
-~/daytoday/discourse/bin/discourse update <post_id> -b "new markdown body"
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest show 297
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest update <post_id> -b "new markdown body"
 ```
 
 VP TechOps = category **74**. Key topics: #296 (project overview), #297
 (ops status), #298 (audit log). The API user is trust-level 4 but **not
 admin** — admin-only ops (category creation, setting the wiki flag) will
 403; surface those to the user rather than retrying. **Gotcha:**
-`update`/`delete` take a post **id**, not a post number (see
-`~/daytoday/discourse/AGENTS.md`). Never create a new topic for an update to
-existing knowledge — edit the wiki post in place.
+`update`/`delete` take a post **id**, not a post number. Never create a new
+topic for an update to existing knowledge — edit the wiki post in place.
 
-The `.env` holding `DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`
-lives (gitignored) at `/home/reachableceo/projects/KNEL-AIMiddleware/discourse-cli/.env`.
+Credentials (`DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`)
+live in the centralized store at `~/.creds/discourse.env`.
 
 ## Key Scripts
 
 | Script | Purpose |
 |--------|---------|
+| [`scripts/check-rules.sh`](scripts/check-rules.sh) | Rule audit engine (shellcheck, image pinning, Discourse pointers, required files) |
+| [`scripts/setup-hooks.sh`](scripts/setup-hooks.sh) | Install git hooks (pre-commit, pre-push) |
 | [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
 | [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
-| `~/daytoday/redmine/bin/redmine` | Redmine CLI wrapper (ticket read/write via Docker container) |
-| `~/daytoday/discourse/bin/discourse` | Discourse CLI wrapper (wiki topic read/write via Docker container) |
+| `redmine-cli` container | Redmine CLI (ticket read/write via `docker run`; see `tooling-cli/redmine/`) |
+| `discourse-cli` container | Discourse CLI (wiki topic read/write via `docker run`; see `tooling-cli/discourse/`) |
+| `dns-cli` container | Technitium DNS CLI (zones, list, add, delete, search, flush; see `tooling-cli/dns/`) |
 | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
 | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
 | [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
@@ -193,7 +465,104 @@ lives (gitignored) at `/home/reachableceo/projects/KNEL-AIMiddleware/discourse-c
 | [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
 | [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
 | [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
-| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift |
+| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |
+
+## Switch Console Access (conman)
+
+Switch configs are pulled via serial console through a conman + ser2net
+stack on pfv-tsys4.
+
+- **conmand server:** pfv-tsys4 (port 7890)
+- **ser2net:** pfv-tsys4, TCP ports 2001-2006
+  (do NOT connect to ser2net directly — it conflicts with conman's
+  persistent sessions; always use the conman client)
+- **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
+  — drives console sessions read-only via PTY. No expect/tcl required.
+- **Command files:** [`netinfra/switches/`](netinfra/switches/) — `.cmds` files
+  with switch-specific show commands
+- **Query available consoles:** `conman -d 100.70.77.93 -q`
+
+```bash
+# Pull a switch config (example):
+CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+| Console name | TCP port | Device |
+|--------------|----------|--------|
+| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
+| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
+| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
+| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
+| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
+| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
+
+## Mandatory: Document ALL Infrastructure Changes
+
+**This is non-negotiable. Every infrastructure change (VM config, disk
+cache, network setting, service config, storage migration) MUST be
+documented BEFORE moving to the next task step — not "later" or "at the
+end."**
+
+Required for EVERY infrastructure change:
+
+1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
+2. **Discourse audit log** — reply to topic
+   [#298](https://community.turnsys.com/t/298) with a dated entry (what
+   changed, why, where).
+3. **Discourse relevant wiki topic** — update the architecture/reference
+   topic if the change affects documented infrastructure facts (storage
+   #300, network #299, k8s #305, etc.).
+4. **Git commit** — if code/config changed in the repo, commit + push
+   immediately per the Git Policy above.
+
+## Tooling
+
+- `gh`, `docker`, `jq` available on the workstation.
+- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
+  `koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
+- For raw API calls not covered by the Redmine/Discourse CLIs, use
+  `python3` inside the CLI Docker containers (escape hatch pattern in
+  `tooling-cli/{discourse,redmine}/AGENTS.md`).
+- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
+  the python-in-Docker escape hatch for HTTP writes.
+
+## Key Commands Quick Reference
+
+```bash
+# Tests + validation:
+bash tests/run-tests.sh                        # test suite
+bash tests/vm-validation.sh                    # VM validation
+bash tests/shellcheck.sh                       # shellcheck whole repo
+bash tests/shellcheck.sh path/to/*.sh          # shellcheck specific files
+
+# Redmine + Discourse sanity checks (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# Proxmox host access (DNS names only — never IPs):
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
+
+# Production VM access (DNS names only — never IPs):
+VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
+
+# DNS infra access:
+bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
+
+# Switch console (read-only config pull — DNS names only):
+CONMAN_SERVER=pfv-tsys4:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+## Credential Management
+
+- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
+- User goal: migrate all keys to Hashicorp Vault
+  (`vault.knownelement.com`). No vault token present yet
+  (`~/.vault-token` missing). Track as high-priority TODO.
 
 ## Key Docs
 
@@ -202,6 +571,12 @@ lives (gitignored) at `/home/reachableceo/projects/KNEL-AIMiddleware/discourse-c
 
 → All `.md` files in this repo are pointers to Discourse topics.
 
+→ **Complete Linux System Inventory:**
+  [Topic #307](https://community.turnsys.com/t/307) — every Linux system
+  (hosts + VMs + physical), with Tailscale IPs, DNS names, SSH access
+  status, and tuned profiles. Reference this for monitoring coverage,
+  access management, and hostname consistency.
+
 ## Project Context
 
 Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.

Makefile

@@ -0,0 +1,50 @@
+# Makefile — convenience dispatch to scripts/.
+#
+# Not required. The scripts in scripts/ are the real entry points and work
+# standalone. This file just gives you short verbs if you're at a terminal.
+#
+# In Mode 2 (Hermes/OWUI/MCP), agents call the scripts directly or via API —
+# they don't need this file.
+
+# Project-specific overrides for check-rules.sh
+export PROJECT_DOC_EXEMPT ?= AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v1.md BASELINE-PROMPT.md PATTERNS.md
+export PROJECT_DISCOURSE_HOST ?= community.turnsys.com
+
+.PHONY: setup validate fast lint test garden up down status clean help
+
+help: ## Show available targets
+	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
+
+setup: ## Install git hooks
+	@bash scripts/setup-hooks.sh
+
+validate: ## Full rule audit (includes tests)
+	@bash scripts/check-rules.sh
+
+fast: ## Fast rule audit (pre-commit equivalent)
+	@bash scripts/check-rules.sh --fast
+
+lint: ## Lint shell scripts (shellcheck via docker)
+	@docker run --rm -v "$$(pwd):/mnt" koalaman/shellcheck:stable \
+		$$(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print | sed 's|^\./|/mnt/|') || true
+
+test: ## Run the test suite (override per project)
+	@bash scripts/test.sh
+
+garden: ## Doc-sprawl / Discourse-migration report
+	@bash scripts/garden.sh
+
+up: ## Bring up the docker-compose stack
+	@bash scripts/up.sh
+
+down: ## Bring down the docker-compose stack
+	@bash scripts/down.sh
+
+status: ## Show repo status snapshot
+	@echo "== branch =="; git branch --show-current 2>/dev/null || echo "(no branch)"
+	@echo "== last commit =="; git log --oneline -1 2>/dev/null || true
+	@echo "== working tree =="; git status --short 2>/dev/null || echo "(not a git repo)"
+	@echo "== STATUS.md head =="; sed -n '1,12p' STATUS.md 2>/dev/null || echo "(no STATUS.md)"
+
+clean: ## Remove build/test artifacts (override per project)
+	@echo "make clean: nothing to clean — override this in your project's Makefile."

WORKING.md

@@ -0,0 +1,11 @@
+# WORKING.md — Active Session Tracker
+
+Agent work only. The human decides when it's done.
+A commit is blocked while any task below remains unchecked.
+
+## Current Tasks
+
+(all done — session complete)
+
+- [x] Add ticket-gate Crush hook (blocks work without active ticket)
+- [x] Document active-ticket workflow in AGENTS.md + meta template

access-matrix.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/bash
+# access-matrix.sh — definitive access verification across all Linux Tailscale nodes.
+# Uses the correct SSH user(s) per system type, checks sudo where applicable.
+# Routes through remote.sh (the only allowed ssh path).
+set -u
+cd /home/reachableceo/projects/PFVCluster || exit 1
+
+# Policy-excluded systems (never attempt access)
+EXCLUDE=':tsys-cloudron:pfv-bms:tsys-umbrel:tsys-ucs-01:tsys-ucs-02:stlpc-bizoffice:ultix-highside:'
+
+# Determine the SSH user(s) for a given hostname and whether sudo is expected.
+# Returns "user1:user2:...:sudoflag" where sudoflag is "yes" or "no".
+users_for() {
+  local name="$1"
+  case "$name" in
+    pfv-tsys[0-9])                   echo "root:no" ;;
+    *-proxmox-datacenter)            echo "root:no" ;;
+    *-proxmox-pve)                   echo "root:no" ;;
+    *-proxmox-pbs)                   echo "root:no" ;;
+    *-proxmox-mailgw*)               echo "root:no" ;;
+    *-proxmox-backup*)               echo "root:no" ;;
+    tsys-ucs-*)                      echo "root:no" ;;
+    ultix-streaming)                 echo "root:no" ;;
+    stlpc-*)                         echo "root:labuser:no" ;;
+    ultix-field)                     echo "ultixfield:yes" ;;
+    subopi*)                         echo "subodev:yes" ;;
+    *)                               echo "localuser:yes" ;;
+  esac
+}
+
+check_user() {
+  local ip="$1" user="$2" expect_sudo="$3"
+  local out sudo
+  out=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'echo SSHOK; id -un' </dev/null 2>&1 | grep -oE 'SSHOK|keyboard-interactive|Connection refused' | head -1)
+  case "$out" in
+    SSHOK)
+      if [ "$expect_sudo" = "yes" ]; then
+        sudo=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'sudo -n true 2>/dev/null && echo SUDOOK || echo SUDONO' </dev/null 2>&1 | grep -oE 'SUDOOK|SUDONO' | head -1)
+        printf '%s(%s)' "$user" "${sudo:-?}"
+      else
+        printf '%s(ok)' "$user"
+      fi
+      ;;
+    keyboard-interactive) printf '%s(2FA)' "$user" ;;
+    'Connection refused')  printf '%s(NOSSH)' "$user" ;;
+    *)                      printf '%s(NOKEY)' "$user" ;;
+  esac
+}
+
+printf '%-32s %-16s %s\n' "NAME" "TS-IP" "ACCESS"
+printf '%-32s %-16s %s\n' "----" "-----" "------"
+
+tailscale status 2>/dev/null | awk '$4=="linux" {print $2, $1}' | sort | while read -r name ip; do
+  [ -n "$name" ] || continue
+  case "$EXCLUDE" in *":$name:"*) printf '%-32s %-16s %s\n' "$name" "$ip" "EXCLUDED"; continue;; esac
+
+  map=$(users_for "$name")
+  expect_sudo="${map##*:}"
+  users="${map%:*}"
+  result=""
+  IFS=':' read -ra user_list <<< "$users"
+  for u in "${user_list[@]}"; do
+    r=$(check_user "$ip" "$u" "$expect_sudo")
+    [ -z "$result" ] && result="$r" || result="$result $r"
+  done
+  printf '%-32s %-16s %s\n' "$name" "$ip" "$result"
+done

agent-bootstrap.sh

@@ -0,0 +1,60 @@
+#!/bin/sh
+# agent-bootstrap.sh
+#
+# Run INSIDE a guest (via SSH, console, or guest-agent) to bring the
+# system fully under agent management in one shot:
+#   1. install + enable qemu-guest-agent  (VMs only, skipped on bare metal)
+#   2. push the agent SSH key to root + AGENT_USER (+ labuser if present)
+#   3. grant AGENT_USER passwordless sudo
+#
+# AGENT_USER defaults to "localuser". Override for systems with a different
+# unprivileged agent user:
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+#
+# After this runs once, the agent has SSH+sudo immediately.
+#
+# Usage (from a root shell in the guest):
+#   bash agent-bootstrap.sh
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+
+set -eu
+
+KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIWms/uCXnjjo4KyxHBcYI2TDHe8OZ2wle6W/0hSRQLu reachableceo@ultix-streaming'
+AGENT_USER="${AGENT_USER:-localuser}"
+
+# 1. guest-agent (skip on bare metal — no virtio-serial device)
+if command -v systemd-detect-virt >/dev/null 2>&1 && \
+   [ "$(systemd-detect-virt --vm 2>/dev/null || echo none)" != "none" ]; then
+  if ! command -v qemu-ga >/dev/null 2>&1; then
+    if command -v apt-get >/dev/null 2>&1; then
+      apt-get update
+      DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
+    elif command -v dnf >/dev/null 2>&1; then
+      dnf install -y qemu-guest-agent
+    elif command -v yum >/dev/null 2>&1; then
+      yum install -y qemu-guest-agent
+    else
+      echo "WARN: no supported package manager; skipping agent install" >&2
+    fi
+  fi
+  systemctl enable --now qemu-guest-agent 2>/dev/null || \
+    systemctl enable --now qemu-ga 2>/dev/null || true
+fi
+
+# 2. SSH key for root + AGENT_USER + labuser (if present)
+for u in root "$AGENT_USER" labuser; do
+  if ! getent passwd "$u" >/dev/null 2>&1; then continue; fi
+  H=$(getent passwd "$u" | cut -d: -f6)
+  mkdir -p "$H/.ssh"; chmod 700 "$H/.ssh"
+  AK="$H/.ssh/authorized_keys"; touch "$AK"; chmod 600 "$AK"
+  grep -qF "$KEY" "$AK" || echo "$KEY" >> "$AK"
+  chown -R "$u": "$H/.ssh"
+done
+
+# 3. passwordless sudo for AGENT_USER only
+if getent passwd "$AGENT_USER" >/dev/null 2>&1 && [ -d /etc/sudoers.d ]; then
+  echo "${AGENT_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/010-agent
+  chmod 440 /etc/sudoers.d/010-agent
+fi
+
+echo BOOTSTRAP-DONE

aug11.md

@@ -0,0 +1,9 @@
+<!-- Discourse: https://community.turnsys.com/t/298 -->
+<!-- Redmine: https://projects.knownelement.com/issues/314 -->
+
+# Session Handoff: 2026-08-11/12 OAM
+
+**Full handoff lives in Redmine #314** (session summary note, 2026-08-12).
+**OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309).
+**Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303).
+**Open questions:** `questions-v2.md` in this repo.

bootstrap-all.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/bash
+# bootstrap-all.sh — push agent SSH key + passwordless sudo to remaining systems.
+#
+# SSH is the ONLY approved access channel (see AGENTS.md "Access-channel
+# policy: SSH only"). This script reaches systems that still allow password
+# auth over sshd. Systems that reject password auth (publickey-only) cannot
+# be reached this way — see the CONSOLE-ONLY section printed at the end.
+#
+# Two escalation methods:
+#   sudo → Ubuntu-style systems (no root pw; localuser has sudo)
+#   su   → Debian-style systems (root has a password)
+#
+# Passes AGENT_USER so agent-bootstrap.sh targets the correct unprivileged
+# user. You enter passwords interactively. Idempotent: safe to re-run.
+#
+# Verified state (access-matrix.sh, 2026-08-10): 68/70 non-excluded systems
+# at intended access state. Only tsys-siem remains below.
+set -u
+cd "$(dirname "$0")" || exit 1
+
+SCRIPT=agent-bootstrap.sh
+SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
+
+run_with_sudo() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (sudo, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "sudo AGENT_USER=${agent_user} bash /tmp/$SCRIPT" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+run_with_su() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (su, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "su -c 'AGENT_USER=${agent_user} bash /tmp/$SCRIPT'" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+# All password-auth-reachable systems have been bootstrapped.
+# Verified state (access-matrix.sh, 2026-08-10): 69/70 non-excluded systems
+# at intended access state. The only remaining NOKEY (stlp-3dscanner) is
+# deferred to [#417] and requires a rename + bring-online first — out of
+# scope here. Run `access-matrix.sh` to re-verify at any time.
+echo "All password-auth-reachable systems are bootstrapped."
+echo "Remaining gap: stlp-3dscanner (deferred to [#417])."
+echo "Run access-matrix.sh to re-verify."
+echo "Deferred (separate ticket):"
+echo "  stlp-3dscanner — rename + bring online first [#417]"
+echo "========================================"
+echo "By design (leave alone):"
+echo "  sectestbed-sandbox — 2FA enforced"
+echo "========================================"
+echo "Excluded by policy (no SSH access):"
+echo "  pfv-bms (API), tsys-cloudron (prod revenue),"
+echo "  tsys-ucs-01/02 (API-managed), tsys-umbrel (treasury)"
+echo "========================================"

dcinfra/console/generate-config.sh

@@ -6,7 +6,7 @@
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
-#      to a named symlink like /dev/consoles/pfv-core-sw01
+#      to a named symlink like /dev/consoles/pfv-r5-core-01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing

dcinfra/console/mapping.txt

@@ -17,13 +17,13 @@
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
-2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
-2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
-2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
+2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
+2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
+2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
-2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
-2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
-2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
+2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
+2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
+# 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare

dcinfra/console/query-remote.sh

@@ -5,7 +5,7 @@
 #
 # Usage:
 #   bash console/query-remote.sh                    # list consoles
-#   bash console/query-remote.sh pfv-core-sw01      # connect to a console
+#   bash console/query-remote.sh pfv-r5-core-01      # connect to a console
 #
 set -euo pipefail
 
@@ -52,7 +52,7 @@ if [ -z "$CONSOLE" ]; then
   conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
   echo ""
   echo "To connect: bash $0 <console-name>"
-  echo "  e.g: bash $0 pfv-core-sw01"
+  echo "  e.g: bash $0 pfv-r5-core-01"
 else
   echo ""
   echo "--- Connecting to: $CONSOLE ---"

dcinfra/console/setup.sh

@@ -202,7 +202,7 @@ echo "    ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
 echo "    conman connects via telnet for logging + multiplexing"
 echo ""
 echo "  Connect from any Tailscale workstation:"
-echo "    conman -d pfv-tsys4:7890 -f pfv-core-sw01"
+echo "    conman -d pfv-tsys4:7890 -f pfv-r5-core-01"
 echo "    conman -d pfv-tsys4:7890 -q              # list consoles"
 echo ""
 echo "  Direct telnet (emergency, conflicts with conman):"

dcinfra/console/validate-conman.sh

@@ -45,7 +45,7 @@ echo "--- 3. Trigger log capture: connect to each console briefly ---"
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
-for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
+for name in pfv-r5-core-01 pfv-r3-tor-mgmt-01 pfv-r3-tor-stor-01 pfv-rrinfra-rtr pfv-r2-tor-01 pfv-r6-mgmt-01; do
   logfile="$LOGDIR/${name}.log"
   if [ -f "$logfile" ]; then
     SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)

dcinfra/powerman/identify-outlets.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/bash
+# powerman/identify-outlets.sh — flash each PDU outlet sequentially for physical tracing
+#
+# Run this from the workstation. It flashes each outlet one at a time so you
+# can walk the rack and see which device's LED blinks. Write down the mapping,
+# then run rename-outlets.sh with that mapping.
+#
+# Usage:
+#   bash dcinfra/powerman/identify-outlets.sh
+#
+# On Friday: run this, walk the rack, note which outlet → which device.
+set -uo pipefail
+PROX_HOST="${PROX_HOST:-pfv-tsys1}"
+REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
+
+echo "PDU Outlet Identification — Flash Sequence"
+echo "============================================"
+echo "Each outlet will flash for 5 seconds. Walk the rack and note the device."
+echo "Press Enter to start..."
+read -r
+
+for i in $(seq 1 10); do
+  echo "--- Outlet $i: FLASHING (5s) ---"
+  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -f outlet-$i" </dev/null 2>/dev/null
+  sleep 5
+  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -u outlet-$i" </dev/null 2>/dev/null
+  echo "    Outlet $i → ? (write it down)"
+  echo ""
+  [ "$i" -lt 10 ] && { echo "Press Enter for next outlet..."; read -r; }
+done
+
+echo "============================================"
+echo "Done. Now create your mapping file and run:"
+echo "  bash dcinfra/powerman/rename-outlets.sh"
+echo ""
+echo "Format: outlet-number:new-name (one per line)"
+echo "Example:"
+echo "  1:pfv-tsys1"
+echo "  2:pfv-tsys3"
+echo "  ..."
+echo "============================================"

dcinfra/powerman/rename-outlets.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/bash
+# powerman/rename-outlets.sh — rename PDU outlets in powerman.conf
+#
+# Takes a mapping file (outlet-number:new-name, one per line) and rewrites
+# the node entries in /etc/powerman/powerman.conf on pfv-tsys1, then
+# restarts powermand.
+#
+# Usage:
+#   bash dcinfra/powerman/rename-outlets.sh <mapping-file>
+#
+# Example mapping file:
+#   1:pfv-tsys1
+#   2:pfv-tsys3
+#   3:pfv-tsys4
+#   ...
+set -euo pipefail
+
+PROX_HOST="${PROX_HOST:-pfv-tsys1}"
+REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
+MAP_FILE="${1:-}"
+
+if [ -z "$MAP_FILE" ] || [ ! -f "$MAP_FILE" ]; then
+  echo "Usage: $0 <mapping-file>"
+  echo "  Format: outlet-number:new-name (one per line)"
+  echo "  Run identify-outlets.sh first to get the mapping."
+  exit 1
+fi
+
+# Build the new node lines
+NODE_LINES=""
+while IFS=: read -r num name; do
+  [ -z "$num" ] && continue
+  NODE_LINES+="node \"$name\" \"cyclades-pm10\" \"$num\""$'\n'
+done < "$MAP_FILE"
+
+# Send to tsys1: backup conf, write new node section, restart powermand
+PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox-file - <<REMOTE_SCRIPT
+set -euo pipefail
+cp /etc/powerman/powerman.conf /etc/powerman/powerman.conf.bak.\$(date +%Y%m%d-%H%M%S)
+
+# Strip existing node lines and append new ones
+grep -v '^node "outlet-' /etc/powerman/powerman.conf > /tmp/powerman.conf.new
+cat >> /tmp/powerman.conf.new <<'NODES'
+$(echo -n "$NODE_LINES")
+NODES
+mv /tmp/powerman.conf.new /etc/powerman/powerman.conf
+systemctl restart powerman
+sleep 1
+powerman -l
+REMOTE_SCRIPT
+
+echo "PDU outlets renamed. Verify with: PROX_HOST=$PROX_HOST bash $REMOTE_SH prox 'powerman -q'"

hooks/ticket-gate.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# hooks/ticket-gate.sh — enforce ticket-first work policy
+#
+# Blocks modifying operations until an active ticket is established.
+# The agent sets the active ticket via:  echo '#NNN' > .crush/active-ticket
+# And clears it when done:               > .crush/active-ticket
+#
+# Exempts read-only and management commands (so you can create tickets,
+# run audits, check status, etc.).
+set -euo pipefail
+
+TICKET_FILE="${CRUSH_PROJECT_DIR}/.crush/active-ticket"
+TOOL="${CRUSH_TOOL_NAME:-}"
+CMD="${CRUSH_TOOL_INPUT_COMMAND:-}"
+
+# Read-only tools — always allowed
+case "$TOOL" in
+  view|ls|grep|glob|agent|sourcegraph|fetch|agentic_fetch|download|lsp_diagnostics|lsp_symbols|lsp_definition|lsp_references|lsp_call_hierarchy|crush_info|crush_logs|question|todos)
+    exit 0
+    ;;
+esac
+
+# For bash tool: exempt read-only and management commands
+if [ "$TOOL" = "bash" ]; then
+  # Ticket/doc/dns management — always allowed
+  case "$CMD" in
+    *"redmine-cli"*|*"discourse-cli"*|*"dns-cli"*|*"technitium"*) exit 0 ;;
+  esac
+  # Read-only git
+  case "$CMD" in
+    *"git status"*|*"git log"*|*"git diff"*|*"git show"*|*"git branch"*) exit 0 ;;
+  esac
+  # Repo hygiene scripts
+  case "$CMD" in
+    *"check-rules"*|*"setup-hooks"*|*"shellcheck"*|*"run-tests"*) exit 0 ;;
+  esac
+  # Monitoring/probe commands
+  case "$CMD" in
+    *"tailscale status"*|*"access-matrix"*) exit 0 ;;
+  esac
+  # Setting/clearing the active ticket
+  case "$CMD" in
+    *active-ticket*) exit 0 ;;
+  esac
+fi
+
+# For edit/write: exempt policy/hook files (these ARE the policy)
+FILE_PATH="${CRUSH_TOOL_INPUT_FILE_PATH:-}"
+case "$FILE_PATH" in
+  */AGENTS.md|*/check-rules.sh|*/crush.json|*/hooks/*)
+    if [ "$TOOL" = "write" ] || [ "$TOOL" = "edit" ] || [ "$TOOL" = "multiedit" ]; then
+      exit 0
+    fi
+    ;;
+esac
+
+# Check for active ticket
+if [ -f "$TICKET_FILE" ] && [ -s "$TICKET_FILE" ]; then
+  TICKET=$(cat "$TICKET_FILE")
+  printf '{"context":"Active ticket: %s"}\n' "$TICKET"
+  exit 0
+fi
+
+# No active ticket — block
+cat >&2 <<'MSG'
+TICKET GATE: No active ticket set.
+
+This project requires ticket-governed work (AGENTS.md Agent Authority).
+Before modifying systems or code, set the active ticket:
+
+  echo '#NNN' > .crush/active-ticket
+
+If no ticket exists yet, create one first (redmine-cli create), then set it.
+Clear the ticket when work is complete:
+
+  > .crush/active-ticket
+MSG
+exit 2

k8s/install-cp.sh

@@ -48,7 +48,9 @@ curl -sfL https://get.k3s.io | sh -s - server \
   $tls_san_flags \
   --flannel-backend=vxlan \
   --etcd-snapshot-schedule-cron='0 */6 * * *' \
-  --egress-selector-mode=agent
+  --egress-selector-mode=agent \
+  --etcd-arg heartbeat-interval=1000 \
+  --etcd-arg election-timeout=5000
 REMOTE_BOOT
 
 echo "  cnode1 install submitted."
@@ -117,7 +119,9 @@ curl -sfL https://get.k3s.io | sh -s - server \
   --advertise-address=$node_ip \
   $tls_san_flags \
   --flannel-backend=vxlan \
-  --egress-selector-mode=agent
+  --egress-selector-mode=agent \
+  --etcd-arg heartbeat-interval=1000 \
+  --etcd-arg election-timeout=5000
 REMOTE_JOIN
 
   echo "  $node_name install submitted."

k8s/post-setup.sh

@@ -66,10 +66,20 @@ echo "    export KUBECONFIG=$KUBECONFIG_FILE"
 echo "    kubectl get nodes"
 
 # -------------------------------------------------------
-# 3. Verify cluster health
+# 3. Deploy tuned (network-latency profile) on all cnodes
 # -------------------------------------------------------
 echo ""
-echo "--- [3/3] Verifying cluster health ---"
+echo "--- [3/4] Deploying tuned (network-latency) on cnodes ---"
+for ip in "${ALL_CNODES[@]}"; do
+  echo "  $ip..."
+  cn "$ip" 'DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; tuned-adm profile network-latency; systemctl enable tuned; systemctl restart tuned; tuned-adm active' 2>&1 | tail -1
+done
+
+# -------------------------------------------------------
+# 4. Verify cluster health
+# -------------------------------------------------------
+echo ""
+echo "--- [4/4] Verifying cluster health ---"
 
 export KUBECONFIG="$KUBECONFIG_FILE"
 

netinfra/dhcp/dhcpd-primary.conf

@@ -45,11 +45,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
 
   # ---- host reservations (fixed-address; not subject to failover pool) ----
 
-  host pfv-r3-mgmt {
+  host pfv-r3-tor-mgmt-01 {
     hardware ethernet 00:14:22:69:1c:37;
     fixed-address 192.168.0.7;
   }
-  host pfv-r3-stor {
+  host pfv-r3-tor-stor-01 {
     hardware ethernet 00:13:72:46:95:e4;
     fixed-address 192.168.0.9;
   }
@@ -57,11 +57,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet 40:9f:38:b0:b5:2f;
     fixed-address 192.168.1.84;
   }
-  host pfv-r2-tor1 {
+  host pfv-r2-tor-01 {
     hardware ethernet 00:0d:56:41:7a:4d;
     fixed-address 192.168.0.10;
   }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
     hardware ethernet a4:ba:db:6f:ce:28;
     fixed-address 192.168.0.12;
   }
@@ -177,11 +177,66 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet a4:bb:6d:e3:56:86;
     fixed-address 192.168.3.58;
   }
+	# umbrel
+	host tsys-umbrel {
+		hardware ethernet 02:2E:FF:8E:A2:D2;
+		fixed-address 192.168.1.97;
+		}
+	# ultix-streaming
+	host ultix-streaming {
+		hardware ethernet bc:24:11:1a:8f:6f;
+		fixed-address 192.168.3.78;
+		}
+	# ultix-offstage
+	host ultix-offstge {
+		hardware ethernet bc:24:11:1f:9d:83;
+		fixed-address 192.168.3.79;
+		}
+	# ultix-highside
+	host ultix-highside {
+		hardware ethernet a0:4a:5e:ca:46:f3;
+		fixed-address 192.168.3.32;
+		}
+	# pfv-k8s-cnode1
+	host pfv-k8s-cnode1 {
+		hardware ethernet bc:24:11:cb:97:10;
+		fixed-address 192.168.1.91;
+		}
+	# pfv-k8s-cnode2
+	host pfv-k8s-cnode2 {
+		hardware ethernet bc:24:11:40:25:f8;
+		fixed-address 192.168.3.113;
+		}
+	# pfv-k8s-cnode3
+	host pfv-k8s-cnode3 {
+		hardware ethernet bc:24:11:38:c0:58;
+		fixed-address 192.168.1.228;
+		}
+	# devbox-cloudron
+	host devbox-cloudron {
+		hardware ethernet bc:24:11:f7:b1:07;
+		fixed-address 192.168.1.6;
+		}
+	# hfnoc-uisp
+	host hfnoc-uisp {
+		hardware ethernet bc:24:11:a3:87:61;
+		fixed-address 192.168.3.193;
+		}
+	# kali-rd
+	host kali-rd {
+		hardware ethernet bc:24:11:9e:1c:e9;
+		fixed-address 192.168.2.37;
+		}
+	# kali-tsys
+	host kali-tsys {
+		hardware ethernet bc:24:11:16:22:d4;
+		fixed-address 192.168.1.114;
+		}
 }
 
 # ---- host declarations outside subnet (global scope, same as netboot) ----
 
-host subodev-torsw01 {
+host pfv-r6-mgmt-01 {
   hardware ethernet 00:14:22:69:18:a7;
   fixed-address 192.168.0.8;
 }
@@ -189,7 +244,80 @@ host pfv-r1-tor-top {
   hardware ethernet 00:23:ae:c1:ad:e8;
   fixed-address 192.168.0.11;
 }
-host tailscale-router {
-  hardware ethernet bc:24:11:8a:69:04;
-  fixed-address 192.168.3.16;
-}
+
+# --- VM DHCP reservations (generated 2026-08-11, ticket #420) ---
+# All pinned to current ARP-observed IPs. No forward DNS needed
+# (forward records point to Tailscale 100.x addresses).
+
+  host tsys-ca {
+    hardware ethernet bc:24:11:32:d0:36;
+    fixed-address 192.168.1.181;
+  }
+  host pfv-netinfra-01 {
+    hardware ethernet bc:24:11:65:b2:ac;
+    fixed-address 192.168.3.252;
+  }
+  host pfv-netinfra-02 {
+    hardware ethernet bc:24:11:e4:37:53;
+    fixed-address 192.168.3.253;
+  }
+  host tsys-librenms {
+    hardware ethernet bc:24:11:5c:96:1e;
+    fixed-address 192.168.3.176;
+  }
+  host tsys-proxmox-datacenter {
+    hardware ethernet bc:24:11:e6:03:2d;
+    fixed-address 192.168.2.44;
+  }
+  host pfv-k8s-wnode-tsys3 {
+    hardware ethernet bc:24:11:ee:7e:7b;
+    fixed-address 192.168.1.98;
+  }
+  host pfv-proxmox-backup-server {
+    hardware ethernet bc:24:11:6e:12:69;
+    fixed-address 192.168.2.193;
+  }
+  host pfv-k8s-wnode-tsys5 {
+    hardware ethernet bc:24:11:c7:a8:6c;
+    fixed-address 192.168.1.5;
+  }
+  host preprod-hfnoc-uisp {
+    hardware ethernet bc:24:11:74:d6:8a;
+    fixed-address 192.168.3.192;
+  }
+  host tsys-awx {
+    hardware ethernet bc:24:11:80:0d:16;
+    fixed-address 192.168.3.115;
+  }
+  host pfv-rr-middleware-02 {
+    hardware ethernet bc:24:11:96:0e:ee;
+    fixed-address 192.168.1.117;
+  }
+  host tsys-proxmox-mailgw-01 {
+    hardware ethernet bc:24:11:56:61:18;
+    fixed-address 192.168.1.11;
+  }
+  host pfv-k8s-wnode-tsys7 {
+    hardware ethernet bc:24:11:30:b8:07;
+    fixed-address 192.168.1.109;
+  }
+  host pfv-rr-middleware-01 {
+    hardware ethernet bc:24:11:1e:61:cf;
+    fixed-address 192.168.1.110;
+  }
+  host tsys-voip {
+    hardware ethernet bc:24:11:23:ce:04;
+    fixed-address 192.168.1.70;
+  }
+  host tsys-proxmox-mailgw-02 {
+    hardware ethernet bc:24:11:5f:e5:2c;
+    fixed-address 192.168.1.10;
+  }
+  host pfv-k8s-wnode-tsys6 {
+    hardware ethernet bc:24:11:fa:6e:b5;
+    fixed-address 192.168.1.111;
+  }
+  host tsys-siem-new {
+    hardware ethernet bc:24:11:ee:67:e2;
+    fixed-address 192.168.1.223;
+  }

netinfra/dhcp/dhcpd-secondary.conf

@@ -59,7 +59,7 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet 00:0d:56:41:7a:4d;
     fixed-address 192.168.0.10;
   }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
     hardware ethernet a4:ba:db:6f:ce:28;
     fixed-address 192.168.0.12;
   }

netinfra/pihole/docker-compose.yml

@@ -4,7 +4,7 @@ services:
     # Root cause of the 2026-08 gravity.db corruption: default /dev/shm (64M)
     # was too small for FTL's shared-memory metrics. 1024M has been stable.
     shm_size: '1024M'
-    image: pihole/pihole:latest
+    image: pihole/pihole:2026.07.0
     hostname: pihole
     entrypoint: ["/usr/local/bin/gravity-validate.sh"]
     ports:
@@ -42,7 +42,7 @@ services:
       - dnsnet
   autoheal:
     container_name: autoheal
-    image: willfarrell/autoheal:latest
+    image: willfarrell/autoheal:1.2.0
     environment:
       AUTOHEAL_CONTAINER_LABEL: autoheal
       AUTOHEAL_INTERVAL: 30

netinfra/switches/core-sw01-ch1-to-lacp.cmds

@@ -0,0 +1,25 @@
+! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
+! Switch: core-sw01 (Dell PowerConnect 5448, rack 5)
+! WHEN: Friday maintenance window — BOTH switches must change together
+! RISK: Brief storage-net outage during transition (seconds)
+!
+! Current: g13-g16 in ch1, mode=on (static, no failure detection)
+! Target:  g13-g16 in ch1, mode=auto (LACP active partner negotiation)
+!
+! NOTE: core-sw01 hash stays layer-2-3 (hardware limit — best available)
+!       tor3-stor hash stays layer-2-3-4 (already set)
+!
+! IMPORTANT: Run this SIMULTANEOUSLY with tor3-stor change.
+!   If one side is LACP and other is static, trunk goes down until
+!   both sides match. Plan for ~30s storage-net outage.
+!
+enable
+configure
+interface range ethernet g13-g16
+  no channel-group
+  channel-group 1 mode auto
+exit
+exit
+show interfaces status port-channel 1
+show lacp port-channel 1
+copy running-config startup-config

netinfra/switches/pfv-r3-tor-stor-01-mac.cmds

@@ -0,0 +1,5 @@
+! pfv-r3-tor-stor-01 — MAC table + port details (Dell PowerConnect 5324 commands)
+enable
+show bridge addressing-table address
+show interfaces description
+show interfaces status

netinfra/switches/pfv-r3-tor-stor-01.cmds

@@ -1,4 +1,4 @@
-! pfv-tor3-stor — Neyland 24T (Radlan-based, rack 3 storage TOR)
+! pfv-r3-tor-stor-01 — Neyland 24T (Radlan-based, rack 3 storage TOR)
 ! Radlan CLI uses different keywords than DNOS
 enable
 show system

netinfra/switches/pfv-r5-core-01-mac.cmds

@@ -0,0 +1,7 @@
+! pfv-r5-core-01 — MAC address table + LLDP neighbors
+terminal datadump
+enable
+show mac-address-table
+show lldp info
+show lldp neighbors
+show interfaces status port-channel

netinfra/switches/pfv-r5-core-01.cmds

@@ -1,4 +1,4 @@
-! pfv-core-sw01 — Dell PowerConnect 5448 (core switch, rack 5)
+! pfv-r5-core-01 — Dell PowerConnect 5448 (core switch, rack 5)
 ! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down)
 terminal datadump
 enable

netinfra/switches/tor3-stor-ch1-to-lacp.cmds

@@ -0,0 +1,24 @@
+! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
+! Switch: pfv-r3-tor-stor (Dell PowerConnect 5324 / "Neyland 24T", rack 3)
+! WHEN: Friday maintenance window — BOTH switches must change together
+! RISK: Brief storage-net outage during transition (seconds)
+!
+! Current: g20-g23 in ch1, mode=on (static, no failure detection)
+! Target:  g20-g23 in ch1, LACP active
+!
+! Radlan CLI syntax (different from DNOS on core-sw01)
+! Verify exact keywords on this switch first — 'show running-config'
+! will show the current ch1 config to confirm syntax.
+!
+! IMPORTANT: Run this SIMULTANEOUSLY with core-sw01 change.
+!
+enable
+configure
+interface range ethernet g20-g23
+  no channel-group
+  channel-group 1 mode active
+exit
+exit
+show interfaces status port-channel 1
+show lacp port-channel 1
+copy running-config startup-config

oam/netdisco/deployment.yml

@@ -0,0 +1,41 @@
+# NetDisco deployment configuration [#337]
+# SNMP community: kn3lmgmt (v2c)
+
+# Database connection (matched to docker-compose services)
+database:
+  host: netdisco-postgresql
+  dbname: netdisco
+  user: netdisco
+  pass: netdisco
+
+# SNMP authentication
+snmp_auth:
+  - tag: default_v2c
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 2
+
+  - tag: default_v1
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 1
+
+# Job scheduling
+schedule:
+  discoverall:
+    first: 60
+    every: 3600
+  macwalk:
+    first: 120
+    every: 900
+  arpwalk:
+    first: 150
+    every: 900
+  nbtwalk:
+    first: 180
+    every: 900
+
+# Node freshness (keep history)
+node_freshness: 0

oam/netdisco/docker-compose.yml

@@ -0,0 +1,44 @@
+services:
+  netdisco-postgresql:
+    image: netdisco/netdisco:2.102001-postgresql
+    container_name: oam-netdisco-db
+    environment:
+      - POSTGRES_USER=netdisco
+      - POSTGRES_PASSWORD=netdisco
+      - POSTGRES_DB=netdisco
+    volumes:
+      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
+    restart: unless-stopped
+
+  netdisco-backend:
+    image: netdisco/netdisco:2.102001-backend
+    container_name: oam-netdisco-backend
+    depends_on:
+      - netdisco-postgresql
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+      - NETDISCO_SNMP_COMMUNITY=kn3lmgmt
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
+    restart: unless-stopped
+
+  netdisco-web:
+    image: netdisco/netdisco:2.102001-web
+    container_name: oam-netdisco-web
+    depends_on:
+      - netdisco-postgresql
+      - netdisco-backend
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    ports:
+      - "8082:5000"
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+    restart: unless-stopped

oam/netdisco/setup.sh

@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy NetDisco on tsys-librenms
+# Part of OAM platform [#337]
+# L2 network discovery and mapping via SNMP
+
+echo "=== Setting up NetDisco ==="
+
+mkdir -p /opt/oam/netdisco/{config,data,pgdata}
+
+# Write deployment config
+cat > /opt/oam/netdisco/config/deployment.yml <<'YMLEOF'
+# NetDisco deployment configuration [#337]
+# SNMP community: kn3lmgmt (v2c)
+
+database:
+  host: netdisco-postgresql
+  dbname: netdisco
+  user: netdisco
+  pass: netdisco
+
+snmp_auth:
+  - tag: default_v2c
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 2
+
+  - tag: default_v1
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 1
+
+schedule:
+  discoverall:
+    first: 60
+    every: 3600
+  macwalk:
+    first: 120
+    every: 900
+  arpwalk:
+    first: 150
+    every: 900
+  nbtwalk:
+    first: 180
+    every: 900
+
+node_freshness: 0
+YMLEOF
+
+# Write docker-compose
+cat > /opt/oam/netdisco/docker-compose.yml <<'DCEOF'
+services:
+  netdisco-postgresql:
+    image: netdisco/netdisco:2.102001-postgresql
+    container_name: oam-netdisco-db
+    environment:
+      - POSTGRES_USER=netdisco
+      - POSTGRES_PASSWORD=netdisco
+      - POSTGRES_DB=netdisco
+    volumes:
+      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
+    restart: unless-stopped
+
+  netdisco-backend:
+    image: netdisco/netdisco:2.102001-backend
+    container_name: oam-netdisco-backend
+    depends_on:
+      - netdisco-postgresql
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
+    restart: unless-stopped
+
+  netdisco-web:
+    image: netdisco/netdisco:2.102001-web
+    container_name: oam-netdisco-web
+    depends_on:
+      - netdisco-postgresql
+      - netdisco-backend
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    ports:
+      - "8082:5000"
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling NetDisco images ==="
+cd /opt/oam/netdisco && docker compose pull 2>&1
+
+echo "=== Starting NetDisco stack ==="
+cd /opt/oam/netdisco && docker compose up -d 2>&1
+
+echo "=== Waiting for PostgreSQL to initialize (30s) ==="
+sleep 30
+
+echo "=== Container status ==="
+docker ps --filter name=oam-netdisco --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
+
+echo "=== Adding devices to NetDisco ==="
+for dev in pfv-r5-core-01.knel.net pfv-r3-tor-mgmt-01.knel.net pfv-r3-tor-stor-01.knel.net pfv-r6-mgmt-01.knel.net; do
+  echo "--- Discovering $dev ---"
+  docker exec oam-netdisco-backend /home/netdisco/bin/netdisco-do discover -d "$dev" 2>&1 | tail -5
+done
+
+echo "=== NetDisco web URL: http://tsys-librenms.knel.net:8082 ==="
+echo "=== Done ==="

oam/oxidized/config

@@ -0,0 +1,38 @@
+---
+username: 'oxidized'
+password: 'see ~/.creds/oxidized.env'
+model: powerconnect
+interval: 3600
+use_syslog: false
+debug: false
+threads: 30
+timeout: 20
+retries: 3
+prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
+rest: 0.0.0.0:8888
+next_adds_job: false
+models: {}
+pid: /home/oxidized/.config/oxidized/pid
+log: /home/oxidized/.config/oxidized/logs/log
+snmp: false
+groups: {}
+models: {}
+resolve_dns: true
+interval: 3600
+output:
+  default: git
+  git:
+    user: oxidized
+    email: oxidized@tsys-librenms.knel.net
+    repo: /home/oxidized/.config/oxidized/git-repos
+source:
+  default: csv
+  csv:
+    file: /home/oxidized/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    gpg: false

oam/oxidized/docker-compose.yml

@@ -0,0 +1,12 @@
+services:
+  oxidized:
+    image: oxidized/oxidized:0.30.1
+    container_name: oam-oxidized
+    environment:
+      - OXIDIZED_CONFIG=/home/oxidized/.config/oxidized/config
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
+    ports:
+      - "8083:8888"
+    restart: unless-stopped

oam/oxidized/hooks/git-push.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Oxidized post-push hook: auto-push to Gitea after each config change
+# Mounted into container at /home/oxidized/.config/oxidized/hooks/checkout/git-push.sh
+set -euo pipefail
+cd "$1"
+export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new -i /home/oxidized/.config/oxidized/.ssh/id_ed25519"
+git push origin master 2>&1 || true

oam/oxidized/router.db

@@ -0,0 +1,4 @@
+pfv-r5-core-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r3-tor-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r3-tor-stor-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r6-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p

oam/oxidized/setup.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy Oxidized on tsys-librenms
+# Part of OAM platform [#337]
+# Network device config backup/versioning
+
+echo "=== Setting up Oxidized ==="
+
+mkdir -p /opt/oam/oxidized/{config,git-repos,logs}
+
+# Config file
+cat > /opt/oam/oxidized/config/config <<'CFGEOF'
+---
+username: 'CHANGE_ME'
+password: 'CHANGE_ME'
+model: powerconnect
+interval: 3600
+use_syslog: false
+debug: false
+threads: 30
+timeout: 20
+retries: 3
+prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
+rest: 0.0.0.0:8888
+next_adds_job: false
+pid: /home/oxidized/.config/oxidized/pid
+log: /home/oxidized/.config/oxidized/logs/log
+snmp: false
+resolve_dns: true
+interval: 3600
+output:
+  default: git
+  git:
+    user: oxidized
+    email: oxidized@tsys-librenms.knel.net
+    repo: /home/oxidized/.config/oxidized/git-repos
+source:
+  default: csv
+  csv:
+    file: /home/oxidized/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    gpg: false
+CFGEOF
+
+# Router DB (device list)
+cat > /opt/oam/oxidized/config/router.db <<'DBEOF'
+pfv-r5-core-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r3-tor-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r3-tor-stor-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r6-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+DBEOF
+
+# Docker compose
+cat > /opt/oam/oxidized/docker-compose.yml <<'DCEOF'
+services:
+  oxidized:
+    image: oxidized/oxidized:0.30.1
+    container_name: oam-oxidized
+    environment:
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
+    ports:
+      - "8083:8888"
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling Oxidized image ==="
+cd /opt/oam/oxidized && timeout 120 docker compose pull 2>&1
+
+echo "=== Starting Oxidized ==="
+cd /opt/oam/oxidized && docker compose up -d 2>&1
+
+echo "=== Status ==="
+docker ps --filter name=oam-oxidized --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
+
+echo ""
+echo "=== IMPORTANT ==="
+echo "Oxidized deployed with placeholder credentials (CHANGE_ME)."
+echo "Update /opt/oam/oxidized/config/router.db with real switch credentials"
+echo "(username:password per device) to enable config backups."
+echo "Then restart: cd /opt/oam/oxidized && docker compose restart"
+echo ""
+echo "=== Oxidized REST API: http://tsys-librenms.knel.net:8083 ==="

oam/smokeping/General

@@ -0,0 +1,84 @@
+*** General ***
+
+owner = Charles N Wyble
+contact = reachableceo@knownelement.com
+mailhost = localhost
+cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
+syslogfacility = local0
+# each probe run every 5 minutes, 300 samples = 25 hours at 5min steps
+pagedir = /data/pages
+piddir  = /tmp
+imgcache = /data/cache
+imgurl  = /smokeping/images
+datadir = /data/_data
+dyndir = /data/dyn
+
+*** Database ***
+
+step     = 300
+pings    = 20
+
+# consfn mrhb steps total
+
+AVERAGE  0.5   1  100800
+AVERAGE  0.5  12  43200
+    MIN  0.5  12  43200
+    MAX  0.5  12  43200
+AVERAGE  0.5 144   7200
+    MAX  0.5 144   7200
+    MIN  0.5 144   7200
+
+*** Presentation ***
+
+template = /etc/smokeping/basepage.html
+
++ charts
+
++ detail
+width = 600
+height = 200
+nradius = 5
+
+*** Probes ***
+
++ FPing
+
+binary = /usr/local/bin/fping
+hostinterval = 1.5
+mininterval = 0.001
+offset = 0%
+
+*** Alerts ***
+
+to = reachableceo@knownelement.com
+from = smokeping-alert@tsys-librenms.knel.net
+
++bigloss
+type = loss
+# in percent
+pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
+comment = suddenly there is packet loss
+
++somedoss
+type = loss
+# in percent
+pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
+comment = incoming packet loss
+
++startloss
+type = loss
+# in percent
+pattern = >0%,>0%,>0%
+comment = loss at startup
+
++rttdetect
+type = rtt
+# in milliseconds
+pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
+comment = routing change?
+
++rttbadd
+type = rtt
+# in milliseconds
+pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
+comment = reroute?

oam/smokeping/Targets

@@ -0,0 +1,111 @@
+*** Targets ***
+
+probe = FPing
+
+menu = Top
+title = Network Latency Monitor
+
++ Infrastructure
+menu = Infrastructure
+title = Core Infrastructure
+
+++ pfv-r5-core-01
+menu = pfv-r5-core-01 (Core Switch)
+title = Dell PowerConnect 5448 - Rack 5 Core
+host = pfv-r5-core-01.knel.net
+
+++ pfv-r3-tor-mgmt-01
+menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
+title = Neyland 24T - Rack 3 Management TOR
+host = pfv-r3-tor-mgmt-01.knel.net
+
+++ pfv-r3-tor-stor-01
+menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
+title = Neyland 24T - Rack 3 Storage TOR
+host = pfv-r3-tor-stor-01.knel.net
+
+++ pfv-r6-mgmt-01
+menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
+title = Neyland 24T - Rack 6 Management Switch
+host = pfv-r6-mgmt-01.knel.net
+
++ Servers
+menu = Servers
+title = Server Infrastructure
+
+++ pfv-stor1
+menu = pfv-stor1
+title = Storage Server 1
+host = pfv-stor1.knel.net
+
+++ pfv-bms
+menu = pfv-bms (HomeAssistant)
+title = Home Assistant / UniFi Controller
+host = pfv-bms.knel.net
+
+++ tsys-librenms
+menu = tsys-librenms
+title = LibreNMS Monitoring Server
+host = tsys-librenms.knel.net
+
++ Wireless
+menu = Wireless
+title = Wireless Access Points
+
+++ ap-tablemount
+menu = ap-tablemount
+title = UniFi AP-AC-Lite (Table Mount)
+host = ap-tablemount.knel.net
+
+++ ap-wallmount
+menu = ap-wallmount
+title = UniFi AP-AC-LR (Wall Mount)
+host = ap-wallmount.knel.net
+
++ ProxmoxHosts
+menu = Proxmox Hosts
+title = Proxmox Hypervisor Nodes
+
+++ pfv-tsys1
+menu = pfv-tsys1
+title = Proxmox Host - Infrastructure
+host = pfv-tsys1.knel.net
+
+++ pfv-tsys3
+menu = pfv-tsys3
+title = Proxmox Host - Compute
+host = pfv-tsys3.knel.net
+
+++ pfv-tsys4
+menu = pfv-tsys4
+title = Proxmox Host - Storage + PBS
+host = pfv-tsys4.knel.net
+
+++ pfv-tsys5
+menu = pfv-tsys5
+title = Proxmox Host - Storage + Sandbox
+host = pfv-tsys5.knel.net
+
+++ pfv-tsys6
+menu = pfv-tsys6
+title = Proxmox Host - RackRental
+host = pfv-tsys6.knel.net
+
+++ pfv-tsys7
+menu = pfv-tsys7
+title = Proxmox Host - RackRental
+host = pfv-tsys7.knel.net
+
+++ pfv-tsys9
+menu = pfv-tsys9
+title = Proxmox Host - Infra + Compute
+host = pfv-tsys9.knel.net
+
++ Power
+menu = Power
+title = Power Devices
+
+++ pfv-garage-pdu-01
+menu = pfv-garage-pdu-1
+title = APC PDU (Garage)
+host = pfv-garage-pdu-1.knel.net

oam/smokeping/docker-compose.yml

@@ -0,0 +1,14 @@
+services:
+  smokeping:
+    image: linuxserver/smokeping:2.9.0
+    container_name: oam-smokeping
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/smokeping/config:/config
+      - /opt/oam/smokeping/data:/data
+    ports:
+      - "8081:80"
+    restart: unless-stopped

oam/smokeping/setup.sh

@@ -0,0 +1,237 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy Smokeping on tsys-librenms
+# Part of OAM platform [#337]
+
+echo "=== Setting up Smokeping ==="
+
+# Ensure directories
+mkdir -p /opt/oam/smokeping/{config,data}
+
+# General config
+cat > /opt/oam/smokeping/config/General <<'GENEOF'
+*** General ***
+
+owner = Charles N Wyble
+contact = reachableceo@knownelement.com
+mailhost = localhost
+cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
+syslogfacility = local0
+pagedir = /data/pages
+piddir  = /tmp
+imgcache = /data/cache
+imgurl  = /smokeping/images
+datadir = /data/_data
+dyndir = /data/dyn
+
+*** Database ***
+
+step     = 300
+pings    = 20
+
+AVERAGE  0.5   1  100800
+AVERAGE  0.5  12  43200
+    MIN  0.5  12  43200
+    MAX  0.5  12  43200
+AVERAGE  0.5 144   7200
+    MAX  0.5 144   7200
+    MIN  0.5 144   7200
+
+*** Presentation ***
+
+template = /etc/smokeping/basepage.html
+
++ charts
+
++ detail
+width = 600
+height = 200
+nradius = 5
+
+*** Probes ***
+
++ FPing
+
+binary = /usr/local/bin/fping
+hostinterval = 1.5
+mininterval = 0.001
+offset = 0%
+
+*** Alerts ***
+
+to = reachableceo@knownelement.com
+from = smokeping-alert@tsys-librenms.knel.net
+
++bigloss
+type = loss
+pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
+comment = suddenly there is packet loss
+
++somedoss
+type = loss
+pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
+comment = incoming packet loss
+
++startloss
+type = loss
+pattern = >0%,>0%,>0%
+comment = loss at startup
+
++rttdetect
+type = rtt
+pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
+comment = routing change?
+
++rttbadd
+type = rtt
+pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
+comment = reroute?
+GENEOF
+
+# Targets config
+cat > /opt/oam/smokeping/config/Targets <<'TGTEOF'
+*** Targets ***
+
+probe = FPing
+
+menu = Top
+title = Network Latency Monitor
+
++ Infrastructure
+menu = Infrastructure
+title = Core Infrastructure
+
+++ pfv-r5-core-01
+menu = pfv-r5-core-01 (Core Switch)
+title = Dell PowerConnect 5448 - Rack 5 Core
+host = pfv-r5-core-01.knel.net
+
+++ pfv-r3-tor-mgmt-01
+menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
+title = Neyland 24T - Rack 3 Management TOR
+host = pfv-r3-tor-mgmt-01.knel.net
+
+++ pfv-r3-tor-stor-01
+menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
+title = Neyland 24T - Rack 3 Storage TOR
+host = pfv-r3-tor-stor-01.knel.net
+
+++ pfv-r6-mgmt-01
+menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
+title = Neyland 24T - Rack 6 Management Switch
+host = pfv-r6-mgmt-01.knel.net
+
++ Servers
+menu = Servers
+title = Server Infrastructure
+
+++ pfv-stor1
+menu = pfv-stor1
+title = Storage Server 1
+host = pfv-stor1.knel.net
+
+++ pfv-bms
+menu = pfv-bms (HomeAssistant)
+title = Home Assistant / UniFi Controller
+host = pfv-bms.knel.net
+
+++ tsys-librenms
+menu = tsys-librenms
+title = LibreNMS Monitoring Server
+host = tsys-librenms.knel.net
+
++ Wireless
+menu = Wireless
+title = Wireless Access Points
+
+++ ap-tablemount
+menu = ap-tablemount
+title = UniFi AP-AC-Lite (Table Mount)
+host = ap-tablemount.knel.net
+
+++ ap-wallmount
+menu = ap-wallmount
+title = UniFi AP-AC-LR (Wall Mount)
+host = ap-wallmount.knel.net
+
++ ProxmoxHosts
+menu = Proxmox Hosts
+title = Proxmox Hypervisor Nodes
+
+++ pfv-tsys1
+menu = pfv-tsys1
+title = Proxmox Host - Infrastructure
+host = pfv-tsys1.knel.net
+
+++ pfv-tsys3
+menu = pfv-tsys3
+title = Proxmox Host - Compute
+host = pfv-tsys3.knel.net
+
+++ pfv-tsys4
+menu = pfv-tsys4
+title = Proxmox Host - Storage + PBS
+host = pfv-tsys4.knel.net
+
+++ pfv-tsys5
+menu = pfv-tsys5
+title = Proxmox Host - Storage + Sandbox
+host = pfv-tsys5.knel.net
+
+++ pfv-tsys6
+menu = pfv-tsys6
+title = Proxmox Host - RackRental
+host = pfv-tsys6.knel.net
+
+++ pfv-tsys7
+menu = pfv-tsys7
+title = Proxmox Host - RackRental
+host = pfv-tsys7.knel.net
+
+++ pfv-tsys9
+menu = pfv-tsys9
+title = Proxmox Host - Infra + Compute
+host = pfv-tsys9.knel.net
+
++ Power
+menu = Power
+title = Power Devices
+
+++ pfv-garage-pdu-01
+menu = pfv-garage-pdu-1
+title = APC PDU (Garage)
+host = pfv-garage-pdu-1.knel.net
+TGTEOF
+
+# Docker compose
+cat > /opt/oam/smokeping/docker-compose.yml <<'DCEOF'
+services:
+  smokeping:
+    image: linuxserver/smokeping:2.9.0
+    container_name: oam-smokeping
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/smokeping/config:/config
+      - /opt/oam/smokeping/data:/data
+    ports:
+      - "8081:80"
+    restart: unless-stopped
+DCEOF
+
+echo "=== Starting Smokeping container ==="
+cd /opt/oam/smokeping && docker compose up -d 2>&1
+
+echo "=== Waiting for startup ==="
+sleep 5
+
+echo "=== Container status ==="
+docker ps --filter name=oam-smokeping --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
+
+echo "=== Smokeping logs (last 20) ==="
+docker logs oam-smokeping --tail 20 2>&1
+
+echo "=== Done ==="

oam/unpoller/setup.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy UNPoller on tsys-librenms
+# Part of OAM platform [#337]
+# UniFi controller polling for metrics export
+
+echo "=== Setting up UNPoller ==="
+
+mkdir -p /opt/oam/unpoller
+
+# Write config
+cat > /opt/oam/unpoller/up.conf <<'CONFEOF'
+[poller]
+debug = false
+quiet = false
+plugins = []
+
+[unifi.defaults]
+url = "https://pfv-bms.knel.net:8443"
+user = "CHANGE_ME"
+pass = "CHANGE_ME"
+sites = ["all"]
+ssl_verify = false
+
+[unifi.defaults.save_dpi]
+enabled = true
+
+[influxdb]
+enable = false
+url = "http://tsys-librenms.knel.net:8086"
+db = "unifi"
+
+[datadog]
+enable = false
+
+[prometheus]
+enable = true
+http_listen = ":9130"
+report_errors = true
+
+[loki]
+enable = false
+CONFEOF
+
+# Docker compose
+cat > /opt/oam/unpoller/docker-compose.yml <<'DCEOF'
+services:
+  unpoller:
+    image: ghcr.io/unpoller/unpoller:latest
+    container_name: oam-unpoller
+    environment:
+      - UP_UNIFI_DEFAULT_URL=https://pfv-bms.knel.net:8443
+      - UP_UNIFI_DEFAULT_USER=CHANGE_ME
+      - UP_UNIFI_DEFAULT_PASS=CHANGE_ME
+      - UP_UNIFI_DEFAULT_VERIFY_SSL=false
+      - UP_PROMETHEUS_ENABLE=true
+      - UP_PROMETHEUS_HTTP_LISTEN=:9130
+      - TZ=America/Chicago
+    ports:
+      - "8084:9130"
+    volumes:
+      - /opt/oam/unpoller/up.conf:/config/unifi-poller.conf:ro
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling UNPoller image ==="
+cd /opt/oam/unpoller && timeout 120 docker compose pull 2>&1
+
+echo "=== Starting UNPoller ==="
+cd /opt/oam/unpoller && docker compose up -d 2>&1
+
+echo "=== Status ==="
+docker ps --filter name=oam-unpoller --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
+
+echo ""
+echo "=== IMPORTANT ==="
+echo "UNPoller deployed with placeholder UniFi credentials (CHANGE_ME)."
+echo "Update UP_UNIFI_DEFAULT_USER and UP_UNINI_DEFAULT_PASS env vars"
+echo "in /opt/oam/unpoller/docker-compose.yml with real UniFi controller"
+echo "credentials, then restart: cd /opt/oam/unpoller && docker compose up -d"
+echo ""
+echo "=== UNPoller Prometheus metrics: http://tsys-librenms.knel.net:8084 ==="

oam/weathermap/network.conf

@@ -0,0 +1,57 @@
+WIDTH 800
+HEIGHT 600
+HTMLOUTPUTFILE output/network.html
+IMAGEOUTPUTFILE output/network.png
+BACKGROUNDCOLOR 255 255 255
+TITLE "PFV Cluster Network Map"
+TIMEPOS 0 0
+TITLECOLOR 0 0 0
+TIMECOLOR 0 0 0
+HTMLSTYLE overlib
+KEYPOS DEFAULT 400 580
+KEYTEXTCOLOR 0 0 0
+KEYOUTLINECOLOR 0 0 0
+KEYBGCOLOR 255 255 255
+SCALE DEFAULT 0 10 0 255 0
+SCALE DEFAULT 10 50 255 255 0
+SCALE DEFAULT 50 100 255 128 0
+SCALE DEFAULT 100 1000 255 0 0
+NODE core_sw01
+  LABEL pfv-r5-core-01
+  POSITION 400 200
+NODE tor_mgmt
+  LABEL pfv-r3-tor-mgmt-01
+  POSITION 200 350
+NODE tor_stor
+  LABEL pfv-r3-tor-stor-01
+  POSITION 400 350
+NODE r6_mgmt
+  LABEL pfv-r6-mgmt-01
+  POSITION 600 350
+NODE pfv_tsys1
+  LABEL pfv-tsys1
+  POSITION 200 100
+NODE pfv_tsys9
+  LABEL pfv-tsys9
+  POSITION 600 100
+NODE pfv_bms
+  LABEL pfv-bms
+  POSITION 100 200
+LINK core_to_mgmt
+  NODES core_sw01 tor_mgmt
+  BANDWIDTH 1000M
+LINK core_to_stor
+  NODES core_sw01 tor_stor
+  BANDWIDTH 1000M
+LINK core_to_r6
+  NODES core_sw01 r6_mgmt
+  BANDWIDTH 1000M
+LINK tsys1_to_core
+  NODES pfv_tsys1 core_sw01
+  BANDWIDTH 1000M
+LINK tsys9_to_core
+  NODES pfv_tsys9 core_sw01
+  BANDWIDTH 1000M
+LINK bms_to_core
+  NODES pfv_bms core_sw01
+  BANDWIDTH 1000M

proxmox/perf/iperf-full-matrix.sh

@@ -288,6 +288,6 @@ echo "===== CONTEXT ====="
 echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
 echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
 echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
-echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
+echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor-01 → pfv-r5-core-01)"
 echo ""
 echo "All logs in: $LOG_DIR/"

proxmox/perf/scripts/add-datanet-nics.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+###############################################################################
+# add-datanet-nics.sh
+#
+# Adds a second virtio NIC (net1) to all k8s + ultix VMs, bridged to the
+# storage/datanet network (VLAN 1000). Records the assigned IP in phpIPAM.
+#
+# Runs from the workstation — uses tests/remote.sh to reach each hypervisor.
+# Hot-adds the NIC (no VM downtime). Guest-side IP config must be done
+# separately (the guest OS needs the IP configured on the new interface).
+#
+# Usage:
+#   bash add-datanet-nics.sh             # add NICs (hot-add, no reboot)
+#   bash add-datanet-nics.sh --status    # show current state only
+#
+# IP allocations (phpIPAM VPTechOps, subnet 10.100.100.0/24):
+#   .10 = cnode1    (102/tsys1)
+#   .11 = cnode2    (705/tsys7)
+#   .12 = cnode3    (603/tsys6)
+#   .13 = wnode-tsys3 (313/tsys3)
+#   .14 = wnode-tsys5 (500/tsys5)
+#   .15 = wnode-tsys6 (601/tsys6)
+#   .16 = wnode-tsys7 (701/tsys7)
+#   .17 = wnode-tsys9 (905/tsys9)
+#   .18 = ultix-streaming (5111/tsys5)
+#   .19 = ultix-offstage  (5112/tsys5)
+#
+# Related: Redmine [#396]
+###############################################################################
+set -euo pipefail
+
+# VM definitions: VMID HOST BRIDGE IP HOSTNAME
+VMS=(
+    "102  tsys1  datanet      10.100.100.10  pfv-k8s-cnode1"
+    "705  tsys7  datanet      10.100.100.11  pfv-k8s-cnode2"
+    "603  tsys6  storagenet   10.100.100.12  pfv-k8s-cnode3"
+    "313  tsys3  datanet      10.100.100.13  pfv-k8s-wnode-tsys3"
+    "500  tsys5  datanet      10.100.100.14  pfv-k8s-wnode-tsys5"
+    "601  tsys6  storagenet   10.100.100.15  pfv-k8s-wnode-tsys6"
+    "701  tsys7  datanet      10.100.100.16  pfv-k8s-wnode-tsys7"
+    "905  tsys9  datanet      10.100.100.17  pfv-k8s-wnode-tsys9"
+    "5111 tsys5  datanet      10.100.100.18  ultix-streaming"
+    "5112 tsys5  datanet      10.100.100.19  ultix-offstage"
+)
+
+ACTION="${1:-add}"
+
+echo "==================================================================="
+echo " add-datanet-nics — [#396]"
+echo " mode: ${ACTION}"
+echo "==================================================================="
+echo ""
+
+for entry in "${VMS[@]}"; do
+    read -r vmid host bridge ip hostname <<< "$entry"
+    prox_host="pfv-${host}"
+
+    echo "--- ${hostname} (VM ${vmid} on ${prox_host}) ---"
+
+    if [ "$ACTION" = "--status" ]; then
+        # Show current NIC state
+        PROX_HOST="$prox_host" bash tests/remote.sh prox \
+            "qm config ${vmid} 2>/dev/null | grep -E '^net|^name'" 2>&1
+        echo ""
+        continue
+    fi
+
+    # Check if net1 already exists
+    existing=$(PROX_HOST="$prox_host" bash tests/remote.sh prox \
+        "qm config ${vmid} 2>/dev/null | grep '^net1'" 2>&1 || true)
+
+    if [ -n "$existing" ]; then
+        echo "  net1 already exists: ${existing}"
+        echo "  Skipping."
+        echo ""
+        continue
+    fi
+
+    # Hot-add net1 bridged to the storage network
+    echo "  Adding net1 (bridge=${bridge}, IP=${ip})..."
+    if PROX_HOST="$prox_host" bash tests/remote.sh prox \
+        "qm set ${vmid} -net1 virtio,bridge=${bridge}" 2>&1; then
+        echo "  NIC added. Verify with: qm config ${vmid} | grep net1"
+    else
+        echo "  FAILED — check error above"
+    fi
+    echo ""
+done
+
+if [ "$ACTION" != "--status" ]; then
+    echo "==================================================================="
+    echo " NICs added. Guest-side IP config still needed."
+    echo " Each guest needs the IP configured on the new interface."
+    echo " IPs are allocated in phpIPAM (VPTechOps / 10.100.100.0/24)."
+    echo "==================================================================="
+fi

proxmox/perf/scripts/audit-guest-io.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/bash
+# Audit VM disk cache from inside guests — confirm writeback visible + I/O perf
+set -uo pipefail
+source "$(cd "$(dirname "$0")/../../.." && pwd)/k8s/env.sh"
+
+echo "=== CNODES (via Tailscale) ==="
+for ip in "${ALL_CNODES[@]}"; do
+  echo "--- $ip ---"
+  # shellcheck disable=SC2016 # heredoc-style remote command uses $() on the remote side
+  cn "$ip" '
+    echo "  write_cache: $(cat /sys/block/sda/queue/write_cache 2>/dev/null)"
+    echo "  scheduler: $(cat /sys/block/sda/queue/scheduler 2>/dev/null)"
+    echo "  fsync (5x 1KB):"
+    for i in 1 2 3 4 5; do
+      t0=$(date +%s%N)
+      dd if=/dev/zero of=/tmp/.ft bs=1k count=1 conv=fsync 2>/dev/null
+      t1=$(date +%s%N)
+      echo -n " $(( (t1-t0)/1000000 )) ms"
+    done
+    echo
+    rm -f /tmp/.ft
+  ' 2>&1 || echo "  UNREACHABLE"
+done
+
+echo ""
+echo "=== DNS INFRA VMs ==="
+for host in netinfra01 netinfra02; do
+  echo "--- $host ---"
+  # shellcheck disable=SC2016 # heredoc-style remote command uses $() on the remote side
+  netinfra/dns-cluster-setup/remote-dns.sh "$host-root" '
+    echo "  write_cache: $(cat /sys/block/sda/queue/write_cache 2>/dev/null)"
+    echo "  fsync (3x 1KB):"
+    for i in 1 2 3; do
+      t0=$(date +%s%N)
+      dd if=/dev/zero of=/tmp/.ft bs=1k count=1 conv=fsync 2>/dev/null
+      t1=$(date +%s%N)
+      echo -n " $(( (t1-t0)/1000000 )) ms"
+    done
+    echo
+    rm -f /tmp/.ft
+  ' 2>&1 || echo "  UNREACHABLE"
+done

proxmox/perf/scripts/audit-vm-disks.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/bash
+# Audit all VM disk configs on a Proxmox host
+set -uo pipefail
+HOST="$1"
+PROX_HOST="$HOST" bash tests/remote.sh prox '
+  qm list 2>/dev/null | tail -n +2 | while read -r line; do
+    vmid=$(echo "$line" | awk "{print \$1}")
+    name=$(echo "$line" | awk "{print \$2}")
+    status=$(echo "$line" | awk "{print \$3}")
+    echo "VMID=$vmid NAME=$name STATUS=$status"
+    qm config "$vmid" 2>/dev/null | grep -E "^(scsi|virtio|ide)[0-9]+:" | sed "s/^/  /"
+    echo ""
+  done
+'

proxmox/perf/scripts/conman-console.py

@@ -22,7 +22,7 @@ any network with a conman server:
 
 Usage:
     CONMAN_SERVER=console-host:7890 \\
-    python3 conman-console.py --console pfv-core-sw01 --cmds switches/pfv-core-sw01.cmds
+    python3 conman-console.py --console pfv-r5-core-01 --cmds switches/pfv-r5-core-01.cmds
 
 Lines starting with '!' or '#' in the cmds file are comments (skipped).
 Blank lines are skipped. The conman escape sequence (&.) is sent automatically
@@ -51,7 +51,7 @@ def main():
     ap = argparse.ArgumentParser(
         description="Drive a conman console session read-only via PTY")
     ap.add_argument("--console", required=True,
-                    help="console name (e.g. pfv-core-sw01)")
+                    help="console name (e.g. pfv-r5-core-01)")
     ap.add_argument("--cmds", required=True,
                     help="command file (one command per line; !/# = comment)")
     ap.add_argument("--server",

proxmox/perf/scripts/deploy-tuned-guests.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/bash
+# Deploy tuned inside VMs over SSH (sshd is the only approved access channel —
+# see AGENTS.md "Access-channel policy: SSH only"). Formerly used the
+# qemu guest-agent channel; converted to SSH now that all VMs have key + sudo.
+set -uo pipefail
+
+# Profile mapping: most VMs get throughput-performance, k8s/network-sensitive get network-latency
+PROFILE="${1:-throughput-performance}"
+REMOTE_SH="${REMOTE_SH:-$(cd "$(dirname "$0")/../../.." && pwd)/tests/remote.sh}"
+VM_USER="${VM_USER:-localuser}"
+
+deploy_vm() {
+  local host="$1" vmid="$2" name="$3"
+  # name is the Tailscale hostname (SSH target); vmid retained for reference.
+  echo -n "  VMID $vmid ($name) on $host: "
+  local result
+  result=$(VM_IP="$name" VM_USER="$VM_USER" bash "$REMOTE_SH" vmroot \
+    "DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; \
+     DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; \
+     tuned-adm profile $PROFILE 2>/dev/null; \
+     systemctl enable --now tuned 2>/dev/null; \
+     tuned-adm active 2>/dev/null" </dev/null 2>&1)
+  if echo "$result" | grep -q 'Current active'; then
+    echo "$result" | grep -o 'Current active.*' | head -1
+  elif echo "$result" | grep -qi 'permission denied\|no route\|timed out'; then
+    echo "SSH FAILED (no key/no sudo) — run bootstrap-all.sh first"
+  else
+    echo "INSTALL FAILED (apt issue or no network)"
+  fi
+}
+
+echo "============================================"
+echo "  Deploying tuned ($PROFILE) to VMs via SSH"
+echo "============================================"
+echo ""
+
+# tsys1 VMs
+echo "--- pfv-tsys1 ---"
+deploy_vm pfv-tsys1 100 pfv-bms
+deploy_vm pfv-tsys1 101 tsys-ca
+deploy_vm pfv-tsys1 102 pfv-k8s-cnode1
+deploy_vm pfv-tsys1 104 tsys-librenms
+deploy_vm pfv-tsys1 105 tsys-proxmox-datacenter
+deploy_vm pfv-tsys1 108 tsys-ucs-01
+
+# tsys3 VMs
+echo "--- pfv-tsys3 ---"
+deploy_vm pfv-tsys3 313 pfv-k8s-wnode-tsys3
+
+# tsys6 VMs
+echo "--- pfv-tsys6 ---"
+deploy_vm pfv-tsys6 600 tsys-awx
+deploy_vm pfv-tsys6 601 pfv-k8s-wnode-tsys6
+deploy_vm pfv-tsys6 602 pfv-rr-middleware-02
+deploy_vm pfv-tsys6 603 pfv-k8s-cnode3
+deploy_vm pfv-tsys6 604 tsys-proxmox-mailgw-01
+
+# tsys7 VMs
+echo "--- pfv-tsys7 ---"
+deploy_vm pfv-tsys7 701 pfv-k8s-wnode-tsys7
+deploy_vm pfv-tsys7 702 hfnoc-uisp
+deploy_vm pfv-tsys7 703 pfv-rr-middleware-01
+deploy_vm pfv-tsys7 705 pfv-k8s-cnode2
+deploy_vm pfv-tsys7 706 kali-rd
+deploy_vm pfv-tsys7 707 tsys-siem
+deploy_vm pfv-tsys7 708 kali-tsys
+deploy_vm pfv-tsys7 709 tsys-voip
+deploy_vm pfv-tsys7 711 tsys-proxmox-mailgw-02
+
+# tsys9 VMs
+echo "--- pfv-tsys9 ---"
+deploy_vm pfv-tsys9 902 tsys-ucs-02
+deploy_vm pfv-tsys9 905 pfv-k8s-wnode-tsys9
+
+echo ""
+echo "============================================"
+echo "  Deployment complete."
+echo "  NOT deployed (do-not-reboot VMs):"
+echo "    ultix-streaming, ultix-offstage,"
+echo "    pfv-netinfra-01, pfv-netinfra-02"
+echo "============================================"

proxmox/perf/scripts/fix-e1000e-offload.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+###############################################################################
+# fix-e1000e-offload.sh — Disable offload on e1000e NICs to prevent hangs
+#
+# The Intel e1000e driver has a known bug where TSO/GSO/GRO offload causes
+# "hardware unit hang" resets on certain Intel NICs (I217-LM, I219-LM, 82571EB).
+# This script disables offload features on all e1000e interfaces and installs
+# a systemd service to persist across reboots.
+#
+# References:
+#   https://forum.proxmox.com/threads/e1000-driver-hang.58284/
+#   https://serverfault.com/questions/616485
+#
+# Run on each Proxmox host:
+#   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file proxmox/perf/scripts/fix-e1000e-offload.sh
+###############################################################################
+set -euo pipefail
+
+echo "=== e1000e Offload Fix on $(hostname) ==="
+
+# Find all e1000e physical NICs (skip bridges, bonds, virtual interfaces)
+AFFECTED_NICS=()
+for nic_path in /sys/class/net/*; do
+    nic=$(basename "$nic_path")
+    [ "$nic" = "lo" ] && continue
+    # Skip bridges, bonds, virtual interfaces
+    [ -d "${nic_path}/bridge" ] && continue
+    [ -d "${nic_path}/bonding" ] && continue
+    case "$nic" in
+        tap*|veth*|fwpr*|fwln*|vmbr*|datanet*|storagenet*|tailscale*) continue ;;
+    esac
+
+    driver=$(ethtool -i "$nic" 2>/dev/null | awk '/^driver:/{print $2}')
+    if [ "$driver" = "e1000e" ]; then
+        AFFECTED_NICS+=("$nic")
+        echo "  Found e1000e NIC: $nic"
+    fi
+done
+
+if [ "${#AFFECTED_NICS[@]}" -eq 0 ]; then
+    echo "  No e1000e NICs found. Nothing to do."
+    exit 0
+fi
+
+# Apply fix live
+echo ""
+echo "--- Disabling offload features ---"
+for nic in "${AFFECTED_NICS[@]}"; do
+    echo "  $nic:"
+    ethtool -K "$nic" tso off gro off gso off tx off rx off 2>&1 | sed 's/^/    /' || true
+    tso_state=$(ethtool -k "$nic" 2>/dev/null | awk '/tcp-segmentation-offload/{print $2}' | head -1)
+    echo "    tso=$tso_state"
+done
+
+# Install systemd service for persistence
+echo ""
+echo "--- Installing systemd service ---"
+{
+    echo "[Unit]"
+    echo "Description=Disable offload on e1000e NICs (prevent hardware unit hang)"
+    echo "After=network.target"
+    echo "Wants=network.target"
+    echo ""
+    echo "[Service]"
+    echo "Type=oneshot"
+    echo "RemainAfterExit=yes"
+    for nic in "${AFFECTED_NICS[@]}"; do
+        echo "ExecStart=/sbin/ethtool -K $nic tso off gro off gso off tx off rx off"
+    done
+    echo ""
+    echo "[Install]"
+    echo "WantedBy=multi-user.target"
+} > /etc/systemd/system/fix-e1000e-offload.service
+
+systemctl daemon-reload
+systemctl enable fix-e1000e-offload.service
+echo "  Service installed and enabled (fix-e1000e-offload.service)"
+echo ""
+echo "=== Done. Affected NICs: ${AFFECTED_NICS[*]} ==="

proxmox/perf/scripts/fix-tsys4-storage-bond.sh

@@ -0,0 +1,283 @@
+#!/bin/bash
+###############################################################################
+# fix-tsys4-storage-bond.sh
+#
+# Reconfigures pfv-tsys4 storage network bond from broken 802.3ad (LACP) to
+# active-backup mode. USB NICs cannot participate in LACP — the driver does
+# not report speed/duplex to the kernel (ethtool shows "Speed: Unknown!"),
+# so the bonding driver never transmits LACP PDUs. active-backup requires no
+# switch-side LACP and works correctly with a single port.
+#
+# Run ON pfv-tsys4. Designed for maintenance-window execution.
+#
+# Safety:
+#   - Dry-run by default (--apply to commit)
+#   - Full backup of /etc/network/interfaces
+#   - 5 health checks with automatic rollback on failure
+#   - SSH survives (management on vmbr0/tailscale0, not bond0/datanet)
+#   - Pre-generated rollback script for manual recovery
+#
+# Switch side (core-sw01): NO changes needed. g31 is already a standalone
+# access port in VLAN 1000 — correct for active-backup (no LACP required).
+# g32/ch2 cleanup (dead NIC) is left for separate maintenance.
+#
+# Usage:
+#   bash fix-tsys4-storage-bond.sh             # dry-run (show changes only)
+#   bash fix-tsys4-storage-bond.sh --apply     # commit with auto-rollback
+#
+# Related: Redmine [#394] BUG 4
+###############################################################################
+set -euo pipefail
+
+STORAGE_PEER="10.100.100.6"
+INTERFACES="/etc/network/interfaces"
+TS_SHORT="$(date +%Y%m%d-%H%M%S)"
+BACKUP_DIR="/root/tsys4-bondfix-backup-${TS_SHORT}"
+ROLLBACK="/root/tsys4-bondfix-rollback-${TS_SHORT}.sh"
+ACTION="${1:-dryrun}"
+
+[ "${ACTION}" = "--apply" ] && ACTION="apply" || ACTION="dryrun"
+
+echo "==================================================================="
+echo " fix-tsys4-storage-bond — $(hostname -s)"
+echo " mode: ${ACTION}"
+echo " time: $(date)"
+echo "==================================================================="
+echo ""
+
+# -------------------------------------------------------------------------
+# Pre-flight checks
+# -------------------------------------------------------------------------
+echo "--- Pre-flight checks ---"
+
+# Must be pfv-tsys4
+HOSTNAME_S="$(hostname -s)"
+if [ "${HOSTNAME_S}" != "pfv-tsys4" ]; then
+    echo "FATAL: This script targets pfv-tsys4 (this host: ${HOSTNAME_S})"
+    exit 1
+fi
+echo "  Host: OK (pfv-tsys4)"
+
+# bond0 must exist
+if [ ! -d /sys/class/net/bond0 ]; then
+    echo "FATAL: bond0 not found — no bond to fix"
+    exit 1
+fi
+echo "  bond0: present"
+
+# ifreload must be available
+if ! command -v ifreload >/dev/null 2>&1; then
+    echo "FATAL: ifreload not found (need ifupdown2)"
+    exit 1
+fi
+echo "  ifreload: available"
+
+# SSH must NOT be on bond0/datanet (check incoming route)
+SSH_SRC="$(echo "${SSH_CLIENT:-}" | awk '{print $1}')"
+if [ -n "${SSH_SRC}" ]; then
+    SSH_IFACE="$(ip route get "${SSH_SRC}" 2>/dev/null | grep -oP 'dev \K\S+' || echo "unknown")"
+    echo "  SSH ingress: ${SSH_IFACE}"
+    if echo "${SSH_IFACE}" | grep -qE 'bond0|datanet'; then
+        echo "FATAL: SSH is on storage network — cannot safely reload."
+        echo "       Use physical console (pfv-tsys4) to run this script."
+        exit 1
+    fi
+else
+    echo "  SSH ingress: (local/console session — OK)"
+fi
+echo "  SSH safety: OK (not on storage network)"
+echo ""
+
+# -------------------------------------------------------------------------
+# Show current state
+# -------------------------------------------------------------------------
+echo "--- Current bond0 state ---"
+grep -E "Bonding Mode|Transmit Hash|MII Status|Number of ports" /proc/net/bonding/bond0
+echo ""
+grep -E "Slave Interface|MII Status|Speed" /proc/net/bonding/bond0
+echo ""
+
+echo "--- Current bond0 stanza in /etc/network/interfaces ---"
+awk '/^auto bond0/,/^$/' "${INTERFACES}"
+echo ""
+
+# -------------------------------------------------------------------------
+# Baseline connectivity
+# -------------------------------------------------------------------------
+echo "--- Baseline connectivity ---"
+echo -n "  Ping ${STORAGE_PEER}: "
+if ping -c 1 -W 2 "${STORAGE_PEER}" >/dev/null 2>&1; then
+    echo "OK"
+else
+    echo "UNREACHABLE (baseline already broken — proceed with caution)"
+fi
+echo -n "  NFS server: "
+systemctl is-active nfs-server 2>/dev/null || echo "(not active)"
+echo ""
+
+# -------------------------------------------------------------------------
+# Backup
+# -------------------------------------------------------------------------
+mkdir -p "${BACKUP_DIR}"
+cp -a "${INTERFACES}" "${BACKUP_DIR}/interfaces"
+echo "Backup: ${BACKUP_DIR}/interfaces"
+
+# Generate rollback script
+cat > "${ROLLBACK}" <<ROLLBACKEOF
+#!/bin/bash
+set -euo pipefail
+echo "Rolling back tsys4 bond config..."
+cp -a "${BACKUP_DIR}/interfaces" "${INTERFACES}"
+echo "Restored ${INTERFACES}"
+ifreload -a 2>&1 || systemctl restart networking 2>&1 || true
+sleep 3
+echo "Post-rollback bond0 state:"
+grep -E "Bonding Mode|MII Status|Slave Interface|Speed" /proc/net/bonding/bond0 2>/dev/null
+echo "Rollback complete."
+ROLLBACKEOF
+chmod +x "${ROLLBACK}"
+echo "Rollback: ${ROLLBACK}"
+echo ""
+
+# -------------------------------------------------------------------------
+# Describe the change
+# -------------------------------------------------------------------------
+echo "==================================================================="
+echo " PROPOSED CHANGE"
+echo "==================================================================="
+echo "  1. bond-mode:  802.3ad → active-backup"
+echo "  2. Remove bond-xmit-hash-policy (unused in active-backup)"
+echo "  3. bond-slaves: keep enx8cae4ccda926 (active NIC only)"
+echo ""
+echo "  Rationale: USB NICs cannot do LACP. ethtool reports Speed: Unknown,"
+echo "  so the bonding driver never sends LACP PDUs (verified via tcpdump:"
+echo "  0 LACP PDUs in 65s on both slave and bond master). active-backup"
+echo "  needs no LACP and works with the single working port."
+echo ""
+echo "  Switch: NO changes needed. g31 is standalone access VLAN 1000."
+echo ""
+
+if [ "${ACTION}" != "apply" ]; then
+    echo "==================================================================="
+    echo " DRY RUN — no changes made."
+    echo " Commit:  bash \$0 --apply"
+    echo "==================================================================="
+    exit 0
+fi
+
+# =========================================================================
+# APPLY
+# =========================================================================
+echo "==================================================================="
+echo " APPLYING — auto-rollback on health check failure"
+echo "==================================================================="
+echo ""
+
+# 1. Change bond-mode
+echo "Changing bond-mode 802.3ad → active-backup..."
+sed -i 's/bond-mode 802\.3ad/bond-mode active-backup/' "${INTERFACES}"
+
+# 2. Remove bond-xmit-hash-policy (not used by active-backup)
+echo "Removing bond-xmit-hash-policy..."
+sed -i '/bond-xmit-hash-policy/d' "${INTERFACES}"
+
+# Show updated stanza
+echo ""
+echo "--- Updated bond0 stanza ---"
+awk '/^auto bond0/,/^$/' "${INTERFACES}"
+echo ""
+
+# 3. Reload networking
+echo "--- Reloading networking (ifreload -a) ---"
+echo "SSH safe on vmbr0/tailscale0 (not bond0)..."
+ifreload -a 2>&1 || echo "WARNING: ifreload returned non-zero — checking state..."
+
+# Wait for bond to settle
+echo "Waiting 5s for bond to settle..."
+sleep 5
+
+# =========================================================================
+# HEALTH CHECKS
+# =========================================================================
+echo ""
+echo "==================================================================="
+echo " HEALTH CHECKS (5 tests)"
+echo "==================================================================="
+
+HEALTH_OK=true
+
+# Check 1: bond0 is UP
+echo -n "  [1/5] bond0 MII up: "
+if grep -q "MII Status: up" /proc/net/bonding/bond0 2>/dev/null; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 2: Active slave assigned
+echo -n "  [2/5] Active slave: "
+ACTIVE_SLAVE="$(cat /sys/class/net/bond0/bonding/active_slave 2>/dev/null || echo "")"
+if [ -n "${ACTIVE_SLAVE}" ]; then
+    echo "PASS (${ACTIVE_SLAVE})"
+else
+    echo "FAIL (no active slave)"
+    HEALTH_OK=false
+fi
+
+# Check 3: datanet bridge UP
+echo -n "  [3/5] datanet bridge up: "
+if ip link show datanet 2>/dev/null | grep -q "state UP"; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 4: Storage network IP present on datanet
+echo -n "  [4/5] Storage IP (10.100.100.4): "
+if ip addr show datanet 2>/dev/null | grep -q "10.100.100.4"; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 5: Storage peer reachable
+echo -n "  [5/5] Ping ${STORAGE_PEER}: "
+if ping -c 3 -W 2 "${STORAGE_PEER}" >/dev/null 2>&1; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# =========================================================================
+# COMMIT OR ROLLBACK
+# =========================================================================
+echo ""
+
+if [ "${HEALTH_OK}" = "true" ]; then
+    echo "==================================================================="
+    echo " SUCCESS — all 5 health checks passed"
+    echo "==================================================================="
+    echo ""
+    echo "--- Final bond0 state ---"
+    grep -E "Bonding Mode|MII Status|Slave Interface|Speed|Active" /proc/net/bonding/bond0
+    echo ""
+    echo "Backup:   ${BACKUP_DIR}/interfaces"
+    echo "Rollback: ${ROLLBACK}"
+    echo ""
+    echo "Config change is LIVE but not yet reboot-tested."
+    echo "Verify NFS clients are healthy before next maintenance window."
+    exit 0
+else
+    echo "==================================================================="
+    echo " HEALTH CHECK FAILED — auto-rolling back"
+    echo "==================================================================="
+    bash "${ROLLBACK}"
+    echo ""
+    echo "Auto-rollback complete."
+    echo "Original config restored. Manual investigation needed."
+    exit 1
+fi

proxmox/perf/scripts/perf-matrix.sh

@@ -0,0 +1,242 @@
+#!/bin/bash
+###############################################################################
+# perf-matrix.sh — Any-to-any performance testing across the PFVCluster
+#
+# Tests three network planes:
+#   A. Datanet (VLAN 1000): hypervisor-to-hypervisor over storage network
+#   B. Guest-to-guest: k8s/ultix VMs over datanet (10.100.100.x)
+#   C. Storage I/O: dd read/write to NFS mounts
+#
+# Prerequisites:
+#   - iperf3 installed on all hosts (systemd service: iperf3-server)
+#   - iperf3 installed inside guest VMs
+#   - SSH key + passwordless sudo on all guest VMs (remote.sh; sshd is the
+#     only approved access channel — see AGENTS.md)
+#
+# Usage:
+#   bash perf-matrix.sh                  # run all tests
+#   bash perf-matrix.sh datanet          # host-to-host datanet only
+#   bash perf-matrix.sh guests           # guest-to-guest datanet only
+#   bash perf-matrix.sh storage          # NFS I/O only
+#
+# Environment:
+#   REMOTE_SH   path to tests/remote.sh (auto-detected)
+###############################################################################
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REMOTE_SH="${REMOTE_SH:-$(cd "$SCRIPT_DIR/../../.." && pwd)/tests/remote.sh}"
+LOG_DIR="$(cd "$SCRIPT_DIR/.." && pwd)/returned-logs"
+mkdir -p "$LOG_DIR"
+TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
+
+# --- Host datanet IPs (VLAN 1000) ---
+declare -A DATANET_IP
+DATANET_IP[pfv-tsys1]="10.100.100.1"
+DATANET_IP[pfv-tsys3]="10.100.100.3"
+DATANET_IP[pfv-tsys4]="10.100.100.4"
+DATANET_IP[pfv-tsys5]="10.100.100.5"
+DATANET_IP[pfv-tsys6]="10.100.100.6"
+DATANET_IP[pfv-tsys7]="10.100.100.7"
+DATANET_IP[pfv-tsys9]="10.100.100.9"
+
+HOSTS="pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9"
+
+# --- k8s/ultix VM targets (SSH for control; datanet_ip for iperf traffic) ---
+# Format: prox_host:vmid:datanet_ip:ssh_host:label
+GUEST_TARGETS="
+pfv-tsys1:102:10.100.100.10:pfv-k8s-cnode1:cnode1
+pfv-tsys7:705:10.100.100.11:pfv-k8s-cnode2:cnode2
+pfv-tsys6:603:10.100.100.12:pfv-k8s-cnode3:cnode3
+pfv-tsys3:313:10.100.100.13:pfv-k8s-wnode-tsys3:wnode-tsys3
+pfv-tsys5:500:10.100.100.14:pfv-k8s-wnode-tsys5:wnode-tsys5
+pfv-tsys6:601:10.100.100.15:pfv-k8s-wnode-tsys6:wnode-tsys6
+pfv-tsys7:701:10.100.100.16:pfv-k8s-wnode-tsys7:wnode-tsys7
+pfv-tsys9:905:10.100.100.17:pfv-k8s-wnode-tsys9:wnode-tsys9
+pfv-tsys5:5111:10.100.100.18:ultix-streaming:ultix-streaming
+pfv-tsys5:5112:10.100.100.19:ultix-offstage:ultix-offstage
+"
+
+DURATION="${DURATION:-3}"    # seconds per iperf3 test
+STREAMS="${STREAMS:-4}"      # parallel streams
+
+# ============================================================================
+# Helpers
+# ============================================================================
+start_iperf_servers() {
+    echo "--- Starting iperf3 servers on all hosts ---"
+    for h in $HOSTS; do
+        PROX_HOST="$h" bash "$REMOTE_SH" prox \
+            'systemctl start iperf3-server 2>/dev/null || iperf3 -s -D; echo ok' \
+            >/dev/null 2>&1 &
+    done
+    wait
+    echo "    All servers started."
+}
+
+stop_iperf_servers() {
+    echo "--- Stopping iperf3 servers on all hosts ---"
+    for h in $HOSTS; do
+        PROX_HOST="$h" bash "$REMOTE_SH" prox \
+            'systemctl stop iperf3-server 2>/dev/null; pkill iperf3 2>/dev/null; true' \
+            >/dev/null 2>&1 &
+    done
+    wait
+    echo "    All servers stopped."
+}
+
+# ============================================================================
+# A. Host-to-host datanet matrix
+# ============================================================================
+test_datanet() {
+    echo ""
+    echo "==================================================================="
+    echo "  DATANET (VLAN 1000) — Host-to-Host Bandwidth Matrix"
+    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
+    echo "==================================================================="
+    start_iperf_servers
+
+    local outfile="$LOG_DIR/datanet-host-${TIMESTAMP}.csv"
+    echo "host_from,host_to,mbps" > "$outfile"
+
+    for client in $HOSTS; do
+        for server in $HOSTS; do
+            [ "$client" = "$server" ] && continue
+            local sip="${DATANET_IP[$server]}"
+            local result
+            result=$(PROX_HOST="$client" bash "$REMOTE_SH" prox \
+                "iperf3 -c $sip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
+                | awk '/SUM.*receiver/{printf "%.0f", $6}')
+            if [ -n "$result" ]; then
+                printf "  %-14s → %-14s : %s Mbps\n" "$client" "$server" "$result"
+                echo "$client,$server,$result" >> "$outfile"
+            else
+                printf "  %-14s → %-14s : FAIL\n" "$client" "$server"
+                echo "$client,$server,FAIL" >> "$outfile"
+            fi
+        done
+    done
+
+    stop_iperf_servers
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# B. Guest-to-guest datanet
+# ============================================================================
+test_guests() {
+    echo ""
+    echo "==================================================================="
+    echo "  DATANET (VLAN 1000) — Guest-to-Guest (k8s + ultix VMs)"
+    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
+    echo "==================================================================="
+
+    # Start iperf3 server on the first guest (cnode1)
+    local server_entry
+    server_entry=$(echo "$GUEST_TARGETS" | head -2 | tail -1)
+    local s_ip s_ssh s_label
+    s_ip=$(echo "$server_entry" | cut -d: -f3)
+    s_ssh=$(echo "$server_entry" | cut -d: -f4)
+    s_label=$(echo "$server_entry" | cut -d: -f5)
+
+    echo "  Starting iperf3 server on $s_label ($s_ip)..."
+    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
+        'pkill iperf3 2>/dev/null; iperf3 -s -D' >/dev/null 2>&1
+    sleep 1
+
+    local outfile="$LOG_DIR/datanet-guest-${TIMESTAMP}.csv"
+    echo "guest_from,guest_to,mbps" > "$outfile"
+
+    while read -r entry; do
+        [ -z "$entry" ] && continue
+        local c_ip c_ssh c_label
+        c_ip=$(echo "$entry" | cut -d: -f3)
+        c_ssh=$(echo "$entry" | cut -d: -f4)
+        c_label=$(echo "$entry" | cut -d: -f5)
+        [ "$c_ip" = "$s_ip" ] && continue
+
+        local result
+        result=$(VM_IP="$c_ssh" bash "$REMOTE_SH" vmroot \
+            "iperf3 -c $s_ip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
+            | awk '/SUM.*receiver/{printf "%.0f", $6}')
+        if [ -n "$result" ]; then
+            printf "  %-18s → %-18s : %s Mbps\n" "$c_label" "$s_label" "$result"
+            echo "$c_label,$s_label,$result" >> "$outfile"
+        else
+            printf "  %-18s → %-18s : FAIL\n" "$c_label" "$s_label"
+            echo "$c_label,$s_label,FAIL" >> "$outfile"
+        fi
+    done <<< "$GUEST_TARGETS"
+
+    # Cleanup
+    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
+        'pkill iperf3' >/dev/null 2>&1
+
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# C. Storage I/O (NFS read/write)
+# ============================================================================
+test_storage() {
+    echo ""
+    echo "==================================================================="
+    echo "  STORAGE I/O — NFS Read/Write (100MB dd)"
+    echo "==================================================================="
+
+    local outfile="$LOG_DIR/storage-io-${TIMESTAMP}.csv"
+    echo "host,mount,write_mbps,read_mbps" > "$outfile"
+
+    local ddscript="/tmp/perf-dd-$$.sh"
+    cat > "$ddscript" <<'DDSCRIPT'
+#!/bin/bash
+nfs=$(mount | awk '/type nfs/{print $3}' | grep -v proc)
+for m in $nfs; do
+    tf="$m/.perf-$$"
+    w=$(dd if=/dev/zero of="$tf" bs=1M count=100 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
+    r=$(dd if="$tf" of=/dev/null bs=1M 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
+    rm -f "$tf" 2>/dev/null
+    echo "$m write=${w:-FAIL}MB/s read=${r:-N/A}MB/s"
+done
+DDSCRIPT
+
+    for h in $HOSTS; do
+        echo ""
+        echo "  --- $h ---"
+        PROX_HOST="$h" bash "$REMOTE_SH" prox-file "$ddscript" 2>&1 | while read -r line; do
+            [ -n "$line" ] && echo "    $line"
+        done
+    done
+    rm -f "$ddscript"
+
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# Main
+# ============================================================================
+MODE="${1:-all}"
+
+case "$MODE" in
+    datanet|a)  test_datanet ;;
+    guests|b)   test_guests ;;
+    storage|c)  test_storage ;;
+    all|"")     test_datanet; test_guests; test_storage ;;
+    *)
+        echo "Usage: $0 [datanet|guests|storage|all]"
+        echo ""
+        echo "  datanet  — host-to-host bandwidth matrix over VLAN 1000"
+        echo "  guests   — guest-to-guest (k8s/ultix VMs over VLAN 1000)"
+        echo "  storage  — NFS read/write I/O"
+        echo "  all      — run all three (default)"
+        exit 1
+        ;;
+esac
+
+echo ""
+echo "==================================================================="
+echo "  Perf testing complete. Logs in: $LOG_DIR/"
+echo "==================================================================="

proxmox/perf/scripts/probe-vm-dns.sh

@@ -13,7 +13,7 @@ for vmid in $(qm list 2>/dev/null | awk 'NR>1 && $4=="running" {print $1}'); do
   net0=$(qm config "$vmid" 2>/dev/null | awk '/^net0:/{print $0}')
   echo ""
   echo "--- VMID $vmid: $name ---"
-  echo "  net0: $(echo "$net0" | sed 's/net0: //')"
+  echo "  net0: ${net0//net0: /}"
 
   # Try to get IP via guest-agent
   if qm config "$vmid" 2>/dev/null | grep -q 'agent:.*enabled=1\|^agent: 1'; then

questions-v1.md

@@ -0,0 +1,61 @@
+# questions-v1.md
+
+> Git-tracked question log. The agent writes; the human reviews/edits inline.
+> Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine.
+> See BASELINE-PROMPT.md §9.
+
+## Open questions
+
+### Q1. Git remote for meta?
+- **Context:** meta is now a git repo (locally) but has no remote configured. The auto-commit+push policy (baseline §4) can't complete without one.
+- **Options:** (a) new Gitea repo under reachableceo; (b) nest under an existing repo; (c) keep local-only for now.
+- **Question:** Where should meta push?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+Go with option a. The tea command is setup on this workstation (and on ultix-offstage). I guess, also capture that the tea command (and docker login) are setup on my workstations, so that in the future, projects know they can use tea to setup a repo. Also, i want this to be TSYS wide, so it should go under the TSYSGroupCorporate organization. Call the repo: TSYSGroupAIOS . Make it a template repository.
+
+### Q2. The bin/ wrapper gap (redmine-cli / discourse-cli) — RESOLVED (discourse)
+- **Context:** PFVCluster's operational.md and AGENTS.md referenced `~/daytoday/redmine/bin/redmine` and `~/daytoday/discourse/bin/discourse` as the entrypoints — thin shortcut wrappers around the real CLI containers. The actual CLI source lived in `~/projects/KNEL-AIMiddleware/{redmine,discourse}-cli/`.
+- **Question:** Are the `bin/` wrappers something that should exist, or is the documentation aspirational? Should the template reference these CLIs at all, or stay tool-agnostic?
+- **Answer:** Reference the real container invocation (full path/container name/invoke notes); no duplicate code via shortcut wrappers.
+- **Decision:** No `bin/` wrappers. Invoke the real container with `docker run`. Both CLI sources have been merged into `~/projects/KNEL-AIMiddleware/tooling-cli/{discourse,redmine}/` and PFVCluster's AGENTS.md now invokes them via `docker run --env-file ~/.creds/{discourse,redmine}.env`. The old `~/daytoday/{discourse,redmine}` workspaces and the `{discourse,redmine}-cli/` subdirs were removed.
+- **Synthesized to:** PFVCluster `AGENTS.md` §CLI access; `tooling-cli/{discourse,redmine}/`
+
+### Q3. Should the template ship the Discourse pointer-header pattern?
+- **Context:** PFVCluster migrated 36 in-repo `.md` files to 10-line pointer stubs citing `https://community.turnsys.com/t/<N>`. The template currently has `scripts/garden.sh` that *warns* about oversized non-Discourse `.md`, but doesn't enforce the pointer-header format.
+- **Options:** (a) keep it advisory (garden.sh warn only); (b) add an opt-in check-rule that fails if a tracked `.md` lacks a Discourse URL (excluding AGENTS.md/STATUS.md/etc.); (c) leave it project-local — infra projects want it, personal/business projects don't.
+- **Question:** Which option, and is the assumption in (c) right?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+All projects need it. Discourse/redmine is MANDATORY. No exceptions. What is project specific is which categories to use, and maybe some tagging/topic guidelines etc.
+
+### Q4. Sub-agent nudge hook — wanted?
+- **Context:** A sub-agent proposed a non-blocking Crush hook (`hooks/nudge-subagent.sh`) that emits a stderr reminder after the Nth sequential file read, nudging toward dispatching a sub-agent. Mirrors football's "never read 10+ files sequentially" rule.
+- **Options:** (a) add it (non-blocking, advisory); (b) leave sub-agent use as prose policy only.
+- **Question:** Worth adding, or too noisy?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+Preseving tokens/quota burn is a HUGE priority. It lets me and you do far more work for much longer. Also, I want to move away from harness specific hooks. Git hooks/strong AGENTS.md protocols are strongly preferred. Ill be shifting away from crush over next few weeks to using OpenWebUi/Hermes and a whole swarm of agents with reporting/working relationships etc etc. So anything that is harness specific, get rid of it and make it portable.
+
+
+### Q5. JOURNAL.md vs Discourse audit-log for infra projects
+- **Context:** The template ships `docs/JOURNAL.md` as the append-only decision log. But PFVCluster (the most mature infra project) has NO JOURNAL.md — it uses Discourse topic #298 as the audit log and Redmine for work tracking. PATTERNS.md §5 noted this divergence.
+- **Question:** Should the template keep JOURNAL.md as the default, with infra projects swapping it for the Discourse-audit-log pattern? Or drop JOURNAL.md entirely in favor of "Discourse is the SoR"?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+No more JOURNAL.md . Redmine is the system of record. JOURNAL.md was a hack I was using until redmine integration was in place. And, yes, discourse can also be used as well. Its a bit of a tricky decision, what should go to redmine vs discourse. I usually keep working notes/evolving status etc in Redmine and then synthesize to Discourse. But thats me as a lowly human :) You figure it out as you go and per project.
+
+### Q6. RackRental split-network design on pfv-tsys6/7 [#433]
+- **Context:** TSYS6 and TSYS7 need trunk ports for RackRental. Servers will run a "split network configuration — half on the non-RackRental switch/router side, half on the RackRental switch/router side." RackRental router prevents network loops.
+- **Question:** Is the split per-NIC (one physical NIC to normal network, one to RackRental) or per-VM (some VMs bridge to RackRental, some to normal)? Both TSYS6/7 currently have bond0 with 2 slaves for storage network. How should the bonds be reconfigured?
+- **Answer:** Keep the storage bonds. We need that bandwidth. I believe tsys6,7 have six ethernet ports each? Two on a riser card, and a four port add-on? Or maybe its four total? If four total, the 2 for storage, one for mgmt non rackrental, one trunked to rackrental would be how I want to go.
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —

questions-v2.md

@@ -0,0 +1,53 @@
+# questions-v2.md
+
+> Git-tracked question log. The agent writes; the human reviews/edits inline.
+> Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine.
+> See BASELINE-PROMPT.md §9.
+
+## Resolved (from v1)
+
+### Q1. Git remote for meta? — RESOLVED
+- **Decision:** New Gitea repo under TSYSGroupCorporate org: `TSYSGroupAIOS`. Template repo. Created and pushed.
+- **Synthesized to:** Meta repo created, `tea` + `docker login` noted as available on workstations.
+
+### Q2. The bin/ wrapper gap — RESOLVED
+- **Decision:** No bin/ wrappers. Invoke real containers with `docker run --env-file`.
+- **Synthesized to:** PFVCluster AGENTS.md §CLI access; meta AGENTS.md §CLI invocation.
+
+### Q3. Discourse pointer-header pattern — RESOLVED
+- **Decision:** Mandatory for ALL projects. No exceptions. Discourse/Redmine required.
+- **Synthesized to:** Meta AGENTS.md, check-rules.sh enforces.
+
+### Q4. Sub-agent nudge hook — RESOLVED
+- **Decision:** No harness-specific hooks. Move to portable git hooks + AGENTS.md protocols. User shifting away from Crush to OpenWebUI/Hermes.
+- **Synthesized to:** All hooks ported to git hooks. No Crush-specific dependencies.
+
+### Q5. JOURNAL.md vs Discourse — RESOLVED
+- **Decision:** No more JOURNAL.md. Redmine is SoR for work. Discourse for synthesized knowledge.
+- **Synthesized to:** Meta AGENTS.md, PFVCluster AGENTS.md.
+
+### Q6. RackRental split-network design — RESOLVED
+- **Decision:** Keep storage bonds. If 4 NICs total: 2 storage, 1 mgmt non-RackRental, 1 trunked to RackRental VLAN 2.
+- **Synthesized to:** #433 (trunk ports), #432 (VLAN config).
+
+## Open questions
+
+### Q7. Close #167 (Small UPS Units Re-build)?
+- **Context:** #167 is an old (Dec 2024) vague ticket about rebuilding garage UPS units. It's been superseded by #372 (Tripp Lite physical reseat) and #439 (UPS data integration). Dates are stale (Aug 2025).
+- **Question:** Should we close #167 as superseded?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+### Q8. Friday onsite work order?
+- **Context:** Friday 2026-08-14 onsite batch. Tickets: #341 (TEMPer deploy), #372 (Tripp Lite UPS reseat), #431 (Cisco fan replacement), #414 (tsys5 nic2 cable swap), #374 (PDU outlet renaming), #439 (UPS integration).
+- **Question:** What order do you want to tackle these? Cisco fan (#431) blocks #432/#433/#434/#436 — should it go first? TEMPer is the "fun" project — save for last?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+### Q9. Session-start gate: relax for continuation sessions?
+- **Context:** You mentioned last session wanting to discuss relaxing the ticket gate for handoff/continuation sessions. The scope-alignment gate now handles this (presents handoff priorities, waits for user to name target). Is this sufficient, or do you want further changes?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —

scripts/check-rules.sh

@@ -0,0 +1,266 @@
+#!/usr/bin/env bash
+# check-rules.sh — project rule audit engine.
+#
+# Usage:
+#   bash scripts/check-rules.sh           # full audit (verbose, includes slow checks)
+#   bash scripts/check-rules.sh --fast    # fast audit (quiet, skips slow checks) — for pre-commit
+#   bash scripts/check-rules.sh --quiet   # full audit, only prints failures
+#
+# Exit code: 0 = all rules pass (warnings are non-fatal), 1 = one or more FAILED.
+#
+# This is a generalized version of the rules engine proven in the
+# RCEO-PersonalAssistant project. Add project-specific checks by appending
+# `check "<desc>" "<pass|warn|fail>"` calls below.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+REPO_ROOT="$(repo_root)"
+cd "$REPO_ROOT"
+
+# --- argument parsing ---
+RULE_FAST=false
+RULE_VERBOSE=true
+for arg in "$@"; do
+    case "$arg" in
+        --fast)  RULE_FAST=true; RULE_VERBOSE=false ;;
+        --quiet) RULE_VERBOSE=false ;;
+        *) die "check-rules.sh: unknown argument '$arg'" ;;
+    esac
+done
+export RULE_FAST RULE_VERBOSE
+
+init_counters
+$RULE_VERBOSE && echo "=== Project Rule Audit ==="
+
+TODAY="$(date +%Y-%m-%d)"
+
+# ----------------------------------------------------------------------------
+# 1. Shellcheck — every .sh/.bash must pass (zero warnings, incl. info-level).
+#    Runs in Docker so the host stays clean (no native shellcheck required).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Shell scripts (shellcheck)"
+mapfile -d '' SH_FILES < <(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print0 2>/dev/null)
+if [ "${#SH_FILES[@]}" -gt 0 ]; then
+    if have shellcheck; then
+        if shellcheck "${SH_FILES[@]}" >/dev/null 2>&1; then
+            check "All shell scripts pass shellcheck (host)" "pass"
+        else
+            check "shellcheck reports violations — run: shellcheck <file>" "fail"
+        fi
+    elif have docker; then
+        MNT_FILES=()
+        for f in "${SH_FILES[@]}"; do MNT_FILES+=("/mnt/${f#./}"); done
+        if docker run --rm -v "$REPO_ROOT:/mnt" koalaman/shellcheck:stable "${MNT_FILES[@]}" >/dev/null 2>&1; then
+            check "All shell scripts pass shellcheck (docker)" "pass"
+        else
+            check "shellcheck (docker) reports violations" "fail"
+        fi
+    else
+        check "No shellcheck or docker available to lint scripts" "warn"
+    fi
+else
+    check "No shell scripts to lint" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 2. Docker image pinning — no ':latest' tags in compose or Dockerfiles.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Docker image pinning"
+if grep -rqE '(image:|FROM).*:latest' --include='docker-compose*.y*ml' --include='Dockerfile*' . 2>/dev/null; then
+    check "No ':latest' image tags (pin everything)" "fail"
+else
+    check "No ':latest' image tags" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 2b. Container naming — every service in a docker-compose file MUST set an
+#     explicit container_name (never rely on Docker's default <dir>_<n>).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Container naming"
+COMPOSE_FILES="$(find . -path ./.git -prune -o \( -name 'docker-compose*.yml' -o -name 'docker-compose*.yaml' -o -name 'compose.yml' -o -name 'compose.yaml' \) -print 2>/dev/null || true)"
+if [ -n "$COMPOSE_FILES" ]; then
+    BAD=0
+    while IFS= read -r cf; do
+        [ -n "$cf" ] || continue
+        # Count top-level service keys (2-space indent under services:) and
+        # compare against the number of container_name: declarations.
+        svc_count=$(awk '/^services:/{f=1;next} f&&/^[^[:space:]]/{f=0} f&&/^[[:space:]]{2}[[:alnum:]_-]+:[[:space:]]*$/{c++} END{print c+0}' "$cf")
+        cn_count=$(grep -cE '^[[:space:]]*container_name:' "$cf" 2>/dev/null || echo 0)
+        if [ "${svc_count:-0}" -gt 0 ] && [ "$cn_count" -lt "$svc_count" ]; then
+            BAD=$((BAD + 1))
+        fi
+    done <<EOF
+$COMPOSE_FILES
+EOF
+    if [ "$BAD" -eq 0 ]; then
+        check "All compose services set container_name" "pass"
+    else
+        check "$BAD compose file(s) with services missing container_name" "fail"
+    fi
+else
+    check "No compose files (container-name check skipped)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 3. Required-files manifest — the files every project using this template owns.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Required files"
+REQUIRED_FILES="AGENTS.md STATUS.md questions-v1.md .env.example scripts/check-rules.sh scripts/setup-hooks.sh"
+REQUIRED_FILES="$REQUIRED_FILES ${PROJECT_REQUIRED_FILES:-}"
+for f in $REQUIRED_FILES; do
+    if [ -f "$f" ]; then check "$f exists" "pass"; else check "$f MISSING" "fail"; fi
+done
+
+# ----------------------------------------------------------------------------
+# 4. Doc freshness — STATUS.md touched today.
+#    Warning (not failure): staleness is a signal, not a break.
+#    Redmine is the system of record for work; Discourse for docs. STATUS.md is
+#    a scratchpad only — see BASELINE-PROMPT.md §3, §8.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Doc freshness"
+if [ -f STATUS.md ]; then
+    STATUS_DATE="$(grep -oE 'Last updated: [0-9]{4}-[0-9]{2}-[0-9]{2}' STATUS.md | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' || echo unknown)"
+    if [ "$STATUS_DATE" = "$TODAY" ]; then
+        check "STATUS.md updated today ($STATUS_DATE)" "pass"
+    else
+        check "STATUS.md is stale (last: $STATUS_DATE, today: $TODAY) — update it" "warn"
+    fi
+else
+    check "STATUS.md MISSING" "fail"
+fi
+
+# ----------------------------------------------------------------------------
+# 4b. Discourse pointer-header policy (MANDATORY).
+#     Discourse is the system of record for documentation. In-repo .md files are
+#     stubs that point to a Discourse topic URL. Operational files exempt.
+#     Override exemptions via PROJECT_DOC_EXEMPT (space-separated globs of
+#     basenames) and the Discourse host via PROJECT_DISCOURSE_HOST.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Discourse pointer-header"
+DISCOURSE_HOST="${PROJECT_DISCOURSE_HOST:-community.turnsys.com}"
+DOC_EXEMPT="${PROJECT_DOC_EXEMPT:-AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v*.md BASELINE-PROMPT.md PATTERNS.md}"
+POINTER_MISSING=0
+while IFS= read -r -d '' f; do
+    base="$(basename "$f")"
+    exempt=false
+    for pat in $DOC_EXEMPT; do
+        # shellcheck disable=SC2254
+        case "$base" in $pat) exempt=true; break ;; esac
+    done
+    [ "$exempt" = true ] && continue
+    if ! grep -qF "$DISCOURSE_HOST" "$f" 2>/dev/null; then
+        if [ "$POINTER_MISSING" -eq 0 ]; then
+            $RULE_VERBOSE && printf '  %s\n' "Missing $DISCOURSE_HOST URL in:"
+        fi
+        POINTER_MISSING=$((POINTER_MISSING + 1))
+        $RULE_VERBOSE && printf '    %s\n' "$f"
+    fi
+done < <(find . -path ./.git -prune -o -path ./.crush -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -name '*.md' -print0 2>/dev/null)
+if [ "$POINTER_MISSING" -eq 0 ]; then
+    check "All non-exempt .md cite Discourse ($DISCOURSE_HOST)" "pass"
+else
+    check "$POINTER_MISSING .md file(s) missing Discourse pointer (see BASELINE-PROMPT.md §3)" "fail"
+fi
+
+# ----------------------------------------------------------------------------
+# 5. Git state — uncommitted changes are a warning (the pre-push hook hardens
+#    this where it matters).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Git state"
+if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
+    if git diff --quiet && git diff --cached --quiet; then
+        check "Working tree clean" "pass"
+    else
+        check "Uncommitted changes present" "warn"
+    fi
+else
+    check "Not a git repo (git checks skipped)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 6. Hooks installed — self-check that git hooks were set up.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Git hooks"
+if [ -f .git/hooks/pre-commit ]; then
+    check "pre-commit hook installed" "pass"
+else
+    check "pre-commit NOT installed (run: bash scripts/setup-hooks.sh)" "warn"
+fi
+if [ -f .git/hooks/pre-push ]; then
+    check "pre-push hook installed" "pass"
+else
+    check "pre-push NOT installed (run: bash scripts/setup-hooks.sh)" "warn"
+fi
+
+# ----------------------------------------------------------------------------
+# 7. WORKING.md completion — no unchecked tasks may remain at commit time.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Task completion"
+if [ -f WORKING.md ]; then
+    UNCHECKED="$(grep -cF -- '- [ ]' WORKING.md || true)"
+    if [ "$UNCHECKED" -eq 0 ]; then
+        check "WORKING.md has no unchecked tasks" "pass"
+    else
+        check "WORKING.md has ${UNCHECKED} unchecked task(s) — finish them before committing" "fail"
+    fi
+else
+    check "WORKING.md absent (no active task tracker)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 8. CNW markers — empty `CNW:` markers flag unresolved questions for the human.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Unresolved questions"
+EMPTY_CNW="$(grep -rn 'CNW:$' . --include='*.md' 2>/dev/null | head -20 || true)"
+if [ -z "$EMPTY_CNW" ]; then
+    check "No empty CNW: markers (unresolved questions)" "pass"
+else
+    CNW_COUNT="$(printf '%s\n' "$EMPTY_CNW" | grep -c . || true)"
+    check "${CNW_COUNT} unresolved CNW: marker(s) — needs user input" "warn"
+fi
+
+# ----------------------------------------------------------------------------
+# 9. Hygiene — merge-conflict markers and trailing whitespace must never land.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "File hygiene"
+if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
+    CONFLICT="$(git diff --cached --name-only --diff-filter=ACM 2>/dev/null | xargs -r grep -lE '^(<<<<<<<|=======|>>>>>>>)' 2>/dev/null || true)"
+    if [ -z "$CONFLICT" ]; then check "No merge-conflict markers staged" "pass"; else check "Merge-conflict markers staged: $CONFLICT" "fail"; fi
+fi
+
+# ----------------------------------------------------------------------------
+# 10. (slow, skipped in --fast) Project test suite via scripts/test.sh.
+# ----------------------------------------------------------------------------
+if [ "$RULE_FAST" = false ] && [ -x scripts/test.sh ]; then
+    $RULE_VERBOSE && log_step "Test suite (scripts/test.sh)"
+    if bash scripts/test.sh >/dev/null 2>&1; then
+        check "scripts/test.sh passes" "pass"
+    else
+        check "scripts/test.sh FAILS" "fail"
+    fi
+fi
+
+# ----------------------------------------------------------------------------
+# 11. Access-channel policy — qemu guest-agent must NEVER be used as an
+#     execution or key-delivery channel. SSH (sshd) is the ONLY approved
+#     remote access path; all commands must be logged through standard
+#     auth/audit infrastructure. ITAR/CMMC environment — non-negotiable.
+#     Allowed: installing/checking qemu-guest-agent for Proxmox state
+#     visibility. Forbidden: `qm guest exec` + any `vm-guest` wrapper.
+#     Scans CODE only (not .md) so docs may describe the ban.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Access-channel policy (no guest-agent exec)"
+GA_HITS="$(grep -rnE 'qm guest exec|vm-guest|_vm_guest' \
+    --include='*.sh' --include='*.bash' --include='*.py' \
+    . 2>/dev/null | grep -vE 'scripts/check-rules\.sh|/archive/' || true)"
+if [ -z "$GA_HITS" ]; then
+    check "No guest-agent exec / vm-guest access patterns in code" "pass"
+else
+    GA_COUNT="$(printf '%s\n' "$GA_HITS" | grep -c . || true)"
+    $RULE_VERBOSE && printf '%s\n' "$GA_HITS" | sed 's/^/    /'
+    check "${GA_COUNT} guest-agent exec / vm-guest reference(s) — SSH-only access policy (AGENTS.md)" "fail"
+fi
+
+print_summary_and_exit

scripts/docker-run.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# docker-run.sh — canonical ephemeral-container wrapper.
+#
+# Keeps the host clean: every build/test/generation runs inside a pinned image.
+# Ensures output files are owned by the invoking user (not root).
+#
+# Usage:
+#   docker-run.sh <image> [command...]
+#       Runs <command> in <image> with the repo mounted at /data, cwd /data.
+#       With no command, drops into the image's default entrypoint.
+#   docker-run.sh --shell <image>
+#       Interactive shell inside the container (for debugging).
+#
+# Examples:
+#   docker-run.sh python:3.12-slim python3 -m pytest
+#   docker-run.sh pandoc/extra report.md -o report.pdf
+#   docker-run.sh --shell node:20
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+
+SHELL_MODE=false
+case "${1:-}" in
+    --shell) SHELL_MODE=true; shift ;;
+    -h|--help)
+        sed -n '2,18p' "$0"; exit 0 ;;
+esac
+
+[ "$#" -ge 1 ] || { sed -n '2,18p' "$0"; exit 1; }
+
+if [ "$SHELL_MODE" = true ]; then
+    # ${SHELL:-sh} must expand inside the container, not in this outer shell.
+    # shellcheck disable=SC2016
+    docker_run "$1" sh -c 'exec "${SHELL:-sh}"'
+else
+    docker_run "$@"
+fi

scripts/garden.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# garden.sh — the gardening loop.
+#
+# Reports doc sprawl and files that violate the "Discourse is the system of
+# record for documentation; gitea .md files are stubs" policy. Run via
+# `bash scripts/garden.sh`. Findings are WARNINGS (advisory); fix them at a natural break.
+#
+# What it checks:
+#   1. Markdown sprawl: count of .md files per directory (top-10 by count).
+#   2. Oversized .md files (default >300 lines) that don't cite a Discourse URL
+#      — candidates to migrate to Discourse, leaving a stub.
+#   3. .md files with no Discourse link at all (informational; exempt: the
+#      operational files in EXEMPT_FILES).
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+REPO_ROOT="$(repo_root)"
+cd "$REPO_ROOT"
+
+SIZE_LIMIT="${GARDEN_MD_LINE_LIMIT:-300}"
+# Operational files that legitimately live next to code, not in Discourse.
+EXEMPT_FILES="${GARDEN_EXEMPT:-AGENTS.md STATUS.md WORKING.md questions-v.*.md PATTERNS.md BASELINE-PROMPT.md README.md}"
+
+log_step "Gardening report for $REPO_ROOT"
+
+# --- 1. sprawl by directory -------------------------------------------------
+log_info "Markdown file count by directory (top 10):"
+find . -path ./.git -prune -o -name '*.md' -print 2>/dev/null \
+    | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn | head -10 | sed 's/^/    /'
+
+# --- 2. oversized .md without a Discourse link ------------------------------
+log_info "Oversized .md (>${SIZE_LIMIT} lines) lacking a Discourse URL — migrate candidates:"
+OVERSIZED=0
+while IFS= read -r -d '' f; do
+    # skip exempt files (glob match against basename and relative path)
+    exempt=false
+    base=$(basename "$f")
+    rel=${f#./}
+    for pat in $EXEMPT_FILES; do
+        # shellcheck disable=SC2254  # glob match is intentional
+        case "$base" in $pat) exempt=true; break ;; esac
+        # shellcheck disable=SC2254
+        case "$rel" in $pat) exempt=true; break ;; esac
+    done
+    [ "$exempt" = true ] && continue
+    lines=$(wc -l < "$f" 2>/dev/null || echo 0)
+    if [ "$lines" -gt "$SIZE_LIMIT" ]; then
+        if ! grep -qiE 'community\.turnsys\.com|discourse' "$f" 2>/dev/null; then
+            printf '    %-60s %s lines\n' "$f" "$lines"
+            OVERSIZED=$((OVERSIZED + 1))
+        fi
+    fi
+done < <(find . -path ./.git -prune -o -name '*.md' -print0 2>/dev/null)
+[ "$OVERSIZED" -eq 0 ] && echo "    (none)"
+
+# --- 3. summary -------------------------------------------------------------
+log_step "Gardening summary"
+echo "    Oversized non-Discourse .md files: $OVERSIZED"
+if [ "$OVERSIZED" -eq 0 ]; then
+    log_ok "no migration candidates"
+else
+    log_warn "$OVERSIZED file(s) to migrate to Discourse"
+fi

scripts/lib/common.sh

@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# lib/common.sh — shared helpers for shell scripts and hooks in this repo.
+#
+# Source it from any script:
+#   #!/usr/bin/env bash
+#   set -euo pipefail
+#   HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+#   # shellcheck source=lib/common.sh
+#   source "$HERE/lib/common.sh"   # or the appropriate relative path
+#
+# This library exists to drive a known cross-project inconsistency to zero:
+# every repo used to re-paste the ANSI color block, redefine log_* helpers,
+# pick one of three incompatible shebangs, and roll its own docker wrapper.
+# Import this once instead.
+
+# Do NOT set -euo pipefail here unconditionally — some callers (git hooks)
+# source this file and rely on controlling their own shell options. We only
+# guarantee the functions below are defined.
+
+###############################################################################
+# Config — override via environment before sourcing if needed
+###############################################################################
+: "${TEMPLATE_ROOT:=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
+export TEMPLATE_ROOT
+
+###############################################################################
+# ANSI colors (defined once, used everywhere)
+###############################################################################
+if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then
+    RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
+    BLUE='\033[0;34m'; BOLD='\033[1m'; NC='\033[0m'
+else
+    RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; NC=''
+fi
+export RED GREEN YELLOW BLUE BOLD NC
+
+###############################################################################
+# Logging
+###############################################################################
+log_info()  { printf "${BLUE}›${NC} %s\n" "$*"; }
+log_ok()    { printf "${GREEN}✓${NC} %s\n" "$*"; }
+log_warn()  { printf "${YELLOW}⚠${NC} %s\n" "$*" >&2; }
+log_error() { printf "${RED}✗${NC} %s\n" "$*" >&2; }
+log_step()  { printf "\n${BOLD}== %s ==${NC}\n" "$*"; }
+
+die() { log_error "$*"; exit 1; }
+
+###############################################################################
+# Predicates
+###############################################################################
+# have <cmd> — return 0 if <cmd> is on PATH
+have() { command -v "$1" >/dev/null 2>&1; }
+
+###############################################################################
+# Path helpers
+###############################################################################
+repo_root() {
+    # Prefer git's notion of the repo root, fall back to $TEMPLATE_ROOT, then pwd.
+    if git rev-parse --show-toplevel >/dev/null 2>&1; then
+        git rev-parse --show-toplevel
+    else
+        printf '%s\n' "${TEMPLATE_ROOT:-$(pwd)}"
+    fi
+}
+
+###############################################################################
+# Privilege helpers
+###############################################################################
+# as_root — run the remaining args as root via sudo, or directly if already root.
+as_root() {
+    if [ "$(id -u)" -eq 0 ]; then "$@"; else sudo "$@"; fi
+}
+
+###############################################################################
+# Docker wrapper
+###############################################################################
+# docker_run <image> <args...>
+# Ephemeral container, host-uid ownership, repo mounted at /data, cwd /data.
+# Drives the "host stays clean; everything runs in containers" policy and
+# ensures output files are owned by the invoking user, not root.
+docker_run() {
+    [ "$#" -ge 1 ] || die "docker_run: image required"
+    local image="$1"; shift
+    have docker || die "docker not found on PATH"
+    local root
+    root="$(repo_root)"
+    docker run --rm \
+        --user "$(id -u):$(id -g)" \
+        -e HOME=/tmp \
+        -v "$root:/data" \
+        -w /data \
+        "$image" "$@"
+}
+
+###############################################################################
+# Rule-audit accumulator (used by scripts/check-rules.sh)
+# Globals read/written: RULE_PASS RULE_WARN RULE_FAIL
+###############################################################################
+init_counters() { RULE_PASS=0; RULE_WARN=0; RULE_FAIL=0; }
+
+# check <description> <pass|warn|fail>
+check() {
+    local desc="$1" result="$2"
+    case "$result" in
+        pass)
+            RULE_PASS=$((RULE_PASS + 1))
+            if [ "${RULE_VERBOSE:-true}" = true ]; then printf "  ${GREEN}PASS${NC}  %s\n" "$desc"; fi
+            ;;
+        warn)
+            RULE_WARN=$((RULE_WARN + 1))
+            if [ "${RULE_VERBOSE:-true}" = true ]; then printf "  ${YELLOW}WARN${NC}  %s\n" "$desc"; fi
+            ;;
+        fail)
+            RULE_FAIL=$((RULE_FAIL + 1))
+            printf "  ${RED}FAIL${NC}  %s\n" "$desc"
+            ;;
+        *)
+            die "check(): invalid result '$result' (use pass|warn|fail)"
+            ;;
+    esac
+}
+
+# print_summary_and_exit
+print_summary_and_exit() {
+    if [ "${RULE_VERBOSE:-true}" = true ]; then
+        printf "\n=== Summary ===\n  PASS: %s\n  WARN: %s\n  FAIL: %s\n\n" \
+            "$RULE_PASS" "$RULE_WARN" "$RULE_FAIL"
+    fi
+    if [ "$RULE_FAIL" -gt 0 ]; then
+        if [ "${RULE_VERBOSE:-true}" = true ]; then
+            printf "AUDIT FAILED — %s rule(s) violated.\n" "$RULE_FAIL"
+        fi
+        exit 1
+    fi
+    if [ "${RULE_VERBOSE:-true}" = true ]; then printf "AUDIT PASSED.\n"; fi
+    exit 0
+}

scripts/pre-commit

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# pre-commit — fast rule audit (< 1s typical).
+# Hot-path bypass: commits that ONLY touch STATUS.md / WORKING.md skip the
+# audit so frequent status/task commits stay frictionless.
+set -euo pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+cd "$REPO_ROOT"
+
+CHANGED="$(git diff --cached --name-only)"
+HOT_PATHS="$(printf '%s\n' "$CHANGED" | grep -vE '^(STATUS.md|WORKING.md)$' || true)"
+
+if [ -z "$HOT_PATHS" ]; then
+    echo "hot-path files only (STATUS/WORKING) — skipping rule audit"
+    exit 0
+fi
+
+if ! bash scripts/check-rules.sh --fast; then
+    echo ""
+    echo "pre-commit audit FAILED. Fix the violations above before committing."
+    echo "Full audit:  bash scripts/check-rules.sh"
+    echo "Bypass:      git commit --no-verify   (emergencies only)"
+    exit 1
+fi
+exit 0

scripts/pre-push

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# pre-push — full rule audit + clean-working-tree gate before pushing.
+# Installed via: bash scripts/setup-hooks.sh
+#
+# Combines two proven policies observed across projects:
+#   - KNEL-AIMiddleware: block push if the working tree is dirty.
+#   - RCEO-PersonalAssistant: block push if the full test suite fails.
+set -euo pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+cd "$REPO_ROOT"
+
+echo "pre-push: running full rule audit..."
+
+# Full audit (non-fast): runs the slow test suite via `make test` if present.
+if ! bash scripts/check-rules.sh --quiet; then
+    echo ""
+    echo "pre-push audit FAILED. Push blocked."
+    echo "Re-run with output:  bash scripts/check-rules.sh"
+    echo "Bypass:              git push --no-verify   (emergencies only)"
+    exit 1
+fi
+
+echo "pre-push: all rules and tests passed."
+exit 0

scripts/setup-hooks.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# setup-hooks.sh — install this repo's git hooks.
+#
+# Mechanism: copy scripts/pre-commit and scripts/pre-push into .git/hooks/ and
+# make them executable. This is the most portable pattern (works on any clone,
+# no `git config core.hooksPath` mutation, survives config resets, idempotent).
+#
+# Run once after cloning:   bash scripts/setup-hooks.sh
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+REPO_ROOT="$(repo_root)"
+cd "$REPO_ROOT"
+
+[ -d .git ] || die "no .git directory here — run this from a git checkout"
+
+HOOKS_DIR=".git/hooks"
+HOOK_NAMES="pre-commit pre-push"
+
+log_step "Installing git hooks"
+for name in $HOOK_NAMES; do
+    src="scripts/$name"
+    dst="$HOOKS_DIR/$name"
+    [ -f "$src" ] || die "source hook not found: $src"
+    cp "$src" "$dst"
+    chmod +x "$dst"
+    log_ok "installed $dst"
+done
+
+cat <<EOF
+
+Git hooks installed. The following now run automatically:
+
+  pre-commit  fast rule audit (shellcheck, image pinning, container naming,
+              required files, doc freshness, Discourse pointers, WORKING.md
+              completion, hygiene).
+              Hot-path bypass for STATUS.md / WORKING.md.
+  pre-push    full rule audit (includes scripts/test.sh) + clean-working-tree gate.
+
+Bypass either with \`git commit --no-verify\` / \`git push --no-verify\`
+(emergencies only).
+EOF

tests/phase1-access-audit.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/bash
+# phase1-access-audit.sh — Try SSH to every Tailscale Linux system
+# Reports: reachable as localuser, reachable as root, or needs key
+set -uo pipefail
+
+SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=8)
+
+# All Linux systems from tailscale status (excluding Windows, offline, switches)
+SYSTEMS=(
+  # Proxmox hosts (try root first)
+  pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9
+  # K8s nodes
+  pfv-k8s-cnode1 pfv-k8s-cnode2 pfv-k8s-cnode3
+  pfv-k8s-wnode-tsys3 pfv-k8s-wnode-tsys5 pfv-k8s-wnode-tsys6
+  pfv-k8s-wnode-tsys7 pfv-k8s-wnode-tsys9
+  # Netinfra
+  pfv-netinfra-01 pfv-netinfra-02
+  # Production VMs
+  pfv-bms devbox-cloudron hfnoc-uisp kali-rd kali-tsys
+  pfv-rr-middleware-01 pfv-rr-middleware-02
+  # Ultix (do not modify, just probe)
+  ultix-streaming ultix-offstage
+  # Preprod
+  preprod-awx preprod-ca preprod-cloudron preprod-hfnoc-uisp
+  preprod-librenms preprod-proxmox-datacenter preprod-proxmox-mailgw
+  preprod-rancherplatform preprod-siem preprod-voip
+  # Sectestbed
+  sectestbed-awx sectestbed-ca sectestbed-cloudron sectestbed-hfnoc-uisp
+  sectestbed-k8s-cnode sectestbed-k8s-wnode sectestbed-librenms
+  sectestbed-netinfra sectestbed-proxmox-datacenter sectestbed-proxmox-mailgw
+  sectestbed-proxmox-pbs sectestbed-proxmox-pve sectestbed-rancherplatform
+  sectestbed-sandbox sectestbed-siem sectestbed-voip
+  # Other Linux
+  netbird pfv-jetson-nano-1 pfv-proxmox-backup-server pfvsvrpi
+  subopi-dev-3 stlpc-artroom stlpc-garage
+)
+
+echo "system	localuser	root	status"
+echo "------	---------	----	------"
+
+for sys in "${SYSTEMS[@]}"; do
+  lu_ok="no"
+  root_ok="no"
+  ts_ip=""
+
+  # Try localuser first
+  if ssh "${SSH_OPTS[@]}" "localuser@${sys}" 'echo ok' 2>/dev/null | grep -q ok; then
+    lu_ok="yes"
+    ts_ip=$(ssh "${SSH_OPTS[@]}" "localuser@${sys}" 'tailscale ip -4 2>/dev/null || echo "?"' 2>/dev/null)
+  fi
+
+  # Try root
+  if ssh "${SSH_OPTS[@]}" "root@${sys}" 'echo ok' 2>/dev/null | grep -q ok; then
+    root_ok="yes"
+    [ -z "$ts_ip" ] && ts_ip=$(ssh "${SSH_OPTS[@]}" "root@${sys}" 'tailscale ip -4 2>/dev/null || echo "?"' 2>/dev/null)
+  fi
+
+  # Determine status
+  if [ "$lu_ok" = "yes" ] && [ "$root_ok" = "yes" ]; then
+    status="OK (both)"
+  elif [ "$lu_ok" = "yes" ]; then
+    status="OK (localuser only)"
+  elif [ "$root_ok" = "yes" ]; then
+    status="OK (root only)"
+  else
+    status="NO ACCESS"
+  fi
+
+  echo "$sys	$lu_ok	$root_ok	$status	${ts_ip:-}"
+done

tests/remote.sh

@@ -30,8 +30,6 @@ PROX_HOST="${PROX_HOST:-pfv-tsys5}"
 PROX_USER="${PROX_USER:-root}"
 VM_IP="${VM_IP:-192.168.3.50}"
 VM_USER="${VM_USER:-localuser}"
-VM_ID="${VM_ID:-}"
-GUEST_TIMEOUT="${GUEST_TIMEOUT:-900}"
 
 SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
 
@@ -55,33 +53,6 @@ _copy() {
   fi
 }
 
-# Out-of-band VM access via the Proxmox qemu-guest-agent. This runs commands
-# as root inside the VM and does NOT depend on SSH, so it works even after
-# secharden-ssh replaces authorized_keys and secharden-2fa enforces
-# publickey+keyboard-interactive (which blocks non-interactive SSH).
-GUEST_PARSER="/root/.knel-guest-parse.py"
-GUEST_PARSER_SRC="import sys, json
-try:
-    d = json.load(sys.stdin)
-except Exception:
-    sys.exit(3)
-sys.stdout.write(d.get('out-data', '') or '')
-sys.stderr.write(d.get('err-data', '') or '')
-ec = d.get('exitcode', 1)
-sys.exit(ec if ec is not None else 1)"
-
-_ensure_guest_parser() {
-  if _prox "test -f '$GUEST_PARSER'" >/dev/null 2>&1; then return 0; fi
-  printf '%s\n' "$GUEST_PARSER_SRC" | _prox "cat > '$GUEST_PARSER'" >/dev/null 2>&1
-}
-
-_vm_guest() {
-  [ -n "$VM_ID" ] || die "vm-guest requires VM_ID"
-  _ensure_guest_parser
-  local cmdb64; cmdb64="$(printf '%s' "$*" | base64 -w0)"
-  _prox "qm guest exec $VM_ID --timeout ${GUEST_TIMEOUT} -- /bin/sh -c 'echo $cmdb64 | base64 -d | /bin/sh' 2>/dev/null | python3 '$GUEST_PARSER'"
-}
-
 mode="${1:-}"; shift || true
 case "$mode" in
   prox)         [ "$#" -ge 0 ] || die "need command"; _prox "$*" ;;
@@ -91,7 +62,6 @@ case "$mode" in
   vm-file)      [ -f "${1:-}" ] || die "need local script file"; _vm "bash -s" < "$1" ;;
   vm-copy)      [ -f "${1:-}" ] || die "need local file"; _copy "${VM_USER}@${VM_IP}" "$1" "${2:-}" ;;
   prox-copy)    [ -f "${1:-}" ] || die "need local file"; _copy "${PROX_USER}@${PROX_HOST}" "$1" "${2:-}" ;;
-  vm-guest)     [ "$#" -ge 1 ] || die "need command"; _vm_guest "$*" ;;
   ""|-h|--help|help) sed -n '2,40p' "${BASH_SOURCE[0]}" >&2; exit 0 ;;
   *) die "unknown mode '$mode'. Run '$0 help'." ;;
 esac

tests/vm-validation.sh

@@ -54,10 +54,9 @@ VM_ID="${VM_ID:-}"
 REPO_URL="${REPO_URL:-https://git.knownelement.com/KNEL/KNELServerBuild.git}"
 REMOTE_REPO="${REMOTE_REPO:-KNELServerBuild}"
 SNAP_PREFIX="${SNAP_PREFIX:-pre-knel-deploy}"
-ACCESS_PUBKEY="${ACCESS_PUBKEY:-$HOME/.ssh/id_ed25519.pub}"
-# Re-inject the validation pubkey after each deploy (secharden-ssh replaces
-# authorized_keys with the managed production key set, locking out the
-# bootstrap/dev key). Set RESTORE_ACCESS=0 to disable.
+# SSH-only access policy: if SSH access is lost after a deploy, the agent does
+# NOT re-inject keys via a back-channel. Set RESTORE_ACCESS=1 to have the
+# script check and warn (it cannot self-restore).
 RESTORE_ACCESS="${RESTORE_ACCESS:-1}"
 
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -80,7 +79,6 @@ die()  { log "ERROR: $*"; exit 1; }
 vm()      { bash "$REMOTE" vm "$@"; }           # as $VM_USER (SSH)
 vmroot()  { bash "$REMOTE" vmroot "$@"; }       # as root via sudo (SSH)
 vmfile()  { bash "$REMOTE" vm-file "$@"; }      # run local script on VM (SSH)
-vmguest() { bash "$REMOTE" vm-guest "$@"; }     # as root via guest agent (no SSH/2FA)
 prox()    { bash "$REMOTE" prox "$@"; }         # as $PROX_USER on Proxmox
 
 require_vm_id() {
@@ -104,45 +102,21 @@ wait_for_vm_ssh() {
 resolve_remote_repo() {
   local p
   p="$(vm "cd ~/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
-  [[ -n "$p" ]] || p="$(vmguest "cd ~${VM_USER}/${REMOTE_REPO} 2>/dev/null && pwd" 2>/dev/null)"
   printf '%s' "$p"
 }
 
-# Re-inject the validation pubkey into ~$VM_USER/.ssh/authorized_keys OUT OF
-# BAND via the Proxmox guest agent (qm guest exec runs as root inside the VM
-# and does not depend on SSH). This is necessary because secharden-ssh replaces
-# authorized_keys with the managed production key set, which would otherwise
-# lock out the bootstrap key used to drive validation. No-op if SSH still works.
+# SSH-only access policy (AGENTS.md): if SSH access is lost (e.g. secharden-ssh
+# replaced authorized_keys), the agent does NOT re-inject keys via guest-agent
+# or any other back-channel. Surface the loss for manual restore.
 restore_vm_access() {
   [[ "$RESTORE_ACCESS" = "1" ]] || { log "RESTORE_ACCESS=0; skipping access restore."; return 0; }
-  [[ -f "$ACCESS_PUBKEY" ]] || { log "WARN: ACCESS_PUBKEY not found ($ACCESS_PUBKEY); cannot restore access."; return 0; }
   if vm 'true' >/dev/null 2>&1; then
-    log "SSH access already works; no need to restore."
+    log "SSH access works; no restore needed."
     return 0
   fi
-  log "SSH access lost (expected after secharden-ssh). Restoring via Proxmox guest agent..."
-  local payload_b64
-  # Leading newline guards against the managed authorized_keys lacking a
-  # trailing newline (which would otherwise concatenate two keys into one).
-  payload_b64="$(printf '\n%s' "$(cat "$ACCESS_PUBKEY")" | base64 -w0)"
-  prox "qm guest exec $VM_ID -- /bin/sh -c 'echo $payload_b64 | base64 -d >> /home/${VM_USER}/.ssh/authorized_keys'" \
-    >/dev/null 2>&1 || { log "WARN: guest-agent key append failed."; return 0; }
-  prox "qm guest exec $VM_ID -- /bin/sh -c 'chown ${VM_USER}:${VM_USER} /home/${VM_USER}/.ssh/authorized_keys; chmod 600 /home/${VM_USER}/.ssh/authorized_keys'" \
-    >/dev/null 2>&1 || true
-  if vm 'true' >/dev/null 2>&1; then
-    log "Access restored."
-    return 0
-  fi
-  # If SSH still fails after re-injecting the key, 2FA is almost certainly the
-  # cause (secharden-2fa enforces publickey+keyboard-interactive, which no
-  # non-interactive SSH client can satisfy). That is expected and not fatal:
-  # the guest agent still gives us full out-of-band access for log fetch and
-  # the validation suite.
-  if vmguest 'grep -q "^AuthenticationMethods" /etc/ssh/sshd_config' >/dev/null 2>&1; then
-    log "SSH requires 2FA (expected after secharden-2fa); using guest agent for further access."
-  else
-    log "WARN: access still not working after restore and 2FA not detected. Check sshd_config."
-  fi
+  log "WARN: SSH access lost (e.g. secharden-ssh replaced authorized_keys)."
+  log "      Per SSH-only access policy (AGENTS.md), no back-channel re-inject."
+  log "      Restore manually: console login + re-add the bootstrap key, then re-run."
 }
 
 # ---------------------------------------------------------------------------