docs: session cleanup — coverage matrix, questions v2, handoff pointer [#343]

- OAM coverage matrix published on Discourse #309
- questions-v2.md: Q1-Q6 resolved, Q7-Q9 added
- aug11.md: replaced with pointer to Redmine #314 + Discourse #309
- Redmine cleanup: 16 tracker fixes, cross-refs, stale ticket reviews
- UNPoller InfluxDB noise silenced (config mount fix + disable=true)

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

.crush/.gitignore

@@ -0,0 +1,3 @@
+*
+!crush.json
+!.gitignore

.crush/crush.json

@@ -0,0 +1,11 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "^(bash|edit|write|multiedit|lsp_replace_symbol|lsp_rename)$",
+        "command": "./hooks/ticket-gate.sh",
+        "timeout": 5
+      }
+    ]
+  }
+}

.env.example

@@ -0,0 +1,7 @@
+# PFVCluster environment variables
+# Copy to .env and fill in values for local development/testing.
+
+# Pi-hole (netinfra/pihole/docker-compose.yml)
+PIHOLE_WEB_PASSWORD=changeme
+
+# Shellcheck wrapper (tests/shellcheck.sh) — no config needed, uses Docker.

.gitignore

@@ -1,5 +1,9 @@
-# Crush internal state
+# Crush internal state (track crush.json config, ignore session data)
-.crush/
+.crush/crush.db
+.crush/crush.db-*
+.crush/logs/
+.crush/memory/
+.crush/active-ticket
 
 # OS/editor
 .DS_Store
@@ -37,3 +41,6 @@ returned-logs/
 # Kubernetes secrets (kubeconfig contains embedded client certs)
 k8s/kubeconfig.yaml
 k8s/*.token
+
+# Pi-hole web UI password (real value only in on-box .env, never committed)
+netinfra/pihole/.env

AGENTS.md

@@ -1,40 +1,311 @@
 # Agent Guidelines
 
-**Top-level files:** [`README.md`](README.md) (project overview),
+## Agent Authority (NON-NEGOTIABLE)
-[`STATUS.md`](STATUS.md) (living status, agent-maintained),
+
-[`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else
+**No work is permissible on any system without an approved Redmine ticket.
-lives in subdirectories.
+There are no exceptions to this rule.**
+
+### Scope of authority
+
+1. **Tickets govern all work.** The agent performs ONLY the work described in
+   the approved Redmine ticket. Anything outside that scope — no matter how
+   small, helpful, or "obvious" — is prohibited.
+
+2. **No autonomous system changes.** The agent does not modify, configure,
+   create, delete, or grant anything on a production system unless it is
+   explicitly directed by an approved ticket. "Production system" means
+   every system in the fleet — there is no "test" exception unless the ticket
+   says so.
+
+3. **Propose, never implement.** If the agent discovers additional work that
+   should be done — a bug, a misconfiguration, a missing dependency, an
+   enhancement — it does NOT implement it. Instead, it creates a Redmine
+   sub-ticket (status Feedback) describing the finding and surfaces it to
+   the user for approval.
+
+4. **Security and access changes require extra scrutiny.** Changes to sudoers,
+   SSH keys, user accounts, firewall rules, authentication policy, file
+   permissions, or any privilege-related configuration are treated as
+   policy decisions, not implementation details. The agent may suggest
+   them but NEVER implements them without explicit user direction in the
+   ticket or a sub-ticket the user has approved.
+
+5. **The user makes policy. The agent implements policy.** The agent does
+   not decide who gets sudo, what keys go where, what services run, or what
+   the access model is. The agent executes the user's decisions, exactly
+   as specified.
+
+6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
+   require something not explicitly authorized, or if the agent is unsure
+   whether an action is in scope — STOP and ask the user via the ticket
+   or directly. Asking is always acceptable. Overstepping is never
+   acceptable.
+
+7. **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user. This applies to ALL tickets,
+   no exceptions.
+8. **User acceptance testing is MANDATORY before declaring work done.**
+   The agent performs implementation and technical validation (services
+   running, configs correct, APIs responding). The user performs UAT —
+   visually confirming dashboards render data, alerts deliver, tools are
+   usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
+   closing, and MUST NOT move to the next ticket until the user explicitly
+   accepts the work. "Technically wired but blank dashboard" is NOT done.
+9. **NEVER access a database directly if an API exists.** APIs are the
+   stable contract; databases change schemas without warning. If a tool
+   has an API, use it — exhaust all API endpoints, check the docs, try
+   alternative methods. Only fall back to direct DB access as a last
+   resort AND with explicit user approval for that specific instance.
+   Soon all DB access will route through a proxy under zero trust;
+   building API-first habits now ensures that transition is clean.
+
+### Access-channel policy: SSH only (NON-NEGOTIABLE)
+
+**The qemu guest-agent is NEVER an access, execution, or key-delivery
+channel.** SSH (`sshd`) is the only approved remote access path. Every
+command must flow through sshd so it is captured by the standard
+auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
+environment — there is no back-door exception, ever.
+
+- **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
+  over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
+  mode). This includes using guest-agent to *deliver* an SSH key, even if
+  the resulting SSH login is itself audited — the delivery bypassed audit.
+- **Allowed:** installing or checking qemu-guest-agent for its intended
+  purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
+  `agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
+  never execution.
+- **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
+  any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
+  no guest-exec mode.
+- **If a system is locked out** (no SSH key, no guest-exec path): surface
+  it to the user. Do NOT improvise an alternate back-channel. The user
+  authorizes the unblock method (console login, credential, etc.).
+
+### Ticket-closing policy (NON-NEGOTIABLE)
+
+**NEVER close a Redmine ticket without explicit user permission.** You
+may SUGGEST a close when the result is clearly scoped and delivered. If
+it's ambiguous whether the work is truly complete, don't suggest a close
+— leave that decision to the user. This applies to ALL tickets, no
+exceptions.
+
+### What this means in practice
+
+- Discovered a typo in a config during approved work? **Finish the approved
+  work. Create a sub-ticket for the typo. Do not fix it inline.**
+- Think a system should also have localuser sudo configured? **Do not add
+  it. Propose it in a sub-ticket.**
+- Need to install a package the ticket didn't mention? **Ask first.**
+- Found a security issue? **Create a ticket immediately with full details.
+  Do not remediate without approval.**
+
+This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
+traceable to an approved ticket. There is no "I thought it would help."
+
+## Quick Start
+
+**You are an AI agent working on this project. Your first actions, in order:**
+
+> **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
+> check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
+> the scope-alignment gate. You MUST NOT begin any task work until the user
+> (a) confirms your rule summary AND (b) names the ticket to work on. This runs
+> **every session, automatically** — the user should never have to ask for it.
+
+1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
+2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
+3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
+4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
+   back to the user before doing any work: Agent Authority (ticket-governed,
+   propose-never-implement), Access-Channel SSH-only policy, Remote access
+   (remote.sh chokepoint mandatory, DNS names only — never IP literals),
+   Questions policy (no harness question-tools), Documentation policy
+   (Discourse is SoR), Redmine tracking, Git policy (always commit+push,
+   shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
+   routing chokepoints (`tests/remote.sh`), the mechanically enforced
+   `check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
+   and wait for the user to confirm. This checkpoint guarantees every
+   session starts aligned.
+5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
+6. **Check current state:** `git log --oneline -10`.
+7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
+8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
+   priority list (or the Redmine queue if no handoff exists). Do NOT scan the
+   full ticket queue and pick work on your own — **the user directs what gets
+   worked on, always.** Note any new or urgent items from the Redmine check,
+   then ask which ticket to work on tonight. **STOP and wait.** Do not set
+   `.crush/active-ticket`, do not read systems, do not run diagnostics, until
+   the user names the target. This gate prevents the agent from burning
+   context on work the user didn't ask for.
+
+## Enforcement Model
+
+Git hooks (`scripts/pre-commit`, `scripts/pre-push`) enforce the rules defined in
+`scripts/check-rules.sh`. The rules engine checks: shellcheck (zero warnings
+including info-level), Docker image pinning (no `:latest`), container naming,
+required files, Discourse pointer headers, and more. Run `bash scripts/check-rules.sh`
+for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
+(emergencies only).
+
+## Task Tracking
+
+- **Redmine is the system of record for all work.**
+- **NEVER close a ticket without explicit user permission.** Suggest a
+  close when clearly scoped/delivered; if ambiguous, don't suggest.
+- **Ticket-first enforcement (mechanically enforced).** Before starting
+  any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
+  The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
+  until this file exists. If no ticket exists, CREATE ONE FIRST via
+  redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
+- **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
+  session. The pre-commit hook blocks commits while any task remains unchecked.
+- Clear WORKING.md before responding to the user.
+
+## Rolling HUD (session-scoped, NOT persisted)
+
+The rolling HUD is a **live status display** the agent maintains throughout
+the session to help the human follow along. It is NOT a system of record —
+Redmine, Discourse, and git are the durable systems. The HUD exists purely
+for the human's situational awareness during the session.
+
+- **Format:** a compact block shown at the end of each significant response
+  (after completing a step, hitting a blocker, or pivoting). Example:
+
+  ```
+  ┌─ SESSION HUD ────────────────────────────────────────
+  │ Active: #343 (Monitoring coverage matrix)
+  │ Done:   ✓ scope-alignment gate added to both AGENTS.md
+  │         ✓ committed + pushed (6a2550b)
+  │ Now:    drafting coverage matrix on Discourse
+  │ Next:   → #341 TEMPer USB (deploy on pfv-tsys1)
+  │         → #338 LibreNMS alerts (BLOCKED: pushover.env)
+  ├─ USER ACTION ITEMS ──────────────────────────────────
+  │ • Populate ~/.creds/pushover.env (unblocks #338, #428)
+  │ • Populate ~/.creds/prometheus.env + grafana.env (#430)
+  └──────────────────────────────────────────────────────
+  ```
+
+- **Placement:** may be written to `.crush/hud.md` on disk to keep context
+  window smaller (re-read and update rather than hold in memory). Never
+  committed to git. Wiped at session end.
+- **Sections:**
+  - **Active:** current ticket number + one-line description
+  - **Done:** ✓ items completed this session (append as work progresses)
+  - **Now:** what the agent is actively doing
+  - **Next:** the queued items (per handoff priority or user direction)
+  - **User action items:** things ONLY the user can do (populate creds,
+    physical work, manual deploys) with the tickets they unblock
+- **When to show it:** after each logical unit of work, at blockers, and
+    when pivoting between tickets. Not every trivial response — use judgment.
+- **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
+    work completes, update the durable systems (ticket notes, Discourse wiki,
+    commits). The HUD just tracks the live narrative for the human.
+
+## Working Style
+
+- **Stop over-thinking.** Get to code and output faster. Explore with code;
+  gather ground truth. Do not burn tokens reasoning about things a quick command
+  answers.
+- **Farm work out to deterministic tooling:** linters, LSPs, formatters, test
+  runners. If an LSP is wired up, use it; otherwise pull a Docker image and lint
+  inside it.
+- **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
+  Never read 10+ files sequentially; batch into agent calls.
+- **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
+  system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
+  (status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
+  If a command hits the timeout, STOP and investigate root cause — never
+  blindly retry. A hung command is a failed command. Detect failure fast,
+  diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
+  This applies to ALL tools — bash, docker, CLIs, sub-agents.
+
+## Questions (NON-NEGOTIABLE)
+
+**NEVER use a harness "question"/"ask user" tool** (structured prompts,
+modal forms, tabbed questions). Banned across every project, every harness.
+They are not portable, not version-controlled, and bypass the git record.
+**All questions go in the current `questions-v(N).md` file** — write the
+question; the human edits the answer inline in the same file. **Version up
+the filename each time answers land** (v1 → v2 → v3...): create
+`questions-v2.md` with resolved Q&A marked, new questions appended. This
+preserves the history of each Q&A round. Synthesize resolved Q&A into
+Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
+§10.
+
+## Documentation policy (IMPORTANT)
+
+**Discourse is the canonical source of truth for all knowledge documentation.**
+Knowledge docs (architecture, runbooks, references, audits, policies) have been
+migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops)
+as wiki topics in the **VP TechOps** category.
+
+All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now
+**pointers** that link to their corresponding Discourse topic. **Do not update
+documentation content in git** — edit the Discourse wiki topic instead. Git
+edit history no longer serves as the documentation changelog; Discourse
+preserves wiki edit history automatically.
+
+Code (scripts, configs, playbooks) still lives in git as the source of truth
+for executables. Only *documentation* moved to Discourse.
+
+## Top-level files
+
+All `.md` files now point to Discourse. The key pointers:
+
+| File | Points to | Discourse topic |
+|------|-----------|-----------------|
+| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) |
+| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) |
+| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) |
+
+Work tracking stays in [Redmine](https://projects.knownelement.com).
 
 ## Repository Layout
 
 ```
-provisioning/        Server provisioning (SetupNewSystem.sh, security, 2FA)
+dcinfra/            Data-center infra: PDU (powerman), serial console (console), UPS (ups)
-tests/               Test suite + VM validation harness
+netinfra/           DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup);
-dns-cluster-setup/   Technitium DNS cluster replication
+                    switch captures (switches); DHCP config (dhcp)
-k8s/                 k3s cluster setup scripts (3-node HA over Tailscale) + docs/
+k8s/                k3s cluster setup scripts (HA control plane over Tailscale) + docs/
-powerman/            Cyclades PM10i PDU management via powerman
+proxmox/            Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
-console/             Serial console management (ser2net + conman) for switches
-ups/                 UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1
-perf/                Proxmox perf tuning, fleet audit, iperf
-proxmox/             Proxmox fleet docs (hardware audit, capacity, k8s host planning)
-netinfra/            pfv-netinfra-01/02 DNS/NTP/DHCP setup
-switches/            Switch configuration captures
 awx/                Ansible AWX deployment (k3s + AWX Operator)
+tests/              Test suite + VM validation harness + remote.sh SSH chokepoint
+scripts/            Framework: git hooks, rule engine (check-rules.sh), shared lib
 docs/               Server-build docs, docmap index, and archive
+archive/            Historical/superseded code (provisioning -> replaced by KNELIAC project)
 vendor/             Vendored KNELShellFramework
 ```
 
-- **Self-locating scripts**: All provisioning scripts derive their own
+- **Server provisioning moved to KNELIAC**: The
-  location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run
+  [`archive/provisioning/`](archive/provisioning/) tree is historical. Active
-  from anywhere.
+  server provisioning lives in the **KNELIAC** project at
-- **Local configs are the source of truth**: Files in
+  `/home/reachableceo/projects/KNELIAC`.
-  [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
+- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
-  `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
-- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
   extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
-- **SSH in Crush**: Direct ssh/scp is blocked. Use
+- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
-  [`tests/remote.sh`](tests/remote.sh) or
+  through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
-  [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh).
+  (Proxmox hosts + all VMs) or
+  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh)
+  (DNS infra hosts: netinfra-01/02, tsrouter, netboot). NEVER call
+  `ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
+  rejects it. There are no exceptions.
+- **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
+  (neither LAN nor Tailscale IPs) in any command, script, or config.
+  ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
+  `VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
+  DNS name does not resolve, fix it in DNS (Technitium) or consult the
+  [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  Do NOT fall back to IP literals. This rule eliminates the per-session
+  discovery tax of finding the right IP for each host.
+- **How to access a production VM:**
+  1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
+  3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
+     and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
+     NOT execution) to find the Tailscale DNS name, then access via that.
 
 ## Git Policy
 
@@ -60,53 +331,257 @@ vendor/              Vendored KNELShellFramework
    justification. A script that emits any diagnostic is a protocol violation.
    Non-bash scripts (PHP with `.sh` shebang `#!/usr/bin/php`, etc.) are exempt.
 
-## Automatic Gardening Protocol
+## Redmine Tracking Policy
 
-**Docs and code must be kept in sync.** After any work session, an agent MUST:
+**Redmine is the system of record for all work.** Do not track status,
+checklists, or TODOs in repo files. Use Redmine tickets instead.
 
-1. **Update [`STATUS.md`](STATUS.md)** — reflect completed work, new issues,
+- **URL:** https://projects.knownelement.com
-   changed infrastructure state. This file is human read-only; agents own it.
+- **Version:** Potential to Kinetic Ready (due 2026-09-30)
-2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added,
+- **Project:** Known Element Enterprises - Technology & Facility Services (id 55)
-   removed, or substantively changed, update the table and "Last Reviewed"
+
-   date.
+### Rules
-3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after
+1. **Every piece of work** (feature, fix, deployment, config change) gets a
-   any rename or restructure. Fix all references in the same commit.
+   Redmine ticket. If one doesn't exist, create it.
-4. **Verify new docs are linked** — every new `.md` file must appear in
+2. **Reference tickets in docs and commits** using `[#NNN]` notation.
-   [`docs/docmap.md`](docs/docmap.md) and be linked from at least one other
+   Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
-   doc.
+3. **When work completes**, update the ticket: set done ratio to 100%,
-5. **If a new top-level directory was created, update ALL directory listings:**
+   add a note describing what was done and where the code lives.
-   - [`README.md`](README.md) → "Directory Structure" table
+   **NEVER close a ticket without explicit user permission.** You may
-   - [`AGENTS.md`](AGENTS.md) → "Repository Layout" code block
+   SUGGEST a close when the result is clearly scoped and delivered. If
-   - [`AGENTS.md`](AGENTS.md) → "Key Scripts" table (if the directory has
+   it's ambiguous whether the work is truly complete, don't suggest a
-     an entrypoint script)
+   close — leave that decision to the user.
-   Missing any one of these is a protocol violation.
+4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
-6. **Self-audit before commit.** Before committing, run:
+5. **Link code to tickets** — ticket descriptions and notes should reference
+   the relevant file paths in this repo (e.g., `dcinfra/ups/`).
+
+### CLI access (read + write)
+
+Tickets are managed via the `redmine-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the subtask escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/redmine/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+
+# Your queue (project 55):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55
+
+# Show / create / update / close:
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest show 367
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest create -p 55 -s "Subject" -d "desc"
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest update 367 -n "Done: committed in abc123" --done-ratio 100
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest close 367
 ```
-   grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md
+
+Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
+Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
+Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
+**Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
+`python-redmine` escape hatch inside the container (see
+`tooling-cli/redmine/AGENTS.md`). Always `show` a ticket before updating it.
+
+Credentials (`REDMINE_URL`/`REDMINE_API_KEY`) live in the centralized store
+at `~/.creds/redmine.env`.
+
+## Documentation Workflow
+
+**Discourse is the source of truth for all knowledge docs.** After any work
+session, an agent MUST:
+
+1. **Update Discourse wiki topics** — if infrastructure facts changed (new
+   VM, IP change, host retired), edit the relevant wiki topic at
+   [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops).
+2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297))
+   if tickets were opened or closed.
+3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'`
+   after any rename or restructure. Fix all references in the same commit.
+4. **If a new top-level directory was created**, update:
+   - `AGENTS.md` → "Repository Layout" code block
+   - `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script)
+   - Create a new Discourse wiki topic for any documentation
+5. **Self-audit before commit.** Code changes must be internally consistent.
+   Documentation changes go to Discourse, not git.
+
+### CLI access (read + write)
+
+Wiki topics are managed via the `discourse-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the raw-API escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/discourse/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# List VP TechOps topics:
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest ls -c vp-techops
+
+# Show a topic / edit a wiki post (find post id via `show`):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest show 297
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest update <post_id> -b "new markdown body"
 ```
-   Every new top-level directory must appear in all four files.
+
+VP TechOps = category **74**. Key topics: #296 (project overview), #297
+(ops status), #298 (audit log). The API user is trust-level 4 but **not
+admin** — admin-only ops (category creation, setting the wiki flag) will
+403; surface those to the user rather than retrying. **Gotcha:**
+`update`/`delete` take a post **id**, not a post number. Never create a new
+topic for an update to existing knowledge — edit the wiki post in place.
+
+Credentials (`DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`)
+live in the centralized store at `~/.creds/discourse.env`.
 
 ## Key Scripts
 
 | Script | Purpose |
 |--------|---------|
-| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning |
+| [`scripts/check-rules.sh`](scripts/check-rules.sh) | Rule audit engine (shellcheck, image pinning, Discourse pointers, required files) |
+| [`scripts/setup-hooks.sh`](scripts/setup-hooks.sh) | Install git hooks (pre-commit, pre-push) |
+| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
+| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
+| `redmine-cli` container | Redmine CLI (ticket read/write via `docker run`; see `tooling-cli/redmine/`) |
+| `discourse-cli` container | Discourse CLI (wiki topic read/write via `docker run`; see `tooling-cli/discourse/`) |
+| `dns-cli` container | Technitium DNS CLI (zones, list, add, delete, search, flush; see `tooling-cli/dns/`) |
 | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
 | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
-| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
+| [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
 | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
-| [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman |
+| [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman |
-| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
+| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
-| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
+| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
-| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
+| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
+| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |
+
+## Switch Console Access (conman)
+
+Switch configs are pulled via serial console through a conman + ser2net
+stack on pfv-tsys4.
+
+- **conmand server:** pfv-tsys4 (port 7890)
+- **ser2net:** pfv-tsys4, TCP ports 2001-2006
+  (do NOT connect to ser2net directly — it conflicts with conman's
+  persistent sessions; always use the conman client)
+- **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
+  — drives console sessions read-only via PTY. No expect/tcl required.
+- **Command files:** [`netinfra/switches/`](netinfra/switches/) — `.cmds` files
+  with switch-specific show commands
+- **Query available consoles:** `conman -d 100.70.77.93 -q`
+
+```bash
+# Pull a switch config (example):
+CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+| Console name | TCP port | Device |
+|--------------|----------|--------|
+| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
+| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
+| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
+| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
+| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
+| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
+
+## Mandatory: Document ALL Infrastructure Changes
+
+**This is non-negotiable. Every infrastructure change (VM config, disk
+cache, network setting, service config, storage migration) MUST be
+documented BEFORE moving to the next task step — not "later" or "at the
+end."**
+
+Required for EVERY infrastructure change:
+
+1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
+2. **Discourse audit log** — reply to topic
+   [#298](https://community.turnsys.com/t/298) with a dated entry (what
+   changed, why, where).
+3. **Discourse relevant wiki topic** — update the architecture/reference
+   topic if the change affects documented infrastructure facts (storage
+   #300, network #299, k8s #305, etc.).
+4. **Git commit** — if code/config changed in the repo, commit + push
+   immediately per the Git Policy above.
+
+## Tooling
+
+- `gh`, `docker`, `jq` available on the workstation.
+- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
+  `koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
+- For raw API calls not covered by the Redmine/Discourse CLIs, use
+  `python3` inside the CLI Docker containers (escape hatch pattern in
+  `tooling-cli/{discourse,redmine}/AGENTS.md`).
+- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
+  the python-in-Docker escape hatch for HTTP writes.
+
+## Key Commands Quick Reference
+
+```bash
+# Tests + validation:
+bash tests/run-tests.sh                        # test suite
+bash tests/vm-validation.sh                    # VM validation
+bash tests/shellcheck.sh                       # shellcheck whole repo
+bash tests/shellcheck.sh path/to/*.sh          # shellcheck specific files
+
+# Redmine + Discourse sanity checks (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# Proxmox host access (DNS names only — never IPs):
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
+
+# Production VM access (DNS names only — never IPs):
+VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
+
+# DNS infra access:
+bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
+
+# Switch console (read-only config pull — DNS names only):
+CONMAN_SERVER=pfv-tsys4:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+## Credential Management
+
+- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
+- User goal: migrate all keys to Hashicorp Vault
+  (`vault.knownelement.com`). No vault token present yet
+  (`~/.vault-token` missing). Track as high-priority TODO.
 
 ## Key Docs
 
-→ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.**
+→ **All documentation lives on Discourse:**
+  [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops)
+
+→ All `.md` files in this repo are pointers to Discourse topics.
+
+→ **Complete Linux System Inventory:**
+  [Topic #307](https://community.turnsys.com/t/307) — every Linux system
+  (hosts + VMs + physical), with Tailscale IPs, DNS names, SSH access
+  status, and tuned profiles. Reference this for monitoring coverage,
+  access management, and hostname consistency.
 
 ## Project Context
 
 Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
-Production lives on a Cloudron VPS in Reston VA. See
+Production lives on a Cloudron VPS in Reston VA. See the
-[`STATUS.md`](STATUS.md) for current state and
+[Operations Status topic](https://community.turnsys.com/t/297) for the ticket
-[`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) for the fleet report.
+index and infrastructure summary.
+All work is tracked in [Redmine](https://projects.knownelement.com)
+(version: Potential to Kinetic Ready, due 2026-09-30).

Makefile

@@ -0,0 +1,50 @@
+# Makefile — convenience dispatch to scripts/.
+#
+# Not required. The scripts in scripts/ are the real entry points and work
+# standalone. This file just gives you short verbs if you're at a terminal.
+#
+# In Mode 2 (Hermes/OWUI/MCP), agents call the scripts directly or via API —
+# they don't need this file.
+
+# Project-specific overrides for check-rules.sh
+export PROJECT_DOC_EXEMPT ?= AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v1.md BASELINE-PROMPT.md PATTERNS.md
+export PROJECT_DISCOURSE_HOST ?= community.turnsys.com
+
+.PHONY: setup validate fast lint test garden up down status clean help
+
+help: ## Show available targets
+	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
+
+setup: ## Install git hooks
+	@bash scripts/setup-hooks.sh
+
+validate: ## Full rule audit (includes tests)
+	@bash scripts/check-rules.sh
+
+fast: ## Fast rule audit (pre-commit equivalent)
+	@bash scripts/check-rules.sh --fast
+
+lint: ## Lint shell scripts (shellcheck via docker)
+	@docker run --rm -v "$$(pwd):/mnt" koalaman/shellcheck:stable \
+		$$(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print | sed 's|^\./|/mnt/|') || true
+
+test: ## Run the test suite (override per project)
+	@bash scripts/test.sh
+
+garden: ## Doc-sprawl / Discourse-migration report
+	@bash scripts/garden.sh
+
+up: ## Bring up the docker-compose stack
+	@bash scripts/up.sh
+
+down: ## Bring down the docker-compose stack
+	@bash scripts/down.sh
+
+status: ## Show repo status snapshot
+	@echo "== branch =="; git branch --show-current 2>/dev/null || echo "(no branch)"
+	@echo "== last commit =="; git log --oneline -1 2>/dev/null || true
+	@echo "== working tree =="; git status --short 2>/dev/null || echo "(not a git repo)"
+	@echo "== STATUS.md head =="; sed -n '1,12p' STATUS.md 2>/dev/null || echo "(no STATUS.md)"
+
+clean: ## Remove build/test artifacts (override per project)
+	@echo "make clean: nothing to clean — override this in your project's Makefile."

README.md

@@ -1,54 +1,10 @@
-# PFVCluster
+# README.md
 
-Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)**
+> **Project overview, architecture, quick start**
-
+>
-## Directory Structure
+> **Read it here:** https://community.turnsys.com/t/296
-
+>
-| Directory | Description |
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-|-----------|-------------|
+> Do not update content here — edit the Discourse wiki topic instead.*
-| [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) |
-| [`tests/`](tests/) | Test suite + VM validation harness |
-| [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts |
-| [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs |
-| [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 |
-| [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
-| [`ups/`](ups/) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale |
-| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
-| [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
-| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |
-| [`switches/`](switches/) | Switch configuration captures |
-| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) |
-| [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive |
-| [`vendor/`](vendor/) | Vendored KNELShellFramework |
-
-## Quick Start
-
-### Provision a new server
-```bash
-sudo bash provisioning/SetupNewSystem.sh
-```
-
-### Validate on the sandbox VM
-```bash
-VM_ID=6000 ./tests/vm-validation.sh all
-```
-
-### Deploy DNS cluster
-```bash
-cd dns-cluster-setup/ && ./setup.sh all
-```
-
-### Deploy perf tunings
-```bash
-cd perf/ && ./deploy-tuning.sh
-```
-
-## Architecture
-
-- **Proxmox hosts**: 7 standalone PVE installs managed via PDM
-- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
-- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
-- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
-- **Backups**: Proxmox Backup Server (PBS)

STATUS.md

@@ -1,189 +1,10 @@
-# Project Status
+# STATUS.md
 
-> **Human read-only. Agents maintain this file automatically after each work
+> **Documentation moved to Discourse — the canonical source of truth.**
-> session.** Do not edit by hand — the next agent run will overwrite it.
-
-> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — HA NUT integration live
 >
-> **Capacity targets are now codified** in
+> **Ticket index + infrastructure summary**
-> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
+>
-> steady-state, ~50% idle CPU. The current placement analysis lives in
+> **Read it here:** https://community.turnsys.com/t/297
-> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md).
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-## Current State: STABLE
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-The merged PFVCluster repo is fully operational across provisioning, DNS
-infrastructure, Proxmox cluster ops, and k8s control plane.
-
-## Completed Work
-
-### Server Provisioning (validated on sectestbed-sandbox)
-- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
-- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
-- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
-- [x] DNS resolv.conf: managed static file pointing at netinfra pair
-- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
-- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
-
-### DNS Cluster (pfv-netinfra-01/02)
-- [x] Production Technitium config replicated from tailscale-router (read-only)
-- [x] 124 zones on both nodes (knel.net + reverse DNS)
-- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
-- [x] Zone replication via rsync systemd timer (every 60s)
-- [x] Credentials + 2FA replicated identically to production
-- [x] Both LAN IPs resolve knel.net device names + recurse externally
-- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
-
-### Proxmox Cluster Ops
-- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
-- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
-- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73%
-- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
-- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg
-- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
-- [x] **tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator** (verified: 8/8 NFS stores active on both)
-- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
-- [ ] tsys4: blocked on PCIe NIC + RAM install
-- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
-- [ ] tsys2: pending rebuild from Win10 to Proxmox
-
-### Kubernetes Control Plane (k3s HA — LIVE)
-- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
-- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
-- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
-- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
-  Tailscale IPs, CoreDNS, API server, workload isolation
-- [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
-- [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
-- [ ] Workers (wnodes) not yet joined to this cluster
-- [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
-  preserved in `k8s/docs/` for future ITAR/classified cluster.
-
-### PDU Management (powerman on pfv-tsys1 — LIVE)
-- [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
-- [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink
-  `/dev/cyclades-pm10`
-- [x] powermand listening on `127.0.0.1:10101` + Tailscale `100.121.189.98:10101`
-- [x] All 10 outlets defined as `outlet-1` through `outlet-10`
-- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
-- [ ] Rename outlets to match physical devices (Friday onsite)
-
-### UPS Management (NUT on pfv-tsys1 — LIVE)
-- [x] **APC Smart-UPS C 1500** fully reporting via NUT usbhid-ups + APC HID 0.100
-- [x] upsd listening on Tailscale `100.121.189.98:3493` (port 3493)
-- [x] Local upsmon (master) configured for graceful hypervisor shutdown
-- [x] Home Assistant credentials generated (`homeassistant` user in upsd.users)
-- [x] Scripts in `ups/`: discover, setup (idempotent), status
-- [x] Home Assistant NUT integration **configured via REST API** (`ups/setup-ha-nut.sh`)
-- [x] **Key fix:** upsd now listens on LAN IP (192.168.3.11) in addition to
-  Tailscale — HAOS runs Tailscale as an isolated add-on, so the HA container
-  cannot route to Tailscale IPs. Shared vmbr0 bridge provides instant LAN path.
-- [x] **Key fix:** udev rule covers BOTH `usb` and `hidraw` subsystems (driver
-  opens raw `/dev/bus/usb` after setuid drop, not `/dev/hidraw`)
-- [ ] **Tripp Lite UPS blocked:** driver matches TrippLite HID 0.85 subdriver
-  but fails reading 878-byte HID Report Descriptor (EAGAIN via USB hub).
-  Needs physical reseat to direct motherboard USB port or cable replacement.
-  Driver masked to prevent restart-loop spam.
-- [ ] **Home Assistant NUT integration:** configured and live (3 sensors:
-  battery charge, status, status data). This APC (FW 02.2, mfg 2012) does not
-  report `ups.load` over USB HID — confirmed via both NUT and apcupsd tests.
-  No wattage for Energy Dashboard; smart plug on UPS output is the practical
-  path. apcupsd package conflicts with nut-server (mutually exclusive).
-
-### Console Management (ser2net telnet(rfc2217) + conman on pfv-tsys4 — LIVE)
-- [x] 7 network switch/router consoles managed via ser2net + conman
-- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
-  ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
-  symlinks that survive reboot regardless of enumeration order
-- [x] ser2net exposes all 7 consoles on telnet(rfc2217) TCP ports (2001-2007)
-  bound to **Tailscale IP only** (`100.70.77.93`)
-- [x] conman connects to TCP ports for logging + multiplexing (7 log
-  files active in `/var/log/conman/`)
-- [x] Both ser2net + conmand enabled via systemd (survive reboot)
-- [x] conmand systemd unit created (Debian package doesn't ship one)
-- [x] conmand remote access enabled (loopback=off, port 7890)
-- [x] Old `/root/conmap` + manual `screen` workflow replaced
-- [x] **Stair-stepping fix:** ser2net accepter changed from raw `tcp` to
-  `telnet(rfc2217)` so conman's telnet NVT negotiates binary mode properly
-- [x] **Validated 6/7 consoles working** via conman from workstation
-  (core-sw01, tor3-mgmt, tor3-stor, rrinfra-rtr, r2-tor-top, subodev-torsw)
-- [ ] **pfv-r2-sw (port 2007): DEAD** — zero bytes at all baud rates
-  (1200-115200), adapter present and mapped correctly (/dev/ttyUSB3).
-  Physical issue: device powered off, console cable disconnected, or
-  console port disabled in device config. Requires onsite investigation.
-
-### Ansible AWX (tsys-awx — LIVE)
-- [x] AWX 24.6.1 deployed via AWX Operator 2.19.1 on k3s single-node
-- [x] Accessible at `http://tsys-awx.knel.net` (port 80, LoadBalancer via k3s ServiceLB)
-- [x] Admin login: `admin` / `REDACTED_PASSWORD`
-- [x] PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
-- [x] VM resized: 32→60 GB disk, swap converted to swapfile
-- [x] Scripts in `awx/`: install-k3s, deploy-awx, verify-awx
-- [ ] HTTPS/TLS not yet configured (currently HTTP on port 80)
-- [ ] Execution environments not yet customized
-
-### Repo Merge
-- [x] KNELServerBuild merged into PFVCluster (history preserved)
-- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
-- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
-
-## Known Issues
-
-| Issue | Impact | Status |
-|-------|--------|--------|
-| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
-| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 73%/37%/62% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §11 |
-| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
-| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ |
-| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
-| **D2 IOPS hotspot** | 17 of 26 NFS-backed VMs on one spindle; S1/S4 underused | Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11) |
-| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
-| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
-
-## Pending (next session priorities)
-
-> **Full phased action checklist with current/target inventory is in**
-> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §11.
-> Summary below. SSD/NVMe tiering is deferred to post-Friday.
-
-**Phase 0 — Storage config fixes — DONE ✓** (tsys6 S1 typo, tsys9 S1 missing)
-
-**Phase 1 — Cleanup:**
-- Delete cnode4 (601) on tsys6 (stopped, stale)
-- cnode5 (706) already deleted ✓
-
-**Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):**
-- wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4
-- rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2
-
-**Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):**
-- tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117)
-- tsys9→tsys6: tsys-siem(901)
-
-**Phase 4 — Grow wnode-tsys9:** 4c/4GB → 6c/8GB (after siem leaves tsys9)
-
-**Phase 5 — Join wnodes to k3s** (after migrations free resources)
-
-**Other:**
-- **Friday maintenance:** tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
-- **PDU:** Rename outlets in powerman.conf (Friday onsite)
-- **k8s deferred:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out
-- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. HA NUT integration live (battery charge, status sensors). No ups.load over USB HID (firmware limitation, confirmed via apcupsd test). Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port).
-
-## Infrastructure Summary
-
-| Component | Details |
-|-----------|---------|
-| Proxmox hosts | 7 standalone PVE, managed via PDM |
-| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
-| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
-| DNS production | tailscale-router (read-only source of truth) |
-| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
-| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
-| Backup | Proxmox Backup Server (PBS) |
-| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
-| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
-| **AWX** | **tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)** |
-| **Console** | **7 switch consoles via ser2net telnet(rfc2217)+conman on pfv-tsys4 (conmand:7890 on Tailscale)** |
-| **UPS** | **APC Smart-UPS C 1500 via NUT on pfv-tsys1 (upsd:3493 on Tailscale)** |
-| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |

WORKING.md

@@ -0,0 +1,11 @@
+# WORKING.md — Active Session Tracker
+
+Agent work only. The human decides when it's done.
+A commit is blocked while any task below remains unchecked.
+
+## Current Tasks
+
+(all done — session complete)
+
+- [x] Add ticket-gate Crush hook (blocks work without active ticket)
+- [x] Document active-ticket workflow in AGENTS.md + meta template

access-matrix.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/bash
+# access-matrix.sh — definitive access verification across all Linux Tailscale nodes.
+# Uses the correct SSH user(s) per system type, checks sudo where applicable.
+# Routes through remote.sh (the only allowed ssh path).
+set -u
+cd /home/reachableceo/projects/PFVCluster || exit 1
+
+# Policy-excluded systems (never attempt access)
+EXCLUDE=':tsys-cloudron:pfv-bms:tsys-umbrel:tsys-ucs-01:tsys-ucs-02:stlpc-bizoffice:ultix-highside:'
+
+# Determine the SSH user(s) for a given hostname and whether sudo is expected.
+# Returns "user1:user2:...:sudoflag" where sudoflag is "yes" or "no".
+users_for() {
+  local name="$1"
+  case "$name" in
+    pfv-tsys[0-9])                   echo "root:no" ;;
+    *-proxmox-datacenter)            echo "root:no" ;;
+    *-proxmox-pve)                   echo "root:no" ;;
+    *-proxmox-pbs)                   echo "root:no" ;;
+    *-proxmox-mailgw*)               echo "root:no" ;;
+    *-proxmox-backup*)               echo "root:no" ;;
+    tsys-ucs-*)                      echo "root:no" ;;
+    ultix-streaming)                 echo "root:no" ;;
+    stlpc-*)                         echo "root:labuser:no" ;;
+    ultix-field)                     echo "ultixfield:yes" ;;
+    subopi*)                         echo "subodev:yes" ;;
+    *)                               echo "localuser:yes" ;;
+  esac
+}
+
+check_user() {
+  local ip="$1" user="$2" expect_sudo="$3"
+  local out sudo
+  out=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'echo SSHOK; id -un' </dev/null 2>&1 | grep -oE 'SSHOK|keyboard-interactive|Connection refused' | head -1)
+  case "$out" in
+    SSHOK)
+      if [ "$expect_sudo" = "yes" ]; then
+        sudo=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'sudo -n true 2>/dev/null && echo SUDOOK || echo SUDONO' </dev/null 2>&1 | grep -oE 'SUDOOK|SUDONO' | head -1)
+        printf '%s(%s)' "$user" "${sudo:-?}"
+      else
+        printf '%s(ok)' "$user"
+      fi
+      ;;
+    keyboard-interactive) printf '%s(2FA)' "$user" ;;
+    'Connection refused')  printf '%s(NOSSH)' "$user" ;;
+    *)                      printf '%s(NOKEY)' "$user" ;;
+  esac
+}
+
+printf '%-32s %-16s %s\n' "NAME" "TS-IP" "ACCESS"
+printf '%-32s %-16s %s\n' "----" "-----" "------"
+
+tailscale status 2>/dev/null | awk '$4=="linux" {print $2, $1}' | sort | while read -r name ip; do
+  [ -n "$name" ] || continue
+  case "$EXCLUDE" in *":$name:"*) printf '%-32s %-16s %s\n' "$name" "$ip" "EXCLUDED"; continue;; esac
+
+  map=$(users_for "$name")
+  expect_sudo="${map##*:}"
+  users="${map%:*}"
+  result=""
+  IFS=':' read -ra user_list <<< "$users"
+  for u in "${user_list[@]}"; do
+    r=$(check_user "$ip" "$u" "$expect_sudo")
+    [ -z "$result" ] && result="$r" || result="$result $r"
+  done
+  printf '%-32s %-16s %s\n' "$name" "$ip" "$result"
+done

agent-bootstrap.sh

@@ -0,0 +1,60 @@
+#!/bin/sh
+# agent-bootstrap.sh
+#
+# Run INSIDE a guest (via SSH, console, or guest-agent) to bring the
+# system fully under agent management in one shot:
+#   1. install + enable qemu-guest-agent  (VMs only, skipped on bare metal)
+#   2. push the agent SSH key to root + AGENT_USER (+ labuser if present)
+#   3. grant AGENT_USER passwordless sudo
+#
+# AGENT_USER defaults to "localuser". Override for systems with a different
+# unprivileged agent user:
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+#
+# After this runs once, the agent has SSH+sudo immediately.
+#
+# Usage (from a root shell in the guest):
+#   bash agent-bootstrap.sh
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+
+set -eu
+
+KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIWms/uCXnjjo4KyxHBcYI2TDHe8OZ2wle6W/0hSRQLu reachableceo@ultix-streaming'
+AGENT_USER="${AGENT_USER:-localuser}"
+
+# 1. guest-agent (skip on bare metal — no virtio-serial device)
+if command -v systemd-detect-virt >/dev/null 2>&1 && \
+   [ "$(systemd-detect-virt --vm 2>/dev/null || echo none)" != "none" ]; then
+  if ! command -v qemu-ga >/dev/null 2>&1; then
+    if command -v apt-get >/dev/null 2>&1; then
+      apt-get update
+      DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
+    elif command -v dnf >/dev/null 2>&1; then
+      dnf install -y qemu-guest-agent
+    elif command -v yum >/dev/null 2>&1; then
+      yum install -y qemu-guest-agent
+    else
+      echo "WARN: no supported package manager; skipping agent install" >&2
+    fi
+  fi
+  systemctl enable --now qemu-guest-agent 2>/dev/null || \
+    systemctl enable --now qemu-ga 2>/dev/null || true
+fi
+
+# 2. SSH key for root + AGENT_USER + labuser (if present)
+for u in root "$AGENT_USER" labuser; do
+  if ! getent passwd "$u" >/dev/null 2>&1; then continue; fi
+  H=$(getent passwd "$u" | cut -d: -f6)
+  mkdir -p "$H/.ssh"; chmod 700 "$H/.ssh"
+  AK="$H/.ssh/authorized_keys"; touch "$AK"; chmod 600 "$AK"
+  grep -qF "$KEY" "$AK" || echo "$KEY" >> "$AK"
+  chown -R "$u": "$H/.ssh"
+done
+
+# 3. passwordless sudo for AGENT_USER only
+if getent passwd "$AGENT_USER" >/dev/null 2>&1 && [ -d /etc/sudoers.d ]; then
+  echo "${AGENT_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/010-agent
+  chmod 440 /etc/sudoers.d/010-agent
+fi
+
+echo BOOTSTRAP-DONE

archive/provisioning/ConfigFiles/99-pfv-nfs.conf

@@ -0,0 +1,23 @@
+# PFV NFS tuning sysctl overrides
+#
+# Applied AFTER tuned via pfv-nfs-tuning.service (systemd oneshot).
+# These override tuned's network-throughput/virtual-host 16MB TCP buffer
+# caps with 128MB for high-BDP NFS over 1-4 GbE LACP links.
+#
+# Install on ALL Proxmox hosts:
+#   cp 99-pfv-nfs.conf /etc/sysctl.d/99-pfv-nfs.conf
+#   cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service
+#   systemctl daemon-reload && systemctl enable --now pfv-nfs-tuning.service
+#
+# Created: 2026-07-31
+# Deployed: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
+
+net.core.rmem_max = 134217728
+net.core.wmem_max = 134217728
+net.core.rmem_default = 26214400
+net.core.wmem_default = 26214400
+net.core.netdev_max_backlog = 250000
+net.core.somaxconn = 65535
+net.ipv4.tcp_rmem = 4096 87380 134217728
+net.ipv4.tcp_wmem = 4096 65536 134217728
+net.ipv4.tcp_max_syn_backlog = 4096

archive/provisioning/ConfigFiles/pfv-nfs-tuning.service

@@ -0,0 +1,28 @@
+# PFV NFS tuning service
+#
+# Systemd oneshot that runs AFTER tuned.service to apply TCP buffer
+# overrides. The tuned daemon's profiles (network-throughput for storage
+# hosts, virtual-host for compute hosts) set 16MB TCP buffer caps which
+# are too small for high-BDP NFS over LACP links. This service force-
+# applies 128MB buffers after tuned has finished its configuration.
+#
+# Install:
+#   cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service
+#   systemctl daemon-reload
+#   systemctl enable --now pfv-nfs-tuning.service
+#
+# Created: 2026-07-31
+# Deployed: all 7 Proxmox hosts (tsys1/3/4/5/6/7/9)
+
+[Unit]
+Description=PFV NFS tuning (override tuned TCP buffer caps)
+After=tuned.service
+Requires=tuned.service
+
+[Service]
+Type=oneshot
+ExecStart=/sbin/sysctl -p /etc/sysctl.d/99-pfv-nfs.conf
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

aug11.md

@@ -0,0 +1,9 @@
+<!-- Discourse: https://community.turnsys.com/t/298 -->
+<!-- Redmine: https://projects.knownelement.com/issues/314 -->
+
+# Session Handoff: 2026-08-11/12 OAM
+
+**Full handoff lives in Redmine #314** (session summary note, 2026-08-12).
+**OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309).
+**Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303).
+**Open questions:** `questions-v2.md` in this repo.

awx/README.md

@@ -1,96 +1,10 @@
-# AWX — tsys-awx.knel.net
+# awx/README.md
 
-Ansible AWX deployment on a dedicated k3s single-node VM.
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-**[→ Deployment Plan](DEPLOYMENT.md)** · **[→ Status](../../STATUS.md)**
+> **Ansible AWX deployment on k3s**
-
+>
-## Access
+> **Read it here:** https://community.turnsys.com/t/302
-
+>
-| | |
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-|---|---|
+> Do not update content here — edit the Discourse wiki topic instead.*
-| URL | `http://tsys-awx.knel.net` (LAN) or `http://100.91.39.53` (Tailscale) |
-| Port | 80 (LoadBalancer via k3s ServiceLB) |
-| User | `admin` |
-| Password | `REDACTED_PASSWORD` (fleet standard) |
-
-## Architecture
-
-```
-tsys-awx (VMID 600 on pfv-tsys6)
-├── k3s v1.36.2+k3s1 (single-node, no Traefik)
-├── AWX Operator 2.19.1 (manages the AWX lifecycle)
-└── AWX instance "tsys-awx"
-    ├── AWX web (UI/API)
-    ├── AWX task (Celery workers)
-    ├── PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
-    ├── Redis (ephemeral)
-    └── ServiceLB (binds port 80 to host IP)
-```
-
-## VM Resources
-
-| Resource | Value |
-|----------|-------|
-| Host | pfv-tsys6 (2× Xeon E5530, 126 GB RAM) |
-| vCPU | 4 (2 sockets × 2 cores) |
-| RAM | 12 GB |
-| Disk | 60 GB (D5 on tsys4 NFS) |
-| OS | Debian 13 trixie |
-| Network | 192.168.3.200 (LAN), 100.91.39.53 (Tailscale) |
-
-## Files
-
-| File | Purpose |
-|------|---------|
-| [`install-k3s.sh`](install-k3s.sh) | Install k3s single-node (no Traefik) |
-| [`deploy-awx.sh`](deploy-awx.sh) | Deploy AWX Operator 2.19.1 + AWX instance |
-| [`verify-awx.sh`](verify-awx.sh) | Verify deployment status + access |
-| [`namespace.yaml`](namespace.yaml) | Kubernetes namespace manifest |
-| [`awx-instance.yaml`](awx-instance.yaml) | AWX Custom Resource (CR) manifest |
-
-## Deployment (from scratch)
-
-```bash
-# 1. Push scripts to VM
-for f in awx/install-k3s.sh awx/deploy-awx.sh awx/verify-awx.sh \
-         awx/namespace.yaml awx/awx-instance.yaml; do
-    VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-        "sudo tee /tmp/awx/$(basename $f) > /dev/null" < "$f"
-done
-
-# 2. Install k3s
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "sudo bash /tmp/awx/install-k3s.sh"
-
-# 3. Deploy AWX
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "sudo bash /tmp/awx/deploy-awx.sh"
-
-# 4. Verify
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "bash /tmp/awx/verify-awx.sh"
-```
-
-## Operating
-
-```bash
-# kubectl access
-sudo k3s kubectl -n awx get pods
-sudo k3s kubectl -n awx logs -f deployment/awx-operator-controller-manager
-
-# Restart AWX instance
-sudo k3s kubectl -n awx delete awx tsys-awx  # operator will recreate
-
-# Get admin password
-sudo k3s kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d
-```
-
-## Known Issues
-
-1. **kube-rbac-proxy image**: The AWX Operator 2.19.1 references
-   `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` which was removed from GCR.
-   The deploy script patches this to `quay.io/brancz/kube-rbac-proxy:v0.15.0`.
-2. **Image pulls are slow**: First deployment pulls ~2 GB of container images.
-   Postgres image alone is ~400 MB and takes several minutes.
-3. **Disk space**: AWX + k3s uses ~10 GB of the 60 GB disk. PostgreSQL data
-   grows over time — monitor `df -h /`.

bootstrap-all.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/bash
+# bootstrap-all.sh — push agent SSH key + passwordless sudo to remaining systems.
+#
+# SSH is the ONLY approved access channel (see AGENTS.md "Access-channel
+# policy: SSH only"). This script reaches systems that still allow password
+# auth over sshd. Systems that reject password auth (publickey-only) cannot
+# be reached this way — see the CONSOLE-ONLY section printed at the end.
+#
+# Two escalation methods:
+#   sudo → Ubuntu-style systems (no root pw; localuser has sudo)
+#   su   → Debian-style systems (root has a password)
+#
+# Passes AGENT_USER so agent-bootstrap.sh targets the correct unprivileged
+# user. You enter passwords interactively. Idempotent: safe to re-run.
+#
+# Verified state (access-matrix.sh, 2026-08-10): 68/70 non-excluded systems
+# at intended access state. Only tsys-siem remains below.
+set -u
+cd "$(dirname "$0")" || exit 1
+
+SCRIPT=agent-bootstrap.sh
+SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
+
+run_with_sudo() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (sudo, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "sudo AGENT_USER=${agent_user} bash /tmp/$SCRIPT" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+run_with_su() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (su, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "su -c 'AGENT_USER=${agent_user} bash /tmp/$SCRIPT'" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+# All password-auth-reachable systems have been bootstrapped.
+# Verified state (access-matrix.sh, 2026-08-10): 69/70 non-excluded systems
+# at intended access state. The only remaining NOKEY (stlp-3dscanner) is
+# deferred to [#417] and requires a rename + bring-online first — out of
+# scope here. Run `access-matrix.sh` to re-verify at any time.
+echo "All password-auth-reachable systems are bootstrapped."
+echo "Remaining gap: stlp-3dscanner (deferred to [#417])."
+echo "Run access-matrix.sh to re-verify."
+echo "Deferred (separate ticket):"
+echo "  stlp-3dscanner — rename + bring online first [#417]"
+echo "========================================"
+echo "By design (leave alone):"
+echo "  sectestbed-sandbox — 2FA enforced"
+echo "========================================"
+echo "Excluded by policy (no SSH access):"
+echo "  pfv-bms (API), tsys-cloudron (prod revenue),"
+echo "  tsys-ucs-01/02 (API-managed), tsys-umbrel (treasury)"
+echo "========================================"

console/README.md

@@ -1,130 +0,0 @@
-# Console Management (ser2net + conman)
-
-Network-accessible serial console management for all production network
-switches and routers, running on **pfv-tsys4** (storage server).
-
-## Architecture
-
-```
-USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
-```
-
-ser2net owns the physical serial devices and exposes them on TCP ports
-using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
-only** (`100.70.77.93:200X`). conman connects to those TCP ports via
-telnet for session logging, output capture, and multi-user console
-sharing.
-
-**Why telnet(rfc2217)?** The serial devices send `
-␍` (LF+CR) line
-endings instead of standard `
-`. Raw TCP transport caused conman's
-telnet NVT to strip bare CR characters, producing stair-stepped output.
-With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
-translation is handled correctly by the telnet layer.
-
-**conman and ser2net do NOT share ports** — only one process can open a
-serial device at a time. ser2net owns the physical device; conman connects
-over TCP.
-
-## The USB Enumeration Problem (SOLVED)
-
-The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
-unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
-enumeration order, which shifts on every boot. This made the old
-`/root/conmap` + manual `screen` workflow break after every reboot.
-
-**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
-topology), which is stable across reboots regardless of enumeration order.
-Each adapter gets a named symlink in `/dev/consoles/` that never changes.
-
-The udev rules are generated from `mapping.txt`, which maps each adapter's
-ID_PATH to a console name and TCP port. To re-map after physically moving
-an adapter, update `mapping.txt` and re-run `setup.sh`.
-
-**Fallback:** if udev trigger doesn't create symlinks for already-discovered
-devices (common on first run), `setup.sh` creates them manually by matching
-ID_PATH. On subsequent boots, udev creates them automatically.
-
-## Port Assignments
-
-| TCP Port | Console Name | ID_PATH | Description |
-|----------|-------------|---------|-------------|
-| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
-| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
-| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
-| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
-| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
-| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
-| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
-
-All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
-
-## Scripts
-
-| Script | Purpose |
-|--------|---------|
-| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
-| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
-| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
-| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
-
-## Usage
-
-### Connect to a console
-
-**Primary method — conman client (with logging + multiplexing):**
-
-```bash
-# From any Tailscale-connected workstation:
-conman -d pfv-tsys4:7890 -f pfv-core-sw01     # connect to console
-conman -d pfv-tsys4:7890 -q                    # list all consoles
-```
-
-Escape sequence: `&.` to disconnect, `&?` for help.
-
-**Direct telnet (emergency only — conflicts with conman):**
-
-```bash
-# Direct telnet to ser2net works ONLY when conmand is stopped, because
-# conmand maintains persistent connections to all 7 TCP ports. Use:
-ssh pfv-tsys4 'systemctl stop conmand'
-telnet pfv-tsys4 2001                          # pfv-core-sw01
-ssh pfv-tsys4 'systemctl start conmand'        # restart when done
-```
-
-**Do NOT use telnet while conmand is running** — conmand will reconnect
-and kick your telnet session immediately ("Connection closed by foreign host").
-The correct workflow is conman client → conmand → ser2net → device.
-
-### Re-deploy after changing mapping.txt
-
-```bash
-PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
-```
-
-### Find the ID_PATH for a new adapter
-
-```bash
-PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
-```
-
-Then match the new adapter's ID_PATH to its physical location and add a line
-to `mapping.txt`.
-
-## Files on pfv-tsys4
-
-| File | Purpose |
-|------|---------|
-| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
-| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
-| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
-| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
-| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
-| `/root/console/setup.sh` | Setup script (re-runnable) |
-| `/root/console/generate-config.sh` | Config generator |
-
-## Old workflow (replaced)
-
-The old `/root/conmap` file and manual `screen` sessions are no longer
-needed. The new setup is fully automated and survives reboots.

dcinfra/console/README.md

@@ -0,0 +1,10 @@
+# dcinfra/console/README.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Serial console management (ser2net + conman)**
+>
+> **Read it here:** https://community.turnsys.com/t/301
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*

dcinfra/console/generate-config.sh

@@ -6,7 +6,7 @@
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
-#      to a named symlink like /dev/consoles/pfv-core-sw01
+#      to a named symlink like /dev/consoles/pfv-r5-core-01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing
@@ -121,7 +121,7 @@ for entry in "${ENTRIES[@]}"; do
   {
     echo ""
     echo "# $name (TCP $tcp_port): $comment"
-    echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"console/$name\""
+    echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"consoles/$name\""
   } >> "$UDEV_RULES"
 done
 

dcinfra/console/mapping.txt

@@ -17,13 +17,13 @@
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
-2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
+2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
-2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
+2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
-2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
+2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
-2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
+2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
-2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
+2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
-2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
+# 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare

dcinfra/console/query-remote.sh

@@ -5,7 +5,7 @@
 #
 # Usage:
 #   bash console/query-remote.sh                    # list consoles
-#   bash console/query-remote.sh pfv-core-sw01      # connect to a console
+#   bash console/query-remote.sh pfv-r5-core-01      # connect to a console
 #
 set -euo pipefail
 
@@ -52,7 +52,7 @@ if [ -z "$CONSOLE" ]; then
   conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
   echo ""
   echo "To connect: bash $0 <console-name>"
-  echo "  e.g: bash $0 pfv-core-sw01"
+  echo "  e.g: bash $0 pfv-r5-core-01"
 else
   echo ""
   echo "--- Connecting to: $CONSOLE ---"

dcinfra/console/setup.sh

@@ -202,7 +202,7 @@ echo "    ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
 echo "    conman connects via telnet for logging + multiplexing"
 echo ""
 echo "  Connect from any Tailscale workstation:"
-echo "    conman -d pfv-tsys4:7890 -f pfv-core-sw01"
+echo "    conman -d pfv-tsys4:7890 -f pfv-r5-core-01"
 echo "    conman -d pfv-tsys4:7890 -q              # list consoles"
 echo ""
 echo "  Direct telnet (emergency, conflicts with conman):"

dcinfra/console/validate-conman.sh

@@ -45,7 +45,7 @@ echo "--- 3. Trigger log capture: connect to each console briefly ---"
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
-for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
+for name in pfv-r5-core-01 pfv-r3-tor-mgmt-01 pfv-r3-tor-stor-01 pfv-rrinfra-rtr pfv-r2-tor-01 pfv-r6-mgmt-01; do
   logfile="$LOGDIR/${name}.log"
   if [ -f "$logfile" ]; then
     SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)

dcinfra/powerman/README.md

@@ -0,0 +1,10 @@
+# dcinfra/powerman/README.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Cyclades PM10i PDU management via powerman**
+>
+> **Read it here:** https://community.turnsys.com/t/301
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*