docs: session cleanup — coverage matrix, questions v2, handoff pointer [#343 ]

- OAM coverage matrix published on Discourse #309 - questions-v2.md: Q1-Q6 resolved, Q7-Q9 added - aug11.md: replaced with pointer to Redmine #314 + Discourse #309 - Redmine cleanup: 16 tracker fixes, cross-refs, stale ticket reviews - UNPoller InfluxDB noise silenced (config mount fix + disable=true) 💘 Generated with Crush Assisted-by: Crush:glm-5.2
docs(agents): add no-direct-DB-access rule — API-first always
2026-08-12 11:44:58 -05:00 · 2026-08-11 22:50:27 -05:00 · 2026-08-11 21:41:47 -05:00 · 2026-08-11 20:34:38 -05:00 · 2026-08-11 20:32:32 -05:00 · 2026-08-11 20:20:45 -05:00
293 changed files with 25442 additions and 26559 deletions
@@ -0,0 +1,3 @@
+*
+!crush.json
+!.gitignore
@@ -0,0 +1,11 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "^(bash|edit|write|multiedit|lsp_replace_symbol|lsp_rename)$",
+        "command": "./hooks/ticket-gate.sh",
+        "timeout": 5
+      }
+    ]
+  }
+}
@@ -0,0 +1,7 @@
+# PFVCluster environment variables
+# Copy to .env and fill in values for local development/testing.
+
+# Pi-hole (netinfra/pihole/docker-compose.yml)
+PIHOLE_WEB_PASSWORD=changeme
+
+# Shellcheck wrapper (tests/shellcheck.sh) — no config needed, uses Docker.
@@ -1,5 +1,9 @@
-# Crush internal state
-.crush/
+# Crush internal state (track crush.json config, ignore session data)
+.crush/crush.db
+.crush/crush.db-*
+.crush/logs/
+.crush/memory/
+.crush/active-ticket

 # OS/editor
 .DS_Store
@@ -33,3 +37,10 @@ __pycache__/
 # Sensitive exports / runtime data
 dns-cluster-setup/.export/
 returned-logs/
+
+# Kubernetes secrets (kubeconfig contains embedded client certs)
+k8s/kubeconfig.yaml
+k8s/*.token
+
+# Pi-hole web UI password (real value only in on-box .env, never committed)
+netinfra/pihole/.env
@@ -0,0 +1,17 @@
+# ShellCheck configuration for PFVCluster
+# (used when running `shellcheck` directly; tests/shellcheck.sh applies the
+#  same disables via -e for consistent results under Docker)
+#
+# These checks are DISABLED because they flag intentional conventions of this
+# codebase, not bugs:
+#
+# SC1090 / SC1091 — cannot follow dynamically-computed `source` paths. The KNEL
+#   framework (vendor/) and test harness source helpers via computed include
+#   dirs, which shellcheck cannot resolve statically.
+# SC2029 — ssh orchestration (tests/remote.sh and perf/k8s/dns scripts)
+#   deliberately builds and expands the remote command on the CLIENT side before
+#   sending it. That is the whole point of the single-chokepoint remote pattern.
+disable=SC1090,SC1091,SC2029
+
+# Treat external-sourced files as bash (matches #!/usr/bin/env bash framework).
+external-sources=true
@@ -1,63 +1,587 @@
 # Agent Guidelines

-This repo combines two formerly-separate projects:
- **Server provisioning** (formerly KNELServerBuild): `provisioning/`, `tests/`,
-  `vendor/`, `dns-cluster-setup/`
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): `perf/`, `netinfra/`,
-  `switches/`, `returned-logs/`
+## Agent Authority (NON-NEGOTIABLE)
+
+**No work is permissible on any system without an approved Redmine ticket.
+There are no exceptions to this rule.**
+
+### Scope of authority
+
+1. **Tickets govern all work.** The agent performs ONLY the work described in
+   the approved Redmine ticket. Anything outside that scope — no matter how
+   small, helpful, or "obvious" — is prohibited.
+
+2. **No autonomous system changes.** The agent does not modify, configure,
+   create, delete, or grant anything on a production system unless it is
+   explicitly directed by an approved ticket. "Production system" means
+   every system in the fleet — there is no "test" exception unless the ticket
+   says so.
+
+3. **Propose, never implement.** If the agent discovers additional work that
+   should be done — a bug, a misconfiguration, a missing dependency, an
+   enhancement — it does NOT implement it. Instead, it creates a Redmine
+   sub-ticket (status Feedback) describing the finding and surfaces it to
+   the user for approval.
+
+4. **Security and access changes require extra scrutiny.** Changes to sudoers,
+   SSH keys, user accounts, firewall rules, authentication policy, file
+   permissions, or any privilege-related configuration are treated as
+   policy decisions, not implementation details. The agent may suggest
+   them but NEVER implements them without explicit user direction in the
+   ticket or a sub-ticket the user has approved.
+
+5. **The user makes policy. The agent implements policy.** The agent does
+   not decide who gets sudo, what keys go where, what services run, or what
+   the access model is. The agent executes the user's decisions, exactly
+   as specified.
+
+6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
+   require something not explicitly authorized, or if the agent is unsure
+   whether an action is in scope — STOP and ask the user via the ticket
+   or directly. Asking is always acceptable. Overstepping is never
+   acceptable.
+
+7. **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user. This applies to ALL tickets,
+   no exceptions.
+8. **User acceptance testing is MANDATORY before declaring work done.**
+   The agent performs implementation and technical validation (services
+   running, configs correct, APIs responding). The user performs UAT —
+   visually confirming dashboards render data, alerts deliver, tools are
+   usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
+   closing, and MUST NOT move to the next ticket until the user explicitly
+   accepts the work. "Technically wired but blank dashboard" is NOT done.
+9. **NEVER access a database directly if an API exists.** APIs are the
+   stable contract; databases change schemas without warning. If a tool
+   has an API, use it — exhaust all API endpoints, check the docs, try
+   alternative methods. Only fall back to direct DB access as a last
+   resort AND with explicit user approval for that specific instance.
+   Soon all DB access will route through a proxy under zero trust;
+   building API-first habits now ensures that transition is clean.
+
+### Access-channel policy: SSH only (NON-NEGOTIABLE)
+
+**The qemu guest-agent is NEVER an access, execution, or key-delivery
+channel.** SSH (`sshd`) is the only approved remote access path. Every
+command must flow through sshd so it is captured by the standard
+auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
+environment — there is no back-door exception, ever.
+
+- **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
+  over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
+  mode). This includes using guest-agent to *deliver* an SSH key, even if
+  the resulting SSH login is itself audited — the delivery bypassed audit.
+- **Allowed:** installing or checking qemu-guest-agent for its intended
+  purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
+  `agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
+  never execution.
+- **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
+  any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
+  no guest-exec mode.
+- **If a system is locked out** (no SSH key, no guest-exec path): surface
+  it to the user. Do NOT improvise an alternate back-channel. The user
+  authorizes the unblock method (console login, credential, etc.).
+
+### Ticket-closing policy (NON-NEGOTIABLE)
+
+**NEVER close a Redmine ticket without explicit user permission.** You
+may SUGGEST a close when the result is clearly scoped and delivered. If
+it's ambiguous whether the work is truly complete, don't suggest a close
+— leave that decision to the user. This applies to ALL tickets, no
+exceptions.
+
+### What this means in practice
+
+- Discovered a typo in a config during approved work? **Finish the approved
+  work. Create a sub-ticket for the typo. Do not fix it inline.**
+- Think a system should also have localuser sudo configured? **Do not add
+  it. Propose it in a sub-ticket.**
+- Need to install a package the ticket didn't mention? **Ask first.**
+- Found a security issue? **Create a ticket immediately with full details.
+  Do not remediate without approval.**
+
+This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
+traceable to an approved ticket. There is no "I thought it would help."
+
+## Quick Start
+
+**You are an AI agent working on this project. Your first actions, in order:**
+
+> **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
+> check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
+> the scope-alignment gate. You MUST NOT begin any task work until the user
+> (a) confirms your rule summary AND (b) names the ticket to work on. This runs
+> **every session, automatically** — the user should never have to ask for it.
+
+1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
+2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
+3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
+4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
+   back to the user before doing any work: Agent Authority (ticket-governed,
+   propose-never-implement), Access-Channel SSH-only policy, Remote access
+   (remote.sh chokepoint mandatory, DNS names only — never IP literals),
+   Questions policy (no harness question-tools), Documentation policy
+   (Discourse is SoR), Redmine tracking, Git policy (always commit+push,
+   shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
+   routing chokepoints (`tests/remote.sh`), the mechanically enforced
+   `check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
+   and wait for the user to confirm. This checkpoint guarantees every
+   session starts aligned.
+5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
+6. **Check current state:** `git log --oneline -10`.
+7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
+8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
+   priority list (or the Redmine queue if no handoff exists). Do NOT scan the
+   full ticket queue and pick work on your own — **the user directs what gets
+   worked on, always.** Note any new or urgent items from the Redmine check,
+   then ask which ticket to work on tonight. **STOP and wait.** Do not set
+   `.crush/active-ticket`, do not read systems, do not run diagnostics, until
+   the user names the target. This gate prevents the agent from burning
+   context on work the user didn't ask for.
+
+## Enforcement Model
+
+Git hooks (`scripts/pre-commit`, `scripts/pre-push`) enforce the rules defined in
+`scripts/check-rules.sh`. The rules engine checks: shellcheck (zero warnings
+including info-level), Docker image pinning (no `:latest`), container naming,
+required files, Discourse pointer headers, and more. Run `bash scripts/check-rules.sh`
+for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
+(emergencies only).
+
+## Task Tracking
+
+- **Redmine is the system of record for all work.**
+- **NEVER close a ticket without explicit user permission.** Suggest a
+  close when clearly scoped/delivered; if ambiguous, don't suggest.
+- **Ticket-first enforcement (mechanically enforced).** Before starting
+  any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
+  The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
+  until this file exists. If no ticket exists, CREATE ONE FIRST via
+  redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
+- **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
+  session. The pre-commit hook blocks commits while any task remains unchecked.
+- Clear WORKING.md before responding to the user.
+
+## Rolling HUD (session-scoped, NOT persisted)
+
+The rolling HUD is a **live status display** the agent maintains throughout
+the session to help the human follow along. It is NOT a system of record —
+Redmine, Discourse, and git are the durable systems. The HUD exists purely
+for the human's situational awareness during the session.
+
+- **Format:** a compact block shown at the end of each significant response
+  (after completing a step, hitting a blocker, or pivoting). Example:
+
+  ```
+  ┌─ SESSION HUD ────────────────────────────────────────
+  │ Active: #343 (Monitoring coverage matrix)
+  │ Done:   ✓ scope-alignment gate added to both AGENTS.md
+  │         ✓ committed + pushed (6a2550b)
+  │ Now:    drafting coverage matrix on Discourse
+  │ Next:   → #341 TEMPer USB (deploy on pfv-tsys1)
+  │         → #338 LibreNMS alerts (BLOCKED: pushover.env)
+  ├─ USER ACTION ITEMS ──────────────────────────────────
+  │ • Populate ~/.creds/pushover.env (unblocks #338, #428)
+  │ • Populate ~/.creds/prometheus.env + grafana.env (#430)
+  └──────────────────────────────────────────────────────
+  ```
+
+- **Placement:** may be written to `.crush/hud.md` on disk to keep context
+  window smaller (re-read and update rather than hold in memory). Never
+  committed to git. Wiped at session end.
+- **Sections:**
+  - **Active:** current ticket number + one-line description
+  - **Done:** ✓ items completed this session (append as work progresses)
+  - **Now:** what the agent is actively doing
+  - **Next:** the queued items (per handoff priority or user direction)
+  - **User action items:** things ONLY the user can do (populate creds,
+    physical work, manual deploys) with the tickets they unblock
+- **When to show it:** after each logical unit of work, at blockers, and
+    when pivoting between tickets. Not every trivial response — use judgment.
+- **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
+    work completes, update the durable systems (ticket notes, Discourse wiki,
+    commits). The HUD just tracks the live narrative for the human.
+
+## Working Style
+
+- **Stop over-thinking.** Get to code and output faster. Explore with code;
+  gather ground truth. Do not burn tokens reasoning about things a quick command
+  answers.
+- **Farm work out to deterministic tooling:** linters, LSPs, formatters, test
+  runners. If an LSP is wired up, use it; otherwise pull a Docker image and lint
+  inside it.
+- **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
+  Never read 10+ files sequentially; batch into agent calls.
+- **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
+  system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
+  (status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
+  If a command hits the timeout, STOP and investigate root cause — never
+  blindly retry. A hung command is a failed command. Detect failure fast,
+  diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
+  This applies to ALL tools — bash, docker, CLIs, sub-agents.
+
+## Questions (NON-NEGOTIABLE)
+
+**NEVER use a harness "question"/"ask user" tool** (structured prompts,
+modal forms, tabbed questions). Banned across every project, every harness.
+They are not portable, not version-controlled, and bypass the git record.
+**All questions go in the current `questions-v(N).md` file** — write the
+question; the human edits the answer inline in the same file. **Version up
+the filename each time answers land** (v1 → v2 → v3...): create
+`questions-v2.md` with resolved Q&A marked, new questions appended. This
+preserves the history of each Q&A round. Synthesize resolved Q&A into
+Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
+§10.
+
+## Documentation policy (IMPORTANT)
+
+**Discourse is the canonical source of truth for all knowledge documentation.**
+Knowledge docs (architecture, runbooks, references, audits, policies) have been
+migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops)
+as wiki topics in the **VP TechOps** category.
+
+All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now
+**pointers** that link to their corresponding Discourse topic. **Do not update
+documentation content in git** — edit the Discourse wiki topic instead. Git
+edit history no longer serves as the documentation changelog; Discourse
+preserves wiki edit history automatically.
+
+Code (scripts, configs, playbooks) still lives in git as the source of truth
+for executables. Only *documentation* moved to Discourse.
+
+## Top-level files
+
+All `.md` files now point to Discourse. The key pointers:
+
+| File | Points to | Discourse topic |
+|------|-----------|-----------------|
+| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) |
+| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) |
+| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) |
+
+Work tracking stays in [Redmine](https://projects.knownelement.com).

 ## Repository Layout

- **Vendored framework**: `KNELShellFramework` lives at
-  `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes
-  are under `Framework-Includes/` there. Never assume `./Framework-Includes`
-  exists relative to the repo root.
- **Self-locating scripts**: All provisioning scripts derive their own location
-  via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
-  depend on the current working directory. Run from anywhere.
- **Local config files are the source of truth**: Configs in
-  `provisioning/ConfigFiles/` are read with `cat`/`cp`. Do NOT re-introduce
-  `curl ${DL_ROOT}/...` downloads — that CDN is deprecated.
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
-  extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
-  checkers must skip these.
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
-  Datacenter Manager). SSH keys deployed to root on all hosts.
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
-  Use the wrapper scripts: `tests/remote.sh`, `dns-cluster-setup/remote-dns.sh`,
-  or the `deploy-check.sh` / `deploy-tuning.sh` patterns.
+```
+dcinfra/            Data-center infra: PDU (powerman), serial console (console), UPS (ups)
+netinfra/           DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup);
+                    switch captures (switches); DHCP config (dhcp)
+k8s/                k3s cluster setup scripts (HA control plane over Tailscale) + docs/
+proxmox/            Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
+awx/                Ansible AWX deployment (k3s + AWX Operator)
+tests/              Test suite + VM validation harness + remote.sh SSH chokepoint
+scripts/            Framework: git hooks, rule engine (check-rules.sh), shared lib
+docs/               Server-build docs, docmap index, and archive
+archive/            Historical/superseded code (provisioning -> replaced by KNELIAC project)
+vendor/             Vendored KNELShellFramework
+```

-## Git Commit Requirements
+- **Server provisioning moved to KNELIAC**: The
+  [`archive/provisioning/`](archive/provisioning/) tree is historical. Active
+  server provisioning lives in the **KNELIAC** project at
+  `/home/reachableceo/projects/KNELIAC`.
+- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
+  extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
+- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
+  through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
+  (Proxmox hosts + all VMs) or
+  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh)
+  (DNS infra hosts: netinfra-01/02, tsrouter, netboot). NEVER call
+  `ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
+  rejects it. There are no exceptions.
+- **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
+  (neither LAN nor Tailscale IPs) in any command, script, or config.
+  ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
+  `VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
+  DNS name does not resolve, fix it in DNS (Technitium) or consult the
+  [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  Do NOT fall back to IP literals. This rule eliminates the per-session
+  discovery tax of finding the right IP for each host.
+- **How to access a production VM:**
+  1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
+  3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
+     and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
+     NOT execution) to find the Tailscale DNS name, then access via that.

-1. **Commit atomically**: each logical change its own commit.
-2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`,
-   `docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`.
-3. **Verbose messages**: title (50 chars max), blank line, body explaining
-   WHAT and WHY, footer with attribution.
+## Git Policy

-## Autonomous Git Workflow
+1. **ALWAYS commit + push. Never wait. Ever.** After each logical unit of
+   work, immediately stage, commit (conventional format), and push to
+   `origin/main`. Do NOT pause for review. Do NOT ask permission. Do NOT
+   stop to "let the user read it first." The user reviews rendered markdown
+   on Gitea AFTER it is pushed. **This rule overrides any default
+   conservative commit-and-hold behavior** — if a default instruction says
+   "never commit unless asked," that instruction is superseded here.
+2. **Atomic commits.** Each commit coherent on its own.
+3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
+   `docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
+4. **All shell scripts MUST pass `shellcheck` before commit.** No exceptions.
+   Run it via the wrapper:
+   ```bash
+   bash tests/shellcheck.sh                 # whole repo
+   bash tests/shellcheck.sh ups/*.sh        # specific files
+   ```
+   This invokes `koalaman/shellcheck:stable` through Docker (no native binary
+   needed). Fix every reported finding — including `info`-level — or add a
+   targeted `# shellcheck disable=SCxxxx  # <reason>` directive with a
+   justification. A script that emits any diagnostic is a protocol violation.
+   Non-bash scripts (PHP with `.sh` shebang `#!/usr/bin/php`, etc.) are exempt.

-Agents are authorized to commit AND push autonomously. After each logical unit
-of work: stage, commit, push to `origin/main`. Group changes so each commit is
-coherent on its own.
+## Redmine Tracking Policy

-## Key scripts
+**Redmine is the system of record for all work.** Do not track status,
+checklists, or TODOs in repo files. Use Redmine tickets instead.
+
+- **URL:** https://projects.knownelement.com
+- **Version:** Potential to Kinetic Ready (due 2026-09-30)
+- **Project:** Known Element Enterprises - Technology & Facility Services (id 55)
+
+### Rules
+1. **Every piece of work** (feature, fix, deployment, config change) gets a
+   Redmine ticket. If one doesn't exist, create it.
+2. **Reference tickets in docs and commits** using `[#NNN]` notation.
+   Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
+3. **When work completes**, update the ticket: set done ratio to 100%,
+   add a note describing what was done and where the code lives.
+   **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user.
+4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
+5. **Link code to tickets** — ticket descriptions and notes should reference
+   the relevant file paths in this repo (e.g., `dcinfra/ups/`).
+
+### CLI access (read + write)
+
+Tickets are managed via the `redmine-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the subtask escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/redmine/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+
+# Your queue (project 55):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55
+
+# Show / create / update / close:
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest show 367
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest create -p 55 -s "Subject" -d "desc"
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest update 367 -n "Done: committed in abc123" --done-ratio 100
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest close 367
+```
+
+Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
+Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
+Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
+**Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
+`python-redmine` escape hatch inside the container (see
+`tooling-cli/redmine/AGENTS.md`). Always `show` a ticket before updating it.
+
+Credentials (`REDMINE_URL`/`REDMINE_API_KEY`) live in the centralized store
+at `~/.creds/redmine.env`.
+
+## Documentation Workflow
+
+**Discourse is the source of truth for all knowledge docs.** After any work
+session, an agent MUST:
+
+1. **Update Discourse wiki topics** — if infrastructure facts changed (new
+   VM, IP change, host retired), edit the relevant wiki topic at
+   [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops).
+2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297))
+   if tickets were opened or closed.
+3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'`
+   after any rename or restructure. Fix all references in the same commit.
+4. **If a new top-level directory was created**, update:
+   - `AGENTS.md` → "Repository Layout" code block
+   - `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script)
+   - Create a new Discourse wiki topic for any documentation
+5. **Self-audit before commit.** Code changes must be internally consistent.
+   Documentation changes go to Discourse, not git.
+
+### CLI access (read + write)
+
+Wiki topics are managed via the `discourse-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the raw-API escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/discourse/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# List VP TechOps topics:
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest ls -c vp-techops
+
+# Show a topic / edit a wiki post (find post id via `show`):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest show 297
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest update <post_id> -b "new markdown body"
+```
+
+VP TechOps = category **74**. Key topics: #296 (project overview), #297
+(ops status), #298 (audit log). The API user is trust-level 4 but **not
+admin** — admin-only ops (category creation, setting the wiki flag) will
+403; surface those to the user rather than retrying. **Gotcha:**
+`update`/`delete` take a post **id**, not a post number. Never create a new
+topic for an update to existing knowledge — edit the wiki post in place.
+
+Credentials (`DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`)
+live in the centralized store at `~/.creds/discourse.env`.
+
+## Key Scripts

 | Script | Purpose |
 |--------|---------|
-| `provisioning/SetupNewSystem.sh` | Full server provisioning (packages, hardening, 2FA) |
-| `tests/vm-validation.sh` | End-to-end deploy + validate on sandbox VM |
-| `tests/run-tests.sh` | Project test suite (unit/security/validation) |
-| `dns-cluster-setup/setup.sh` | Technitium DNS cluster replication |
-| `perf/deploy-check.sh` | Deploy read-only data collector to hosts |
-| `perf/deploy-tuning.sh` | Deploy perf tunings to hosts |
-| `perf/validate-fixes.sh` | Validate applied tuning changes |
-| `perf/iperf-full-matrix.sh` | Full iperf throughput suite |
+| [`scripts/check-rules.sh`](scripts/check-rules.sh) | Rule audit engine (shellcheck, image pinning, Discourse pointers, required files) |
+| [`scripts/setup-hooks.sh`](scripts/setup-hooks.sh) | Install git hooks (pre-commit, pre-push) |
+| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
+| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
+| `redmine-cli` container | Redmine CLI (ticket read/write via `docker run`; see `tooling-cli/redmine/`) |
+| `discourse-cli` container | Discourse CLI (wiki topic read/write via `docker run`; see `tooling-cli/discourse/`) |
+| `dns-cli` container | Technitium DNS CLI (zones, list, add, delete, search, flush; see `tooling-cli/dns/`) |
+| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
+| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
+| [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
+| [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
+| [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman |
+| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
+| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
+| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
+| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |

-## Project context
+## Switch Console Access (conman)

-This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
-budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
-(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
-`docs/PROJECT.md` for the comprehensive fleet report and `docs/TODO.md` for
-pending hardware work.
+Switch configs are pulled via serial console through a conman + ser2net
+stack on pfv-tsys4.
+
+- **conmand server:** pfv-tsys4 (port 7890)
+- **ser2net:** pfv-tsys4, TCP ports 2001-2006
+  (do NOT connect to ser2net directly — it conflicts with conman's
+  persistent sessions; always use the conman client)
+- **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
+  — drives console sessions read-only via PTY. No expect/tcl required.
+- **Command files:** [`netinfra/switches/`](netinfra/switches/) — `.cmds` files
+  with switch-specific show commands
+- **Query available consoles:** `conman -d 100.70.77.93 -q`
+
+```bash
+# Pull a switch config (example):
+CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+| Console name | TCP port | Device |
+|--------------|----------|--------|
+| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
+| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
+| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
+| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
+| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
+| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
+
+## Mandatory: Document ALL Infrastructure Changes
+
+**This is non-negotiable. Every infrastructure change (VM config, disk
+cache, network setting, service config, storage migration) MUST be
+documented BEFORE moving to the next task step — not "later" or "at the
+end."**
+
+Required for EVERY infrastructure change:
+
+1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
+2. **Discourse audit log** — reply to topic
+   [#298](https://community.turnsys.com/t/298) with a dated entry (what
+   changed, why, where).
+3. **Discourse relevant wiki topic** — update the architecture/reference
+   topic if the change affects documented infrastructure facts (storage
+   #300, network #299, k8s #305, etc.).
+4. **Git commit** — if code/config changed in the repo, commit + push
+   immediately per the Git Policy above.
+
+## Tooling
+
+- `gh`, `docker`, `jq` available on the workstation.
+- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
+  `koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
+- For raw API calls not covered by the Redmine/Discourse CLIs, use
+  `python3` inside the CLI Docker containers (escape hatch pattern in
+  `tooling-cli/{discourse,redmine}/AGENTS.md`).
+- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
+  the python-in-Docker escape hatch for HTTP writes.
+
+## Key Commands Quick Reference
+
+```bash
+# Tests + validation:
+bash tests/run-tests.sh                        # test suite
+bash tests/vm-validation.sh                    # VM validation
+bash tests/shellcheck.sh                       # shellcheck whole repo
+bash tests/shellcheck.sh path/to/*.sh          # shellcheck specific files
+
+# Redmine + Discourse sanity checks (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# Proxmox host access (DNS names only — never IPs):
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
+
+# Production VM access (DNS names only — never IPs):
+VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
+
+# DNS infra access:
+bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
+
+# Switch console (read-only config pull — DNS names only):
+CONMAN_SERVER=pfv-tsys4:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+## Credential Management
+
+- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
+- User goal: migrate all keys to Hashicorp Vault
+  (`vault.knownelement.com`). No vault token present yet
+  (`~/.vault-token` missing). Track as high-priority TODO.
+
+## Key Docs
+
+→ **All documentation lives on Discourse:**
+  [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops)
+
+→ All `.md` files in this repo are pointers to Discourse topics.
+
+→ **Complete Linux System Inventory:**
+  [Topic #307](https://community.turnsys.com/t/307) — every Linux system
+  (hosts + VMs + physical), with Tailscale IPs, DNS names, SSH access
+  status, and tuned profiles. Reference this for monitoring coverage,
+  access management, and hostname consistency.
+
+## Project Context
+
+Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
+Production lives on a Cloudron VPS in Reston VA. See the
+[Operations Status topic](https://community.turnsys.com/t/297) for the ticket
+index and infrastructure summary.
+All work is tracked in [Redmine](https://projects.knownelement.com)
+(version: Potential to Kinetic Ready, due 2026-09-30).
@@ -0,0 +1,50 @@
+# Makefile — convenience dispatch to scripts/.
+#
+# Not required. The scripts in scripts/ are the real entry points and work
+# standalone. This file just gives you short verbs if you're at a terminal.
+#
+# In Mode 2 (Hermes/OWUI/MCP), agents call the scripts directly or via API —
+# they don't need this file.
+
+# Project-specific overrides for check-rules.sh
+export PROJECT_DOC_EXEMPT ?= AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v1.md BASELINE-PROMPT.md PATTERNS.md
+export PROJECT_DISCOURSE_HOST ?= community.turnsys.com
+
+.PHONY: setup validate fast lint test garden up down status clean help
+
+help: ## Show available targets
+	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
+
+setup: ## Install git hooks
+	@bash scripts/setup-hooks.sh
+
+validate: ## Full rule audit (includes tests)
+	@bash scripts/check-rules.sh
+
+fast: ## Fast rule audit (pre-commit equivalent)
+	@bash scripts/check-rules.sh --fast
+
+lint: ## Lint shell scripts (shellcheck via docker)
+	@docker run --rm -v "$$(pwd):/mnt" koalaman/shellcheck:stable \
+		$$(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print | sed 's|^\./|/mnt/|') || true
+
+test: ## Run the test suite (override per project)
+	@bash scripts/test.sh
+
+garden: ## Doc-sprawl / Discourse-migration report
+	@bash scripts/garden.sh
+
+up: ## Bring up the docker-compose stack
+	@bash scripts/up.sh
+
+down: ## Bring down the docker-compose stack
+	@bash scripts/down.sh
+
+status: ## Show repo status snapshot
+	@echo "== branch =="; git branch --show-current 2>/dev/null || echo "(no branch)"
+	@echo "== last commit =="; git log --oneline -1 2>/dev/null || true
+	@echo "== working tree =="; git status --short 2>/dev/null || echo "(not a git repo)"
+	@echo "== STATUS.md head =="; sed -n '1,12p' STATUS.md 2>/dev/null || echo "(no STATUS.md)"
+
+clean: ## Remove build/test artifacts (override per project)
+	@echo "make clean: nothing to clean — override this in your project's Makefile."
@@ -1,70 +1,10 @@
-# PFVCluster
+# README.md

-Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
-Combines server provisioning, Proxmox cluster operations, and DNS infrastructure.
-
-## Directory Structure
-
-```
-provisioning/        Server provisioning (SetupNewSystem.sh, security hardening,
-                     2FA, NTP/DNS config, SNMP, Dell OMSA)
-tests/               Test suite + VM validation harness
-dns-cluster-setup/   Technitium DNS cluster replication scripts
-perf/                Proxmox performance tuning, fleet audit, iperf, switch diagnostics
-netinfra/            pfv-netinfra-01/02 DNS/NTP setup + audit scripts
-switches/            Switch configuration captures
-docs/                All documentation (PROJECT.md, SECURITY.md, tailscale.md, etc.)
-vendor/              Vendored KNELShellFramework
-```
-
-## Quick Start
-
-### Provision a new server
-```bash
-sudo bash provisioning/SetupNewSystem.sh
-```
-Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh),
-configures NTP/DNS/SNMP/syslog/postfix.
-
-### Validate provisioning on the sandbox VM
-```bash
-VM_ID=6000 ./tests/vm-validation.sh all
-```
-Snapshots, deploys, runs the test suite, auto-rolls back on failure.
-
-### Run the test suite
-```bash
-./tests/run-tests.sh all
-```
-
-### Deploy DNS cluster setup
-```bash
-cd dns-cluster-setup/
-./setup.sh all
-```
-
-### Deploy perf tunings to hosts
-```bash
-cd perf/
-./deploy-check.sh    # read-only data collection
-./deploy-tuning.sh   # apply sysctl/tuned/NFS tunings
-```
-
-## Key Documentation
-
-| Doc | Contents |
-|-----|----------|
-| `docs/PROJECT.md` | Comprehensive fleet report (7 hosts, VM inventory, storage) |
-| `docs/SECURITY.md` | Security architecture and hardening details |
-| `docs/tailscale.md` | Tailscale vs managed DNS analysis |
-| `docs/DEPLOYMENT.md` | Deployment procedures |
-| `docs/TODO.md` | Pending hardware work (tsys2/4/5) |
-| `dns-cluster-setup/README.md` | DNS cluster setup guide |
-
-## Architecture
-
- **Proxmox hosts**: 7 standalone PVE installs managed via PDM
- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs)
- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
- **Backups**: Proxmox Backup Server (PBS)
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Project overview, architecture, quick start**
+>
+> **Read it here:** https://community.turnsys.com/t/296
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*
@@ -0,0 +1,10 @@
+# STATUS.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Ticket index + infrastructure summary**
+>
+> **Read it here:** https://community.turnsys.com/t/297
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*
@@ -0,0 +1,11 @@
+# WORKING.md — Active Session Tracker
+
+Agent work only. The human decides when it's done.
+A commit is blocked while any task below remains unchecked.
+
+## Current Tasks
+
+(all done — session complete)
+
+- [x] Add ticket-gate Crush hook (blocks work without active ticket)
+- [x] Document active-ticket workflow in AGENTS.md + meta template
@@ -0,0 +1,67 @@
+#!/usr/bin/bash
+# access-matrix.sh — definitive access verification across all Linux Tailscale nodes.
+# Uses the correct SSH user(s) per system type, checks sudo where applicable.
+# Routes through remote.sh (the only allowed ssh path).
+set -u
+cd /home/reachableceo/projects/PFVCluster || exit 1
+
+# Policy-excluded systems (never attempt access)
+EXCLUDE=':tsys-cloudron:pfv-bms:tsys-umbrel:tsys-ucs-01:tsys-ucs-02:stlpc-bizoffice:ultix-highside:'
+
+# Determine the SSH user(s) for a given hostname and whether sudo is expected.
+# Returns "user1:user2:...:sudoflag" where sudoflag is "yes" or "no".
+users_for() {
+  local name="$1"
+  case "$name" in
+    pfv-tsys[0-9])                   echo "root:no" ;;
+    *-proxmox-datacenter)            echo "root:no" ;;
+    *-proxmox-pve)                   echo "root:no" ;;
+    *-proxmox-pbs)                   echo "root:no" ;;
+    *-proxmox-mailgw*)               echo "root:no" ;;
+    *-proxmox-backup*)               echo "root:no" ;;
+    tsys-ucs-*)                      echo "root:no" ;;
+    ultix-streaming)                 echo "root:no" ;;
+    stlpc-*)                         echo "root:labuser:no" ;;
+    ultix-field)                     echo "ultixfield:yes" ;;
+    subopi*)                         echo "subodev:yes" ;;
+    *)                               echo "localuser:yes" ;;
+  esac
+}
+
+check_user() {
+  local ip="$1" user="$2" expect_sudo="$3"
+  local out sudo
+  out=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'echo SSHOK; id -un' </dev/null 2>&1 | grep -oE 'SSHOK|keyboard-interactive|Connection refused' | head -1)
+  case "$out" in
+    SSHOK)
+      if [ "$expect_sudo" = "yes" ]; then
+        sudo=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'sudo -n true 2>/dev/null && echo SUDOOK || echo SUDONO' </dev/null 2>&1 | grep -oE 'SUDOOK|SUDONO' | head -1)
+        printf '%s(%s)' "$user" "${sudo:-?}"
+      else
+        printf '%s(ok)' "$user"
+      fi
+      ;;
+    keyboard-interactive) printf '%s(2FA)' "$user" ;;
+    'Connection refused')  printf '%s(NOSSH)' "$user" ;;
+    *)                      printf '%s(NOKEY)' "$user" ;;
+  esac
+}
+
+printf '%-32s %-16s %s\n' "NAME" "TS-IP" "ACCESS"
+printf '%-32s %-16s %s\n' "----" "-----" "------"
+
+tailscale status 2>/dev/null | awk '$4=="linux" {print $2, $1}' | sort | while read -r name ip; do
+  [ -n "$name" ] || continue
+  case "$EXCLUDE" in *":$name:"*) printf '%-32s %-16s %s\n' "$name" "$ip" "EXCLUDED"; continue;; esac
+
+  map=$(users_for "$name")
+  expect_sudo="${map##*:}"
+  users="${map%:*}"
+  result=""
+  IFS=':' read -ra user_list <<< "$users"
+  for u in "${user_list[@]}"; do
+    r=$(check_user "$ip" "$u" "$expect_sudo")
+    [ -z "$result" ] && result="$r" || result="$result $r"
+  done
+  printf '%-32s %-16s %s\n' "$name" "$ip" "$result"
+done
@@ -0,0 +1,60 @@
+#!/bin/sh
+# agent-bootstrap.sh
+#
+# Run INSIDE a guest (via SSH, console, or guest-agent) to bring the
+# system fully under agent management in one shot:
+#   1. install + enable qemu-guest-agent  (VMs only, skipped on bare metal)
+#   2. push the agent SSH key to root + AGENT_USER (+ labuser if present)
+#   3. grant AGENT_USER passwordless sudo
+#
+# AGENT_USER defaults to "localuser". Override for systems with a different
+# unprivileged agent user:
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+#
+# After this runs once, the agent has SSH+sudo immediately.
+#
+# Usage (from a root shell in the guest):
+#   bash agent-bootstrap.sh
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+
+set -eu
+
+KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIWms/uCXnjjo4KyxHBcYI2TDHe8OZ2wle6W/0hSRQLu reachableceo@ultix-streaming'
+AGENT_USER="${AGENT_USER:-localuser}"
+
+# 1. guest-agent (skip on bare metal — no virtio-serial device)
+if command -v systemd-detect-virt >/dev/null 2>&1 && \
+   [ "$(systemd-detect-virt --vm 2>/dev/null || echo none)" != "none" ]; then
+  if ! command -v qemu-ga >/dev/null 2>&1; then
+    if command -v apt-get >/dev/null 2>&1; then
+      apt-get update
+      DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
+    elif command -v dnf >/dev/null 2>&1; then
+      dnf install -y qemu-guest-agent
+    elif command -v yum >/dev/null 2>&1; then
+      yum install -y qemu-guest-agent
+    else
+      echo "WARN: no supported package manager; skipping agent install" >&2
+    fi
+  fi
+  systemctl enable --now qemu-guest-agent 2>/dev/null || \
+    systemctl enable --now qemu-ga 2>/dev/null || true
+fi
+
+# 2. SSH key for root + AGENT_USER + labuser (if present)
+for u in root "$AGENT_USER" labuser; do
+  if ! getent passwd "$u" >/dev/null 2>&1; then continue; fi
+  H=$(getent passwd "$u" | cut -d: -f6)
+  mkdir -p "$H/.ssh"; chmod 700 "$H/.ssh"
+  AK="$H/.ssh/authorized_keys"; touch "$AK"; chmod 600 "$AK"
+  grep -qF "$KEY" "$AK" || echo "$KEY" >> "$AK"
+  chown -R "$u": "$H/.ssh"
+done
+
+# 3. passwordless sudo for AGENT_USER only
+if getent passwd "$AGENT_USER" >/dev/null 2>&1 && [ -d /etc/sudoers.d ]; then
+  echo "${AGENT_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/010-agent
+  chmod 440 /etc/sudoers.d/010-agent
+fi
+
+echo BOOTSTRAP-DONE
@@ -0,0 +1,23 @@
+# PFV NFS tuning sysctl overrides
+#
+# Applied AFTER tuned via pfv-nfs-tuning.service (systemd oneshot).
+# These override tuned's network-throughput/virtual-host 16MB TCP buffer
+# caps with 128MB for high-BDP NFS over 1-4 GbE LACP links.
+#
+# Install on ALL Proxmox hosts:
+#   cp 99-pfv-nfs.conf /etc/sysctl.d/99-pfv-nfs.conf
+#   cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service
+#   systemctl daemon-reload && systemctl enable --now pfv-nfs-tuning.service
+#
+# Created: 2026-07-31
+# Deployed: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
+
+net.core.rmem_max = 134217728
+net.core.wmem_max = 134217728
+net.core.rmem_default = 26214400
+net.core.wmem_default = 26214400
+net.core.netdev_max_backlog = 250000
+net.core.somaxconn = 65535
+net.ipv4.tcp_rmem = 4096 87380 134217728
+net.ipv4.tcp_wmem = 4096 65536 134217728
+net.ipv4.tcp_max_syn_backlog = 4096
@@ -0,0 +1,28 @@
+# PFV NFS tuning service
+#
+# Systemd oneshot that runs AFTER tuned.service to apply TCP buffer
+# overrides. The tuned daemon's profiles (network-throughput for storage
+# hosts, virtual-host for compute hosts) set 16MB TCP buffer caps which
+# are too small for high-BDP NFS over LACP links. This service force-
+# applies 128MB buffers after tuned has finished its configuration.
+#
+# Install:
+#   cp pfv-nfs-tuning.service /etc/systemd/system/pfv-nfs-tuning.service
+#   systemctl daemon-reload
+#   systemctl enable --now pfv-nfs-tuning.service
+#
+# Created: 2026-07-31
+# Deployed: all 7 Proxmox hosts (tsys1/3/4/5/6/7/9)
+
+[Unit]
+Description=PFV NFS tuning (override tuned TCP buffer caps)
+After=tuned.service
+Requires=tuned.service
+
+[Service]
+Type=oneshot
+ExecStart=/sbin/sysctl -p /etc/sysctl.d/99-pfv-nfs.conf
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
@@ -5,17 +5,18 @@
 #magic to detect main int
 echo "Determining management interface..."
 #export MAIN_INT=$(brctl show $(netstat -rn|grep 0.0.0.0|head -n1|awk '{print $NF}') | awk '{print $NF}'|tail -1|awk -F '.' '{print $1}')
-export MAIN_INT=$(brctl show|grep vmbr0|awk '{print $NF}'|awk -F '.' '{print $1}')
+MAIN_INT=$(brctl show|grep vmbr0|awk '{print $NF}'|awk -F '.' '{print $1}')
+export MAIN_INT

 echo "Management interface is: $MAIN_INT"

 #fix the issue
 echo "Fixing management interface..."
-ethtool -K $MAIN_INT tso off 
-ethtool -K $MAIN_INT gro off 
-ethtool -K $MAIN_INT gso off
-ethtool -K $MAIN_INT tx off
-ethtool -K $MAIN_INT rx off
+ethtool -K "$MAIN_INT" tso off 
+ethtool -K "$MAIN_INT" gro off 
+ethtool -K "$MAIN_INT" gso off
+ethtool -K "$MAIN_INT" tx off
+ethtool -K "$MAIN_INT" rx off

 #https://forum.proxmox.com/threads/e1000-driver-hang.58284/
 #https://serverfault.com/questions/616485/e1000e-reset-adapter-unexpectedly-detected-hardware-unit-hang
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+# auth-cloudron-ldap.sh — placeholder module (Cloudron LDAP auth integration).
+# Intentionally empty; populated when the auth stack is deployed.
+true
@@ -1,4 +1,5 @@
 #!/bin/bash
+# shellcheck disable=SC2103 # legacy R&D build script; cd/cd- sequence is intentional

 #Made from instructions at https://www.tunetheweb.com/performance/http2/

@@ -24,17 +25,17 @@ CURL_FILE="curl-7.60.0.tar.gz"
 #Download and install latest version of openssl
 wget $OPENSSL_URL_BASE/$OPENSSL_FILE
 tar xzf $OPENSSL_FILE
-cd openssl-1.1.0h 
+cd openssl-1.1.0h || exit
 ./config enable-weak-ssl-ciphers shared zlib-dynamic -DOPENSSL_TLS_SECURITY_LEVEL=0 --prefix=/usr/local/custom-ssl/openssl-1.1.0h ; make ; make install
 ln -s /usr/local/custom-ssl/openssl-1.1.0h /usr/local/openssl
-cd -
+cd - || exit

 #Download and install nghttp2 (needed for mod_http2).
 wget $NGHTTP_URL_BASE/$NGHTTP_FILE
 tar xzf $NGHTTP_FILE
-cd nghttp2-1.31.0 
+cd nghttp2-1.31.0 || exit
 ./configure --prefix=/usr/local/custom-ssl/nghttp ; make ; make install
-cd -
+cd - || exit

 #Updated ldconfig so curl build

@@ -48,34 +49,34 @@ ldconfig
 #Download and install curl
 wget $CURL_URL_BASE/$CURL_FILE
 tar xzf curl-7.60.0.tar.gz
-cd curl-7.60.0
+cd curl-7.60.0 || exit
 ./configure --prefix=/usr/local/custom-ssl/curl --with-nghttp2=/usr/local/custom-ssl/nghttp/ --with-ssl=/usr/local/custom-ssl/openssl-1.1.0h/ ; make ; make install
-cd -
+cd - || exit


 #Download and install latest apr
 wget $APR_URL_BASE/$APR_FILE
 tar xzf $APR_FILE
-cd apr-1.6.3 
+cd apr-1.6.3 || exit
 ./configure --prefix=/usr/local/custom-ssl/apr ; make ; make install
-cd -
+cd - || exit

 #Download and install latest apr-util
 wget $APR_UTIL_URL_BASE/$APR_UTIL_FILE
 tar xzf apr-util-1.6.1.tar.gz
-cd apr-util-1.6.1 
+cd apr-util-1.6.1 || exit
 ./configure --prefix=/usr/local/custom-ssl/apr-util --with-apr=/usr/local/custom-ssl/apr ; make; make install
-cd -
+cd - || exit

 #Download and install apache
 wget $APACHE_URL_BASE/$APACHE_FILE
 tar xzf httpd-2.4.33.tar.gz
-cd httpd-2.4.33
+cd httpd-2.4.33 || exit
 cp -r ../apr-1.6.3 srclib/apr
 cp -r ../apr-util-1.6.1 srclib/apr-util
 ./configure --prefix=/usr/local/custom-ssl/apache  --with-ssl=/usr/local/custom-ssl/openssl-1.1.0h/ --with-pcre=/usr/bin/pcre-config --enable-unique-id --enable-ssl --enable-so --with-included-apr --enable-http2 --with-nghttp2=/usr/local/custom-ssl/nghttp/
 make
 make install
 ln -s /usr/local/custom-ssl/apache /usr/local/apache
-cd -
+cd - || exit

@@ -79,7 +79,7 @@ if [ "$SUBODEV_CHECK" = 1 ]; then
 fi

 export DEV_WORKSTATION_CHECK
-DEV_WORKSTATION_CHECK="$(hostname | egrep -c 'subopi-dev|CharlesDevServer' || true)"
+DEV_WORKSTATION_CHECK="$(hostname | grep -Ec 'subopi-dev|CharlesDevServer' || true)"

 if [ "$DEV_WORKSTATION_CHECK" -eq 0 ]; then
    
@@ -9,7 +9,7 @@ export user_check
 user_check="$(echo "$curr_user" | grep -c root)"


-if [ $user_check -ne 1 ]; then
+if [ "$user_check" -ne 1 ]; then
    print_error "Must run as root."
    error_out
 fi
@@ -1,7 +1,7 @@
-
+# shellcheck shell=bash disable=SC2148 # sourced function file (no shebang by design)
 function pi-detect()
 {
-print_info Now running "$FUNCNAME"....
+print_info Now running "${FUNCNAME[0]}"....
 if [ -f /sys/firmware/devicetree/base/model ] ; then
 export IS_RASPI="1"
 fi
@@ -9,5 +9,5 @@ fi
 if [ ! -f /sys/firmware/devicetree/base/model ] ; then
 export IS_RASPI="0"
 fi
-print_info Completed running "$FUNCNAME"
+print_info Completed running "${FUNCNAME[0]}"
 }
@@ -60,18 +60,18 @@ LOCALUSER_CHECK="$(getent passwd | grep -c localuser || true)"
 #######################

 function global-oam() {
-  print_info "Now running $FUNCNAME...."
+  print_info "Now running ${FUNCNAME[0]}...."

  cat "$SCRIPTS_PATH/up2date.sh" >/usr/local/bin/up2date.sh && chmod +x /usr/local/bin/up2date.sh

  bash "$MODULES_PATH/OAM/oam-librenms.sh"

-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"

 }

 function global-systemServiceConfigurationFiles() {
-  print_info "Now running $FUNCNAME...."
+  print_info "Now running ${FUNCNAME[0]}...."

  cat "$CONFIGFILES_PATH/ZSH/tsys-zshrc" >/etc/zshrc
  cat "$CONFIGFILES_PATH/SMTP/aliases" >/etc/aliases
@@ -79,11 +79,11 @@ function global-systemServiceConfigurationFiles() {

  newaliases

-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function global-installPackages() {
-  print_info "Now running $FUNCNAME...."
+  print_info "Now running ${FUNCNAME[0]}...."

  # Setup webmin repo, used for RBAC/2fa PAM

@@ -195,7 +195,7 @@ function global-installPackages() {
  VIRT_TYPE="$(virt-what)"

  export IS_VIRT_GUEST
-  IS_VIRT_GUEST="$(echo "$VIRT_TYPE" | egrep -c 'hyperv|kvm' || true)"
+  IS_VIRT_GUEST="$(echo "$VIRT_TYPE" | grep -Ec 'hyperv|kvm' || true)"

  export IS_KVM_GUEST
  IS_KVM_GUEST="$(echo "$VIRT_TYPE" | grep -c 'kvm' || true)"
@@ -227,12 +227,12 @@ function global-installPackages() {

 # vault cli

-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function global-postPackageConfiguration() {

-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"

  systemctl --now enable auditd

@@ -253,7 +253,7 @@ function global-postPackageConfiguration() {
  #This is under test/dev and may fail
  echo "hi from root to root" | mail -s "hi directly to root from $(hostname)" root

-  chsh -s $(which zsh) root
+  chsh -s "$(which zsh)" root

  if [ "$LOCALUSER_CHECK" -gt 0 ]; then
    chsh -s "$(which zsh)" localuser
@@ -312,7 +312,7 @@ function global-postPackageConfiguration() {
  fi

  export NTP_SERVER_CHECK
-  NTP_SERVER_CHECK="$(hostname | egrep -c 'pfv-netboot|pfvsvrpi|pfv-netinfra' || true)"
+  NTP_SERVER_CHECK="$(hostname | grep -Ec 'pfv-netboot|pfvsvrpi|pfv-netinfra' || true)"

  if [ "$NTP_SERVER_CHECK" -eq 0 ]; then

@@ -341,7 +341,7 @@ function global-postPackageConfiguration() {
    tuned-adm profile virtual-guest
  fi

-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 ####################################################################################################
@@ -355,41 +355,41 @@ function global-postPackageConfiguration() {
 # SSH

 function secharden-ssh() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"

  bash "$MODULES_PATH/Security/secharden-ssh.sh"

-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function secharden-wazuh() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"
  bash "$MODULES_PATH/Security/secharden-wazuh.sh"
-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function secharden-2fa() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"
  bash "$MODULES_PATH/Security/secharden-2fa.sh"
-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function secharden-scap-stig() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"
  bash "$MODULES_PATH/Security/secharden-scap-stig.sh"
-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function secharden-agents() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"
  bash "$MODULES_PATH/Security/secharden-audit-agents.sh"
-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 function secharden-auto-upgrades() {
-  print_info "Now running $FUNCNAME"
+  print_info "Now running ${FUNCNAME[0]}"
  #curl --silent ${DL_ROOT}/Modules/Security/secharden-ssh.sh|$(which bash)
-  print_info "Completed running $FUNCNAME"
+  print_info "Completed running ${FUNCNAME[0]}"
 }


@@ -400,16 +400,16 @@ function secharden-auto-upgrades() {
 ####################################################################################################

 function auth-cloudron-ldap() {
-  print_info "Now running "$FUNCNAME""
+  print_info "Now running ${FUNCNAME[0]}"
  #curl --silent ${DL_ROOT}/Modules/Auth/auth-cloudron-ldap.sh|$(which bash)
-  print_info "Completed running "$FUNCNAME""
+  print_info "Completed running ${FUNCNAME[0]}"
 }

 ####################################################################################################
 # RUn the various functions in the correct order
 ####################################################################################################

-echo >$LOGFILENAME
+echo >"$LOGFILENAME"

 print_info "Execution starting at $CURRENT_TIMESTAMP..."

@@ -1,3 +1,4 @@
+# shellcheck shell=bash disable=SC2148 # sourced .bashrc profile fragment
 if command -v tmux &> /dev/null && [ -n "$PS1" ] && [[ ! "$TERM" =~ screen ]] && [[ ! "$TERM" =~ tmux ]] && [ -z "$TMUX" ]; then
          tmux a -t default || exec tmux new -s default && exit;
 fi
@@ -0,0 +1,2 @@
+# shellcheck shell=bash disable=SC2148 # sourced .bashrc profile fragment
+export HISTTIMEFORMAT="%m/%d/%Y %T "
@@ -0,0 +1,9 @@
+<!-- Discourse: https://community.turnsys.com/t/298 -->
+<!-- Redmine: https://projects.knownelement.com/issues/314 -->
+
+# Session Handoff: 2026-08-11/12 OAM
+
+**Full handoff lives in Redmine #314** (session summary note, 2026-08-12).
+**OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309).
+**Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303).
+**Open questions:** `questions-v2.md` in this repo.
@@ -0,0 +1,10 @@
+# awx/README.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Ansible AWX deployment on k3s**
+>
+> **Read it here:** https://community.turnsys.com/t/302
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*
@@ -0,0 +1,46 @@
+---
+# AWX namespace
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: awx
+---
+# Admin password secret — the password is 'REDACTED_PASSWORD' (fleet standard)
+apiVersion: v1
+kind: Secret
+metadata:
+  name: awx-admin-password
+  namespace: awx
+type: Opaque
+stringData:
+  password: REDACTED_PASSWORD
+---
+# AWX Custom Resource — single instance, LoadBalancer service
+apiVersion: awx.ansible.com/v1beta1
+kind: AWX
+metadata:
+  name: tsys-awx
+  namespace: awx
+spec:
+  service_type: LoadBalancer
+  ingress_type: none
+
+  admin_user: admin
+  admin_password_secret: awx-admin-password
+
+  # PostgreSQL — bundled, stored on local disk via PVC (k3s local-path)
+  postgres_storage_class: local-path
+  postgres_storage_requirements:
+    requests:
+      storage: 8Gi
+  postgres_resource_requirements:
+    requests:
+      memory: 1Gi
+
+  # Resource limits — fit within 12 GB host RAM
+  web_resource_requirements:
+    requests:
+      memory: 1Gi
+  task_resource_requirements:
+    requests:
+      memory: 1Gi
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+###############################################################################
+# deploy-awx.sh — Deploy AWX Operator + instance on k3s.
+#
+# Prerequisites: k3s must be installed and running (install-k3s.sh).
+# Intended to run ON the target VM (tsys-awx.knel.net) as root or via sudo.
+#
+# Usage:  sudo bash deploy-awx.sh
+###############################################################################
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}"
+
+OPERATOR_VERSION="${OPERATOR_VERSION:-2.19.1}"
+
+echo "=========================================================="
+echo " AWX Operator deployment — version ${OPERATOR_VERSION}"
+echo "=========================================================="
+
+# ---------------------------------------------------------------------------
+# Step 1: Create namespace
+# ---------------------------------------------------------------------------
+echo ""
+echo "=== Step 1: Create namespace ==="
+kubectl apply -f "${SCRIPT_DIR}/namespace.yaml"
+
+# ---------------------------------------------------------------------------
+# Step 2: Deploy AWX Operator
+# ---------------------------------------------------------------------------
+echo ""
+echo "=== Step 2: Deploy AWX Operator ${OPERATOR_VERSION} ==="
+
+# Clone the operator to get kustomize manifests
+OPERATOR_DIR="/tmp/awx-operator-${OPERATOR_VERSION}"
+rm -rf "${OPERATOR_DIR}"
+git clone --branch "${OPERATOR_VERSION}" --depth 1 \
+    "https://github.com/ansible/awx-operator.git" "${OPERATOR_DIR}" 2>&1 | tail -3
+
+# The operator's default namespace is 'awx' — matches our setup
+# Apply the operator via kustomize (config/default has the full manifest set)
+kubectl apply -k "${OPERATOR_DIR}/config/default" 2>&1 || {
+    echo "kustomize apply failed, trying raw manifests..."
+    kubectl apply -f "https://raw.githubusercontent.com/ansible/awx-operator/${OPERATOR_VERSION}/deploy/awx-operator.yaml"
+}
+
+# Fix kube-rbac-proxy image (gcr.io/kubebuilder/kube-rbac-proxy was removed;
+# quay.io/brancz/kube-rbac-proxy is the maintained replacement)
+echo ""
+echo "=== Patching kube-rbac-proxy image ==="
+kubectl set image deployment/awx-operator-controller-manager -n awx \
+    kube-rbac-proxy=quay.io/brancz/kube-rbac-proxy:v0.15.0 2>&1 || true
+
+# Scale down any old replicasets that still reference the broken image
+for rs in $(kubectl -n awx get rs -l control-plane=controller-manager -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do
+    img=$(kubectl -n awx get rs "${rs}" -o jsonpath='{.spec.template.spec.containers[?(@.name=="kube-rbac-proxy")].image}' 2>/dev/null)
+    if [[ "${img}" == *"gcr.io/kubebuilder"* ]]; then
+        echo "Scaling down old RS ${rs} (has broken gcr.io image)"
+        kubectl -n awx scale rs "${rs}" --replicas=0 2>&1
+    fi
+done
+
+echo ""
+echo "Waiting for AWX Operator deployment to be ready..."
+kubectl -n awx wait --for=condition=Available deployment/awx-operator-controller-manager \
+    --timeout=300s 2>&1 || {
+    echo "Operator not ready yet — checking status..."
+    kubectl -n awx get pods
+}
+
+# ---------------------------------------------------------------------------
+# Step 3: Deploy AWX instance
+# ---------------------------------------------------------------------------
+echo ""
+echo "=== Step 3: Deploy AWX instance ==="
+kubectl apply -f "${SCRIPT_DIR}/awx-instance.yaml"
+
+echo ""
+echo "AWX instance created. Operator will now reconcile."
+echo "This typically takes 5-10 minutes for the first deployment."
+echo ""
+echo "Monitor progress with:"
+echo "  kubectl -n awx get awx tsys-awx -o jsonpath='{.status.conditions}' | jq ."
+echo "  kubectl -n awx get pods -w"
+echo "  kubectl -n awx logs deployment/awx-operator-controller-manager -f"
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+###############################################################################
+# install-k3s.sh — Install k3s single-node on the tsys-awx VM.
+#
+# Intended to run ON the target VM (tsys-awx.knel.net) as root or via sudo.
+# Installs k3s without Traefik (we use NodePort/LoadBalancer directly).
+#
+# Usage:  sudo bash install-k3s.sh
+###############################################################################
+set -euo pipefail
+
+NODE_NAME="${NODE_NAME:-tsys-awx}"
+
+echo "=========================================================="
+echo " k3s single-node install — ${NODE_NAME}"
+echo "=========================================================="
+
+if command -v k3s >/dev/null 2>&1 && k3s kubectl get nodes >/dev/null 2>&1; then
+    echo "k3s already installed and running. Skipping."
+    k3s kubectl get nodes
+    exit 0
+fi
+
+echo ""
+echo "=== Installing k3s (this takes 1-2 minutes) ==="
+curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik --write-kubeconfig-mode=644" sh -
+
+echo ""
+echo "=== Waiting for k3s node to be Ready ==="
+export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+for i in $(seq 1 30); do
+    if k3s kubectl get nodes 2>/dev/null | grep -q ' Ready'; then
+        echo "Node is Ready!"
+        k3s kubectl get nodes
+        break
+    fi
+    echo "  waiting... (${i}/30)"
+    sleep 5
+done
+
+echo ""
+echo "=== k3s install complete ==="
+echo "kubeconfig: /etc/rancher/k3s/k3s.yaml"
+echo "kubectl:    k3s kubectl  (or set KUBECONFIG=/etc/rancher/k3s/k3s.yaml)"
@@ -0,0 +1,6 @@
+---
+# AWX Operator namespace
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: awx
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+###############################################################################
+# verify-awx.sh — Verify AWX deployment status and access.
+#
+# Intended to run ON the target VM (tsys-awx.knel.net).
+# Usage:  bash verify-awx.sh
+###############################################################################
+set -euo pipefail
+
+export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}"
+
+echo "=========================================================="
+echo " AWX Deployment Verification — $(date)"
+echo "=========================================================="
+
+echo ""
+echo "=== 1. k3s node ==="
+kubectl get nodes
+
+echo ""
+echo "=== 2. AWX pods ==="
+kubectl -n awx get pods
+
+echo ""
+echo "=== 3. AWX CR status ==="
+kubectl -n awx get awx tsys-awx -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' 2>/dev/null || echo "AWX CR not found"
+
+echo ""
+echo "=== 4. Services ==="
+kubectl -n awx get svc
+
+echo ""
+echo "=== 5. LoadBalancer / NodePort access ==="
+LB_IP=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "")
+LB_HOST=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' 2>/dev/null || echo "")
+NODE_PORT=$(kubectl -n awx get svc tsys-awx-service -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null || echo "")
+
+if [ -n "${LB_IP}" ]; then
+    echo "LoadBalancer IP: ${LB_IP}"
+    ACCESS_URL="http://${LB_IP}"
+elif [ -n "${LB_HOST}" ]; then
+    echo "LoadBalancer hostname: ${LB_HOST}"
+    ACCESS_URL="http://${LB_HOST}"
+elif [ -n "${NODE_PORT}" ]; then
+    echo "NodePort: ${NODE_PORT}"
+    ACCESS_URL="http://$(hostname -I | awk '{print $1}'):${NODE_PORT}"
+else
+    echo "Service not ready yet"
+    ACCESS_URL=""
+fi
+
+echo ""
+echo "=== 6. Admin password ==="
+ADMIN_PASS=$(kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' 2>/dev/null | base64 -d 2>/dev/null || echo "")
+if [ -n "${ADMIN_PASS}" ]; then
+    echo "User: admin"
+    echo "Password: ${ADMIN_PASS}"
+else
+    echo "Admin password secret not found"
+fi
+
+echo ""
+echo "=== 7. HTTP check ==="
+if [ -n "${ACCESS_URL}" ]; then
+    echo "Testing ${ACCESS_URL}..."
+    HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "${ACCESS_URL}" 2>/dev/null || echo "failed")
+    echo "HTTP response: ${HTTP_CODE}"
+    if [ "${HTTP_CODE}" = "200" ] || [ "${HTTP_CODE}" = "302" ] || [ "${HTTP_CODE}" = "301" ]; then
+        echo "✓ AWX is accessible at ${ACCESS_URL}"
+    else
+        echo "✗ AWX not yet responding (HTTP ${HTTP_CODE})"
+    fi
+fi
+
+echo ""
+echo "=========================================================="
+if [ -n "${ACCESS_URL}" ]; then
+    echo " AWX Access URL: ${ACCESS_URL}"
+fi
+echo "=========================================================="
@@ -0,0 +1,65 @@
+#!/usr/bin/bash
+# bootstrap-all.sh — push agent SSH key + passwordless sudo to remaining systems.
+#
+# SSH is the ONLY approved access channel (see AGENTS.md "Access-channel
+# policy: SSH only"). This script reaches systems that still allow password
+# auth over sshd. Systems that reject password auth (publickey-only) cannot
+# be reached this way — see the CONSOLE-ONLY section printed at the end.
+#
+# Two escalation methods:
+#   sudo → Ubuntu-style systems (no root pw; localuser has sudo)
+#   su   → Debian-style systems (root has a password)
+#
+# Passes AGENT_USER so agent-bootstrap.sh targets the correct unprivileged
+# user. You enter passwords interactively. Idempotent: safe to re-run.
+#
+# Verified state (access-matrix.sh, 2026-08-10): 68/70 non-excluded systems
+# at intended access state. Only tsys-siem remains below.
+set -u
+cd "$(dirname "$0")" || exit 1
+
+SCRIPT=agent-bootstrap.sh
+SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
+
+run_with_sudo() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (sudo, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "sudo AGENT_USER=${agent_user} bash /tmp/$SCRIPT" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+run_with_su() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (su, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "su -c 'AGENT_USER=${agent_user} bash /tmp/$SCRIPT'" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+# All password-auth-reachable systems have been bootstrapped.
+# Verified state (access-matrix.sh, 2026-08-10): 69/70 non-excluded systems
+# at intended access state. The only remaining NOKEY (stlp-3dscanner) is
+# deferred to [#417] and requires a rename + bring-online first — out of
+# scope here. Run `access-matrix.sh` to re-verify at any time.
+echo "All password-auth-reachable systems are bootstrapped."
+echo "Remaining gap: stlp-3dscanner (deferred to [#417])."
+echo "Run access-matrix.sh to re-verify."
+echo "Deferred (separate ticket):"
+echo "  stlp-3dscanner — rename + bring online first [#417]"
+echo "========================================"
+echo "By design (leave alone):"
+echo "  sectestbed-sandbox — 2FA enforced"
+echo "========================================"
+echo "Excluded by policy (no SSH access):"
+echo "  pfv-bms (API), tsys-cloudron (prod revenue),"
+echo "  tsys-ucs-01/02 (API-managed), tsys-umbrel (treasury)"
+echo "========================================"
@@ -0,0 +1,10 @@
+# dcinfra/console/README.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Serial console management (ser2net + conman)**
+>
+> **Read it here:** https://community.turnsys.com/t/301
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*
@@ -0,0 +1,103 @@
+#!/usr/bin/bash
+# shellcheck disable=SC2010,SC2012 # diagnostic script; ls|grep/ls -la on sysfs & log dirs is intentional for human-readable output
+#
+# console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4
+#
+# Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
+#
+# This script is strictly read-only. No writes to the system.
+#
+set -uo pipefail
+
+echo "============================================"
+echo "  Console Setup Discovery"
+echo "  Host: $(hostname)"
+echo "  Date: $(date)"
+echo "  READ-ONLY"
+echo "============================================"
+
+echo ""
+echo "=== 1. USB devices ==="
+lsusb 2>/dev/null || echo "(lsusb not available)"
+
+echo ""
+echo "=== 2. All ttyUSB* devices (with major/minor) ==="
+ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
+
+echo ""
+echo "=== 3. USB-serial driver bindings ==="
+echo "-- pl2303 --"
+ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
+echo "-- cp210x --"
+ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
+echo "-- ftdi_sio --"
+ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
+echo "-- ch341 --"
+ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
+
+echo ""
+echo "=== 4. USB serial adapter details (vendor/model/serial per port) ==="
+for tty in /dev/ttyUSB*; do
+  [ -e "$tty" ] || continue
+  echo "--- $tty ---"
+  udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/  /'
+done
+
+echo ""
+echo "=== 5. Existing /root/conmap ==="
+if [ -f /root/conmap ]; then
+  cat /root/conmap
+else
+  echo "(no /root/conmap)"
+fi
+ls -la /root/conmap* 2>/dev/null
+
+echo ""
+echo "=== 6. Screen sessions (running) ==="
+screen -ls 2>&1 || echo "(screen not running or not installed)"
+
+echo ""
+echo "=== 7. Existing screen wrappers/scripts in /root ==="
+ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)"
+
+echo ""
+echo "=== 8. ser2net ==="
+which ser2net 2>/dev/null || echo "(ser2net not installed)"
+dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)"
+cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)"
+systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)"
+
+echo ""
+echo "=== 9. conman ==="
+which conman 2>/dev/null || echo "(conman not installed)"
+which conmand 2>/dev/null || echo "(conmand not installed)"
+dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)"
+echo "--- /etc/conman.conf (console lines only) ---"
+grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)"
+echo "--- conmand service ---"
+systemctl is-active conmand 2>/dev/null || echo "(conmand not running)"
+systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)"
+
+echo ""
+echo "=== 10. Existing console logs ==="
+ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)"
+ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)"
+
+echo ""
+echo "=== 11. udev rules for ttyUSB ==="
+grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)"
+grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true
+
+echo ""
+echo "=== 12. expect availability ==="
+command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
+command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed"
+
+echo ""
+echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ==="
+ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)"
+
+echo ""
+echo "============================================"
+echo "  Discovery complete (read-only)."
+echo "============================================"
@@ -0,0 +1,254 @@
+#!/usr/bin/bash
+#
+# console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf
+#
+# Reads console/mapping.txt (the source of truth) and generates all three
+# config files. This is the fix for the USB enumeration shift problem:
+#
+#   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
+#      to a named symlink like /dev/consoles/pfv-r5-core-01
+#   2. ser2net opens those stable symlinks and exposes them on TCP ports
+#      (2001, 2002, ...) bound to the Tailscale IP
+#   3. conman connects to those TCP ports for logging + multiplexing
+#
+# Run this script ON the target host. It writes to:
+#   /etc/udev/rules.d/99-console-ports.rules
+#   /etc/ser2net.yaml
+#   /etc/conman/console-consoles.conf  (included by /etc/conman.conf)
+#
+# Usage:
+#   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh
+#
+# Environment overrides:
+#   MAPPING_FILE  — path to mapping.txt (default: auto-detect next to this script)
+#   TS_IP         — Tailscale IP to bind ser2net on (default: auto-detect)
+#   CONMAN_LOGDIR — conman log directory (default: /var/log/conman)
+#
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}"
+CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}"
+
+UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
+SER2NET_CONF="/etc/ser2net.yaml"
+CONMAN_CONF="/etc/conman.conf"
+
+echo "============================================"
+echo "  Console Config Generator"
+echo "  Host: $(hostname)   $(date)"
+echo "============================================"
+
+# --- Locate mapping file ---
+# When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong.
+# Search common locations.
+if [ ! -f "$MAPPING_FILE" ]; then
+  for candidate in \
+    "/root/console/mapping.txt" \
+    "/tmp/mapping.txt" \
+    "$(dirname "$0")/mapping.txt"; do
+    if [ -f "$candidate" ]; then
+      MAPPING_FILE="$candidate"
+      break
+    fi
+  done
+fi
+
+if [ ! -f "$MAPPING_FILE" ]; then
+  echo "FATAL: mapping file not found. Tried: $MAPPING_FILE"
+  echo "Copy mapping.txt to the target host first."
+  exit 1
+fi
+echo "  Mapping file: $MAPPING_FILE"
+
+# --- Auto-detect Tailscale IP ---
+if [ -z "${TS_IP:-}" ]; then
+  TS_IP=$(tailscale ip -4 2>/dev/null || true)
+  if [ -z "$TS_IP" ]; then
+    echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually."
+    exit 1
+  fi
+fi
+echo "  Tailscale IP: $TS_IP"
+echo "  ser2net will bind to: $TS_IP"
+
+# --- Parse mapping file (skip comments and blank lines) ---
+echo ""
+echo "--- Parsing mapping file ---"
+ENTRIES=()
+while IFS= read -r line; do
+  # Skip comments and blank lines
+  line="${line%%#*}"
+  line="$(echo "$line" | xargs)"  # trim whitespace
+  [ -z "$line" ] && continue
+  ENTRIES+=("$line")
+  echo "  $line"
+done < "$MAPPING_FILE"
+
+if [ "${#ENTRIES[@]}" -eq 0 ]; then
+  echo "FATAL: no entries found in mapping file."
+  exit 1
+fi
+
+echo ""
+echo "  ${#ENTRIES[@]} console ports configured."
+
+# ============================================================
+# 1. Generate udev rules
+# ============================================================
+echo ""
+echo "--- [1/3] Generating udev rules: $UDEV_RULES ---"
+
+cat > "$UDEV_RULES" <<'UDEV_HEADER'
+# Stable symlinks for USB-DB9 console adapters
+# Generated by console/generate-config.sh
+# DO NOT EDIT — edit mapping.txt and re-run generate-config.sh
+#
+# These rules pin each adapter to a named symlink based on its physical
+# USB port path (ID_PATH), which is stable across reboots regardless of
+# enumeration order. This is the fix for the "USB adapters shift on reboot"
+# problem.
+#
+# To find the ID_PATH for a device:
+#   udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH
+UDEV_HEADER
+
+for entry in "${ENTRIES[@]}"; do
+  IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
+  # Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4"
+  # The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0"
+  # We match on the substring to be portable across PCI bus changes.
+  {
+    echo ""
+    echo "# $name (TCP $tcp_port): $comment"
+    echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"consoles/$name\""
+  } >> "$UDEV_RULES"
+done
+
+echo "  Written: $UDEV_RULES"
+echo "  Symlinks: /dev/consoles/<name> for each device"
+
+# ============================================================
+# 2. Generate ser2net.yaml
+# ============================================================
+echo ""
+echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---"
+
+# Backup existing config if not already backed up
+if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then
+  cp "$SER2NET_CONF" "${SER2NET_CONF}.orig"
+  echo "  Backed up original to ${SER2NET_CONF}.orig"
+fi
+
+{
+  echo "%YAML 1.1"
+  echo "---"
+  echo "# ser2net configuration for pfv-tsys4 console ports"
+  echo "# Generated by console/generate-config.sh on $(date)"
+  echo "#"
+  echo "# All ports use telnet(rfc2217) accepter so conman and telnet clients"
+  echo "# negotiate proper telnet binary mode — this prevents CR stripping"
+  printf '%s\n' "# and stair-stepping on devices that send \\n\\r (LF+CR) line endings."
+  echo "# Ports bound to Tailscale IP ($TS_IP) for secure remote access."
+  echo "#"
+  echo "# Direct telnet:  telnet $TS_IP 2001"
+  echo "# Via conman:     conman -f <name>"
+  echo ""
+  printf '%s\n' "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
+  echo ""
+
+  for entry in "${ENTRIES[@]}"; do
+    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
+    # ser2net connection block — telnet(rfc2217) accepter so conman and
+    # telnet clients negotiate proper telnet binary mode. This prevents
+    # CR stripping that occurs with raw TCP + conman's telnet NVT.
+    echo "connection: &con${tcp_port}"
+    echo "    accepter: telnet(rfc2217),tcp,${TS_IP},${tcp_port}"
+    echo "    enable: on"
+    echo "    options:"
+    echo "      banner: *banner"
+    echo "      kickolduser: true"
+    echo "      telnet-brk-on-sync: true"
+    echo "    connector: serialdev,"
+    echo "              /dev/consoles/${name},"
+    echo "              ${baud},local"
+    echo ""
+  done
+} > "$SER2NET_CONF"
+
+echo "  Written: $SER2NET_CONF"
+echo "  ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)"
+
+# ============================================================
+# 3. Write conman console entries directly into conman.conf
+# ============================================================
+# conman 0.3.x does NOT support the 'include' directive, so we write
+# CONSOLE entries directly into /etc/conman.conf between idempotent markers.
+echo ""
+echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---"
+
+# Ensure logdir exists
+mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true
+
+# Ensure LOGDIR is set in conman.conf (server-level directive for log file paths)
+if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
+  # Insert near the top, after the first SERVER directives
+  sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF"
+  echo "  Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
+fi
+
+# Ensure loopback=off so conmand is reachable over Tailscale (not localhost-only)
+if ! grep -qiE '^\s*server\s+loopback\s*=' "$CONMAN_CONF" 2>/dev/null; then
+  sed -i "/^server logdir/a server loopback=off" "$CONMAN_CONF"
+  echo "  Added server loopback=off to $CONMAN_CONF (enables remote access)"
+fi
+
+# Remove any previous auto-generated block (between markers)
+# Then append the new block
+MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
+MARKER_END="# END PFV CONSOLE DEFINITIONS"
+
+# Strip old block if present
+if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then
+  sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF"
+  echo "  Removed previous console definitions."
+fi
+
+# Append new block
+{
+  echo ""
+  echo "$MARKER_BEGIN"
+  echo "# Generated by console/generate-config.sh on $(date)"
+  echo "# Each console connects to a ser2net TCP port via telnet protocol."
+  echo "# ser2net uses telnet(rfc2217) accepter so binary mode is negotiated"
+  echo "# and CR/LF translation is handled correctly by the telnet NVT layer."
+  echo "# Access: conman -f <name>"
+  echo ""
+  for entry in "${ENTRIES[@]}"; do
+    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
+    echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
+  done
+  echo "$MARKER_END"
+} >> "$CONMAN_CONF"
+
+CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0)
+echo "  Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF"
+
+# ============================================================
+# Summary
+# ============================================================
+echo ""
+echo "============================================"
+echo "  Configuration generated successfully."
+echo ""
+echo "  Files written:"
+echo "    $UDEV_RULES       ($(wc -l < "$UDEV_RULES") lines)"
+echo "    $SER2NET_CONF     ($(wc -l < "$SER2NET_CONF") lines)"
+echo "    $CONMAN_CONF      (CONSOLE entries appended between markers)"
+echo ""
+echo "  Next steps:"
+echo "    1. Reload udev:   udevadm control --reload-rules && udevadm trigger"
+echo "    2. Restart ser2net: systemctl restart ser2net"
+echo "    3. Start conman:  systemctl enable --now conmand"
+echo "    4. Or run:       bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
+echo "============================================"
@@ -0,0 +1,29 @@
+# console/mapping.txt — Source of Truth for console port assignments
+#
+# Format: <tcp_port>|<name>|<id_path_substring>|<baud>|<comment>
+#
+# Delimiter is | (pipe) because ID_PATH values contain colons.
+#
+# - tcp_port:     TCP port ser2net listens on (also the conman console name suffix)
+# - name:         Device name (used for /dev/console/<name> symlink, conman console name)
+# - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH`
+#                      These are STABLE across reboots as long as adapters aren't moved
+#                      to different physical USB ports.
+# - baud:         Serial baud rate (9600n81 = 9600 8N1, no flow control)
+# - comment:      Free-form description
+#
+# To RE-MAP after physically moving an adapter:
+#   1. Run: bash console/discover.sh   (find the new ID_PATH for the device)
+#   2. Update the id_path_substring in this file
+#   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
+#
+2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
+2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
+2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
+2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
+2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
+2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
+# 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
+# Unassigned (no device detected):
+# 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
+# 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
--- a/Show More
+++ b/Show More