docs: session cleanup — coverage matrix, questions v2, handoff pointer [#343]

- OAM coverage matrix published on Discourse #309
- questions-v2.md: Q1-Q6 resolved, Q7-Q9 added
- aug11.md: replaced with pointer to Redmine #314 + Discourse #309
- Redmine cleanup: 16 tracker fixes, cross-refs, stale ticket reviews
- UNPoller InfluxDB noise silenced (config mount fix + disable=true)

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

.crush/.gitignore

@@ -0,0 +1,3 @@
+*
+!crush.json
+!.gitignore

.crush/crush.json

@@ -0,0 +1,11 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "^(bash|edit|write|multiedit|lsp_replace_symbol|lsp_rename)$",
+        "command": "./hooks/ticket-gate.sh",
+        "timeout": 5
+      }
+    ]
+  }
+}

.env.example

@@ -0,0 +1,7 @@
+# PFVCluster environment variables
+# Copy to .env and fill in values for local development/testing.
+
+# Pi-hole (netinfra/pihole/docker-compose.yml)
+PIHOLE_WEB_PASSWORD=changeme
+
+# Shellcheck wrapper (tests/shellcheck.sh) — no config needed, uses Docker.

.gitignore

@@ -1,5 +1,9 @@
-# Crush internal state
+# Crush internal state (track crush.json config, ignore session data)
-.crush/
+.crush/crush.db
+.crush/crush.db-*
+.crush/logs/
+.crush/memory/
+.crush/active-ticket
 
 # OS/editor
 .DS_Store
@@ -37,3 +41,6 @@ returned-logs/
 # Kubernetes secrets (kubeconfig contains embedded client certs)
 k8s/kubeconfig.yaml
 k8s/*.token
+
+# Pi-hole web UI password (real value only in on-box .env, never committed)
+netinfra/pihole/.env

AGENTS.md

@@ -1,9 +1,267 @@
 # Agent Guidelines
 
-**Top-level files:** [`README.md`](README.md) (project overview),
+## Agent Authority (NON-NEGOTIABLE)
-[`STATUS.md`](STATUS.md) (living status, agent-maintained),
+
-[`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else
+**No work is permissible on any system without an approved Redmine ticket.
-lives in subdirectories.
+There are no exceptions to this rule.**
+
+### Scope of authority
+
+1. **Tickets govern all work.** The agent performs ONLY the work described in
+   the approved Redmine ticket. Anything outside that scope — no matter how
+   small, helpful, or "obvious" — is prohibited.
+
+2. **No autonomous system changes.** The agent does not modify, configure,
+   create, delete, or grant anything on a production system unless it is
+   explicitly directed by an approved ticket. "Production system" means
+   every system in the fleet — there is no "test" exception unless the ticket
+   says so.
+
+3. **Propose, never implement.** If the agent discovers additional work that
+   should be done — a bug, a misconfiguration, a missing dependency, an
+   enhancement — it does NOT implement it. Instead, it creates a Redmine
+   sub-ticket (status Feedback) describing the finding and surfaces it to
+   the user for approval.
+
+4. **Security and access changes require extra scrutiny.** Changes to sudoers,
+   SSH keys, user accounts, firewall rules, authentication policy, file
+   permissions, or any privilege-related configuration are treated as
+   policy decisions, not implementation details. The agent may suggest
+   them but NEVER implements them without explicit user direction in the
+   ticket or a sub-ticket the user has approved.
+
+5. **The user makes policy. The agent implements policy.** The agent does
+   not decide who gets sudo, what keys go where, what services run, or what
+   the access model is. The agent executes the user's decisions, exactly
+   as specified.
+
+6. **When in doubt, ask.** If the ticket is ambiguous, if a task seems to
+   require something not explicitly authorized, or if the agent is unsure
+   whether an action is in scope — STOP and ask the user via the ticket
+   or directly. Asking is always acceptable. Overstepping is never
+   acceptable.
+
+7. **NEVER close a ticket without explicit user permission.** You may
+   SUGGEST a close when the result is clearly scoped and delivered. If
+   it's ambiguous whether the work is truly complete, don't suggest a
+   close — leave that decision to the user. This applies to ALL tickets,
+   no exceptions.
+8. **User acceptance testing is MANDATORY before declaring work done.**
+   The agent performs implementation and technical validation (services
+   running, configs correct, APIs responding). The user performs UAT —
+   visually confirming dashboards render data, alerts deliver, tools are
+   usable. The agent MUST NOT set done-ratio to 100%, MUST NOT suggest
+   closing, and MUST NOT move to the next ticket until the user explicitly
+   accepts the work. "Technically wired but blank dashboard" is NOT done.
+9. **NEVER access a database directly if an API exists.** APIs are the
+   stable contract; databases change schemas without warning. If a tool
+   has an API, use it — exhaust all API endpoints, check the docs, try
+   alternative methods. Only fall back to direct DB access as a last
+   resort AND with explicit user approval for that specific instance.
+   Soon all DB access will route through a proxy under zero trust;
+   building API-first habits now ensures that transition is clean.
+
+### Access-channel policy: SSH only (NON-NEGOTIABLE)
+
+**The qemu guest-agent is NEVER an access, execution, or key-delivery
+channel.** SSH (`sshd`) is the only approved remote access path. Every
+command must flow through sshd so it is captured by the standard
+auth/logging/audit infrastructure. This is an ITAR/CMMC/TS/SCI
+environment — there is no back-door exception, ever.
+
+- **Forbidden:** `qm guest exec` (runs arbitrary commands inside a guest
+  over an unaudited channel) and any wrapper around it (e.g. a `vm-guest`
+  mode). This includes using guest-agent to *deliver* an SSH key, even if
+  the resulting SSH login is itself audited — the delivery bypassed audit.
+- **Allowed:** installing or checking qemu-guest-agent for its intended
+  purpose — letting Proxmox see guest state (`qm guest cmd <id> ping`,
+  `agent: 1` config, `apt install qemu-guest-agent`). Visibility only;
+  never execution.
+- **Enforced mechanically:** `scripts/check-rules.sh` rule #11 fails on
+  any `qm guest exec` / `vm-guest` pattern in code. `tests/remote.sh` has
+  no guest-exec mode.
+- **If a system is locked out** (no SSH key, no guest-exec path): surface
+  it to the user. Do NOT improvise an alternate back-channel. The user
+  authorizes the unblock method (console login, credential, etc.).
+
+### Ticket-closing policy (NON-NEGOTIABLE)
+
+**NEVER close a Redmine ticket without explicit user permission.** You
+may SUGGEST a close when the result is clearly scoped and delivered. If
+it's ambiguous whether the work is truly complete, don't suggest a close
+— leave that decision to the user. This applies to ALL tickets, no
+exceptions.
+
+### What this means in practice
+
+- Discovered a typo in a config during approved work? **Finish the approved
+  work. Create a sub-ticket for the typo. Do not fix it inline.**
+- Think a system should also have localuser sudo configured? **Do not add
+  it. Propose it in a sub-ticket.**
+- Need to install a package the ticket didn't mention? **Ask first.**
+- Found a security issue? **Create a ticket immediately with full details.
+  Do not remediate without approval.**
+
+This environment operates in ITAR/CMMC/TS/SCI space. Every action must be
+traceable to an approved ticket. There is no "I thought it would help."
+
+## Quick Start
+
+**You are an AI agent working on this project. Your first actions, in order:**
+
+> **SESSION-START GATE (NON-NEGOTIABLE):** Steps 1-3 orient you. Step 4 is the
+> check-for-understanding gate (rule summary). Steps 5-7 gather state. Step 8 is
+> the scope-alignment gate. You MUST NOT begin any task work until the user
+> (a) confirms your rule summary AND (b) names the ticket to work on. This runs
+> **every session, automatically** — the user should never have to ask for it.
+
+1. **Set up the environment:** `bash scripts/setup-hooks.sh` (installs git hooks — idempotent).
+2. **Read this file** (`AGENTS.md`) — project policy and domain knowledge.
+3. **Read the latest questions file** (`questions-v*.md`) — open questions awaiting human input. The version number increments each round (v1, v2, v3...).
+4. **Check for understanding — GATE.** In your own words, summarize ALL the rules
+   back to the user before doing any work: Agent Authority (ticket-governed,
+   propose-never-implement), Access-Channel SSH-only policy, Remote access
+   (remote.sh chokepoint mandatory, DNS names only — never IP literals),
+   Questions policy (no harness question-tools), Documentation policy
+   (Discourse is SoR), Redmine tracking, Git policy (always commit+push,
+   shellcheck), Mandatory infra-change documentation (#298 audit log), SSH
+   routing chokepoints (`tests/remote.sh`), the mechanically enforced
+   `check-rules.sh` rules, and Credentials (Vault migration TODO). Then stop
+   and wait for the user to confirm. This checkpoint guarantees every
+   session starts aligned.
+5. **Check Redmine** — `docker run --rm --env-file ~/.creds/redmine.env git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55` for active work.
+6. **Check current state:** `git log --oneline -10`.
+7. **Run rule audit:** `bash scripts/check-rules.sh --fast`.
+8. **Scope-alignment — GATE.** Present the session handoff's "What's Left"
+   priority list (or the Redmine queue if no handoff exists). Do NOT scan the
+   full ticket queue and pick work on your own — **the user directs what gets
+   worked on, always.** Note any new or urgent items from the Redmine check,
+   then ask which ticket to work on tonight. **STOP and wait.** Do not set
+   `.crush/active-ticket`, do not read systems, do not run diagnostics, until
+   the user names the target. This gate prevents the agent from burning
+   context on work the user didn't ask for.
+
+## Enforcement Model
+
+Git hooks (`scripts/pre-commit`, `scripts/pre-push`) enforce the rules defined in
+`scripts/check-rules.sh`. The rules engine checks: shellcheck (zero warnings
+including info-level), Docker image pinning (no `:latest`), container naming,
+required files, Discourse pointer headers, and more. Run `bash scripts/check-rules.sh`
+for a full audit or `--fast` for pre-commit speed. Bypass with `--no-verify`
+(emergencies only).
+
+## Task Tracking
+
+- **Redmine is the system of record for all work.**
+- **NEVER close a ticket without explicit user permission.** Suggest a
+  close when clearly scoped/delivered; if ambiguous, don't suggest.
+- **Ticket-first enforcement (mechanically enforced).** Before starting
+  any work, set the active ticket: `echo '#NNN' > .crush/active-ticket`.
+  The Crush hook (`hooks/ticket-gate.sh`) blocks modifying operations
+  until this file exists. If no ticket exists, CREATE ONE FIRST via
+  redmine-cli, then set it. Clear when done: `> .crush/active-ticket`.
+- **WORKING.md** is the only in-repo task tracker — a scratchpad for the current
+  session. The pre-commit hook blocks commits while any task remains unchecked.
+- Clear WORKING.md before responding to the user.
+
+## Rolling HUD (session-scoped, NOT persisted)
+
+The rolling HUD is a **live status display** the agent maintains throughout
+the session to help the human follow along. It is NOT a system of record —
+Redmine, Discourse, and git are the durable systems. The HUD exists purely
+for the human's situational awareness during the session.
+
+- **Format:** a compact block shown at the end of each significant response
+  (after completing a step, hitting a blocker, or pivoting). Example:
+
+  ```
+  ┌─ SESSION HUD ────────────────────────────────────────
+  │ Active: #343 (Monitoring coverage matrix)
+  │ Done:   ✓ scope-alignment gate added to both AGENTS.md
+  │         ✓ committed + pushed (6a2550b)
+  │ Now:    drafting coverage matrix on Discourse
+  │ Next:   → #341 TEMPer USB (deploy on pfv-tsys1)
+  │         → #338 LibreNMS alerts (BLOCKED: pushover.env)
+  ├─ USER ACTION ITEMS ──────────────────────────────────
+  │ • Populate ~/.creds/pushover.env (unblocks #338, #428)
+  │ • Populate ~/.creds/prometheus.env + grafana.env (#430)
+  └──────────────────────────────────────────────────────
+  ```
+
+- **Placement:** may be written to `.crush/hud.md` on disk to keep context
+  window smaller (re-read and update rather than hold in memory). Never
+  committed to git. Wiped at session end.
+- **Sections:**
+  - **Active:** current ticket number + one-line description
+  - **Done:** ✓ items completed this session (append as work progresses)
+  - **Now:** what the agent is actively doing
+  - **Next:** the queued items (per handoff priority or user direction)
+  - **User action items:** things ONLY the user can do (populate creds,
+    physical work, manual deploys) with the tickets they unblock
+- **When to show it:** after each logical unit of work, at blockers, and
+    when pivoting between tickets. Not every trivial response — use judgment.
+- **NOT a substitute for Redmine/Discourse/git.** The HUD is ephemeral. When
+    work completes, update the durable systems (ticket notes, Discourse wiki,
+    commits). The HUD just tracks the live narrative for the human.
+
+## Working Style
+
+- **Stop over-thinking.** Get to code and output faster. Explore with code;
+  gather ground truth. Do not burn tokens reasoning about things a quick command
+  answers.
+- **Farm work out to deterministic tooling:** linters, LSPs, formatters, test
+  runners. If an LSP is wired up, use it; otherwise pull a Docker image and lint
+  inside it.
+- **Use sub-agents as subcontractors:** scoped spec in, distilled deliverable out.
+  Never read 10+ files sequentially; batch into agent calls.
+- **Command timeouts (NON-NEGOTIABLE):** Every command that touches a remote
+  system MUST be wrapped with `timeout`. Hard limits: 30s for quick reads
+  (status, ps, ls), 120s for standard operations, 300s for deployments/pulls.
+  If a command hits the timeout, STOP and investigate root cause — never
+  blindly retry. A hung command is a failed command. Detect failure fast,
+  diagnose, fix, move on. Example: `timeout 120 bash tests/remote.sh vm 'cmd'`.
+  This applies to ALL tools — bash, docker, CLIs, sub-agents.
+
+## Questions (NON-NEGOTIABLE)
+
+**NEVER use a harness "question"/"ask user" tool** (structured prompts,
+modal forms, tabbed questions). Banned across every project, every harness.
+They are not portable, not version-controlled, and bypass the git record.
+**All questions go in the current `questions-v(N).md` file** — write the
+question; the human edits the answer inline in the same file. **Version up
+the filename each time answers land** (v1 → v2 → v3...): create
+`questions-v2.md` with resolved Q&A marked, new questions appended. This
+preserves the history of each Q&A round. Synthesize resolved Q&A into
+Discourse (decisions) and Redmine (work items). See `BASELINE-PROMPT.md`
+§10.
+
+## Documentation policy (IMPORTANT)
+
+**Discourse is the canonical source of truth for all knowledge documentation.**
+Knowledge docs (architecture, runbooks, references, audits, policies) have been
+migrated to [community.turnsys.com](https://community.turnsys.com/c/vp-techops)
+as wiki topics in the **VP TechOps** category.
+
+All `.md` files in this repo (except `AGENTS.md` and `LICENSE`) are now
+**pointers** that link to their corresponding Discourse topic. **Do not update
+documentation content in git** — edit the Discourse wiki topic instead. Git
+edit history no longer serves as the documentation changelog; Discourse
+preserves wiki edit history automatically.
+
+Code (scripts, configs, playbooks) still lives in git as the source of truth
+for executables. Only *documentation* moved to Discourse.
+
+## Top-level files
+
+All `.md` files now point to Discourse. The key pointers:
+
+| File | Points to | Discourse topic |
+|------|-----------|-----------------|
+| `README.md` | Project overview | [#296](https://community.turnsys.com/t/296) |
+| `STATUS.md` | Ticket index + infra summary | [#297](https://community.turnsys.com/t/297) |
+| `docs/docmap.md` | Documentation index | [#296](https://community.turnsys.com/t/296) |
+
+Work tracking stays in [Redmine](https://projects.knownelement.com).
 
 ## Repository Layout
 
@@ -15,6 +273,7 @@ k8s/                k3s cluster setup scripts (HA control plane over Tailscale)
 proxmox/            Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
 awx/                Ansible AWX deployment (k3s + AWX Operator)
 tests/              Test suite + VM validation harness + remote.sh SSH chokepoint
+scripts/            Framework: git hooks, rule engine (check-rules.sh), shared lib
 docs/               Server-build docs, docmap index, and archive
 archive/            Historical/superseded code (provisioning -> replaced by KNELIAC project)
 vendor/             Vendored KNELShellFramework
@@ -26,9 +285,27 @@ vendor/             Vendored KNELShellFramework
   `/home/reachableceo/projects/KNELIAC`.
 - **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
   extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
-- **SSH in Crush**: Direct ssh/scp is blocked. Use
+- **Remote access (NON-NEGOTIABLE):** ALL SSH/SCP to ANY host MUST go
-  [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
+  through the chokepoint scripts — [`tests/remote.sh`](tests/remote.sh)
-  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
+  (Proxmox hosts + all VMs) or
+  [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh)
+  (DNS infra hosts: netinfra-01/02, tsrouter, netboot). NEVER call
+  `ssh`/`scp` directly — the harness blocks raw ssh and the command scanner
+  rejects it. There are no exceptions.
+- **DNS names ONLY (NON-NEGOTIABLE):** NEVER use IP address literals
+  (neither LAN nor Tailscale IPs) in any command, script, or config.
+  ALWAYS use DNS names. For Proxmox hosts: `PROX_HOST=<dns-name>`. For VMs:
+  `VM_IP=<dns-name>`. For conman/SNMP/any tool: pass the DNS name. If a
+  DNS name does not resolve, fix it in DNS (Technitium) or consult the
+  [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  Do NOT fall back to IP literals. This rule eliminates the per-session
+  discovery tax of finding the right IP for each host.
+- **How to access a production VM:**
+  1. Look up the DNS name in the [system inventory — Discourse #307](https://community.turnsys.com/t/307).
+  2. `VM_IP=<dns-name> VM_USER=root bash tests/remote.sh vm '<command>'`
+  3. If the name does not resolve from the workstation, use `PROX_HOST=<proxmox-node>`
+     and run `qm guest cmd <vmid> network-get-interfaces` (visibility only —
+     NOT execution) to find the Tailscale DNS name, then access via that.
 
 ## Git Policy
 
@@ -54,38 +331,132 @@ vendor/             Vendored KNELShellFramework
    justification. A script that emits any diagnostic is a protocol violation.
    Non-bash scripts (PHP with `.sh` shebang `#!/usr/bin/php`, etc.) are exempt.
 
-## Automatic Gardening Protocol
+## Redmine Tracking Policy
 
-**Docs and code must be kept in sync.** After any work session, an agent MUST:
+**Redmine is the system of record for all work.** Do not track status,
+checklists, or TODOs in repo files. Use Redmine tickets instead.
 
-1. **Update [`STATUS.md`](STATUS.md)** — reflect completed work, new issues,
+- **URL:** https://projects.knownelement.com
-   changed infrastructure state. This file is human read-only; agents own it.
+- **Version:** Potential to Kinetic Ready (due 2026-09-30)
-2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added,
+- **Project:** Known Element Enterprises - Technology & Facility Services (id 55)
-   removed, or substantively changed, update the table and "Last Reviewed"
+
-   date.
+### Rules
-3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after
+1. **Every piece of work** (feature, fix, deployment, config change) gets a
-   any rename or restructure. Fix all references in the same commit.
+   Redmine ticket. If one doesn't exist, create it.
-4. **Verify new docs are linked** — every new `.md` file must appear in
+2. **Reference tickets in docs and commits** using `[#NNN]` notation.
-   [`docs/docmap.md`](docs/docmap.md) and be linked from at least one other
+   Example: `[#367] Rebuilt k3s control plane after cnode wipe`.
-   doc.
+3. **When work completes**, update the ticket: set done ratio to 100%,
-5. **If a new top-level directory was created, update ALL directory listings:**
+   add a note describing what was done and where the code lives.
-   - [`README.md`](README.md) → "Directory Structure" table
+   **NEVER close a ticket without explicit user permission.** You may
-   - [`AGENTS.md`](AGENTS.md) → "Repository Layout" code block
+   SUGGEST a close when the result is clearly scoped and delivered. If
-   - [`AGENTS.md`](AGENTS.md) → "Key Scripts" table (if the directory has
+   it's ambiguous whether the work is truly complete, don't suggest a
-     an entrypoint script)
+   close — leave that decision to the user.
-   Missing any one of these is a protocol violation.
+4. **Operations Status** lives on Discourse ([topic #297](https://community.turnsys.com/t/297)) — update that wiki topic if the infrastructure summary needs refreshing. The `STATUS.md` file in git is now a pointer only.
-6. **Self-audit before commit.** Before committing, run:
+5. **Link code to tickets** — ticket descriptions and notes should reference
-   ```
+   the relevant file paths in this repo (e.g., `dcinfra/ups/`).
-   grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md
+
-   ```
+### CLI access (read + write)
-   Every new top-level directory must appear in all four files.
+
+Tickets are managed via the `redmine-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the subtask escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/redmine/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+
+# Your queue (project 55):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest list --assigned-to-me -p 55
+
+# Show / create / update / close:
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest show 367
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest create -p 55 -s "Subject" -d "desc"
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest update 367 -n "Done: committed in abc123" --done-ratio 100
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest close 367
+```
+
+Key IDs: project **55** (`technicaloperations`), user **5** (`reachableceo`).
+Statuses: New(1), In Progress(2), Resolved(3,closed), Feedback(4), Closed(5),
+Rejected(6). New subtasks go to **Feedback (4)**. Tracker **3** = Support.
+**Gotcha:** `create` has no `--parent` flag — to make a subtask, use the
+`python-redmine` escape hatch inside the container (see
+`tooling-cli/redmine/AGENTS.md`). Always `show` a ticket before updating it.
+
+Credentials (`REDMINE_URL`/`REDMINE_API_KEY`) live in the centralized store
+at `~/.creds/redmine.env`.
+
+## Documentation Workflow
+
+**Discourse is the source of truth for all knowledge docs.** After any work
+session, an agent MUST:
+
+1. **Update Discourse wiki topics** — if infrastructure facts changed (new
+   VM, IP change, host retired), edit the relevant wiki topic at
+   [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops).
+2. **Update the Operations Status topic** ([#297](https://community.turnsys.com/t/297))
+   if tickets were opened or closed.
+3. **Grep for stale paths in code** — `grep -rn 'old/path' --include='*.sh'`
+   after any rename or restructure. Fix all references in the same commit.
+4. **If a new top-level directory was created**, update:
+   - `AGENTS.md` → "Repository Layout" code block
+   - `AGENTS.md` → "Key Scripts" table (if it has an entrypoint script)
+   - Create a new Discourse wiki topic for any documentation
+5. **Self-audit before commit.** Code changes must be internally consistent.
+   Documentation changes go to Discourse, not git.
+
+### CLI access (read + write)
+
+Wiki topics are managed via the `discourse-cli` container, invoked directly
+with `docker run` (no wrapper script). Full command reference, patterns, and
+the raw-API escape hatch live in the CLI's own `AGENTS.md`
+(`~/projects/KNEL-AIMiddleware/tooling-cli/discourse/AGENTS.md`) — read it
+for anything beyond the basics.
+
+```bash
+# Connection sanity check (run first in any session):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# List VP TechOps topics:
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest ls -c vp-techops
+
+# Show a topic / edit a wiki post (find post id via `show`):
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest show 297
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest update <post_id> -b "new markdown body"
+```
+
+VP TechOps = category **74**. Key topics: #296 (project overview), #297
+(ops status), #298 (audit log). The API user is trust-level 4 but **not
+admin** — admin-only ops (category creation, setting the wiki flag) will
+403; surface those to the user rather than retrying. **Gotcha:**
+`update`/`delete` take a post **id**, not a post number. Never create a new
+topic for an update to existing knowledge — edit the wiki post in place.
+
+Credentials (`DISCOURSE_URL`/`DISCOURSE_API_KEY`/`DISCOURSE_API_USERNAME`)
+live in the centralized store at `~/.creds/discourse.env`.
 
 ## Key Scripts
 
 | Script | Purpose |
 |--------|---------|
+| [`scripts/check-rules.sh`](scripts/check-rules.sh) | Rule audit engine (shellcheck, image pinning, Discourse pointers, required files) |
+| [`scripts/setup-hooks.sh`](scripts/setup-hooks.sh) | Install git hooks (pre-commit, pre-push) |
 | [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
 | [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
+| `redmine-cli` container | Redmine CLI (ticket read/write via `docker run`; see `tooling-cli/redmine/`) |
+| `discourse-cli` container | Discourse CLI (wiki topic read/write via `docker run`; see `tooling-cli/discourse/`) |
+| `dns-cli` container | Technitium DNS CLI (zones, list, add, delete, search, flush; see `tooling-cli/dns/`) |
 | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
 | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
 | [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
@@ -94,15 +465,123 @@ vendor/             Vendored KNELShellFramework
 | [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
 | [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
 | [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
-| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift |
+| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift, audit-vm-disks, audit-guest-io, deploy-tuned-guests |
+
+## Switch Console Access (conman)
+
+Switch configs are pulled via serial console through a conman + ser2net
+stack on pfv-tsys4.
+
+- **conmand server:** pfv-tsys4 (port 7890)
+- **ser2net:** pfv-tsys4, TCP ports 2001-2006
+  (do NOT connect to ser2net directly — it conflicts with conman's
+  persistent sessions; always use the conman client)
+- **Script:** [`proxmox/perf/scripts/conman-console.py`](proxmox/perf/scripts/conman-console.py)
+  — drives console sessions read-only via PTY. No expect/tcl required.
+- **Command files:** [`netinfra/switches/`](netinfra/switches/) — `.cmds` files
+  with switch-specific show commands
+- **Query available consoles:** `conman -d 100.70.77.93 -q`
+
+```bash
+# Pull a switch config (example):
+CONMAN_SERVER=100.70.77.93:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+| Console name | TCP port | Device |
+|--------------|----------|--------|
+| pfv-r5-core-01 | 2001 | Dell PowerConnect 5448 (rack 5 core, mgmt+storage) |
+| pfv-r3-tor-mgmt-01 | 2002 | Dell PowerConnect 5324 (rack 3 mgmt TOR) |
+| pfv-r3-tor-stor-01 | 2003 | Dell PowerConnect 5324 (rack 3 storage TOR) |
+| pfv-rrinfra-rtr | 2004 | Cisco router (rrinfra) |
+| pfv-r2-tor-01 | 2005 | Rack 2 TOR switch |
+| pfv-r6-mgmt-01 | 2006 | Rack 6 management switch |
+
+## Mandatory: Document ALL Infrastructure Changes
+
+**This is non-negotiable. Every infrastructure change (VM config, disk
+cache, network setting, service config, storage migration) MUST be
+documented BEFORE moving to the next task step — not "later" or "at the
+end."**
+
+Required for EVERY infrastructure change:
+
+1. **Redmine ticket** — create one if none exists. Reference as `[#NNN]`.
+2. **Discourse audit log** — reply to topic
+   [#298](https://community.turnsys.com/t/298) with a dated entry (what
+   changed, why, where).
+3. **Discourse relevant wiki topic** — update the architecture/reference
+   topic if the change affects documented infrastructure facts (storage
+   #300, network #299, k8s #305, etc.).
+4. **Git commit** — if code/config changed in the repo, commit + push
+   immediately per the Git Policy above.
+
+## Tooling
+
+- `gh`, `docker`, `jq` available on the workstation.
+- No native shellcheck — use `bash tests/shellcheck.sh` (Docker wrapper
+  `koalaman/shellcheck:stable`). ALL scripts must pass including info-level.
+- For raw API calls not covered by the Redmine/Discourse CLIs, use
+  `python3` inside the CLI Docker containers (escape hatch pattern in
+  `tooling-cli/{discourse,redmine}/AGENTS.md`).
+- `curl`/`wget`/`httpie` may be blocked by some harnesses. Use the CLIs or
+  the python-in-Docker escape hatch for HTTP writes.
+
+## Key Commands Quick Reference
+
+```bash
+# Tests + validation:
+bash tests/run-tests.sh                        # test suite
+bash tests/vm-validation.sh                    # VM validation
+bash tests/shellcheck.sh                       # shellcheck whole repo
+bash tests/shellcheck.sh path/to/*.sh          # shellcheck specific files
+
+# Redmine + Discourse sanity checks (run first in any session):
+docker run --rm --env-file ~/.creds/redmine.env \
+    git.knownelement.com/reachableceo/redmine-cli:latest whoami
+docker run --rm --env-file ~/.creds/discourse.env \
+    git.knownelement.com/reachableceo/discourse-cli:latest whoami
+
+# Proxmox host access (DNS names only — never IPs):
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'qm list'
+PROX_HOST=pfv-tsys5 bash tests/remote.sh prox 'pvesm status'
+
+# Production VM access (DNS names only — never IPs):
+VM_IP=tsys-librenms VM_USER=root bash tests/remote.sh vm 'systemctl status cron'
+
+# DNS infra access:
+bash netinfra/dns-cluster-setup/remote-dns.sh netinfra01-root 'systemctl status docker'
+
+# Switch console (read-only config pull — DNS names only):
+CONMAN_SERVER=pfv-tsys4:7890 python3 proxmox/perf/scripts/conman-console.py \
+  --console pfv-r5-core-01 --cmds netinfra/switches/pfv-r5-core-01.cmds
+```
+
+## Credential Management
+
+- API keys currently in `.env` files under KNEL-AIMiddleware (gitignored).
+- User goal: migrate all keys to Hashicorp Vault
+  (`vault.knownelement.com`). No vault token present yet
+  (`~/.vault-token` missing). Track as high-priority TODO.
 
 ## Key Docs
 
-→ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.**
+→ **All documentation lives on Discourse:**
+  [community.turnsys.com/c/vp-techops](https://community.turnsys.com/c/vp-techops)
+
+→ All `.md` files in this repo are pointers to Discourse topics.
+
+→ **Complete Linux System Inventory:**
+  [Topic #307](https://community.turnsys.com/t/307) — every Linux system
+  (hosts + VMs + physical), with Tailscale IPs, DNS names, SSH access
+  status, and tuned profiles. Reference this for monitoring coverage,
+  access management, and hostname consistency.
 
 ## Project Context
 
 Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
-Production lives on a Cloudron VPS in Reston VA. See
+Production lives on a Cloudron VPS in Reston VA. See the
-[`STATUS.md`](STATUS.md) for current state and
+[Operations Status topic](https://community.turnsys.com/t/297) for the ticket
-[`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) for the fleet report.
+index and infrastructure summary.
+All work is tracked in [Redmine](https://projects.knownelement.com)
+(version: Potential to Kinetic Ready, due 2026-09-30).

Makefile

@@ -0,0 +1,50 @@
+# Makefile — convenience dispatch to scripts/.
+#
+# Not required. The scripts in scripts/ are the real entry points and work
+# standalone. This file just gives you short verbs if you're at a terminal.
+#
+# In Mode 2 (Hermes/OWUI/MCP), agents call the scripts directly or via API —
+# they don't need this file.
+
+# Project-specific overrides for check-rules.sh
+export PROJECT_DOC_EXEMPT ?= AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v1.md BASELINE-PROMPT.md PATTERNS.md
+export PROJECT_DISCOURSE_HOST ?= community.turnsys.com
+
+.PHONY: setup validate fast lint test garden up down status clean help
+
+help: ## Show available targets
+	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
+
+setup: ## Install git hooks
+	@bash scripts/setup-hooks.sh
+
+validate: ## Full rule audit (includes tests)
+	@bash scripts/check-rules.sh
+
+fast: ## Fast rule audit (pre-commit equivalent)
+	@bash scripts/check-rules.sh --fast
+
+lint: ## Lint shell scripts (shellcheck via docker)
+	@docker run --rm -v "$$(pwd):/mnt" koalaman/shellcheck:stable \
+		$$(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print | sed 's|^\./|/mnt/|') || true
+
+test: ## Run the test suite (override per project)
+	@bash scripts/test.sh
+
+garden: ## Doc-sprawl / Discourse-migration report
+	@bash scripts/garden.sh
+
+up: ## Bring up the docker-compose stack
+	@bash scripts/up.sh
+
+down: ## Bring down the docker-compose stack
+	@bash scripts/down.sh
+
+status: ## Show repo status snapshot
+	@echo "== branch =="; git branch --show-current 2>/dev/null || echo "(no branch)"
+	@echo "== last commit =="; git log --oneline -1 2>/dev/null || true
+	@echo "== working tree =="; git status --short 2>/dev/null || echo "(not a git repo)"
+	@echo "== STATUS.md head =="; sed -n '1,12p' STATUS.md 2>/dev/null || echo "(no STATUS.md)"
+
+clean: ## Remove build/test artifacts (override per project)
+	@echo "make clean: nothing to clean — override this in your project's Makefile."

README.md

@@ -1,49 +1,10 @@
-# PFVCluster
+# README.md
 
-Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)**
+> **Project overview, architecture, quick start**
-
+>
-## Directory Structure
+> **Read it here:** https://community.turnsys.com/t/296
-
+>
-| Directory | Description |
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-|-----------|-------------|
+> Do not update content here — edit the Discourse wiki topic instead.*
-| [`dcinfra/`](dcinfra/) | Data-center infrastructure: PDU (`powerman/`), serial console (`console/`), UPS (`ups/`) |
-| [`netinfra/`](netinfra/) | DNS/NTP/DHCP setup + audit scripts, DNS cluster replication (`dns-cluster-setup/`), switch captures (`switches/`), DHCP config (`dhcp/`) |
-| [`k8s/`](k8s/) | k3s cluster setup scripts (HA control plane over Tailscale) + co-located docs |
-| [`proxmox/`](proxmox/) | Proxmox fleet docs (hardware audit, capacity, storage, k8s host planning) + performance tuning (`perf/`) |
-| [`awx/`](awx/) | Ansible AWX deployment (k3s + AWX Operator) |
-| [`tests/`](tests/) | Test suite + VM validation harness + `remote.sh` SSH chokepoint |
-| [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive |
-| [`archive/`](archive/) | Historical/superseded code (provisioning → replaced by KNELIAC project) |
-| [`vendor/`](vendor/) | Vendored KNELShellFramework |
-
-## Quick Start
-
-### Provision a new server
-```bash
-sudo bash provisioning/SetupNewSystem.sh
-```
-
-### Validate on the sandbox VM
-```bash
-VM_ID=6000 ./tests/vm-validation.sh all
-```
-
-### Deploy DNS cluster
-```bash
-cd dns-cluster-setup/ && ./setup.sh all
-```
-
-### Deploy perf tunings
-```bash
-cd perf/ && ./deploy-tuning.sh
-```
-
-## Architecture
-
-- **Proxmox hosts**: 7 standalone PVE installs managed via PDM
-- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
-- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
-- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
-- **Backups**: Proxmox Backup Server (PBS)

STATUS.md

@@ -1,224 +1,10 @@
-# Project Status
+# STATUS.md
 
-> **Human read-only. Agents maintain this file automatically after each work
+> **Documentation moved to Discourse — the canonical source of truth.**
-> session.** Do not edit by hand — the next agent run will overwrite it.
-
-> **Last updated:** 2026-08-05 by Crush (GLM-5.2) — full read-only ground-truth
-> re-orientation (nmap + SNMP + SSH + API queries).
 >
-> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)
+> **Ticket index + infrastructure summary**
-> — repo restructure, 4 switches up / 1 down, 63 VMs, 167 DNS zones, 753 DHCP
+>
-> leases, k8s cluster DOWN (cnode wiped), netinfra-02 wiped, tsys5 storage
+> **Read it here:** https://community.turnsys.com/t/297
-> configured, tsys4 upgraded, 3-tier Tailscale fleet (80 nodes).
+>
-> Capacity targets are codified in
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
+> Do not update content here — edit the Discourse wiki topic instead.*
-> steady-state (fleet PASSES, highest tsys1 at 71%).
-
-## Current State: MIXED (infra stable; k8s + netinfra-02 need rebuild)
-
-Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack
-are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum)
-and **netinfra-02 is wiped** (needs rebuild as redundant secondary). Repo was
-restructured into a module/subproject layout (see
-[`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)).
-
-## Completed Work
-
-### Server Provisioning (validated on sectestbed-sandbox)
-- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
-- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
-- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
-- [x] DNS resolv.conf: managed static file pointing at netinfra pair
-- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
-- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
-
-### DNS Cluster (pfv-netinfra-01/02)
-- [x] Production Technitium config replicated from tailscale-router (read-only)
-- [x] 124 zones on both nodes (knel.net + reverse DNS)
-- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
-- [x] Zone replication via rsync systemd timer (every 60s)
-- [x] Credentials + 2FA replicated identically to production
-- [x] Both LAN IPs resolve knel.net device names + recurse externally
-- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, **753 active leases** (was 333). DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
-- [ ] **netinfra-02 WIPED 2026-08-05** (storage move accident). Reachable on
-  Tailscale (100.71.171.20) but SSH key auth fails — needs full rebuild as
-  redundant secondary (key deploy + zone sync + DHCP failover-peer).
-- [x] **Technitium now runs as Docker container** (`tsys-dns`) on netinfra-01,
-  not systemd dnsServer. Config at `/home/localuser/services/technitium/config`.
-  **167 zones** (was 124). NTP = `ntpsec` (not chrony).
-- [x] **Pi-hole `/dev/shm` = 1.0 GB** (operator hotfix, was too small → prod
-  outage). Healthcheck = `dig pi.hole` (DNS-based; gravity.db-specific check
-  not yet added).
-
-### Proxmox Cluster Ops
-- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
-- [x] **Full ground-truth audit 2026-07-30** (network + storage + compute + switches)
-- [x] **Cross-rack NFS root cause identified:** core-sw01 LAG hash=layer-2-3 (no L4),
-  g16/g17 cable mismatch (3 of 4 links active), static mode (no LACP)
-- [x] **tsys4/tsys5 storage changes documented:** new NVMe (CT500P1 500GB), relocated
-  SK hynix SSD, D3 changed to USB HDD, /mnt/albert + /mnt/backup added
-- [x] **Console symlink bug fixed:** udev rule singular→plural mismatch broke all 7
-  console ports after reboot
-- [x] **Portable audit tooling built:** probe-storage.sh, probe-network.sh,
-  conman-console.py, snmp-switch-audit.py (AGPLv3-friendly, config-driven)
-- [ ] **Memory goal (≤80% actual): PASS fleet-wide** — highest is tsys9 at 69%
-- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
-- [ ] tsys4: PVE 9.1.5 (behind fleet 9.2.5), needs upgrade
-- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
-- [ ] tsys2: pending rebuild from Win10 to Proxmox
-
-### Kubernetes Control Plane (k3s HA — DOWN, needs rebuild)
-- [ ] **Cluster DOWN 2026-08-05**: operator wiped a cnode + shut cnodes down
-  (100% RAM on one). cnode2 has k3s API up, cnode1/cnode3 do not — no etcd
-  quorum. Kubeconfig points to stale IP (100.97.178.106, matches no cnode).
-- [x] Was: 3-node k3s HA control plane (cnode1/2/3, embedded etcd, Tailscale)
-- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
-- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
-- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
-  Tailscale IPs, CoreDNS, API server, workload isolation
-- [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
-- [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
-- [ ] Workers (wnodes) not yet joined to this cluster
-- [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
-  preserved in `k8s/docs/` for future ITAR/classified cluster.
-
-### PDU Management (powerman on pfv-tsys1 — LIVE)
-- [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
-- [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink
-  `/dev/cyclades-pm10`
-- [x] powermand listening on `127.0.0.1:10101` + Tailscale `100.121.189.98:10101`
-- [x] All 10 outlets defined as `outlet-1` through `outlet-10`
-- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
-- [ ] Rename outlets to match physical devices (Friday onsite)
-
-### UPS Management (NUT on pfv-tsys1 — LIVE)
-- [x] **APC Smart-UPS C 1500** fully reporting via NUT usbhid-ups + APC HID 0.100
-- [x] upsd listening on Tailscale `100.121.189.98:3493` (port 3493)
-- [x] Local upsmon (master) configured for graceful hypervisor shutdown
-- [x] Home Assistant credentials generated (`homeassistant` user in upsd.users)
-- [x] Scripts in `ups/`: discover, setup (idempotent), status
-- [x] Home Assistant NUT integration **configured via REST API** (`ups/setup-ha-nut.sh`)
-- [x] **Key fix:** upsd now listens on LAN IP (192.168.3.11) in addition to
-  Tailscale — HAOS runs Tailscale as an isolated add-on, so the HA container
-  cannot route to Tailscale IPs. Shared vmbr0 bridge provides instant LAN path.
-- [x] **Key fix:** udev rule covers BOTH `usb` and `hidraw` subsystems (driver
-  opens raw `/dev/bus/usb` after setuid drop, not `/dev/hidraw`)
-- [ ] **Tripp Lite UPS blocked:** driver matches TrippLite HID 0.85 subdriver
-  but fails reading 878-byte HID Report Descriptor (EAGAIN via USB hub).
-  Needs physical reseat to direct motherboard USB port or cable replacement.
-  Driver masked to prevent restart-loop spam.
-- [ ] **Home Assistant NUT integration:** configured and live (3 sensors:
-  battery charge, status, status data). This APC (FW 02.2, mfg 2012) does not
-  report `ups.load` over USB HID — confirmed via both NUT and apcupsd tests.
-  No wattage for Energy Dashboard; smart plug on UPS output is the practical
-  path. apcupsd package conflicts with nut-server (mutually exclusive).
-
-### Console Management (ser2net telnet(rfc2217) + conman on pfv-tsys4 — LIVE)
-- [x] 7 network switch/router consoles managed via ser2net + conman
-- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
-  ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
-  symlinks that survive reboot regardless of enumeration order
-- [x] ser2net exposes all 7 consoles on telnet(rfc2217) TCP ports (2001-2007)
-  bound to **Tailscale IP only** (`100.70.77.93`)
-- [x] conman connects to TCP ports for logging + multiplexing (7 log
-  files active in `/var/log/conman/`)
-- [x] Both ser2net + conmand enabled via systemd (survive reboot)
-- [x] conmand systemd unit created (Debian package doesn't ship one)
-- [x] conmand remote access enabled (loopback=off, port 7890)
-- [x] Old `/root/conmap` + manual `screen` workflow replaced
-- [x] **Stair-stepping fix:** ser2net accepter changed from raw `tcp` to
-  `telnet(rfc2217)` so conman's telnet NVT negotiates binary mode properly
-- [x] **Validated 6/7 consoles working** via conman from workstation
-  (core-sw01, tor3-mgmt, tor3-stor, rrinfra-rtr, r2-tor-top, subodev-torsw)
-- [ ] **pfv-r2-sw (port 2007): DEAD** — zero bytes at all baud rates
-  (1200-115200), adapter present and mapped correctly (/dev/ttyUSB3).
-  Physical issue: device powered off, console cable disconnected, or
-  console port disabled in device config. Requires onsite investigation.
-
-### Ansible AWX (tsys-awx — LIVE)
-- [x] AWX 24.6.1 deployed via AWX Operator 2.19.1 on k3s single-node
-- [x] Accessible at `http://tsys-awx.knel.net` (port 80, LoadBalancer via k3s ServiceLB)
-- [x] Admin login: `admin` / `REDACTED_PASSWORD`
-- [x] PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
-- [x] VM resized: 32→60 GB disk, swap converted to swapfile
-- [x] Scripts in `awx/`: install-k3s, deploy-awx, verify-awx
-- [ ] HTTPS/TLS not yet configured (currently HTTP on port 80)
-- [ ] Execution environments not yet customized
-
-### Repo Merge
-- [x] KNELServerBuild merged into PFVCluster (history preserved)
-- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
-- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
-
-## Known Issues
-
-| Issue | Impact | Status |
-|-------|--------|--------|
-| **core-sw01 LAG hash = layer-2-3** | NFS cross-rack capped at 1 Gbps per host pair | Fix: change to layer-2-3-4 (switch change, pending approval) |
-| **core-sw01 ch1 g16/g17 mismatch** | Only 3 of 4 cross-rack links active | Fix: change member list g(13-15,17)→g(13-16) |
-| **Cross-rack trunk uses `mode on`** | No LACP failure detection | Fix: change to `mode active` (both switches) |
-| **tsys5 bond0 degraded** (1 of 2 slaves) | Single link, no redundancy | Physical: 2nd ethernet cable needed |
-| **tsys5 NVMe + SSD not configured** | Best local storage unused | Config: mount + add to Proxmox storage.cfg |
-| **D2 carries 13 running VMs** | IOPS hotspot persists | Migrate to S2/S3/new disks |
-| **tsys4 PVE 9.1.5** | Behind fleet (9.2.5) | Upgrade needed |
-| **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Offload to new SK hynix SSD |
-| **Port errors on core-sw01** | g9:2902, g38:590, g26:73 inErr | Investigate CRC/cable |
-| **tsys5 is single-CPU** | CPU2 socket unpopulated | Consider seating spare LGA771 Xeon |
-| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; grow via workloads |
-
-## Pending (next session priorities)
-
-> **Full audit with findings + recommendations is in**
-> [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md).
-
-**Awaiting operator review of AUDIT-2026-07-30 before executing any fixes.**
-
-**Switch changes (need approval):**
-- Fix core-sw01 ch1 member: g(13-15,17) → g(13-16) [restores 4th link]
-- Fix core-sw01 LAG hash: layer-2-3 → layer-2-3-4 [unlocks multi-link NFS]
-- Convert trunk to LACP: mode on → mode active [failure detection]
-- Remove stale tsys8 config from tor3-stor (ch4/g13/g14)
-
-**Host-side changes (no switch access needed):**
-- Mount + configure tsys5 NVMe (CT500P1 500GB) as LVM-thin
-- Mount + configure tsys5 SK hynix SSD (512GB, relocated from tsys4)
-- Clean stale S3/S4 exports + storage.cfg on tsys5
-- Upgrade tsys4 to PVE 9.2.5 / kernel 7.0.14
-- Apply apply-tunings.sh to tsys4/5 (nconnect, bond hash, rmem/wmem)
-- Audit VM cache modes (writethrough → writeback for NFS)
-
-**Other:**
-- **Schedule:** Operator onsite Monday 0700. Until then: software fixes,
-  documentation, and OAM only (no physical access). Sunday off. Saturday =
-  OAM Day (2026-08-01).
-- **Remote-capable (now–Saturday):** tsys5 NVMe/SSD mount+config, fleet tuning
-  (tsys4/5 nconnect/bond hash/rmem), VM config audit, OAM Day prep, documentation
-- **Onsite-only (Monday+):** switch config changes (core-sw01 ch1 hash+member,
-  tor3-stor LACP conversion), tsys5 2nd ethernet cable, physical hardware work
-- **Saturday OAM Day (2026-08-01):** 100% instrumentation coverage — Beszel
-  (metrics) on all hosts, Uptime Kuma (up/down) → Pushover alerts, LibreNMS
-  (SNMP) for switches, Home Assistant (power/temp) for physical infra,
-  centralized syslog (tool TBD)
-- Investigate core-sw01 port errors (g9/g38/g26)
-- k8s: join wnodes, label by workload class
-- k8s deferred: ETL tooling, HPC scheduler, vcluster policy
-
-## Infrastructure Summary
-
-| Component | Details |
-|-----------|---------|
-| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) |
-| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + new NVMe unconfigured) |
-| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3), + 1 unidentified (192.168.0.8) |
-| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
-| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
-| DNS production | tailscale-router (read-only source of truth) |
-| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
-| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
-| Backup | Proxmox Backup Server (PBS) |
-| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
-| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
-| **AWX** | **tsys-awx: AWX 24.6.1 on k3s (http://tsys-awx.knel.net:80)** |
-| **Console** | **7 switch consoles via ser2net telnet(rfc2217)+conman on pfv-tsys4 (conmand:7890 on Tailscale)** |
-| **UPS** | **APC Smart-UPS C 1500 via NUT on pfv-tsys1 (upsd:3493 on Tailscale)** |
-| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |

WORKING.md

@@ -0,0 +1,11 @@
+# WORKING.md — Active Session Tracker
+
+Agent work only. The human decides when it's done.
+A commit is blocked while any task below remains unchecked.
+
+## Current Tasks
+
+(all done — session complete)
+
+- [x] Add ticket-gate Crush hook (blocks work without active ticket)
+- [x] Document active-ticket workflow in AGENTS.md + meta template

access-matrix.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/bash
+# access-matrix.sh — definitive access verification across all Linux Tailscale nodes.
+# Uses the correct SSH user(s) per system type, checks sudo where applicable.
+# Routes through remote.sh (the only allowed ssh path).
+set -u
+cd /home/reachableceo/projects/PFVCluster || exit 1
+
+# Policy-excluded systems (never attempt access)
+EXCLUDE=':tsys-cloudron:pfv-bms:tsys-umbrel:tsys-ucs-01:tsys-ucs-02:stlpc-bizoffice:ultix-highside:'
+
+# Determine the SSH user(s) for a given hostname and whether sudo is expected.
+# Returns "user1:user2:...:sudoflag" where sudoflag is "yes" or "no".
+users_for() {
+  local name="$1"
+  case "$name" in
+    pfv-tsys[0-9])                   echo "root:no" ;;
+    *-proxmox-datacenter)            echo "root:no" ;;
+    *-proxmox-pve)                   echo "root:no" ;;
+    *-proxmox-pbs)                   echo "root:no" ;;
+    *-proxmox-mailgw*)               echo "root:no" ;;
+    *-proxmox-backup*)               echo "root:no" ;;
+    tsys-ucs-*)                      echo "root:no" ;;
+    ultix-streaming)                 echo "root:no" ;;
+    stlpc-*)                         echo "root:labuser:no" ;;
+    ultix-field)                     echo "ultixfield:yes" ;;
+    subopi*)                         echo "subodev:yes" ;;
+    *)                               echo "localuser:yes" ;;
+  esac
+}
+
+check_user() {
+  local ip="$1" user="$2" expect_sudo="$3"
+  local out sudo
+  out=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'echo SSHOK; id -un' </dev/null 2>&1 | grep -oE 'SSHOK|keyboard-interactive|Connection refused' | head -1)
+  case "$out" in
+    SSHOK)
+      if [ "$expect_sudo" = "yes" ]; then
+        sudo=$(VM_IP="$ip" VM_USER="$user" bash tests/remote.sh vm 'sudo -n true 2>/dev/null && echo SUDOOK || echo SUDONO' </dev/null 2>&1 | grep -oE 'SUDOOK|SUDONO' | head -1)
+        printf '%s(%s)' "$user" "${sudo:-?}"
+      else
+        printf '%s(ok)' "$user"
+      fi
+      ;;
+    keyboard-interactive) printf '%s(2FA)' "$user" ;;
+    'Connection refused')  printf '%s(NOSSH)' "$user" ;;
+    *)                      printf '%s(NOKEY)' "$user" ;;
+  esac
+}
+
+printf '%-32s %-16s %s\n' "NAME" "TS-IP" "ACCESS"
+printf '%-32s %-16s %s\n' "----" "-----" "------"
+
+tailscale status 2>/dev/null | awk '$4=="linux" {print $2, $1}' | sort | while read -r name ip; do
+  [ -n "$name" ] || continue
+  case "$EXCLUDE" in *":$name:"*) printf '%-32s %-16s %s\n' "$name" "$ip" "EXCLUDED"; continue;; esac
+
+  map=$(users_for "$name")
+  expect_sudo="${map##*:}"
+  users="${map%:*}"
+  result=""
+  IFS=':' read -ra user_list <<< "$users"
+  for u in "${user_list[@]}"; do
+    r=$(check_user "$ip" "$u" "$expect_sudo")
+    [ -z "$result" ] && result="$r" || result="$result $r"
+  done
+  printf '%-32s %-16s %s\n' "$name" "$ip" "$result"
+done

agent-bootstrap.sh

@@ -0,0 +1,60 @@
+#!/bin/sh
+# agent-bootstrap.sh
+#
+# Run INSIDE a guest (via SSH, console, or guest-agent) to bring the
+# system fully under agent management in one shot:
+#   1. install + enable qemu-guest-agent  (VMs only, skipped on bare metal)
+#   2. push the agent SSH key to root + AGENT_USER (+ labuser if present)
+#   3. grant AGENT_USER passwordless sudo
+#
+# AGENT_USER defaults to "localuser". Override for systems with a different
+# unprivileged agent user:
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+#
+# After this runs once, the agent has SSH+sudo immediately.
+#
+# Usage (from a root shell in the guest):
+#   bash agent-bootstrap.sh
+#   AGENT_USER=subodev bash agent-bootstrap.sh
+
+set -eu
+
+KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIWms/uCXnjjo4KyxHBcYI2TDHe8OZ2wle6W/0hSRQLu reachableceo@ultix-streaming'
+AGENT_USER="${AGENT_USER:-localuser}"
+
+# 1. guest-agent (skip on bare metal — no virtio-serial device)
+if command -v systemd-detect-virt >/dev/null 2>&1 && \
+   [ "$(systemd-detect-virt --vm 2>/dev/null || echo none)" != "none" ]; then
+  if ! command -v qemu-ga >/dev/null 2>&1; then
+    if command -v apt-get >/dev/null 2>&1; then
+      apt-get update
+      DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
+    elif command -v dnf >/dev/null 2>&1; then
+      dnf install -y qemu-guest-agent
+    elif command -v yum >/dev/null 2>&1; then
+      yum install -y qemu-guest-agent
+    else
+      echo "WARN: no supported package manager; skipping agent install" >&2
+    fi
+  fi
+  systemctl enable --now qemu-guest-agent 2>/dev/null || \
+    systemctl enable --now qemu-ga 2>/dev/null || true
+fi
+
+# 2. SSH key for root + AGENT_USER + labuser (if present)
+for u in root "$AGENT_USER" labuser; do
+  if ! getent passwd "$u" >/dev/null 2>&1; then continue; fi
+  H=$(getent passwd "$u" | cut -d: -f6)
+  mkdir -p "$H/.ssh"; chmod 700 "$H/.ssh"
+  AK="$H/.ssh/authorized_keys"; touch "$AK"; chmod 600 "$AK"
+  grep -qF "$KEY" "$AK" || echo "$KEY" >> "$AK"
+  chown -R "$u": "$H/.ssh"
+done
+
+# 3. passwordless sudo for AGENT_USER only
+if getent passwd "$AGENT_USER" >/dev/null 2>&1 && [ -d /etc/sudoers.d ]; then
+  echo "${AGENT_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/010-agent
+  chmod 440 /etc/sudoers.d/010-agent
+fi
+
+echo BOOTSTRAP-DONE

aug11.md

@@ -0,0 +1,9 @@
+<!-- Discourse: https://community.turnsys.com/t/298 -->
+<!-- Redmine: https://projects.knownelement.com/issues/314 -->
+
+# Session Handoff: 2026-08-11/12 OAM
+
+**Full handoff lives in Redmine #314** (session summary note, 2026-08-12).
+**OAM coverage matrix:** Discourse [#309](https://community.turnsys.com/t/309).
+**Architecture rules:** Discourse [#303](https://community.turnsys.com/t/303).
+**Open questions:** `questions-v2.md` in this repo.

awx/README.md

@@ -1,96 +1,10 @@
-# AWX — tsys-awx.knel.net
+# awx/README.md
 
-Ansible AWX deployment on a dedicated k3s single-node VM.
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-**[→ Deployment Plan](DEPLOYMENT.md)** · **[→ Status](../../STATUS.md)**
+> **Ansible AWX deployment on k3s**
-
+>
-## Access
+> **Read it here:** https://community.turnsys.com/t/302
-
+>
-| | |
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-|---|---|
+> Do not update content here — edit the Discourse wiki topic instead.*
-| URL | `http://tsys-awx.knel.net` (LAN) or `http://100.91.39.53` (Tailscale) |
-| Port | 80 (LoadBalancer via k3s ServiceLB) |
-| User | `admin` |
-| Password | `REDACTED_PASSWORD` (fleet standard) |
-
-## Architecture
-
-```
-tsys-awx (VMID 600 on pfv-tsys6)
-├── k3s v1.36.2+k3s1 (single-node, no Traefik)
-├── AWX Operator 2.19.1 (manages the AWX lifecycle)
-└── AWX instance "tsys-awx"
-    ├── AWX web (UI/API)
-    ├── AWX task (Celery workers)
-    ├── PostgreSQL 15 (bundled, 8 GiB PVC on local-path)
-    ├── Redis (ephemeral)
-    └── ServiceLB (binds port 80 to host IP)
-```
-
-## VM Resources
-
-| Resource | Value |
-|----------|-------|
-| Host | pfv-tsys6 (2× Xeon E5530, 126 GB RAM) |
-| vCPU | 4 (2 sockets × 2 cores) |
-| RAM | 12 GB |
-| Disk | 60 GB (D5 on tsys4 NFS) |
-| OS | Debian 13 trixie |
-| Network | 192.168.3.200 (LAN), 100.91.39.53 (Tailscale) |
-
-## Files
-
-| File | Purpose |
-|------|---------|
-| [`install-k3s.sh`](install-k3s.sh) | Install k3s single-node (no Traefik) |
-| [`deploy-awx.sh`](deploy-awx.sh) | Deploy AWX Operator 2.19.1 + AWX instance |
-| [`verify-awx.sh`](verify-awx.sh) | Verify deployment status + access |
-| [`namespace.yaml`](namespace.yaml) | Kubernetes namespace manifest |
-| [`awx-instance.yaml`](awx-instance.yaml) | AWX Custom Resource (CR) manifest |
-
-## Deployment (from scratch)
-
-```bash
-# 1. Push scripts to VM
-for f in awx/install-k3s.sh awx/deploy-awx.sh awx/verify-awx.sh \
-         awx/namespace.yaml awx/awx-instance.yaml; do
-    VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-        "sudo tee /tmp/awx/$(basename $f) > /dev/null" < "$f"
-done
-
-# 2. Install k3s
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "sudo bash /tmp/awx/install-k3s.sh"
-
-# 3. Deploy AWX
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "sudo bash /tmp/awx/deploy-awx.sh"
-
-# 4. Verify
-VM_IP=100.91.39.53 VM_USER=localuser bash tests/remote.sh vm \
-    "bash /tmp/awx/verify-awx.sh"
-```
-
-## Operating
-
-```bash
-# kubectl access
-sudo k3s kubectl -n awx get pods
-sudo k3s kubectl -n awx logs -f deployment/awx-operator-controller-manager
-
-# Restart AWX instance
-sudo k3s kubectl -n awx delete awx tsys-awx  # operator will recreate
-
-# Get admin password
-sudo k3s kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 -d
-```
-
-## Known Issues
-
-1. **kube-rbac-proxy image**: The AWX Operator 2.19.1 references
-   `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` which was removed from GCR.
-   The deploy script patches this to `quay.io/brancz/kube-rbac-proxy:v0.15.0`.
-2. **Image pulls are slow**: First deployment pulls ~2 GB of container images.
-   Postgres image alone is ~400 MB and takes several minutes.
-3. **Disk space**: AWX + k3s uses ~10 GB of the 60 GB disk. PostgreSQL data
-   grows over time — monitor `df -h /`.

bootstrap-all.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/bash
+# bootstrap-all.sh — push agent SSH key + passwordless sudo to remaining systems.
+#
+# SSH is the ONLY approved access channel (see AGENTS.md "Access-channel
+# policy: SSH only"). This script reaches systems that still allow password
+# auth over sshd. Systems that reject password auth (publickey-only) cannot
+# be reached this way — see the CONSOLE-ONLY section printed at the end.
+#
+# Two escalation methods:
+#   sudo → Ubuntu-style systems (no root pw; localuser has sudo)
+#   su   → Debian-style systems (root has a password)
+#
+# Passes AGENT_USER so agent-bootstrap.sh targets the correct unprivileged
+# user. You enter passwords interactively. Idempotent: safe to re-run.
+#
+# Verified state (access-matrix.sh, 2026-08-10): 68/70 non-excluded systems
+# at intended access state. Only tsys-siem remains below.
+set -u
+cd "$(dirname "$0")" || exit 1
+
+SCRIPT=agent-bootstrap.sh
+SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10)
+
+run_with_sudo() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (sudo, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "sudo AGENT_USER=${agent_user} bash /tmp/$SCRIPT" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+run_with_su() {
+  local name="$1" ip="$2" user="$3" agent_user="${4:-localuser}"
+  echo "========================================"
+  echo "  $name ($ip) — $user (su, agent=${agent_user})"
+  echo "========================================"
+  scp "${SSH_OPTS[@]}" "$SCRIPT" "${user}@${ip}:/tmp/" \
+    && ssh -t "${SSH_OPTS[@]}" "${user}@${ip}" "su -c 'AGENT_USER=${agent_user} bash /tmp/$SCRIPT'" \
+    && echo "  -> $name DONE" \
+    || echo "  -> $name FAILED"
+  echo
+}
+
+# All password-auth-reachable systems have been bootstrapped.
+# Verified state (access-matrix.sh, 2026-08-10): 69/70 non-excluded systems
+# at intended access state. The only remaining NOKEY (stlp-3dscanner) is
+# deferred to [#417] and requires a rename + bring-online first — out of
+# scope here. Run `access-matrix.sh` to re-verify at any time.
+echo "All password-auth-reachable systems are bootstrapped."
+echo "Remaining gap: stlp-3dscanner (deferred to [#417])."
+echo "Run access-matrix.sh to re-verify."
+echo "Deferred (separate ticket):"
+echo "  stlp-3dscanner — rename + bring online first [#417]"
+echo "========================================"
+echo "By design (leave alone):"
+echo "  sectestbed-sandbox — 2FA enforced"
+echo "========================================"
+echo "Excluded by policy (no SSH access):"
+echo "  pfv-bms (API), tsys-cloudron (prod revenue),"
+echo "  tsys-ucs-01/02 (API-managed), tsys-umbrel (treasury)"
+echo "========================================"

dcinfra/console/README.md

@@ -1,130 +1,10 @@
-# Console Management (ser2net + conman)
+# dcinfra/console/README.md
 
-Network-accessible serial console management for all production network
+> **Documentation moved to Discourse — the canonical source of truth.**
-switches and routers, running on **pfv-tsys4** (storage server).
+>
-
+> **Serial console management (ser2net + conman)**
-## Architecture
+>
-
+> **Read it here:** https://community.turnsys.com/t/301
-```
+>
-USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-```
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-ser2net owns the physical serial devices and exposes them on TCP ports
-using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
-only** (`100.70.77.93:200X`). conman connects to those TCP ports via
-telnet for session logging, output capture, and multi-user console
-sharing.
-
-**Why telnet(rfc2217)?** The serial devices send `
-␍` (LF+CR) line
-endings instead of standard `
-`. Raw TCP transport caused conman's
-telnet NVT to strip bare CR characters, producing stair-stepped output.
-With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
-translation is handled correctly by the telnet layer.
-
-**conman and ser2net do NOT share ports** — only one process can open a
-serial device at a time. ser2net owns the physical device; conman connects
-over TCP.
-
-## The USB Enumeration Problem (SOLVED)
-
-The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
-unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
-enumeration order, which shifts on every boot. This made the old
-`/root/conmap` + manual `screen` workflow break after every reboot.
-
-**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
-topology), which is stable across reboots regardless of enumeration order.
-Each adapter gets a named symlink in `/dev/consoles/` that never changes.
-
-The udev rules are generated from `mapping.txt`, which maps each adapter's
-ID_PATH to a console name and TCP port. To re-map after physically moving
-an adapter, update `mapping.txt` and re-run `setup.sh`.
-
-**Fallback:** if udev trigger doesn't create symlinks for already-discovered
-devices (common on first run), `setup.sh` creates them manually by matching
-ID_PATH. On subsequent boots, udev creates them automatically.
-
-## Port Assignments
-
-| TCP Port | Console Name | ID_PATH | Description |
-|----------|-------------|---------|-------------|
-| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
-| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
-| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
-| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
-| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
-| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
-| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
-
-All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
-
-## Scripts
-
-| Script | Purpose |
-|--------|---------|
-| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
-| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
-| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
-| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
-
-## Usage
-
-### Connect to a console
-
-**Primary method — conman client (with logging + multiplexing):**
-
-```bash
-# From any Tailscale-connected workstation:
-conman -d pfv-tsys4:7890 -f pfv-core-sw01     # connect to console
-conman -d pfv-tsys4:7890 -q                    # list all consoles
-```
-
-Escape sequence: `&.` to disconnect, `&?` for help.
-
-**Direct telnet (emergency only — conflicts with conman):**
-
-```bash
-# Direct telnet to ser2net works ONLY when conmand is stopped, because
-# conmand maintains persistent connections to all 7 TCP ports. Use:
-ssh pfv-tsys4 'systemctl stop conmand'
-telnet pfv-tsys4 2001                          # pfv-core-sw01
-ssh pfv-tsys4 'systemctl start conmand'        # restart when done
-```
-
-**Do NOT use telnet while conmand is running** — conmand will reconnect
-and kick your telnet session immediately ("Connection closed by foreign host").
-The correct workflow is conman client → conmand → ser2net → device.
-
-### Re-deploy after changing mapping.txt
-
-```bash
-PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
-```
-
-### Find the ID_PATH for a new adapter
-
-```bash
-PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
-```
-
-Then match the new adapter's ID_PATH to its physical location and add a line
-to `mapping.txt`.
-
-## Files on pfv-tsys4
-
-| File | Purpose |
-|------|---------|
-| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
-| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
-| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
-| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
-| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
-| `/root/console/setup.sh` | Setup script (re-runnable) |
-| `/root/console/generate-config.sh` | Config generator |
-
-## Old workflow (replaced)
-
-The old `/root/conmap` file and manual `screen` sessions are no longer
-needed. The new setup is fully automated and survives reboots.

dcinfra/console/generate-config.sh

@@ -6,7 +6,7 @@
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
-#      to a named symlink like /dev/consoles/pfv-core-sw01
+#      to a named symlink like /dev/consoles/pfv-r5-core-01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing

dcinfra/console/mapping.txt

@@ -17,13 +17,13 @@
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
-2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
+2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
-2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
+2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
-2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
+2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
-2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
+2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
-2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
+2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
-2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
+# 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare

dcinfra/console/query-remote.sh

@@ -5,7 +5,7 @@
 #
 # Usage:
 #   bash console/query-remote.sh                    # list consoles
-#   bash console/query-remote.sh pfv-core-sw01      # connect to a console
+#   bash console/query-remote.sh pfv-r5-core-01      # connect to a console
 #
 set -euo pipefail
 
@@ -52,7 +52,7 @@ if [ -z "$CONSOLE" ]; then
   conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
   echo ""
   echo "To connect: bash $0 <console-name>"
-  echo "  e.g: bash $0 pfv-core-sw01"
+  echo "  e.g: bash $0 pfv-r5-core-01"
 else
   echo ""
   echo "--- Connecting to: $CONSOLE ---"

dcinfra/console/setup.sh

@@ -202,7 +202,7 @@ echo "    ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
 echo "    conman connects via telnet for logging + multiplexing"
 echo ""
 echo "  Connect from any Tailscale workstation:"
-echo "    conman -d pfv-tsys4:7890 -f pfv-core-sw01"
+echo "    conman -d pfv-tsys4:7890 -f pfv-r5-core-01"
 echo "    conman -d pfv-tsys4:7890 -q              # list consoles"
 echo ""
 echo "  Direct telnet (emergency, conflicts with conman):"

dcinfra/console/validate-conman.sh

@@ -45,7 +45,7 @@ echo "--- 3. Trigger log capture: connect to each console briefly ---"
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
-for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
+for name in pfv-r5-core-01 pfv-r3-tor-mgmt-01 pfv-r3-tor-stor-01 pfv-rrinfra-rtr pfv-r2-tor-01 pfv-r6-mgmt-01; do
   logfile="$LOGDIR/${name}.log"
   if [ -f "$logfile" ]; then
     SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)

dcinfra/powerman/README.md

@@ -1,111 +1,10 @@
-# Powerman PDU Management
+# dcinfra/powerman/README.md
 
-Centralized power management for the Cyclades AlterPath PM10i PDU via
+> **Documentation moved to Discourse — the canonical source of truth.**
-[Powerman](https://github.com/chaos/powerman), running on pfv-tsys1.
+>
-
+> **Cyclades PM10i PDU management via powerman**
-## Hardware
+>
-
+> **Read it here:** https://community.turnsys.com/t/301
-| Component | Details |
+>
-|-----------|---------|
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-| **PDU** | Cyclades AlterPath PM10i (10 controllable AC outlets) |
+> Do not update content here — edit the Discourse wiki topic instead.*
-| **Firmware** | v1.9.0 (Aug 4, 2006) |
-| **Connection** | USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07) |
-| **Host** | pfv-tsys1 (OptiPlex 9020, Proxmox) |
-| **Serial** | 9600 baud, 8N1, raw mode |
-| **Credentials** | Factory defaults: `admin` / `pm8` (in cyclades-pm10.dev) |
-| **Network access** | powermand listens on `127.0.0.1:10101` (local) + `100.121.189.98:10101` (Tailscale) |
-
-## Device mapping
-
-```
-USB adapter (067b:23a3, serial BJAAb144J07)
-  └─ pl2303 driver → /dev/ttyUSB1
-       └─ udev symlink → /dev/cyclades-pm10 (stable across reboots)
-            └─ powermand reads/writes serial → Cyclades PM10i
-                 └─ 10 outlets (factory default names: 1-10)
-```
-
-The udev rule (`/etc/udev/rules.d/99-cyclades-pdu.rules`) pins the adapter
-by its USB serial number, so the symlink survives replugs and reboots.
-
-## Scripts
-
-All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`:
-
-```bash
-# Setup (idempotent — safe to re-run):
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh
-
-# Validate PDU control (cycles outlet 10 off → on):
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh
-
-# Status check:
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh
-```
-
-### Customizing for other hosts/PDUs
-
-The setup script accepts environment overrides:
-
-```bash
-PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \
-  PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh
-```
-
-## Usage (daily operations)
-
-From pfv-tsys1 (or any host with network access to port 10101):
-
-```bash
-# List all outlets
-powerman -l
-
-# Query status (all outlets)
-powerman -q
-
-# Turn outlet off
-powerman -0 outlet-10
-
-# Turn outlet on
-powerman -1 outlet-10
-
-# Cycle outlet (off → 4s delay → on)
-powerman -c outlet-10
-
-# Query a specific outlet
-powerman -q outlet-10
-```
-
-### Remote access from other hosts
-
-powermand listens on `0.0.0.0:10101`. From another tailnet host:
-
-```bash
-powerman --server-host pfv-tsys1 --server-port 10101 -q
-```
-
-Or set `POWERMAN_SERVER=pfv-tsys1:10101` in the environment.
-
-## Configuration files on pfv-tsys1
-
-| File | Purpose |
-|------|---------|
-| `/etc/udev/rules.d/99-cyclades-pdu.rules` | Stable symlink for USB-DB9 adapter |
-| `/etc/powerman/powerman.conf` | Device definition + 10 outlet nodes |
-| `/etc/powerman/cyclades-pm10.dev` | Cyclades PM10 protocol spec (shipped with powerman) |
-
-## Validation results
-
-2026-07-28: All 8 checks passed.
-Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled.
-
-## TODO (Friday onsite)
-
-- [ ] **Rename outlets** in `/etc/powerman/powerman.conf` to match the
-      physical devices plugged into each outlet (e.g., `node "tsys4-psu"
-      "cyclades-pm10" "3"`). Currently all outlets are generically named
-      `outlet-1` through `outlet-10`.
-- [ ] **Change PDU admin password** from factory default (`pm8`) if
-      security-sensitive. Update `/etc/powerman/cyclades-pm10.dev` login
-      script to match.
-- [ ] **Verify all 10 outlets** individually once device mapping is known.

dcinfra/powerman/identify-outlets.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/bash
+# powerman/identify-outlets.sh — flash each PDU outlet sequentially for physical tracing
+#
+# Run this from the workstation. It flashes each outlet one at a time so you
+# can walk the rack and see which device's LED blinks. Write down the mapping,
+# then run rename-outlets.sh with that mapping.
+#
+# Usage:
+#   bash dcinfra/powerman/identify-outlets.sh
+#
+# On Friday: run this, walk the rack, note which outlet → which device.
+set -uo pipefail
+PROX_HOST="${PROX_HOST:-pfv-tsys1}"
+REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
+
+echo "PDU Outlet Identification — Flash Sequence"
+echo "============================================"
+echo "Each outlet will flash for 5 seconds. Walk the rack and note the device."
+echo "Press Enter to start..."
+read -r
+
+for i in $(seq 1 10); do
+  echo "--- Outlet $i: FLASHING (5s) ---"
+  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -f outlet-$i" </dev/null 2>/dev/null
+  sleep 5
+  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -u outlet-$i" </dev/null 2>/dev/null
+  echo "    Outlet $i → ? (write it down)"
+  echo ""
+  [ "$i" -lt 10 ] && { echo "Press Enter for next outlet..."; read -r; }
+done
+
+echo "============================================"
+echo "Done. Now create your mapping file and run:"
+echo "  bash dcinfra/powerman/rename-outlets.sh"
+echo ""
+echo "Format: outlet-number:new-name (one per line)"
+echo "Example:"
+echo "  1:pfv-tsys1"
+echo "  2:pfv-tsys3"
+echo "  ..."
+echo "============================================"

dcinfra/powerman/rename-outlets.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/bash
+# powerman/rename-outlets.sh — rename PDU outlets in powerman.conf
+#
+# Takes a mapping file (outlet-number:new-name, one per line) and rewrites
+# the node entries in /etc/powerman/powerman.conf on pfv-tsys1, then
+# restarts powermand.
+#
+# Usage:
+#   bash dcinfra/powerman/rename-outlets.sh <mapping-file>
+#
+# Example mapping file:
+#   1:pfv-tsys1
+#   2:pfv-tsys3
+#   3:pfv-tsys4
+#   ...
+set -euo pipefail
+
+PROX_HOST="${PROX_HOST:-pfv-tsys1}"
+REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
+MAP_FILE="${1:-}"
+
+if [ -z "$MAP_FILE" ] || [ ! -f "$MAP_FILE" ]; then
+  echo "Usage: $0 <mapping-file>"
+  echo "  Format: outlet-number:new-name (one per line)"
+  echo "  Run identify-outlets.sh first to get the mapping."
+  exit 1
+fi
+
+# Build the new node lines
+NODE_LINES=""
+while IFS=: read -r num name; do
+  [ -z "$num" ] && continue
+  NODE_LINES+="node \"$name\" \"cyclades-pm10\" \"$num\""$'\n'
+done < "$MAP_FILE"
+
+# Send to tsys1: backup conf, write new node section, restart powermand
+PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox-file - <<REMOTE_SCRIPT
+set -euo pipefail
+cp /etc/powerman/powerman.conf /etc/powerman/powerman.conf.bak.\$(date +%Y%m%d-%H%M%S)
+
+# Strip existing node lines and append new ones
+grep -v '^node "outlet-' /etc/powerman/powerman.conf > /tmp/powerman.conf.new
+cat >> /tmp/powerman.conf.new <<'NODES'
+$(echo -n "$NODE_LINES")
+NODES
+mv /tmp/powerman.conf.new /etc/powerman/powerman.conf
+systemctl restart powerman
+sleep 1
+powerman -l
+REMOTE_SCRIPT
+
+echo "PDU outlets renamed. Verify with: PROX_HOST=$PROX_HOST bash $REMOTE_SH prox 'powerman -q'"

dcinfra/ups/README.md

@@ -1,210 +1,10 @@
-# UPS Management (NUT — Network UPS Tools)
+# dcinfra/ups/README.md
 
-Centralized UPS monitoring for the server room via
+> **Documentation moved to Discourse — the canonical source of truth.**
-[NUT](https://networkupstools.org/), running on **pfv-tsys1**. USB HID UPS
+>
-units feed one `upsd` network server; Home Assistant polls it over Tailscale for
+> **UPS management (NUT) for APC Smart-UPS C 1500**
-real-time power/load/runtime tracking, and a local `upsmon` shuts the hypervisor
+>
-down gracefully when battery is low.
+> **Read it here:** https://community.turnsys.com/t/301
-
+>
-> **Why NUT (not apcupsd)?** Two different UPS brands (APC + Tripp Lite) must be
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-> covered. `apcupsd` only supports APC, so it would require a second daemon
+> Do not update content here — edit the Discourse wiki topic instead.*
-> stack. NUT's `usbhid-ups` driver speaks to **both** via the USB HID Power
-> Device class, and Home Assistant ships a first-class NUT integration.
-
-## Hardware
-
-| UPS | Model | VID:PID | USB Serial | Status |
-|-----|-------|---------|------------|--------|
-| **APC** | Smart-UPS C 1500 (FW 02.2) | `051d:0003` | `AS1213210423` | **LIVE** |
-| **Tripp Lite** | UPS (HID PDC) | `09ae:3016` | `2352CVLSM871900694` | **Blocked** — see below |
-
-## Current State (2026-07-30)
-
-### APC Smart-UPS C 1500 — OPERATIONAL
-
-Fully reporting via `usbhid-ups` + `APC HID 0.100` subdriver. Data validated:
-
-```
-battery.charge: 100         battery.runtime: 1800     battery.voltage: 27.4
-ups.status: OL              ups.load: (via HA)         ups.model: Smart-UPS C 1500
-```
-
-### Tripp Lite UPS — BLOCKED (hardware issue)
-
-The driver finds the device, matches the `TrippLite HID 0.85` subdriver, claims
-the interface, and reads the HID descriptor — but **fails reading the 878-byte
-HID Report Descriptor** (`Resource temporarily unavailable` / EAGAIN after 5s).
-The driver is masked to prevent restart-loop spam.
-
-USB descriptors (manufacturer, product, serial) are readable via `lsusb -v` and
-`nut-scanner`, but the bulk control transfer for the full report descriptor
-times out. Likely causes:
-
-1. **USB hub** — the Tripp Lite is behind a Genesys Logic hub (`05e3:0608`).
-   Try plugging directly into a motherboard USB port.
-2. **USB cable** — try a high-quality data cable (not charge-only).
-3. **UPS firmware** — the USB controller may not properly implement all HID
-   endpoints.
-
-**To retry after physical reseat:**
-```bash
-# On pfv-tsys1:
-systemctl unmask nut-driver@tripp-lite-ups
-systemctl start nut-driver@tripp-lite-ups
-upsc tripp-lite-ups@localhost
-```
-
-## Architecture
-
-```
-pfv-tsys1 (192.168.3.11 / Tailscale 100.121.189.98)
- ├─ APC Smart-UPS C 1500      ──┐
- └─ Tripp Lite UPS (masked)  ──┤  USB HID
-                                ▼
-                       nut-driver@apc-smartups-c1500 (usbhid-ups)
-                                ▼
-          upsd  :3493  (LISTEN 127.0.0.1 + Tailscale + LAN)
-                                ▼            ▼
-                  upsmon (local)        Home Assistant (NUT integration)
-                  graceful shutdown      via LAN 192.168.3.11 (HAOS can't
-                                         route to Tailscale IPs)
-```
-
-- **Driver layer** — `usbhid-ups` process, pinned by USB serial. Debian uses
-  templated `nut-driver@<upsname>.service` units managed by
-  `nut-driver-enumerator`.
-- **Server layer** — `upsd` exposes UPS data on TCP 3493 (localhost + Tailscale
-  + LAN). Clients authenticate via `upsd.users`.
-- **Monitor layer** — `upsmon` runs locally as `master` to trigger
-  `SHUTDOWNCMD` (`/sbin/shutdown -h now`) when a UPS reports `LOWBATT`.
-- **Home Assistant** — native NUT integration connects to `upsd` over Tailscale
-  and exposes `ups.load`, `battery.runtime`, `ups.status`, etc. as sensors.
-
-### Key deployment lesson: udev must cover raw USB devices
-
-The `usbhid-ups` driver opens `/dev/bus/usb/BBB/DDD` (raw USB device files),
-**not** `/dev/hidraw*`. After calling `setuid(111)` to drop to the `nut` user,
-it needs write access to those raw USB files. The udev rule must match
-`SUBSYSTEM=="usb"` by vendor/product ID to set `GROUP="nut"` — matching only
-`hidraw` is insufficient. See `/etc/udev/rules.d/99-nut-ups.rules`.
-
-## Scripts
-
-NUT host scripts run on pfv-tsys1 via `tests/remote.sh`:
-
-```bash
-# Idempotent install + configure (safe to re-run):
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
-
-# Discover USB UPS + NUT state (read-only diagnostic):
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
-
-# Query UPS data + service health:
-PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
-```
-
-The HA integration script runs from your workstation (needs HA API access):
-
-```bash
-# Add the NUT integration to Home Assistant (idempotent):
-bash ups/setup-ha-nut.sh
-```
-
-`setup.sh` accepts environment overrides for serials/VIDs/PIDs/usernames, so it
-can be repurposed for other hosts or UPS units. Passwords for `monuser` and
-`homeassistant` are auto-generated on first run and reused on subsequent runs
-(stored in `/etc/nut/upsd.users`).
-
-Set `TRIPP_ENABLED=0` to skip the Tripp Lite entirely (useful if it's physically
-unplugged).
-
-## Configuration files on pfv-tsys1
-
-| File | Purpose |
-|------|---------|
-| `/etc/udev/rules.d/99-nut-ups.rules` | Grant nut group rw on raw USB + hidraw devices (both subsystems) |
-| `/etc/nut/ups.conf` | `usbhid-ups` device(s), pinned by serial + subdriver |
-| `/etc/nut/upsd.conf` | `LISTEN 127.0.0.1` + `LISTEN <tailscale>` + `LISTEN <lan>` on port 3493 |
-| `/etc/nut/upsd.users` | `monuser` (master) + `homeassistant` (read-only) credentials |
-| `/etc/nut/upsmon.conf` | Local master monitor + `SHUTDOWNCMD` |
-| `/etc/nut/nut.conf` | `MODE=netserver` |
-
-## Home Assistant integration
-
-The NUT integration is added automatically by `setup-ha-nut.sh`, which drives
-HA's REST config-flow API. It is idempotent (skips if the entry exists).
-
-```bash
-# Prerequisites: create token + password files (one-time):
-mkdir -p ~/.config/pfvcluster
-# HA → Profile → Long-Lived Access Tokens → Create Token:
-echo -n 'YOUR_HA_TOKEN' > ~/.config/pfvcluster/ha-token
-# Password is in /etc/nut/upsd.users on pfv-tsys1 (the homeassistant user):
-echo -n 'YOUR_NUT_PASS' > ~/.config/pfvcluster/nut-password
-chmod 600 ~/.config/pfvcluster/{ha-token,nut-password}
-
-# Run:
-bash ups/setup-ha-nut.sh
-```
-
-### Why LAN IP, not Tailscale
-
-upsd listens on **both** the Tailscale IP (`100.121.189.98`) **and** the LAN IP
-(`192.168.3.11`). The HA NUT integration uses the **LAN IP** because HAOS runs
-Tailscale as an isolated add-on container — the HA core container cannot route
-to Tailscale IPs. Since pfv-bms (HA, `192.168.3.12`) and pfv-tsys1 (`192.168.3.11`)
-share the same vmbr0 bridge, LAN connectivity is instant and reliable.
-
-### Manual UI alternative
-
-In Home Assistant → **Settings → Devices & Services → Add Integration → NUT**:
-
-| Field | Value |
-|-------|-------|
-| Host | `192.168.3.11` (LAN — HAOS can't reach Tailscale IPs from the HA container) |
-| Port | `3493` |
-| Username | `homeassistant` |
-| Password | *(stored in `/etc/nut/upsd.users` on pfv-tsys1)* |
-| UPS | `apc-smartups-c1500` |
-
-### Live sensors
-
-HA exposes UPS data as sensors (prefix `sensor.apc_smartups_c1500_`):
-`battery_charge`, `status` (Online/On Battery), `status_data` (OL/OB/DISCHRG).
-Additional sensors (load, runtime, voltage) populate as the UPS reports them.
-
-## Daily operations
-
-From pfv-tsys1 (or any tailnet host with NUT client installed):
-
-```bash
-# List UPS units served by upsd
-upsc -l pfv-tsys1
-
-# Full variable dump for one UPS
-upsc apc-smartups-c1500@pfv-tsys1
-
-# Battery runtime (the only runtime/charge data this UPS exposes)
-upsc apc-smartups-c1500@pfv-tsys1 battery.runtime
-```
-
-## Notes
-
-- **No USB passthrough to the HA VM.** Keeping the UPS on the host preserves
-  hypervisor graceful-shutdown capability and matches the `powerman/` pattern
-  (PDU managed on the host where the adapter physically lives).
-- **No `ups.load` / `ups.realpower` on this UPS (FW 02.2, mfg 2012):** The
-  APC Smart-UPS C 1500 does not expose load or power data over USB HID.
-  Both NUT `usbhid-ups` and `apcupsd` (USB mode, tested 2026-07-30) read the
-  same HID descriptor — the variable simply isn't there. This means the HA
-  NUT integration provides **battery/runtime/status sensors only**, not
-  wattage for the Energy Dashboard.
-  - **apcupsd test note:** Debian's `apcupsd` package conflicts with
-    `nut-server` (mutually exclusive). apcupsd USB mode returned `COMMLOST`
-    even before we could check load. The APC Smart Serial protocol (serial
-    cable, AP940-1524C, ~$30) DOES report load%, but this requires a serial
-    port on the UPS and on the host.
-  - **Energy Dashboard path:** A smart plug (Shelly Plug S / TP-Link Kasa,
-    ~$15-25) on the UPS output reports real watts natively and feeds the
-    Energy Dashboard with zero UPS-driver hacking. The NUT sensors remain
-    valuable for outage detection and graceful-shutdown automations.

docs/docmap.md

@@ -1,108 +1,10 @@
-# Documentation Map
+# docs/docmap.md
 
-> **Index of all documentation in this repo.** Agents must update this file
+> **Documentation moved to Discourse — the canonical source of truth.**
-> whenever a doc is added, removed, or substantively changed.
+>
-
+> **Documentation index — now lives in the Project Overview topic**
-> **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra)
+>
-
+> **Read it here:** https://community.turnsys.com/t/296
-## Kubernetes Architecture ([`k8s/`](k8s/))
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-Distro decision, target architecture, control-plane design, bootstrap and DR
+> Do not update content here — edit the Discourse wiki topic instead.*
-procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
-
-| Document | Description | Last Reviewed |
-|----------|-------------|---------------|
-| [`k8s/docs/README.md`](k8s/docs/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
-| [`k8s/docs/DISTRO-DECISION.md`](k8s/docs/DISTRO-DECISION.md) | Talos vs k3s analysis. Decision: k3s for regular R&D cluster; Talos preserved for future ITAR/classified | 2026-07-28 |
-| [`k8s/docs/ARCHITECTURE.md`](k8s/docs/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
-
-## Proxmox Cluster ([`proxmox/`](proxmox/))
-
-Fleet operations, hardware, performance tuning, storage architecture.
-
-| Document | Description | Last Reviewed |
-|----------|-------------|---------------|
-| [`proxmox/docs/NETWORK-TOPOLOGY.md`](proxmox/docs/NETWORK-TOPOLOGY.md) | **Full network reference**: Mermaid topology diagrams, complete switch port maps, VLAN/subnet map, cross-rack trunk analysis, NFS data-flow, per-host NIC reference | 2026-07-30 |
-| [`proxmox/docs/STORAGE-ARCHITECTURE.md`](proxmox/docs/STORAGE-ARCHITECTURE.md) | **Full storage reference**: tier definitions, physical disk inventory (all 14 disks), NFS export/mount map, per-spindle VM placement, capacity summary | 2026-07-30 |
-| [`proxmox/docs/HOST-REFERENCE.md`](proxmox/docs/HOST-REFERENCE.md) | **Per-host reference**: hardware, network, storage, tuning, role, running VMs for all 7 nodes | 2026-07-30 |
-| [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 |
-| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
-| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
-| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | Prior audit (network + storage + compute + switches). **Superseded by AUDIT-2026-08-05** | 2026-07-30 |
-| [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) | **Current & authoritative.** Read-only ground-truth via nmap/SNMP/SSH/API: 4 switches up / 1 down, 63 VMs, 167 zones, 753 DHCP leases, k8s DOWN, netinfra-02 wiped, repo restructure, 3-tier Tailscale fleet | 2026-08-05 |
-| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 |
-| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
-| [`proxmox/docs/NAMING-POLICY.md`](proxmox/docs/NAMING-POLICY.md) | Canonical naming policy: VM name = source of truth (DNS/Tailscale/monitoring must match) | 2026-07-31 |
-| [`proxmox/docs/PHYSICAL-SYSTEMS.md`](proxmox/docs/PHYSICAL-SYSTEMS.md) | Physical infrastructure: Pi's, network devices, UniFi APs, SNMP-discovered hosts | 2026-07-31 |
-| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
-
-## Server Build ([`docs/server-build/`](server-build/))
-
-Server provisioning, security hardening, DNS/NTP configuration.
-
-| Document | Description | Last Reviewed |
-|----------|-------------|---------------|
-| [`docs/server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
-| [`docs/server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
-| [`docs/server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
-| [`docs/server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
-| [`docs/server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
-
-## Operational Guides (outside docs/)
-
-| Document | Description | Last Reviewed |
-|----------|-------------|---------------|
-| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
-| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
-| [`../dcinfra/powerman/README.md`](../dcinfra/powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
-| [`../dcinfra/console/README.md`](../dcinfra/console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
-| [`../dcinfra/ups/README.md`](../dcinfra/ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
-| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
-| [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
-| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
-| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
-| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
-
-## Archive ([`archive/`](archive/))
-
-Historical AI reviews, completed task lists, and pre-refactor examples. Read-only
-context — do not update; link to active docs instead.
-
-| Document | Description |
-|----------|-------------|
-| [`archive/CODE-REVIEW-FINDINGS.md`](archive/CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) |
-| [`archive/REFACTORING-EXAMPLES.md`](archive/REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) |
-| [`archive/Claude-Review.md`](archive/Claude-Review.md) | Claude's initial code review |
-| [`archive/AIReview-QWEN.md`](archive/AIReview-QWEN.md) | Qwen AI review |
-| [`archive/AiOverview-Gemini.md`](archive/AiOverview-Gemini.md) | Gemini project overview |
-| [`archive/AiOverview-OpenCode.md`](archive/AiOverview-OpenCode.md) | OpenCode project overview |
-| [`archive/AiSecurityAudit-Gemini.md`](archive/AiSecurityAudit-Gemini.md) | Gemini security audit |
-| [`archive/charles-todo.md`](archive/charles-todo.md) | Charles's early task list (completed) |
-| [`archive/claude-todo.md`](archive/claude-todo.md) | Claude's early task list (completed) |
-
-## Top-Level Files
-
-| File | Description |
-|------|-------------|
-| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, gardening protocol) |
-| [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) |
-| [`../README.md`](../README.md) | Project overview and quick start |
-| [`../LICENSE`](../LICENSE) | License |
-
----
-
-## Agent Gardening Protocol
-
-When making changes to this repo, agents MUST:
-
-1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure
-   state, completes/starts a task, or discovers a new issue.
-2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a
-   substantive content change. Update the "Last Reviewed" date.
-3. **Verify cross-references** — any new `.md` file must be linked from at
-   least one existing doc or this map.
-4. **Check for stale paths** — after any directory rename or file move,
-   `grep -rn 'old/path' --include='*.md'` and fix all references.
-5. **Keep code and docs in sync** — if you change a script's interface,
-   behavior, or location, update every doc that references it in the same
-   commit.

docs/server-build/DEPLOYMENT.md

@@ -1,336 +1,10 @@
-# TSYS FetchApply Deployment Guide
+# docs/server-build/DEPLOYMENT.md
-
+
-## Overview
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-This guide provides comprehensive instructions for deploying the TSYS FetchApply infrastructure provisioning system on Linux servers.
+> **Server deployment procedures**
-
+>
-## Prerequisites
+> **Read it here:** https://community.turnsys.com/t/302
-
+>
-### System Requirements
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-- **Operating System:** Ubuntu 18.04+ or Debian 10+ (recommended)
+> Do not update content here — edit the Discourse wiki topic instead.*
-- **RAM:** Minimum 2GB, recommended 4GB
-- **Disk Space:** Minimum 10GB free space
-- **Network:** Internet connectivity for package downloads
-- **Privileges:** Root or sudo access required
-
-### Required Tools
-- `git` - Version control system
-- `curl` - HTTP client for downloads
-- `wget` - Alternative download tool
-- `systemctl` - System service management
-- `apt-get` - Package management (Debian/Ubuntu)
-
-### Network Requirements
-- **HTTPS access** to:
-  - `https://archive.ubuntu.com` (Ubuntu packages)
-  - `https://linux.dell.com` (Dell hardware support)
-  - `https://download.proxmox.com` (Proxmox packages)
-  - `https://github.com` (Git repositories)
-
-## Pre-Deployment Validation
-
-### 1. System Compatibility Check
-```bash
-# Clone repository
-git clone [repository-url]
-cd FetchApply
-
-# Run system validation
-./Project-Tests/validation/system-requirements.sh
-```
-
-### 2. Network Connectivity Test
-```bash
-# Test network connectivity
-curl -I https://archive.ubuntu.com
-curl -I https://linux.dell.com
-curl -I https://download.proxmox.com
-```
-
-### 3. Permission Verification
-```bash
-# Verify write permissions
-test -w /etc && echo "✅ /etc writable" || echo "❌ /etc not writable"
-test -w /usr/local/bin && echo "✅ /usr/local/bin writable" || echo "❌ /usr/local/bin not writable"
-```
-
-## Deployment Methods
-
-### Method 1: Standard Deployment (Recommended)
-```bash
-# 1. Clone repository
-git clone [repository-url]
-cd FetchApply
-
-# 2. Run pre-deployment tests
-./Project-Tests/run-tests.sh validation
-
-# 3. Execute deployment
-cd ProjectCode
-sudo bash SetupNewSystem.sh
-```
-
-### Method 2: Dry Run Mode
-```bash
-# 1. Clone repository
-git clone [repository-url]
-cd FetchApply
-
-# 2. Review configuration
-cat provisioning/SetupNewSystem.sh
-
-# 3. Execute with manual review
-cd ProjectCode
-sudo bash -x SetupNewSystem.sh  # Debug mode
-```
-
-## Deployment Process
-
-### Phase 1: Framework Initialization
-1. **Environment Setup**
-   - Load framework variables
-   - Source framework includes
-   - Initialize logging system
-
-2. **System Detection**
-   - Detect physical vs virtual hardware
-   - Identify operating system
-   - Check for existing users
-
-### Phase 2: Base System Configuration
-1. **Package Installation**
-   - Update package repositories
-   - Install essential packages
-   - Configure package sources
-
-2. **User Management**
-   - Create required user accounts
-   - Configure SSH access
-   - Set up sudo permissions
-
-### Phase 3: Security Hardening
-1. **SSH Configuration**
-   - Deploy hardened SSH configuration
-   - Install SSH keys
-   - Disable password authentication
-
-2. **System Hardening**
-   - Configure firewall rules
-   - Enable audit logging
-   - Install security tools
-
-### Phase 4: Monitoring and Management
-1. **Monitoring Agents**
-   - Deploy LibreNMS agents
-   - Configure SNMP
-   - Set up system monitoring
-
-2. **Management Tools**
-   - Install Cockpit dashboard
-   - Configure remote access
-   - Set up maintenance scripts
-
-## Post-Deployment Verification
-
-### 1. Security Validation
-```bash
-# Run security tests
-./Project-Tests/run-tests.sh security
-
-# Verify SSH configuration
-ssh -T [server-ip]  # Should work with key authentication
-```
-
-### 2. Service Status Check
-```bash
-# Check critical services
-sudo systemctl status ssh
-sudo systemctl status auditd
-sudo systemctl status snmpd
-```
-
-### 3. Network Connectivity
-```bash
-# Test internal services
-curl -k https://localhost:9090  # Cockpit
-snmpwalk -v2c -c public localhost system
-```
-
-## Troubleshooting
-
-### Common Issues
-
-#### 1. Permission Denied Errors
-```bash
-# Solution: Run with sudo
-sudo bash SetupNewSystem.sh
-```
-
-#### 2. Network Connectivity Issues
-```bash
-# Check DNS resolution
-nslookup archive.ubuntu.com
-
-# Test direct IP access
-curl -I 91.189.91.26  # Ubuntu archive IP
-```
-
-#### 3. Package Installation Failures
-```bash
-# Update package cache
-sudo apt-get update
-
-# Fix broken packages
-sudo apt-get -f install
-```
-
-#### 4. SSH Key Issues
-```bash
-# Verify key permissions
-ls -la ~/.ssh/
-chmod 600 ~/.ssh/id_rsa
-chmod 644 ~/.ssh/id_rsa.pub
-```
-
-### Debug Mode
-```bash
-# Enable debug logging
-export DEBUG=1
-bash -x SetupNewSystem.sh
-```
-
-### Log Analysis
-```bash
-# Check deployment logs
-tail -f /var/log/fetchapply/deployment.log
-
-# Review system logs
-journalctl -u ssh
-journalctl -u auditd
-```
-
-## Environment-Specific Configurations
-
-### Physical Dell Servers
-- **OMSA Installation:** Dell OpenManage Server Administrator
-- **Hardware Monitoring:** iDRAC configuration
-- **Performance Tuning:** CPU and memory optimizations
-
-### Virtual Machines
-- **Guest Additions:** VMware tools or VirtualBox additions
-- **Resource Limits:** Memory and CPU constraints
-- **Network Configuration:** Bridge vs NAT settings
-
-### Development Environments
-- **SSH Configuration:** Less restrictive settings
-- **Development Tools:** Additional packages for development
-- **Testing Access:** Enhanced logging and debugging
-
-## Maintenance and Updates
-
-### Regular Maintenance
-```bash
-# Update system packages
-sudo apt-get update && sudo apt-get upgrade
-
-# Update monitoring scripts
-cd /usr/local/bin
-sudo wget https://[repository]/scripts/up2date.sh
-sudo chmod +x up2date.sh
-```
-
-### Security Updates
-```bash
-# Check for security updates
-sudo apt-get update
-sudo apt list --upgradable | grep -i security
-
-# Apply security patches
-sudo apt-get upgrade
-```
-
-### Configuration Updates
-```bash
-# Update FetchApply
-cd FetchApply
-git pull origin main
-
-# Re-run specific modules
-cd provisioning/Modules/Security
-sudo bash secharden-ssh.sh
-```
-
-## Best Practices
-
-### 1. Pre-Deployment
-- Always test in non-production environment first
-- Review all scripts before execution
-- Validate network connectivity
-- Ensure proper backup procedures
-
-### 2. During Deployment
-- Monitor deployment progress
-- Check for errors and warnings
-- Document any customizations
-- Validate each phase completion
-
-### 3. Post-Deployment
-- Run full security test suite
-- Verify all services are running
-- Test remote access
-- Document deployment specifics
-
-### 4. Ongoing Operations
-- Regular security updates
-- Monitor system performance
-- Review audit logs
-- Maintain deployment documentation
-
-## Support and Resources
-
-### Documentation
-- **README.md:** Basic usage instructions
-- **SECURITY.md:** Security architecture and guidelines
-- **tests/README.md:** Testing framework documentation
-
-### Community Support
-- **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
-- **Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26
-
-### Professional Support
-- **Technical Support:** [Contact information to be added]
-- **Consulting Services:** [Contact information to be added]
-
-## Deployment Checklist
-
-### Pre-Deployment
-- [ ] System requirements validated
-- [ ] Network connectivity tested
-- [ ] Backup procedures in place
-- [ ] Security review completed
-
-### Deployment
-- [ ] Repository cloned successfully
-- [ ] Pre-deployment tests passed
-- [ ] Deployment executed without errors
-- [ ] Post-deployment verification completed
-
-### Post-Deployment
-- [ ] Security tests passed
-- [ ] All services running
-- [ ] Remote access verified
-- [ ] Documentation updated
-
-### Maintenance
-- [ ] Update schedule established
-- [ ] Monitoring configured
-- [ ] Backup procedures tested
-- [ ] Incident response plan activated
-
-## Version History
-
-- **v1.0:** Initial deployment framework
-- **v1.1:** Added security hardening and secrets management
-- **v1.2:** Enhanced testing framework and documentation
-
-Last updated: July 14, 2025

docs/server-build/DEVELOPMENT-GUIDELINES.md

@@ -1,407 +1,10 @@
-<!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
+# docs/server-build/DEVELOPMENT-GUIDELINES.md
-# TSYS PFVCluster Development Guidelines
+
-
+> **Documentation moved to Discourse — the canonical source of truth.**
-## Overview
+>
-
+> **Coding standards, commit conventions, script patterns**
-This document contains development standards and best practices for the TSYS PFVCluster infrastructure provisioning system.
+>
-
+> **Read it here:** https://community.turnsys.com/t/302
-## Package Management Best Practices
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-### Combine apt-get Install Commands
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-**Rule:** Always combine multiple package installations into a single `apt-get install` command for performance.
-
-**Rationale:** Single command execution is significantly faster than multiple separate commands due to:
-- Reduced package cache processing
-- Single dependency resolution
-- Fewer network connections
-- Optimized package download ordering
-
-#### ✅ Correct Implementation
-```bash
-# Install all packages in one command
-apt-get install -y package1 package2 package3 package4
-
-# Real example from 2FA script
-apt-get install -y libpam-google-authenticator qrencode
-```
-
-#### ❌ Incorrect Implementation
-```bash
-# Don't use separate commands for each package
-apt-get install -y package1
-apt-get install -y package2
-apt-get install -y package3
-```
-
-#### Complex Package Installation Pattern
-```bash
-function install_security_packages() {
-    print_info "Installing security packages..."
-    
-    # Update package cache once
-    apt-get update
-    
-    # Install all packages in single command
-    apt-get install -y \
-        auditd \
-        fail2ban \
-        libpam-google-authenticator \
-        lynis \
-        rkhunter \
-        aide \
-        chkrootkit \
-        clamav \
-        clamav-daemon
-    
-    print_success "Security packages installed successfully"
-}
-```
-
-## Script Development Standards
-
-### Error Handling
-- Always use `set -euo pipefail` at script start
-- Implement proper error trapping
-- Use framework error handling functions
-- Return appropriate exit codes
-
-### Function Structure
-```bash
-function function_name() {
-    print_info "Description of what function does..."
-    
-    # Local variables
-    local var1="value"
-    local var2="value"
-    
-    # Function logic
-    if [[ condition ]]; then
-        print_success "Success message"
-        return 0
-    else
-        print_error "Error message"
-        return 1
-    fi
-}
-```
-
-### Framework Integration
-- Source framework includes at script start
-- Use framework logging and pretty print functions
-- Follow existing patterns for consistency
-- Include proper PROJECT_ROOT path resolution
-
-```bash
-# Standard framework sourcing pattern
-PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.."
-source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh"
-source "$PROJECT_ROOT/Framework-Includes/Logging.sh"
-source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh"
-```
-
-## Code Quality Standards
-
-### ShellCheck Compliance
-- All scripts must pass shellcheck validation
-- Address shellcheck warnings appropriately
-- Use proper quoting for variables
-- Handle edge cases and error conditions
-
-### Variable Naming
-- Use UPPERCASE for global constants
-- Use lowercase for local variables
-- Use descriptive names
-- Quote all variable expansions
-
-```bash
-# Global constants
-declare -g BACKUP_DIR="/root/backup"
-declare -g CONFIG_FILE="/etc/ssh/sshd_config"
-
-# Local variables
-local user_name="localuser"
-local temp_file="/tmp/config.tmp"
-
-# Proper quoting
-if [[ -f "$CONFIG_FILE" ]]; then
-    cp "$CONFIG_FILE" "$BACKUP_DIR/"
-fi
-```
-
-### Function Documentation
-- Include purpose description
-- Document parameters if any
-- Document return values
-- Include usage examples for complex functions
-
-```bash
-# Configure SSH hardening settings
-# Parameters: none
-# Returns: 0 on success, 1 on failure
-# Usage: configure_ssh_hardening
-function configure_ssh_hardening() {
-    print_info "Configuring SSH hardening..."
-    # Implementation
-}
-```
-
-## Testing Requirements
-
-### Test Coverage
-- Every new module must include corresponding tests
-- Test both success and failure scenarios
-- Validate configurations after changes
-- Include integration tests for complex workflows
-
-### Test Categories
-1. **Unit Tests:** Individual function validation
-2. **Integration Tests:** Module interaction testing
-3. **Security Tests:** Security configuration validation
-4. **Validation Tests:** System requirement checking
-
-### Test Implementation Pattern
-```bash
-function test_function_name() {
-    echo "🔍 Testing specific functionality..."
-    
-    local failed=0
-    
-    # Test implementation
-    if [[ condition ]]; then
-        echo "✅ Test passed"
-    else
-        echo "❌ Test failed"
-        ((failed++))
-    fi
-    
-    return $failed
-}
-```
-
-## Security Standards
-
-### Configuration Backup
-- Always backup configurations before modification
-- Use timestamped backup directories
-- Provide restore instructions
-- Test backup/restore procedures
-
-### Service Management
-- Test configurations before restarting services
-- Provide rollback procedures
-- Validate service status after changes
-- Include service dependency handling
-
-### User Safety
-- Use `nullok` for gradual 2FA rollout
-- Provide clear setup instructions
-- Include emergency access procedures
-- Test all access methods before enforcement
-
-## Documentation Standards
-
-### Script Headers
-```bash
-#!/bin/bash
-
-# TSYS Module Name - Brief Description
-# Longer description of what this script does
-# Author: TSYS Development Team
-# Version: 1.0
-# Last Updated: YYYY-MM-DD
-
-set -euo pipefail
-```
-
-### Inline Documentation
-- Comment complex logic
-- Explain non-obvious decisions
-- Document external dependencies
-- Include troubleshooting notes
-
-### User Documentation
-- Create comprehensive guides for complex features
-- Include step-by-step procedures
-- Provide troubleshooting sections
-- Include examples and use cases
-
-## Performance Optimization
-
-### Package Management
-- Single apt-get commands (as noted above)
-- Cache package lists appropriately
-- Use specific package versions when stability required
-- Clean up package cache when appropriate
-
-### Network Operations
-- Use connection timeouts for external requests
-- Implement retry logic with backoff
-- Cache downloaded resources when possible
-- Validate download integrity
-
-### File Operations
-- Use efficient file processing tools
-- Minimize file system operations
-- Use appropriate file permissions
-- Clean up temporary files
-
-## Version Control Practices
-
-### Commit Messages
-- Use descriptive commit messages
-- Include scope of changes
-- Reference related issues/requirements
-- Follow established commit message format
-
-### Branch Management
-- Test changes in feature branches
-- Use pull requests for review
-- Maintain clean commit history
-- Tag releases appropriately
-
-### Code Review Requirements
-- All changes require review
-- Security changes require security team review
-- Test coverage must be maintained
-- Documentation must be updated
-
-## Deployment Practices
-
-### Pre-Deployment
-- Run full test suite
-- Validate in test environment
-- Review security implications
-- Update documentation
-
-### Deployment Process
-- Use configuration validation
-- Implement gradual rollout when possible
-- Monitor for issues during deployment
-- Have rollback procedures ready
-
-### Post-Deployment
-- Validate deployment success
-- Monitor system performance
-- Update operational documentation
-- Gather feedback for improvements
-
-## Example Implementation
-
-### Complete Module Template
-```bash
-#!/bin/bash
-
-# TSYS Security Module - Template
-# Template for creating new security modules
-# Author: TSYS Development Team
-
-set -euo pipefail
-
-# Source framework functions
-PROJECT_ROOT="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/../.."
-source "$PROJECT_ROOT/Framework-Includes/PrettyPrint.sh"
-source "$PROJECT_ROOT/Framework-Includes/Logging.sh"
-source "$PROJECT_ROOT/Framework-Includes/ErrorHandling.sh"
-
-# Module configuration
-BACKUP_DIR="/root/backup/module-$(date +%Y%m%d-%H%M%S)"
-CONFIG_FILE="/etc/example.conf"
-
-# Create backup directory
-mkdir -p "$BACKUP_DIR"
-
-print_header "TSYS Module Template"
-
-function backup_configs() {
-    print_info "Creating configuration backup..."
-    
-    if [[ -f "$CONFIG_FILE" ]]; then
-        cp "$CONFIG_FILE" "$BACKUP_DIR/"
-        print_success "Configuration backed up"
-    fi
-}
-
-function install_packages() {
-    print_info "Installing required packages..."
-    
-    # Update package cache
-    apt-get update
-    
-    # Install all packages in single command
-    apt-get install -y package1 package2 package3
-    
-    print_success "Packages installed successfully"
-}
-
-function configure_module() {
-    print_info "Configuring module..."
-    
-    # Configuration logic here
-    
-    print_success "Module configured successfully"
-}
-
-function validate_configuration() {
-    print_info "Validating configuration..."
-    
-    local failed=0
-    
-    # Validation logic here
-    
-    if [[ $failed -eq 0 ]]; then
-        print_success "Configuration validation passed"
-        return 0
-    else
-        print_error "Configuration validation failed"
-        return 1
-    fi
-}
-
-function main() {
-    # Check if running as root
-    if [[ $EUID -ne 0 ]]; then
-        print_error "This script must be run as root"
-        exit 1
-    fi
-    
-    # Execute module steps
-    backup_configs
-    install_packages
-    configure_module
-    validate_configuration
-    
-    print_success "Module setup completed successfully!"
-}
-
-# Run main function
-main "$@"
-```
-
-## Continuous Improvement
-
-### Regular Reviews
-- Review guidelines quarterly
-- Update based on lessons learned
-- Incorporate new best practices
-- Gather team feedback
-
-### Tool Updates
-- Keep development tools current
-- Adopt new security practices
-- Update testing frameworks
-- Improve automation
-
-### Knowledge Sharing
-- Document lessons learned
-- Share best practices
-- Provide training materials
-- Maintain knowledge base
-
----
-
-**Last Updated:** July 14, 2025  
-**Version:** 1.0  
-**Author:** TSYS Development Team
-
-**Note:** These guidelines are living documents and should be updated as the project evolves and new best practices are identified.

docs/server-build/SECURITY.md

@@ -1,190 +1,10 @@
-# PFVCluster Security Documentation
+# docs/server-build/SECURITY.md
 
-## Security Architecture
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-The PFVCluster infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management.
+> **Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd**
-
+>
-## Current Security Features
+> **Read it here:** https://community.turnsys.com/t/303
-
+>
-### 1. Secure Deployment Method ✅
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-- **Git-based deployment:** Uses `git clone` instead of `curl | bash`
+> Do not update content here — edit the Discourse wiki topic instead.*
-- **Local execution:** Scripts run locally after inspection
-- **Version control:** Full audit trail of changes
-- **Code review:** Changes require explicit approval
-
-### 2. HTTPS Enforcement ✅
-- **All downloads use HTTPS:** Eliminates man-in-the-middle attacks
-- **SSL certificate validation:** Automatic certificate checking
-- **Secure repositories:** Ubuntu archive, Dell, Proxmox all use HTTPS
-- **No HTTP fallbacks:** No insecure download methods
-
-### 3. SSH Hardening
-- **Key-only authentication:** Password login disabled
-- **Secure ciphers:** Modern encryption algorithms only
-- **Fail2ban protection:** Automated intrusion prevention
-- **Custom SSH configuration:** Hardened sshd_config
-
-### 4. System Security
-- **Firewall configuration:** Automated iptables rules
-- **Audit logging:** auditd with custom rules
-- **SIEM integration:** Wazuh agent deployment
-- **Compliance scanning:** SCAP-STIG automated checks
-
-### 5. Error Handling
-- **Bash strict mode:** `set -euo pipefail` prevents errors
-- **Centralized logging:** All operations logged with timestamps
-- **Graceful failures:** Proper cleanup on errors
-- **Line-level debugging:** Error reporting with line numbers
-
-## Security Testing
-
-### Automated Security Validation
-```bash
-# Run security test suite
-./tests/run-tests.sh security
-
-# Specific security tests
-./tests/security/https-enforcement.sh
-```
-
-### Security Test Categories
-1. **HTTPS Enforcement:** Validates all URLs use HTTPS
-2. **Deployment Security:** Checks for secure deployment methods
-3. **SSL Certificate Validation:** Tests certificate authenticity
-4. **Permission Validation:** Verifies proper file permissions
-
-## Threat Model
-
-### Mitigated Threats
-- **Supply Chain Attacks:** Git-based deployment with review
-- **Man-in-the-Middle:** HTTPS-only downloads
-- **Privilege Escalation:** Proper permission models
-- **Unauthorized Access:** SSH hardening and key management
-
-### Remaining Risks
-- **Secrets in Repository:** SSH keys stored in git (planned for removal)
-- **No Integrity Verification:** Downloads lack checksum validation
-- **No Backup/Recovery:** No rollback capability implemented
-
-## Security Recommendations
-
-### High Priority
-1. **Implement Secrets Management**
-   - Remove SSH keys from repository
-   - Use Bitwarden/Vault for secret storage
-   - Implement key rotation procedures
-
-2. **Add Download Integrity Verification**
-   - SHA256 checksum validation for all downloads
-   - GPG signature verification where available
-   - Fail-safe on integrity check failures
-
-3. **Enhance Audit Logging**
-   - Centralized log collection
-   - Real-time security monitoring
-   - Automated threat detection
-
-### Medium Priority
-1. **Configuration Backup**
-   - System state snapshots before changes
-   - Rollback capability for failed deployments
-   - Configuration drift detection
-
-2. **Network Security**
-   - VPN-based deployment (where applicable)
-   - Network segmentation for management
-   - Encrypted communication channels
-
-## Compliance
-
-### Security Standards
-- **CIS Benchmarks:** Automated compliance checking
-- **STIG Guidelines:** SCAP-based validation
-- **Industry Best Practices:** Following NIST cybersecurity framework
-
-### Audit Requirements
-- **Change Tracking:** All modifications logged
-- **Access Control:** Permission-based system access
-- **Vulnerability Management:** Regular security assessments
-
-## Incident Response
-
-### Security Event Handling
-1. **Detection:** Automated monitoring and alerting
-2. **Containment:** Immediate isolation procedures
-3. **Investigation:** Log analysis and forensics
-4. **Recovery:** System restoration procedures
-5. **Lessons Learned:** Process improvement
-
-### Contact Information
-- **Security Team:** [To be defined]
-- **Incident Response:** [To be defined]
-- **Escalation Path:** [To be defined]
-
-## Security Development Lifecycle
-
-### Code Review Process
-1. **Static Analysis:** Automated security scanning
-2. **Peer Review:** Manual code inspection
-3. **Security Testing:** Automated security test suite
-4. **Approval:** Security team sign-off
-
-### Deployment Security
-1. **Pre-deployment Validation:** Security test execution
-2. **Secure Deployment:** Authorized personnel only
-3. **Post-deployment Verification:** Security configuration validation
-4. **Monitoring:** Continuous security monitoring
-
-## Security Tools and Integrations
-
-### Current Tools
-- **Wazuh:** SIEM and security monitoring
-- **Lynis:** Security auditing
-- **auditd:** System call auditing
-- **Fail2ban:** Intrusion prevention
-
-### Planned Integrations
-- **Vault/Bitwarden:** Secrets management
-- **OSSEC:** Host-based intrusion detection
-- **Nessus/OpenVAS:** Vulnerability scanning
-- **ELK Stack:** Log aggregation and analysis
-
-## Vulnerability Management
-
-### Vulnerability Scanning
-- **Regular scans:** Monthly vulnerability assessments
-- **Automated patching:** Security update automation
-- **Exception handling:** Risk-based patch management
-- **Reporting:** Executive security dashboards
-
-### Disclosure Process
-1. **Internal Discovery:** Report to security team
-2. **Assessment:** Risk and impact evaluation
-3. **Remediation:** Patch development and testing
-4. **Deployment:** Coordinated security updates
-5. **Verification:** Post-patch validation
-
-## Security Metrics
-
-### Key Performance Indicators
-- **Deployment Success Rate:** Percentage of successful secure deployments
-- **Vulnerability Response Time:** Time to patch critical vulnerabilities
-- **Security Test Coverage:** Percentage of code covered by security tests
-- **Incident Response Time:** Time to detect and respond to security events
-
-### Monitoring and Reporting
-- **Real-time Dashboards:** Security status monitoring
-- **Executive Reports:** Monthly security summaries
-- **Compliance Reports:** Quarterly compliance assessments
-- **Trend Analysis:** Security posture improvement tracking
-
-## Contact and Support
-
-For security-related questions or incidents:
-- **Repository Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
-- **Community Discussion:** https://community.turnsys.com/c/chieftechnologyandproductofficer/26
-- **Security Team:** [Contact information to be added]
-
-## Security Updates
-
-This document is updated as security features are implemented and threats evolve. Last updated: July 14, 2025.

docs/server-build/TSYS-2FA-GUIDE.md

@@ -1,329 +1,10 @@
-# TSYS Two-Factor Authentication Implementation Guide
+# docs/server-build/TSYS-2FA-GUIDE.md
-
+
-## Overview
+> **Documentation moved to Discourse — the canonical source of truth.**
-
+>
-This guide provides complete instructions for implementing and managing two-factor authentication (2FA) on TSYS servers using Google Authenticator (TOTP).
+> **End-user guide for 2FA setup**
-
+>
-## What This Implementation Provides
+> **Read it here:** https://community.turnsys.com/t/303
-
+>
-### Services Protected by 2FA
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-- **SSH Access:** Requires SSH key + 2FA token
+> Do not update content here — edit the Discourse wiki topic instead.*
-- **Cockpit Web Interface:** Requires password + 2FA token
-- **Webmin Administration:** Requires password + 2FA token (if installed)
-
-### Security Features
-- **Time-based One-Time Passwords (TOTP):** Standard 6-digit codes
-- **Backup Codes:** Emergency access codes
-- **Gradual Rollout:** Optional nullok mode for phased deployment
-- **Configuration Backup:** Automatic backup of all configs
-
-## Implementation Steps
-
-### Step 1: Run the 2FA Setup Script
-```bash
-# Navigate to the security modules directory
-cd provisioning/Modules/Security
-
-# Run the 2FA setup script as root
-sudo bash secharden-2fa.sh
-```
-
-### Step 2: Validate Installation
-```bash
-# Run 2FA validation tests
-./Project-Tests/security/2fa-validation.sh
-
-# Run specific 2FA security test
-./Project-Tests/run-tests.sh security
-```
-
-### Step 3: Setup Individual Users
-For each user that needs 2FA access:
-
-```bash
-# Check setup instructions
-cat /home/username/2fa-setup-instructions.txt
-
-# Run user setup script
-sudo /tmp/setup-2fa-username.sh
-```
-
-### Step 4: Test 2FA Access
-1. **Test SSH access** from another terminal
-2. **Test Cockpit access** via web browser
-3. **Test Webmin access** if installed
-
-## User Setup Process
-
-### Installing Authenticator Apps
-Users need one of these apps on their phone:
-- **Google Authenticator** (Android/iOS)
-- **Authy** (Android/iOS)
-- **Microsoft Authenticator** (Android/iOS)
-- **1Password** (with TOTP support)
-
-### Setting Up 2FA for a User
-1. **Run setup script:**
-   ```bash
-   sudo /tmp/setup-2fa-username.sh
-   ```
-
-2. **Follow prompts:**
-   - Answer "y" to update time-based token
-   - Scan QR code with authenticator app
-   - Save emergency backup codes securely
-   - Answer "y" to remaining security questions
-
-3. **Test immediately:**
-   ```bash
-   # Test SSH from another terminal
-   ssh username@server-ip
-   # You'll be prompted for 6-digit code
-   ```
-
-## Configuration Details
-
-### SSH Configuration Changes
-File: `/etc/ssh/sshd_config`
-```
-ChallengeResponseAuthentication yes
-UsePAM yes
-AuthenticationMethods publickey,keyboard-interactive
-```
-
-### PAM Configuration
-File: `/etc/pam.d/sshd`
-```
-auth required pam_google_authenticator.so nullok
-```
-
-### Cockpit Configuration
-File: `/etc/cockpit/cockpit.conf`
-```
-[WebService]
-LoginTitle = TSYS Server Management
-LoginTo = 300
-RequireHost = true
-
-[Session]
-Banner = /etc/cockpit/issue.cockpit
-IdleTimeout = 15
-```
-
-### Webmin Configuration
-File: `/etc/webmin/miniserv.conf`
-```
-twofactor_provider=totp
-twofactor=1
-```
-
-## Security Considerations
-
-### Gradual vs Strict Enforcement
-
-#### Gradual Enforcement (Default)
-- Uses `nullok` option in PAM
-- Users without 2FA can still log in
-- Allows phased rollout
-- Good for initial deployment
-
-#### Strict Enforcement
-- Remove `nullok` from PAM configuration
-- All users must have 2FA configured
-- Immediate security enforcement
-- Risk of lockout if misconfigured
-
-### Backup and Recovery
-
-#### Emergency Access
-- **Backup codes:** Generated during setup
-- **Root access:** Can disable 2FA if needed
-- **Console access:** Physical/virtual console bypasses SSH
-
-#### Configuration Backup
-- Automatic backup to `/root/backup/2fa-TIMESTAMP/`
-- Includes all modified configuration files
-- Can be restored if needed
-
-## Troubleshooting
-
-### Common Issues
-
-#### 1. User Cannot Generate QR Code
-```bash
-# Ensure qrencode is installed
-sudo apt-get install qrencode
-
-# Re-run user setup
-sudo /tmp/setup-2fa-username.sh
-```
-
-#### 2. SSH Connection Fails
-```bash
-# Check SSH service status
-sudo systemctl status sshd
-
-# Test SSH configuration
-sudo sshd -t
-
-# Check logs
-sudo journalctl -u sshd -f
-```
-
-#### 3. 2FA Code Not Accepted
-- **Check time synchronization** on server and phone
-- **Verify app setup** - rescan QR code if needed
-- **Try backup codes** if available
-
-#### 4. Locked Out of Server
-```bash
-# Access via console (physical/virtual)
-# Disable 2FA temporarily
-sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd
-sudo systemctl restart sshd
-```
-
-### Debug Commands
-
-```bash
-# Check 2FA status
-./Project-Tests/security/2fa-validation.sh
-
-# Check SSH configuration
-sudo sshd -T | grep -E "(Challenge|PAM|Authentication)"
-
-# Check PAM configuration
-cat /etc/pam.d/sshd | grep google-authenticator
-
-# Check user 2FA status
-ls -la ~/.google_authenticator
-```
-
-## Management and Maintenance
-
-### Adding New Users
-1. Ensure user account exists
-2. Run setup script for new user
-3. Provide setup instructions
-4. Test access
-
-### Removing User 2FA
-```bash
-# Remove user's 2FA configuration
-sudo rm /home/username/.google_authenticator
-
-# User will need to re-setup 2FA
-```
-
-### Disabling 2FA System-Wide
-```bash
-# Restore original configurations
-sudo cp /root/backup/2fa-*/sshd_config.bak /etc/ssh/sshd_config
-sudo cp /root/backup/2fa-*/pam.d.bak/sshd /etc/pam.d/sshd
-sudo systemctl restart sshd
-```
-
-### Updating 2FA Configuration
-```bash
-# Re-run setup script
-sudo bash secharden-2fa.sh
-
-# Validate changes
-./Project-Tests/security/2fa-validation.sh
-```
-
-## Best Practices
-
-### Deployment Strategy
-1. **Test in non-production** environment first
-2. **Enable gradual rollout** (nullok) initially
-3. **Train users** on 2FA setup process
-4. **Test emergency procedures** before strict enforcement
-5. **Monitor logs** for authentication issues
-
-### Security Recommendations
-- **Enforce strict mode** after successful rollout
-- **Regular backup code rotation**
-- **Monitor failed authentication attempts**
-- **Document emergency procedures**
-- **Regular security audits**
-
-### User Training
-- **Provide clear instructions**
-- **Demonstrate setup process**
-- **Explain backup code importance**
-- **Test login process with users**
-- **Establish support procedures**
-
-## Monitoring and Logging
-
-### Authentication Logs
-```bash
-# SSH authentication logs
-sudo journalctl -u sshd | grep -i "authentication"
-
-# PAM authentication logs
-sudo journalctl | grep -i "pam_google_authenticator"
-
-# Failed login attempts
-sudo journalctl | grep -i "failed"
-```
-
-### Security Monitoring
-- Monitor for repeated failed 2FA attempts
-- Alert on successful logins without 2FA (during gradual rollout)
-- Track user 2FA setup completion
-- Monitor for emergency access usage
-
-## Integration with Existing Systems
-
-### LDAP/Active Directory
-- 2FA works with existing authentication systems
-- Users still need local 2FA setup
-- Consider centralized 2FA solutions for large deployments
-
-### Monitoring Systems
-- LibreNMS: Will continue to work with SNMP
-- Wazuh: Will log 2FA authentication events
-- Cockpit: Enhanced with 2FA protection
-
-### Backup Systems
-- Ensure backup procedures account for 2FA
-- Test restore procedures with 2FA enabled
-- Document emergency access procedures
-
-## Support and Resources
-
-### Files Created by Setup
-- `/tmp/setup-2fa-*.sh` - User setup scripts
-- `/home/*/2fa-setup-instructions.txt` - User instructions
-- `/root/backup/2fa-*/` - Configuration backups
-
-### Validation Tools
-- `./Project-Tests/security/2fa-validation.sh` - Complete 2FA validation
-- `./Project-Tests/run-tests.sh security` - Security test suite
-
-### Emergency Contacts
-- System Administrator: [Contact Info]
-- Security Team: [Contact Info]
-- 24/7 Support: [Contact Info]
-
-## Compliance and Audit
-
-### Security Benefits
-- Significantly reduces risk of unauthorized access
-- Meets multi-factor authentication requirements
-- Provides audit trail of authentication events
-- Complies with security frameworks (NIST, ISO 27001)
-
-### Audit Trail
-- All authentication attempts logged
-- 2FA setup events recorded
-- Configuration changes tracked
-- Emergency access documented
-
----
-
-**Last Updated:** July 14, 2025  
-**Version:** 1.0  
-**Author:** TSYS Security Team

docs/server-build/tailscale.md

@@ -1,112 +1,10 @@
-# Tailscale vs. Managed DNS — Architecture Analysis
+# docs/server-build/tailscale.md
 
-> **Status:** **RESOLVED.** The pfv-netinfra-01/02 pair now runs production
+> **Documentation moved to Discourse — the canonical source of truth.**
-> Technitium DNS with all `knel.net` records replicated from tailscale-router
+>
-> via the DNS cluster setup. Both LAN IPs serve authoritative records for
+> **Tailscale vs managed DNS analysis (resolved)**
-> `knel.net` and recurse externally. This document records the original
+>
-> conflict, how it was resolved, and the recommended client configuration.
+> **Read it here:** https://community.turnsys.com/t/306
-
+>
-## 1. Executive summary
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-Every host in this build runs the Tailscale client, and Tailscale's MagicDNS
-manages `/etc/resolv.conf` by default (pointing at `100.100.100.100`). This
-previously conflicted with a managed `resolv.conf` pointing at the LAN
-resolvers. The root cause was that the LAN Technitium instances did not have
-the `knel.net` zone populated — **that is now fixed.**
-
-The pfv-netinfra-01/02 pair now serves identical, authoritative `knel.net`
-records (replicated from production via [`dns-cluster-setup/`](../dns-cluster-setup/README.md)).
-Both LAN IPs resolve `knel.net` device names and recurse externally. The
-managed `resolv.conf` is now safe to deploy.
-
-**Recommendation:** Deploy the managed `resolv.conf` (`.252`/`.253`) on hosts
-where you want tunnel-independent DNS. Leave Tailscale managing DNS on hosts
-where MagicDNS device names must resolve without a LAN path (e.g. laptops off
--network). See [§5](#5-recommendation) for details.
-
-## 2. How name resolution works today (post-cluster-setup)
-
-Probed from `sectestbed-sandbox` (192.168.3.50) after the DNS cluster was
-deployed:
-
-| Query path | External name (`github.com`) | `knel.net` device name (`pfv-netinfra-01.knel.net`) |
-|---|---|---|
-| Via Tailscale resolver (`100.100.100.100`) | resolves | resolves → `100.70.181.72` (Tailscale CGNAT) |
-| Direct `dig @192.168.3.252` (Technitium primary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
-| Direct `dig @192.168.3.253` (Technitium secondary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
-
-**Both LAN resolvers now serve `knel.net` records identically.** The
-Technitium zone is no longer stale — it was replicated from production
-(tailscale-router) as part of the DNS cluster setup.
-
-### What changed
-
-Previously (before the DNS cluster setup), querying the LAN IPs returned
-NXDOMAIN for `knel.net` device names because the Technitium `knel.net` zone
-was empty (SOA serial `2025062313`, dated 2025-06-23). After replicating
-production config to both netinfra hosts, all 124 zones — including
-`knel.net` with all current device records — are served authoritatively on
-both `.252` and `.253`.
-
-## 3. The DNS server pair
-
-| Host | IP | Role | Services |
-|------|----|------|----------|
-| pfv-netinfra-01 | 192.168.3.252 | **Primary** | Technitium (authoritative, port 5300) + Pi-hole (recursive, port 53) |
-| pfv-netinfra-02 | 192.168.3.253 | **Secondary** | Technitium (replicated via rsync, port 5300) + Pi-hole (recursive, port 53) |
-
-Zone replication is rsync-based (every 60s via systemd timer) because
-Technitium's AXFR uses port 53, which is occupied by Pi-hole on these hosts.
-See [`dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) for
-full details.
-
-## 4. NTP (fully resolved)
-
-NTP is independent of DNS: `provisioning/ConfigFiles/NTP/ntp.conf` points
-directly at the LAN IPs with no DNS dependency:
-
-```
-server 192.168.3.252 iburst
-server 192.168.3.253 iburst
-```
-
-Both servers respond with stratum 2/3. The client config uses `restrict`
-rules (not `interface listen`) to avoid the loopback-binding bug that
-prevented sync. This is safe under both Tailscale-managed and LAN-pinned
-resolver configurations.
-
-## 5. Recommendation
-
-### On fixed servers (always on-LAN)
-
-**Deploy the managed `resolv.conf`** (`provisioning/ConfigFiles/Resolv/`):
-- Points at `.252`/`.253` with failover
-- `knel.net` records resolve on both servers
-- External names recurse on both servers
-- DNS survives `tailscaled` outages (unlike Tailscale-managed DNS)
-
-To prevent Tailscale from overwriting the managed file:
-```bash
-tailscale up --accept-dns=false
-```
-
-### On laptops / roaming hosts
-
-**Let Tailscale manage DNS** (default `accept-dns=true`):
-- MagicDNS resolves `knel.net` device names via the tunnel
-- No dependency on LAN reachability
-- Accept the `tailscaled` dependency (if the tunnel is down, you're off-network anyway)
-
-## 6. Known items / future work
-
-1. **Pi-hole upstream configuration.** Pi-hole on both hosts should forward
-   to the local Technitium instance (port 5300) for `knel.net` and to an
-   external resolver for everything else. Verify this is configured on both
-   nodes.
-2. **Zone transfer via AXFR.** Currently using rsync because Technitium's
-   AXFR expects port 53. If Technitium's listen port can be changed, or
-   Pi-hole can be configured to proxy AXFR, the rsync timer could be
-   replaced with native DNS zone transfer.
-3. **`accept-dns=false` automation.** The provisioning code should set
-   `--accept-dns=false` on Tailscale during setup (after deploying the
-   managed `resolv.conf`) so Tailscale doesn't overwrite it on reboot.

hooks/ticket-gate.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# hooks/ticket-gate.sh — enforce ticket-first work policy
+#
+# Blocks modifying operations until an active ticket is established.
+# The agent sets the active ticket via:  echo '#NNN' > .crush/active-ticket
+# And clears it when done:               > .crush/active-ticket
+#
+# Exempts read-only and management commands (so you can create tickets,
+# run audits, check status, etc.).
+set -euo pipefail
+
+TICKET_FILE="${CRUSH_PROJECT_DIR}/.crush/active-ticket"
+TOOL="${CRUSH_TOOL_NAME:-}"
+CMD="${CRUSH_TOOL_INPUT_COMMAND:-}"
+
+# Read-only tools — always allowed
+case "$TOOL" in
+  view|ls|grep|glob|agent|sourcegraph|fetch|agentic_fetch|download|lsp_diagnostics|lsp_symbols|lsp_definition|lsp_references|lsp_call_hierarchy|crush_info|crush_logs|question|todos)
+    exit 0
+    ;;
+esac
+
+# For bash tool: exempt read-only and management commands
+if [ "$TOOL" = "bash" ]; then
+  # Ticket/doc/dns management — always allowed
+  case "$CMD" in
+    *"redmine-cli"*|*"discourse-cli"*|*"dns-cli"*|*"technitium"*) exit 0 ;;
+  esac
+  # Read-only git
+  case "$CMD" in
+    *"git status"*|*"git log"*|*"git diff"*|*"git show"*|*"git branch"*) exit 0 ;;
+  esac
+  # Repo hygiene scripts
+  case "$CMD" in
+    *"check-rules"*|*"setup-hooks"*|*"shellcheck"*|*"run-tests"*) exit 0 ;;
+  esac
+  # Monitoring/probe commands
+  case "$CMD" in
+    *"tailscale status"*|*"access-matrix"*) exit 0 ;;
+  esac
+  # Setting/clearing the active ticket
+  case "$CMD" in
+    *active-ticket*) exit 0 ;;
+  esac
+fi
+
+# For edit/write: exempt policy/hook files (these ARE the policy)
+FILE_PATH="${CRUSH_TOOL_INPUT_FILE_PATH:-}"
+case "$FILE_PATH" in
+  */AGENTS.md|*/check-rules.sh|*/crush.json|*/hooks/*)
+    if [ "$TOOL" = "write" ] || [ "$TOOL" = "edit" ] || [ "$TOOL" = "multiedit" ]; then
+      exit 0
+    fi
+    ;;
+esac
+
+# Check for active ticket
+if [ -f "$TICKET_FILE" ] && [ -s "$TICKET_FILE" ]; then
+  TICKET=$(cat "$TICKET_FILE")
+  printf '{"context":"Active ticket: %s"}\n' "$TICKET"
+  exit 0
+fi
+
+# No active ticket — block
+cat >&2 <<'MSG'
+TICKET GATE: No active ticket set.
+
+This project requires ticket-governed work (AGENTS.md Agent Authority).
+Before modifying systems or code, set the active ticket:
+
+  echo '#NNN' > .crush/active-ticket
+
+If no ticket exists yet, create one first (redmine-cli create), then set it.
+Clear the ticket when work is complete:
+
+  > .crush/active-ticket
+MSG
+exit 2

k8s/README.md

@@ -1,55 +1,10 @@
-# k8s/ — pfv-k8s Cluster Setup Scripts
+# k8s/README.md
 
-Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
+> **Documentation moved to Discourse — the canonical source of truth.**
-All cluster communication goes over Tailscale IPs — no LAN traffic.
+>
-
+> **k3s cluster setup scripts: wipe, bootstrap, taint, verify**
-## Current State
+>
-
+> **Read it here:** https://community.turnsys.com/t/305
-3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-| Node | Tailscale IP | Role | Tainted |
+> Do not update content here — edit the Discourse wiki topic instead.*
-|------|-------------|------|---------|
-| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
-| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
-| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
-
-## Scripts
-
-| Script | Purpose |
-|--------|---------|
-| [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
-| [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). |
-| [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
-| [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). |
-| [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. |
-| [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
-| [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. |
-
-## Usage
-
-```bash
-# Full bootstrap from scratch:
-bash k8s/wipe.sh
-bash k8s/install-cp.sh
-bash k8s/post-setup.sh
-bash k8s/verify.sh
-
-# Access the cluster:
-export KUBECONFIG=~/.kube/config.pfv-k8s
-kubectl get nodes
-```
-
-## Design Decisions
-
-- **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified.
-  Talos architecture is documented in [`docs/`](docs/) for when
-  that requirement comes online. k3s on stock Debian is simpler to operate.
-- **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and
-  `--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node
-  status or certificate.
-- **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of
-  Tailscale's WireGuard. Double-encrypted, but functional and reliable.
-- **NoSchedule taint:** All 3 cnodes are tainted so no user workloads
-  schedule on the control plane. Only system components (CoreDNS,
-  metrics-server, flannel, kube-proxy) with built-in tolerations run here.
-- **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure.

k8s/docs/ARCHITECTURE.md

@@ -1,673 +1,10 @@
-# pfv-k8s Target Architecture (Talos)
+# k8s/docs/ARCHITECTURE.md
-
+
-> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
+> **Documentation moved to Discourse — the canonical source of truth.**
-> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
+>
-> Proxmox audit).
+> **k8s target architecture: control plane, network, identity, storage, DR**
-
+>
-**Last updated:** 2026-07-28
+> **Read it here:** https://community.turnsys.com/t/305
-
+>
----
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## Table of Contents
-
-1. [High-Level Architecture](#1-high-level-architecture)
-2. [Control Plane](#2-control-plane)
-3. [Network Topology](#3-network-topology)
-4. [CNI: Cilium](#4-cni-cilium)
-5. [Identity and Trust](#5-identity-and-trust)
-6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
-7. [Storage Integration](#7-storage-integration)
-8. [Local Image Registry](#8-local-image-registry)
-9. [Bootstrap Procedure](#9-bootstrap-procedure)
-10. [Disaster Recovery](#10-disaster-recovery)
-11. [Migration from Current State](#11-migration-from-current-state)
-
----
-
-## 1. High-Level Architecture
-
-```mermaid
-flowchart TB
-    subgraph RESIDENCE["Residence — Proxmox LAN"]
-        subgraph CP["Talos Control Plane (3 cnodes)"]
-            C1[cnode1<br/>tsys9 · local-SSD]
-            C2[cnode2<br/>tsys9 · local-SSD]
-            C3[cnode3<br/>tsys1 · local-HDD]
-        end
-        subgraph WP["Talos Worker Plane"]
-            W3[wnode-tsys3<br/>NVMe · 28GB]
-            W5[wnode-tsys5<br/>NVMe · 32-64GB]
-            W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
-            W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
-            W9[wnode-tsys9<br/>local-SSD · 4-8GB]
-        end
-        ETCD[(etcd<br/>raft, mTLS)]
-        REG[(Harbor registry<br/>on D3 SSD · tsys5)]
-        BASTION[tailscale-router VM<br/>subnet router]
-    end
-
-    subgraph TAILNET["Tailscale overlay"]
-        OP[Operator devices]
-    end
-
-    subgraph CLOUDRON["Cloudron production — Reston VA"]
-        KC[Keycloak OIDC IdP]
-    end
-
-    C1 ---|mTLS LAN| ETCD
-    C2 ---|mTLS LAN| ETCD
-    C3 ---|mTLS LAN| ETCD
-    CP -->|pull images| REG
-    WP -->|pull images| REG
-
-    OP -->|Talos API :50000<br/>via subnet route| BASTION
-    BASTION -.->|LAN| CP
-    CP -->|OIDC| KC
-    WP -->|OIDC| KC
-
-    classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
-    classDef infra fill:#0f3460,stroke:#e94560,color:#fff
-    classDef external fill:#16213e,stroke:#533483,color:#fff
-    class CP,WP,ETCD talos
-    class REG,BASTION infra
-    class OP,KC,EXTERNAL external
-```
-
-### Design principles
-
-1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
-   Strongest posture for ITAR/classified.
-2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
-   advertises the cluster LAN subnet. Operator reaches Talos API from
-   anywhere.
-3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
-   for etcd). Workers boot from local disk where available; NFS for bulk
-   data only.
-4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
-   of the Talos host cluster.
-5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
-   tokens. No long-lived static credentials for humans.
-
----
-
-## 2. Control Plane
-
-### 2.1 Recommendation: 3 cnodes (down from 5)
-
-| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
-|--------|--------|-------------------|-----------------|---------------|
-| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
-| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
-
-For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
-failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
-quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
-host capacity) are better spent on tenant worker allocations.
-
-**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
-tolerance on the control plane, keep 5. Otherwise 3.
-
-### 2.2 Cnode placement
-
-Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
-**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
-
-| cnode | Host | Boot disk | Type | Why |
-|-------|------|-----------|------|-----|
-| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
-| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
-| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
-
-**Quorum survival:**
-
-| Failure | cnodes lost | Quorum OK? |
-|---------|-------------|------------|
-| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
-| tsys1 host dies | cnode3 | YES (2 of 3) |
-| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
-
-**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
-loses quorum. Alternative: spread cnodes across 3 different hosts. See
-"open question" at end of this section.
-
-### 2.3 Machine config strategy
-
-Talos nodes are configured by **machine configs** (YAML). Two flavors:
-
-- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
-  controller-manager, API server.
-- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
-
-Strategy for this cluster:
-
-1. **One shared `talosconfig`** (client identity) — stored in 1Password
-   and in the Proxmox Backup Server (PBS) encrypted backup target.
-2. **Per-node machine config patches** — small patches on top of the base
-   `controlplane.yaml` / `worker.yaml` for node-specific settings:
-   - Hostname
-   - Network interface + IP (DHCP or static — recommend static for cnodes)
-   - Schematic image digest (pinned Talos version)
-   - System extensions (e.g., `tailscale` — only if running Pattern A
-     instead of recommended Pattern C)
-3. **All machine configs in Git** under a future `k8s/talos-configs/`
-   directory. Secrets are templated in at apply-time from 1Password / sops.
-
-```mermaid
-flowchart LR
-    BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
-    BASE --> PATCH2[patch: cnode2]
-    BASE --> PATCH3[patch: cnode3]
-    BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
-    PATCH1 --> APPLY1[talosctl apply]
-    PATCH2 --> APPLY2[talosctl apply]
-    PATCH3 --> APPLY3[talosctl apply]
-    PATCHW --> APPLYW[talosctl apply]
-```
-
-### 2.4 Open question: cnode host spread
-
-If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
-cluster, tsys9 is brand-new hardware, single digit annual failure
-probability), the layout in §2.2 is fine.
-
-If not, alternative spread across 3 hosts:
-
-| cnode | Host | Boot disk |
-|-------|------|-----------|
-| cnode1 | tsys9 | local-lvm SSD |
-| cnode2 | tsys1 | local-lvm HDD |
-| cnode3 | tsys3 | local-lvm NVMe |
-
-tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
-[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
-would make an excellent etcd disk.
-
-**This 3-host spread survives any single host failure with quorum intact.
-Recommended.**
-
----
-
-## 3. Network Topology
-
-### 3.1 Zones
-
-```mermaid
-flowchart TB
-    subgraph INTERNET["Internet"]
-        FIBER[Gigabit symmetric fiber]
-    end
-
-    subgraph RESLAN["Residence LAN 192.168.x.x/24"]
-        subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
-            CNODES[Cnodes 192.168.3.x]
-            WNODES[Wnodes 192.168.3.x]
-        end
-        BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
-        REG[Harbor registry<br/>192.168.3.x]
-        STORAGE[NFS servers<br/>tsys4, tsys5]
-    end
-
-    subgraph TSNET["Tailscale 100.x.x.x/8"]
-        OPS[Operator devices]
-        KC[Keycloak<br/>via Cloudron prod]
-    end
-
-    FIBER --> BASTION
-    BASTION <-. subnet route .-> CLUSTERNET
-    OPS -->|TCP 50000 talos API| BASTION
-    BASTION -->|LAN forward| CNODES
-    CNODES -->|LAN mTLS| WNODES
-    CNODES -->|OIDC HTTPS| KC
-    WNODES -->|pull images| REG
-    WNODES -->|bulk data IO| STORAGE
-    CNODES -->|pull images| REG
-```
-
-### 3.2 Address plan (suggested)
-
-Reserve a small contiguous block in the residence LAN for cluster nodes:
-
-| Role | Range | Count |
-|------|-------|-------|
-| Cnodes | `192.168.3.31-33` | 3 |
-| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
-| Bastion | existing `tailscale-router` | 1 |
-| Registry | `192.168.3.50` | 1 (Harbor) |
-
-**Static IPs are strongly recommended for cnodes** (etcd cluster membership
-is hostname-based; stable IPs make `talosctl` targeting simple). Workers
-can DHCP.
-
-### 3.3 Firewall posture
-
-Each cnode/wnode has:
-
-- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
-  cnodes only), plus CNI ports (varies by CNI — see §4).
-- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
-- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
-  firewall for these IPs. ITAR workloads must not be able to phone home.
-
-The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
-via the subnet route.
-
----
-
-## 4. CNI: Cilium
-
-**Recommendation: Cilium** (eBPF-based CNI).
-
-| Property | Why it matters here |
-|----------|---------------------|
-| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
-| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
-| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
-| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
-| **Talos integration** | First-class. Talos docs document the install path. |
-
-Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
-enabled. Default-deny NetworkPolicy applied per namespace.
-
----
-
-## 5. Identity and Trust
-
-### 5.1 Trust flow
-
-```mermaid
-sequenceDiagram
-    autonumber
-    participant Human as Operator
-    participant TAIL as Tailscale
-    participant BAST as Bastion
-    participant TALOS as Talos API :50000
-    participant KC as Keycloak (Cloudron)
-    participant K8S as Kubernetes API :6443
-
-    Human->>TAIL: Authenticate (device + SSO)
-    TAIL-->>Human: Tailnet IP
-    Human->>BAST: Reach bastion via tailnet
-    BAST->>TALOS: Forward to LAN node :50000
-    Human->>KC: OIDC login (browser)
-    KC-->>Human: Bearer token (short-lived)
-    Human->>TALOS: talosctl (mTLS with client cert)
-    Human->>K8S: kubectl --oidc (Keycloak token)
-    K8S->>KC: Validate token (introspection)
-    KC-->>K8S: Valid + claims
-    K8S-->>Human: Authorized response
-```
-
-### 5.2 Two distinct identity layers
-
-| Layer | Mechanism | Audience |
-|-------|-----------|----------|
-| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
-| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
-
-The **Talos secrets bundle** is the root of trust for the cluster. Lose it
-and you cannot operate the cluster; an attacker with it owns the cluster.
-Storage:
-
-1. **Primary:** 1Password (or equivalent) — operator-accessible.
-2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
-3. **NOT in Git.** Machine configs go in Git; secrets stay out.
-
-### 5.3 Keycloak client configuration
-
-On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
-
-- **Authorization Code + PKCE flow** (no implicit, no password).
-- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
-  Omni/Rancher URLs if/when those are added.
-- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
-  `k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
-
----
-
-## 6. Tenant Isolation (vcluster)
-
-### 6.1 Why vcluster
-
-[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
-plane** (API server, scheduler, controller-manager, etcd) inside a namespace
-of the host cluster. Tenant workloads run on the host's worker nodes but
-are isolated by:
-
-- Separate API server (tenant cannot see host cluster objects).
-- Separate RBAC and admission control.
-- Separate network policies (per-namespace).
-- Separate resource quotas.
-
-This aligns with the user's per-tenant plan from
-[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
-
-### 6.2 Tenant registry
-
-| Tenant | Compliance | Workload example | vcluster name |
-|--------|-----------|------------------|---------------|
-| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
-| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
-| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
-| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
-
-```mermaid
-flowchart TB
-    subgraph HOST["Talos host cluster"]
-        CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
-        subgraph NS["Host cluster namespaces"]
-            NS_RR[ns: vc-rackrental]
-            NS_SO[ns: vc-suborbital-open]
-            NS_SI[ns: vc-suborbital-itar]
-            NS_SLP[ns: vc-slp]
-        end
-    end
-
-    subgraph VRR["vcluster: vc-rackrental"]
-        API_RR[k8s API + etcd]
-    end
-    subgraph VSO["vcluster: vc-suborbital-open"]
-        API_SO[k8s API + etcd]
-    end
-    subgraph VSI["vcluster: vc-suborbital-itar"]
-        API_SI[k8s API + etcd]
-    end
-    subgraph VSLP["vcluster: vc-slp"]
-        API_SLP[k8s API + etcd]
-    end
-
-    CP --> NS_RR & NS_SO & NS_SI & NS_SLP
-    NS_RR --> API_RR
-    NS_SO --> API_SO
-    NS_SI --> API_SI
-    NS_SLP --> API_SLP
-
-    classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
-    class NS_SI,API_SI itar
-```
-
-### 6.3 ITAR enforcement at host layer
-
-For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
-controls:
-
-- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
-  ITAR workloads (subset of wnodes, marked in node labels).
-- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
-  namespace. Allow only explicit destinations (registry, NFS for ITAR
-  data tier, Keycloak).
-- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
-  `D3-itar` on tsys5) that no other tenant can mount.
-- **Audit:** Hubble flows + auditd on the host worker nodes capture all
-  access to ITAR data.
-
-Rancher (or Sidero Omni) sits above this, presenting each tenant's
-vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
-access per tenant group claim.
-
----
-
-## 7. Storage Integration
-
-Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
-
-| StorageClass | Provisioner | Backing | Speed | Use |
-|--------------|------------|---------|-------|-----|
-| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
-| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
-| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
-
-### 7.1 CSI driver notes
-
-- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
-  (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
-  NFS server + base export path.
-- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
-  with one manifest. Uses wnode's kubelet root dir.
-
-### 7.2 ITAR data isolation
-
-The ITAR tenant should target a dedicated NFS export, not shared
-`nfs-hdd`. Recommended:
-
-- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
-  `nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
-  namespace via RBAC + NetworkPolicy.
-
----
-
-## 8. Local Image Registry
-
-### 8.1 Recommendation: Harbor on D3 SSD
-
-D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
-for a **Harbor** instance:
-
-| Property | Value |
-|----------|-------|
-| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
-| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
-| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
-| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
-
-### 8.2 Pull-through cache benefit
-
-Cluster nodes have zero internet egress (per §3.3). Without a local cache,
-image pulls fail. With Harbor as a pull-through cache:
-
-```mermaid
-sequenceDiagram
-    WNODE->>HARBOR: docker pull nginx:1.25
-    alt cache hit
-        HARBOR-->>WNODE: layer bytes (LAN-speed)
-    else cache miss
-        HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
-        DOCKERHUB-->>HARBOR: layer bytes
-        HARBOR-->>WNODE: layer bytes (cached for next time)
-    end
-```
-
-Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
-machine in the cluster with container-registry internet egress, and that
-egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
-ghcr.io).
-
-### 8.3 Supply-chain integrity (future)
-
-Harbor + Cosign lets you require that all images deployed to the ITAR
-tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
-control. Implementation deferred to a later session.
-
----
-
-## 9. Bootstrap Procedure
-
-### 9.1 One-time setup
-
-```mermaid
-sequenceDiagram
-    autonumber
-    participant OP as Operator
-    participant GIT as Git repo
-    participant ONEPW as 1Password
-    participant PBS as PBS (tsys4)
-    OP->>GIT: Clone PFVCluster repo
-    OP->>ONEPW: Generate Talos secrets bundle (offline)
-    ONEPW-->>OP: secrets.yaml
-    OP->>PBS: Backup secrets.yaml (encrypted)
-    OP->>GIT: Write machine configs (no secrets)
-```
-
-### 9.2 Provision first cnode (bootstrap)
-
-```mermaid
-sequenceDiagram
-    autonumber
-    participant OP as Operator
-    participant PX as Proxmox host
-    participant C1 as cnode1
-    participant ETCD as etcd (new)
-
-    OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
-    OP->>PX: qm start VMID
-    C1->>C1: Boots Talos (no config yet, "maintenance mode")
-    OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
-    C1->>C1: Applies config, restarts services
-    OP->>C1: talosctl bootstrap
-    C1->>ETCD: Initialize single-node raft
-    ETCD-->>C1: ready
-    OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
-    OP->>C1: talosctl etcd snapshot (initial backup → PBS)
-```
-
-### 9.3 Add second and third cnodes
-
-```mermaid
-sequenceDiagram
-    autonumber
-    participant OP as Operator
-    participant PX as Proxmox host
-    participant C2 as cnode2
-    participant C3 as cnode3
-    participant C1 as cnode1 (existing)
-
-    OP->>PX: qm create + start cnode2 VM
-    C2->>C2: Boots Talos maintenance mode
-    OP->>C2: talosctl apply --patch cnode2.yaml
-    C2->>C1: Join etcd cluster
-    OP->>PX: qm create + start cnode3 VM
-    C3->>C3: Boots Talos maintenance mode
-    OP->>C3: talosctl apply --patch cnode3.yaml
-    C3->>C1: Join etcd cluster
-    Note over C1,C3: etcd now has 3/3 members → HA quorum
-```
-
-### 9.4 Post-bootstrap cluster configuration
-
-Once 3 cnodes are up and joined:
-
-1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
-2. **Install CSI drivers** — nfs-csi + local-path provisioner.
-3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
-4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
-5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
-6. **Apply default-deny NetworkPolicy** in all namespaces.
-7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
-8. **First etcd snapshot** + automated daily snapshot cron → PBS.
-
-### 9.5 Add workers
-
-Workers are simpler (no etcd):
-
-```mermaid
-sequenceDiagram
-    OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
-    WNODE->>WNODE: Boots maintenance mode
-    OP->>WNODE: talosctl apply --patch worker-X.yaml
-    WNODE->>C1: Kubelet registers with API server
-    C1-->>WNODE: Approved (auto via bootstrap token)
-    Note over WNODE: Joins cluster, becomes Ready
-```
-
----
-
-## 10. Disaster Recovery
-
-### 10.1 Backup strategy
-
-| Artifact | Frequency | Storage | Tool |
-|----------|-----------|---------|------|
-| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
-| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
-| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
-| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
-| **Harbor metadata** | Daily | PBS | Harbor built-in backup |
-
-### 10.2 Restore scenarios
-
-**Lost 1 cnode (e.g., tsys9 disk failure):**
-
-1. Provision new VM on tsys9 (or other host with local SSD).
-2. Apply cnode2 machine config patch.
-3. New cnode joins etcd, syncs state from survivors.
-4. Quorum was never lost (2 of 3 alive throughout).
-
-**Lost 2 cnodes simultaneously (quorum lost):**
-
-1. Use surviving cnode's etcd snapshot.
-2. Provision 3 new cnode VMs.
-3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
-4. Join other 2 cnodes.
-5. Workers reconnect automatically once API server is back.
-
-**Total cluster loss (all 3 cnodes):**
-
-1. Restore from latest PBS etcd snapshot.
-2. Provision new cnode VMs.
-3. `talosctl bootstrap --recover-from=snapshot.db`.
-4. Re-join workers.
-5. Verify tenant vclusters restored.
-
-### 10.3 Recovery time objectives
-
-| Scenario | RTO | RPO |
-|----------|-----|-----|
-| Single cnode failure | < 30 min | 0 (no data loss) |
-| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
-| Total cluster loss | < 4 hours | ≤ 24 hours |
-
----
-
-## 11. Migration from Current State
-
-### 11.1 Current state
-
-- 5 cnode VMs exist (Debian stock + Tailscale).
-- **No k3s deployed yet.** Cluster was never bootstrapped.
-- 6 wnode VMs exist (some stopped).
-- No workloads running in k8s.
-
-### 11.2 Migration: clean cutover (not a migration)
-
-Since there is no etcd data and no workloads to preserve, the path is a
-**clean rebuild**:
-
-| Phase | Action | Risk |
-|-------|--------|------|
-| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
-| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
-| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
-| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
-| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
-| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
-
-### 11.3 Open dependency: Friday hardware work
-
-Phases 1-2 require:
-
-- **tsys3 local-lvm available.** Per
-  [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
-  free NVMe local-lvm. Currently unused. **Ready.**
-- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
-- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
-  [`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
-  Harbor depends on D3 being available on tsys5.
-
-Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
-Harbor registry does.
-
----
-
-## Appendix: Open questions for next session
-
-1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
-2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
-   (tsys9, tsys1, tsys3).
-3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
-4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
-   Defer until cluster is up.
-5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
-   approval of `192.168.3.0/24` route advertisement.
-6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
-   Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
-   general capacity; tsys3 + tsys5 (local fast storage) reserved for
-   non-ITAR HPC.

k8s/docs/DISTRO-DECISION.md

@@ -1,188 +1,10 @@
-# Distro Decision: Talos Linux vs k3s
+# k8s/docs/DISTRO-DECISION.md
 
-> **Recommendation: Talos Linux.**
+> **Documentation moved to Discourse — the canonical source of truth.**
-> The k3s-on-Debian plan was sound before the ITAR/classified requirement
+>
-> entered scope. Once classified workloads are on the table, Talos's
+> **Talos vs k3s distro analysis and decision**
-> immutable, API-only, measured-boot-capable posture is materially easier
+>
-> to certify and defend.
+> **Read it here:** https://community.turnsys.com/t/305
-
+>
-**Last updated:** 2026-07-28
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
----
-
-## 1. Decision context
-
-| Factor | Constraint |
-|--------|-----------|
-| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
-| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
-| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
-| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
-| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
-| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
-
----
-
-## 2. Head-to-head comparison
-
-### 2.1 ITAR / classified posture
-
-| Property | Talos Linux | k3s on Debian |
-|----------|-------------|---------------|
-| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
-| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
-| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
-| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
-| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
-| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
-| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
-| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
-
-**Bottom line:** For classified workloads, an auditor's first question is
-"how do you prevent unauthorized changes to a node?" Talos's answer is
-"the OS is immutable and the only path is a signed API call." k3s's answer
-is"SSH is locked down and we scan with STIG." The first is structurally
-stronger; the second is operationally maintained.
-
-### 2.2 Operational considerations
-
-| Property | Talos | k3s |
-|----------|-------|-----|
-| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
-| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
-| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
-| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
-| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
-| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
-| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
-
-### 2.3 Cost of choosing Talos over the existing k3s plan
-
-The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
-**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
-
-- **No etcd data to migrate.** Clean cutover, not a migration.
-- **No workloads to drain.** The cluster is empty.
-- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
-  either way it's a `qm` script, not a stateful migration).
-- **Tailscale config shifts** from "installed via apt" to "Talos system
-  extension." (Or, per our recommendation in
-  [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
-  nodes entirely** and onto the existing `tailscale-router` bastion as a
-  subnet router. Cluster nodes become LAN-only.)
-
-**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
-machine config YAML. The hardening investment already encoded in
-`provisioning/Modules/Security/` is **not wasted** — it still applies to
-every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
-the cnodes/wnodes move to Talos.
-
----
-
-## 3. Tailscale compatibility (deep-dive)
-
-Tailscale on Talos is well-supported but introduces a configuration dimension
-worth being explicit about. Three patterns exist:
-
-### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
-
-Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
-internet egress (to Tailscale DERP servers and for coordinate).
-
-- **Talos implementation:** add `siderolabs/tailscale` system extension to
-  each machine config, configure `machine.network.interfaces`.
-- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
-- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
-  finding (data exfiltration path).
-
-### Pattern B — Tailscale on bastion only, SSH/API jump
-
-Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
-`tailscale-router` VM), then runs `talosctl` from the bastion.
-
-- **Pros:** Zero internet egress from cluster nodes.
-- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
-  action originates from the bastion (auditable but clunky).
-
-### Pattern C — Tailscale subnet router on bastion (recommended)
-
-The existing `tailscale-router` VM advertises the cluster LAN subnet
-(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
-Tailscale client transparently routes cluster-bound traffic through the
-bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
-"just works."
-
-- **Pros:**
-  - Cluster nodes have **zero internet egress** (strongest ITAR posture).
-  - Operator UX is unchanged from direct LAN access.
-  - All access is mediated by Tailscale's identity + ACLs (already integrated
-    with your env).
-  - Audit trail lives in Tailscale + bastion logs.
-- **Cons:**
-  - Bastion becomes a dependency for remote admin (LAN-local admin still
-    works without it).
-  - Must enable IP forwarding + subnet route approval in Tailscale ACLs.
-
-**Recommendation: Pattern C.** Documented in
-[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
-
----
-
-## 4. Recommendation
-
-**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
-
-### Justification
-
-1. **Compliance posture is structural, not operational.** "Immutable,
-   API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
-   is a property of how Debian is operated. The first is dramatically
-   easier to argue to an ITAR counsel or classified accreditation officer
-   (DSS, DCSA) than the second.
-2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
-   3 cnodes with Talos is a `qm` script invocation, not a stateful
-   migration. The sunk cost of "we planned k3s" is **zero deployed state**.
-3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
-   `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
-   learn but more error-prone to operate at HA.
-
-### Acknowledged tradeoffs
-
-- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
-  replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
-- **No shell debugging.** When something breaks on a node, you cannot `ssh`
-  in. Mitigation: `talosctl logs/support` produces a support bundle
-  equivalent to a sosreport.
-- **Hardware/module surprises.** Talos ships a curated kernel. Anything
-  beyond virtio + common NIC drivers needs a system extension. On Proxmox
-  VMs this is **not expected to be a problem** — virtio is the path.
-- **Tailscale via system extension.** Adds one config dimension per node.
-  Mitigated by Pattern C (above), which removes Tailscale from cluster
-  nodes entirely.
-
-### What we keep from the k3s mental model
-
-- **Single binary on each node** semantics (Talos is conceptually similar).
-- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
-  API. `kubectl`, `helm`, `kustomize` all work as-is.
-- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
-  distro-independent.
-
----
-
-## 5. What we are NOT deciding here
-
-| Topic | Deferred to |
-|--------|-------------|
-| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
-| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
-| Per-tenant vcluster policy templates | Future session, post-bootstrap |
-| Solar-aware scale-out hosts | Future capacity planning session |
-| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
-
----
-
-## 6. Next step
-
-Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
-design, network topology, identity flow, and bootstrap procedure.

k8s/docs/README.md

@@ -1,44 +1,10 @@
-# Kubernetes Architecture & Build Plan
+# k8s/docs/README.md
 
-> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
+> **Documentation moved to Discourse — the canonical source of truth.**
-> (which captured the storage/host analysis from the Proxmox audit).
+>
-> This directory takes the next step: **which distro, how to build it,
+> **k8s docs index + TL;DR**
-> how to operate it.**
+>
-
+> **Read it here:** https://community.turnsys.com/t/305
-**Last updated:** 2026-07-28
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-## Documents in this directory
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-| Document | Purpose |
-|----------|---------|
-| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
-| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
-
-## TL;DR
-
-| Decision | Recommendation | Why |
-|----------|----------------|-----|
-| **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. |
-| **Runtime** | **containerd** | Talos/k3s default. |
-| **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. |
-| **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. |
-| **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. |
-| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. |
-| **Multi-tenancy** | **vcluster** (per tenant) | Future work. |
-| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. |
-| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. |
-
-## What this directory does NOT cover (deferred)
-
-- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
-- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
-- vcluster per-tenant policy templates.
-- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
-
-These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
-
-## Open question for the user
-
-1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
-   [`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
-   2-failure tolerance on the control plane, keep 5.

k8s/env.sh

@@ -4,9 +4,9 @@
 #
 # All cluster communication goes over Tailscale IPs. No LAN IPs, ever.
 
-# --- Nodes (Tailscale 100.x addresses) ---
+# --- Control plane nodes (Tailscale 100.x addresses) ---
 CNODE1_NAME="pfv-k8s-cnode1"
-CNODE1_IP="100.97.178.106"
+CNODE1_IP="100.125.134.53"
 
 CNODE2_NAME="pfv-k8s-cnode2"
 CNODE2_IP="100.109.34.72"
@@ -21,6 +21,32 @@ ALL_CNODE_NAMES=("$CNODE1_NAME" "$CNODE2_NAME" "$CNODE3_NAME")
 BOOTSTRAP_IP="$CNODE1_IP"
 BOOTSTRAP_NAME="$CNODE1_NAME"
 
+# --- Worker nodes (Tailscale 100.x addresses) ---
+WNODE1_NAME="pfv-k8s-wnode-tsys3"
+WNODE1_IP="100.126.9.112"
+
+WNODE2_NAME="pfv-k8s-wnode-tsys5"
+WNODE2_IP="100.122.252.116"
+
+WNODE3_NAME="pfv-k8s-wnode-tsys6"
+WNODE3_IP="100.83.49.75"
+
+WNODE4_NAME="pfv-k8s-wnode-tsys7"
+WNODE4_IP="100.119.240.11"
+
+WNODE5_NAME="pfv-k8s-wnode-tsys9"
+WNODE5_IP="100.95.201.66"
+
+WNODE6_NAME="ultix-offstage"
+WNODE6_IP="100.70.119.59"
+
+# ultix-streaming: SSH key not yet deployed — join after setup
+# WNODE7_NAME="ultix-streaming"
+# WNODE7_IP="100.101.187.119"
+
+ALL_WNODES=("$WNODE1_IP" "$WNODE2_IP" "$WNODE3_IP" "$WNODE4_IP" "$WNODE5_IP" "$WNODE6_IP")
+ALL_WNODE_NAMES=("$WNODE1_NAME" "$WNODE2_NAME" "$WNODE3_NAME" "$WNODE4_NAME" "$WNODE5_NAME" "$WNODE6_NAME")
+
 # --- SSH ---
 SSH_USER="localuser"
 SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
@@ -44,5 +70,5 @@ cn() {
 # Helper: run a heredoc script on a node
 cn_file() {
   local ip="$1"
-  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${ip}" "sudo -n bash -s" 
+  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${ip}" "sudo -n bash -s"
 }

k8s/install-cp.sh

@@ -48,7 +48,9 @@ curl -sfL https://get.k3s.io | sh -s - server \
   $tls_san_flags \
   --flannel-backend=vxlan \
   --etcd-snapshot-schedule-cron='0 */6 * * *' \
-  --egress-selector-mode=agent
+  --egress-selector-mode=agent \
+  --etcd-arg heartbeat-interval=1000 \
+  --etcd-arg election-timeout=5000
 REMOTE_BOOT
 
 echo "  cnode1 install submitted."
@@ -117,7 +119,9 @@ curl -sfL https://get.k3s.io | sh -s - server \
   --advertise-address=$node_ip \
   $tls_san_flags \
   --flannel-backend=vxlan \
-  --egress-selector-mode=agent
+  --egress-selector-mode=agent \
+  --etcd-arg heartbeat-interval=1000 \
+  --etcd-arg election-timeout=5000
 REMOTE_JOIN
 
   echo "  $node_name install submitted."

k8s/join-workers.sh

@@ -0,0 +1,103 @@
+#!/usr/bin/bash
+#
+# k8s/join-workers.sh — join worker nodes to the k3s cluster
+#
+# Joins all worker nodes defined in env.sh as k3s agents. Worker nodes
+# run user workloads; control plane nodes are tainted NoSchedule.
+#
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck source=./env.sh
+source "$SCRIPT_DIR/env.sh"
+
+echo "============================================"
+echo "  Joining worker nodes to cluster"
+echo "  Server: $BOOTSTRAP_NAME ($BOOTSTRAP_IP)"
+echo "  Workers: ${#ALL_WNODE_NAMES[@]}"
+echo "============================================"
+
+# -------------------------------------------------------
+# 1. Fetch join token from bootstrap node
+# -------------------------------------------------------
+echo ""
+echo "--- [1/3] Fetching join token from $BOOTSTRAP_NAME ---"
+JOIN_TOKEN=$(cn "$BOOTSTRAP_IP" 'cat /var/lib/rancher/k3s/server/token')
+if [ -z "$JOIN_TOKEN" ] || [[ "$JOIN_TOKEN" == cat:* ]]; then
+  echo "FATAL: could not fetch token. Got: ${JOIN_TOKEN:0:40}"
+  exit 1
+fi
+echo "  Token OK (masked: ${JOIN_TOKEN:0:12}***)"
+
+SERVER_URL="https://${BOOTSTRAP_IP}:${K3S_API_PORT}"
+
+# -------------------------------------------------------
+# 2. Install k3s-agent on each worker
+# -------------------------------------------------------
+for i in "${!ALL_WNODES[@]}"; do
+  node_ip="${ALL_WNODES[$i]}"
+  node_name="${ALL_WNODE_NAMES[$i]}"
+
+  echo ""
+  echo "--- [2/3] Joining worker: $node_name ($node_ip) ---"
+
+  # Wipe any existing k3s first
+  cn "$node_ip" '
+    systemctl stop k3s-agent 2>/dev/null || true
+    if [ -x /usr/local/bin/k3s-agent-uninstall.sh ]; then
+      /usr/local/bin/k3s-agent-uninstall.sh
+    fi
+    rm -rf /etc/rancher/k3s /var/lib/rancher/k3s /var/lib/kubelet /var/lib/cni
+    rm -f /etc/systemd/system/k3s-agent.service
+    systemctl daemon-reload
+    ip link delete cni0 2>/dev/null || true
+    ip link delete flannel.1 2>/dev/null || true
+  ' 2>/dev/null || true
+
+  # Install as agent
+  # shellcheck disable=SC2087 # heredoc intentionally expands local config
+  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node_ip}" "sudo -n bash -s" <<REMOTE_AGENT
+set -euo pipefail
+export INSTALL_K3S_VERSION="$K3S_VERSION"
+export K3S_URL="$SERVER_URL"
+export K3S_TOKEN="$JOIN_TOKEN"
+export K3S_NODE_NAME="$node_name"
+curl -sfL https://get.k3s.io | sh -s - agent \
+  --node-name=$node_name \
+  --node-ip=$node_ip
+REMOTE_AGENT
+
+  echo "  $node_name agent install submitted."
+done
+
+# -------------------------------------------------------
+# 3. Wait for all workers to appear Ready
+# -------------------------------------------------------
+TOTAL_NODES=$(( ${#ALL_CNODES[@]} + ${#ALL_WNODES[@]} ))
+echo ""
+echo "--- [3/3] Waiting for all $TOTAL_NODES nodes (${#ALL_CNODES[@]} cp + ${#ALL_WNODES[@]} workers) ---"
+for i in $(seq 1 60); do
+  READY_NODES=$(cn "$BOOTSTRAP_IP" 'k3s kubectl get nodes --no-headers 2>/dev/null | grep -c " Ready"' 2>/dev/null || echo 0)
+  if [ "$READY_NODES" -ge "$TOTAL_NODES" ]; then
+    echo "  All $TOTAL_NODES nodes Ready."
+    break
+  fi
+  echo "  ...waiting ($i/60, $READY_NODES/$TOTAL_NODES ready)"
+  sleep 10
+done
+
+echo ""
+echo "============================================"
+echo "  Node status:"
+echo "============================================"
+cn "$BOOTSTRAP_IP" 'k3s kubectl get nodes -o wide'
+
+if [ "$READY_NODES" -ge "$TOTAL_NODES" ]; then
+  echo ""
+  echo "============================================"
+  echo "  All workers joined. Cluster fully operational."
+  echo "============================================"
+else
+  echo ""
+  echo "WARN: $READY_NODES/$TOTAL_NODES ready. Check failing nodes."
+  exit 1
+fi

k8s/post-setup.sh

@@ -66,10 +66,20 @@ echo "    export KUBECONFIG=$KUBECONFIG_FILE"
 echo "    kubectl get nodes"
 
 # -------------------------------------------------------
-# 3. Verify cluster health
+# 3. Deploy tuned (network-latency profile) on all cnodes
 # -------------------------------------------------------
 echo ""
-echo "--- [3/3] Verifying cluster health ---"
+echo "--- [3/4] Deploying tuned (network-latency) on cnodes ---"
+for ip in "${ALL_CNODES[@]}"; do
+  echo "  $ip..."
+  cn "$ip" 'DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; tuned-adm profile network-latency; systemctl enable tuned; systemctl restart tuned; tuned-adm active' 2>&1 | tail -1
+done
+
+# -------------------------------------------------------
+# 4. Verify cluster health
+# -------------------------------------------------------
+echo ""
+echo "--- [4/4] Verifying cluster health ---"
 
 export KUBECONFIG="$KUBECONFIG_FILE"
 

netinfra/dhcp-migration.md

@@ -1,125 +1,10 @@
-# DHCP Migration: pfv-netboot → pfv-netinfra-01/02
+# netinfra/dhcp-migration.md
 
-**Date:** 2026-07-29
+> **Documentation moved to Discourse — the canonical source of truth.**
-**Status:** ✅ **COMPLETE** — both servers in `normal` failover state, 333
+>
-active leases load-balanced. Netboot DHCP stopped + disabled.
+> **DHCP migration to netinfra-01/02**
-
+>
-> **Safety nets:** pfv-netboot, pfv-netinfra-01/02, sectestbed-sandbox, and
+> **Read it here:** https://community.turnsys.com/t/306
-> ultix-streaming all have static IPs (reachable via Tailscale even if DHCP
+>
-> fails). sectestbed-sandbox snapshot `pre-dhcp-migration` exists as rollback.
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## Migration results (2026-07-29 17:20 CDT)
-
-| Check | Result |
-|-------|--------|
-| netinfra-01 (primary) | `active`, failover `normal`, 244 active leases |
-| netinfra-02 (secondary) | `active`, failover `normal`, 89 active leases |
-| Failover partnership | "Both servers normal" — load-balanced at split=128 |
-| DHCP test (sandbox) | Lease obtained from 192.168.3.252, IP 192.168.3.50/22 |
-| DNS handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
-| NTP handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
-| Domain | `knel.net` |
-| Gateway | `192.168.3.254` |
-| Webmin | Port 10000 (SSL) on both nodes; DHCP module configured |
-| Netboot DHCP | Stopped + disabled (won't start on reboot) |
-| Netmask fix | Both nodes corrected /24 → /22 |
-
-## 1. Current state (discovered 2026-07-29)
-
-| | pfv-netboot (SOURCE) | pfv-netinfra-01 (TARGET primary) | pfv-netinfra-02 (TARGET secondary) |
-|---|---|---|---|
-| IP | 192.168.3.250/22 | 192.168.3.252/24 | 192.168.3.253/24 |
-| isc-dhcp-server | 4.4.3, **active** | NOT installed | NOT installed |
-| Webmin | 2.652 (port 10000, SSL) | NOT installed | NOT installed |
-| DHCP leases | **532 active** | — | — |
-| DHCP config | `/etc/dhcp/dhcpd.conf` | — | — |
-
-**Source DHCP config summary:**
-- Subnet: `192.168.0.0/22` (covers .0.x/.1.x/.2.x/.3.x)
-- Pool range: `192.168.0.1 — 192.168.3.200`
-- Router/GW: `192.168.3.254`
-- DNS: `192.168.3.250` (netboot itself — **changing to 252/253**)
-- Domain: `knel.net`
-- Lease times: 600s default, 7200s max
-- DDNS: `ddns-update-style none` (globally disabled)
-- ~37 host reservations (MAC → fixed IP)
-- 3 global-scope host declarations (outside subnet)
-
-## 2. What changes in the migrated config
-
-| Setting | Old (netboot) | New (netinfra) | Why |
-|---------|--------------|----------------|-----|
-| DNS servers handed out | `192.168.3.250` | `192.168.3.252, 192.168.3.253` | Point clients at the netinfra Pi-hole pair (netboot will be decommissioned) |
-| Failover | none (single server) | ISC DHCP failover-peer | Primary/secondary HA |
-| DHCP interface | `eth0` | `ens18` | Different NIC name on netinfra VMs |
-| Netmask on servers | /22 | **fix to /22** (currently /24) | Match the /22 network |
-
-Everything else (subnet, pool range, reservations, lease times, domain) stays
-identical. The host reservation list is copied verbatim.
-
-## 3. Configs prepared
-
-| File | Destination |
-|------|-------------|
-| [`dhcp/dhcpd-primary.conf`](dhcp/dhcpd-primary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-01 |
-| [`dhcp/dhcpd-secondary.conf`](dhcp/dhcpd-secondary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-02 |
-
-Both are identical except for the `failover peer` block (primary vs secondary).
-
-## 4. Execution sequence
-
-### Phase 1 — pre-flight (non-disruptive)
-
-1. **Snapshot sectestbed-sandbox** (VMID 6000 on tsys5) — rollback point
-2. **Fix netmask** on netinfra-01/02: `/24` → `/22` in `/etc/network/interfaces`
-   (then `ifdown ens18 && ifup ens18`, or `systemctl restart networking`)
-3. Verify Tailscale still reachable after netmask change
-
-### Phase 2 — install prerequisites (non-disruptive, nothing serving yet)
-
-On **both** netinfra-01 and netinfra-02:
-1. `apt-get update && apt-get install -y isc-dhcp-server`
-2. Install Webmin via official setup script
-3. Copy `dhcpd-{primary,secondary}.conf` → `/etc/dhcp/dhcpd.conf`
-4. Write `/etc/default/isc-dhcp-server` with `INTERFACESv4="ens18"`
-5. Syntax check: `dhcpd -t -cf /etc/dhcp/dhcpd.conf`
-6. **Do NOT start the DHCP service yet**
-
-### Phase 3 — cutover (the critical moment)
-
-**Goal:** minimize the DHCP gap. Existing clients hold valid leases for 600
-seconds — they won't notice a brief outage.
-
-1. **Stop netboot DHCP:** `ssh netboot 'sudo systemctl stop isc-dhcp-server'`
-2. **Immediately start netinfra-01:** `systemctl start isc-dhcp-server`
-3. **Immediately start netinfra-02:** `systemctl start isc-dhcp-server`
-4. **Verify failover partnership:** check `journalctl -u isc-dhcp-server` on both
-   — look for "failover peer pfv-dhcp: I move from recover to recover-done"
-5. Wait ~10 seconds for lease sync
-
-### Phase 4 — test
-
-1. On sectestbed-sandbox: edit `/etc/network/interfaces` to DHCP
-2. `dhclient -r` (release) then `dhclient` (request new lease)
-3. Verify IP assigned, correct DNS, correct gateway
-4. If fails: `qm rollback 6000 <snapshot>` and debug
-5. If succeeds: change back to static (sandbox's normal config)
-
-### Phase 5 — cleanup
-
-1. **Disable netboot DHCP:** `ssh netboot 'sudo systemctl disable isc-dhcp-server'`
-2. Verify Webmin DHCP module visible at `https://192.168.3.252:10000`
-3. Document + commit
-
-## 5. Failover mechanics (reference)
-
-ISC DHCP failover uses a TCP connection on port **647** between the two servers.
-
-- **Primary** owns the pool initially and can assign new leases immediately.
-- **Secondary** enters `recover` state, syncs the lease database from primary,
-  then transitions to `recover-done` → `normal`.
-- If primary dies, secondary continues serving after the MCLT timeout (600s).
-- If secondary dies, primary continues serving alone.
-- **split 128** = load balance 50/50 (primary handles even hashes, secondary odd).
-- Host reservations (fixed-address) are NOT subject to failover — both servers
-  can independently assign reserved IPs.

netinfra/dhcp/dhcpd-primary.conf

@@ -45,11 +45,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
 
   # ---- host reservations (fixed-address; not subject to failover pool) ----
 
-  host pfv-r3-mgmt {
+  host pfv-r3-tor-mgmt-01 {
     hardware ethernet 00:14:22:69:1c:37;
     fixed-address 192.168.0.7;
   }
-  host pfv-r3-stor {
+  host pfv-r3-tor-stor-01 {
     hardware ethernet 00:13:72:46:95:e4;
     fixed-address 192.168.0.9;
   }
@@ -57,11 +57,11 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet 40:9f:38:b0:b5:2f;
     fixed-address 192.168.1.84;
   }
-  host pfv-r2-tor1 {
+  host pfv-r2-tor-01 {
     hardware ethernet 00:0d:56:41:7a:4d;
     fixed-address 192.168.0.10;
   }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
     hardware ethernet a4:ba:db:6f:ce:28;
     fixed-address 192.168.0.12;
   }
@@ -177,11 +177,66 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet a4:bb:6d:e3:56:86;
     fixed-address 192.168.3.58;
   }
+	# umbrel
+	host tsys-umbrel {
+		hardware ethernet 02:2E:FF:8E:A2:D2;
+		fixed-address 192.168.1.97;
+		}
+	# ultix-streaming
+	host ultix-streaming {
+		hardware ethernet bc:24:11:1a:8f:6f;
+		fixed-address 192.168.3.78;
+		}
+	# ultix-offstage
+	host ultix-offstge {
+		hardware ethernet bc:24:11:1f:9d:83;
+		fixed-address 192.168.3.79;
+		}
+	# ultix-highside
+	host ultix-highside {
+		hardware ethernet a0:4a:5e:ca:46:f3;
+		fixed-address 192.168.3.32;
+		}
+	# pfv-k8s-cnode1
+	host pfv-k8s-cnode1 {
+		hardware ethernet bc:24:11:cb:97:10;
+		fixed-address 192.168.1.91;
+		}
+	# pfv-k8s-cnode2
+	host pfv-k8s-cnode2 {
+		hardware ethernet bc:24:11:40:25:f8;
+		fixed-address 192.168.3.113;
+		}
+	# pfv-k8s-cnode3
+	host pfv-k8s-cnode3 {
+		hardware ethernet bc:24:11:38:c0:58;
+		fixed-address 192.168.1.228;
+		}
+	# devbox-cloudron
+	host devbox-cloudron {
+		hardware ethernet bc:24:11:f7:b1:07;
+		fixed-address 192.168.1.6;
+		}
+	# hfnoc-uisp
+	host hfnoc-uisp {
+		hardware ethernet bc:24:11:a3:87:61;
+		fixed-address 192.168.3.193;
+		}
+	# kali-rd
+	host kali-rd {
+		hardware ethernet bc:24:11:9e:1c:e9;
+		fixed-address 192.168.2.37;
+		}
+	# kali-tsys
+	host kali-tsys {
+		hardware ethernet bc:24:11:16:22:d4;
+		fixed-address 192.168.1.114;
+		}
 }
 
 # ---- host declarations outside subnet (global scope, same as netboot) ----
 
-host subodev-torsw01 {
+host pfv-r6-mgmt-01 {
   hardware ethernet 00:14:22:69:18:a7;
   fixed-address 192.168.0.8;
 }
@@ -189,7 +244,80 @@ host pfv-r1-tor-top {
   hardware ethernet 00:23:ae:c1:ad:e8;
   fixed-address 192.168.0.11;
 }
-host tailscale-router {
+
-  hardware ethernet bc:24:11:8a:69:04;
+# --- VM DHCP reservations (generated 2026-08-11, ticket #420) ---
-  fixed-address 192.168.3.16;
+# All pinned to current ARP-observed IPs. No forward DNS needed
-}
+# (forward records point to Tailscale 100.x addresses).
+
+  host tsys-ca {
+    hardware ethernet bc:24:11:32:d0:36;
+    fixed-address 192.168.1.181;
+  }
+  host pfv-netinfra-01 {
+    hardware ethernet bc:24:11:65:b2:ac;
+    fixed-address 192.168.3.252;
+  }
+  host pfv-netinfra-02 {
+    hardware ethernet bc:24:11:e4:37:53;
+    fixed-address 192.168.3.253;
+  }
+  host tsys-librenms {
+    hardware ethernet bc:24:11:5c:96:1e;
+    fixed-address 192.168.3.176;
+  }
+  host tsys-proxmox-datacenter {
+    hardware ethernet bc:24:11:e6:03:2d;
+    fixed-address 192.168.2.44;
+  }
+  host pfv-k8s-wnode-tsys3 {
+    hardware ethernet bc:24:11:ee:7e:7b;
+    fixed-address 192.168.1.98;
+  }
+  host pfv-proxmox-backup-server {
+    hardware ethernet bc:24:11:6e:12:69;
+    fixed-address 192.168.2.193;
+  }
+  host pfv-k8s-wnode-tsys5 {
+    hardware ethernet bc:24:11:c7:a8:6c;
+    fixed-address 192.168.1.5;
+  }
+  host preprod-hfnoc-uisp {
+    hardware ethernet bc:24:11:74:d6:8a;
+    fixed-address 192.168.3.192;
+  }
+  host tsys-awx {
+    hardware ethernet bc:24:11:80:0d:16;
+    fixed-address 192.168.3.115;
+  }
+  host pfv-rr-middleware-02 {
+    hardware ethernet bc:24:11:96:0e:ee;
+    fixed-address 192.168.1.117;
+  }
+  host tsys-proxmox-mailgw-01 {
+    hardware ethernet bc:24:11:56:61:18;
+    fixed-address 192.168.1.11;
+  }
+  host pfv-k8s-wnode-tsys7 {
+    hardware ethernet bc:24:11:30:b8:07;
+    fixed-address 192.168.1.109;
+  }
+  host pfv-rr-middleware-01 {
+    hardware ethernet bc:24:11:1e:61:cf;
+    fixed-address 192.168.1.110;
+  }
+  host tsys-voip {
+    hardware ethernet bc:24:11:23:ce:04;
+    fixed-address 192.168.1.70;
+  }
+  host tsys-proxmox-mailgw-02 {
+    hardware ethernet bc:24:11:5f:e5:2c;
+    fixed-address 192.168.1.10;
+  }
+  host pfv-k8s-wnode-tsys6 {
+    hardware ethernet bc:24:11:fa:6e:b5;
+    fixed-address 192.168.1.111;
+  }
+  host tsys-siem-new {
+    hardware ethernet bc:24:11:ee:67:e2;
+    fixed-address 192.168.1.223;
+  }

netinfra/dhcp/dhcpd-secondary.conf

@@ -59,7 +59,7 @@ subnet 192.168.0.0 netmask 255.255.252.0 {
     hardware ethernet 00:0d:56:41:7a:4d;
     fixed-address 192.168.0.10;
   }
-  host pfv-core-sw01 {
+  host pfv-r5-core-01 {
     hardware ethernet a4:ba:db:6f:ce:28;
     fixed-address 192.168.0.12;
   }

netinfra/dns-cluster-setup/README.md

@@ -1,183 +1,10 @@
-# Technitium DNS Cluster Setup
+# netinfra/dns-cluster-setup/README.md
 
-Replicates the production Technitium DNS Server from `tailscale-router` to the
+> **Documentation moved to Discourse — the canonical source of truth.**
-`pfv-netinfra-01/02` pair and configures them as a primary/secondary cluster
+>
-with automatic zone transfers.
+> **Technitium DNS cluster setup**
-
+>
-## Architecture
+> **Read it here:** https://community.turnsys.com/t/306
-
+>
-```
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
- tailscale-router (PRODUCTION — READ ONLY)
+> Do not update content here — edit the Discourse wiki topic instead.*
-   └─ tsys-dns container (technitium/dns-server)
-      └─ 124 zones (knel.net + reverse DNS)
-      └─ Users + 2FA in auth.config
-                │
-          docker cp (export)
-                │
-                ▼
- ┌─ pfv-netinfra-01 (192.168.3.252) ──── PRIMARY ──────────┐
- │  tsys-dns container (Technitium on :5300)                │
- │  pihole container (Pi-hole on :53 → Technitium :5300)   │
- │  All zones are Primary                                   │
- │  Zone transfer allowed from 192.168.3.253               │
- └──────────────────────────────────────────────────────────┘
-                │
-          AXFR / IXFR + NOTIFY (DNS zone transfer, port 5300)
-                │
-                ▼
- ┌─ pfv-netinfra-02 (192.168.3.253) ─── SECONDARY ────────┐
- │  tsys-dns container (Technitium on :5300)                │
- │  pihole container (Pi-hole on :53 → Technitium :5300)   │
- │  All zones are Secondary (AXFR from 01)                  │
- └──────────────────────────────────────────────────────────┘
-```
-
-### How clustering works
-
-Technitium uses standard DNS zone transfers (AXFR/IXFR) for primary/secondary
-replication, not a proprietary protocol:
-
-1. **Primary (01)** holds all zones as authoritative primary zones.
-2. **Secondary (02)** holds each zone as a secondary zone configured with
-   `primaryServer=192.168.3.252:5300`.
-3. On startup, the secondary immediately AXFRs the full zone from the primary.
-4. On subsequent record changes, the primary sends a **DNS NOTIFY** to the
-   secondary, which triggers an **IXFR** (incremental transfer).
-5. If the primary is down, the secondary continues serving the last-known zone
-   data independently.
-
-### Credentials and 2FA
-
-The production `auth.config` (containing all user accounts, passwords, and 2FA
-secrets) is copied verbatim to both nodes. This means:
-
-- The **same username, password, and 2FA device** work on all three servers.
-- The web console is at `http://<host>:5380/` on each node.
-- No credential changes are needed.
-
-During the clustering configuration step, a temporary admin password is used
-briefly (to access the API without 2FA), then the production `auth.config` is
-restored. See "Security notes" below.
-
-## Prerequisites
-
-- SSH key access to all hosts as `localuser` with passwordless sudo.
-- The `remote-dns.sh` wrapper must be able to reach all hosts via Tailscale FQDN.
-- Docker + Docker Compose on netinfra-01/02 (already installed).
-- The production Technitium on tailscale-router must be running.
-
-## Usage
-
-```bash
-cd dns-cluster-setup/
-
-# Step-by-step (recommended for first run):
-./setup.sh export      # 1. Export config from tailscale-router (READ-ONLY)
-./setup.sh deploy01    # 2. Deploy to netinfra-01 as primary
-./setup.sh deploy02    # 3. Deploy to netinfra-02 as secondary clone
-./setup.sh cluster     # 4. Configure clustering (01→02 zone transfers)
-./setup.sh verify      # 5. Run all verification tests
-
-# Or all at once:
-./setup.sh all
-```
-
-### Configuration overrides
-
-All defaults can be overridden via environment variables:
-
-| Variable | Default | Description |
-|---|---|---|
-| `PRIMARY_IP` | `192.168.3.252` | netinfra-01 LAN IP |
-| `SECONDARY_IP` | `192.168.3.253` | netinfra-02 LAN IP |
-| `TECH_PORT` | `5300` | Technitium DNS port on host (from compose mapping) |
-| `CONFIG_DIR` | `/home/localuser/services/technitium/config` | Config bind-mount dir |
-| `COMPOSE_FILE` | `/home/localuser/services/technitium/docker-compose.yml` | Compose file |
-| `TEMP_ADMIN_PW` | `KnelClusterSetup!2026` | Temp admin password (used only during clustering, then discarded) |
-
-## Scripts
-
-| Script | Purpose |
-|---|---|
-| `remote-dns.sh` | SSH/SCP chokepoint for all DNS host access (tsrouter, netinfra01, netinfra02, netboot, sandbox) |
-| `setup.sh` | Master orchestrator: export → deploy → cluster → verify |
-| `verify.sh` | Comprehensive 10-section verification suite |
-| `discover*.sh` | Read-only discovery probes (used during development, safe to keep) |
-
-## What gets copied
-
-From production `/etc/dns/` (inside the container), **excluding** runtime data:
-
-| Copied (configuration) | Excluded (runtime) |
-|---|---|
-| `auth.config` (users, passwords, 2FA) | `cache.bin` (DNS cache) |
-| `dns.config` (server settings) | `stats/` (query statistics) |
-| `webservice.config` (web console) | `logs/` (log files) |
-| `allowed.config` (zone transfer ACL) | |
-| `blocked.config` (blocked domains) | |
-| `blocklist.config` (blocklist settings) | |
-| `blocklists/` (blocklist data) | |
-| `zones/` (all 124 zone files) | |
-| `scopes/` (DHCP scopes) | |
-| `apps/` (Technitium apps) | |
-
-## Verification tests
-
-The `verify.sh` script runs 10 categories of tests:
-
-1. **Container health** — both Technitium containers are Up
-2. **API responds** — web console API is reachable on both nodes
-3. **Zone count** — primary matches production; secondary matches primary
-4. **Forward DNS** — known knel.net records resolve identically on both nodes
-5. **External DNS** — both nodes can resolve external domains (github.com)
-6. **Zone transfer (AXFR)** — secondary can AXFR knel.net from primary
-7. **Reverse DNS** — PTR zones have SOA records on both nodes
-8. **Production untouched** — container still running, zone count unchanged
-9. **Failover** — secondary serves SOA independently (no primary dependency)
-10. **Credentials** — `auth.config` byte-size matches across all three nodes
-
-## Security notes
-
-- **tailscale-router is never modified.** The only operation is `docker cp`
-  (read) to export the config. No writes, no restarts, no config changes.
-- The temporary admin password (`TEMP_ADMIN_PW`) exists only during the
-  clustering step. After configuration, the production `auth.config` (with 2FA)
-  is restored. The temp password is never persisted.
-- The export tarball (`.export/technitium-production-config.tar.gz`) contains
-  production credentials. It is in `.gitignore` and should be deleted after
-  setup: `rm -rf dns-cluster-setup/.export/`
-- Each node's existing config is backed up to `config.backup-<timestamp>` before
-  replacement, so the change is reversible.
-
-## Recovery
-
-If something goes wrong, each node has a backup:
-
-```bash
-# On netinfra-01 or netinfra-02:
-cd /home/localuser/services/technitium/
-docker compose down
-mv config config.failed
-mv config.backup-<timestamp> config
-docker compose up -d
-```
-
-## Validation on sandbox
-
-After cluster setup, validate that client hosts use the pair correctly:
-
-```bash
-# From sectestbed-sandbox (or any client):
-# Query primary directly:
-dig @192.168.3.252 pfv-netinfra-01.knel.net
-
-# Query secondary directly:
-dig @192.168.3.253 pfv-netinfra-01.knel.net
-
-# Both should return the same answer.
-```
-
-The KNELServerBuild provisioning code (`provisioning/ConfigFiles/NTP/ntp.conf`
-and `provisioning/ConfigFiles/Resolv/resolv.conf`) points clients at both
-servers for DNS and NTP redundancy. See `docs/server-build/tailscale.md` for the
-full DNS architecture analysis.

netinfra/pfv-netboot-setup.md

@@ -1,116 +1,10 @@
-# pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
+# netinfra/pfv-netboot-setup.md
 
-> **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
+> **Documentation moved to Discourse — the canonical source of truth.**
-> Do **not** modify it. This document describes it as audited so its services can
+>
-> be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
+> **pfv-netboot reference node setup**
-> collected by **read-only** audit scripts (`audit-netboot.sh`,
+>
-> `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
+> **Read it here:** https://community.turnsys.com/t/306
-
+>
-## 1. Host
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-| Item | Value |
-|---|---|
-| Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
-| OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
-| Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
-| Timezone | `America/Chicago` (US/Central) |
-| LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
-| Tailscale | `100.103.64.82` (`tailscale0`) |
-| DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
-| Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
-| Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
-
-`eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
-`ntp_default`, `dns_default`, and several stale ones).
-
-## 2. Services overview
-
-| Service | Form | Running? |
-|---|---|---|
-| **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
-| **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
-| **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
-| **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
-
-## 3. Pi-hole (container)
-
-- **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
-- **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
-- **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
-- **Ports (host):**
-
-  | Host | Container | Purpose |
-  |---|---|---|
-  | `53/tcp`, `53/udp` | 53 | DNS |
-  | `10002/tcp` | 80 | Web admin (HTTP) |
-  | `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
-
-- **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=REDACTED_PASSWORD`, `FTLCONF_dns_listeningMode=all`
-- **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
-- **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
-  - Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
-  - `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
-  - `queryLogging = true`, DNSSEC off
-- **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
-- **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
-- **Web admin:** `http://pfv-netboot:10002/admin/` (password `REDACTED_PASSWORD`)
-- Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
-
-## 4. NTP (two layers)
-
-### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
-- **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
-- **Image:** `dockurr/chrony`
-- **Env:** `NTP_SERVERS=pool.ntp.org`
-- **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
-- `restart: always`
-- chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
-- On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
-
-### 4b. bare-metal `ntpsec`
-- Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
-- **Config** (`/etc/ntpsec/ntp.conf`):
-  ```
-  driftfile /var/lib/ntp/ntp.drift
-  leapfile /usr/share/zoneinfo/leap-seconds.list
-  server pfvsvrpi.knel.net
-  restrict 127.0.0.1
-  restrict ::1
-  ```
-- Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
-
-## 5. Technitium DNS (currently stopped)
-
-- **Not running** — no container and **no compose file** exists for it.
-- A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
-  (mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
-  are intact (last activity 2025-06-23). A second typo'd volume
-  `dns_tyss-dns-config` is empty.
-- Config files are **binary** (Technitium's own serialization), but copy verbatim:
-  `dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
-  `self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
-- **Zones present** (12 reverse + 1 forward):
-  - `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
-  - Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
-- **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
-- The compose project name historically was `dns` (network `dns_default` still exists).
-
-## 6. Firewall / misc
-
-- nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
-  `DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
-  `OUTPUT ACCEPT`. No UFW / firewalld.
-- Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
-  (2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
-  usermin (10000/10002/20000), Tailscale (41641).
-- SELinux absent; AppArmor default docker profile.
-
-## 7. How it was audited (no changes made)
-
-```bash
-ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh        # broad read-only sweep
-ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh   # docker inspect + compose
-ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh       # pihole.toml + technitium
-```
-
-Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.

netinfra/pfv-netinfra-setup.md

@@ -1,220 +1,10 @@
-# pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
+# netinfra/pfv-netinfra-setup.md
 
-These two nodes replicate the network-infrastructure services of **pfv-netboot**
+> **Documentation moved to Discourse — the canonical source of truth.**
-(Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
+>
-reads config from pfv-netboot (read-only) and relays it to each target.
+> **pfv-netinfra-01/02 initial setup**
-
+>
-## 1. Nodes
+> **Read it here:** https://community.turnsys.com/t/306
-
+>
-| | pfv-netinfra-01 | pfv-netinfra-02 |
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-|---|---|---|
+> Do not update content here — edit the Discourse wiki topic instead.*
-| OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
-| LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
-| Tailscale | `100.70.181.72` | `100.93.194.82` |
-| RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
-| Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
-| Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
-| Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
-
-## 2. Service layout
-
-All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
-so the compose files are directly editable; data dirs keep container uids):
-
-```
-/home/localuser/services/
-├── pihole/
-│   ├── docker-compose.yml
-│   └── etc-pihole/          # copied from netboot /root/pihole/etc-pihole
-│       ├── pihole.toml      # Pi-hole v6 config (upstreams, etc.)
-│       ├── gravity.db       # adlists / domainlists / clients / groups
-│       ├── adlists.list
-│       ├── dnsmasq.conf
-│       ├── tls.{crt,pem,crt_ca}
-│       └── versions
-├── ntp/
-│   └── docker-compose.yml   # chrony container (see §5 — not used; host ntpsec serves)
-└── technitium/
-    ├── docker-compose.yml
-    └── config/              # copied from netboot orphaned volume dns_tsys-dns-config/_data
-        ├── dns.config
-        ├── auth.config
-        ├── scopes/Default.scope
-        ├── self-signed-cert.pfx
-        └── zones/           # knel.net.zone + 12 Tailscale reverse zones
-```
-
-## 3. Pi-hole (container `pihole`)
-
-Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
-
-| Host port | Container | Purpose |
-|---|---|---|
-| `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
-| `10002/tcp` | 80 | Web admin (HTTP) |
-| `10003/tcp` | 443 | Web admin (HTTPS) |
-
-`docker-compose.yml`:
-```yaml
-services:
-  pihole:
-    container_name: pihole
-    image: pihole/pihole:latest
-    hostname: pihole
-    ports:
-      - "53:53/tcp"
-      - "53:53/udp"
-      - "10002:80/tcp"
-      - "10003:443/tcp"
-    environment:
-      TZ: 'America/Chicago'
-      FTLCONF_webserver_api_password: 'REDACTED_PASSWORD'
-      FTLCONF_dns_listeningMode: 'all'
-    volumes:
-      - './etc-pihole:/etc/pihole'
-    cap_add:
-      - SYS_NICE
-    restart: always
-```
-- Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
-- Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
-- `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
-- Web admin: `http://<node>:10002/admin/` — password **`REDACTED_PASSWORD`** (same as netboot).
-- Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
-
-## 4. Technitium DNS (container `tsys-dns`)
-
-Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
-`knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
-orphaned `dns_tsys-dns-config` volume.
-
-| Host port | Container | Purpose |
-|---|---|---|
-| `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
-| `5380/tcp` | 5380 | Web console (HTTP) |
-| `53443/tcp` | 53443 | Web console (HTTPS) |
-
-`docker-compose.yml`:
-```yaml
-services:
-  technitium:
-    image: technitium/dns-server
-    container_name: tsys-dns
-    ports:
-      - "5300:53/tcp"
-      - "5300:53/udp"
-      - "5380:5380/tcp"
-      - "53443:53443/tcp"
-    volumes:
-      - './config:/etc/dns'
-    restart: always
-```
-- Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
-- Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
-  password (carried over via `auth.config`). If the password is unknown, reset it
-  from the console or by removing `config/auth.config` and recreating the container.
-- **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
-  because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
-  query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
-  Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
-  Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
-  from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
-  `knel.net` A-records directly in Pi-hole's Local DNS).
-
-## 5. NTP (host `ntpsec`, not a container)
-
-Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
-serves NTP on every local address — including the Tailscale IP — and keeps the
-system clock synced. This is the **same daemon family as netboot's own bare-metal
-ntpsec**.
-
-- **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
-  Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
-  does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
-  **does** bind the Tailscale IP, so the container cannot claim it (`address
-  already in use`) and would be a non-functional duplicate (verified: the
-  container started but never synced — Stratum 0). It is therefore intentionally
-  **omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
-  host NTP unit and removes any stale `tsys-ntp` container.
-- ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
-  `restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
-- Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
-  leap normal.
-
-The `ntp/docker-compose.yml` is still written on each node for parity/reference
-(and in case the host NTP is ever disabled — then `sudo docker compose -f
-/home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
-
-## 6. Verification results (2026-07-28)
-
-| Check | pfv-netinfra-01 | pfv-netinfra-02 |
-|---|---|---|
-| `pihole` health | healthy | healthy |
-| `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
-| Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
-| `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
-| Technitium web `:5380` | HTTP 200 | HTTP 200 |
-| NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
-
-## 7. Operating the services
-
-```bash
-# status
-sudo docker ps
-
-# Pi-hole
-sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
-sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
-sudo docker exec pihole pihole -v          # version
-sudo docker exec pihole pihole -g          # rebuild gravity
-sudo docker exec pihole pihole -a -p       # set/change web password
-
-# Technitium
-sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
-sudo docker exec tsys-dns sh               # explore /etc/dns
-
-# NTP (host)
-systemctl status ntpsec
-ntpq -pn
-```
-
-## 8. Differences from pfv-netboot (intentional)
-
-1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
-   can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
-2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
-3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
-   (the container cannot bind the Tailscale IP here; see §5).
-4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
-   on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
-5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
-   copied (transient); gravity DB and all configuration are.
-
-## 9. Re-running / reproducing
-
-`setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
-present and uses `docker compose up -d` (no-ops when unchanged). It reads
-pfv-netboot read-only and never mutates it.
-
-```bash
-./setup-netinfra.sh                       # deploy to both nodes
-./setup-netinfra.sh pfv-netinfra-01       # deploy one node
-./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
-```
-
-Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
-sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
-internet for image pulls.
-
-## 10. Files in this directory
-
-| File | Purpose |
-|---|---|
-| `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
-| `audit-netboot.sh` | broad read-only audit of pfv-netboot |
-| `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
-| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
-| `baseline.sh` | read-only baseline of a target node |
-| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
-| [`pfv-netboot-setup.md`](pfv-netboot-setup.md) | reference-node documentation |
-| [`pfv-netinfra-setup.md`](pfv-netinfra-setup.md) | this document |

netinfra/pihole/.env.example

@@ -0,0 +1,3 @@
+# Pi-hole web UI password. NEVER commit the real .env — only this template.
+# Copy to .env and set the value before `docker compose up -d`.
+PIHOLE_WEB_PASSWORD=changeme

netinfra/pihole/README.md

@@ -0,0 +1,10 @@
+# netinfra/pihole/README.md
+
+> **Documentation moved to Discourse — the canonical source of truth.**
+>
+> **Pi-hole recursive DNS hardening**
+>
+> **Read it here:** https://community.turnsys.com/t/306
+>
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
+> Do not update content here — edit the Discourse wiki topic instead.*

netinfra/pihole/docker-compose.yml

@@ -0,0 +1,55 @@
+services:
+  pihole:
+    container_name: pihole
+    # Root cause of the 2026-08 gravity.db corruption: default /dev/shm (64M)
+    # was too small for FTL's shared-memory metrics. 1024M has been stable.
+    shm_size: '1024M'
+    image: pihole/pihole:2026.07.0
+    hostname: pihole
+    entrypoint: ["/usr/local/bin/gravity-validate.sh"]
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: '${PIHOLE_WEB_PASSWORD}'
+      FTLCONF_dns_listeningMode: 'all'
+      # Rate-limiting disabled (count=0). Uptime Kuma on Cloudron VPS sends
+      # high-volume DNS queries for monitoring; default 1000/60s limit was
+      # causing intermittent REFUSED responses → Uptime Kuma flapping.
+      FTLCONF_dns_rateLimit_count: '0'
+      FTLCONF_dns_rateLimit_interval: '0'
+      FTLCONF_dns_upstreams: '["8.8.8.8"]'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+      - './etc-dnsmasq.d:/etc/dnsmasq.d'
+      - './gravity-validate.sh:/usr/local/bin/gravity-validate.sh:ro'
+    cap_add:
+      - SYS_NICE
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "dig +short +norecurse @127.0.0.1 pi.hole >/dev/null 2>&1 && test -s /etc/pihole/gravity.db || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+    labels:
+      autoheal: "true"
+    networks:
+      - default
+      - dnsnet
+  autoheal:
+    container_name: autoheal
+    image: willfarrell/autoheal:1.2.0
+    environment:
+      AUTOHEAL_CONTAINER_LABEL: autoheal
+      AUTOHEAL_INTERVAL: 30
+      AUTOHEAL_START_PERIOD: 60
+    volumes:
+      - '/var/run/docker.sock:/var/run/docker.sock:ro'
+    restart: always
+networks:
+  dnsnet:
+    external: true

netinfra/pihole/gravity-validate.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# gravity-validate.sh — pre-start integrity check for Pi-hole's gravity.db
+#
+# Runs as the container entrypoint. If gravity.db is empty or has an invalid
+# SQLite header (the symptom of the /dev/shm corruption outage), move it aside
+# so Pi-hole regenerates a clean DB on start instead of crashing.
+set -e
+GRAVITY_DB="/etc/pihole/gravity.db"
+TIMESTAMP=$(date +%Y%m%d%H%M%S)
+
+if [ -f "$GRAVITY_DB" ]; then
+    if [ ! -s "$GRAVITY_DB" ]; then
+        echo "[gravity-validate] gravity.db is empty, moving aside"
+        mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+    else
+        HEADER=$(head -c 15 "$GRAVITY_DB" 2>/dev/null || true)
+        if [ "$HEADER" != "SQLite format 3" ]; then
+            echo "[gravity-validate] gravity.db invalid header, moving aside"
+            mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+        fi
+    fi
+fi
+# Keep only the 3 most recent corrupt backups (names carry a timestamp,
+# so lexical reverse-sort = newest-first).
+find /etc/pihole -maxdepth 1 -name 'gravity.db.corrupt.*' -print 2>/dev/null \
+  | sort -r | tail -n +4 | xargs -r rm -f
+echo "[gravity-validate] OK, starting Pi-hole"
+exec /usr/bin/start.sh "$@"

netinfra/switches/core-sw01-ch1-to-lacp.cmds

@@ -0,0 +1,25 @@
+! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
+! Switch: core-sw01 (Dell PowerConnect 5448, rack 5)
+! WHEN: Friday maintenance window — BOTH switches must change together
+! RISK: Brief storage-net outage during transition (seconds)
+!
+! Current: g13-g16 in ch1, mode=on (static, no failure detection)
+! Target:  g13-g16 in ch1, mode=auto (LACP active partner negotiation)
+!
+! NOTE: core-sw01 hash stays layer-2-3 (hardware limit — best available)
+!       tor3-stor hash stays layer-2-3-4 (already set)
+!
+! IMPORTANT: Run this SIMULTANEOUSLY with tor3-stor change.
+!   If one side is LACP and other is static, trunk goes down until
+!   both sides match. Plan for ~30s storage-net outage.
+!
+enable
+configure
+interface range ethernet g13-g16
+  no channel-group
+  channel-group 1 mode auto
+exit
+exit
+show interfaces status port-channel 1
+show lacp port-channel 1
+copy running-config startup-config

netinfra/switches/pfv-r3-tor-stor-01-mac.cmds

@@ -0,0 +1,5 @@
+! pfv-r3-tor-stor-01 — MAC table + port details (Dell PowerConnect 5324 commands)
+enable
+show bridge addressing-table address
+show interfaces description
+show interfaces status

netinfra/switches/pfv-r3-tor-stor-01.cmds

@@ -1,4 +1,4 @@
-! pfv-tor3-stor — Neyland 24T (Radlan-based, rack 3 storage TOR)
+! pfv-r3-tor-stor-01 — Neyland 24T (Radlan-based, rack 3 storage TOR)
 ! Radlan CLI uses different keywords than DNOS
 enable
 show system

netinfra/switches/pfv-r5-core-01-mac.cmds

@@ -0,0 +1,7 @@
+! pfv-r5-core-01 — MAC address table + LLDP neighbors
+terminal datadump
+enable
+show mac-address-table
+show lldp info
+show lldp neighbors
+show interfaces status port-channel

netinfra/switches/pfv-r5-core-01.cmds

@@ -1,4 +1,4 @@
-! pfv-core-sw01 — Dell PowerConnect 5448 (core switch, rack 5)
+! pfv-r5-core-01 — Dell PowerConnect 5448 (core switch, rack 5)
 ! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down)
 terminal datadump
 enable

netinfra/switches/tor3-stor-ch1-to-lacp.cmds

@@ -0,0 +1,24 @@
+! #369/#394: Convert cross-rack trunk ch1 from static (mode=on) to LACP
+! Switch: pfv-r3-tor-stor (Dell PowerConnect 5324 / "Neyland 24T", rack 3)
+! WHEN: Friday maintenance window — BOTH switches must change together
+! RISK: Brief storage-net outage during transition (seconds)
+!
+! Current: g20-g23 in ch1, mode=on (static, no failure detection)
+! Target:  g20-g23 in ch1, LACP active
+!
+! Radlan CLI syntax (different from DNOS on core-sw01)
+! Verify exact keywords on this switch first — 'show running-config'
+! will show the current ch1 config to confirm syntax.
+!
+! IMPORTANT: Run this SIMULTANEOUSLY with core-sw01 change.
+!
+enable
+configure
+interface range ethernet g20-g23
+  no channel-group
+  channel-group 1 mode active
+exit
+exit
+show interfaces status port-channel 1
+show lacp port-channel 1
+copy running-config startup-config

oam/netdisco/deployment.yml

@@ -0,0 +1,41 @@
+# NetDisco deployment configuration [#337]
+# SNMP community: kn3lmgmt (v2c)
+
+# Database connection (matched to docker-compose services)
+database:
+  host: netdisco-postgresql
+  dbname: netdisco
+  user: netdisco
+  pass: netdisco
+
+# SNMP authentication
+snmp_auth:
+  - tag: default_v2c
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 2
+
+  - tag: default_v1
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 1
+
+# Job scheduling
+schedule:
+  discoverall:
+    first: 60
+    every: 3600
+  macwalk:
+    first: 120
+    every: 900
+  arpwalk:
+    first: 150
+    every: 900
+  nbtwalk:
+    first: 180
+    every: 900
+
+# Node freshness (keep history)
+node_freshness: 0

oam/netdisco/docker-compose.yml

@@ -0,0 +1,44 @@
+services:
+  netdisco-postgresql:
+    image: netdisco/netdisco:2.102001-postgresql
+    container_name: oam-netdisco-db
+    environment:
+      - POSTGRES_USER=netdisco
+      - POSTGRES_PASSWORD=netdisco
+      - POSTGRES_DB=netdisco
+    volumes:
+      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
+    restart: unless-stopped
+
+  netdisco-backend:
+    image: netdisco/netdisco:2.102001-backend
+    container_name: oam-netdisco-backend
+    depends_on:
+      - netdisco-postgresql
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+      - NETDISCO_SNMP_COMMUNITY=kn3lmgmt
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
+    restart: unless-stopped
+
+  netdisco-web:
+    image: netdisco/netdisco:2.102001-web
+    container_name: oam-netdisco-web
+    depends_on:
+      - netdisco-postgresql
+      - netdisco-backend
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    ports:
+      - "8082:5000"
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+    restart: unless-stopped

oam/netdisco/setup.sh

@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy NetDisco on tsys-librenms
+# Part of OAM platform [#337]
+# L2 network discovery and mapping via SNMP
+
+echo "=== Setting up NetDisco ==="
+
+mkdir -p /opt/oam/netdisco/{config,data,pgdata}
+
+# Write deployment config
+cat > /opt/oam/netdisco/config/deployment.yml <<'YMLEOF'
+# NetDisco deployment configuration [#337]
+# SNMP community: kn3lmgmt (v2c)
+
+database:
+  host: netdisco-postgresql
+  dbname: netdisco
+  user: netdisco
+  pass: netdisco
+
+snmp_auth:
+  - tag: default_v2c
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 2
+
+  - tag: default_v1
+    community: kn3lmgmt
+    read: true
+    write: false
+    version: 1
+
+schedule:
+  discoverall:
+    first: 60
+    every: 3600
+  macwalk:
+    first: 120
+    every: 900
+  arpwalk:
+    first: 150
+    every: 900
+  nbtwalk:
+    first: 180
+    every: 900
+
+node_freshness: 0
+YMLEOF
+
+# Write docker-compose
+cat > /opt/oam/netdisco/docker-compose.yml <<'DCEOF'
+services:
+  netdisco-postgresql:
+    image: netdisco/netdisco:2.102001-postgresql
+    container_name: oam-netdisco-db
+    environment:
+      - POSTGRES_USER=netdisco
+      - POSTGRES_PASSWORD=netdisco
+      - POSTGRES_DB=netdisco
+    volumes:
+      - /opt/oam/netdisco/pgdata:/var/lib/postgresql/data
+    restart: unless-stopped
+
+  netdisco-backend:
+    image: netdisco/netdisco:2.102001-backend
+    container_name: oam-netdisco-backend
+    depends_on:
+      - netdisco-postgresql
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+      - /opt/oam/netdisco/data:/home/netdisco/netdisco-sqlite
+    restart: unless-stopped
+
+  netdisco-web:
+    image: netdisco/netdisco:2.102001-web
+    container_name: oam-netdisco-web
+    depends_on:
+      - netdisco-postgresql
+      - netdisco-backend
+    environment:
+      - NETDISCO_DB_HOST=netdisco-postgresql
+      - NETDISCO_DB_NAME=netdisco
+      - NETDISCO_DB_USER=netdisco
+      - NETDISCO_DB_PASS=netdisco
+    ports:
+      - "8082:5000"
+    volumes:
+      - /opt/oam/netdisco/config/deployment.yml:/home/netdisco/environments/deployment.yml
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling NetDisco images ==="
+cd /opt/oam/netdisco && docker compose pull 2>&1
+
+echo "=== Starting NetDisco stack ==="
+cd /opt/oam/netdisco && docker compose up -d 2>&1
+
+echo "=== Waiting for PostgreSQL to initialize (30s) ==="
+sleep 30
+
+echo "=== Container status ==="
+docker ps --filter name=oam-netdisco --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
+
+echo "=== Adding devices to NetDisco ==="
+for dev in pfv-r5-core-01.knel.net pfv-r3-tor-mgmt-01.knel.net pfv-r3-tor-stor-01.knel.net pfv-r6-mgmt-01.knel.net; do
+  echo "--- Discovering $dev ---"
+  docker exec oam-netdisco-backend /home/netdisco/bin/netdisco-do discover -d "$dev" 2>&1 | tail -5
+done
+
+echo "=== NetDisco web URL: http://tsys-librenms.knel.net:8082 ==="
+echo "=== Done ==="

oam/oxidized/config

@@ -0,0 +1,38 @@
+---
+username: 'oxidized'
+password: 'see ~/.creds/oxidized.env'
+model: powerconnect
+interval: 3600
+use_syslog: false
+debug: false
+threads: 30
+timeout: 20
+retries: 3
+prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
+rest: 0.0.0.0:8888
+next_adds_job: false
+models: {}
+pid: /home/oxidized/.config/oxidized/pid
+log: /home/oxidized/.config/oxidized/logs/log
+snmp: false
+groups: {}
+models: {}
+resolve_dns: true
+interval: 3600
+output:
+  default: git
+  git:
+    user: oxidized
+    email: oxidized@tsys-librenms.knel.net
+    repo: /home/oxidized/.config/oxidized/git-repos
+source:
+  default: csv
+  csv:
+    file: /home/oxidized/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    gpg: false

oam/oxidized/docker-compose.yml

@@ -0,0 +1,12 @@
+services:
+  oxidized:
+    image: oxidized/oxidized:0.30.1
+    container_name: oam-oxidized
+    environment:
+      - OXIDIZED_CONFIG=/home/oxidized/.config/oxidized/config
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
+    ports:
+      - "8083:8888"
+    restart: unless-stopped

oam/oxidized/hooks/git-push.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Oxidized post-push hook: auto-push to Gitea after each config change
+# Mounted into container at /home/oxidized/.config/oxidized/hooks/checkout/git-push.sh
+set -euo pipefail
+cd "$1"
+export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new -i /home/oxidized/.config/oxidized/.ssh/id_ed25519"
+git push origin master 2>&1 || true

oam/oxidized/router.db

@@ -0,0 +1,4 @@
+pfv-r5-core-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r3-tor-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r3-tor-stor-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p
+pfv-r6-mgmt-01.knel.net:powerconnect:oxidized:xAlZI3j7m8Bkgf0p

oam/oxidized/setup.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy Oxidized on tsys-librenms
+# Part of OAM platform [#337]
+# Network device config backup/versioning
+
+echo "=== Setting up Oxidized ==="
+
+mkdir -p /opt/oam/oxidized/{config,git-repos,logs}
+
+# Config file
+cat > /opt/oam/oxidized/config/config <<'CFGEOF'
+---
+username: 'CHANGE_ME'
+password: 'CHANGE_ME'
+model: powerconnect
+interval: 3600
+use_syslog: false
+debug: false
+threads: 30
+timeout: 20
+retries: 3
+prompt: !ruby/regexp /([\w.@-]+[#>]\s?)n/
+rest: 0.0.0.0:8888
+next_adds_job: false
+pid: /home/oxidized/.config/oxidized/pid
+log: /home/oxidized/.config/oxidized/logs/log
+snmp: false
+resolve_dns: true
+interval: 3600
+output:
+  default: git
+  git:
+    user: oxidized
+    email: oxidized@tsys-librenms.knel.net
+    repo: /home/oxidized/.config/oxidized/git-repos
+source:
+  default: csv
+  csv:
+    file: /home/oxidized/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    gpg: false
+CFGEOF
+
+# Router DB (device list)
+cat > /opt/oam/oxidized/config/router.db <<'DBEOF'
+pfv-r5-core-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r3-tor-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r3-tor-stor-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+pfv-r6-mgmt-01.knel.net:powerconnect:CHANGE_ME:CHANGE_ME
+DBEOF
+
+# Docker compose
+cat > /opt/oam/oxidized/docker-compose.yml <<'DCEOF'
+services:
+  oxidized:
+    image: oxidized/oxidized:0.30.1
+    container_name: oam-oxidized
+    environment:
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/oxidized/config:/home/oxidized/.config/oxidized
+    ports:
+      - "8083:8888"
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling Oxidized image ==="
+cd /opt/oam/oxidized && timeout 120 docker compose pull 2>&1
+
+echo "=== Starting Oxidized ==="
+cd /opt/oam/oxidized && docker compose up -d 2>&1
+
+echo "=== Status ==="
+docker ps --filter name=oam-oxidized --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
+
+echo ""
+echo "=== IMPORTANT ==="
+echo "Oxidized deployed with placeholder credentials (CHANGE_ME)."
+echo "Update /opt/oam/oxidized/config/router.db with real switch credentials"
+echo "(username:password per device) to enable config backups."
+echo "Then restart: cd /opt/oam/oxidized && docker compose restart"
+echo ""
+echo "=== Oxidized REST API: http://tsys-librenms.knel.net:8083 ==="

oam/smokeping/General

@@ -0,0 +1,84 @@
+*** General ***
+
+owner = Charles N Wyble
+contact = reachableceo@knownelement.com
+mailhost = localhost
+cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
+syslogfacility = local0
+# each probe run every 5 minutes, 300 samples = 25 hours at 5min steps
+pagedir = /data/pages
+piddir  = /tmp
+imgcache = /data/cache
+imgurl  = /smokeping/images
+datadir = /data/_data
+dyndir = /data/dyn
+
+*** Database ***
+
+step     = 300
+pings    = 20
+
+# consfn mrhb steps total
+
+AVERAGE  0.5   1  100800
+AVERAGE  0.5  12  43200
+    MIN  0.5  12  43200
+    MAX  0.5  12  43200
+AVERAGE  0.5 144   7200
+    MAX  0.5 144   7200
+    MIN  0.5 144   7200
+
+*** Presentation ***
+
+template = /etc/smokeping/basepage.html
+
++ charts
+
++ detail
+width = 600
+height = 200
+nradius = 5
+
+*** Probes ***
+
++ FPing
+
+binary = /usr/local/bin/fping
+hostinterval = 1.5
+mininterval = 0.001
+offset = 0%
+
+*** Alerts ***
+
+to = reachableceo@knownelement.com
+from = smokeping-alert@tsys-librenms.knel.net
+
++bigloss
+type = loss
+# in percent
+pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
+comment = suddenly there is packet loss
+
++somedoss
+type = loss
+# in percent
+pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
+comment = incoming packet loss
+
++startloss
+type = loss
+# in percent
+pattern = >0%,>0%,>0%
+comment = loss at startup
+
++rttdetect
+type = rtt
+# in milliseconds
+pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
+comment = routing change?
+
++rttbadd
+type = rtt
+# in milliseconds
+pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
+comment = reroute?

oam/smokeping/Targets

@@ -0,0 +1,111 @@
+*** Targets ***
+
+probe = FPing
+
+menu = Top
+title = Network Latency Monitor
+
++ Infrastructure
+menu = Infrastructure
+title = Core Infrastructure
+
+++ pfv-r5-core-01
+menu = pfv-r5-core-01 (Core Switch)
+title = Dell PowerConnect 5448 - Rack 5 Core
+host = pfv-r5-core-01.knel.net
+
+++ pfv-r3-tor-mgmt-01
+menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
+title = Neyland 24T - Rack 3 Management TOR
+host = pfv-r3-tor-mgmt-01.knel.net
+
+++ pfv-r3-tor-stor-01
+menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
+title = Neyland 24T - Rack 3 Storage TOR
+host = pfv-r3-tor-stor-01.knel.net
+
+++ pfv-r6-mgmt-01
+menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
+title = Neyland 24T - Rack 6 Management Switch
+host = pfv-r6-mgmt-01.knel.net
+
++ Servers
+menu = Servers
+title = Server Infrastructure
+
+++ pfv-stor1
+menu = pfv-stor1
+title = Storage Server 1
+host = pfv-stor1.knel.net
+
+++ pfv-bms
+menu = pfv-bms (HomeAssistant)
+title = Home Assistant / UniFi Controller
+host = pfv-bms.knel.net
+
+++ tsys-librenms
+menu = tsys-librenms
+title = LibreNMS Monitoring Server
+host = tsys-librenms.knel.net
+
++ Wireless
+menu = Wireless
+title = Wireless Access Points
+
+++ ap-tablemount
+menu = ap-tablemount
+title = UniFi AP-AC-Lite (Table Mount)
+host = ap-tablemount.knel.net
+
+++ ap-wallmount
+menu = ap-wallmount
+title = UniFi AP-AC-LR (Wall Mount)
+host = ap-wallmount.knel.net
+
++ ProxmoxHosts
+menu = Proxmox Hosts
+title = Proxmox Hypervisor Nodes
+
+++ pfv-tsys1
+menu = pfv-tsys1
+title = Proxmox Host - Infrastructure
+host = pfv-tsys1.knel.net
+
+++ pfv-tsys3
+menu = pfv-tsys3
+title = Proxmox Host - Compute
+host = pfv-tsys3.knel.net
+
+++ pfv-tsys4
+menu = pfv-tsys4
+title = Proxmox Host - Storage + PBS
+host = pfv-tsys4.knel.net
+
+++ pfv-tsys5
+menu = pfv-tsys5
+title = Proxmox Host - Storage + Sandbox
+host = pfv-tsys5.knel.net
+
+++ pfv-tsys6
+menu = pfv-tsys6
+title = Proxmox Host - RackRental
+host = pfv-tsys6.knel.net
+
+++ pfv-tsys7
+menu = pfv-tsys7
+title = Proxmox Host - RackRental
+host = pfv-tsys7.knel.net
+
+++ pfv-tsys9
+menu = pfv-tsys9
+title = Proxmox Host - Infra + Compute
+host = pfv-tsys9.knel.net
+
++ Power
+menu = Power
+title = Power Devices
+
+++ pfv-garage-pdu-01
+menu = pfv-garage-pdu-1
+title = APC PDU (Garage)
+host = pfv-garage-pdu-1.knel.net

oam/smokeping/docker-compose.yml

@@ -0,0 +1,14 @@
+services:
+  smokeping:
+    image: linuxserver/smokeping:2.9.0
+    container_name: oam-smokeping
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/smokeping/config:/config
+      - /opt/oam/smokeping/data:/data
+    ports:
+      - "8081:80"
+    restart: unless-stopped

oam/smokeping/setup.sh

@@ -0,0 +1,237 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy Smokeping on tsys-librenms
+# Part of OAM platform [#337]
+
+echo "=== Setting up Smokeping ==="
+
+# Ensure directories
+mkdir -p /opt/oam/smokeping/{config,data}
+
+# General config
+cat > /opt/oam/smokeping/config/General <<'GENEOF'
+*** General ***
+
+owner = Charles N Wyble
+contact = reachableceo@knownelement.com
+mailhost = localhost
+cgiurl = http://tsys-librenms.knel.net:8081/smokeping/smokeping.cgi
+syslogfacility = local0
+pagedir = /data/pages
+piddir  = /tmp
+imgcache = /data/cache
+imgurl  = /smokeping/images
+datadir = /data/_data
+dyndir = /data/dyn
+
+*** Database ***
+
+step     = 300
+pings    = 20
+
+AVERAGE  0.5   1  100800
+AVERAGE  0.5  12  43200
+    MIN  0.5  12  43200
+    MAX  0.5  12  43200
+AVERAGE  0.5 144   7200
+    MAX  0.5 144   7200
+    MIN  0.5 144   7200
+
+*** Presentation ***
+
+template = /etc/smokeping/basepage.html
+
++ charts
+
++ detail
+width = 600
+height = 200
+nradius = 5
+
+*** Probes ***
+
++ FPing
+
+binary = /usr/local/bin/fping
+hostinterval = 1.5
+mininterval = 0.001
+offset = 0%
+
+*** Alerts ***
+
+to = reachableceo@knownelement.com
+from = smokeping-alert@tsys-librenms.knel.net
+
++bigloss
+type = loss
+pattern = ==0%,==0%,==0%,==0%,>0%,>0%,>0%
+comment = suddenly there is packet loss
+
++somedoss
+type = loss
+pattern = ==0%,==0%,==0%,==0%,==0%,>0%,>0%,>0%,>0%,>0%,>0%,>0%
+comment = incoming packet loss
+
++startloss
+type = loss
+pattern = >0%,>0%,>0%
+comment = loss at startup
+
++rttdetect
+type = rtt
+pattern = <10,<10,<10,<10,<10,<10,<10,>10,>10,>10
+comment = routing change?
+
++rttbadd
+type = rtt
+pattern = <50,<50,<50,<50,<50,<50,>50,>50,>50,>50
+comment = reroute?
+GENEOF
+
+# Targets config
+cat > /opt/oam/smokeping/config/Targets <<'TGTEOF'
+*** Targets ***
+
+probe = FPing
+
+menu = Top
+title = Network Latency Monitor
+
++ Infrastructure
+menu = Infrastructure
+title = Core Infrastructure
+
+++ pfv-r5-core-01
+menu = pfv-r5-core-01 (Core Switch)
+title = Dell PowerConnect 5448 - Rack 5 Core
+host = pfv-r5-core-01.knel.net
+
+++ pfv-r3-tor-mgmt-01
+menu = pfv-r3-tor-mgmt-01 (Rack 3 Mgmt TOR)
+title = Neyland 24T - Rack 3 Management TOR
+host = pfv-r3-tor-mgmt-01.knel.net
+
+++ pfv-r3-tor-stor-01
+menu = pfv-r3-tor-stor-01 (Rack 3 Storage TOR)
+title = Neyland 24T - Rack 3 Storage TOR
+host = pfv-r3-tor-stor-01.knel.net
+
+++ pfv-r6-mgmt-01
+menu = pfv-r6-mgmt-01 (Rack 6 Mgmt Switch)
+title = Neyland 24T - Rack 6 Management Switch
+host = pfv-r6-mgmt-01.knel.net
+
++ Servers
+menu = Servers
+title = Server Infrastructure
+
+++ pfv-stor1
+menu = pfv-stor1
+title = Storage Server 1
+host = pfv-stor1.knel.net
+
+++ pfv-bms
+menu = pfv-bms (HomeAssistant)
+title = Home Assistant / UniFi Controller
+host = pfv-bms.knel.net
+
+++ tsys-librenms
+menu = tsys-librenms
+title = LibreNMS Monitoring Server
+host = tsys-librenms.knel.net
+
++ Wireless
+menu = Wireless
+title = Wireless Access Points
+
+++ ap-tablemount
+menu = ap-tablemount
+title = UniFi AP-AC-Lite (Table Mount)
+host = ap-tablemount.knel.net
+
+++ ap-wallmount
+menu = ap-wallmount
+title = UniFi AP-AC-LR (Wall Mount)
+host = ap-wallmount.knel.net
+
++ ProxmoxHosts
+menu = Proxmox Hosts
+title = Proxmox Hypervisor Nodes
+
+++ pfv-tsys1
+menu = pfv-tsys1
+title = Proxmox Host - Infrastructure
+host = pfv-tsys1.knel.net
+
+++ pfv-tsys3
+menu = pfv-tsys3
+title = Proxmox Host - Compute
+host = pfv-tsys3.knel.net
+
+++ pfv-tsys4
+menu = pfv-tsys4
+title = Proxmox Host - Storage + PBS
+host = pfv-tsys4.knel.net
+
+++ pfv-tsys5
+menu = pfv-tsys5
+title = Proxmox Host - Storage + Sandbox
+host = pfv-tsys5.knel.net
+
+++ pfv-tsys6
+menu = pfv-tsys6
+title = Proxmox Host - RackRental
+host = pfv-tsys6.knel.net
+
+++ pfv-tsys7
+menu = pfv-tsys7
+title = Proxmox Host - RackRental
+host = pfv-tsys7.knel.net
+
+++ pfv-tsys9
+menu = pfv-tsys9
+title = Proxmox Host - Infra + Compute
+host = pfv-tsys9.knel.net
+
++ Power
+menu = Power
+title = Power Devices
+
+++ pfv-garage-pdu-01
+menu = pfv-garage-pdu-1
+title = APC PDU (Garage)
+host = pfv-garage-pdu-1.knel.net
+TGTEOF
+
+# Docker compose
+cat > /opt/oam/smokeping/docker-compose.yml <<'DCEOF'
+services:
+  smokeping:
+    image: linuxserver/smokeping:2.9.0
+    container_name: oam-smokeping
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/Chicago
+    volumes:
+      - /opt/oam/smokeping/config:/config
+      - /opt/oam/smokeping/data:/data
+    ports:
+      - "8081:80"
+    restart: unless-stopped
+DCEOF
+
+echo "=== Starting Smokeping container ==="
+cd /opt/oam/smokeping && docker compose up -d 2>&1
+
+echo "=== Waiting for startup ==="
+sleep 5
+
+echo "=== Container status ==="
+docker ps --filter name=oam-smokeping --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
+
+echo "=== Smokeping logs (last 20) ==="
+docker logs oam-smokeping --tail 20 2>&1
+
+echo "=== Done ==="

oam/unpoller/setup.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Deploy UNPoller on tsys-librenms
+# Part of OAM platform [#337]
+# UniFi controller polling for metrics export
+
+echo "=== Setting up UNPoller ==="
+
+mkdir -p /opt/oam/unpoller
+
+# Write config
+cat > /opt/oam/unpoller/up.conf <<'CONFEOF'
+[poller]
+debug = false
+quiet = false
+plugins = []
+
+[unifi.defaults]
+url = "https://pfv-bms.knel.net:8443"
+user = "CHANGE_ME"
+pass = "CHANGE_ME"
+sites = ["all"]
+ssl_verify = false
+
+[unifi.defaults.save_dpi]
+enabled = true
+
+[influxdb]
+enable = false
+url = "http://tsys-librenms.knel.net:8086"
+db = "unifi"
+
+[datadog]
+enable = false
+
+[prometheus]
+enable = true
+http_listen = ":9130"
+report_errors = true
+
+[loki]
+enable = false
+CONFEOF
+
+# Docker compose
+cat > /opt/oam/unpoller/docker-compose.yml <<'DCEOF'
+services:
+  unpoller:
+    image: ghcr.io/unpoller/unpoller:latest
+    container_name: oam-unpoller
+    environment:
+      - UP_UNIFI_DEFAULT_URL=https://pfv-bms.knel.net:8443
+      - UP_UNIFI_DEFAULT_USER=CHANGE_ME
+      - UP_UNIFI_DEFAULT_PASS=CHANGE_ME
+      - UP_UNIFI_DEFAULT_VERIFY_SSL=false
+      - UP_PROMETHEUS_ENABLE=true
+      - UP_PROMETHEUS_HTTP_LISTEN=:9130
+      - TZ=America/Chicago
+    ports:
+      - "8084:9130"
+    volumes:
+      - /opt/oam/unpoller/up.conf:/config/unifi-poller.conf:ro
+    restart: unless-stopped
+DCEOF
+
+echo "=== Pulling UNPoller image ==="
+cd /opt/oam/unpoller && timeout 120 docker compose pull 2>&1
+
+echo "=== Starting UNPoller ==="
+cd /opt/oam/unpoller && docker compose up -d 2>&1
+
+echo "=== Status ==="
+docker ps --filter name=oam-unpoller --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
+
+echo ""
+echo "=== IMPORTANT ==="
+echo "UNPoller deployed with placeholder UniFi credentials (CHANGE_ME)."
+echo "Update UP_UNIFI_DEFAULT_USER and UP_UNINI_DEFAULT_PASS env vars"
+echo "in /opt/oam/unpoller/docker-compose.yml with real UniFi controller"
+echo "credentials, then restart: cd /opt/oam/unpoller && docker compose up -d"
+echo ""
+echo "=== UNPoller Prometheus metrics: http://tsys-librenms.knel.net:8084 ==="

oam/weathermap/network.conf

@@ -0,0 +1,57 @@
+WIDTH 800
+HEIGHT 600
+HTMLOUTPUTFILE output/network.html
+IMAGEOUTPUTFILE output/network.png
+BACKGROUNDCOLOR 255 255 255
+TITLE "PFV Cluster Network Map"
+TIMEPOS 0 0
+TITLECOLOR 0 0 0
+TIMECOLOR 0 0 0
+HTMLSTYLE overlib
+KEYPOS DEFAULT 400 580
+KEYTEXTCOLOR 0 0 0
+KEYOUTLINECOLOR 0 0 0
+KEYBGCOLOR 255 255 255
+SCALE DEFAULT 0 10 0 255 0
+SCALE DEFAULT 10 50 255 255 0
+SCALE DEFAULT 50 100 255 128 0
+SCALE DEFAULT 100 1000 255 0 0
+NODE core_sw01
+  LABEL pfv-r5-core-01
+  POSITION 400 200
+NODE tor_mgmt
+  LABEL pfv-r3-tor-mgmt-01
+  POSITION 200 350
+NODE tor_stor
+  LABEL pfv-r3-tor-stor-01
+  POSITION 400 350
+NODE r6_mgmt
+  LABEL pfv-r6-mgmt-01
+  POSITION 600 350
+NODE pfv_tsys1
+  LABEL pfv-tsys1
+  POSITION 200 100
+NODE pfv_tsys9
+  LABEL pfv-tsys9
+  POSITION 600 100
+NODE pfv_bms
+  LABEL pfv-bms
+  POSITION 100 200
+LINK core_to_mgmt
+  NODES core_sw01 tor_mgmt
+  BANDWIDTH 1000M
+LINK core_to_stor
+  NODES core_sw01 tor_stor
+  BANDWIDTH 1000M
+LINK core_to_r6
+  NODES core_sw01 r6_mgmt
+  BANDWIDTH 1000M
+LINK tsys1_to_core
+  NODES pfv_tsys1 core_sw01
+  BANDWIDTH 1000M
+LINK tsys9_to_core
+  NODES pfv_tsys9 core_sw01
+  BANDWIDTH 1000M
+LINK bms_to_core
+  NODES pfv_bms core_sw01
+  BANDWIDTH 1000M

proxmox/docs/AUDIT-2026-07-29.md

@@ -1,452 +1,10 @@
-# Fleet Audit — 2026-07-29 (consolidated, storage-corrected)
+# proxmox/docs/AUDIT-2026-07-29.md
-
+
-> **The single authoritative audit.** Supersedes all prior audits and the
+> **Documentation moved to Discourse — the canonical source of truth.**
-> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict.
+>
-> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
+> **Audit snapshot 2026-07-29**
-
+>
-**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware
+> **Read it here:** https://community.turnsys.com/t/298
-dumps + ~05:40 CDT capacity probe). Storage configs read directly from
+>
-`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host.
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
+> Do not update content here — edit the Discourse wiki topic instead.*
-**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (retired / permanently offline)
-
----
-
-## 1. Executive summary
-
-| Goal | Status | Detail |
-|------|--------|--------|
-| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. |
-| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. |
-
-| Finding | Severity | § |
-|---------|----------|---|
-| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 |
-| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 |
-| ~~tsys6 S1 NFS export typo~~ (`S21`) — **FIXED by operator 2026-07-29**; all 8 stores active | Resolved | 6 |
-| ~~tsys9 missing S1~~ in storage.cfg — **FIXED by operator 2026-07-29**; S1 active | Resolved | 6 |
-| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 |
-| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 |
-| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 |
-
----
-
-## 2. Host capacity (live, 2026-07-29 ~05:40 CDT)
-
-| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load |
-|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------|
-| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 |
-| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 |
-| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 |
-| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 |
-| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 |
-| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 |
-| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 |
-
-**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6,
-cnode5 (706) deleted from tsys7. No host migrations performed yet.
-
----
-
-## 3. Storage topology (CORRECTED — read from live configs)
-
-> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its
-> own storage entries. The same physical disk can appear under different storage
-> IDs on different hosts. This was the source of the prior audit's errors.
-
-### 3.1 Physical disk → Proxmox storage ID mapping
-
-| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type |
-|---------------|--------|------------|-----------------|---------------------|------|
-| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD |
-| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) |
-| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD |
-| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown |
-| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown |
-| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** |
-| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD |
-| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD |
-| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD |
-| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** |
-
-### 3.2 Critical correction: `local-nonprod` = S1
-
-On tsys5, the storage entry is:
-```
-dir: local-nonprod
-    path /mnt/pfv-tsys5/S1
-```
-
-**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**,
-accessed locally as a directory storage instead of via NFS. The 16 sectestbed
-and preprod VMs "on local-nonprod" are physically on S1. They appear locked to
-tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage
-than `S1` (NFS type), even though they point to the same directory.
-
-**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan
-to move production VMs onto S1 would have stacked them on top of 16 sandboxes.
-
-### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday)
-
-| Tier | Spindles | Speed | Current use |
-|------|----------|-------|-------------|
-| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs |
-| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs |
-| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) |
-| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday |
-| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only |
-
----
-
-## 4. Per-spindle IOPS load (CORRECTED)
-
-| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** |
-|---------|--------|----------------|----------------|------------------------|
-| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** |
-| D5 (2TB) | tsys4 | 3 | 0 | 3 |
-| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** |
-| S2 (916GB) | tsys5 | 4 | 0 | 4 |
-| S3 (916GB) | tsys5 | 3 | 0 | 3 |
-| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity |
-
-**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.**
-
-The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real
-IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI
-pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to
-S1.** Target S2, S3, S4, and D5 for IOPS spreading.
-
----
-
-## 5. Storage bugs and config gaps found
-
-### 5.1 BUG: tsys6 S1 export path typo (S21) — RESOLVED ✓
-
-**Fixed by operator 2026-07-29.** The export path was corrected from `S21` to
-`S1`. Verified: `pvesm status` on tsys6 shows all 8 NFS stores active, including
-S1 (96 GB, 10.3% used by the sectestbed VMs visible via NFS).
-
-### 5.2 GAP: tsys9 missing S1 in storage.cfg — RESOLVED ✓
-
-**Fixed by operator 2026-07-29.** S1 entry added to tsys9's storage.cfg.
-Verified: all 8 NFS stores active on tsys9, S1 listing identical to tsys6.
-
-### 5.3 ORPHANED: D1 and D7 exports on tsys4
-
-tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg.
-- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped
-  templates only, no running VMs.
-- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped.
-
-These aren't causing problems but are underdocumented. Not actionable now.
-
----
-
-## 6. Current VM inventory (host + disk, post-user-changes)
-
-> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from
-> start of session. `Disk` = NFS export or local storage holding the primary
-> qcow2.
-
-### tsys1 — Infrastructure (11 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server |
-|------|------|------|-----|------|--------|
-| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 |
-| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 |
-| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 |
-| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 |
-| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 |
-| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 |
-| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 |
-| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 |
-| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 |
-| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 |
-| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 |
-
-### tsys3 — Compute (1 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server |
-|------|------|------|-----|------|--------|
-| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 |
-
-### tsys4 — Storage (1 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server |
-|------|------|------|-----|------|--------|
-| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL |
-
-### tsys5 — Storage + sandboxes (17 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Migratable? |
-|------|------|------|-----|------|--------|-------------|
-| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) |
-| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) |
-| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) |
-| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) |
-| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No |
-| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No |
-| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No |
-| ×10 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No |
-
-> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk
-> but accessed via dir storage, not NFS — Proxmox will not migrate them. To
-> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the
-> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.*
-
-### tsys6 — RackRental-ready (2 running, 1 stopped)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Status |
-|------|------|------|-----|------|--------|--------|
-| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running |
-| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running |
-| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** |
-
-### tsys7 — RackRental-ready (5 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server |
-|------|------|------|-----|------|--------|
-| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 |
-| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 |
-| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 |
-| 704 | TCTC | 4 | 6 GB | D2 | tsys4 |
-| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 |
-
-### tsys9 — Infra + control plane (6 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server |
-|------|------|------|-----|------|--------|
-| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 |
-| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 |
-| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 |
-| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 |
-| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 |
-| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 |
-
-### k3s control plane (ground truth via `kubectl`)
-
-3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1).
-All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.**
-
----
-
-## 7. Workload placement model
-
-| Workload family | Needs | Best hosts | Why |
-|-----------------|-------|-----------|-----|
-| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe |
-| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
-| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
-
----
-
-## 8. Answers to operator's questions
-
-### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?"
-
-**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB.
-After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**.
-
-**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most
-RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode.
-
-### 8.2 "Are tsys6/7 better for RackRental.net?"
-
-**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need
-more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s)
-— deferred to Friday + StorageClass work.
-
-### 8.3 "How should I deploy the DDR3?"
-
-Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST).
-tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB).
-Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM
-migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5.
-
----
-
-## 9. Hardware corrections
-
-**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has
-one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch —
-useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity
-and activate the CPU2 DIMM branch.
-
-**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but
-monitor after the planned RAM upgrade.
-
----
-
-## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced)
-
-> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve
-> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4
-> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA
-> pairs split across storage servers.
-
-### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Change |
-|------|------|------|-----|------|--------|--------|
-| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none |
-| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none |
-| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none |
-| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none |
-| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none |
-| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none |
-
-### tsys3 — Compute (1 running)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Change |
-|------|------|------|-----|------|--------|--------|
-| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none |
-
-### tsys4 — Storage (1 running)
-
-| VMID | Name | vCPU | RAM | Disk | Change |
-|------|------|------|-----|------|--------|
-| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
-
-### tsys5 — Storage + sandboxes (17 running, unchanged)
-
-All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay
-locked (idle sandboxes, low priority).
-
-### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Change |
-|------|------|------|-----|------|--------|--------|
-| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none |
-| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** |
-| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** |
-| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** |
-
-### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Change |
-|------|------|------|-----|------|--------|--------|
-| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none |
-| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** |
-| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** |
-| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** |
-| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** |
-| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
-| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** |
-| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) |
-| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** |
-| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
-
-### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%)
-
-| VMID | Name | vCPU | RAM | Disk | Server | Change |
-|------|------|------|-----|------|--------|--------|
-| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none |
-| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none |
-| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none |
-| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** |
-| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none |
-| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** |
-
-### End-state per-spindle IOPS load
-
-| Spindle | Server | # VMs | VMs on it |
-|---------|--------|-------|-----------|
-| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
-| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
-| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) |
-| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* |
-| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench |
-| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... |
-
-> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16
-> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7
-> production VMs each is fine (they're 916 GB disks with light per-VM I/O).*
-
-**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.**
-
-HA pairs stay split:
-- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓
-- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓
-
----
-
-## 11. Action checklist (do in this order)
-
-### Phase 0 — fix storage config bugs first — DONE ✓
-
-- ~~Fix tsys6 S1 typo~~ — **operator fixed 2026-07-29** (verified: 8/8 stores active)
-- ~~Add S1 to tsys9 storage.cfg~~ — **operator fixed 2026-07-29** (verified)
-
-### Phase 1 — cleanup
-
-- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster
-- **Confirm cnode5 (706) gone** — already deleted by operator ✓
-
-### Phase 2 — disk migrations off D2 (IOPS rebalancing)
-
-> Do disk moves on VMs that are staying on their current host first.
-
-- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5)
-- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6)
-- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7)
-- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7)
-- **TCTC (704):** disk **D2 → S3** (host stays tsys7)
-- **cnode2 (705):** disk **D2 → S2** (host stays tsys7)
-
-### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs)
-
-- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3**
-- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4**
-- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
-- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2**
-- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3**
-- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2**
-
-### Phase 4 — grow the compute wnode
-
-- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required)
-
-### Phase 5 — join workers to k3s
-
-- Join wnodes via `k8s/join-servers.sh`
-- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
-
-### Verification after each phase
-
-- `qm list` per host — confirm VM count + status
-- `cat /proc/meminfo` — confirm actual RAM ≤ 80%
-- `kubectl top nodes` — confirm cnodes healthy
-- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up
-- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills
-
----
-
-## 12. Future: local storage as ephemeral scratch
-
-> Codified as standing policy in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.
-
-The fleet has substantial **unused local storage IOPS** on several hosts. OS
-drives stay on NFS (no per-node disk redundancy — see policy), but local storage
-is ideal for **ephemeral scratch** (container image cache, build artifacts,
-`emptyDir` volumes) where data loss on node failure is acceptable.
-
-**Per-host opportunity:**
-
-| Host | Local | Available | Scratch-suitable? |
-|------|-------|-----------|-------------------|
-| tsys3 | NVMe (PM961) | 349 GB | **Yes — best existing** |
-| tsys9 | SSD (PNY CS900) | 136 GB | **Yes** |
-| tsys2 | NVMe + SSD | 512 GB + 1 TB | **Yes (incoming)** |
-| tsys5 | HDD (Hitachi) | ~1.7 TB | Yes (bulk) |
-| tsys6 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
-| tsys7 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
-
-**The tsys6/7 problem:** they are the RackRental hosts (most in need of fast
-image pulls) but their local storage is USB 2.0 — slower than NFS. The real fix
-is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5
-empty SATA ports). Deferred to a physical-hardware session.
-
-**Implementation:** post-Friday, once SSD/NVMe tiers are installed and the k8s
-StorageClass design is finalized. Wnodes with local storage get a `/scratch`
-mount for ephemeral data; persistent data stays on NFS. See
-[`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.4 for the StorageClass mapping.

proxmox/docs/AUDIT-2026-07-30.md

@@ -1,478 +1,10 @@
-# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)
+# proxmox/docs/AUDIT-2026-07-30.md
-
+
-> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md)
+> **Documentation moved to Discourse — the canonical source of truth.**
-> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are
+>
-> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
+> **Audit snapshot 2026-07-30**
-
+>
-**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)
+> **Read it here:** https://community.turnsys.com/t/298
-+ switch SNMP/conman captures (~03:50–04:00 UTC).
+>
-**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`,
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all
+> Do not update content here — edit the Discourse wiki topic instead.*
-read-only. No changes made to any system.
-**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
-**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild)
-
----
-
-## 1. Physical network topology
-
-### 1.1 Diagram
-
-```
-  RACK 5 (core/data)                           RACK 3 (compute/RackRental)
-  ┌─────────────────────────┐                  ┌──────────────────────────┐
-  │                         │                  │                          │
-  │  pfv-core-sw01          │    ch1 LAG       │  pfv-tor3-stor           │
-  │  PowerConnect 5448      │◄───3×1G──────────►│  Neyland 24T (PC5324)   │
-  │  (mgmt+storage VLAN)    │   (g13-15)       │  (storage TOR)           │
-  │                         │   g16 UP, not    │                          │
-  │  VLAN1 (mgmt)           │   in LAG; g17    │  g8-9  → tsys6 bond0     │
-  │  VLAN1000 (storage)     │   DOWN           │  g10-11 → tsys7 bond0    │
-  │                         │                  │  g20-23 → ch1 (3 active) │
-  │  g8   → tsys4 nic0      │                  │                          │
-  │  g19  → tsys5 nic0      │                  │  pfv-tor3-mgmt           │
-  │  g31  → tsys4 enx (USB) │                  │  Neyland 24T (PC5324)    │
-  │  g33-34 → tsys5 ch3     │                  │  (mgmt TOR)              │
-  │  ch3 → tsys5 bond (1 ln)│                  │  g7 → tsys6 nic0         │
-  │                         │                  │  g3 → tsys7 nic0         │
-  │  tsys4 (storage+PBS)    │                  │                          │
-  │  tsys5 (storage+sectest)│                  │  tsys6 (RackRental)      │
-  │  tsys1 (infra)          │                  │  tsys7 (RackRental)      │
-  │  tsys3 (compute)        │                  │                          │
-  │  tsys9 (infra+compute)  │                  │                          │
-  └─────────────────────────┘                  └──────────────────────────┘
-```
-
-### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK
-
-| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? |
-|----------|-------------------|--------------------|----------|
-| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below |
-| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG |
-| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate |
-| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 |
-| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection |
-| **VLAN** | access vlan 1000 | access vlan 1000 | OK |
-
-**Three root causes of cross-rack underperformance:**
-
-1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on
-   `g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th
-   physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps,
-   not the intended 4).
-
-2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4`
-   (includes TCP/UDP source+dest ports — correct for NFS with nconnect).
-   core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With
-   layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one
-   physical link**. Even though the clients use `nconnect=4` (4 TCP connections
-   per mount), they share the same src/dst IP → same hash → **max 1 Gbps per
-   server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.
-
-3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the
-   cross-rack trunk. `mode on` provides no link-failure detection, no partner
-   validation, no dynamic negotiation. A dead cable stays "active" until
-   manually discovered.
-
-### 1.3 Per-host network state (ground truth)
-
-| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP |
-|------|------|----------|-------------|-------|------|---------------|-------|------------|
-| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 |
-| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 |
-| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 |
-| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 |
-| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 |
-| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 |
-| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 |
-
-**Key observations:**
-- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously.
-- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy.
-- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is
-  the LACP problem the operator has been struggling with. The switch side (core-sw01
-  ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
-- **tsys1 also uses a USB dongle** for storage.
-- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5
-  use layer2+3 (suboptimal for NFS multi-connection).
-
-### 1.4 Switch inventory (SNMP + conman ground truth)
-
-| Switch | IP | Model | Firmware | Role | Console |
-|--------|----|-------|----------|------|---------|
-| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 |
-| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 |
-| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 |
-| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? |
-
-**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c.
-**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit
-(udev symlink naming bug — singular vs plural — prevented consoles from working
-after reboot).
-
-### 1.5 Port error counters (core-sw01)
-
-| Port | In Errors | Notes |
-|------|-----------|-------|
-| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. |
-| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. |
-| **g26** | **73** | Connected to a management-network device. |
-| All others | 0 | Healthy. |
-
-tor3-stor shows **zero errors** on all active ports.
-
----
-
-## 2. Storage topology (CORRECTED — current ground truth)
-
-### 2.1 Storage tier definitions
-
-| Tier | Definition | Performance | Use case |
-|------|-----------|-------------|----------|
-| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads |
-| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives |
-| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only |
-
-### 2.2 tsys4 disk inventory (CHANGED since last audit)
-
-| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes |
-|--------|-------|------|-----------|-------|------------|------|-------|
-| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. |
-| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. |
-| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** |
-| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
-| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. |
-| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). |
-| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. |
-
-**Changes from AUDIT-2026-07-29:**
-- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB).
-  D3 is now Tier 3 (was previously listed as unreliable USB SSD).
-- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox.
-  Potential new Tier 2 storage.
-- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup.
-- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone.
-
-### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)
-
-| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes |
-|--------|-------|------|-----------|-------|-----------|------|-------|
-| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
-| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
-| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
-| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). |
-| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. |
-| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. |
-| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. |
-
-**Changes from AUDIT-2026-07-29:**
-- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest-
-  performance local storage in the fleet (true NVMe).
-- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal
-  SATA. Not yet mounted/configured.
-- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented
-  out. /etc/exports still exports S3 (stale).
-- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented.
-  storage.cfg still has a broken `dir: S4` entry (points to root filesystem).
-- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate
-  916GB" in old audit).
-- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention.
-
-### 2.4 NFS export map (current, verified)
-
-| Server | Export | NFS ID | Disk | Tier | Used |
-|--------|--------|--------|------|------|------|
-| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% |
-| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) |
-| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% |
-| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) |
-| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) |
-| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) |
-| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — |
-
-### 2.5 NFS client mount status (nconnect)
-
-| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash |
-|------|-------------------|-------------------|-------------|------|
-| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 |
-| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 |
-| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
-| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
-
-**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4).
-**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client).
-
-### 2.6 Per-spindle IOPS load (running VMs only)
-
-| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes |
-|---------|--------|-----------------|----------------|-------|------|-------|
-| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 |
-| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 |
-| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) |
-| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 |
-| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. |
-| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
-| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
-| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) |
-
-**D2 still carries 13 running VMs** — the IOPS hotspot persists.
-
----
-
-## 3. Host capacity (live ground truth, 2026-07-30)
-
-| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy |
-|------|-----|---------|-----|-------------|-------------|------|----------|
-| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% |
-| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% |
-| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% |
-| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% |
-| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% |
-| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% |
-| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% |
-
-**Changes since AUDIT-2026-07-29:**
-- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped.
-- **tsys9 RAM 62%→69%** — approaching 80% goal.
-- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit).
-- **All other hosts healthy** — well under 80%.
-
----
-
-## 4. Fleet consistency audit
-
-### 4.1 PVE/kernel versions
-
-| Host | PVE | Kernel | Status |
-|------|-----|--------|--------|
-| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** |
-| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
-
-### 4.2 Performance tuning state
-
-| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect |
-|------|----------|------------|--------|---------------|---------------|-----------|----------|
-| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) |
-| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A |
-| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** |
-| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** |
-| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) |
-| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** |
-| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A |
-
-**Inconsistencies:**
-- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong
-  (`throughput-performance` should be `network-throughput`). Missing nconnect.
-- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere —
-  expected, not a bug).
-- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7).
-
----
-
-## 5. VM inventory + config audit (running VMs only)
-
-### 5.1 Per-VM disk + CPU + network settings
-
-> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good
-> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS.
-> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled.
-
-| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes |
-|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------|
-| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 |
-| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt |
-| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | |
-| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
-| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
-| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | |
-| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | |
-| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | |
-| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | |
-| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | |
-| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server |
-| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode |
-| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) |
-| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB |
-| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM |
-| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | |
-| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | |
-| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk |
-| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | |
-| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | |
-| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | |
-| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk |
-| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk |
-| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | |
-| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | |
-| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G |
-| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | |
-
-### 5.2 VM optimization findings
-
-| Finding | VMs affected | Impact | Fix |
-|---------|-------------|--------|-----|
-| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
-| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
-| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
-| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
-| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |
-
----
-
-## 6. Console infrastructure status (fixed during this audit)
-
-**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with
-`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
-`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
-console ports failed until `setup.sh`'s manual fallback re-created symlinks.
-
-**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
-all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
-consoles now working (verified via conman-console.py).
-
-**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
-conmand systemd service from starting.
-
----
-
-## 7. Findings summary (prioritized)
-
-### Critical (cross-rack performance)
-
-| # | Finding | Impact | Type |
-|---|---------|--------|------|
-| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
-| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
-| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
-| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |
-
-### High (storage configuration)
-
-| # | Finding | Impact | Type |
-|---|---------|--------|------|
-| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
-| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
-| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
-| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity |
-| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config |
-
-### Medium (fleet consistency)
-
-| # | Finding | Impact | Type |
-|---|---------|--------|------|
-| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade |
-| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning |
-| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning |
-| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning |
-| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config |
-| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup |
-
-### Low (monitoring / investigation)
-
-| # | Finding | Impact | Type |
-|---|---------|--------|------|
-| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate |
-| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring |
-| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config |
-| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup |
-
----
-
-## 9. Observability stack (operator direction)
-
-The operator has defined a clear monitoring architecture with three tools,
-each serving a distinct role. Do NOT duplicate functionality across tools.
-
-### 9.1 Current architecture
-
-| Tool | Where it runs | Scope | Role |
-|------|---------------|-------|------|
-| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts |
-| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover |
-| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent |
-| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) |
-
-### 9.2 Division of responsibility (no duplication)
-
-| Metric class | Tool | Notes |
-|-------------|------|-------|
-| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) |
-| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications |
-| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) |
-| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations |
-
-### 9.3 Saturday "OAM Day" plan (2026-08-01)
-
-The operator has designated Saturday as a dedicated operations/admin/management
-day with the goal of achieving **100% instrumentation coverage**:
-
-- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover)
-- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog
-      relay, or Loki/Grafana stack)
-- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale
-      devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12)
-- [ ] Beszel agent coverage on all 7 Proxmox hosts
-- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU
-      via powerman, any temp sensors)
-- [ ] Verify alerting paths end-to-end (test each alert fires correctly)
-
-> Much of the groundwork is already done: NUT is live on tsys1, powerman is
-> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live
-> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring
-> continuous monitoring + alerting on top of these.
-
----
-
-## 8. Recommendations (for operator review — NO changes applied)
-
-### 8.1 Switch changes (require maintenance window + operator approval)
-
-1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes
-   the 4th live cable (g16) part of the LAG.
-2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor).
-   This unlocks multi-link distribution for nconnect NFS traffic.
-3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches
-   for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
-4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor.
-5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr)
-   on core-sw01.
-
-> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the
-> highest-impact changes with the lowest risk. LACP conversion (#3) is optional
-> if the static trunk is working reliably otherwise. All changes can be done
-> read-only-verified via the conman + SNMP tooling in this repo.
-
-### 8.2 Storage configuration (host-side, no switch changes needed)
-
-1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`)
-   per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact)
-   and sectestbed scratch.
-2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or
-   similar. Offloads the 93%-full Samsung 860 PRO.
-3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken
-   `dir: S4` entry in storage.cfg.
-4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or
-   keep local-only for PBS scratch.
-
-### 8.3 Fleet consistency + tuning (host-side)
-
-1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
-2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5:
-   - Fix rmem/wmem to 128MB (tsys4)
-   - Fix tuned profile (tsys4 → network-throughput)
-   - Add nconnect=4 to tsys5 NFS client mounts
-   - Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
-3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS-
-   backed qcow2 disks (requires VM stop/start).
-
-### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md))
-
-The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty
-Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading
-that didn't exist in the 07-29 audit. Detailed placement plan deferred to the
-operator review of this document.

proxmox/docs/AUDIT-2026-08-05.md

@@ -1,220 +1,10 @@
-# Ground-Truth Audit — 2026-08-05
+# proxmox/docs/AUDIT-2026-08-05.md
 
-> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
+> **Documentation moved to Discourse — the canonical source of truth.**
-> network discovery (nmap, SNMP, SSH, API queries). Supersedes
-> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
 >
-> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
+> **Audit snapshot 2026-08-05 (current authoritative)**
-> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
+>
-> Proxmox `qm list`/`pvecm`, Docker/container inspection.
+> **Read it here:** https://community.turnsys.com/t/298
-
+>
-## TL;DR — What Changed Since 2026-07-30
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-The fleet grew significantly and the repository was restructured into a
-module/subproject layout. Headline deltas:
-
-| Metric | 2026-07-30 | 2026-08-05 | Δ |
-|--------|-----------|-----------|---|
-| VMs (running) | 28 | **63** | +125% |
-| DNS zones | 124 | **167** | +35% |
-| DHCP leases | 333 | **753** | +126% |
-| Proxmox hosts | 7 | 7 | — |
-| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
-| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
-| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
-| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
-
-## Repository Restructure (DONE by operator)
-
-Top-level dirs were consolidated into modules. **Old flat dirs no longer
-exist** at the repo root:
-
-| Was (repo root) | Now |
-|-----------------|-----|
-| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
-| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
-| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
-| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
-
-Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
-
-> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
-> the OLD paths. These are being updated in this same session.
-
-## Network Switches
-
-SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
-the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
-
-| IP | sysName | Model | Status |
-|----|---------|-------|--------|
-| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
-| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
-| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
-| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
-| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
-
-**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
-161 (SNMP) responds with `kn3lmgmt`.
-
-### LACP / Cross-Rack Trunk (core-sw01 ch1)
-
-- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
-- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
-- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
-  aggregate bundle speed via standard ifSpeed, so **member count + hash
-  algorithm need console verification** (conman requires a PTY — run manually):
-  ```bash
-  conman -d pfv-tsys4:7890 -f pfv-core-sw01
-  # then: show interfaces status port-channel 1
-  #       show running-config | begin port channel
-  ```
-
-## Proxmox Hosts (7 standalone, managed via PDM)
-
-All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
-**Not clustered** — standalone installs (corosync Expected votes=1 each).
-
-| Host | RAM used/total | VMs | Uptime | Notes |
-|------|---------------|-----|--------|-------|
-| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
-| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
-| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
-| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
-| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
-| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
-| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
-
-**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
-
-### tsys5 Storage (RESOLVED)
-
-Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
-
-| Disk | Mount | Use |
-|------|-------|-----|
-| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
-| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
-| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
-| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
-| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
-| local-lvm (1.67 TB thin) | LVM | 6.9% |
-
-NFS from tsys4: D2, D3, D5 all mounted and active.
-
-## DNS / NTP / DHCP (netinfra)
-
-### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
-
-| Service | State | Detail |
-|---------|-------|--------|
-| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
-| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
-| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
-| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
-
-### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
-
-**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
-but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
-full rebuild as redundant node — see Pending.
-
-## Kubernetes (pfv-k8s) — DOWN
-
-| Node | Tailscale IP | k3s API :6443 | Location |
-|------|-------------|---------------|----------|
-| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
-| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
-| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
-
-- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
-- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
-  — an IP matching NO current cnode. Cluster must be rebuilt.
-- Worker nodes (5) all online on Tailscale, **not joined**:
-  pfv-k8s-wnode-tsys3/5/6/7/9.
-
-## Monitoring / Observability Stack
-
-| System | Host | Status | Access |
-|--------|------|--------|--------|
-| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
-| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
-| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
-| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
-| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
-
-## Console Management (pfv-tsys4)
-
-ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
-All 7 `/dev/consoles/<name>` symlinks present:
-
-```
-pfv-core-sw01→ttyUSB2  pfv-tor3-mgmt→ttyUSB1  pfv-tor3-stor→ttyUSB6
-pfv-rrinfra-rtr→ttyUSB5  pfv-r2-tor-top→ttyUSB7  subodev-torsw→ttyUSB0
-pfv-r2-sw→ttyUSB3
-```
-
-**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
-interactive terminal, or the conman client directly.
-
-## Tailscale Fleet (80 nodes)
-
-Three full environment tiers now exist, each mirroring the same service set:
-
-- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
-  tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
-  tsys-umbrel, tsys-voip
-- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
-  preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
-  preprod-rancherplatform, preprod-siem, preprod-voip
-- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
-  sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
-  sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
-  sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
-  sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
-
-Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
-pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
-pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
-kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
-stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
-
-> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
-
-## Identified LAN Hosts (non-switch)
-
-| IP | Identity | Evidence |
-|----|----------|----------|
-| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
-| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
-| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
-| 192.168.3.1 | gateway/router | Linux + xrdp |
-
-## Divergences vs STATUS.md (2026-07-30)
-
-1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
-2. **DNS zones 124→167**, **DHCP leases 333→753**
-3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
-4. **tsys5 NVMe+SSD** now configured (was "not configured")
-5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
-6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
-7. **k8s cluster** now DOWN (was LIVE)
-8. **netinfra-02** wiped (was redundant secondary)
-9. **Technitium** now Docker container (was systemd dnsServer)
-10. **NTP** is ntpsec (STATUS assumed chrony)
-11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
-12. **Tailscale fleet** 3× larger with 3 environment tiers
-
-## Pending / Recommended Next Actions
-
-*(From `nextprompt` — reference only; awaiting operator direction)*
-
-- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
-  zone sync + DHCP failover-peer)
-- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
-- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
-- **Vault** migrate all API keys (HIGH priority)
-- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
-- **Home Assistant**: own all environmental monitoring/reporting
-- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
-- **Repo gardening**: README/AGENTS/docmap path fixes (this session)

proxmox/docs/CAPACITY-GOALS.md

@@ -1,220 +1,10 @@
-# Capacity & Sizing Goals
+# proxmox/docs/CAPACITY-GOALS.md
 
-**Date:** 2026-07-29
+> **Documentation moved to Discourse — the canonical source of truth.**
-**Status:** Authoritative. These are the operator's standing targets. Every
+>
-fleet audit and VM-placement decision is evaluated against them.
+> **Capacity targets + workload placement model**
-
+>
-> Companion to [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (the current
+> **Read it here:** https://community.turnsys.com/t/304
-> placement analysis) and [`PROJECT.md`](PROJECT.md) (the fleet hardware report).
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
----
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-## 1. Memory target: 80% steady-state
-
-**Rule:** Keep each node at or below **80% physical RAM utilization at steady
-state.** Occasional bursts above 80% are acceptable (that is what swap is for);
-the *sustained* idle-to-average load must sit at ~80% or below.
-
-**Two metrics must both be tracked:**
-
-| Metric | Definition | Target |
-|--------|-----------|--------|
-| **Actual usage** | `MemTotal - MemAvailable` from `/proc/meminfo` | **≤ 80%** steady state (the real rule) |
-| **Allocation commitment** | sum of running-VM `memory:` vs host RAM | planning ceiling — keep < 100% where possible |
-
-**Why both:** KSM (same-page merging) and virtio-balloon mean *actual* usage is
-often well below the sum of VM allocations (e.g. tsys1 allocates 115% but uses
-72% actual). Actual usage is the steady-state truth; allocation is the
-worst-case ceiling if every VM suddenly fills its reservation. A host is
-"healthy" when actual ≤ 80% **and** allocation isn't so far over 100% that a
-balloon-deflate storm could OOM it.
-
-**Tolerance:** brief actual spikes to 90-100% are fine. Persistent swap use
-(>~1 GB on a steady basis) means the host is over-committed and VMs should
-migrate off.
-
-## 2. CPU target: ~50% utilization at idle
-
-**Rule:** Ideally, nodes sit around **50% CPU utilization at idle.**
-
-This is a *utilization* goal, not a headroom goal. The fleet is
-**CPU-underutilized** (most hosts idle at 0-6%). On a shoestring budget, idle
-silicon is wasted investment and (for the PowerEdge hosts) wasted electricity.
-The intent is to **pack workloads** so the baseline load consumes roughly half
-the compute, reserving the other half for bursts.
-
-**Practical implications:**
-
-- The older, power-hungry hosts (tsys6 Nehalem, tsys5 Westmere) should either be
-  doing useful work at ~50% or be candidates to power down / solar-gate when
-  idle. Running them at 0-6% wastes power per core.
-- The newer, efficient hosts (tsys9 Comet Lake, tsys3 Skylake) are cheap to run
-  idle and are the natural home for always-on control-plane + infra.
-- vCPU overcommit is fine for idle workloads; the 50% target is about *actual*
-  utilization, not vCPU ratios.
-
-## 3. Workload placement model
-
-The fleet serves three distinct workload families. Placement should follow CPU
-quality + storage speed + RAM size, in that order of relevance per family.
-
-### 3.1 ETL / HPC / cross-compile (CPU + storage bound)
-
-Large geospatial/weather ETL, RTL/firmware simulation, cross-compile pipelines.
-**Needs high IPC (modern cores) + fast storage (SSD/NVMe).** RAM is moderate
-(8-32 GB per job).
-
-| Host | CPU | Storage | Fit |
-|------|-----|---------|-----|
-| **tsys9** | i5-10500 Comet Lake (2020) — **best IPC in fleet** | local SSD (PNY CS900) | **Primary compute node** |
-| **tsys3** | Xeon E3-1535M v5 Skylake (2015) | local NVMe (Samsung PM961) | **Fast compute node** |
-| tsys2 (incoming) | i7-7820HQ Kaby Lake (2017) | NVMe 960 PRO + SSD 850 EVO | **Fast compute node** (post-rebuild) |
-
-These three hosts have the newest CPUs and the only fast local storage. Their
-wnodes should be the **compute-tier** workers, sized larger (more cores/RAM) and
-labeled `workload=compute`.
-
-### 3.2 RackRental.net — network labs on demand (RAM + IOPS bound, CPU-light)
-
-Business idea: rent out ephemeral network labs (containerlab topologies) on
-demand. Many containers / network namespaces, short-lived, mostly idle once up.
-**Needs massive RAM + fast container-image pulls. CPU quality is secondary**
-(labs are idle between keystrokes).
-
-| Host | CPU | RAM | Fit |
-|------|-----|-----|-----|
-| **tsys7** | 2× E5-2630 v2 Ivy Bridge (2013), 24 threads | **189 GB** | **Primary RackRental host** |
-| **tsys6** | 2× E5530 Nehalem (2009), 16 threads | **126 GB** | **Secondary RackRental host** |
-
-**Verdict: tsys6/7 are the correct RackRental hosts.** Their weak CPUs are
-irrelevant for idle network labs; their RAM is unmatched. Their wnodes should be
-labeled `workload=rackrental`.
-
-**Caveat (deferred):** their local-lvm is USB 2.0 portable HDD (~30 MB/s), which
-is terrible for container image pulls. This is a *disk* problem, not a RAM/CPU
-problem, and is deferred to the post-Friday storage-class work. RAM-wise they are
-ready today.
-
-### 3.3 Infrastructure + k8s control plane (always-on, low resource)
-
-netinfra, UCS, LibreNMS, SIEM, tailscale-router, PBS, k3s cnodes. Lightweight,
-must stay up. Lives on tsys1 + tsys9 (and the cnodes spread across hosts for
-etcd diversity).
-
----
-
-## 4. Host role summary (target)
-
-| Host | CPU era | RAM | Primary role | Wnode class |
-|------|---------|-----|--------------|-------------|
-| tsys1 | Haswell 2013 | 32 GB (maxed) | Infra + control plane | light (4c/4G) |
-| tsys2 | Kaby Lake 2017 | 32 GB | Compute (incoming) | compute |
-| tsys3 | Skylake 2015 | 32 GB | Compute | compute |
-| tsys4 | Haswell 2013 | 16 GB | Storage (NFS + PBS) | — |
-| tsys5 | Westmere 2010 | 96 GB | Storage + fast-tier hub | compute (NVMe, post-Friday) |
-| tsys6 | Nehalem 2009 | 126 GB | RackRental | rackrental |
-| tsys7 | Ivy Bridge 2013 | 189 GB | RackRental | rackrental |
-| tsys9 | Comet Lake 2020 | 24 GB | Infra + control plane + compute | compute (growable) |
-
-## 5. DDR3 memory policy
-
-**Do not blindly add RAM to tsys6/7.** They idle at 6%/9% actual usage with
-117/171 GB free — they are RAM-saturated in the wrong direction (over-provisioned,
-not starved). The fleet's RAM pressure is on the *infra* hosts (tsys1/5/9
-over-allocated), and that is solved by **migrating VMs** to tsys6/7's existing
-free RAM, not by installing sticks.
-
-Compatibility is decisive and is documented per-host in
-[`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) §5. Short version: tsys5/6/7 require
-**Registered ECC (RDIMM)**; only RDIMM sticks will POST. Consumer UDIMM
-(Patriot / most Kingston) fits essentially nowhere useful here.
-
----
-
-## 6. Local storage policy: ephemeral scratch, not OS drives
-
-**Principle:** Local disk storage is a **scratch tier**, not a boot/OS tier.
-VM OS drives stay on NFS (tsys4/tsys5) where redundancy is handled by spreading
-across physical disks and -01/-02 HA pairs. Local storage is used for ephemeral
-workloads where data loss on node failure is acceptable by definition.
-
-### 6.1 Why OS drives stay on NFS
-
-Individual Proxmox nodes have **no local disk redundancy** (single disk, no
-RAID, no ZFS mirror). If a host's local disk dies, every VM on it is gone.
-Redundancy is instead achieved at the **storage-server + application layer**:
-
-- **-01/-02 HA pairs** (netinfra, UCS): each half on a different NFS server
-- **k8s spreading**: cnode/wnode disks spread across D2/D5/S2/S3/S4 so no
-  single disk or server failure takes down the whole cluster
-- **PBS backups**: all VMs are backed up to Proxmox Backup Server regardless of
-  disk tier
-
-This is deliberate. Moving OS drives to local storage would trade NFS network
-latency for single-disk-failure risk. Not worth it for persistent workloads.
-
-### 6.2 What local storage IS good for: ephemeral scratch
-
-Workloads that produce or consume temporary data that can be discarded on node
-failure. These are exactly the workloads where NFS latency hurts most and local
-IOPS help most:
-
-| Use case | Why local scratch helps | Examples |
-|----------|------------------------|---------|
-| **Container image pulls** | Image layers write to local disk during pull; NFS-over-HDD is seek-bound and slow | RackRental labs, any pod startup |
-| **Build artifacts** | Cross-compile / firmware builds write hundreds of small files; local SSD/NVMe is 10-100x faster than NFS-HDD | ETL, cross-compile, CI pipelines |
-| **Temporary processing** | Intermediate data that doesn't need to survive a reboot | Weather/GIS transform scratch, log rotation |
-| **kubelet ephemeral storage** | `emptyDir` volumes, container writable layers | All k8s pods |
-
-### 6.3 Per-host local storage inventory and suitability
-
-| Host | Local storage | Type | Available | Scratch-suitable? | Notes |
-|------|--------------|------|-----------|-------------------|-------|
-| **tsys2** | NVMe 960 PRO + SSD 850 EVO | NVMe + SSD | 512 GB + 1 TB | **Yes — best in fleet** | Incoming (post-rebuild). No NFS needed at all |
-| **tsys3** | local-lvm (Samsung PM961) | NVMe | 349 GB | **Yes — excellent** | Fastest existing local storage. 0% used |
-| **tsys9** | local-lvm (PNY CS900) | SSD | 136 GB | **Yes — excellent** | Low latency. 0% used |
-| **tsys5** | local-lvm (Hitachi) | HDD | ~1.7 TB | Yes (bulk) | Large capacity, HDD speed. Storage host — scratch is secondary |
-| **tsys1** | local-lvm | HDD | ~90 GB | Marginal (small) | Barely worth it; tsys1 is infra-loaded anyway |
-| **tsys4** | local-lvm | HDD | ~62 GB | No | PBS host; minimal free space |
-| **tsys6** | local-lvm (WD My Passport) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
-| **tsys7** | local-lvm (WD portable) | **USB 2.0 HDD** | 1.7 TB | **NO — 30 MB/s** | Slower than NFS. Do not use for VMs |
-
-**tsys6/7 are the hard problem.** They are the RackRental hosts — exactly the
-workload that benefits most from local scratch (fast container image pulls) —
-but their local storage is USB 2.0 portable HDD, which is slower than
-NFS-over-HDD. Options for the future:
-
-1. **Install internal drives** (tsys6 has a disabled SAS controller; tsys7 has
-   5 empty SATA ports). This is the real fix — gives both RackRental hosts proper
-   local scratch. Deferred to a physical-hardware session.
-2. **Accept NFS for image pulls** and use local-lvm only for swap/temp that
-   tolerates 30 MB/s. Functional but sluggish for lab spin-up.
-
-### 6.4 How this maps to k8s StorageClasses (future, post-Friday)
-
-The scratch tier maps to two k8s concepts:
-
-```
-StorageClass: local-fast
-  provisioner: local-path
-  where: tsys3 (NVMe), tsys9 (SSD), tsys2 (NVMe+SSD), tsys5 (HDD)
-  use: emptyDir replacement, build artifacts, container runtime cache
-  redundancy: none — ephemeral, dies with the pod
-
-StorageClass: nfs-hdd  (existing concept)
-  provisioner: nfs-subdir-external
-  where: D2/D5 (tsys4), S1/S2/S3/S4 (tsys5)
-  use: persistent bulk data, VM OS drives, anything that must survive restarts
-  redundancy: cross-server spreading + PBS backup
-```
-
-A wnode with local scratch gets a second disk (or a partition on local-lvm)
-mounted as `/scratch`. ETL jobs write intermediate data to `/scratch`
-(local-fast) and final output to `/data` (nfs-hdd). Pods that need ephemeral
-fast storage request `local-fast` PVCs; pods that need persistence request
-`nfs-hdd`.
-
-**This is forward-looking.** The concrete implementation (formatting local-lvm,
-adding to Proxmox as VM disk targets, defining StorageClasses, labeling nodes)
-happens after the Friday maintenance window when the SSD/NVMe tiers are
-physically installed and the storage architecture is finalized.

proxmox/docs/DRIFT-2026-07-30.md

@@ -1,175 +1,10 @@
-# Fleet Drift Report
+# proxmox/docs/DRIFT-2026-07-30.md
 
-> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts)
+> **Documentation moved to Discourse — the canonical source of truth.**
-> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel)
+>
-> **Report generated:** 2026-07-30
+> **Fleet drift report 2026-07-30**
-
+>
----
+> **Read it here:** https://community.turnsys.com/t/298
-
+>
-## Executive Briefing
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-### What needs your decision now
-
-| # | Issue | Impact | Recommendation |
-|---|-------|--------|---------------|
-| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` |
-| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing |
-| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` |
-| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` |
-| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` |
-| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` |
-| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 |
-| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) |
-| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog |
-| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS |
-| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide |
-
-### Consistent across the fleet (good)
-
-These items are the SAME on all 7 hosts — no drift:
-
-- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task)
-- **Debian version:** all on trixie/13
-- **tcp_congestion_control:** bbr everywhere
-- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected)
-- **ethtool:** same version everywhere
-- **nfs-common:** same version everywhere
-- **tcpdump:** same version everywhere
-- **smartmontools:** 7.5 everywhere, smartd active everywhere
-- **tuned:** installed everywhere
-- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM)
-- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module
-
-### Quick severity summary
-
-- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5
-- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`)
-- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment
-- **Investigate:** tsys9 SSH key count (2 keys missing)
-
----
-
-## Appendix A: Package Presence Matrix
-
-> MISSING = not installed. Version shown = installed version (truncated).
-
-| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
-|---------|-------|-------|-------|-------|-------|-------|-------|
-| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 |
-| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 |
-| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 |
-| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — |
-| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** |
-| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 |
-| rsyslog | installed | installed | installed | installed | installed | installed | installed |
-| qemu-guest-agent | — | — | — | — | — | — | — |
-| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** |
-| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** |
-| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 |
-| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 |
-| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 |
-| conman | — | — | 0.3.1 | — | — | — | — |
-| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — |
-| nut-server | 2.8.1 | — | — | — | — | — | — |
-| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 |
-
-> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are
-> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config.
-> conman/ser2net on tsys4 is expected (console server). These are not drift.
-
-## Appendix B: Service State Matrix
-
-| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
-|---------|-------|-------|-------|-------|-------|-------|-------|
-| lldpd | **inactive** | active | active | active | active | active | active |
-| smartd | active | active | active | active | active | active | active |
-| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
-| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
-| beszel | — | — | — | — | — | — | — |
-
-> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM
-> Day items, not drift — they haven't been deployed yet.
-
-## Appendix C: Sysctl Tuning Matrix
-
-| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
-|---------|-------|-------|-------|-------|-------|-------|-------|
-| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
-| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
-| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr |
-| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 |
-| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 |
-| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 |
-| governor | perf | perf | perf | N/A | perf | perf | perf |
-
-> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB),
-> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000).
-> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection
-> window size, and the low backlog values can cause packet drops under load.
-> swappiness=1 on tsys4/5 is intentional (storage hosts).
-
-## Appendix D: Tuned Profile Matrix
-
-| Host | Profile | Expected | Match? |
-|------|---------|----------|--------|
-| tsys1 | virtual-host | virtual-host | ✓ |
-| tsys3 | virtual-host | virtual-host | ✓ |
-| **tsys4** | **throughput-performance** | **network-throughput** | **✗** |
-| tsys5 | network-throughput | network-throughput | ✓ |
-| tsys6 | virtual-host | virtual-host | ✓ |
-| tsys7 | virtual-host | virtual-host | ✓ |
-| tsys9 | virtual-host | virtual-host | ✓ |
-
-## Appendix E: SSH + Security State
-
-### Authorized keys (root)
-
-| Host | Key count | Notes |
-|------|-----------|-------|
-| tsys1 | 5 | Standard |
-| tsys3 | 5 | Standard |
-| tsys4 | 5 | Standard |
-| tsys5 | 5 | Standard |
-| tsys6 | 5 | Standard |
-| tsys7 | 5 | Standard |
-| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** |
-
-### SSHD config (consistent across fleet)
-
-| Setting | Value | All hosts |
-|---------|-------|-----------|
-| PermitRootLogin | yes | ✓ (all) |
-| PasswordAuthentication | (default — not set) | ✓ (all) |
-| KbdInteractiveAuthentication | no | ✓ (all) |
-
-> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's
-> KbdInteractiveAuthentication. The PAM approach is consistent.
-
-## Appendix F: Filesystem Mount Options
-
-| Host | root fs mount option | Notes |
-|------|---------------------|-------|
-| tsys1 | relatime | Default |
-| tsys3 | relatime | Default |
-| tsys4 | relatime | Default |
-| **tsys5** | **noatime** | **Only host with noatime on root** |
-| tsys6 | relatime | Default |
-| tsys7 | relatime | Default |
-| tsys9 | relatime | Default |
-
-> Minor: noatime reduces metadata writes (slight improvement on HDD).
-> Storage hosts (tsys4/5) would benefit most from noatime.
-
-## Appendix G: Raw Data Location
-
-All raw drift probe output is stored in `/tmp/<hostname>-drift.txt` on this
-workstation (not committed — ephemeral). Re-gather any time with:
-
-```bash
-for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do
-  PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt
-done
-```
-
-The probe script should be saved to the repo as `perf/scripts/probe-drift.sh`
-for reuse. It is read-only and portable.

proxmox/docs/HOST-REFERENCE.md

@@ -1,173 +1,10 @@
-# Host Reference
+# proxmox/docs/HOST-REFERENCE.md
 
-> **One-page-per-host quick reference for every Proxmox node.** Hardware,
+> **Documentation moved to Discourse — the canonical source of truth.**
-> network, storage, role, and running VMs. Data gathered 2026-07-30.
 >
-> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
+> **Per-host hardware/network/storage/tuning/role reference**
-> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
+>
-> [`VM-INVENTORY.md`](VM-INVENTORY.md).
+> **Read it here:** https://community.turnsys.com/t/304
-
+>
----
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## Fleet summary
-
-```mermaid
-graph TB
-    subgraph "RACK 5 -- Core / Data / Infra"
-        T1["tsys1 -- Infra<br>i7-4770 | 32GB | 9 VMs<br>57% RAM"]
-        T3["tsys3 -- Compute<br>Xeon E3-1535M v5 | 32GB | 1 VM<br>10% RAM"]
-        T4["tsys4 -- Storage+PBS<br>Xeon E3-1246 v3 | 16GB | 1 VM<br>15% RAM"]
-        T5["tsys5 -- Storage+Sandbox<br>Xeon E5620 | 96GB | 4 VMs<br>15% RAM"]
-        T9["tsys9 -- Infra+Compute<br>i5-10500 | 24GB | 6 VMs<br>69% RAM"]
-    end
-
-    subgraph "RACK 3 -- Compute / RackRental"
-        T6["tsys6 -- RackRental<br>2x Xeon E5530 | 126GB | 2 VMs<br>12% RAM"]
-        T7["tsys7 -- RackRental<br>2x Xeon E5-2630 v2 | 189GB | 5 VMs<br>7% RAM"]
-    end
-```
-
-| Host | CPU (era) | Threads | RAM | PVE | Kernel | Role | Running VMs | RAM actual% |
-|------|-----------|---------|-----|-----|--------|------|-------------|-------------|
-| tsys1 | i7-4770 Haswell (2013) | 8 | 32 GB | 9.2.5 | 7.0.14 | Infra | 9 | 57% |
-| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% |
-| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% |
-| tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% |
-| tsys6 | 2x Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% |
-| tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% |
-| tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% |
-
-**Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired)
-
----
-
-## pfv-tsys1 -- Infrastructure host
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | Infrastructure: Home Assistant, CA, netinfra-01, LibreNMS, PDM, k8s cnode3, UCS-01, kali, workbench |
-| **Hardware** | Dell, i7-4770 Haswell (2013), 8 threads, 32 GB RAM (maxed) |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.3.11 (vmbr0, DHCP) |
-| **Tailscale** | 100.121.189.98 |
-| **Storage IP** | 10.100.100.1 (datanet via nic1, USB dongle, no bond) |
-| **NFS mounts** | S1, S2, S3, S4, D2 (all nconnect=4) |
-| **Local storage** | local-lvm (~90 GB, HDD, marginal) |
-| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
-| **Bond** | None (single USB NIC for storage) |
-| **Special services** | NUT (UPS), powerman (PDU) |
-| **Running VMs** | 100, 101, 103, 104, 105, 106, 108, 114, 117 |
-
----
-
-## pfv-tsys3 -- Compute host
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | Compute: k8s wnode-tsys3 (28 GB RAM, 8 vCPU) |
-| **Hardware** | Xeon E3-1535M v5 Skylake (2015), 8 threads, 32 GB RAM |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) |
-| **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) |
-| **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) |
-| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) -- **best existing local storage** |
-| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
-| **Bond** | None (single USB NIC for storage) |
-| **Running VMs** | 313 |
-
----
-
-## pfv-tsys4 -- Storage server + PBS
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | NFS storage server (D1/D2/D3/D5) + Proxmox Backup Server + console management (conman/ser2net) |
-| **Hardware** | Xeon E3-1246 v3 Haswell (2013), 8 threads, 16 GB RAM |
-| **PVE** | **9.1.5** / kernel **6.17.9-1-pve** (BEHIND fleet) |
-| **mgmt IP** | 192.168.3.251 (vmbr0, DHCP) |
-| **Tailscale** | 100.70.77.93 |
-| **Storage IP** | 10.100.100.4 (datanet via enx8cae4ccda926 USB dongle, bond0 with 1 slave) |
-| **NFS mounts** | None (NFS server only) |
-| **Local storage** | local-lvm (WDC WD10EZRX 1TB, ~62 GB free) |
-| **NFS exports** | D1 (WD 1TB USB), D2 (WDC Red 3TB), D3 (Seagate 500GB USB), D5 (Hitachi 2TB) |
-| **Other disks** | /mnt/albert (Hitachi 2TB, not exported), /mnt/backup (WD 5TB USB) |
-| **Tuning** | governor=performance, swappiness=1, bbr, **rmem/wmem=16MB** (should be 128MB), **tuned=throughput-performance** (should be network-throughput) |
-| **Bond** | bond0 (1 USB slave), hash=layer2+3 |
-| **Special services** | ser2net + conman (7 switch consoles on :7890) |
-| **Running VMs** | 400 (PBS) |
-| **Notes** | USB NIC bottleneck. Needs PVE upgrade + tuning alignment. |
-
----
-
-## pfv-tsys5 -- Storage server + sandboxes
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | NFS storage server (S1/S2) + sectestbed sandboxes + ultix streaming |
-| **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) |
-| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** -- nic2 DOWN) |
-| **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) |
-| **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) |
-| **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) |
-| **Unconfigured disks** | **NVMe CT500P1 500GB** (not mounted), **SK hynix SC300B 512GB SSD** (not mounted) |
-| **Tuning** | governor=N/A (no cpufreq driver on Westmere), swappiness=1, bbr, rmem/wmem=128MB, tuned=network-throughput |
-| **Bond** | bond0 (1 of 2 slaves), hash=layer2+3 |
-| **Running VMs** | 509, 515, 5111, 5112 (+ 22 stopped sandboxes) |
-| **Notes** | Bond degraded (needs 2nd cable). NVMe + SSD need mounting/config. SSD 93% full. Single CPU. |
-
----
-
-## pfv-tsys6 -- RackRental host
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX |
-| **Hardware** | 2x Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) |
-| **Tailscale** | 100.73.35.111 |
-| **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) |
-| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
-| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) |
-| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
-| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
-| **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) |
-| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 -- unusable for scratch. |
-
----
-
-## pfv-tsys7 -- RackRental host
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 |
-| **Hardware** | 2x Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) |
-| **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) |
-| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
-| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) |
-| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
-| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
-| **Running VMs** | 701, 702, 703, 704, 705 |
-| **Notes** | Best storage bond in fleet (2G LACP). Most RAM in fleet (189 GB). Local storage USB 2.0. |
-
----
-
-## pfv-tsys9 -- Infra + Compute
-
-| Attribute | Value |
-|-----------|-------|
-| **Role** | Infra + Compute: k8s cnode1 + wnode-tsys9 + siem + UCS-02 + netinfra-02 + kali |
-| **Hardware** | i5-10500 Comet Lake (2020), 12 threads, 24 GB RAM |
-| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
-| **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) |
-| **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) |
-| **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) |
-| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) -- **excellent for scratch** |
-| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
-| **Bond** | None (single USB NIC for storage) |
-| **Running VMs** | 901, 902, 903, 904, 905, 906 |
-| **Notes** | Newest CPU (best IPC in fleet). **69% RAM -- approaching 80% goal.** Target for compute growth (after siem migrates off). |

proxmox/docs/K8S.md

@@ -1,485 +1,10 @@
-# K8S.md -- Kubernetes Architecture Deep-Dive
+# proxmox/docs/K8S.md
-
+
-**Date:** 2026-07-27
+> **Documentation moved to Discourse — the canonical source of truth.**
-**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
+>
-Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment).
+> **Legacy k8s storage/host analysis**
-**Status:** For discussion in a future session. No changes made.
+>
-
+> **Read it here:** https://community.turnsys.com/t/305
----
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-## Table of Contents
+> Do not update content here — edit the Discourse wiki topic instead.*
-
-1. [Workload Profile](#1-workload-profile)
-2. [Current State](#2-current-state)
-3. [Target Architecture](#3-target-architecture)
-4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
-5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
-6. [Storage Class Design](#6-storage-class-design)
-7. [ETL/HPC Considerations](#7-etlhpc-considerations)
-8. [Migration Plan](#8-migration-plan)
-
----
-
-## 1. Workload Profile
-
-This cluster runs **R&D and RackRental (containerlab) workloads** via
-Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
-Reston, VA running Cloudron.
-
-**Workload types expected:**
-
-| Type | Description | Storage need | RAM need | Examples |
-|------|------------|-------------|----------|---------|
-| **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
-| **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
-| **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
-
-**Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
-sequential I/O). HPC and containerlab need low-latency random I/O (local
-SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
-
----
-
-## 2. Current State
-
-### 2.1 pfv-k8s nodes and their storage
-
-| Node | Type | Host | Storage | Disk type | Status |
-|------|------|------|---------|-----------|--------|
-| cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
-| cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
-| cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
-| cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
-| cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
-| wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
-| wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
-| wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
-| wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
-| wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
-
-### 2.2 Problems
-
-1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
-   etcd quorum lost.
-2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
-3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
-4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
-   (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
-5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
-6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
-
----
-
-## 3. Target Architecture
-
-### 3.1 Design principles
-
-1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
-   heavy workers.
-2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
-   storage server failing.
-3. **Wnode boot disks on local storage** -- eliminates NFS latency for
-   container runtime and kubelet.
-4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
-5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
-6. **One wnode per hypervisor host** -- maximize total cluster capacity.
-
-### 3.2 Target node-host-storage matrix
-
-| Node | Type | Host | Boot disk | Data disk | Disk type |
-|------|------|------|-----------|-----------|-----------|
-| cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
-| cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
-| cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
-| cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
-| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
-| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
-| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
-| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
-| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
-| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
-| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
-| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
-
-### 3.3 Storage server distribution after changes
-
-| Storage server | cnodes | wnodes (boot) | wnodes (data) |
-|---------------|--------|---------------|---------------|
-| tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
-| tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
-| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
-| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
-
-**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
-NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
-than NFS-HDD, and the user has chosen not to install internal drives.
-
-**No single disk or server is a quorum-losing failure point.**
-
----
-
-## 4. Control Plane (Cnodes)
-
-### 4.1 Cnode sizing
-
-Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
-kubernetes control plane components (API server, scheduler, controller-manager).
-
-### 4.2 Cnode host placement rationale
-
-| Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
-|------|--------|---------------|---------------|------------------------|
-| tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
-| tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
-| tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
-
-**tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
-
-### 4.3 Cnode storage placement rationale
-
-The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
-
-| Disk | cnodes | Rationale |
-|------|--------|-----------|
-| D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
-| D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
-| S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
-| S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
-
-**If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
-**If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
-**If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
-tsys5. **Only 2 of 5 -- QUORUM LOST.**
-
-Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
-cnodes and only have 2 on tsys5. That loses quorum (need 3).
-
-**Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
-3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
-tsys4. Same problem inverted.
-
-The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
-3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
-holding 2 cnodes is safe (3 survive).
-
-**Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
-other.** After Friday's hardware work:
-- tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
-- tsys5 will have bond0 fixed + NVMe -- more reliable
-
-Either way, 3/2 split means one server failure could lose quorum. **To truly
-solve this, use a 3rd storage target.** Options:
-- Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
-  independent failure domain)
-- Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
-  no NFS dependency at all)
-
-**Best option: put cnode boot disks on local-lvm where available.** This
-eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
-its own host's local disk -- no shared dependency.
-
-| cnode | Host | **Recommended storage** | Type |
-|-------|------|------------------------|------|
-| cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
-| cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
-| cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
-| cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
-| cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
-
-With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
-down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
-survives.
-
-**This is the recommended approach.** Local storage for cnodes wherever
-possible. NFS only as fallback.
-
-### 4.4 etcd performance on local vs NFS
-
-| Storage | Typical fsync latency | etcd commit latency | Impact |
-|---------|----------------------|--------------------|--------|
-| NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
-| NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
-| Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
-| Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
-| NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
-
-etcd is the heartbeat of the kubernetes control plane. Every API call, every
-pod schedule, every controller reconciliation involves an etcd write. Cutting
-etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
-responsive. **This is the single highest-impact change for k8s performance.**
-
----
-
-## 5. Worker Nodes (Wnodes)
-
-### 5.1 One wnode per hypervisor host
-
-| Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
-|------|-------|-----------|-----------|-----------|-----------|------|
-| tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
-| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
-| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
-| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
-| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
-| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
-| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
-
-### 5.2 Why boot disks on local-lvm
-
-Current: all wnodes boot from NFS. Every container image pull, every kubelet
-log write, every ephemeral volume traverses the NFS network path.
-
-With local-lvm boot disks:
-- **Container image pulls** write to local disk (100-150 MB/s HDD, no network
-  hop) instead of NFS-HDD (80-120 MB/s with network latency)
-- **kubelet logs** stay local (no NFS writes for log rotation)
-- **ephemeral storage** (emptyDir volumes) uses local disk by default
-- **NFS server failure does not kill the wnode** -- the VM stays running, only
-  the data disk (if mounted) goes away
-
-### 5.3 Wnode sizing guidance
-
-| Host | Recommended wnode config | Rationale |
-|------|------------------------|-----------|
-| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
-| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
-| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
-| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
-| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
-| tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
-| tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
-
-### 5.4 Tainting and labeling strategy
-
-Label wnodes by capability so the k8s scheduler can target them:
-
-```yaml
-# Heavy RAM hosts (ETL/HPC)
-wnode-tsys6: workload=heavy, ram=128g
-wnode-tsys7: workload=heavy, ram=192g
-
-# NVMe host (I/O-intensive HPC)
-wnode-tsys5: workload=hpc, storage=nvme
-
-# SSD boot host (low-latency)
-wnode-tsys9: workload=light, storage=ssd
-
-# General workers
-wnode-tsys3: workload=general
-wnode-tsys2: workload=storage-fast, storage=nvme
-wnode-tsys1: workload=light
-```
-
-Then use nodeSelector or nodeAffinity in job specs:
-
-```yaml
-# Weather/GIS ETL job -- needs lots of RAM
-spec:
-  nodeSelector:
-    workload: heavy
-
-# Firmware build -- needs fast storage
-spec:
-  nodeSelector:
-    storage: nvme
-```
-
----
-
-## 6. Storage Class Design
-
-### 6.1 Proposed StorageClasses
-
-| StorageClass | Provisioner | Where | Speed | Use case |
-|-------------|------------|-------|-------|----------|
-| `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
-| `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
-| `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
-
-### 6.2 How this maps to wnode disk topology
-
-Each wnode has:
-- **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
-  runtime. k8s `local-fast` StorageClass provisioner points here.
-- **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
-  second block device or filesystem. k8s `nfs-hdd` provisioner points here.
-
-Inside k8s, pods request storage via PVC:
-
-```yaml
-# ETL job: needs bulk storage for weather data
-apiVersion: v1
-kind: PersistentVolumeClaim
-spec:
-  storageClassName: nfs-hdd
-  accessModes: [ReadWriteMany]  # NFS allows RWX
-  resources:
-    requests:
-      storage: 500Gi
-
-# HPC job: needs fast scratch
-spec:
-  storageClassName: local-fast
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests:
-      storage: 50Gi
-```
-
-### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
-
-**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
-scratch space, with the exception of ultix-streaming which stays on T5-SSD.
-Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
-SIEM, etc.).
-
-The SSD NFS exports:
-- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
-  cache, ephemeral volumes). Currently 99% empty.
-- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
-  occupies 83 GB. Remaining 140 GB available for k8s use.
-
-**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
-This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
-tsys5 SSD exports, bulk PVCs target either server.
-
-### 6.4 NFS data distribution across storage servers
-
-To avoid re-creating the "everything on tsys4" problem, distribute NFS data
-disks across both servers:
-
-| wnode | Boot (local) | Bulk data (NFS) | NFS server |
-|-------|-------------|-----------------|------------|
-| wnode-tsys3 | local-lvm | S3 | tsys5 |
-| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
-| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
-| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
-| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
-
-This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
-
----
-
-## 7. ETL/HPC Considerations
-
-### 7.1 Weather/GIS ETL pipeline
-
-Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
-store results.
-
-| Stage | Storage class | Why |
-|-------|-------------|-----|
-| Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
-| Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
-| Store results | `nfs-hdd` | Large sequential writes. Persistent. |
-
-**Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
-wnode. ETL jobs use `/scratch` for intermediate processing and write final
-output to the NFS-mounted `/data`.
-
-### 7.2 HPC workloads (hardware startup)
-
-Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
-
-| Workload | Best wnode | Why |
-|----------|-----------|-----|
-| RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
-| Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
-| Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
-| Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
-
-### 7.3 RackRental/containerlab
-
-Rapid container deployment. Key need: fast container image pulls.
-
-This is where **local-lvm boot disks** shine. Currently, every container image
-pull writes through NFS to a spinning disk -- slow. With local-lvm, images
-cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
-On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
-
-### 7.4 Data locality for ETL
-
-For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
-it on local-lvm of the heavy hosts:
-
-```
-tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/  -- fastest cache tier
-tsys5 NVMe (local): /data/cache/gis/                 -- fastest cache tier
-```
-
-**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
-be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
-
-This avoids re-reading the same data from the same NFS export on every job
-if the data is already cached in the page cache.
-
----
-
-## 8. Migration Plan
-
-**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
-Datacenter Manager (PDM)** manages them collectively and supports VM migration
-between nodes. Storage migration can be done via the PDM/Proxmox UI rather
-than manual disk copies -- the destination node just needs access to the target
-storage (which all nodes have for NFS exports, and local storage can be
-migrated through the UI's "Storage Migrate" function).
-
-### 8.1 Phase 1: Friday (after hardware work)
-
-After tsys5 cable + NVMe and tsys4 NIC + RAM:
-
-1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
-2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
-   do not use for VM storage. Recreate on D2 or D5 NFS.
-3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
-
-### 8.2 Phase 2: Cnode rebalance (maintenance window)
-
-These changes require creating new VMs on target hosts and migrating disks.
-Plan for a maintenance window with the k8s cluster briefly down.
-
-1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
-2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
-3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
-4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
-5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
-
-### 8.3 Phase 3: Wnode local storage migration (maintenance window)
-
-1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
-2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
-3. wnode-tsys7 stays on NFS (same reason).
-4. Recreate wnode-tsys5 with boot disk on NVMe.
-5. Add data disks (NFS) as second SCSI devices where applicable.
-
-### 8.4 Phase 4: tsys2 integration (when rebuilt)
-
-1. Install Proxmox on tsys2.
-2. Run `scripts/check.sh` to inventory.
-3. Run `scripts/apply-tunings.sh --apply`.
-4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
-5. Join to k8s cluster.
-
-### 8.5 Phase 5: Critical VM relocation
-
-1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
-2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
-3. (No change to T5-SSD -- ultix-streaming stays.)
-
----
-
-## Open questions for next session
-
-1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
-   whether live migration is available (huge simplification) or we need manual
-   disk migration. Check `pvecm status` on each host.
-2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
-   nodes are joined/drain and how StorageClasses are configured.
-3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
-4. **Is there a container image registry mirror in the cluster?** Or do all
-   pulls go to Docker Hub / external? A local registry on D3 SSD would speed
-   up all pulls.
-5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
-   whether jobs need shared (RWX) or exclusive (RWO) storage.
-6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
-   how we label and taint nodes.

proxmox/docs/NAMING-POLICY.md

@@ -1,25 +1,10 @@
-# Naming and DNS Policy
+# proxmox/docs/NAMING-POLICY.md
 
-> **Canonical naming policy for the PFVCluster fleet.**
+> **Documentation moved to Discourse — the canonical source of truth.**
-> Established: 2026-07-31
+>
-
+> **Canonical naming policy for VMs/DNS/Tailscale**
-## Rule
+>
-
+> **Read it here:** https://community.turnsys.com/t/304
-**The Proxmox VM name is the canonical identifier.** Everything else
+>
-derives from it:
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-1. **VM name** (set via `qm set <vmid> --name <name>`) is the source of truth
-2. **DNS records** (Technitium, forward + reverse) must match the VM name exactly
-3. **Tailscale node name** must match the VM name
-4. **Monitoring/alerting** (Beszel, Uptime Kuma, LibreNMS) uses the VM name
-5. **Inventory/k8s** references use the VM name
-
-Any mismatch between VM name and DNS/Tailscale name is a **critical bug**
-that must be fixed. The Technitium DNS API key is stored at
-`~/.secrets/api-keys.env` for programmatic DNS management.
-
-## Tools
-
-- Technitium DNS API key: `~/.secrets/api-keys.env`
-- Uptime Kuma API key: `~/.secrets/api-keys.env`
-- DNS audit: `perf/scripts/probe-vm-dns.sh`

proxmox/docs/NETWORK-TOPOLOGY.md

@@ -1,365 +1,10 @@
-# Network Topology
+# proxmox/docs/NETWORK-TOPOLOGY.md
 
-> **Complete physical + logical network reference for the PFVCluster.**
+> **Documentation moved to Discourse — the canonical source of truth.**
-> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct
-> host probes. All data is live ground truth -- not documentation inherited.
 >
-> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
+> **Network reference: topology diagrams, switch ports, VLAN map, NFS data-flow**
-> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
+>
-> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
+> **Read it here:** https://community.turnsys.com/t/299
-
+>
----
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## 1. Physical topology diagram
-
-```mermaid
-graph TB
-    subgraph "RACK 5 - Core / Data / Infra"
-        CORE["pfv-core-sw01<br>PowerConnect 5448<br>192.168.0.12<br>(mgmt + storage VLANs)"]
-
-        TSYS1["pfv-tsys1<br>i7-4770 Haswell<br>Infra host"]
-        TSYS3["pfv-tsys3<br>Xeon E3-1535M v5<br>Compute"]
-        TSYS4["pfv-tsys4<br>Xeon E3-1246 v3<br>NFS Storage + PBS"]
-        TSYS5["pfv-tsys5<br>Xeon E5620 Westmere<br>NFS Storage + Sandboxes"]
-        TSYS9["pfv-tsys9<br>i5-10500 Comet Lake<br>Infra + Compute"]
-
-        TSYS4_CONSOLE["conman + ser2net<br>7 switch consoles<br>Tailscale 100.70.77.93:7890"]
-        TSYS1_PDU["Cyclades PM10i PDU<br>powerman on tsys1"]
-        TSYS1_UPS["APC Smart-UPS C 1500<br>NUT on tsys1"]
-    end
-
-    subgraph "RACK 3 - Compute / RackRental"
-        TOR3STOR["pfv-tor3-stor<br>PowerConnect 5324<br>192.168.0.9<br>storage VLAN"]
-        TOR3MGMT["pfv-tor3-mgmt<br>PowerConnect 5324<br>192.168.0.7<br>mgmt VLAN"]
-
-        TSYS6["pfv-tsys6<br>2x Xeon E5530 Nehalem<br>RackRental"]
-        TSYS7["pfv-tsys7<br>2x Xeon E5-2630 v2<br>RackRental"]
-    end
-
-    %% Cross-rack trunk (two one-way dotted arrows with labels)
-    CORE -. "ch1: 3x1G active (g13-15)<br>hash=layer-2-3<br>mode=on" .-> TOR3STOR
-    TOR3STOR -. "ch1: 4x1G (g20-23)<br>hash=layer-2-3-4<br>mode=on" .-> CORE
-
-    %% Rack 5 connections to core
-    CORE --> TSYS1
-    CORE --> TSYS3
-    CORE --> TSYS4
-    CORE --> TSYS5
-    CORE --> TSYS9
-
-    %% Rack 3 connections
-    TOR3STOR --> TSYS6
-    TOR3STOR --> TSYS7
-    TOR3MGMT --> TSYS6
-    TOR3MGMT --> TSYS7
-```
-
-### How to read this diagram
-
-The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** --
-the most critical link in the cluster. It carries all NFS storage traffic
-between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5).
-The trunk is a 4-port static LAG, but only 3 links are active due to a cable
-mismatch (see §3 below).
-
----
-
-## 2. Subnet / VLAN map
-
-| VLAN | Subnet | Purpose | Where it lives |
-|------|--------|---------|----------------|
-| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) |
-| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) |
-| -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs |
-
-**Management IPs (VLAN 1, via DHCP reservations):**
-
-| Host | mgmt IP (vmbr0) | Tailscale IP |
-|------|-----------------|--------------|
-| tsys1 | 192.168.3.11 | 100.121.189.98 |
-| tsys3 | 192.168.2.5 | (connected) |
-| tsys4 | 192.168.3.251 | 100.70.77.93 |
-| tsys5 | 192.168.3.179 | (connected) |
-| tsys6 | 192.168.3.169 | 100.73.35.111 |
-| tsys7 | 192.168.3.55 | (connected) |
-| tsys9 | 192.168.3.58 | (connected) |
-
-**Storage IPs (VLAN 1000, static):**
-
-| Host | Storage IP | Interface |
-|------|-----------|-----------|
-| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) |
-| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) |
-| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) |
-| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) |
-| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) |
-| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) |
-| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) |
-
-**Switch IPs (VLAN 1, static DHCP reservations):**
-
-| Switch | IP | Model | Console |
-|--------|----|-------|---------|
-| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 |
-| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 |
-| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 |
-| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? |
-
-**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c)
-
----
-
-## 3. Cross-rack trunk detail (ch1) -- the critical link
-
-This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and
-rack 5 (tsys4/5). It is the single most important network path in the cluster.
-
-```mermaid
-graph LR
-    subgraph "core-sw01 (rack5)"
-        G13["g13<br>1G UP"]
-        G14["g14<br>1G UP"]
-        G15["g15<br>1G UP"]
-        G16["g16<br>1G UP<br>NOT in LAG!"]
-        G17["g17<br>DOWN<br>in config but<br>no cable"]
-    end
-
-    subgraph "tor3-stor (rack3)"
-        G20["g20<br>1G UP"]
-        G21["g21<br>1G UP"]
-        G22["g22<br>1G UP"]
-        G23["g23<br>1G UP"]
-    end
-
-    G13 --- G20
-    G14 --- G21
-    G15 --- G22
-    G16 -.- G23
-    G17 -.->|no cable| G23
-```
-
-### Configuration comparison
-
-| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? |
-|----------|-------------------|--------------------|--------|
-| Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** |
-| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- |
-| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** |
-| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** |
-| Channel mode | on (static) | on (static) | Yes (both should be LACP) |
-| VLAN | access vlan 1000 | access vlan 1000 | Yes |
-
-### Problems
-
-1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on
-   core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex)
-   but is NOT in the LAG group. g17 is configured in the LAG but has no cable
-   (DOWN). Result: **only 3 of 4 links are active**.
-
-2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP).
-   tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With
-   layer-2-3 on core-sw01, **all traffic between a given host pair hashes to
-   one link** regardless of how many TCP connections (nconnect) are used.
-
-3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure
-   detection -- a dead cable stays in the LAG until manually discovered.
-
----
-
-## 4. Complete switch port map
-
-### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5
-
-| Port | Speed | Status | VLAN | Description / Connected device |
-|------|-------|--------|------|-------------------------------|
-| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) |
-| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) |
-| g5 | -- | UP | 1 | pfv-tsys1 (mgmt) |
-| g6 | 100 | UP | 1 | pfvsvrpi (secondary) |
-| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt |
-| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** |
-| g10 | 1000 | UP | 1 | (unknown) |
-| g11 | 1000 | UP | 1 | (unknown) |
-| g12 | 1000 | UP | 1 | (unknown) |
-| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** |
-| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** |
-| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** |
-| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** |
-| **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** |
-| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt |
-| g21 | 1000 | UP | 1 | (unknown) |
-| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) |
-| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** |
-| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) |
-| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) |
-| g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) |
-| g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) |
-| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) |
-| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** |
-| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) |
-| g41 | 1000 | UP | 1 | (unknown) |
-| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) |
-| g44 | 1000 | UP | 1 | (unknown) |
-| g45-g48 | -- | DOWN | -- | (Combo ports, unused) |
-
-**Port-channels on core-sw01:**
-
-| Channel | Speed | Status | Members | VLAN | Notes |
-|---------|-------|--------|---------|------|-------|
-| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor |
-| ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused |
-| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) |
-
-**LAG hash:** `port-channel load-balance layer-2-3`
-
-### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3
-
-| Port | Speed | Status | VLAN | Description / Connected device |
-|------|-------|--------|------|-------------------------------|
-| g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) |
-| g2-g7 | 1000 | UP | 1 | (unused but UP) |
-| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** |
-| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** |
-| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** |
-| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** |
-| **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
-| **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
-| g15-g19 | 1000 | UP | 1000 | (unused storage ports) |
-| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** |
-| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** |
-| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** |
-| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** |
-| g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) |
-
-**Port-channels on tor3-stor:**
-
-| Channel | Speed | Status | Members | VLAN | Notes |
-|---------|-------|--------|---------|------|-------|
-| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) |
-| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) |
-| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) |
-| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** |
-
-**LAG hash:** `port-channel load-balance layer-2-3-4`
-
-### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3
-
-| Port | Speed | Status | Connected device |
-|------|-------|--------|-----------------|
-| g3 | -- | UP | tsys7 (mgmt nic0) |
-| g7 | -- | UP | tsys6 (mgmt nic0) |
-
-(Full port map not captured -- this switch carries only VLAN 1 management
-traffic. Low priority for storage performance.)
-
----
-
-## 5. Per-host network interface reference
-
-| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed |
-|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------|
-| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G |
-| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G |
-| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G |
-| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** |
-| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
-| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
-| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G |
-
-### Key observations
-
-- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2
-  active slaves). All NFS traffic from rack 3 uses these bonds.
-- **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy.
-- **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is
-  active via core-sw01 ch3 (g34). This is the LACP problem the operator has been
-  troubleshooting.
-- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS),
-  tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one
-  link even with nconnect's multiple TCP connections).
-
----
-
-## 6. NFS data flow diagram
-
-```mermaid
-graph TB
-    subgraph "Storage Servers (rack 5)"
-        T4D2["tsys4 D2<br>WDC Red 3TB<br>Tier 2"]
-        T4D5["tsys4 D5<br>Hitachi 2TB<br>Tier 2"]
-        T4D1["tsys4 D1<br>WD 1TB USB<br>Tier 3"]
-        T4D3["tsys4 D3<br>Seagate 500GB USB<br>Tier 3"]
-        T5S1["tsys5 S1<br>IronWolf 1TB<br>Tier 2"]
-        T5S2["tsys5 S2<br>IronWolf 1TB<br>Tier 2"]
-        T5SSD["tsys5 SSD<br>Samsung 860 PRO<br>Tier 1 (93% full)"]
-    end
-
-    subgraph "VLAN 1000 storage network"
-        NFS["NFS v4.2<br>nconnect=4<br>rsize/wsize=1MB"]
-    end
-
-    subgraph "NFS Clients"
-        T1["tsys1 (9 mounts)"]
-        T3["tsys3 (5 mounts)"]
-        T5["tsys5 (3 mounts, NO nconnect)"]
-        T6["tsys6 (9 mounts)"]
-        T7["tsys7 (9 mounts)"]
-        T9["tsys9 (5 mounts)"]
-    end
-
-    T4D2 --> NFS
-    T4D5 --> NFS
-    T4D1 --> NFS
-    T4D3 --> NFS
-    T5S1 --> NFS
-    T5S2 --> NFS
-    T5SSD --> NFS
-
-    NFS --> T1
-    NFS --> T3
-    NFS --> T5
-    NFS --> T6
-    NFS --> T7
-    NFS --> T9
-```
-
-### NFS mount inventory (live)
-
-| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
-|--------|--------|-------|-------|-------|-------|-------|-------|-------|
-| tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- |
-| tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
-| tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- |
-| tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
-| tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- |
-| tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- |
-| tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ |
-| tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ |
-| tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ |
-
-(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts.
-tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.)
-
----
-
-## 7. Switch management access
-
-| Method | Details |
-|--------|---------|
-| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. |
-| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` |
-| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) |
-| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) |
-| **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. |
-
-### Console port assignments
-
-| Port | Console name | Switch | ID_PATH (USB port) |
-|------|-------------|--------|---------------------|
-| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 |
-| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 |
-| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 |
-| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 |
-| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 |
-| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 |
-| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 |

proxmox/docs/PHYSICAL-SYSTEMS.md

@@ -1,70 +1,10 @@
-# Physical Systems (Pi's + other hardware)
+# proxmox/docs/PHYSICAL-SYSTEMS.md
 
-> **Access patterns and locations for physical infrastructure devices.**
+> **Documentation moved to Discourse — the canonical source of truth.**
-> Updated: 2026-07-31
+>
-
+> **Physical infrastructure: Pis, network devices, UniFi APs**
-## Raspberry Pi systems
+>
-
+> **Read it here:** https://community.turnsys.com/t/304
-All accessible via Tailscale. Use `localuser` (passwordless sudo to root)
+>
-except where noted.
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-| Name | Tailscale IP | LAN IP | Access | Purpose |
-|------|-------------|--------|--------|---------|
-| pfvsvrpi | 100.91.151.113 | 192.168.1.230 | `localuser` | Server Pi (general purpose) |
-| subopi-dev-3 | 100.64.231.65 | 192.168.1.232 | via subodev jump | Suborbital dev Pi 3 |
-| subopi-dev-4 | 100.65.224.85 | 192.168.1.231 | via subodev jump | Suborbital dev Pi 4 |
-
-> **Note from operator:** "use subodev to get into subopi-*" — subodev is
-> a jump host for the suborbital Pi's.
-
-## Network devices discovered (non-Proxmox)
-
-| IP | Device | Type | Identified via |
-|----|--------|------|---------------|
-| 192.168.0.12 | pfv-core-sw01 | Dell PowerConnect 5448 | SNMP |
-| 192.168.0.7 | pfv-tor3-mgmt | Neyland 24T (PC5324) | SNMP |
-| 192.168.0.8 | (unidentified) | Neyland 24T (PC5324) | SNMP |
-| 192.168.0.9 | pfv-tor3-stor | Neyland 24T (PC5324) | SNMP |
-| 192.168.1.166 | pfv-stor1 | Linux 4.14.22 (Synology?) | SNMP |
-| 192.168.1.182 | UAP-AC-LR | UniFi AP (FW 6.8.2.15592) | SNMP |
-| 192.168.1.230 | pfvsvrpi | Linux 6.12.62 (RPi) | SNMP |
-| 192.168.1.231 | subopi-dev-4 | Linux 6.12.25 (RPi) | SNMP |
-| 192.168.1.232 | subopi-dev-3 | Linux 6.12.9 (RPi) | SNMP |
-| 192.168.1.217 | subopi3 | Linux 6.12.25 (RPi) | SNMP |
-
-## Other identified hosts (SSH open, unidentified)
-
-| IP | Notes |
-|----|-------|
-| 192.168.0.72 | SSH open, no SNMP — investigate |
-| 192.168.0.250 | SSH open, no SNMP — investigate |
-| 192.168.1.138 | SSH open — investigate |
-| 192.168.1.181 | SSH open — investigate |
-| 192.168.1.192 | SSH open — investigate |
-| 192.168.1.201 | SSH open — investigate |
-| 192.168.1.205 | SSH open — investigate |
-| 192.168.1.214 | SSH open — investigate |
-| 192.168.1.223 | SSH open — investigate |
-| 192.168.1.226 | SSH open — investigate |
-| 192.168.1.228 | SSH open — investigate |
-| 192.168.2.5 | SSH open — investigate (tsys3 vmbr0) |
-| 192.168.3.1 | SSH open — investigate (gateway/router?) |
-| 192.168.3.78 | SSH open — investigate |
-
-## 192.168.3.12-24 range
-
-These 13 hosts respond to ping but have no SSH/SNMP. Likely:
-- IoT devices
-- Cameras
-- Phones (VoIP)
-- Other DHCP clients
-
-## UniFi Access Points
-
-| AP | Port (core-sw01) | Model | FW |
-|----|-----------------|-------|-----|
-| UAP-AC-LR | g9 (2902 inErrors!) | UAP-AC-LR | 6.8.2.15592 |
-| UAP-AC-Lite | g38 (590 inErrors!) | UAP-AC-Lite | (unknown) |
-
-Both managed by UniFi controller (likely on Home Assistant VM 100 or
-pfv-stor1). UniPoller exports metrics to Prometheus.

proxmox/docs/PROJECT.md

@@ -1,857 +1,10 @@
-# Proxmox Cluster Project Report
+# proxmox/docs/PROJECT.md
-
+
-**Date:** 2026-07-27 (re-audited)
+> **Documentation moved to Discourse — the canonical source of truth.**
-**Prepared by:** Performance Optimization Engagement
+>
-**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
+> **Comprehensive fleet report**
-
+>
-**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
+> **Read it here:** https://community.turnsys.com/t/304
-`deploy-check.sh`. VM placements reflect live state after user's PDM
+>
-migrations. This is ground truth.
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
----
-
-## Table of Contents
-
-1. [Executive Summary](#1-executive-summary)
-2. [Host Fleet](#2-host-fleet)
-3. [Storage Architecture](#3-storage-architecture)
-4. [VM Fleet Inventory](#4-vm-fleet-inventory)
-5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
-6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
-7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
-8. [Role Alignment Audit](#8-role-alignment-audit)
-9. [Network Findings](#9-network-findings)
-10. [Recommendations](#10-recommendations)
-11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
-12. [Open Items](#12-open-items)
-
----
-
-## 1. Executive Summary
-
-The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
-running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
-performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
-are blocked on physical hardware work scheduled for Friday.
-
-**Progress since initial audit:** The user has been actively rebalancing k8s
-nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
-73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
-needed for etcd quorum survival.
-
-The VM-layer assessment reveals:
-
-| # | Finding | Severity | Status |
-|---|---------|----------|--------|
-| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
-| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
-| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
-| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
-
----
-
-## 2. Host Fleet
-
-### 2.1 Inventory
-
-| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
-|------|-------|-----------|-------|-----|-----------|-----------------|--------|
-| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
-| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
-| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
-| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
-| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
-| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
-| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
-| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
-
-### 2.2 Role taxonomy (per user directive)
-
-| Role | Hosts | Workload |
-|------|-------|----------|
-| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
-| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
-| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
-
-**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
-RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
-(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
-run one per hypervisor host across the fleet to maximize total cluster capacity.
-
-pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
-Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
-running Cloudron -- not in this cluster.
-
----
-
-## 3. Storage Architecture
-
-### 3.1 NFS exports from tsys4 (primary storage server)
-
-| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
-|--------|-----------|------|-----|-------|------|------|------|
-| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
-| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
-| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
-
-Non-exported disks on tsys4:
-- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
-- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
-- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
-
-**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
-tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
-eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
-/sdc/sdd), so tsys5 is the better target. See section 3.2.
-
-### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
-
-| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
-|--------|-----------|------|-----|-------|------|------|------|
-| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
-| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
-| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
-| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
-| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
-| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
-
-**tsys5 storage controllers (plenty of free ports):**
-- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
-  **5 free**
-- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
-- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
-- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
-
-**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
-wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
-(unused, 7 GB). S4 still 99% empty (435 GB free).
-
-T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
-
-**Friday additions:**
-1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
-   USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
-2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
-   boot disk and HPC scratch. The fastest tier in the fleet.
-
-After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
-NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
-natural home for latency-sensitive workloads and the k8s StorageClass design
-center.
-
-### 3.3 Local storage tiers (per host, with utilization)
-
-| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
-|------|-----------|-----------|-----|-------|------|------|-------------|
-| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
-| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
-| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
-| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
-| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
-| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
-| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
-| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
-| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
-| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
-
-**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
-
-The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
-single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
-260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
-(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
-
-**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
-NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
-hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
-for VM storage.
-
-Additionally, both servers have completely empty internal drive bays:
-- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
-drives.
-- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
-DVD-ROM on port 5). No internal drives.
-
-This is a reliability risk beyond performance: the entire host OS boots
-from a consumer-grade portable USB drive not designed for 24/7 server use.
-
-**tsys3 correction:** Previously documented as HDD. Actually boots from a
-**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
-fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
-
-**Critical observation: every k8s host has 0% used local-lvm.** tsys3
-(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
-1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
-storage.
-
-**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
-tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
-NFS-over-HDD and unsuitable for wnode boot disks.
-
-### 3.4 Disk speed tiers summary
-
-| Tier | Where | Speed class | Best for |
-|------|-------|------------|----------|
-| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
-| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
-| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
-| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
-| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
-| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
-
-### 3.5 Storage tier characterization per host
-
-| Host | Storage profile | Detail |
-|------|----------------|--------|
-| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
-| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
-| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
-| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
-| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
-| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
-| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
-
-### 3.6 All exports are single-disk with no redundancy
-
-Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
-mirror, no mdraid. A single disk failure takes down every VM whose disk lives
-on that export. This applies to **both storage servers** and to the **PBS
-backup target** (a single 4.5T SMR drive).
-
----
-
-## 4. VM Fleet Inventory
-
-### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
-
-#### tsys1 (Infrastructure) -- 11 running VMs
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
-| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
-| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
-| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
-| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
-
-#### tsys3 (Kubernetes) -- 1 running VM
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
-
-#### tsys4 (Storage) -- 1 running VM
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
-
-#### tsys5 (Storage) -- 15 running VMs
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
-| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
-| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
-| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
-| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
-| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
-| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
-| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
-
-#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
-| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
-| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
-| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
-
-#### tsys7 (Kubernetes) -- 4 running VMs
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
-| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
-| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
-
-#### tsys9 (Infrastructure) -- 5 running VMs
-
-| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
-|------|------|-------|----------|------|---------|------|
-| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
-| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
-| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
-| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
-| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
-
-### 4.2 Storage concentration summary
-
-| Storage target | # of running VMs | % of fleet |
-|---------------|-----------------|------------|
-| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
-| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
-| local-nonprod (tsys5 local HDD) | 10 | 25% |
-| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
-| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
-| local-lvm (tsys4 local) | 1 | 3% |
-
-**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
-If tsys4 goes offline, two-thirds of the fleet loses its storage.
-
----
-
-## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
-
-### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
-
-| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
-|------|------|------------|---------|-----------|----------|
-| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
-| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
-| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
-| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
-| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
-
-**Storage distribution:**
-
-| Storage server | cnodes | Quorum impact if it fails |
-|---------------|--------|--------------------------|
-| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
-| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
-
-**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
-still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
-
-**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
-tsys9) -- good host diversity.
-
-### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
-
-| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
-|------|------|------------|---------|-----------|-----|--------|----------|
-| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
-| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
-| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
-| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
-| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
-| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
-
-**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
-stopped. wnode-tsys6 restarted.
-
-**Storage distribution:**
-
-| Storage server | wnodes | Notes |
-|---------------|--------|-------|
-| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
-| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
-
-### 5.3 Summary: k8s node storage distribution
-
-| Storage server | cnodes | wnodes | Total k8s nodes |
-|---------------|--------|--------|-----------------|
-| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
-| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
-| Local SSD/NVMe | 0 | 0 | 0 (0%) |
-
-**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
-cnodes on each storage server (60/40 or better) so either server failing
-leaves quorum intact.
-
-### 5.4 Remaining cnode migration needed for etcd quorum survival
-
-To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
-be on tsys5:
-
-| Action | Effect |
-|--------|--------|
-| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
-| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
-| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
-
-**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
-PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
-
-### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
-
-The k8s layer will be tackled soon. Key requirements from user:
-
-- **Platform:** vcluster + Rancher for multi-tenant management
-- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
-- **Workload isolation (vcluster per tenant):**
-  - RackRental workloads (containerlab network labs)
-  - Suborbital ITAR (compliance-restricted)
-  - Suborbital non-ITAR
-  - Starting Line Productions customer workloads
-- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
-  will be brought online during peak solar production for burst capacity.
-  These older cores/ram supplement the main fleet when power is abundant.
-- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
-  both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
-  leftover host capacity) to large (28-32 GB, consuming most of a host).
-- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
-  full final audit will be performed at that time (post-NVMe install on tsys5,
-  post-D3 SSD relocation, post-tsys4 NIC+RAM).
-
-### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
-
-tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
-
-| VMID | Name | Storage |
-|------|------|---------|
-| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
-| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
-
-These are on local storage (good -- no NFS dependency) but on a single host's
-single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
-
----
-
-## 6. Storage Redundancy Analysis
-
-### 6.1 -01/-02 infrastructure pair audit
-
-Two -01/-02 pairs exist in the fleet:
-
-**Pair 1: pfv-netinfra (network infrastructure)**
-
-| Role | VMID | Host | Storage | NFS Server |
-|------|------|------|---------|-----------|
-| -01 | 103 | tsys1 | D5 | **tsys4** |
-| -02 | 904 | tsys9 | D2 | **tsys4** |
-
-**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
-Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
-disks. The -02 half should be on an S2/S3/S4 export from tsys5.
-
-**Pair 2: tsys-ucs (Univention Corporate Server)**
-
-| Role | VMID | Host | Storage | NFS Server |
-|------|------|------|---------|-----------|
-| -01 | 108 | tsys1 | D2 | **tsys4** |
-| -02 | 902 | tsys9 | D5 | **tsys4** |
-
-**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
-Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
-
-### 6.2 Redundancy principle for paired VMs
-
-For any -01/-02 pair to survive a single storage server failure:
-
-```
--01 VM disk -> tsys4 NFS export (D2/D3/D5)
--02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
-```
-
-This ensures that losing either tsys4 or tsys5 takes down only one half of
-the pair. Currently, **both pairs fail this test** because both halves are on
-tsys4.
-
-### 6.3 NFS server failure blast radius
-
-If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
-
-| Impact | Count |
-|--------|-------|
-| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
-| k8s wnodes that lose storage | 4 of 5 |
-| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
-| Total VMs that lose storage | **27 of 40 (68%)** |
-
-If **tsys5** goes offline:
-
-| Impact | Count |
-|--------|-------|
-| k8s cnodes that lose storage | 0 of 5 |
-| k8s wnodes that lose storage | 1 of 5 |
-| Total VMs that lose storage | 1 of 40 (3%) |
-
-**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
-Rebalancing VM storage across both servers dramatically reduces risk.
-
----
-
-## 7. Local SSD/NVMe Opportunity
-
-### 7.1 Available fast tiers (currently unused by k8s)
-
-| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
-|------|--------|------|------|--------------------|-----------------------|
-| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
-| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
-| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
-| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
-
-### 7.2 Why local storage matters for k8s nodes
-
-Kubernetes nodes are latency-sensitive in two specific areas:
-
-1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
-   On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
-   client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
-   latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
-   This directly affects k8s API responsiveness and pod scheduling speed.
-
-2. **Container image pulls:** Worker nodes pull container images frequently.
-   On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
-   eliminates the network hop and reduces seek time. This matters most for
-   RackRental/containerlab workloads that spin up containers rapidly.
-
-### 7.3 Current waste: tsys9 local SSD
-
-tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
-**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
-idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
-its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
-
-### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
-
-The PCI NVMe being added to tsys5 will be the fastest storage tier in the
-fleet. Two placement options:
-
-**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
-VMs that might need migration. Adds the NFS/network overhead back.
-
-**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
-network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
-accessed from other hosts.
-
-**Recommendation:** Option B (local-only). k8s worker nodes do not need
-shared storage -- pods are ephemeral and reschedule on failure. The NVMe
-should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
-and used for local VM images.
-
----
-
-## 8. Role Alignment Audit
-
-Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
-plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
-**correct by design** (keeps heavy hosts free for workers).
-
-### 8.1 VMs that need to move
-
-| VMID | Name | Current host | Issue | Target |
-|------|------|-------------|-------|--------|
-| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
-| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
-
-### 8.2 Host capacity for k8s nodes
-
-| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
-|------|------|-------------------|-------------|-----------|----------|
-| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
-| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
-| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
-| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
-| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
-| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
-
-**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
-free RAM respectively. They should be the primary targets for heavy worker
-nodes and ETL/HPC workloads.
-
-### 8.3 tsys6 wnode-tsys6 is stopped
-
-VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
-NFS). This wnode should be restarted (or recreated on local-lvm) to restore
-cluster capacity.
-
----
-
-## 9. Network Findings
-
-### 9.1 tsys9 storage NIC is a USB dongle (new finding)
-
-Validating tsys9 revealed that its storage network interface
-(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
-anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
-management; storage uses the USB adapter.
-
-**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
-wobble, ESD, and USB controller resets. For an infrastructure host with 5
-VMs, this is a reliability risk.
-
-**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
-PCIe NIC would eliminate this risk (same recommendation as tsys4).
-
-### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
-
-| Host | Blocker | Staged fix |
-|------|---------|-----------|
-| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
-| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
-
-### 9.3 LACP resolved on tsys6/tsys7
-
-tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
-The switch LACP hash change took effect after renegotiation. The 56-106K
-retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
-LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
-
-### 9.4 NFS nconnect=4 + noatime confirmed active
-
-All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
-options. Each host maintains 4 TCP connections per NFS mount to each storage
-server. This was the Tier 0 tuning item from the performance optimization
-engagement and is confirmed working cluster-wide.
-
----
-
-## 10. Recommendations
-
-**No changes have been made. These are assessment-only recommendations.**
-See `K8S.md` for the detailed kubernetes architecture deep-dive.
-
-### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
-
-These are the most critical production VMs in the fleet. They must survive
-any single-point failure (host, storage server, or disk).
-
-**Design principle for -01/-02 HA pairs:**
-- Different hypervisors (already satisfied: tsys1 vs tsys9)
-- Different storage servers (currently FAILED: all on tsys4)
-- Prefer SSD for latency-sensitive services
-
-**Recommended placement:**
-
-| VM | Host | Storage | Tier | Free space | Rationale |
-|----|------|---------|------|-----------|-----------|
-| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
-| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
-| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
-| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
-
-**Failure survival matrix (all single-point failures):**
-
-| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
-|---------|-------------|-------------|--------|--------|--------|
-| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
-| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
-| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
-| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
-| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
-
-**Why this works:** Every row has at least one netinfra and one UCS alive.
-The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
-
-**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
-Friday) is freed for latency-sensitive workloads that actually benefit from
-SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
-registry mirror. Not infrastructure VMs.
-
-### 10.2 CRITICAL: Cnode (control plane) storage split
-
-**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
-quorum (3 of 5 cnodes share D2).
-
-**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
-tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
-
-| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
-|-------|-------------|----------------|----------------|-------------------|-----------|
-| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
-| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
-| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
-| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
-| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
-
-**Result after changes:**
-
-| Host | cnodes | Storage server |
-|------|--------|---------------|
-| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
-| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
-| tsys3 | cnode5 (S3) | tsys5 |
-
-- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
-  etcd keeps quorum (3 of 5 survive).
-- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
-  holds more than 2 cnodes.
-- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
-  at least 3 cnodes alive.
-- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
-
-Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
-the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
-cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
-
-### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
-
-**Target: one wnode per hypervisor host, using local storage where possible
-and NFS-HDD for bulk data.**
-
-| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
-|-------|------|--------------------------|---------------------|-----------|
-| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
-| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
-| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
-| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
-| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
-| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
-| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
-
-**Storage tiering strategy per wnode:**
-
-Each wnode gets two disk tiers mapped to k8s StorageClasses:
-
-1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
-   images, ephemeral storage. This is where local SSD/NVMe shines -- container
-   image pulls and layer extraction are seek-bound and benefit enormously from
-   low-latency storage.
-
-2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
-   areas, large files that do not fit on local storage. Mounted as a second
-   disk in the VM and exposed to k8s as a StorageClass.
-
-This maps to two k8s StorageClasses:
-- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
-- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
-
-### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
-
-**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
-space and ultix-streaming (developer workstation running "cluster of 1"
-pre-production jobs). Spinning rust hosts all other infrastructure VMs**
-(UCS, netinfra, LibreNMS, SIEM, etc.).
-
-SSD allocation after Friday:
-- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
-  the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
-- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
-  (developer workstation, runs single-node test jobs before k8s). Remaining
-  140 GB available for k8s.
-
-**Deep-dive on exact k8s scratch allocation is deferred to the next session**
-(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
-
-### 10.5 MEDIUM: Restart wnode-tsys6
-
-VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
-(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
-the boot disk to local storage simultaneously.
-
-### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
-
-**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
-directory storage.**
-
-Use for:
-- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
-- sectestbed VMs that need fast scratch space
-- Not NFS-exported (avoid adding network overhead to the fastest tier)
-
-### 10.7 LOW: Add PCIe NIC to tsys9
-
-tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
-slots. A $150 PCIe NIC eliminates the USB reliability risk.
-
-### 10.8 LOW: Standardize PVE/kernel versions
-
-tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
-(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
-
----
-
-## 11. Hardware End-of-Support Exposure
-
-| Host | EOS date | Years past | Form factor |
-|------|----------|-----------|-------------|
-| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
-| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
-| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
-| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
-| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
-| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
-| pfv-tsys2 (5520) | TBD | -- | Laptop |
-| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
-
-**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
-The two actual rack servers (R610, R620) are the most overdue for replacement.
-This belongs in the next budget cycle.
-
----
-
-## 12. Open Items
-
-### 12.1 Immediate (do today via PDM)
-
-1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
-   redundancy. UCS stays on HDD.
-2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
-3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
-4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
-5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
-
-### 12.2 Friday maintenance window (user action)
-
-1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
-   reconfigure `/etc/network/interfaces`, reboot.
-2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
-   install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
-   format NVMe as local storage, reboot.
-3. **tsys2:** Load Proxmox (replacing Windows 10).
-4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
-
-### 12.3 Post-Friday validation
-
-1. Re-run iperf matrix: `./iperf-full-matrix.sh`
-2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
-3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
-4. Update PROJECT.md with post-hardware numbers.
-
-### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
-
-Next major workstream. Requirements captured:
-- **vcluster + Rancher** for multi-tenant k8s management
-- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
-- **Workload isolation** via separate vclusters:
-  - RackRental (containerlab)
-  - Suborbital ITAR
-  - Suborbital non-ITAR
-  - Starting Line Productions customer workloads
-- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
-  during peak solar production for burst capacity
-- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
-- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
-- **Spinning rust for all infrastructure VMs**
-
-### 12.5 Data gaps
-
-| Gap | How to close |
-|-----|--------------|
-| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
-| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
-| tsys5 NVMe size and model | Confirm after Friday installation |
-| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
-| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
-
-### 12.6 Scripts and tools available
-
-| Script | Purpose |
-|--------|---------|
-| `scripts/check.sh` | Read-only data collector (run on hosts) |
-| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
-| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
-| `validate-fixes.sh` | Read-only validation of all applied changes |
-| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
-| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |

proxmox/docs/STORAGE-ARCHITECTURE.md

@@ -1,196 +1,10 @@
-# Storage Architecture
+# proxmox/docs/STORAGE-ARCHITECTURE.md
 
-> **Complete storage reference for the PFVCluster.** Covers physical disks,
+> **Documentation moved to Discourse — the canonical source of truth.**
-> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
-> VM placement. Data gathered 2026-07-30 via live host probes.
 >
-> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
+> **Storage reference: tiers, disk inventory, NFS exports, VM placement**
-> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
+>
-> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
+> **Read it here:** https://community.turnsys.com/t/300
-
+>
----
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## 1. Storage tier definitions
-
-| Tier | Definition | Typical speed | Reliability | Use case |
-|------|-----------|---------------|-------------|----------|
-| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch |
-| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage |
-| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. |
-
-**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk
-storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
-(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
-and HA pairs. Local storage is for ephemeral data where loss on node failure
-is acceptable.
-
----
-
-## 2. Physical disk inventory
-
-### tsys4 -- NFS storage server + PBS (7 disks)
-
-| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
-|--------|-------|------|-----|-----------|-------|--------|------|-------|------|
-| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% |
-| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
-| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
-| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
-| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% |
-| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
-| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% |
-
-**Changes from previous audit:**
-- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
-- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
-- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only.
-- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone.
-
-### tsys5 -- NFS storage server + sandboxes (7 disks)
-
-| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
-|--------|-------|------|-----|-----------|-------|-------------|------|-------|------|
-| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- |
-| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** |
-| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk |
-| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** |
-| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) |
-| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) |
-| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |
-
-**Changes from previous audit:**
-- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
-- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
-- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
-- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry.
-- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
-- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention.
-- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).
-
----
-
-## 3. NFS export → Proxmox storage ID map
-
-```mermaid
-graph LR
-    subgraph "tsys4 -- NFS Server"
-        D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
-        D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
-        D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
-        D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
-    end
-
-    subgraph "tsys5 -- NFS Server"
-        S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
-        S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
-        S3stale["S3: (stale export)<br>disk not mounted"]
-    end
-
-    subgraph "All hosts -- NFS Clients"
-        MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
-    end
-
-    D2 -->|NFS v4.2| MNT
-    D5 -->|NFS v4.2| MNT
-    D1 -->|NFS v4.2| MNT
-    D3 -->|NFS v4.2| MNT
-    S1 -->|NFS v4.2| MNT
-    S2 -->|NFS v4.2| MNT
-    S3stale -.->|broken| MNT
-```
-
-### NFS export details
-
-| Server | Export path | NFS ID | Protocol | Options (client-side) |
-|--------|-------------|--------|----------|----------------------|
-| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) |
-| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
-| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) |
-
-> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from
-> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on
-> all mounts.
-
----
-
-## 4. Per-spindle VM placement (running VMs only)
-
-> Shows every running VM's primary disk and the physical spindle it lives on.
-> This is the IOPS distribution map -- the key to storage balancing.
-
-```mermaid
-graph TB
-    subgraph "tsys4 spindles"
-        D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
-        D5["D5 -- Hitachi 2TB<br>4 VMs"]
-        D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
-        D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
-        ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
-    end
-
-    subgraph "tsys5 spindles"
-        S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
-        S2["S2 -- IronWolf 1TB<br>5 VMs"]
-        SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
-        NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
-        HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
-    end
-
-    D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
-    D5 --- D5vms["106 313 600(d0) 903"]
-    S1 --- S1vms["702 hfnoc-uisp"]
-    S2 --- S2vms["103 703 705 902 905"]
-    SSD --- SSDvms["5111 ultix-streaming"]
-```
-
-### Detailed per-spindle load
-
-| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes |
-|---------|--------|--------|------|-------------|-------|-------|
-| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** |
-| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate |
-| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
-| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
-| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** |
-| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty |
-| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty |
-| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox |
-| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
-| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
-
-### Local-only VM disks (not on NFS)
-
-| Host | Storage ID | Disk | VMs | Notes |
-|------|-----------|------|-----|-------|
-| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk |
-| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage |
-| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. |
-| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. |
-
----
-
-## 5. Storage capacity summary
-
-| Spindle | Size | Used | Available | % Used |
-|---------|------|------|-----------|--------|
-| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% |
-| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% |
-| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% |
-| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% |
-| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% |
-| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% |
-| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
-| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
-| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** |
-| NVMe (CT500P1) | 466 GB | -- | -- | Not configured |
-| SK hynix SSD | 477 GB | -- | -- | Not configured |
-
-**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2
-**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup
-**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD

proxmox/docs/TODO.md

@@ -1,242 +1,10 @@
-# TODO.md — Pending User Actions
+# proxmox/docs/TODO.md
 
-**Date:** 2026-07-27
+> **Documentation moved to Discourse — the canonical source of truth.**
-**Items needing user input or physical action.**
+>
-
+> **Pending hardware work — folded into Operations Status**
----
+>
-
+> **Read it here:** https://community.turnsys.com/t/298
-## 1. tsys2 Windows hardware inventory (run on the Windows host)
+>
-
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
+> Do not update content here — edit the Discourse wiki topic instead.*
-hardware data so the architecture plan can account for it.
-
-### Option A: PowerShell (recommended — single command, copy-paste output)
-
-Open **PowerShell as Administrator** and run:
-
-```powershell
-# Full hardware inventory in one shot
-Write-Output "=== COMPUTER ==="
-Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
-
-Write-Output "`n=== CPU ==="
-Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
-
-Write-Output "`n=== MEMORY STICKS ==="
-Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
-
-Write-Output "`n=== DISKS ==="
-Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
-
-Write-Output "`n=== DISK PARTITIONS ==="
-Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
-
-Write-Output "`n=== NETWORK ADAPTERS ==="
-Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
-
-Write-Output "`n=== GPU(s) ==="
-Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
-
-Write-Output "`n=== PCIe SLOTS ==="
-Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
-
-Write-Output "`n=== USB DEVICES (storage + network only) ==="
-Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
-```
-
-Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
-directly into the chat.
-
-### Option B: Command Prompt (cmd.exe) fallbacks
-
-If PowerShell is unavailable for some reason, these cmd commands give a
-subset:
-
-```cmd
-:: Computer model and serial
-wmic computersystem get manufacturer,model
-wmic bios get serialnumber
-
-:: CPU
-wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
-
-:: RAM (total)
-wmic computersystem get totalphysicalmemory
-
-:: RAM sticks (per-slot detail)
-wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
-
-:: Disks
-wmic diskdrive get model,size,interfacetype,mediatype
-
-:: Network adapters
-wmic nic where netenabled=true get name,speed,macaddress
-
-:: GPU
-wmic path win32_videocontroller get name,adapterram,driverversion
-```
-
-### What I'm looking for
-
-- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
-  whether wnode-tsys2 can use local storage like the other wnodes.)
-- **Network adapters**: How many onboard NICs? Model? (Determines whether
-  tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
-  onboard NIC available.)
-- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
-  memory bandwidth for HPC workloads — dual-channel matters.)
-- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
-  planning.
-- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
-- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
-
-### STATUS: Collected 2026-07-27
-
-**Disk inventory — RESOLVED:**
-- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
-- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
-- Both SSDs, no spinning rust. 1.5 TB total local SSD.
-
-**Network adapters — RESOLVED (concerning):**
-- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
-- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
-- Wi-Fi: Intel 8265 (disconnected)
-- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
-  Unavoidable on this laptop form factor — no onboard PCIe NIC available.
-
-**GPU — RESOLVED:**
-- Intel HD Graphics 630 (integrated, 1 GB)
-- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
-
-**PCIe slots — RESOLVED:**
-- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
-  user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
-  by 960 PRO.
-
-**RAM — PARTIAL:**
-- Total 32 GB confirmed (34,097,573,888 bytes).
-- Per-stick detail failed to run (PowerShell line-break split
-  `Format-T` + `able`). Re-run the command below if bandwidth planning
-  needs stick-level detail:
-  ```powershell
-  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
-  ```
-
-**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
-
----
-
-## 2. Friday maintenance window (physical hardware)
-
-### tsys4 — install PCIe NIC + add RAM
-
-1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
-2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
-3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
-4. Power on, then update `/etc/network/interfaces` to replace
-   `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
-5. Reboot to activate new NIC and NFS nconnect.
-6. Run `validate-fixes.sh pfv-tsys4` to confirm.
-
-### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
-
-1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
-2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
-   "Number of ports: 2" and a real partner MAC (not all zeros).
-3. Apply bond hash fix (same as tsys6/7):
-   ```bash
-   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
-   ```
-4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
-   - Power down tsys4
-   - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
-   - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
-   - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
-   - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
-     `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
-   - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
-     empty, so minimal migration)
-5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
-6. Format NVMe as local directory storage (see TODO section 3 below).
-7. Reboot tsys5 to activate NFS nconnect.
-8. Run `validate-fixes.sh pfv-tsys5` to confirm.
-
----
-
-## 3. tsys5 NVMe format/mount decision (after Friday install)
-
-**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
-storage so it shows up as a VM image target in the Proxmox UI.
-
-After the NVMe is physically installed and visible in Proxmox:
-
-1. Identify the device: `lsblk` or `ls /dev/nvme*`
-2. Format and add to Proxmox:
-   ```bash
-   # Option A: LVM-thin (thin provisioning, snapshots)
-   pvcreate /dev/nvme0n1
-   vgcreate nvme-pool /dev/nvme0n1
-   lvcreate -l 100%FREE -T nvme-pool/data
-   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
-   #   ID: nvme-local
-   #   Volume Group: nvme-pool
-   #   Thin Pool: data
-   #   Content: Disk image, Container template
-
-   # Option B: Directory (simpler, no thin provisioning)
-   mkfs.ext4 /dev/nvme0n1
-   mkdir -p /mnt/nvme
-   mount /dev/nvme0n1 /mnt/nvme
-   # Add to /etc/fstab for persistence
-   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
-   #   ID: nvme-local
-   #   Directory: /mnt/nvme
-   #   Content: Disk image, Container template
-   ```
-
-3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
-
----
-
-## 4. Post-hardware validation (run after Friday work)
-
-1. Re-run iperf matrix: `./iperf-full-matrix.sh`
-2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
-3. Update PROJECT.md with post-hardware iperf numbers.
-
----
-
-## 5. UCS storage migration to spinning disk (do today)
-
-UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
-UCS VMs should stay on spinning disk (HDD) and be split across storage
-servers for redundancy.
-
-| VM | Current | Target | Action |
-|----|---------|--------|--------|
-| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
-| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
-
-**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
-1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
-2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
-   (both are NFS exports visible to tsys9, so this is a storage-only migration)
-3. Verify VM 902 boots and LDAP/AD services are healthy after migration
-
-Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
-ucs-02 needs to move -- it should be on a different storage server than
-ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
-
----
-
-## 6. Open questions for next session
-
-- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
-  Determines whether live migration is available.
-- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
-- Container runtime? (containerd, cri-o?)
-- Is there a local container image registry mirror?
-- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
-- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
-- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
-- tsys3 thermal state (laptop in rack for years) — check `sensors`.

proxmox/docs/VM-INVENTORY.md

@@ -1,225 +1,10 @@
-# VM Inventory
+# proxmox/docs/VM-INVENTORY.md
 
-> **Every VM in the cluster, with host placement, disk location, CPU/RAM
+> **Documentation moved to Discourse — the canonical source of truth.**
-> allocation, and optimization status.** Data gathered 2026-07-30.
 >
-> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
+> **Complete VM catalog with disk/CPU/RAM/config**
-> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
+>
-> [`HOST-REFERENCE.md`](HOST-REFERENCE.md).
+> **Read it here:** https://community.turnsys.com/t/304
-
+>
----
+> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
-
+> Do not update content here — edit the Discourse wiki topic instead.*
-## 1. VM placement diagram
-
-```mermaid
-graph TB
-    subgraph tsys1["tsys1 -- Infra (57% RAM)"]
-        VM100["100 pfv-bms<br>2c/4G D2"]
-        VM101["101 tsys-ca<br>2c/2G D2"]
-        VM103["103 netinfra-01<br>2c/4G S2"]
-        VM104["104 librenms<br>2c/2G D2"]
-        VM105["105 proxmox-dc<br>2c/2G D2"]
-        VM106["106 cnode3<br>4c/4G D5"]
-        VM108["108 ucs-01<br>2c/8G D2"]
-        VM114["114 kali-tsys<br>2c/2G D2"]
-        VM117["117 workbench<br>2c/4G D2"]
-    end
-
-    subgraph tsys3["tsys3 -- Compute (10% RAM)"]
-        VM313["313 wnode-tsys3<br>8c/28G D5"]
-    end
-
-    subgraph tsys4["tsys4 -- Storage (15% RAM)"]
-        VM400["400 PBS<br>2c/2G local-lvm"]
-    end
-
-    subgraph tsys5["tsys5 -- Storage+Sandbox (15% RAM)"]
-        VM509["509 wnode-tsys5<br>2c/32G D2"]
-        VM515["515 hfnoc-uisp-preprod<br>2c/2G local-nonprod"]
-        VM5111["5111 ultix-streaming<br>4c/9G SSD"]
-        VM5112["5112 ultix-offstage<br>4c/6G local-lvm"]
-    end
-
-    subgraph tsys6["tsys6 -- RackRental (12% RAM)"]
-        VM100b["100 wnode-tsys6<br>2c/64G S4"]
-        VM600["600 tsys-awx<br>2c/12G D5"]
-    end
-
-    subgraph tsys7["tsys7 -- RackRental (7% RAM)"]
-        VM701["701 wnode-tsys7<br>4c/32G S3"]
-        VM702["702 hfnoc-uisp<br>2c/8G S1"]
-        VM703["703 rr-middleware<br>2c/2G S2"]
-        VM704["704 TCTC<br>4c/6G D2"]
-        VM705["705 cnode2<br>4c/4G S2"]
-    end
-
-    subgraph tsys9["tsys9 -- Infra+Compute (69% RAM)"]
-        VM901["901 tsys-siem<br>2c/8G D2"]
-        VM902["902 ucs-02<br>2c/8G S2"]
-        VM903["903 kali-rd<br>2c/2G D5"]
-        VM904["904 netinfra-02<br>2c/4G D2"]
-        VM905["905 wnode-tsys9<br>4c/4G S2"]
-        VM906["906 cnode1<br>2c/4G S3"]
-    end
-```
-
----
-
-## 2. Complete running VM inventory (28 running VMs)
-
-### Infrastructure (netinfra, UCS, CA, LibreNMS, PDM)
-
-| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
-|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
-| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | -- | -- | ✓ | discard=on, ssd=1. HA for power/temp monitoring. |
-| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | no nested-virt flag. Certificate Authority. |
-| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | -- | DNS primary (Technitium + Pi-hole). |
-| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | SNMP monitoring for switches. |
-| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | PDM (Proxmox Datacenter Manager). |
-| 108 | tsys-ucs-01 | tsys1 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Univention Corporate Server half 1. |
-| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | -- | ✓ | ✓ | UCS half 2 (HA pair with 108). |
-| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | -- | ✓ | -- | DNS secondary. HA pair with 103. |
-
-### k8s control plane (cnode1/2/3)
-
-| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
-|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
-| 106 | pfv-k8s-cnode3 | tsys1 | 4 | host | 4G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. |
-| 705 | pfv-k8s-cnode2 | tsys7 | 4 | host | 4G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. |
-| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | -- | ✓ | -- | etcd member. |
-
-### k8s workers (wnodes)
-
-| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
-|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
-| 313 | pfv-k8s-wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Compute worker. |
-| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | Undersized vCPU (2). |
-| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | -- | ✓ | -- | Undersized vCPU (2), high RAM. |
-| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | -- | ✓ | -- | RackRental worker. |
-| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | ✓ | Target: grow to 6c/8G. |
-
-### Application VMs
-
-| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
-|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
-| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Kali Linux pentest. |
-| 117 | tsys-secure-workbench | tsys1 | 2 | host | 4G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Secure workbench. |
-| 400 | pfv-proxmox-backup-server | tsys4 | 2 | -- | 2G | local-lvm | (local) | raw | -- | -- | -- | PBS -- all VM backups. |
-| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | -- | ✓ | -- | Preprod. Locked to tsys5. |
-| 5111 | ultix-streaming | tsys5 | 4 | -- | 9G | SSD dir | SSD | qcow2 | -- | -- | -- | 288GB on Samsung 860 PRO (**93% full**). |
-| 5112 | ultix-offstage | tsys5 | 4 | -- | 6G | local-lvm | (local) | raw | -- | -- | -- | 288GB. Locked to tsys5. |
-| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Ansible AWX. |
-| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | -- | ✓ | ✓ | 100GB disk. UISP network monitoring. |
-| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | RackRental middleware. |
-| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | |
-| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | 132GB disk. SIEM. |
-| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Kali R&D. |
-
----
-
-## 3. Optimization findings
-
-### 3.1 Cache mode audit
-
-`cache=writethrough` forces synchronous writes to the NFS server on every write
-operation. For NFS-backed qcow2, this is unnecessarily slow -- `writeback` or
-`none` is recommended (qcow2 already provides journaling integrity).
-
-| Cache mode | Count | VMs |
-|-----------|-------|-----|
-| **writethrough** (slow) | **10** | 106, 108, 114, 117, 313, 600, 703, 705, 901, 903 |
-| none/default (optimal) | 18 | all others |
-
-**Fix:** Change `cache=writethrough` → `cache=writeback` or remove (defaults to
-writeback for qcow2). Requires VM stop/start.
-
-### 3.2 CPU type audit
-
-`cpu: host` passes the host's full CPU instruction set to the VM (best
-performance). `x86-64-v2-AES` is a conservative baseline (safe for migration
-but lacks some host instructions).
-
-| CPU type | Count | VMs |
-|----------|-------|-----|
-| **host** | 21 | most production VMs |
-| **x86-64-v2-AES** | 5 | 101, 103, 509, 904, 905 |
-| not set | 2 | 400, 5111, 5112 |
-
-**Note:** Mixed CPU types prevent live migration between hosts with different
-CPU generations. K8s cnodes/wnodes should ideally match within each tier.
-
-### 3.3 Disk controller audit
-
-| Controller | Count | Notes |
-|-----------|-------|-------|
-| **scsi0** (virtio-scsi) | 22 | Optimal for Proxmox |
-| **virtio0** | 4 | 101, 509, 701, 704 -- also good |
-| raw (no virtio) | 2 | 100 (pfv-bms), 500 (stopped) |
-
-All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks.
-
-### 3.4 Network audit
-
-All running VMs use **virtio-net**. Firewall enabled on most (`firewall=1`).
-No legacy e1000/rtl8139 adapters.
-
-### 3.5 Guest agent (qga) audit
-
-| Status | Count | VMs without qga |
-|--------|-------|-----------------|
-| **enabled** | 20 | -- |
-| **not set** | 8 | 101, 103, 509, 100(tsys6), 701, 704, 904, 906 |
-
-**Fix:** Enable `qm set <vmid> --agent 1` on the 8 VMs missing it. No VM
-stop/start required (takes effect on next guest reboot, but the setting itself
-is immediate).
-
-### 3.6 k8s wnode sizing
-
-| VMID | Name | vCPU | RAM | Tier target | Issue |
-|------|------|------|-----|-------------|-------|
-| 509 | wnode-tsys5 | 2 | 32G | compute | **Undersized vCPU** (2 for a compute worker) |
-| 100 | wnode-tsys6 | 2 | 64G | rackrental | **Undersized vCPU** (2 for 64G RAM) |
-| 701 | wnode-tsys7 | 4 | 32G | rackrental | OK for idle labs |
-| 905 | wnode-tsys9 | 4 | 4G | compute | **Target: grow to 6c/8G** after siem migrates off |
-| 313 | wnode-tsys3 | 8 | 28G | compute | OK (best compute worker) |
-
----
-
-## 4. Stopped VMs (not running, for reference)
-
-### Sectestbed / sandboxes (all on tsys5, all stopped)
-
-| VMID | Name | RAM | Disk | Notes |
-|------|------|-----|------|-------|
-| 5100 | sectestbed-template | 4G | local-lvm | Base template |
-| 5101 | sectestbed-siem | 10G | local-nonprod (S1) | 132G disk |
-| 5102 | sectestbed-proxmox-pve | 4G | local-lvm | |
-| 5103 | sectestbed-proxmox-datacenter | 4G | local-lvm | |
-| 5104 | sectestbed-proxmox-pbs | 4G | local-lvm | |
-| 5105 | sectestbed-awx | 4G | local-nonprod (S1) | 288G disk |
-| 5106 | sectestbed-k8s-cnode | 4G | local-nonprod (S1) | |
-| 5107 | sectestbed-k8s-wnode | 4G | local-nonprod (S1) | |
-| 5108 | sectestbed-librenms | 4G | local-nonprod (S1) | |
-| 5109 | sectestbed-netinfra | 4G | local-nonprod (S1) | |
-| 51010 | sectestbed-tctc | 4G | local-nonprod (S1) | |
-| 51011 | sectestbed-cloudron | 4G | local-nonprod (S1) | |
-| 51012 | sectestbed-hfnoc | 4G | local-nonprod (S1) | |
-| 51013 | sectestbed-rancherplatform | 4G | local-nonprod (S1) | |
-| 51014 | sectestbed-proxmox-mailgw | 4G | local-nonprod (S1) | |
-| 53100 | tsys-preprod-awx | 9G | local-nonprod (S1) | 160G disk |
-| 53101 | tsys-preprod-siem | 12G | local-nonprod (S1) | |
-| 53102 | tsys-preprod-rancherplatform | 8G | local-nonprod (S1) | |
-| 53103 | tsys-preprod-proxmoxmailgw | 4G | local-nonprod (S1) | |
-| 6000 | sectestbed-sandbox | 4G | local-nonprod (S1) | Has 3 state snapshots |
-
-### Other stopped VMs
-
-| VMID | Name | Host | RAM | Disk | Notes |
-|------|------|------|-----|------|-------|
-| 102 | pfv-k8s-wnode-tsys1 | tsys1 | 4G | (no disk) | Stopped, no boot disk |
-| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail -- disk export removed** |
-| 5500 | RestoreTemplate | tsys5 | 2G | D2 | Template |
-| 1000 | KNELTemplate | tsys1 | 2G | D2 | Base template |
-| 3000 | KNELTemplate | tsys3 | 2G | D2 | Base template copy |
-| 4000 | KNELTemplate | tsys4 | 2G | (local) | Base template copy |

proxmox/perf/iperf-full-matrix.sh

@@ -288,6 +288,6 @@ echo "===== CONTEXT ====="
 echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
 echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
 echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
-echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
+echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor-01 → pfv-r5-core-01)"
 echo ""
 echo "All logs in: $LOG_DIR/"

proxmox/perf/scripts/add-datanet-nics.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+###############################################################################
+# add-datanet-nics.sh
+#
+# Adds a second virtio NIC (net1) to all k8s + ultix VMs, bridged to the
+# storage/datanet network (VLAN 1000). Records the assigned IP in phpIPAM.
+#
+# Runs from the workstation — uses tests/remote.sh to reach each hypervisor.
+# Hot-adds the NIC (no VM downtime). Guest-side IP config must be done
+# separately (the guest OS needs the IP configured on the new interface).
+#
+# Usage:
+#   bash add-datanet-nics.sh             # add NICs (hot-add, no reboot)
+#   bash add-datanet-nics.sh --status    # show current state only
+#
+# IP allocations (phpIPAM VPTechOps, subnet 10.100.100.0/24):
+#   .10 = cnode1    (102/tsys1)
+#   .11 = cnode2    (705/tsys7)
+#   .12 = cnode3    (603/tsys6)
+#   .13 = wnode-tsys3 (313/tsys3)
+#   .14 = wnode-tsys5 (500/tsys5)
+#   .15 = wnode-tsys6 (601/tsys6)
+#   .16 = wnode-tsys7 (701/tsys7)
+#   .17 = wnode-tsys9 (905/tsys9)
+#   .18 = ultix-streaming (5111/tsys5)
+#   .19 = ultix-offstage  (5112/tsys5)
+#
+# Related: Redmine [#396]
+###############################################################################
+set -euo pipefail
+
+# VM definitions: VMID HOST BRIDGE IP HOSTNAME
+VMS=(
+    "102  tsys1  datanet      10.100.100.10  pfv-k8s-cnode1"
+    "705  tsys7  datanet      10.100.100.11  pfv-k8s-cnode2"
+    "603  tsys6  storagenet   10.100.100.12  pfv-k8s-cnode3"
+    "313  tsys3  datanet      10.100.100.13  pfv-k8s-wnode-tsys3"
+    "500  tsys5  datanet      10.100.100.14  pfv-k8s-wnode-tsys5"
+    "601  tsys6  storagenet   10.100.100.15  pfv-k8s-wnode-tsys6"
+    "701  tsys7  datanet      10.100.100.16  pfv-k8s-wnode-tsys7"
+    "905  tsys9  datanet      10.100.100.17  pfv-k8s-wnode-tsys9"
+    "5111 tsys5  datanet      10.100.100.18  ultix-streaming"
+    "5112 tsys5  datanet      10.100.100.19  ultix-offstage"
+)
+
+ACTION="${1:-add}"
+
+echo "==================================================================="
+echo " add-datanet-nics — [#396]"
+echo " mode: ${ACTION}"
+echo "==================================================================="
+echo ""
+
+for entry in "${VMS[@]}"; do
+    read -r vmid host bridge ip hostname <<< "$entry"
+    prox_host="pfv-${host}"
+
+    echo "--- ${hostname} (VM ${vmid} on ${prox_host}) ---"
+
+    if [ "$ACTION" = "--status" ]; then
+        # Show current NIC state
+        PROX_HOST="$prox_host" bash tests/remote.sh prox \
+            "qm config ${vmid} 2>/dev/null | grep -E '^net|^name'" 2>&1
+        echo ""
+        continue
+    fi
+
+    # Check if net1 already exists
+    existing=$(PROX_HOST="$prox_host" bash tests/remote.sh prox \
+        "qm config ${vmid} 2>/dev/null | grep '^net1'" 2>&1 || true)
+
+    if [ -n "$existing" ]; then
+        echo "  net1 already exists: ${existing}"
+        echo "  Skipping."
+        echo ""
+        continue
+    fi
+
+    # Hot-add net1 bridged to the storage network
+    echo "  Adding net1 (bridge=${bridge}, IP=${ip})..."
+    if PROX_HOST="$prox_host" bash tests/remote.sh prox \
+        "qm set ${vmid} -net1 virtio,bridge=${bridge}" 2>&1; then
+        echo "  NIC added. Verify with: qm config ${vmid} | grep net1"
+    else
+        echo "  FAILED — check error above"
+    fi
+    echo ""
+done
+
+if [ "$ACTION" != "--status" ]; then
+    echo "==================================================================="
+    echo " NICs added. Guest-side IP config still needed."
+    echo " Each guest needs the IP configured on the new interface."
+    echo " IPs are allocated in phpIPAM (VPTechOps / 10.100.100.0/24)."
+    echo "==================================================================="
+fi

proxmox/perf/scripts/audit-guest-io.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/bash
+# Audit VM disk cache from inside guests — confirm writeback visible + I/O perf
+set -uo pipefail
+source "$(cd "$(dirname "$0")/../../.." && pwd)/k8s/env.sh"
+
+echo "=== CNODES (via Tailscale) ==="
+for ip in "${ALL_CNODES[@]}"; do
+  echo "--- $ip ---"
+  # shellcheck disable=SC2016 # heredoc-style remote command uses $() on the remote side
+  cn "$ip" '
+    echo "  write_cache: $(cat /sys/block/sda/queue/write_cache 2>/dev/null)"
+    echo "  scheduler: $(cat /sys/block/sda/queue/scheduler 2>/dev/null)"
+    echo "  fsync (5x 1KB):"
+    for i in 1 2 3 4 5; do
+      t0=$(date +%s%N)
+      dd if=/dev/zero of=/tmp/.ft bs=1k count=1 conv=fsync 2>/dev/null
+      t1=$(date +%s%N)
+      echo -n " $(( (t1-t0)/1000000 )) ms"
+    done
+    echo
+    rm -f /tmp/.ft
+  ' 2>&1 || echo "  UNREACHABLE"
+done
+
+echo ""
+echo "=== DNS INFRA VMs ==="
+for host in netinfra01 netinfra02; do
+  echo "--- $host ---"
+  # shellcheck disable=SC2016 # heredoc-style remote command uses $() on the remote side
+  netinfra/dns-cluster-setup/remote-dns.sh "$host-root" '
+    echo "  write_cache: $(cat /sys/block/sda/queue/write_cache 2>/dev/null)"
+    echo "  fsync (3x 1KB):"
+    for i in 1 2 3; do
+      t0=$(date +%s%N)
+      dd if=/dev/zero of=/tmp/.ft bs=1k count=1 conv=fsync 2>/dev/null
+      t1=$(date +%s%N)
+      echo -n " $(( (t1-t0)/1000000 )) ms"
+    done
+    echo
+    rm -f /tmp/.ft
+  ' 2>&1 || echo "  UNREACHABLE"
+done

proxmox/perf/scripts/audit-vm-disks.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/bash
+# Audit all VM disk configs on a Proxmox host
+set -uo pipefail
+HOST="$1"
+PROX_HOST="$HOST" bash tests/remote.sh prox '
+  qm list 2>/dev/null | tail -n +2 | while read -r line; do
+    vmid=$(echo "$line" | awk "{print \$1}")
+    name=$(echo "$line" | awk "{print \$2}")
+    status=$(echo "$line" | awk "{print \$3}")
+    echo "VMID=$vmid NAME=$name STATUS=$status"
+    qm config "$vmid" 2>/dev/null | grep -E "^(scsi|virtio|ide)[0-9]+:" | sed "s/^/  /"
+    echo ""
+  done
+'

proxmox/perf/scripts/conman-console.py

@@ -22,7 +22,7 @@ any network with a conman server:
 
 Usage:
     CONMAN_SERVER=console-host:7890 \\
-    python3 conman-console.py --console pfv-core-sw01 --cmds switches/pfv-core-sw01.cmds
+    python3 conman-console.py --console pfv-r5-core-01 --cmds switches/pfv-r5-core-01.cmds
 
 Lines starting with '!' or '#' in the cmds file are comments (skipped).
 Blank lines are skipped. The conman escape sequence (&.) is sent automatically
@@ -51,7 +51,7 @@ def main():
     ap = argparse.ArgumentParser(
         description="Drive a conman console session read-only via PTY")
     ap.add_argument("--console", required=True,
-                    help="console name (e.g. pfv-core-sw01)")
+                    help="console name (e.g. pfv-r5-core-01)")
     ap.add_argument("--cmds", required=True,
                     help="command file (one command per line; !/# = comment)")
     ap.add_argument("--server",

proxmox/perf/scripts/deploy-tuned-guests.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/bash
+# Deploy tuned inside VMs over SSH (sshd is the only approved access channel —
+# see AGENTS.md "Access-channel policy: SSH only"). Formerly used the
+# qemu guest-agent channel; converted to SSH now that all VMs have key + sudo.
+set -uo pipefail
+
+# Profile mapping: most VMs get throughput-performance, k8s/network-sensitive get network-latency
+PROFILE="${1:-throughput-performance}"
+REMOTE_SH="${REMOTE_SH:-$(cd "$(dirname "$0")/../../.." && pwd)/tests/remote.sh}"
+VM_USER="${VM_USER:-localuser}"
+
+deploy_vm() {
+  local host="$1" vmid="$2" name="$3"
+  # name is the Tailscale hostname (SSH target); vmid retained for reference.
+  echo -n "  VMID $vmid ($name) on $host: "
+  local result
+  result=$(VM_IP="$name" VM_USER="$VM_USER" bash "$REMOTE_SH" vmroot \
+    "DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>/dev/null; \
+     DEBIAN_FRONTEND=noninteractive apt-get install -y -qq tuned 2>/dev/null; \
+     tuned-adm profile $PROFILE 2>/dev/null; \
+     systemctl enable --now tuned 2>/dev/null; \
+     tuned-adm active 2>/dev/null" </dev/null 2>&1)
+  if echo "$result" | grep -q 'Current active'; then
+    echo "$result" | grep -o 'Current active.*' | head -1
+  elif echo "$result" | grep -qi 'permission denied\|no route\|timed out'; then
+    echo "SSH FAILED (no key/no sudo) — run bootstrap-all.sh first"
+  else
+    echo "INSTALL FAILED (apt issue or no network)"
+  fi
+}
+
+echo "============================================"
+echo "  Deploying tuned ($PROFILE) to VMs via SSH"
+echo "============================================"
+echo ""
+
+# tsys1 VMs
+echo "--- pfv-tsys1 ---"
+deploy_vm pfv-tsys1 100 pfv-bms
+deploy_vm pfv-tsys1 101 tsys-ca
+deploy_vm pfv-tsys1 102 pfv-k8s-cnode1
+deploy_vm pfv-tsys1 104 tsys-librenms
+deploy_vm pfv-tsys1 105 tsys-proxmox-datacenter
+deploy_vm pfv-tsys1 108 tsys-ucs-01
+
+# tsys3 VMs
+echo "--- pfv-tsys3 ---"
+deploy_vm pfv-tsys3 313 pfv-k8s-wnode-tsys3
+
+# tsys6 VMs
+echo "--- pfv-tsys6 ---"
+deploy_vm pfv-tsys6 600 tsys-awx
+deploy_vm pfv-tsys6 601 pfv-k8s-wnode-tsys6
+deploy_vm pfv-tsys6 602 pfv-rr-middleware-02
+deploy_vm pfv-tsys6 603 pfv-k8s-cnode3
+deploy_vm pfv-tsys6 604 tsys-proxmox-mailgw-01
+
+# tsys7 VMs
+echo "--- pfv-tsys7 ---"
+deploy_vm pfv-tsys7 701 pfv-k8s-wnode-tsys7
+deploy_vm pfv-tsys7 702 hfnoc-uisp
+deploy_vm pfv-tsys7 703 pfv-rr-middleware-01
+deploy_vm pfv-tsys7 705 pfv-k8s-cnode2
+deploy_vm pfv-tsys7 706 kali-rd
+deploy_vm pfv-tsys7 707 tsys-siem
+deploy_vm pfv-tsys7 708 kali-tsys
+deploy_vm pfv-tsys7 709 tsys-voip
+deploy_vm pfv-tsys7 711 tsys-proxmox-mailgw-02
+
+# tsys9 VMs
+echo "--- pfv-tsys9 ---"
+deploy_vm pfv-tsys9 902 tsys-ucs-02
+deploy_vm pfv-tsys9 905 pfv-k8s-wnode-tsys9
+
+echo ""
+echo "============================================"
+echo "  Deployment complete."
+echo "  NOT deployed (do-not-reboot VMs):"
+echo "    ultix-streaming, ultix-offstage,"
+echo "    pfv-netinfra-01, pfv-netinfra-02"
+echo "============================================"

proxmox/perf/scripts/fix-e1000e-offload.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+###############################################################################
+# fix-e1000e-offload.sh — Disable offload on e1000e NICs to prevent hangs
+#
+# The Intel e1000e driver has a known bug where TSO/GSO/GRO offload causes
+# "hardware unit hang" resets on certain Intel NICs (I217-LM, I219-LM, 82571EB).
+# This script disables offload features on all e1000e interfaces and installs
+# a systemd service to persist across reboots.
+#
+# References:
+#   https://forum.proxmox.com/threads/e1000-driver-hang.58284/
+#   https://serverfault.com/questions/616485
+#
+# Run on each Proxmox host:
+#   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file proxmox/perf/scripts/fix-e1000e-offload.sh
+###############################################################################
+set -euo pipefail
+
+echo "=== e1000e Offload Fix on $(hostname) ==="
+
+# Find all e1000e physical NICs (skip bridges, bonds, virtual interfaces)
+AFFECTED_NICS=()
+for nic_path in /sys/class/net/*; do
+    nic=$(basename "$nic_path")
+    [ "$nic" = "lo" ] && continue
+    # Skip bridges, bonds, virtual interfaces
+    [ -d "${nic_path}/bridge" ] && continue
+    [ -d "${nic_path}/bonding" ] && continue
+    case "$nic" in
+        tap*|veth*|fwpr*|fwln*|vmbr*|datanet*|storagenet*|tailscale*) continue ;;
+    esac
+
+    driver=$(ethtool -i "$nic" 2>/dev/null | awk '/^driver:/{print $2}')
+    if [ "$driver" = "e1000e" ]; then
+        AFFECTED_NICS+=("$nic")
+        echo "  Found e1000e NIC: $nic"
+    fi
+done
+
+if [ "${#AFFECTED_NICS[@]}" -eq 0 ]; then
+    echo "  No e1000e NICs found. Nothing to do."
+    exit 0
+fi
+
+# Apply fix live
+echo ""
+echo "--- Disabling offload features ---"
+for nic in "${AFFECTED_NICS[@]}"; do
+    echo "  $nic:"
+    ethtool -K "$nic" tso off gro off gso off tx off rx off 2>&1 | sed 's/^/    /' || true
+    tso_state=$(ethtool -k "$nic" 2>/dev/null | awk '/tcp-segmentation-offload/{print $2}' | head -1)
+    echo "    tso=$tso_state"
+done
+
+# Install systemd service for persistence
+echo ""
+echo "--- Installing systemd service ---"
+{
+    echo "[Unit]"
+    echo "Description=Disable offload on e1000e NICs (prevent hardware unit hang)"
+    echo "After=network.target"
+    echo "Wants=network.target"
+    echo ""
+    echo "[Service]"
+    echo "Type=oneshot"
+    echo "RemainAfterExit=yes"
+    for nic in "${AFFECTED_NICS[@]}"; do
+        echo "ExecStart=/sbin/ethtool -K $nic tso off gro off gso off tx off rx off"
+    done
+    echo ""
+    echo "[Install]"
+    echo "WantedBy=multi-user.target"
+} > /etc/systemd/system/fix-e1000e-offload.service
+
+systemctl daemon-reload
+systemctl enable fix-e1000e-offload.service
+echo "  Service installed and enabled (fix-e1000e-offload.service)"
+echo ""
+echo "=== Done. Affected NICs: ${AFFECTED_NICS[*]} ==="

proxmox/perf/scripts/fix-tsys4-storage-bond.sh

@@ -0,0 +1,283 @@
+#!/bin/bash
+###############################################################################
+# fix-tsys4-storage-bond.sh
+#
+# Reconfigures pfv-tsys4 storage network bond from broken 802.3ad (LACP) to
+# active-backup mode. USB NICs cannot participate in LACP — the driver does
+# not report speed/duplex to the kernel (ethtool shows "Speed: Unknown!"),
+# so the bonding driver never transmits LACP PDUs. active-backup requires no
+# switch-side LACP and works correctly with a single port.
+#
+# Run ON pfv-tsys4. Designed for maintenance-window execution.
+#
+# Safety:
+#   - Dry-run by default (--apply to commit)
+#   - Full backup of /etc/network/interfaces
+#   - 5 health checks with automatic rollback on failure
+#   - SSH survives (management on vmbr0/tailscale0, not bond0/datanet)
+#   - Pre-generated rollback script for manual recovery
+#
+# Switch side (core-sw01): NO changes needed. g31 is already a standalone
+# access port in VLAN 1000 — correct for active-backup (no LACP required).
+# g32/ch2 cleanup (dead NIC) is left for separate maintenance.
+#
+# Usage:
+#   bash fix-tsys4-storage-bond.sh             # dry-run (show changes only)
+#   bash fix-tsys4-storage-bond.sh --apply     # commit with auto-rollback
+#
+# Related: Redmine [#394] BUG 4
+###############################################################################
+set -euo pipefail
+
+STORAGE_PEER="10.100.100.6"
+INTERFACES="/etc/network/interfaces"
+TS_SHORT="$(date +%Y%m%d-%H%M%S)"
+BACKUP_DIR="/root/tsys4-bondfix-backup-${TS_SHORT}"
+ROLLBACK="/root/tsys4-bondfix-rollback-${TS_SHORT}.sh"
+ACTION="${1:-dryrun}"
+
+[ "${ACTION}" = "--apply" ] && ACTION="apply" || ACTION="dryrun"
+
+echo "==================================================================="
+echo " fix-tsys4-storage-bond — $(hostname -s)"
+echo " mode: ${ACTION}"
+echo " time: $(date)"
+echo "==================================================================="
+echo ""
+
+# -------------------------------------------------------------------------
+# Pre-flight checks
+# -------------------------------------------------------------------------
+echo "--- Pre-flight checks ---"
+
+# Must be pfv-tsys4
+HOSTNAME_S="$(hostname -s)"
+if [ "${HOSTNAME_S}" != "pfv-tsys4" ]; then
+    echo "FATAL: This script targets pfv-tsys4 (this host: ${HOSTNAME_S})"
+    exit 1
+fi
+echo "  Host: OK (pfv-tsys4)"
+
+# bond0 must exist
+if [ ! -d /sys/class/net/bond0 ]; then
+    echo "FATAL: bond0 not found — no bond to fix"
+    exit 1
+fi
+echo "  bond0: present"
+
+# ifreload must be available
+if ! command -v ifreload >/dev/null 2>&1; then
+    echo "FATAL: ifreload not found (need ifupdown2)"
+    exit 1
+fi
+echo "  ifreload: available"
+
+# SSH must NOT be on bond0/datanet (check incoming route)
+SSH_SRC="$(echo "${SSH_CLIENT:-}" | awk '{print $1}')"
+if [ -n "${SSH_SRC}" ]; then
+    SSH_IFACE="$(ip route get "${SSH_SRC}" 2>/dev/null | grep -oP 'dev \K\S+' || echo "unknown")"
+    echo "  SSH ingress: ${SSH_IFACE}"
+    if echo "${SSH_IFACE}" | grep -qE 'bond0|datanet'; then
+        echo "FATAL: SSH is on storage network — cannot safely reload."
+        echo "       Use physical console (pfv-tsys4) to run this script."
+        exit 1
+    fi
+else
+    echo "  SSH ingress: (local/console session — OK)"
+fi
+echo "  SSH safety: OK (not on storage network)"
+echo ""
+
+# -------------------------------------------------------------------------
+# Show current state
+# -------------------------------------------------------------------------
+echo "--- Current bond0 state ---"
+grep -E "Bonding Mode|Transmit Hash|MII Status|Number of ports" /proc/net/bonding/bond0
+echo ""
+grep -E "Slave Interface|MII Status|Speed" /proc/net/bonding/bond0
+echo ""
+
+echo "--- Current bond0 stanza in /etc/network/interfaces ---"
+awk '/^auto bond0/,/^$/' "${INTERFACES}"
+echo ""
+
+# -------------------------------------------------------------------------
+# Baseline connectivity
+# -------------------------------------------------------------------------
+echo "--- Baseline connectivity ---"
+echo -n "  Ping ${STORAGE_PEER}: "
+if ping -c 1 -W 2 "${STORAGE_PEER}" >/dev/null 2>&1; then
+    echo "OK"
+else
+    echo "UNREACHABLE (baseline already broken — proceed with caution)"
+fi
+echo -n "  NFS server: "
+systemctl is-active nfs-server 2>/dev/null || echo "(not active)"
+echo ""
+
+# -------------------------------------------------------------------------
+# Backup
+# -------------------------------------------------------------------------
+mkdir -p "${BACKUP_DIR}"
+cp -a "${INTERFACES}" "${BACKUP_DIR}/interfaces"
+echo "Backup: ${BACKUP_DIR}/interfaces"
+
+# Generate rollback script
+cat > "${ROLLBACK}" <<ROLLBACKEOF
+#!/bin/bash
+set -euo pipefail
+echo "Rolling back tsys4 bond config..."
+cp -a "${BACKUP_DIR}/interfaces" "${INTERFACES}"
+echo "Restored ${INTERFACES}"
+ifreload -a 2>&1 || systemctl restart networking 2>&1 || true
+sleep 3
+echo "Post-rollback bond0 state:"
+grep -E "Bonding Mode|MII Status|Slave Interface|Speed" /proc/net/bonding/bond0 2>/dev/null
+echo "Rollback complete."
+ROLLBACKEOF
+chmod +x "${ROLLBACK}"
+echo "Rollback: ${ROLLBACK}"
+echo ""
+
+# -------------------------------------------------------------------------
+# Describe the change
+# -------------------------------------------------------------------------
+echo "==================================================================="
+echo " PROPOSED CHANGE"
+echo "==================================================================="
+echo "  1. bond-mode:  802.3ad → active-backup"
+echo "  2. Remove bond-xmit-hash-policy (unused in active-backup)"
+echo "  3. bond-slaves: keep enx8cae4ccda926 (active NIC only)"
+echo ""
+echo "  Rationale: USB NICs cannot do LACP. ethtool reports Speed: Unknown,"
+echo "  so the bonding driver never sends LACP PDUs (verified via tcpdump:"
+echo "  0 LACP PDUs in 65s on both slave and bond master). active-backup"
+echo "  needs no LACP and works with the single working port."
+echo ""
+echo "  Switch: NO changes needed. g31 is standalone access VLAN 1000."
+echo ""
+
+if [ "${ACTION}" != "apply" ]; then
+    echo "==================================================================="
+    echo " DRY RUN — no changes made."
+    echo " Commit:  bash \$0 --apply"
+    echo "==================================================================="
+    exit 0
+fi
+
+# =========================================================================
+# APPLY
+# =========================================================================
+echo "==================================================================="
+echo " APPLYING — auto-rollback on health check failure"
+echo "==================================================================="
+echo ""
+
+# 1. Change bond-mode
+echo "Changing bond-mode 802.3ad → active-backup..."
+sed -i 's/bond-mode 802\.3ad/bond-mode active-backup/' "${INTERFACES}"
+
+# 2. Remove bond-xmit-hash-policy (not used by active-backup)
+echo "Removing bond-xmit-hash-policy..."
+sed -i '/bond-xmit-hash-policy/d' "${INTERFACES}"
+
+# Show updated stanza
+echo ""
+echo "--- Updated bond0 stanza ---"
+awk '/^auto bond0/,/^$/' "${INTERFACES}"
+echo ""
+
+# 3. Reload networking
+echo "--- Reloading networking (ifreload -a) ---"
+echo "SSH safe on vmbr0/tailscale0 (not bond0)..."
+ifreload -a 2>&1 || echo "WARNING: ifreload returned non-zero — checking state..."
+
+# Wait for bond to settle
+echo "Waiting 5s for bond to settle..."
+sleep 5
+
+# =========================================================================
+# HEALTH CHECKS
+# =========================================================================
+echo ""
+echo "==================================================================="
+echo " HEALTH CHECKS (5 tests)"
+echo "==================================================================="
+
+HEALTH_OK=true
+
+# Check 1: bond0 is UP
+echo -n "  [1/5] bond0 MII up: "
+if grep -q "MII Status: up" /proc/net/bonding/bond0 2>/dev/null; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 2: Active slave assigned
+echo -n "  [2/5] Active slave: "
+ACTIVE_SLAVE="$(cat /sys/class/net/bond0/bonding/active_slave 2>/dev/null || echo "")"
+if [ -n "${ACTIVE_SLAVE}" ]; then
+    echo "PASS (${ACTIVE_SLAVE})"
+else
+    echo "FAIL (no active slave)"
+    HEALTH_OK=false
+fi
+
+# Check 3: datanet bridge UP
+echo -n "  [3/5] datanet bridge up: "
+if ip link show datanet 2>/dev/null | grep -q "state UP"; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 4: Storage network IP present on datanet
+echo -n "  [4/5] Storage IP (10.100.100.4): "
+if ip addr show datanet 2>/dev/null | grep -q "10.100.100.4"; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# Check 5: Storage peer reachable
+echo -n "  [5/5] Ping ${STORAGE_PEER}: "
+if ping -c 3 -W 2 "${STORAGE_PEER}" >/dev/null 2>&1; then
+    echo "PASS"
+else
+    echo "FAIL"
+    HEALTH_OK=false
+fi
+
+# =========================================================================
+# COMMIT OR ROLLBACK
+# =========================================================================
+echo ""
+
+if [ "${HEALTH_OK}" = "true" ]; then
+    echo "==================================================================="
+    echo " SUCCESS — all 5 health checks passed"
+    echo "==================================================================="
+    echo ""
+    echo "--- Final bond0 state ---"
+    grep -E "Bonding Mode|MII Status|Slave Interface|Speed|Active" /proc/net/bonding/bond0
+    echo ""
+    echo "Backup:   ${BACKUP_DIR}/interfaces"
+    echo "Rollback: ${ROLLBACK}"
+    echo ""
+    echo "Config change is LIVE but not yet reboot-tested."
+    echo "Verify NFS clients are healthy before next maintenance window."
+    exit 0
+else
+    echo "==================================================================="
+    echo " HEALTH CHECK FAILED — auto-rolling back"
+    echo "==================================================================="
+    bash "${ROLLBACK}"
+    echo ""
+    echo "Auto-rollback complete."
+    echo "Original config restored. Manual investigation needed."
+    exit 1
+fi

proxmox/perf/scripts/perf-matrix.sh

@@ -0,0 +1,242 @@
+#!/bin/bash
+###############################################################################
+# perf-matrix.sh — Any-to-any performance testing across the PFVCluster
+#
+# Tests three network planes:
+#   A. Datanet (VLAN 1000): hypervisor-to-hypervisor over storage network
+#   B. Guest-to-guest: k8s/ultix VMs over datanet (10.100.100.x)
+#   C. Storage I/O: dd read/write to NFS mounts
+#
+# Prerequisites:
+#   - iperf3 installed on all hosts (systemd service: iperf3-server)
+#   - iperf3 installed inside guest VMs
+#   - SSH key + passwordless sudo on all guest VMs (remote.sh; sshd is the
+#     only approved access channel — see AGENTS.md)
+#
+# Usage:
+#   bash perf-matrix.sh                  # run all tests
+#   bash perf-matrix.sh datanet          # host-to-host datanet only
+#   bash perf-matrix.sh guests           # guest-to-guest datanet only
+#   bash perf-matrix.sh storage          # NFS I/O only
+#
+# Environment:
+#   REMOTE_SH   path to tests/remote.sh (auto-detected)
+###############################################################################
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REMOTE_SH="${REMOTE_SH:-$(cd "$SCRIPT_DIR/../../.." && pwd)/tests/remote.sh}"
+LOG_DIR="$(cd "$SCRIPT_DIR/.." && pwd)/returned-logs"
+mkdir -p "$LOG_DIR"
+TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
+
+# --- Host datanet IPs (VLAN 1000) ---
+declare -A DATANET_IP
+DATANET_IP[pfv-tsys1]="10.100.100.1"
+DATANET_IP[pfv-tsys3]="10.100.100.3"
+DATANET_IP[pfv-tsys4]="10.100.100.4"
+DATANET_IP[pfv-tsys5]="10.100.100.5"
+DATANET_IP[pfv-tsys6]="10.100.100.6"
+DATANET_IP[pfv-tsys7]="10.100.100.7"
+DATANET_IP[pfv-tsys9]="10.100.100.9"
+
+HOSTS="pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9"
+
+# --- k8s/ultix VM targets (SSH for control; datanet_ip for iperf traffic) ---
+# Format: prox_host:vmid:datanet_ip:ssh_host:label
+GUEST_TARGETS="
+pfv-tsys1:102:10.100.100.10:pfv-k8s-cnode1:cnode1
+pfv-tsys7:705:10.100.100.11:pfv-k8s-cnode2:cnode2
+pfv-tsys6:603:10.100.100.12:pfv-k8s-cnode3:cnode3
+pfv-tsys3:313:10.100.100.13:pfv-k8s-wnode-tsys3:wnode-tsys3
+pfv-tsys5:500:10.100.100.14:pfv-k8s-wnode-tsys5:wnode-tsys5
+pfv-tsys6:601:10.100.100.15:pfv-k8s-wnode-tsys6:wnode-tsys6
+pfv-tsys7:701:10.100.100.16:pfv-k8s-wnode-tsys7:wnode-tsys7
+pfv-tsys9:905:10.100.100.17:pfv-k8s-wnode-tsys9:wnode-tsys9
+pfv-tsys5:5111:10.100.100.18:ultix-streaming:ultix-streaming
+pfv-tsys5:5112:10.100.100.19:ultix-offstage:ultix-offstage
+"
+
+DURATION="${DURATION:-3}"    # seconds per iperf3 test
+STREAMS="${STREAMS:-4}"      # parallel streams
+
+# ============================================================================
+# Helpers
+# ============================================================================
+start_iperf_servers() {
+    echo "--- Starting iperf3 servers on all hosts ---"
+    for h in $HOSTS; do
+        PROX_HOST="$h" bash "$REMOTE_SH" prox \
+            'systemctl start iperf3-server 2>/dev/null || iperf3 -s -D; echo ok' \
+            >/dev/null 2>&1 &
+    done
+    wait
+    echo "    All servers started."
+}
+
+stop_iperf_servers() {
+    echo "--- Stopping iperf3 servers on all hosts ---"
+    for h in $HOSTS; do
+        PROX_HOST="$h" bash "$REMOTE_SH" prox \
+            'systemctl stop iperf3-server 2>/dev/null; pkill iperf3 2>/dev/null; true' \
+            >/dev/null 2>&1 &
+    done
+    wait
+    echo "    All servers stopped."
+}
+
+# ============================================================================
+# A. Host-to-host datanet matrix
+# ============================================================================
+test_datanet() {
+    echo ""
+    echo "==================================================================="
+    echo "  DATANET (VLAN 1000) — Host-to-Host Bandwidth Matrix"
+    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
+    echo "==================================================================="
+    start_iperf_servers
+
+    local outfile="$LOG_DIR/datanet-host-${TIMESTAMP}.csv"
+    echo "host_from,host_to,mbps" > "$outfile"
+
+    for client in $HOSTS; do
+        for server in $HOSTS; do
+            [ "$client" = "$server" ] && continue
+            local sip="${DATANET_IP[$server]}"
+            local result
+            result=$(PROX_HOST="$client" bash "$REMOTE_SH" prox \
+                "iperf3 -c $sip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
+                | awk '/SUM.*receiver/{printf "%.0f", $6}')
+            if [ -n "$result" ]; then
+                printf "  %-14s → %-14s : %s Mbps\n" "$client" "$server" "$result"
+                echo "$client,$server,$result" >> "$outfile"
+            else
+                printf "  %-14s → %-14s : FAIL\n" "$client" "$server"
+                echo "$client,$server,FAIL" >> "$outfile"
+            fi
+        done
+    done
+
+    stop_iperf_servers
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# B. Guest-to-guest datanet
+# ============================================================================
+test_guests() {
+    echo ""
+    echo "==================================================================="
+    echo "  DATANET (VLAN 1000) — Guest-to-Guest (k8s + ultix VMs)"
+    echo "  ${STREAMS} parallel streams, ${DURATION}s per test"
+    echo "==================================================================="
+
+    # Start iperf3 server on the first guest (cnode1)
+    local server_entry
+    server_entry=$(echo "$GUEST_TARGETS" | head -2 | tail -1)
+    local s_ip s_ssh s_label
+    s_ip=$(echo "$server_entry" | cut -d: -f3)
+    s_ssh=$(echo "$server_entry" | cut -d: -f4)
+    s_label=$(echo "$server_entry" | cut -d: -f5)
+
+    echo "  Starting iperf3 server on $s_label ($s_ip)..."
+    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
+        'pkill iperf3 2>/dev/null; iperf3 -s -D' >/dev/null 2>&1
+    sleep 1
+
+    local outfile="$LOG_DIR/datanet-guest-${TIMESTAMP}.csv"
+    echo "guest_from,guest_to,mbps" > "$outfile"
+
+    while read -r entry; do
+        [ -z "$entry" ] && continue
+        local c_ip c_ssh c_label
+        c_ip=$(echo "$entry" | cut -d: -f3)
+        c_ssh=$(echo "$entry" | cut -d: -f4)
+        c_label=$(echo "$entry" | cut -d: -f5)
+        [ "$c_ip" = "$s_ip" ] && continue
+
+        local result
+        result=$(VM_IP="$c_ssh" bash "$REMOTE_SH" vmroot \
+            "iperf3 -c $s_ip -t $DURATION -P $STREAMS -f m 2>&1" </dev/null \
+            | awk '/SUM.*receiver/{printf "%.0f", $6}')
+        if [ -n "$result" ]; then
+            printf "  %-18s → %-18s : %s Mbps\n" "$c_label" "$s_label" "$result"
+            echo "$c_label,$s_label,$result" >> "$outfile"
+        else
+            printf "  %-18s → %-18s : FAIL\n" "$c_label" "$s_label"
+            echo "$c_label,$s_label,FAIL" >> "$outfile"
+        fi
+    done <<< "$GUEST_TARGETS"
+
+    # Cleanup
+    VM_IP="$s_ssh" bash "$REMOTE_SH" vmroot \
+        'pkill iperf3' >/dev/null 2>&1
+
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# C. Storage I/O (NFS read/write)
+# ============================================================================
+test_storage() {
+    echo ""
+    echo "==================================================================="
+    echo "  STORAGE I/O — NFS Read/Write (100MB dd)"
+    echo "==================================================================="
+
+    local outfile="$LOG_DIR/storage-io-${TIMESTAMP}.csv"
+    echo "host,mount,write_mbps,read_mbps" > "$outfile"
+
+    local ddscript="/tmp/perf-dd-$$.sh"
+    cat > "$ddscript" <<'DDSCRIPT'
+#!/bin/bash
+nfs=$(mount | awk '/type nfs/{print $3}' | grep -v proc)
+for m in $nfs; do
+    tf="$m/.perf-$$"
+    w=$(dd if=/dev/zero of="$tf" bs=1M count=100 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
+    r=$(dd if="$tf" of=/dev/null bs=1M 2>&1 | awk '/copied/{printf "%.0f", 100/($8+0.001)}')
+    rm -f "$tf" 2>/dev/null
+    echo "$m write=${w:-FAIL}MB/s read=${r:-N/A}MB/s"
+done
+DDSCRIPT
+
+    for h in $HOSTS; do
+        echo ""
+        echo "  --- $h ---"
+        PROX_HOST="$h" bash "$REMOTE_SH" prox-file "$ddscript" 2>&1 | while read -r line; do
+            [ -n "$line" ] && echo "    $line"
+        done
+    done
+    rm -f "$ddscript"
+
+    echo ""
+    echo "  Results saved: $outfile"
+}
+
+# ============================================================================
+# Main
+# ============================================================================
+MODE="${1:-all}"
+
+case "$MODE" in
+    datanet|a)  test_datanet ;;
+    guests|b)   test_guests ;;
+    storage|c)  test_storage ;;
+    all|"")     test_datanet; test_guests; test_storage ;;
+    *)
+        echo "Usage: $0 [datanet|guests|storage|all]"
+        echo ""
+        echo "  datanet  — host-to-host bandwidth matrix over VLAN 1000"
+        echo "  guests   — guest-to-guest (k8s/ultix VMs over VLAN 1000)"
+        echo "  storage  — NFS read/write I/O"
+        echo "  all      — run all three (default)"
+        exit 1
+        ;;
+esac
+
+echo ""
+echo "==================================================================="
+echo "  Perf testing complete. Logs in: $LOG_DIR/"
+echo "==================================================================="

proxmox/perf/scripts/probe-vm-dns.sh

@@ -13,7 +13,7 @@ for vmid in $(qm list 2>/dev/null | awk 'NR>1 && $4=="running" {print $1}'); do
   net0=$(qm config "$vmid" 2>/dev/null | awk '/^net0:/{print $0}')
   echo ""
   echo "--- VMID $vmid: $name ---"
-  echo "  net0: $(echo "$net0" | sed 's/net0: //')"
+  echo "  net0: ${net0//net0: /}"
 
   # Try to get IP via guest-agent
   if qm config "$vmid" 2>/dev/null | grep -q 'agent:.*enabled=1\|^agent: 1'; then

questions-v1.md

@@ -0,0 +1,61 @@
+# questions-v1.md
+
+> Git-tracked question log. The agent writes; the human reviews/edits inline.
+> Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine.
+> See BASELINE-PROMPT.md §9.
+
+## Open questions
+
+### Q1. Git remote for meta?
+- **Context:** meta is now a git repo (locally) but has no remote configured. The auto-commit+push policy (baseline §4) can't complete without one.
+- **Options:** (a) new Gitea repo under reachableceo; (b) nest under an existing repo; (c) keep local-only for now.
+- **Question:** Where should meta push?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+Go with option a. The tea command is setup on this workstation (and on ultix-offstage). I guess, also capture that the tea command (and docker login) are setup on my workstations, so that in the future, projects know they can use tea to setup a repo. Also, i want this to be TSYS wide, so it should go under the TSYSGroupCorporate organization. Call the repo: TSYSGroupAIOS . Make it a template repository.
+
+### Q2. The bin/ wrapper gap (redmine-cli / discourse-cli) — RESOLVED (discourse)
+- **Context:** PFVCluster's operational.md and AGENTS.md referenced `~/daytoday/redmine/bin/redmine` and `~/daytoday/discourse/bin/discourse` as the entrypoints — thin shortcut wrappers around the real CLI containers. The actual CLI source lived in `~/projects/KNEL-AIMiddleware/{redmine,discourse}-cli/`.
+- **Question:** Are the `bin/` wrappers something that should exist, or is the documentation aspirational? Should the template reference these CLIs at all, or stay tool-agnostic?
+- **Answer:** Reference the real container invocation (full path/container name/invoke notes); no duplicate code via shortcut wrappers.
+- **Decision:** No `bin/` wrappers. Invoke the real container with `docker run`. Both CLI sources have been merged into `~/projects/KNEL-AIMiddleware/tooling-cli/{discourse,redmine}/` and PFVCluster's AGENTS.md now invokes them via `docker run --env-file ~/.creds/{discourse,redmine}.env`. The old `~/daytoday/{discourse,redmine}` workspaces and the `{discourse,redmine}-cli/` subdirs were removed.
+- **Synthesized to:** PFVCluster `AGENTS.md` §CLI access; `tooling-cli/{discourse,redmine}/`
+
+### Q3. Should the template ship the Discourse pointer-header pattern?
+- **Context:** PFVCluster migrated 36 in-repo `.md` files to 10-line pointer stubs citing `https://community.turnsys.com/t/<N>`. The template currently has `scripts/garden.sh` that *warns* about oversized non-Discourse `.md`, but doesn't enforce the pointer-header format.
+- **Options:** (a) keep it advisory (garden.sh warn only); (b) add an opt-in check-rule that fails if a tracked `.md` lacks a Discourse URL (excluding AGENTS.md/STATUS.md/etc.); (c) leave it project-local — infra projects want it, personal/business projects don't.
+- **Question:** Which option, and is the assumption in (c) right?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+All projects need it. Discourse/redmine is MANDATORY. No exceptions. What is project specific is which categories to use, and maybe some tagging/topic guidelines etc.
+
+### Q4. Sub-agent nudge hook — wanted?
+- **Context:** A sub-agent proposed a non-blocking Crush hook (`hooks/nudge-subagent.sh`) that emits a stderr reminder after the Nth sequential file read, nudging toward dispatching a sub-agent. Mirrors football's "never read 10+ files sequentially" rule.
+- **Options:** (a) add it (non-blocking, advisory); (b) leave sub-agent use as prose policy only.
+- **Question:** Worth adding, or too noisy?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+Preseving tokens/quota burn is a HUGE priority. It lets me and you do far more work for much longer. Also, I want to move away from harness specific hooks. Git hooks/strong AGENTS.md protocols are strongly preferred. Ill be shifting away from crush over next few weeks to using OpenWebUi/Hermes and a whole swarm of agents with reporting/working relationships etc etc. So anything that is harness specific, get rid of it and make it portable.
+
+
+### Q5. JOURNAL.md vs Discourse audit-log for infra projects
+- **Context:** The template ships `docs/JOURNAL.md` as the append-only decision log. But PFVCluster (the most mature infra project) has NO JOURNAL.md — it uses Discourse topic #298 as the audit log and Redmine for work tracking. PATTERNS.md §5 noted this divergence.
+- **Question:** Should the template keep JOURNAL.md as the default, with infra projects swapping it for the Discourse-audit-log pattern? Or drop JOURNAL.md entirely in favor of "Discourse is the SoR"?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+No more JOURNAL.md . Redmine is the system of record. JOURNAL.md was a hack I was using until redmine integration was in place. And, yes, discourse can also be used as well. Its a bit of a tricky decision, what should go to redmine vs discourse. I usually keep working notes/evolving status etc in Redmine and then synthesize to Discourse. But thats me as a lowly human :) You figure it out as you go and per project.
+
+### Q6. RackRental split-network design on pfv-tsys6/7 [#433]
+- **Context:** TSYS6 and TSYS7 need trunk ports for RackRental. Servers will run a "split network configuration — half on the non-RackRental switch/router side, half on the RackRental switch/router side." RackRental router prevents network loops.
+- **Question:** Is the split per-NIC (one physical NIC to normal network, one to RackRental) or per-VM (some VMs bridge to RackRental, some to normal)? Both TSYS6/7 currently have bond0 with 2 slaves for storage network. How should the bonds be reconfigured?
+- **Answer:** Keep the storage bonds. We need that bandwidth. I believe tsys6,7 have six ethernet ports each? Two on a riser card, and a four port add-on? Or maybe its four total? If four total, the 2 for storage, one for mgmt non rackrental, one trunked to rackrental would be how I want to go.
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —

questions-v2.md

@@ -0,0 +1,53 @@
+# questions-v2.md
+
+> Git-tracked question log. The agent writes; the human reviews/edits inline.
+> Version up when a round of answers lands. Synthesize resolved Q&A to Discourse/Redmine.
+> See BASELINE-PROMPT.md §9.
+
+## Resolved (from v1)
+
+### Q1. Git remote for meta? — RESOLVED
+- **Decision:** New Gitea repo under TSYSGroupCorporate org: `TSYSGroupAIOS`. Template repo. Created and pushed.
+- **Synthesized to:** Meta repo created, `tea` + `docker login` noted as available on workstations.
+
+### Q2. The bin/ wrapper gap — RESOLVED
+- **Decision:** No bin/ wrappers. Invoke real containers with `docker run --env-file`.
+- **Synthesized to:** PFVCluster AGENTS.md §CLI access; meta AGENTS.md §CLI invocation.
+
+### Q3. Discourse pointer-header pattern — RESOLVED
+- **Decision:** Mandatory for ALL projects. No exceptions. Discourse/Redmine required.
+- **Synthesized to:** Meta AGENTS.md, check-rules.sh enforces.
+
+### Q4. Sub-agent nudge hook — RESOLVED
+- **Decision:** No harness-specific hooks. Move to portable git hooks + AGENTS.md protocols. User shifting away from Crush to OpenWebUI/Hermes.
+- **Synthesized to:** All hooks ported to git hooks. No Crush-specific dependencies.
+
+### Q5. JOURNAL.md vs Discourse — RESOLVED
+- **Decision:** No more JOURNAL.md. Redmine is SoR for work. Discourse for synthesized knowledge.
+- **Synthesized to:** Meta AGENTS.md, PFVCluster AGENTS.md.
+
+### Q6. RackRental split-network design — RESOLVED
+- **Decision:** Keep storage bonds. If 4 NICs total: 2 storage, 1 mgmt non-RackRental, 1 trunked to RackRental VLAN 2.
+- **Synthesized to:** #433 (trunk ports), #432 (VLAN config).
+
+## Open questions
+
+### Q7. Close #167 (Small UPS Units Re-build)?
+- **Context:** #167 is an old (Dec 2024) vague ticket about rebuilding garage UPS units. It's been superseded by #372 (Tripp Lite physical reseat) and #439 (UPS data integration). Dates are stale (Aug 2025).
+- **Question:** Should we close #167 as superseded?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+### Q8. Friday onsite work order?
+- **Context:** Friday 2026-08-14 onsite batch. Tickets: #341 (TEMPer deploy), #372 (Tripp Lite UPS reseat), #431 (Cisco fan replacement), #414 (tsys5 nic2 cable swap), #374 (PDU outlet renaming), #439 (UPS integration).
+- **Question:** What order do you want to tackle these? Cisco fan (#431) blocks #432/#433/#434/#436 — should it go first? TEMPer is the "fun" project — save for last?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —
+
+### Q9. Session-start gate: relax for continuation sessions?
+- **Context:** You mentioned last session wanting to discuss relaxing the ticket gate for handoff/continuation sessions. The scope-alignment gate now handles this (presents handoff priorities, waits for user to name target). Is this sufficient, or do you want further changes?
+- **Answer:** _(human)_
+- **Decision:** _(human/agent)_
+- **Synthesized to:** —

scripts/check-rules.sh

@@ -0,0 +1,266 @@
+#!/usr/bin/env bash
+# check-rules.sh — project rule audit engine.
+#
+# Usage:
+#   bash scripts/check-rules.sh           # full audit (verbose, includes slow checks)
+#   bash scripts/check-rules.sh --fast    # fast audit (quiet, skips slow checks) — for pre-commit
+#   bash scripts/check-rules.sh --quiet   # full audit, only prints failures
+#
+# Exit code: 0 = all rules pass (warnings are non-fatal), 1 = one or more FAILED.
+#
+# This is a generalized version of the rules engine proven in the
+# RCEO-PersonalAssistant project. Add project-specific checks by appending
+# `check "<desc>" "<pass|warn|fail>"` calls below.
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+REPO_ROOT="$(repo_root)"
+cd "$REPO_ROOT"
+
+# --- argument parsing ---
+RULE_FAST=false
+RULE_VERBOSE=true
+for arg in "$@"; do
+    case "$arg" in
+        --fast)  RULE_FAST=true; RULE_VERBOSE=false ;;
+        --quiet) RULE_VERBOSE=false ;;
+        *) die "check-rules.sh: unknown argument '$arg'" ;;
+    esac
+done
+export RULE_FAST RULE_VERBOSE
+
+init_counters
+$RULE_VERBOSE && echo "=== Project Rule Audit ==="
+
+TODAY="$(date +%Y-%m-%d)"
+
+# ----------------------------------------------------------------------------
+# 1. Shellcheck — every .sh/.bash must pass (zero warnings, incl. info-level).
+#    Runs in Docker so the host stays clean (no native shellcheck required).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Shell scripts (shellcheck)"
+mapfile -d '' SH_FILES < <(find . -path ./.git -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -path ./node_modules -prune -o \( -name '*.sh' -o -name '*.bash' \) -print0 2>/dev/null)
+if [ "${#SH_FILES[@]}" -gt 0 ]; then
+    if have shellcheck; then
+        if shellcheck "${SH_FILES[@]}" >/dev/null 2>&1; then
+            check "All shell scripts pass shellcheck (host)" "pass"
+        else
+            check "shellcheck reports violations — run: shellcheck <file>" "fail"
+        fi
+    elif have docker; then
+        MNT_FILES=()
+        for f in "${SH_FILES[@]}"; do MNT_FILES+=("/mnt/${f#./}"); done
+        if docker run --rm -v "$REPO_ROOT:/mnt" koalaman/shellcheck:stable "${MNT_FILES[@]}" >/dev/null 2>&1; then
+            check "All shell scripts pass shellcheck (docker)" "pass"
+        else
+            check "shellcheck (docker) reports violations" "fail"
+        fi
+    else
+        check "No shellcheck or docker available to lint scripts" "warn"
+    fi
+else
+    check "No shell scripts to lint" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 2. Docker image pinning — no ':latest' tags in compose or Dockerfiles.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Docker image pinning"
+if grep -rqE '(image:|FROM).*:latest' --include='docker-compose*.y*ml' --include='Dockerfile*' . 2>/dev/null; then
+    check "No ':latest' image tags (pin everything)" "fail"
+else
+    check "No ':latest' image tags" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 2b. Container naming — every service in a docker-compose file MUST set an
+#     explicit container_name (never rely on Docker's default <dir>_<n>).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Container naming"
+COMPOSE_FILES="$(find . -path ./.git -prune -o \( -name 'docker-compose*.yml' -o -name 'docker-compose*.yaml' -o -name 'compose.yml' -o -name 'compose.yaml' \) -print 2>/dev/null || true)"
+if [ -n "$COMPOSE_FILES" ]; then
+    BAD=0
+    while IFS= read -r cf; do
+        [ -n "$cf" ] || continue
+        # Count top-level service keys (2-space indent under services:) and
+        # compare against the number of container_name: declarations.
+        svc_count=$(awk '/^services:/{f=1;next} f&&/^[^[:space:]]/{f=0} f&&/^[[:space:]]{2}[[:alnum:]_-]+:[[:space:]]*$/{c++} END{print c+0}' "$cf")
+        cn_count=$(grep -cE '^[[:space:]]*container_name:' "$cf" 2>/dev/null || echo 0)
+        if [ "${svc_count:-0}" -gt 0 ] && [ "$cn_count" -lt "$svc_count" ]; then
+            BAD=$((BAD + 1))
+        fi
+    done <<EOF
+$COMPOSE_FILES
+EOF
+    if [ "$BAD" -eq 0 ]; then
+        check "All compose services set container_name" "pass"
+    else
+        check "$BAD compose file(s) with services missing container_name" "fail"
+    fi
+else
+    check "No compose files (container-name check skipped)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 3. Required-files manifest — the files every project using this template owns.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Required files"
+REQUIRED_FILES="AGENTS.md STATUS.md questions-v1.md .env.example scripts/check-rules.sh scripts/setup-hooks.sh"
+REQUIRED_FILES="$REQUIRED_FILES ${PROJECT_REQUIRED_FILES:-}"
+for f in $REQUIRED_FILES; do
+    if [ -f "$f" ]; then check "$f exists" "pass"; else check "$f MISSING" "fail"; fi
+done
+
+# ----------------------------------------------------------------------------
+# 4. Doc freshness — STATUS.md touched today.
+#    Warning (not failure): staleness is a signal, not a break.
+#    Redmine is the system of record for work; Discourse for docs. STATUS.md is
+#    a scratchpad only — see BASELINE-PROMPT.md §3, §8.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Doc freshness"
+if [ -f STATUS.md ]; then
+    STATUS_DATE="$(grep -oE 'Last updated: [0-9]{4}-[0-9]{2}-[0-9]{2}' STATUS.md | grep -oE '[0-9]{4}-[0-9]{2}-[0-9]{2}' || echo unknown)"
+    if [ "$STATUS_DATE" = "$TODAY" ]; then
+        check "STATUS.md updated today ($STATUS_DATE)" "pass"
+    else
+        check "STATUS.md is stale (last: $STATUS_DATE, today: $TODAY) — update it" "warn"
+    fi
+else
+    check "STATUS.md MISSING" "fail"
+fi
+
+# ----------------------------------------------------------------------------
+# 4b. Discourse pointer-header policy (MANDATORY).
+#     Discourse is the system of record for documentation. In-repo .md files are
+#     stubs that point to a Discourse topic URL. Operational files exempt.
+#     Override exemptions via PROJECT_DOC_EXEMPT (space-separated globs of
+#     basenames) and the Discourse host via PROJECT_DISCOURSE_HOST.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Discourse pointer-header"
+DISCOURSE_HOST="${PROJECT_DISCOURSE_HOST:-community.turnsys.com}"
+DOC_EXEMPT="${PROJECT_DOC_EXEMPT:-AGENTS.md STATUS.md WORKING.md README.md ADOPTING.md LICENSE .env.example questions-v*.md BASELINE-PROMPT.md PATTERNS.md}"
+POINTER_MISSING=0
+while IFS= read -r -d '' f; do
+    base="$(basename "$f")"
+    exempt=false
+    for pat in $DOC_EXEMPT; do
+        # shellcheck disable=SC2254
+        case "$base" in $pat) exempt=true; break ;; esac
+    done
+    [ "$exempt" = true ] && continue
+    if ! grep -qF "$DISCOURSE_HOST" "$f" 2>/dev/null; then
+        if [ "$POINTER_MISSING" -eq 0 ]; then
+            $RULE_VERBOSE && printf '  %s\n' "Missing $DISCOURSE_HOST URL in:"
+        fi
+        POINTER_MISSING=$((POINTER_MISSING + 1))
+        $RULE_VERBOSE && printf '    %s\n' "$f"
+    fi
+done < <(find . -path ./.git -prune -o -path ./.crush -prune -o -path ./.tmp -prune -o -path ./vendor -prune -o -path ./archive -prune -o -name '*.md' -print0 2>/dev/null)
+if [ "$POINTER_MISSING" -eq 0 ]; then
+    check "All non-exempt .md cite Discourse ($DISCOURSE_HOST)" "pass"
+else
+    check "$POINTER_MISSING .md file(s) missing Discourse pointer (see BASELINE-PROMPT.md §3)" "fail"
+fi
+
+# ----------------------------------------------------------------------------
+# 5. Git state — uncommitted changes are a warning (the pre-push hook hardens
+#    this where it matters).
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Git state"
+if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
+    if git diff --quiet && git diff --cached --quiet; then
+        check "Working tree clean" "pass"
+    else
+        check "Uncommitted changes present" "warn"
+    fi
+else
+    check "Not a git repo (git checks skipped)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 6. Hooks installed — self-check that git hooks were set up.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Git hooks"
+if [ -f .git/hooks/pre-commit ]; then
+    check "pre-commit hook installed" "pass"
+else
+    check "pre-commit NOT installed (run: bash scripts/setup-hooks.sh)" "warn"
+fi
+if [ -f .git/hooks/pre-push ]; then
+    check "pre-push hook installed" "pass"
+else
+    check "pre-push NOT installed (run: bash scripts/setup-hooks.sh)" "warn"
+fi
+
+# ----------------------------------------------------------------------------
+# 7. WORKING.md completion — no unchecked tasks may remain at commit time.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Task completion"
+if [ -f WORKING.md ]; then
+    UNCHECKED="$(grep -cF -- '- [ ]' WORKING.md || true)"
+    if [ "$UNCHECKED" -eq 0 ]; then
+        check "WORKING.md has no unchecked tasks" "pass"
+    else
+        check "WORKING.md has ${UNCHECKED} unchecked task(s) — finish them before committing" "fail"
+    fi
+else
+    check "WORKING.md absent (no active task tracker)" "pass"
+fi
+
+# ----------------------------------------------------------------------------
+# 8. CNW markers — empty `CNW:` markers flag unresolved questions for the human.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Unresolved questions"
+EMPTY_CNW="$(grep -rn 'CNW:$' . --include='*.md' 2>/dev/null | head -20 || true)"
+if [ -z "$EMPTY_CNW" ]; then
+    check "No empty CNW: markers (unresolved questions)" "pass"
+else
+    CNW_COUNT="$(printf '%s\n' "$EMPTY_CNW" | grep -c . || true)"
+    check "${CNW_COUNT} unresolved CNW: marker(s) — needs user input" "warn"
+fi
+
+# ----------------------------------------------------------------------------
+# 9. Hygiene — merge-conflict markers and trailing whitespace must never land.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "File hygiene"
+if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
+    CONFLICT="$(git diff --cached --name-only --diff-filter=ACM 2>/dev/null | xargs -r grep -lE '^(<<<<<<<|=======|>>>>>>>)' 2>/dev/null || true)"
+    if [ -z "$CONFLICT" ]; then check "No merge-conflict markers staged" "pass"; else check "Merge-conflict markers staged: $CONFLICT" "fail"; fi
+fi
+
+# ----------------------------------------------------------------------------
+# 10. (slow, skipped in --fast) Project test suite via scripts/test.sh.
+# ----------------------------------------------------------------------------
+if [ "$RULE_FAST" = false ] && [ -x scripts/test.sh ]; then
+    $RULE_VERBOSE && log_step "Test suite (scripts/test.sh)"
+    if bash scripts/test.sh >/dev/null 2>&1; then
+        check "scripts/test.sh passes" "pass"
+    else
+        check "scripts/test.sh FAILS" "fail"
+    fi
+fi
+
+# ----------------------------------------------------------------------------
+# 11. Access-channel policy — qemu guest-agent must NEVER be used as an
+#     execution or key-delivery channel. SSH (sshd) is the ONLY approved
+#     remote access path; all commands must be logged through standard
+#     auth/audit infrastructure. ITAR/CMMC environment — non-negotiable.
+#     Allowed: installing/checking qemu-guest-agent for Proxmox state
+#     visibility. Forbidden: `qm guest exec` + any `vm-guest` wrapper.
+#     Scans CODE only (not .md) so docs may describe the ban.
+# ----------------------------------------------------------------------------
+$RULE_VERBOSE && log_step "Access-channel policy (no guest-agent exec)"
+GA_HITS="$(grep -rnE 'qm guest exec|vm-guest|_vm_guest' \
+    --include='*.sh' --include='*.bash' --include='*.py' \
+    . 2>/dev/null | grep -vE 'scripts/check-rules\.sh|/archive/' || true)"
+if [ -z "$GA_HITS" ]; then
+    check "No guest-agent exec / vm-guest access patterns in code" "pass"
+else
+    GA_COUNT="$(printf '%s\n' "$GA_HITS" | grep -c . || true)"
+    $RULE_VERBOSE && printf '%s\n' "$GA_HITS" | sed 's/^/    /'
+    check "${GA_COUNT} guest-agent exec / vm-guest reference(s) — SSH-only access policy (AGENTS.md)" "fail"
+fi
+
+print_summary_and_exit

scripts/docker-run.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# docker-run.sh — canonical ephemeral-container wrapper.
+#
+# Keeps the host clean: every build/test/generation runs inside a pinned image.
+# Ensures output files are owned by the invoking user (not root).
+#
+# Usage:
+#   docker-run.sh <image> [command...]
+#       Runs <command> in <image> with the repo mounted at /data, cwd /data.
+#       With no command, drops into the image's default entrypoint.
+#   docker-run.sh --shell <image>
+#       Interactive shell inside the container (for debugging).
+#
+# Examples:
+#   docker-run.sh python:3.12-slim python3 -m pytest
+#   docker-run.sh pandoc/extra report.md -o report.pdf
+#   docker-run.sh --shell node:20
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+
+SHELL_MODE=false
+case "${1:-}" in
+    --shell) SHELL_MODE=true; shift ;;
+    -h|--help)
+        sed -n '2,18p' "$0"; exit 0 ;;
+esac
+
+[ "$#" -ge 1 ] || { sed -n '2,18p' "$0"; exit 1; }
+
+if [ "$SHELL_MODE" = true ]; then
+    # ${SHELL:-sh} must expand inside the container, not in this outer shell.
+    # shellcheck disable=SC2016
+    docker_run "$1" sh -c 'exec "${SHELL:-sh}"'
+else
+    docker_run "$@"
+fi

scripts/garden.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# garden.sh — the gardening loop.
+#
+# Reports doc sprawl and files that violate the "Discourse is the system of
+# record for documentation; gitea .md files are stubs" policy. Run via
+# `bash scripts/garden.sh`. Findings are WARNINGS (advisory); fix them at a natural break.
+#
+# What it checks:
+#   1. Markdown sprawl: count of .md files per directory (top-10 by count).
+#   2. Oversized .md files (default >300 lines) that don't cite a Discourse URL
+#      — candidates to migrate to Discourse, leaving a stub.
+#   3. .md files with no Discourse link at all (informational; exempt: the
+#      operational files in EXEMPT_FILES).
+set -euo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+# shellcheck disable=SC1091
+source "$HERE/lib/common.sh"
+REPO_ROOT="$(repo_root)"
+cd "$REPO_ROOT"
+
+SIZE_LIMIT="${GARDEN_MD_LINE_LIMIT:-300}"
+# Operational files that legitimately live next to code, not in Discourse.
+EXEMPT_FILES="${GARDEN_EXEMPT:-AGENTS.md STATUS.md WORKING.md questions-v.*.md PATTERNS.md BASELINE-PROMPT.md README.md}"
+
+log_step "Gardening report for $REPO_ROOT"
+
+# --- 1. sprawl by directory -------------------------------------------------
+log_info "Markdown file count by directory (top 10):"
+find . -path ./.git -prune -o -name '*.md' -print 2>/dev/null \
+    | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn | head -10 | sed 's/^/    /'
+
+# --- 2. oversized .md without a Discourse link ------------------------------
+log_info "Oversized .md (>${SIZE_LIMIT} lines) lacking a Discourse URL — migrate candidates:"
+OVERSIZED=0
+while IFS= read -r -d '' f; do
+    # skip exempt files (glob match against basename and relative path)
+    exempt=false
+    base=$(basename "$f")
+    rel=${f#./}
+    for pat in $EXEMPT_FILES; do
+        # shellcheck disable=SC2254  # glob match is intentional
+        case "$base" in $pat) exempt=true; break ;; esac
+        # shellcheck disable=SC2254
+        case "$rel" in $pat) exempt=true; break ;; esac
+    done
+    [ "$exempt" = true ] && continue
+    lines=$(wc -l < "$f" 2>/dev/null || echo 0)
+    if [ "$lines" -gt "$SIZE_LIMIT" ]; then
+        if ! grep -qiE 'community\.turnsys\.com|discourse' "$f" 2>/dev/null; then
+            printf '    %-60s %s lines\n' "$f" "$lines"
+            OVERSIZED=$((OVERSIZED + 1))
+        fi
+    fi
+done < <(find . -path ./.git -prune -o -name '*.md' -print0 2>/dev/null)
+[ "$OVERSIZED" -eq 0 ] && echo "    (none)"
+
+# --- 3. summary -------------------------------------------------------------
+log_step "Gardening summary"
+echo "    Oversized non-Discourse .md files: $OVERSIZED"
+if [ "$OVERSIZED" -eq 0 ]; then
+    log_ok "no migration candidates"
+else
+    log_warn "$OVERSIZED file(s) to migrate to Discourse"
+fi

scripts/lib/common.sh

@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# lib/common.sh — shared helpers for shell scripts and hooks in this repo.
+#
+# Source it from any script:
+#   #!/usr/bin/env bash
+#   set -euo pipefail
+#   HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+#   # shellcheck source=lib/common.sh
+#   source "$HERE/lib/common.sh"   # or the appropriate relative path
+#
+# This library exists to drive a known cross-project inconsistency to zero:
+# every repo used to re-paste the ANSI color block, redefine log_* helpers,
+# pick one of three incompatible shebangs, and roll its own docker wrapper.
+# Import this once instead.
+
+# Do NOT set -euo pipefail here unconditionally — some callers (git hooks)
+# source this file and rely on controlling their own shell options. We only
+# guarantee the functions below are defined.
+
+###############################################################################
+# Config — override via environment before sourcing if needed
+###############################################################################
+: "${TEMPLATE_ROOT:=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
+export TEMPLATE_ROOT
+
+###############################################################################
+# ANSI colors (defined once, used everywhere)
+###############################################################################
+if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then
+    RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
+    BLUE='\033[0;34m'; BOLD='\033[1m'; NC='\033[0m'
+else
+    RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; NC=''
+fi
+export RED GREEN YELLOW BLUE BOLD NC
+
+###############################################################################
+# Logging
+###############################################################################
+log_info()  { printf "${BLUE}›${NC} %s\n" "$*"; }
+log_ok()    { printf "${GREEN}✓${NC} %s\n" "$*"; }
+log_warn()  { printf "${YELLOW}⚠${NC} %s\n" "$*" >&2; }
+log_error() { printf "${RED}✗${NC} %s\n" "$*" >&2; }
+log_step()  { printf "\n${BOLD}== %s ==${NC}\n" "$*"; }
+
+die() { log_error "$*"; exit 1; }
+
+###############################################################################
+# Predicates
+###############################################################################
+# have <cmd> — return 0 if <cmd> is on PATH
+have() { command -v "$1" >/dev/null 2>&1; }
+
+###############################################################################
+# Path helpers
+###############################################################################
+repo_root() {
+    # Prefer git's notion of the repo root, fall back to $TEMPLATE_ROOT, then pwd.
+    if git rev-parse --show-toplevel >/dev/null 2>&1; then
+        git rev-parse --show-toplevel
+    else
+        printf '%s\n' "${TEMPLATE_ROOT:-$(pwd)}"
+    fi
+}
+
+###############################################################################
+# Privilege helpers
+###############################################################################
+# as_root — run the remaining args as root via sudo, or directly if already root.
+as_root() {
+    if [ "$(id -u)" -eq 0 ]; then "$@"; else sudo "$@"; fi
+}
+
+###############################################################################
+# Docker wrapper
+###############################################################################
+# docker_run <image> <args...>
+# Ephemeral container, host-uid ownership, repo mounted at /data, cwd /data.
+# Drives the "host stays clean; everything runs in containers" policy and
+# ensures output files are owned by the invoking user, not root.
+docker_run() {
+    [ "$#" -ge 1 ] || die "docker_run: image required"
+    local image="$1"; shift
+    have docker || die "docker not found on PATH"
+    local root
+    root="$(repo_root)"
+    docker run --rm \
+        --user "$(id -u):$(id -g)" \
+        -e HOME=/tmp \
+        -v "$root:/data" \
+        -w /data \
+        "$image" "$@"
+}
+
+###############################################################################
+# Rule-audit accumulator (used by scripts/check-rules.sh)
+# Globals read/written: RULE_PASS RULE_WARN RULE_FAIL
+###############################################################################
+init_counters() { RULE_PASS=0; RULE_WARN=0; RULE_FAIL=0; }
+
+# check <description> <pass|warn|fail>
+check() {
+    local desc="$1" result="$2"
+    case "$result" in
+        pass)
+            RULE_PASS=$((RULE_PASS + 1))
+            if [ "${RULE_VERBOSE:-true}" = true ]; then printf "  ${GREEN}PASS${NC}  %s\n" "$desc"; fi
+            ;;
+        warn)
+            RULE_WARN=$((RULE_WARN + 1))
+            if [ "${RULE_VERBOSE:-true}" = true ]; then printf "  ${YELLOW}WARN${NC}  %s\n" "$desc"; fi
+            ;;
+        fail)
+            RULE_FAIL=$((RULE_FAIL + 1))
+            printf "  ${RED}FAIL${NC}  %s\n" "$desc"
+            ;;
+        *)
+            die "check(): invalid result '$result' (use pass|warn|fail)"
+            ;;
+    esac
+}
+
+# print_summary_and_exit
+print_summary_and_exit() {
+    if [ "${RULE_VERBOSE:-true}" = true ]; then
+        printf "\n=== Summary ===\n  PASS: %s\n  WARN: %s\n  FAIL: %s\n\n" \
+            "$RULE_PASS" "$RULE_WARN" "$RULE_FAIL"
+    fi
+    if [ "$RULE_FAIL" -gt 0 ]; then
+        if [ "${RULE_VERBOSE:-true}" = true ]; then
+            printf "AUDIT FAILED — %s rule(s) violated.\n" "$RULE_FAIL"
+        fi
+        exit 1
+    fi
+    if [ "${RULE_VERBOSE:-true}" = true ]; then printf "AUDIT PASSED.\n"; fi
+    exit 0
+}

scripts/pre-commit

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# pre-commit — fast rule audit (< 1s typical).
+# Hot-path bypass: commits that ONLY touch STATUS.md / WORKING.md skip the
+# audit so frequent status/task commits stay frictionless.
+set -euo pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+cd "$REPO_ROOT"
+
+CHANGED="$(git diff --cached --name-only)"
+HOT_PATHS="$(printf '%s\n' "$CHANGED" | grep -vE '^(STATUS.md|WORKING.md)$' || true)"
+
+if [ -z "$HOT_PATHS" ]; then
+    echo "hot-path files only (STATUS/WORKING) — skipping rule audit"
+    exit 0
+fi
+
+if ! bash scripts/check-rules.sh --fast; then
+    echo ""
+    echo "pre-commit audit FAILED. Fix the violations above before committing."
+    echo "Full audit:  bash scripts/check-rules.sh"
+    echo "Bypass:      git commit --no-verify   (emergencies only)"
+    exit 1
+fi
+exit 0