fix(powerman): use -h flag for remote server in query script

Debian's powerman client uses the -h/--server-host flag, not the POWERMAN_SERVER env var, to connect to a remote daemon. Update query-remote.sh to pass -h explicitly. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
feat(powerman): add remote query script for workstation PDU access
2026-07-28 19:48:02 -05:00 · 2026-07-28 19:47:01 -05:00 · 2026-07-28 19:45:17 -05:00 · 2026-07-28 19:44:02 -05:00 · 2026-07-28 18:49:39 -05:00 · 2026-07-28 18:22:47 -05:00
244 changed files with 127806 additions and 380 deletions
@@ -1,8 +1,27 @@
 # Crush internal state
 .crush/
 # OS/editor
 .DS_Store
 *.swp
 *.swo
 *~
 \#*\#
 .\#*
 # Python bytecode
 __pycache__/
 *.pyc
 # Temporary files
 *.tmp
 *.bak
 *.log.tmp
 # LOGFILENAME artifacts: the framework (Logging.sh + PrettyPrint.sh) appends
 # every print_info/print_error line to LOGFILENAME, defined as
 # "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$". Running any script that sources the
-# framework therefore drops a timestamped log file next to it. Ignore these
+# framework therefore drops a timestamped log file next to it.
 # everywhere in the repo.
 *.Monday-*
 *.Tuesday-*
 *.Wednesday-*
@@ -10,4 +29,11 @@
 *.Friday-*
 *.Saturday-*
 *.Sunday-*
 # Sensitive exports / runtime data
 dns-cluster-setup/.export/
 returned-logs/
 # Kubernetes secrets (kubeconfig contains embedded client certs)
 k8s/kubeconfig.yaml
 k8s/*.token
@@ -1,76 +1,97 @@
 # Agent Guidelines
 **Top-level files:** [`README.md`](README.md) (project overview),
 [`STATUS.md`](STATUS.md) (living status, agent-maintained),
 [`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else
 lives in subdirectories.
 ## Repository Layout
-Knowing where things live prevents broken edits:
+```
 provisioning/        Server provisioning (SetupNewSystem.sh, security, 2FA)
 tests/               Test suite + VM validation harness
 dns-cluster-setup/   Technitium DNS cluster replication
 k8s/                 k3s cluster setup scripts (3-node HA over Tailscale)
 powerman/            Cyclades PM10i PDU management via powerman
 console/             Serial console management (ser2net + conman) for switches
 perf/                Proxmox perf tuning, fleet audit, iperf
 netinfra/            pfv-netinfra-01/02 DNS/NTP setup
 switches/            Switch configuration captures
 docs/                All documentation (see docs/docmap.md)
 vendor/              Vendored KNELShellFramework
 ```
 - **Vendored framework**: `KNELShellFramework` lives at
  `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`, **not** at
  the repo root. Its includes are under `Framework-Includes/` there. Never
  assume `./Framework-Includes` exists relative to the repo root.
 - **Self-locating scripts**: All provisioning scripts derive their own
-  location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must
+  location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run
-  **never** depend on the current working directory or on `cd`/`realpath ..`
+  from anywhere.
-  chains. Run them from anywhere — `sudo bash SetupNewSystem.sh` works.
+- **Local configs are the source of truth**: Files in
- **Local config files are the source of truth**: Configs in
+  [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
-  `ProjectCode/ConfigFiles/` are read with `cat`/`cp`. Do **not** re-introduce
+  `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
-  `curl ${DL_ROOT}/...` downloads from `dl.knownelement.com` — that CDN is
+- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
-  deprecated for this repo.
+  extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **Path variables**: Scripts export `PROJECT_ROOT_PATH`, `CONFIGFILES_PATH`,
+- **SSH in Crush**: Direct ssh/scp is blocked. Use
-  `MODULES_PATH`, `SCRIPTS_PATH`, and `AGENTS_PATH` for locating repo content.
+  [`tests/remote.sh`](tests/remote.sh) or
- **Non-bash agents**: Some files under `ProjectCode/Agents/` carry a `.sh`
+  [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh).
  extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
  checkers must skip these.
-## Git Commit Requirements
+## Git Policy
-When making changes to this repository, ALWAYS:
+1. **ALWAYS commit + push. Never wait. Ever.** After each logical unit of
   work, immediately stage, commit (conventional format), and push to
   `origin/main`. Do NOT pause for review. Do NOT ask permission. Do NOT
   stop to "let the user read it first." The user reviews rendered markdown
   on Gitea AFTER it is pushed. **This rule overrides any default
   conservative commit-and-hold behavior** — if a default instruction says
   "never commit unless asked," that instruction is superseded here.
 2. **Atomic commits.** Each commit coherent on its own.
 3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
   `docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
-1. **Commit atomically**: Each logical change should be its own commit
+## Automatic Gardening Protocol
 2. **Use conventional commit format**:
   - `feat(scope): description` - New feature
   - `fix(scope): description` - Bug fix
   - `docs: description` - Documentation changes
   - `refactor(scope): description` - Code refactoring
   - `test(scope): description` - Test additions/changes
   - `chore: description` - Maintenance tasks
 3. **Write verbose, beautifully formatted messages**:
   - Title line (50 chars max)
   - Blank line
   - Body explaining WHAT and WHY (not how)
   - Reference related files/issues
   - Include footer with attribution
-## Example Commit
+**Docs and code must be kept in sync.** After any work session, an agent MUST:
-```
+1. **Update [`STATUS.md`](STATUS.md)** — reflect completed work, new issues,
-feat(security-hardening): implement SCAP-STIG compliance logic
+   changed infrastructure state. This file is human read-only; agents own it.
 2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added,
   removed, or substantively changed, update the table and "Last Reviewed"
   date.
 3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after
   any rename or restructure. Fix all references in the same commit.
 4. **Verify new docs are linked** — every new `.md` file must appear in
   [`docs/docmap.md`](docs/docmap.md) and be linked from at least one other
   doc.
 5. **If a new top-level directory was created, update ALL directory listings:**
   - [`README.md`](README.md) → "Directory Structure" table
   - [`AGENTS.md`](AGENTS.md) → "Repository Layout" code block
   - [`AGENTS.md`](AGENTS.md) → "Key Scripts" table (if the directory has
     an entrypoint script)
   Missing any one of these is a protocol violation.
 6. **Self-audit before commit.** Before committing, run:
   ```
   grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md
   ```
   Every new top-level directory must appear in all four files.
-Refactor apply script to implement comprehensive security hardening:
+## Key Scripts
- Add GRUB bootloader permission hardening (root:root, mode 0400)
+| Script | Purpose |
- Disable and remove autofs service per STIG requirements
+|--------|---------|
- Deploy modprobe configurations for kernel module blacklisting
+| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning |
- Create STIG-compliant network protocol blacklist
+| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
 | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
 | [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
 | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
 | [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman |
 | [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
 | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
-This ensures servers meet DoD security requirements for production
+## Key Docs
 deployment.
-🤖 Generated with [Crush](https://github.com/charmassociates/crush)
+→ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.**
-Assisted-by: GLM-5 via Crush <crush@charm.land>
+## Project Context
 ```
-## Autonomous Git Workflow
+Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
-
+Production lives on a Cloudron VPS in Reston VA. See
-**Agents are authorized to commit AND push autonomously. Do not wait to be
+[`STATUS.md`](STATUS.md) for current state and
-asked.** After each logical unit of work:
+[`docs/proxmox/PROJECT.md`](docs/proxmox/PROJECT.md) for the fleet report.
 1. Stage only the files belonging to that logical change.
 2. Commit with a conventional, well-formed message (see above).
 3. Push to `origin` (`git push`). The branch tracks `origin/main`.
 4. Repeat per logical unit.
 Group changes so each commit is coherent on its own (a reader should
 understand the commit without seeing the others). Never batch unrelated
 changes into one commit.
@@ -1,217 +0,0 @@
 # Tailscale vs. Managed DNS — Architecture Analysis
 > **Status:** analysis for review. No code decisions are final. Read the
 > "Known issues" section before acting on the managed-resolv.conf change.
 ## 1. Executive summary
 Every host in this build runs the Tailscale client, and Tailscale — by default —
 **manages `/etc/resolv.conf` itself**, pointing it at `100.100.100.100`
 (Tailscale's MagicDNS resolver). This directly conflicts with the managed
 `resolv.conf` (pointing at `192.168.3.252`/`192.168.3.253`) that
 `SetupNewSystem.sh` deploys: whichever runs last wins, and Tailscale's daemon
 re-wins on every `tailscale up` and on reboot.
 Worse, a probe of the live network shows that **knel.net device records only
 resolve through the Tailscale 100.100.100.100 path** — querying the LAN IPs of
 the DNS servers directly returns NXDOMAIN for current hostnames (the Technitium
 `knel.net` zone has the SOA but is stale/empty of actual records). So pointing
 `resolv.conf` at the LAN IPs would break resolution of the very names this
 project's modules depend on (`tsys-nsm.knel.net`, `tsys-cloudron.knel.net`,
 `tsys-librenms.knel.net`).
 This document lays out the options and a recommended path forward.
 ## 2. How name resolution actually works today (as measured)
 Probed from `sectestbed-sandbox` (192.168.3.50):
 | Query path | External name (`github.com`) | knel.net device name (`pfv-netinfra-01.knel.net`) |
 |---|---|---|
 | Via current resolver = `100.100.100.100` (Tailscale) | resolves | **resolves** → `100.70.181.72` (Tailscale CGNAT) |
 | Direct `dig @192.168.3.252` (Technitium, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) |
 | Direct `dig @192.168.3.253` (Pi-hole, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) |
 Other measured facts:
 - `dig @192.168.3.252 knel.net SOA` → `NOERROR`, returns
  `knel.net. 900 IN SOA dns.knel.net. hostadmin.knel.net. 2025062313 …`
  (serial dated **2025-06-23** — the zone exists but is stale).
 - NTP on both `.252` and `.253` answers time queries (stratum 2/3).
 - The live `/etc/resolv.conf` on a deployed host reads:
  ```
  # resolv.conf(5) file generated by tailscale
  # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
  nameserver 100.100.100.100
  nameserver fd7a:115c:a1e0::53
  search knel.net
  ```
 **Interpretation:** the `knel.net` device→Tailscale-IP mappings are synthesised
 by Tailscale's MagicDNS from the tailnet device registry (every device that
 joins the tailnet gets `hostname.knel.net` → its `100.x.x.x` address). The
 Technitium `knel.net` zone is a separate, manually-maintained zone that has
 fallen out of date. The two are not the same source of truth.
 ## 3. The core tension
 | Goal | Who provides it today |
 |---|---|
 | Resolve `*.knel.net` device names (→ Tailscale IPs) | Tailscale MagicDNS via `100.100.100.100` |
 | Resolve external names with ad-blocking | Pi-hole (`.253`), reachable via Tailscale → Technitium → Pi-hole chain |
 | Redundant, low-latency, tunnel-independent DNS | LAN resolvers `.252`/`.253` — **but these lack knel.net records** |
 | Authoritative time | NTP on `.252`/`.253` (works on either path) |
 The conflict: you cannot simply point `resolv.conf` at the LAN resolvers,
 because they do not know about the current `knel.net` device records, and
 several modules in this project resolve `knel.net` hostnames at runtime
 (wazuh manager, postfix relay, syslog target). You also cannot ignore Tailscale,
 because it is the only thing that resolves those names today.
 ## 4. Options
 ### Option A — Let Tailscale own DNS (status quo, `accept-dns=true`)
 Leave the default. Tailscale writes `100.100.100.100` to `resolv.conf`; the
 control-plane forwarding (`100.100.100.100` → Technitium → Pi-hole) handles
 external names and ad-blocking; MagicDNS handles `knel.net` device names.
 | Pros | Cons |
 |---|---|
 | Zero per-host config; new machines "just work" on `tailscale up` | **All DNS depends on the Tailscale daemon being up.** If `tailscaled` dies, every name lookup fails — including the ones you need to SSH in and fix it. |
 | MagicDNS + knel.net names resolve automatically | Latency: every query goes host→tailscaled→100.100.100.100→(tunnel)→Technitium→Pi-hole→upstream |
 | Ad-blocking preserved (via the Pi-hole hop) | Overwrites the managed `resolv.conf` — the `.252`/`.253` redundancy is lost |
 | Centralised in the Tailscale admin console | Single resolver in `resolv.conf` (`100.100.100.100`); no glibc-level failover |
 | | Boot-order risk: early-boot processes have no DNS until `tailscaled` is up |
 ### Option B — Pin resolv.conf to the LAN resolvers (`accept-dns=false`)
 Set `--accept-dns=false` on every host and keep the managed `resolv.conf`
 pointing at `.252`/`.253`.
 | Pros | Cons |
 |---|---|
 | DNS independent of Tailscale — survives `tailscaled` outages | **`*.knel.net` device names break (NXDOMAIN)** because the LAN resolvers' knel.net zone is stale. This breaks wazuh/postfix/syslog hostname resolution. |
 | Lowest latency, full glibc-level failover across two servers | MagicDNS names (`*.ts.net`) do not resolve |
 | Managed `resolv.conf` wins uncontested | Requires fixing the Technitium/Pi-hole `knel.net` zone to mirror the Tailscale device records before this is viable |
 | Boot-time DNS works immediately | Off-LAN hosts (laptops) can't reach `.252`/`.253` without the tunnel — back to needing Tailscale |
 > **Not recommended as-is.** Only viable **after** the `knel.net` zone on
 > `.252`/`.253` is repopulated with current device records (see §6).
 ### Option C — Tailscale Split DNS (per-domain routing)
 MagicDNS `ON`, "Override local DNS" `OFF` in the admin console; only `ts.net`
 (and explicitly split domains) route to `100.100.100.100`, everything else stays
 on the system resolver.
 | Pros | Cons |
 |---|---|
 | Best of both worlds: MagicDNS names resolve AND general queries go direct | Requires `systemd-resolved` (or NetworkManager `dns=dnsmasq`) for per-domain routing. These hosts use a **plain `/etc/resolv.conf`** — on which Tailscale **cannot** do per-domain split; it replaces the whole file. |
 | Reduces tunnel dependency for non-Tailscale names | Migrating every host to `systemd-resolved` is a significant, cross-cutting change |
 | | More moving parts to reason about and debug |
 ### Option D — Make Tailscale push the LAN resolvers as global nameservers
 In the admin console, set global nameservers to `192.168.3.252`/`192.168.3.253`,
 keep `accept-dns=true`.
 | Pros | Cons |
 |---|---|
 | Clients get the LAN resolvers via Tailscale config (consistent) | Tailscale still overwrites `resolv.conf` |
 | MagicDNS still works (100.100.100.100 added for `ts.net`/`knel.net`) | On-LAN hosts don't need Tailscale to find `.252`/`.253` — pure indirection |
 | Centralised management | Still depends on `tailscaled` for DNS |
 | | `knel.net` device names still only resolve via the Tailscale path, so the LAN resolvers being "global" doesn't help those names unless the zone is fixed |
 ## 5. Recommendation
 **Short term (unblock now): Option A — let Tailscale own DNS.** Revert/disable
 the managed-`resolv.conf` deployment so provisioning stops fighting Tailscale.
 Today, `knel.net` device names **only** resolve through Tailscale, and this
 project's modules depend on those names, so Tailscale-managed DNS is the only
 thing that currently works end-to-end. Keep the NTP change (LAN IPs, no DNS
 dependency) — that part is safe and beneficial regardless.
 **Medium term (the real fix): populate the `knel.net` zone on the LAN
 resolvers**, then choose B or C. Concretely:
 1. Make Technitium (`.252`) authoritative for `knel.net` **with current records**
   (mirror the Tailscale device→IP mappings, or enable a zone-transfer/sync from
   the Tailscale device registry, or use Technitium's "Tailscale" DNS app if
   available). Confirm `dig @192.168.3.252 pfv-netinfra-01.knel.net` returns an
   answer, not NXDOMAIN.
 2. Make Pi-hole (`.253`) forward `knel.net` to Technitium (or also serve the
   zone), so both resolvers in the pair can answer internal names — otherwise
   glibc failover to `.253` would silently break knel.net lookups.
 3. *Then* pin `resolv.conf` to `.252`/`.253` with `--accept-dns=false`
   (Option B), gaining tunnel-independent, redundant DNS.
 **Long term (optional, if per-domain routing is wanted): Option C** — adopt
 `systemd-resolved` and configure Tailscale Split DNS so `ts.net`/`knel.net` go
 to MagicDNS and everything else goes direct. Only worth the migration cost if
 you specifically need `*.ts.net` short-name resolution alongside direct LAN DNS.
 ### Why not just force `.252`/`.253` today?
 Because it regresses name resolution for the hostnames this project already
 uses. Concretely, with `resolv.conf` pinned to the LAN resolvers the following
 would fail to resolve:
 - `ProjectCode/Modules/Security/secharden-wazuh.sh` → `WAZUH_MANAGER="tsys-nsm.knel.net"`
 - `ProjectCode/SetupNewSystem.sh` → `postconf -e "relayhost = tsys-cloudron.knel.net"`
 - `ProjectCode/ConfigFiles/Syslog/rsyslog.conf` → `*.* @tsys-librenms.knel.net:514`
 All three resolve cleanly via `100.100.100.100` today and return NXDOMAIN via
 `.252`/`.253`. Pinning the LAN resolvers before the zone is fixed would break
 wazuh, mail relay, and syslog.
 ## 6. Known issues / action items
 1. **Technitium `knel.net` zone is stale.** SOA serial `2025062313`
   (2025-06-23); current device names return NXDOMAIN from the LAN interface.
   Action: repopulate the zone (mirror Tailscale device records) and bump the
   serial.
 2. **Pi-hole (`.253`) has no `knel.net` device records either.** For the pair
   to be truly redundant for internal names, `.253` must either serve the same
   zone or conditional-forward `knel.net` to `.252`. Action: configure Pi-hole
   to forward `knel.net` to Technitium.
 3. **The managed-`resolv.conf` change (commit f010fa9) conflicts with
   Tailscale.** As written, `SetupNewSystem.sh` writes `resolv.conf` with
   `.252`/`.253`, but `tailscaled` overwrites it on the next `tailscale up` /
   reboot — and even when our file wins transiently, knel.net names break. See
   §5 for the recommended handling.
 4. **NTP change is safe and good.** `ntp.conf` now uses LAN IPs
   (`192.168.3.252`/`192.168.3.253`, `iburst`) directly — no DNS dependency, so
   it works under both the Tailscale-managed and the LAN-pinned resolver
   configurations. Keep this regardless of the DNS decision.
 5. **Split-horizon possibility (unconfirmed).** It is possible Technitium serves
   a richer `knel.net` zone on its Tailscale interface (`100.x`) than on its LAN
   interface (`192.168.3.252`). If so, the fix is to make the LAN view match the
   Tailscale view. Worth confirming with `dig @<technitium-tailscale-ip> knel.net host`.
 ## 7. Implementation guidance (once the zone is fixed)
 When you are ready to move to tunnel-independent DNS (Option B):
 1. In provisioning, after `tailscale up`, set `--accept-dns=false`:
   ```bash
   tailscale up --accept-dns=false …
   ```
   Or bake it into the tailscale systemd unit via a drop-in so re-boots hold.
 2. *Then* deploy the managed `resolv.conf` (`.252`/`.253`). Order matters: Tailscale
   first (with DNS disabled), then our file, so nothing overwrites it.
 3. Add a watchdog (timer) that restores `resolv.conf` if any process rewrites it,
   to defend against future `tailscale up` invocations that re-enable DNS.
 4. Validate with `Project-Tests/validation/dns-ntp-redundancy.sh` — and extend
   its probe to assert `*.knel.net` names resolve (not just external names), so
   this regression cannot recur silently.
 ## 8. TL;DR
 - **DNS**: don't fight Tailscale yet. Today `knel.net` names only resolve via
  Tailscale, and this project depends on them. Fix the Technitium/Pi-hole
  `knel.net` zone first, *then* pin the LAN resolvers.
 - **NTP**: the LAN-IP change is correct and safe; keep it.
 - **The managed `resolv.conf` (`.252`/`.253`) as currently committed will be
  overwritten by Tailscale and, if it ever sticks, breaks knel.net resolution —
  see §5/§6 before relying on it.**
@@ -1,21 +1,51 @@
-# KNEL FetchApply
+# PFVCluster
-## Repo Issues
+Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
 https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
-## Repo Discussion
+**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)**
 https://community.turnsys.com/c/chieftechnologyandproductofficer/26
 ## Directory Structure
-## Repo discription 
+| Directory | Description |
-Known Element Enterprises (the entity serving as the TSYS Group management company) (through it’s executive leader, the COO) provides core IT/back office systems/services/support on a hands off/fully delegated authortity basis to the CCO and the orgs/members.
+|-----------|-------------|
 | [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) |
 | [`tests/`](tests/) | Test suite + VM validation harness |
 | [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts |
 | [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) |
 | [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 |
 | [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
 | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
 | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
 | [`switches/`](switches/) | Switch configuration captures |
 | [`docs/`](docs/) | All documentation ([see docmap](docs/docmap.md)) |
 | [`vendor/`](vendor/) | Vendored KNELShellFramework |
-One of those functions is the provisoning of Linux servers. This repository is the Infrastructure As Code (IAC) repository for TSYS. 
+## Quick Start
-In the future it will be used via FetchApply https://github.com/P5vc/fetch-apply
+### Provision a new server
 ```bash
 sudo bash provisioning/SetupNewSystem.sh
 ```
-## Usage
+### Validate on the sandbox VM
 ```bash
 VM_ID=6000 ./tests/vm-validation.sh all
 ```
-git clone this repo
+### Deploy DNS cluster
-cd FetchApply/ProjectCode
+```bash
-bash SetupNewSystem.sh
+cd dns-cluster-setup/ && ./setup.sh all
 ```
 ### Deploy perf tunings
 ```bash
 cd perf/ && ./deploy-tuning.sh
 ```
 ## Architecture
 - **Proxmox hosts**: 7 standalone PVE installs managed via PDM
 - **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
 - **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
 - **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
 - **Backups**: Proxmox Backup Server (PBS)
@@ -0,0 +1,116 @@
 # Project Status
 > **Human read-only. Agents maintain this file automatically after each work
 > session.** Do not edit by hand — the next agent run will overwrite it.
 > **Last updated:** 2026-07-28 by Crush (GLM-5.2)
 ## Current State: STABLE
 The merged PFVCluster repo is fully operational across provisioning, DNS
 infrastructure, Proxmox cluster ops, and k8s control plane.
 ## Completed Work
 ### Server Provisioning (validated on sectestbed-sandbox)
 - [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
 - [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
 - [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
 - [x] DNS resolv.conf: managed static file pointing at netinfra pair
 - [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
 - [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
 ### DNS Cluster (pfv-netinfra-01/02)
 - [x] Production Technitium config replicated from tailscale-router (read-only)
 - [x] 124 zones on both nodes (knel.net + reverse DNS)
 - [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
 - [x] Zone replication via rsync systemd timer (every 60s)
 - [x] Credentials + 2FA replicated identically to production
 - [x] Both LAN IPs resolve knel.net device names + recurse externally
 ### Proxmox Cluster Ops
 - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
 - [x] Fleet audit complete (PROJECT.md has ground truth)
 - [ ] tsys4: blocked on PCIe NIC + RAM install
 - [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
 - [ ] tsys2: pending rebuild from Win10 to Proxmox
 ### Kubernetes Control Plane (k3s HA — LIVE)
 - [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
 - [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
 - [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
 - [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
  Tailscale IPs, CoreDNS, API server, workload isolation
 - [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
 - [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
 - [ ] Workers (wnodes) not yet joined to this cluster
 - [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
  preserved in `docs/k8s/` for future ITAR/classified cluster.
 ### PDU Management (powerman on pfv-tsys1 — LIVE)
 - [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
 - [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink
  `/dev/cyclades-pm10`
 - [x] powermand listening on `127.0.0.1:10101` + Tailscale `100.121.189.98:10101`
 - [x] All 10 outlets defined as `outlet-1` through `outlet-10`
 - [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
 - [ ] Rename outlets to match physical devices (Friday onsite)
 ### Console Management (ser2net + conman on pfv-tsys4 — LIVE)
 - [x] 7 network switch/router consoles managed via ser2net + conman
 - [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
  ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
  symlinks that survive reboot regardless of enumeration order
 - [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
  **Tailscale IP only** (`100.70.77.93`)
 - [x] conman connects to TCP ports for logging + multiplexing (7 log
  files active in `/var/log/conman/`)
 - [x] Both ser2net + conmand enabled via systemd (survive reboot)
 - [x] conmand systemd unit created (Debian package doesn't ship one)
 - [x] Old `/root/conmap` + manual `screen` workflow replaced
 ### Repo Merge
 - [x] KNELServerBuild merged into PFVCluster (history preserved)
 - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
 - [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
 ## Known Issues
 | Issue | Impact | Status |
 |-------|--------|--------|
 | Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
 | `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
 | tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
 ## Pending (next session priorities)
 1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical
   devices (Friday onsite)
 2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
 2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
   - Cnode count: 3 (recommended) vs 5
   - Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
   - Tailscale pattern: subnet router on `tailscale-router` (recommended)
 3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
 4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
 5. Perf: complete tsys4/5 tuning after hardware install (Friday)
 6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
 7. **k8s deferred topics** (tracked for future sessions): ETL tooling
   (GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
   per-tenant vcluster policy templates, solar-aware scale-out
 ## Infrastructure Summary
 | Component | Details |
 |-----------|---------|
 | Proxmox hosts | 7 standalone PVE, managed via PDM |
 | DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
 | DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
 | DNS production | tailscale-router (read-only source of truth) |
 | NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
 | Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
 | Backup | Proxmox Backup Server (PBS) |
 | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
 | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
 | **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** |
 | Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
@@ -0,0 +1,109 @@
 # Console Management (ser2net + conman)
 Network-accessible serial console management for all production network
 switches and routers, running on **pfv-tsys4** (storage server).
 ## Architecture
 ```
 USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net (TCP) → conman (logging + multiplexing)
 ```
 ser2net owns the physical serial devices and exposes them on TCP ports
 bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman
 connects to those TCP ports for session logging, output capture, and
 multi-user console sharing.
 **conman and ser2net do NOT share ports** — only one process can open a
 serial device at a time. ser2net owns the physical device; conman connects
 over TCP.
 ## The USB Enumeration Problem (SOLVED)
 The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
 unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
 enumeration order, which shifts on every boot. This made the old
 `/root/conmap` + manual `screen` workflow break after every reboot.
 **Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
 topology), which is stable across reboots regardless of enumeration order.
 Each adapter gets a named symlink in `/dev/consoles/` that never changes.
 The udev rules are generated from `mapping.txt`, which maps each adapter's
 ID_PATH to a console name and TCP port. To re-map after physically moving
 an adapter, update `mapping.txt` and re-run `setup.sh`.
 **Fallback:** if udev trigger doesn't create symlinks for already-discovered
 devices (common on first run), `setup.sh` creates them manually by matching
 ID_PATH. On subsequent boots, udev creates them automatically.
 ## Port Assignments
 | TCP Port | Console Name | ID_PATH | Description |
 |----------|-------------|---------|-------------|
 | 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
 | 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
 | 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
 | 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
 | 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
 | 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
 | 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
 All ports listen on the Tailscale IP (`100.70.77.93`).
 ## Scripts
 | Script | Purpose |
 |--------|---------|
 | [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
 | [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
 | [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
 | [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
 ## Usage
 ### Connect to a console
 ```bash
 # Via conman (with logging + multiplexing):
 conman -f pfv-core-sw01        # attach to console
 conman -q                      # query status of all consoles
 # Via telnet (direct, no logging):
 telnet 100.70.77.93 2001       # pfv-core-sw01
 # From any Tailscale-connected device:
 telnet pfv-tsys4 2001
 ```
 ### Re-deploy after changing mapping.txt
 ```bash
 PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
 ```
 ### Find the ID_PATH for a new adapter
 ```bash
 PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
 ```
 Then match the new adapter's ID_PATH to its physical location and add a line
 to `mapping.txt`.
 ## Files on pfv-tsys4
 | File | Purpose |
 |------|---------|
 | `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
 | `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) |
 | `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
 | `/etc/systemd/system/conmand.service` | systemd unit for conmand |
 | `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
 | `/root/console/setup.sh` | Setup script (re-runnable) |
 | `/root/console/generate-config.sh` | Config generator |
 ## Old workflow (replaced)
 The old `/root/conmap` file and manual `screen` sessions are no longer
 needed. The new setup is fully automated and survives reboots.
@@ -0,0 +1,102 @@
 #!/usr/bin/bash
 #
 # console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4
 #
 # Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
 #
 # This script is strictly read-only. No writes to the system.
 #
 set -uo pipefail
 echo "============================================"
 echo "  Console Setup Discovery"
 echo "  Host: $(hostname)"
 echo "  Date: $(date)"
 echo "  READ-ONLY"
 echo "============================================"
 echo ""
 echo "=== 1. USB devices ==="
 lsusb 2>/dev/null || echo "(lsusb not available)"
 echo ""
 echo "=== 2. All ttyUSB* devices (with major/minor) ==="
 ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
 echo ""
 echo "=== 3. USB-serial driver bindings ==="
 echo "-- pl2303 --"
 ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- cp210x --"
 ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ftdi_sio --"
 ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ch341 --"
 ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo ""
 echo "=== 4. USB serial adapter details (vendor/model/serial per port) ==="
 for tty in /dev/ttyUSB*; do
  [ -e "$tty" ] || continue
  echo "--- $tty ---"
  udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/  /'
 done
 echo ""
 echo "=== 5. Existing /root/conmap ==="
 if [ -f /root/conmap ]; then
  cat /root/conmap
 else
  echo "(no /root/conmap)"
 fi
 ls -la /root/conmap* 2>/dev/null
 echo ""
 echo "=== 6. Screen sessions (running) ==="
 screen -ls 2>&1 || echo "(screen not running or not installed)"
 echo ""
 echo "=== 7. Existing screen wrappers/scripts in /root ==="
 ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)"
 echo ""
 echo "=== 8. ser2net ==="
 which ser2net 2>/dev/null || echo "(ser2net not installed)"
 dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)"
 cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)"
 systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)"
 echo ""
 echo "=== 9. conman ==="
 which conman 2>/dev/null || echo "(conman not installed)"
 which conmand 2>/dev/null || echo "(conmand not installed)"
 dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)"
 echo "--- /etc/conman.conf (console lines only) ---"
 grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)"
 echo "--- conmand service ---"
 systemctl is-active conmand 2>/dev/null || echo "(conmand not running)"
 systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)"
 echo ""
 echo "=== 10. Existing console logs ==="
 ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)"
 ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)"
 echo ""
 echo "=== 11. udev rules for ttyUSB ==="
 grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)"
 grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true
 echo ""
 echo "=== 12. expect availability ==="
 command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
 command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed"
 echo ""
 echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ==="
 ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)"
 echo ""
 echo "============================================"
 echo "  Discovery complete (read-only)."
 echo "============================================"
@@ -0,0 +1,244 @@
 #!/usr/bin/bash
 #
 # console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf
 #
 # Reads console/mapping.txt (the source of truth) and generates all three
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
 #      to a named symlink like /dev/consoles/pfv-core-sw01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing
 #
 # Run this script ON the target host. It writes to:
 #   /etc/udev/rules.d/99-console-ports.rules
 #   /etc/ser2net.yaml
 #   /etc/conman/console-consoles.conf  (included by /etc/conman.conf)
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh
 #
 # Environment overrides:
 #   MAPPING_FILE  — path to mapping.txt (default: auto-detect next to this script)
 #   TS_IP         — Tailscale IP to bind ser2net on (default: auto-detect)
 #   CONMAN_LOGDIR — conman log directory (default: /var/log/conman)
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}"
 CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}"
 UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
 SER2NET_CONF="/etc/ser2net.yaml"
 CONMAN_CONF="/etc/conman.conf"
 CONMAN_CONSOLES="/etc/conman/console-consoles.conf"
 CONSOLE_DEV_DIR="/dev/console"
 echo "============================================"
 echo "  Console Config Generator"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- Locate mapping file ---
 # When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong.
 # Search common locations.
 if [ ! -f "$MAPPING_FILE" ]; then
  for candidate in \
    "/root/console/mapping.txt" \
    "/tmp/mapping.txt" \
    "$(dirname "$0")/mapping.txt"; do
    if [ -f "$candidate" ]; then
      MAPPING_FILE="$candidate"
      break
    fi
  done
 fi
 if [ ! -f "$MAPPING_FILE" ]; then
  echo "FATAL: mapping file not found. Tried: $MAPPING_FILE"
  echo "Copy mapping.txt to the target host first."
  exit 1
 fi
 echo "  Mapping file: $MAPPING_FILE"
 # --- Auto-detect Tailscale IP ---
 if [ -z "${TS_IP:-}" ]; then
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -z "$TS_IP" ]; then
    echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually."
    exit 1
  fi
 fi
 echo "  Tailscale IP: $TS_IP"
 echo "  ser2net will bind to: $TS_IP"
 # --- Parse mapping file (skip comments and blank lines) ---
 echo ""
 echo "--- Parsing mapping file ---"
 ENTRIES=()
 while IFS= read -r line; do
  # Skip comments and blank lines
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"  # trim whitespace
  [ -z "$line" ] && continue
  ENTRIES+=("$line")
  echo "  $line"
 done < "$MAPPING_FILE"
 if [ "${#ENTRIES[@]}" -eq 0 ]; then
  echo "FATAL: no entries found in mapping file."
  exit 1
 fi
 echo ""
 echo "  ${#ENTRIES[@]} console ports configured."
 # ============================================================
 # 1. Generate udev rules
 # ============================================================
 echo ""
 echo "--- [1/3] Generating udev rules: $UDEV_RULES ---"
 cat > "$UDEV_RULES" <<'UDEV_HEADER'
 # Stable symlinks for USB-DB9 console adapters
 # Generated by console/generate-config.sh
 # DO NOT EDIT — edit mapping.txt and re-run generate-config.sh
 #
 # These rules pin each adapter to a named symlink based on its physical
 # USB port path (ID_PATH), which is stable across reboots regardless of
 # enumeration order. This is the fix for the "USB adapters shift on reboot"
 # problem.
 #
 # To find the ID_PATH for a device:
 #   udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH
 UDEV_HEADER
 for entry in "${ENTRIES[@]}"; do
  IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
  # Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4"
  # The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0"
  # We match on the substring to be portable across PCI bus changes.
  echo "" >> "$UDEV_RULES"
  echo "# $name (TCP $tcp_port): $comment" >> "$UDEV_RULES"
  echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"console/$name\"" >> "$UDEV_RULES"
 done
 echo "  Written: $UDEV_RULES"
 echo "  Symlinks: /dev/consoles/<name> for each device"
 # ============================================================
 # 2. Generate ser2net.yaml
 # ============================================================
 echo ""
 echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---"
 # Backup existing config if not already backed up
 if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then
  cp "$SER2NET_CONF" "${SER2NET_CONF}.orig"
  echo "  Backed up original to ${SER2NET_CONF}.orig"
 fi
 {
  echo "%YAML 1.1"
  echo "---"
  echo "# ser2net configuration for pfv-tsys4 console ports"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "#"
  echo "# All ports bound to Tailscale IP ($TS_IP) for secure remote access."
  echo "# Physical devices are accessed via stable udev symlinks in /dev/consoles/."
  echo "#"
  echo "# To connect directly:   telnet $TS_IP 2001"
  echo "# To connect via conman: conman -f <name>"
  echo ""
  echo "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    # ser2net connection block
    echo "connection: &con${tcp_port}"
    echo "    accepter: tcp,${TS_IP},${tcp_port}"
    echo "    enable: on"
    echo "    options:"
    echo "      banner: *banner"
    echo "      kickolduser: true"
    echo "      telnet-brk-on-sync: true"
    echo "    connector: serialdev,"
    echo "              /dev/consoles/${name},"
    echo "              ${baud},local"
    echo ""
  done
 } > "$SER2NET_CONF"
 echo "  Written: $SER2NET_CONF"
 echo "  ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)"
 # ============================================================
 # 3. Write conman console entries directly into conman.conf
 # ============================================================
 # conman 0.3.x does NOT support the 'include' directive, so we write
 # CONSOLE entries directly into /etc/conman.conf between idempotent markers.
 echo ""
 echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---"
 # Ensure logdir exists
 mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true
 # Ensure LOGDIR is set in conman.conf (server-level directive for log file paths)
 if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
  # Insert near the top, after the first SERVER directives
  sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF"
  echo "  Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
 fi
 # Remove any previous auto-generated block (between markers)
 # Then append the new block
 MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
 MARKER_END="# END PFV CONSOLE DEFINITIONS"
 # Strip old block if present
 if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then
  sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF"
  echo "  Removed previous console definitions."
 fi
 # Append new block
 {
  echo ""
  echo "$MARKER_BEGIN"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "# Each console connects to a ser2net TCP port via telnet protocol."
  echo "# ser2net owns the physical serial device; conman provides logging"
  echo "# and multiplexing on top."
  echo "# Access: conman -f <name>"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
  done
  echo "$MARKER_END"
 } >> "$CONMAN_CONF"
 CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0)
 echo "  Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF"
 # ============================================================
 # Summary
 # ============================================================
 echo ""
 echo "============================================"
 echo "  Configuration generated successfully."
 echo ""
 echo "  Files written:"
 echo "    $UDEV_RULES       ($(wc -l < "$UDEV_RULES") lines)"
 echo "    $SER2NET_CONF     ($(wc -l < "$SER2NET_CONF") lines)"
 echo "    $CONMAN_CONF      (CONSOLE entries appended between markers)"
 echo ""
 echo "  Next steps:"
 echo "    1. Reload udev:   udevadm control --reload-rules && udevadm trigger"
 echo "    2. Restart ser2net: systemctl restart ser2net"
 echo "    3. Start conman:  systemctl enable --now conmand"
 echo "    4. Or run:       bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
 echo "============================================"
@@ -0,0 +1,29 @@
 # console/mapping.txt — Source of Truth for console port assignments
 #
 # Format: <tcp_port>|<name>|<id_path_substring>|<baud>|<comment>
 #
 # Delimiter is | (pipe) because ID_PATH values contain colons.
 #
 # - tcp_port:     TCP port ser2net listens on (also the conman console name suffix)
 # - name:         Device name (used for /dev/console/<name> symlink, conman console name)
 # - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH`
 #                      These are STABLE across reboots as long as adapters aren't moved
 #                      to different physical USB ports.
 # - baud:         Serial baud rate (9600n81 = 9600 8N1, no flow control)
 # - comment:      Free-form description
 #
 # To RE-MAP after physically moving an adapter:
 #   1. Run: bash console/discover.sh   (find the new ID_PATH for the device)
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
 2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
 2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
 2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
 2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
 2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
@@ -0,0 +1,216 @@
 #!/usr/bin/bash
 #
 # console/setup.sh — deploy console management on pfv-tsys4
 #
 # Orchestrates the full setup:
 #   1. Ensures ser2net + conman are installed
 #   2. Copies mapping.txt to the target host (if running remotely)
 #   3. Runs generate-config.sh to produce udev rules + ser2net.yaml + conman.conf
 #   4. Reloads udev, creates /dev/consoles/ symlinks
 #   5. Restarts ser2net (TCP ports on Tailscale IP)
 #   6. Enables + starts conmand (logging + multiplexing)
 #   7. Verifies
 #
 # This script is IDEMPOTENT — safe to run multiple times.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/setup.sh
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
 SER2NET_CONF="/etc/ser2net.yaml"
 CONMAN_CONSOLES="/etc/conman/console-consoles.conf"
 CONSOLE_DEV_DIR="/dev/consoles"
 echo "============================================"
 echo "  Console Management Setup"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- 1. Install dependencies ---
 echo ""
 echo "--- [1/7] Checking dependencies ---"
 NEED_INSTALL=()
 dpkg -l ser2net 2>/dev/null | grep -q '^ii' && echo "  ser2net: installed" || NEED_INSTALL+=(ser2net)
 dpkg -l conman  2>/dev/null | grep -q '^ii' && echo "  conman: installed"  || NEED_INSTALL+=(conman)
 if [ "${#NEED_INSTALL[@]}" -gt 0 ]; then
  echo "  Installing: ${NEED_INSTALL[*]}"
  apt-get update -qq
  apt-get install -y -qq "${NEED_INSTALL[@]}"
 else
  echo "  All dependencies present."
 fi
 # --- 2. Ensure mapping file is available ---
 echo ""
 echo "--- [2/7] Locating mapping file ---"
 MAPPING_FILE=""
 for candidate in \
  "$SCRIPT_DIR/mapping.txt" \
  "$(dirname "$0")/mapping.txt" \
  "/root/console/mapping.txt" \
  "/tmp/mapping.txt"; do
  if [ -f "$candidate" ]; then
    MAPPING_FILE="$candidate"
    break
  fi
 done
 if [ -z "$MAPPING_FILE" ]; then
  echo "FATAL: mapping.txt not found. Copy it to the target host."
  exit 1
 fi
 echo "  Using: $MAPPING_FILE"
 # --- 3. Generate configs ---
 echo ""
 echo "--- [3/7] Generating configs ---"
 export MAPPING_FILE
 bash "$(dirname "$0")/generate-config.sh" 2>&1 || bash "$SCRIPT_DIR/generate-config.sh" 2>&1 || {
  echo "FATAL: generate-config.sh failed."
  exit 1
 }
 # --- 4. Reload udev + create symlinks ---
 echo ""
 echo "--- [4/7] Reloading udev rules ---"
 udevadm control --reload-rules
 # Try trigger first (works on some systems)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty" ] && udevadm trigger --action=add "$tty" 2>/dev/null || true
 done
 # Also try writing to uevent (forces udev reprocessing)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty/uevent" ] && echo "add" > "$tty/uevent" 2>/dev/null || true
 done
 sleep 2
 # FALLBACK: if udev symlinks don't exist (common when devices are already
 # discovered — udev trigger doesn't always re-create symlinks for existing
 # devices), create them manually by matching ID_PATH. The udev rules will
 # handle future boots/hotplugs automatically.
 if [ ! -d /dev/consoles ] || [ -z "$(ls /dev/consoles/ 2>/dev/null)" ]; then
  echo "  udev trigger didn't create symlinks. Creating manually..."
  mkdir -p /dev/consoles
  while IFS= read -r line; do
    line="${line%%#*}"
    line="$(echo "$line" | xargs)"
    [ -z "$line" ] && continue
    IFS='|' read -r tcp_port name id_path baud comment <<< "$line"
    # Find the ttyUSB whose ID_PATH contains the mapping's id_path substring
    for tty in /dev/ttyUSB*; do
      [ -e "$tty" ] || continue
      DEV_IDPATH=$(udevadm info -q property -n "$tty" 2>/dev/null | grep ^ID_PATH= | cut -d= -f2)
      if echo "$DEV_IDPATH" | grep -q "$id_path"; then
        ln -sf "$tty" "/dev/consoles/$name"
        echo "    ln -s $tty -> /dev/consoles/$name"
        break
      fi
    done
  done < "$MAPPING_FILE"
 fi
 echo "  Stable symlinks:"
 ls -la /dev/consoles/ 2>/dev/null | grep -v '^total\|^d' | sed 's/^/    /' || echo "    (none created)"
 # Verify each symlink resolves
 echo ""
 echo "  Symlink verification:"
 while IFS= read -r line; do
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"
  [ -z "$line" ] && continue
  IFS='|' read -r tcp_port name id_path baud comment <<< "$line"
  if [ -e "/dev/consoles/$name" ]; then
    TARGET=$(readlink -f "/dev/consoles/$name")
    echo "    [OK] /dev/consoles/$name -> $TARGET"
  else
    echo "    [MISSING] /dev/consoles/$name (adapter unplugged or ID_PATH changed)"
  fi
 done < "$MAPPING_FILE"
 # --- 5. Restart ser2net ---
 echo ""
 echo "--- [5/7] Restarting ser2net ---"
 systemctl enable ser2net
 systemctl restart ser2net
 sleep 2
 if systemctl is-active --quiet ser2net; then
  echo "  ser2net is running."
  TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1")
  echo "  Listening ports:"
  ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/    /'
 else
  echo "  WARNING: ser2net failed to start. Checking journal..."
  journalctl -u ser2net --no-pager -n 20
 fi
 # --- 6. Enable + start conmand ---
 echo ""
 echo "--- [6/7] Starting conmand ---"
 # conman package on Debian may not ship a systemd unit. Create one if missing.
 if ! systemctl cat conmand >/dev/null 2>&1; then
  echo "  No systemd unit for conmand — creating one..."
  cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT'
 [Unit]
 Description=ConMan (Console Manager)
 After=network.target ser2net.service
 Requires=ser2net.service
 [Service]
 Type=forking
 ExecStart=/usr/sbin/conmand -c /etc/conman.conf
 Restart=on-failure
 RestartSec=5
 [Install]
 WantedBy=multi-user.target
 CONMAND_UNIT
  systemctl daemon-reload
  echo "  Created /etc/systemd/system/conmand.service"
 fi
 # Kill any manually-started conmand first
 pkill -x conmand 2>/dev/null || true
 sleep 1
 systemctl enable conmand 2>/dev/null || true
 systemctl restart conmand 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet conmand; then
  echo "  conmand is running."
  echo "  Consoles:"
  conman -q 2>&1 | sed 's/^/    /' || true
 else
  echo "  WARNING: conmand failed to start. Checking journal..."
  journalctl -u conmand --no-pager -n 20 2>/dev/null || true
  # Try manual start as fallback
  echo "  Attempting manual start..."
  /usr/sbin/conmand -c /etc/conman.conf 2>&1 || true
 fi
 # --- 7. Summary ---
 echo ""
 echo "--- [7/7] Setup complete ---"
 echo ""
 echo "  ser2net TCP ports (connect directly):"
 echo "    telnet <tailscale-ip> 2001    # pfv-core-sw01"
 echo "    telnet <tailscale-ip> 2002    # pfv-tor3-mgmt"
 echo "    ..."
 echo ""
 echo "  conman consoles (with logging):"
 echo "    conman -f pfv-core-sw01"
 echo "    conman -q                   # query status"
 echo ""
 echo "  To regenerate after changing mapping.txt:"
 echo "    bash generate-config.sh"
 echo "    udevadm trigger"
 echo "    systemctl restart ser2net conmand"
 echo "============================================"
@@ -0,0 +1,89 @@
 #!/usr/bin/bash
 #
 # console/validate-conman.sh — verify conman can actually reach devices via
 # ser2net TCP ports and is capturing log output to files.
 #
 # This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device
 #
 set -uo pipefail
 TS_IP=$(tailscale ip -4)
 LOGDIR="/var/log/conman"
 echo "============================================"
 echo "  Conman Data Path + Log Validation"
 echo "  Host: $(hostname)   TS IP: $TS_IP"
 echo "============================================"
 echo ""
 echo "--- 1. conman.conf log settings ---"
 grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo "  (no explicit logdir — defaults to /var/log/conman)"
 echo "  Log dir: $LOGDIR"
 ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo "  ($LOGDIR does not exist yet)"
 echo ""
 echo "--- 2. CONSOLE entries: each has a log= directive? ---"
 # Extract the auto-generated block and check each CONSOLE line has log=
 sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do
  name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p')
  if echo "$line" | grep -q 'log='; then
    logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p')
    echo "  [OK] $name → log=$logfile"
  else
    echo "  [FAIL] $name has NO log= directive"
  fi
 done
 echo ""
 echo "--- 3. Trigger log capture: connect to each console briefly ---"
 # conman -e changes the escape char. We use -j (join, read-only) with a timeout.
 # Actually, conman doesn't have a built-in "connect for N seconds" — but conmand
 # connects to each device ON STARTUP and keeps the connection open for logging.
 # The log files should already be created. Let's check timestamps.
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
 for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
  logfile="$LOGDIR/${name}.log"
  if [ -f "$logfile" ]; then
    SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
    MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1)
    echo "  [OK] $logfile ($SIZE bytes, modified $MTIME)"
  else
    echo "  [MISSING] $logfile — conmand may not be writing yet"
  fi
 done
 echo ""
 echo "--- 5. conmand connection status (journal) ---"
 # conmand logs connection attempts/errors to syslog
 journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo "  (no relevant journal entries)"
 echo ""
 echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---"
 # Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded)
 echo "  Probing TCP $TS_IP:2007 for data..."
 RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
 if [ -n "$RESPONSE" ]; then
  echo "  [OK] Data flowing through ser2net TCP 2007:"
  echo "$RESPONSE" | sed 's/^/    /'
 else
  echo "  (no immediate response — device may need more interaction)"
 fi
 echo ""
 echo "--- 7. Check if conmand has open connections to ser2net ports ---"
 CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
 if [ -n "$CONMAND_PID" ]; then
  echo "  conmand PID: $CONMAND_PID"
  echo "  Open connections to ser2net (expect 7 to 100.x:200X):"
  ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/    /'
  COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
  echo "  Total conmand→ser2net connections: $COUNT (expect 7)"
 else
  echo "  [FAIL] conmand not running"
 fi
 echo ""
 echo "============================================"
@@ -177,7 +177,7 @@ dig @192.168.3.253 pfv-netinfra-01.knel.net
 # Both should return the same answer.
 ```
-The KNELServerBuild provisioning code (`ProjectCode/ConfigFiles/NTP/ntp.conf`
+The KNELServerBuild provisioning code (`provisioning/ConfigFiles/NTP/ntp.conf`
-and `ProjectCode/ConfigFiles/Resolv/resolv.conf`) points clients at both
+and `provisioning/ConfigFiles/Resolv/resolv.conf`) points clients at both
-servers for DNS and NTP redundancy. See `ProjectDocs/tailscale.md` for the
+servers for DNS and NTP redundancy. See `docs/server-build/tailscale.md` for the
 full DNS architecture analysis.
@@ -8,7 +8,7 @@
 #
 # WHY: one place to configure host aliases/users/keys, one place to audit,
 # and the command scanner only permits ssh when invoked indirectly via a
-# script. Mirrors the pattern of Project-Tests/remote.sh.
+# script. Mirrors the pattern of tests/remote.sh.
 #
 # HOSTS (override IPs via env if needed):
 #   tsrouter       tailscale-router.knel.net  (PRODUCTION — READ-ONLY here)
@@ -1,4 +1,5 @@
-# AI Review: KNELServerBuild (FetchApply) Project
+<!-- Historical AI-generated review. Paths updated to current structure. -->
 # AI Review: KNELServerBuild (PFVCluster) Project
 ## Executive Summary
@@ -6,7 +7,7 @@ The KNELServerBuild project is a comprehensive Infrastructure-as-Code (IaC) solu
 ## Project Overview
-The FetchApply project is a shell-based automation framework that provisions Linux servers with:
+The PFVCluster project is a shell-based automation framework that provisions Linux servers with:
 - Security hardening (SSH, 2FA, Wazuh, STIG compliance)
 - Operational monitoring (LibreNMS, cockpit, SNMP)
 - System packages and configurations for enterprise operations
@@ -15,10 +16,10 @@ The FetchApply project is a shell-based automation framework that provisions Lin
 ## Architecture and Structure
 ### Key Components
- **ProjectCode/**: Main setup and configuration scripts
+- **provisioning/**: Main setup and configuration scripts
 - **Project-ConfigFiles/**: Configuration variables and parameters
 - **Project-Includes/**: Reusable shell functions and utilities
- **Project-Tests/**: Comprehensive testing framework
+- **tests/**: Comprehensive testing framework
 - **Modules/**: Functional modules for security, operations, etc.
 - **vendor/**: External dependencies and frameworks
@@ -134,6 +135,6 @@ The `SetupNewSystem.sh` orchestrates:
 ## Conclusion
-The FetchApply project represents a solid foundation for automated server provisioning with good security practices and testing. However, there are significant opportunities to improve security, maintainability, and operational resilience. Prioritizing security improvements and configuration management would provide the greatest value to the project's stability and long-term viability.
+The PFVCluster project represents a solid foundation for automated server provisioning with good security practices and testing. However, there are significant opportunities to improve security, maintainability, and operational resilience. Prioritizing security improvements and configuration management would provide the greatest value to the project's stability and long-term viability.
 The modular architecture and comprehensive testing framework provide a strong foundation for future enhancements and improvements.
@@ -1,3 +1,4 @@
 <!-- Historical AI-generated review. Paths updated to current structure. -->
 # AI Overview of KNELServerBuild
 This is an AI-generated overview of the KNELServerBuild project. The analysis is based on a read-only review of the project's files.
@@ -6,7 +7,7 @@ This is an AI-generated overview of the KNELServerBuild project. The analysis is
 The KNELServerBuild project is an Infrastructure as Code (IAC) repository for provisioning and configuring Linux servers. It is based on a collection of bash scripts that automate the installation of packages, configuration of services, and security hardening of the system. The project is designed to be used with the `FetchApply` tool, which is not included in this repository.
-The main entry point of the project is the `ProjectCode/SetupNewSystem.sh` script. This script performs the following actions:
+The main entry point of the project is the `provisioning/SetupNewSystem.sh` script. This script performs the following actions:
 *   **Initializes the environment:** Sets up project paths and sources a shell framework (`KNELShellFramework`) and project-specific includes.
 *   **Installs packages:** Installs a wide range of packages, including monitoring agents (check_mk, snmp), security tools (auditd, aide, lynis, clamav), administration tools (cockpit, webmin), and common utilities (tmux, vim, zsh).
@@ -1,3 +1,4 @@
 <!-- Historical AI-generated review. Paths updated to current structure. -->
 # AI Overview: KNEL Server Build (FetchApply) Project
 **Date:** December 26, 2025  
@@ -1,3 +1,5 @@
 <!-- Historical AI-generated security review. Paths updated where actionable. -->
 <!-- Historical AI-generated review. Paths updated to current structure. -->
 # AI Security Audit of KNELServerBuild
 This is an AI-generated security audit of the KNELServerBuild project. The analysis is based on a read-only review of the project's files.
@@ -8,7 +10,7 @@ The KNELServerBuild project has a good security posture overall, but there are a
 ### High-Risk Findings
-*   **SSH Authorized Keys in Repository:** The `ProjectCode/ConfigFiles/SSH/AuthorizedKeys` directory contains SSH authorized keys for the `localuser` and `root` users. This is a security risk, as it allows anyone with access to the repository to know which public keys are authorized to access the servers.
+*   **SSH Authorized Keys in Repository:** The `provisioning/ConfigFiles/SSH/AuthorizedKeys` directory contains SSH authorized keys for the `localuser` and `root` users. This is a security risk, as it allows anyone with access to the repository to know which public keys are authorized to access the servers.
 ### Medium-Risk Findings
@@ -16,12 +18,12 @@ The KNELServerBuild project has a good security posture overall, but there are a
 ### Low-Risk Findings
-*   **Potential for Password on Command Line:** The `ProjectCode/Agents/librenms/mysql.sh` script has a `--pass` argument for a MySQL password. This is a potential security risk if the password is provided on the command line, as it could be logged in the shell history.
+*   **Potential for Password on Command Line:** The `provisioning/Agents/librenms/mysql.sh` script has a `--pass` argument for a MySQL password. This is a potential security risk if the password is provided on the command line, as it could be logged in the shell history.
 ## Recommendations
 *   **Remove SSH Authorized Keys from Repository:** The SSH authorized keys should be removed from the repository and managed using a secrets management tool like HashiCorp Vault or AWS Secrets Manager.
 *   **Use Variables for Hostnames:** The hardcoded hostnames should be replaced with variables that are defined in a central configuration file. This will make it easier to update the hostnames if they change.
-*   **Avoid Passwords on Command Line:** The `ProjectCode/Agents/librenms/mysql.sh` script should be modified to avoid passing the MySQL password on the command line. For example, the script could prompt the user for the password or read it from a configuration file.
+*   **Avoid Passwords on Command Line:** The `provisioning/Agents/librenms/mysql.sh` script should be modified to avoid passing the MySQL password on the command line. For example, the script could prompt the user for the password or read it from a configuration file.
 Overall, the KNELServerBuild project is a good starting point for an IAC repository. By addressing the security risks identified in this audit, the project can be made more secure and reliable.
@@ -1,4 +1,5 @@
-# TSYS FetchApply Code Review Findings
+<!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
 # TSYS PFVCluster Code Review Findings
 **Review Date:** July 14, 2025  
 **Reviewer:** Claude (Anthropic)  
@@ -11,7 +12,7 @@ The repository shows good architectural structure with centralized framework com
 ## Critical Issues (High Priority)
 ### 1. Package Installation Performance ⚠️
-**Location:** `ProjectCode/SetupNewSystem.sh:27` and `Lines 117-183`
+**Location:** `provisioning/SetupNewSystem.sh:27` and `Lines 117-183`
 **Issue:** Multiple separate package installation commands causing performance bottlenecks
 ```bash
 # Current inefficient pattern
@@ -23,7 +24,7 @@ DEBIAN_FRONTEND="noninteractive" apt-get -qq --yes install virt-what auditd ...
 **Fix:** Combine all package installations into single command
 ### 2. Network Operations Lack Error Handling 🔴
-**Location:** `ProjectCode/SetupNewSystem.sh:61-63`, multiple modules
+**Location:** `provisioning/SetupNewSystem.sh:61-63`, multiple modules
 **Issue:** curl commands without timeout or error handling
 ```bash
 # Vulnerable pattern
@@ -33,7 +34,7 @@ curl --silent ${DL_ROOT}/path/file >/etc/config
 **Fix:** Add timeout, error handling, and retry logic
 ### 3. Unquoted Variable Expansions 🔴
-**Location:** Multiple files, including `ProjectCode/SetupNewSystem.sh:244`
+**Location:** Multiple files, including `provisioning/SetupNewSystem.sh:244`
 **Issue:** Variables used without proper quoting creating security risks
 ```bash
 # Risky pattern
@@ -57,7 +58,7 @@ chsh -s $(which zsh) root
 ## Performance Optimization Opportunities
 ### 6. Individual File Downloads 🟡
-**Location:** `ProjectCode/Modules/Security/secharden-scap-stig.sh:66-77`
+**Location:** `provisioning/Modules/Security/secharden-scap-stig.sh:66-77`
 **Issue:** 12+ individual curl commands for config files
 ```bash
 curl --silent ${DL_ROOT}/path1 > /etc/file1
@@ -171,9 +172,9 @@ function download_configurations() {
    print_info "Downloading configuration files..."
    local -A configs=(
-        ["${DL_ROOT}/ProjectCode/ConfigFiles/ZSH/tsys-zshrc"]="/etc/zshrc"
+        ["${DL_ROOT}/provisioning/ConfigFiles/ZSH/tsys-zshrc"]="/etc/zshrc"
-        ["${DL_ROOT}/ProjectCode/ConfigFiles/SMTP/aliases"]="/etc/aliases"
+        ["${DL_ROOT}/provisioning/ConfigFiles/SMTP/aliases"]="/etc/aliases"
-        ["${DL_ROOT}/ProjectCode/ConfigFiles/Syslog/rsyslog.conf"]="/etc/rsyslog.conf"
+        ["${DL_ROOT}/provisioning/ConfigFiles/Syslog/rsyslog.conf"]="/etc/rsyslog.conf"
    )
    for url in "${!configs[@]}"; do
@@ -261,7 +262,7 @@ function test_network_error_handling() {
 ## Conclusion
-The TSYS FetchApply repository has a solid foundation but requires systematic improvements to meet production reliability standards. The recommended fixes will significantly enhance:
+The TSYS PFVCluster repository has a solid foundation but requires systematic improvements to meet production reliability standards. The recommended fixes will significantly enhance:
 - **Deployment reliability** through better error handling
 - **Security posture** through integrity verification
@@ -1,4 +1,5 @@
-# Claude Code Review - TSYS FetchApply Infrastructure
+<!-- Historical AI-generated review. Paths updated to current structure. -->
 # Claude Code Review - TSYS PFVCluster Infrastructure
 **Review Date:** July 14, 2025 (Updated)  
 **Reviewed by:** Claude (Anthropic)  
@@ -1,3 +1,4 @@
 <!-- Historical document: paths and patterns shown are pre-refactor. See provisioning/ for current code. -->
 # Code Refactoring Examples
 This document provides specific examples of how to apply the code review findings to improve performance, security, and reliability.
@@ -1,4 +1,4 @@
-# Charles TODO - TSYS FetchApply Security Improvements
+# Charles TODO - PFVCluster Security Improvements
 **Priority Order:** High → Medium → Low  
 **Target:** Address security vulnerabilities and operational improvements
@@ -8,7 +8,7 @@
 ### ✅ 1. Replace Insecure Deployment Method - RESOLVED
 **Previous Issue:** `curl https://dl.knownelement.com/KNEL/FetchApply/SetupNewSystem.sh | bash`
 **Status:** Fixed in README.md - now uses secure git clone approach
-**Current Method:** `git clone this repo` → `cd FetchApply/ProjectCode` → `bash SetupNewSystem.sh`
+**Current Method:** `git clone this repo` → `cd PFVCluster/provisioning` → `bash SetupNewSystem.sh`
 **Remaining considerations:**
 - Consider implementing GPG signature verification for tagged releases
@@ -17,9 +17,9 @@
 ### ✅ 2. Enforce HTTPS for All Downloads - RESOLVED
 **Previous Issue:** HTTP URLs in Dell OMSA and some repository setups
 **Status:** All HTTP URLs converted to HTTPS across:
-  - `ProjectCode/Dell/Server/omsa.sh` - Ubuntu archive and Dell repo URLs
+  - `provisioning/Dell/Server/omsa.sh` - Ubuntu archive and Dell repo URLs
-  - `ProjectCode/legacy/prox7.sh` - Proxmox download URLs
+  - `provisioning/legacy/prox7.sh` - Proxmox download URLs
-  - `ProjectCode/Modules/RandD/sslStackFromSource.sh` - Apache source URLs
+  - `provisioning/Modules/RandD/sslStackFromSource.sh` - Apache source URLs
 **Remaining considerations:**
 - SSL certificate validation is enabled by default in wget/curl
@@ -35,7 +35,7 @@
 - Add environment variable support for sensitive data
 **Files to secure:**
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/` (entire directory)
+- `provisioning/ConfigFiles/SSH/AuthorizedKeys/` (entire directory)
 - Hard-coded hostnames in various scripts
 ## 🔶 MEDIUM PRIORITY (Operational Security)
@@ -1,4 +1,4 @@
-# Claude TODO - TSYS FetchApply Automation Tasks
+# Claude TODO - TSYS PFVCluster Automation Tasks
 **Purpose:** Actionable items optimized for AI assistant implementation  
 **Priority:** Critical → High → Medium → Low
@@ -11,9 +11,9 @@
 ### ✅ RESOLVED: Replace HTTP URLs with HTTPS
 **Files modified:**
- `ProjectCode/Dell/Server/omsa.sh` - Converted 11 HTTP URLs to HTTPS (Ubuntu archive, Dell repo)
+- `provisioning/Dell/Server/omsa.sh` - Converted 11 HTTP URLs to HTTPS (Ubuntu archive, Dell repo)
- `ProjectCode/legacy/prox7.sh` - Converted 2 HTTP URLs to HTTPS (Proxmox downloads)
+- `provisioning/legacy/prox7.sh` - Converted 2 HTTP URLs to HTTPS (Proxmox downloads)
- `ProjectCode/Modules/RandD/sslStackFromSource.sh` - Converted 3 HTTP URLs to HTTPS (Apache sources)
+- `provisioning/Modules/RandD/sslStackFromSource.sh` - Converted 3 HTTP URLs to HTTPS (Apache sources)
 **Status:** All HTTP URLs in active scripts converted to HTTPS. Only remaining HTTP references are in comments and LibreNMS agent files (external dependencies).
@@ -39,7 +39,7 @@ function verify_download() {
 ```
 ### TASK-003: Create Secure Deployment Script
-**Create:** `ProjectCode/SecureSetupNewSystem.sh`
+**Create:** `provisioning/SecureSetupNewSystem.sh`
 **Features to implement:**
 - GPG signature verification
 - SHA256 checksum validation
@@ -50,9 +50,9 @@ function verify_download() {
 ### TASK-004: Remove Hardcoded SSH Keys
 **Files to modify:**
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/root-ssh-authorized-keys`
+- `provisioning/ConfigFiles/SSH/AuthorizedKeys/root-ssh-authorized-keys`
- `ProjectCode/ConfigFiles/SSH/AuthorizedKeys/localuser-ssh-authorized-keys`
+- `provisioning/ConfigFiles/SSH/AuthorizedKeys/localuser-ssh-authorized-keys`
- `ProjectCode/Modules/Security/secharden-ssh.sh:31,40,51`
+- `provisioning/Modules/Security/secharden-ssh.sh:31,40,51`
 **Implementation approach:**
 1. Create environment variable support: `SSH_KEYS_URL` or `SSH_KEYS_VAULT_PATH`
@@ -119,7 +119,7 @@ function update_status() { }        # Update current operation
 ```
 ### TASK-012: Dry Run Mode
-**Add to:** `ProjectCode/SetupNewSystem.sh`
+**Add to:** `provisioning/SetupNewSystem.sh`
 **Implementation:**
 - `--dry-run` flag support
 - Preview of changes without execution
@@ -139,9 +139,9 @@ function update_status() { }        # Update current operation
 ## File Location Patterns
 - **Framework components:** `Framework-Includes/*.sh`
- **Security modules:** `ProjectCode/Modules/Security/*.sh`
+- **Security modules:** `provisioning/Modules/Security/*.sh`
- **Configuration files:** `ProjectCode/ConfigFiles/*/`
+- **Configuration files:** `provisioning/ConfigFiles/*/`
- **Main entry point:** `ProjectCode/SetupNewSystem.sh`
+- **Main entry point:** `provisioning/SetupNewSystem.sh`
 ## Testing Strategy
@@ -0,0 +1,95 @@
 # Documentation Map
 > **Index of all documentation in this repo.** Agents must update this file
 > whenever a doc is added, removed, or substantively changed.
 > **Last updated:** 2026-07-28
 ## Kubernetes Architecture ([`k8s/`](k8s/))
 Distro decision, target architecture, control-plane design, bootstrap and DR
 procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`k8s/README.md`](k8s/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
 | [`k8s/DISTRO-DECISION.md`](k8s/DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: Talos, driven by ITAR/classified requirement | 2026-07-28 |
 | [`k8s/ARCHITECTURE.md`](k8s/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
 ## Proxmox Cluster ([`proxmox/`](proxmox/))
 Fleet operations, hardware, performance tuning, storage architecture.
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`proxmox/PROJECT.md`](proxmox/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
 | [`proxmox/TODO.md`](proxmox/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
 | [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
 ## Server Build ([`server-build/`](server-build/))
 Server provisioning, security hardening, DNS/NTP configuration.
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
 | [`server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
 | [`server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
 | [`server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
 | [`server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
 ## Operational Guides (outside docs/)
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
 | [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 |
 | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
 | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
 | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
 | [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
 | [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
 ## Archive ([`archive/`](archive/))
 Historical AI reviews, completed task lists, and pre-refactor examples. Read-only
 context — do not update; link to active docs instead.
 | Document | Description |
 |----------|-------------|
 | [`archive/CODE-REVIEW-FINDINGS.md`](archive/CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) |
 | [`archive/REFACTORING-EXAMPLES.md`](archive/REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) |
 | [`archive/Claude-Review.md`](archive/Claude-Review.md) | Claude's initial code review |
 | [`archive/AIReview-QWEN.md`](archive/AIReview-QWEN.md) | Qwen AI review |
 | [`archive/AiOverview-Gemini.md`](archive/AiOverview-Gemini.md) | Gemini project overview |
 | [`archive/AiOverview-OpenCode.md`](archive/AiOverview-OpenCode.md) | OpenCode project overview |
 | [`archive/AiSecurityAudit-Gemini.md`](archive/AiSecurityAudit-Gemini.md) | Gemini security audit |
 | [`archive/charles-todo.md`](archive/charles-todo.md) | Charles's early task list (completed) |
 | [`archive/claude-todo.md`](archive/claude-todo.md) | Claude's early task list (completed) |
 ## Top-Level Files
 | File | Description |
 |------|-------------|
 | [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, gardening protocol) |
 | [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) |
 | [`../README.md`](../README.md) | Project overview and quick start |
 | [`../LICENSE`](../LICENSE) | License |
 ---
 ## Agent Gardening Protocol
 When making changes to this repo, agents MUST:
 1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure
   state, completes/starts a task, or discovers a new issue.
 2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a
   substantive content change. Update the "Last Reviewed" date.
 3. **Verify cross-references** — any new `.md` file must be linked from at
   least one existing doc or this map.
 4. **Check for stale paths** — after any directory rename or file move,
   `grep -rn 'old/path' --include='*.md'` and fix all references.
 5. **Keep code and docs in sync** — if you change a script's interface,
   behavior, or location, update every doc that references it in the same
   commit.
@@ -0,0 +1,673 @@
 # pfv-k8s Target Architecture (Talos)
 > **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
 > [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
 > Proxmox audit).
 **Last updated:** 2026-07-28
 ---
 ## Table of Contents
 1. [High-Level Architecture](#1-high-level-architecture)
 2. [Control Plane](#2-control-plane)
 3. [Network Topology](#3-network-topology)
 4. [CNI: Cilium](#4-cni-cilium)
 5. [Identity and Trust](#5-identity-and-trust)
 6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
 7. [Storage Integration](#7-storage-integration)
 8. [Local Image Registry](#8-local-image-registry)
 9. [Bootstrap Procedure](#9-bootstrap-procedure)
 10. [Disaster Recovery](#10-disaster-recovery)
 11. [Migration from Current State](#11-migration-from-current-state)
 ---
 ## 1. High-Level Architecture
 ```mermaid
 flowchart TB
    subgraph RESIDENCE["Residence — Proxmox LAN"]
        subgraph CP["Talos Control Plane (3 cnodes)"]
            C1[cnode1<br/>tsys9 · local-SSD]
            C2[cnode2<br/>tsys9 · local-SSD]
            C3[cnode3<br/>tsys1 · local-HDD]
        end
        subgraph WP["Talos Worker Plane"]
            W3[wnode-tsys3<br/>NVMe · 28GB]
            W5[wnode-tsys5<br/>NVMe · 32-64GB]
            W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
            W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
            W9[wnode-tsys9<br/>local-SSD · 4-8GB]
        end
        ETCD[(etcd<br/>raft, mTLS)]
        REG[(Harbor registry<br/>on D3 SSD · tsys5)]
        BASTION[tailscale-router VM<br/>subnet router]
    end
    subgraph TAILNET["Tailscale overlay"]
        OP[Operator devices]
    end
    subgraph CLOUDRON["Cloudron production — Reston VA"]
        KC[Keycloak OIDC IdP]
    end
    C1 ---|mTLS LAN| ETCD
    C2 ---|mTLS LAN| ETCD
    C3 ---|mTLS LAN| ETCD
    CP -->|pull images| REG
    WP -->|pull images| REG
    OP -->|Talos API :50000<br/>via subnet route| BASTION
    BASTION -.->|LAN| CP
    CP -->|OIDC| KC
    WP -->|OIDC| KC
    classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
    classDef infra fill:#0f3460,stroke:#e94560,color:#fff
    classDef external fill:#16213e,stroke:#533483,color:#fff
    class CP,WP,ETCD talos
    class REG,BASTION infra
    class OP,KC,EXTERNAL external
 ```
 ### Design principles
 1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
   Strongest posture for ITAR/classified.
 2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
   advertises the cluster LAN subnet. Operator reaches Talos API from
   anywhere.
 3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
   for etcd). Workers boot from local disk where available; NFS for bulk
   data only.
 4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
   of the Talos host cluster.
 5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
   tokens. No long-lived static credentials for humans.
 ---
 ## 2. Control Plane
 ### 2.1 Recommendation: 3 cnodes (down from 5)
 | Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
 |--------|--------|-------------------|-----------------|---------------|
 | **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
 | 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
 For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
 failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
 quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
 host capacity) are better spent on tenant worker allocations.
 **Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
 tolerance on the control plane, keep 5. Otherwise 3.
 ### 2.2 Cnode placement
 Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
 **local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
 | cnode | Host | Boot disk | Type | Why |
 |-------|------|-----------|------|-----|
 | cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
 | cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
 | cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
 **Quorum survival:**
 | Failure | cnodes lost | Quorum OK? |
 |---------|-------------|------------|
 | tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
 | tsys1 host dies | cnode3 | YES (2 of 3) |
 | Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
 **Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
 loses quorum. Alternative: spread cnodes across 3 different hosts. See
 "open question" at end of this section.
 ### 2.3 Machine config strategy
 Talos nodes are configured by **machine configs** (YAML). Two flavors:
 - **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
  controller-manager, API server.
 - **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
 Strategy for this cluster:
 1. **One shared `talosconfig`** (client identity) — stored in 1Password
   and in the Proxmox Backup Server (PBS) encrypted backup target.
 2. **Per-node machine config patches** — small patches on top of the base
   `controlplane.yaml` / `worker.yaml` for node-specific settings:
   - Hostname
   - Network interface + IP (DHCP or static — recommend static for cnodes)
   - Schematic image digest (pinned Talos version)
   - System extensions (e.g., `tailscale` — only if running Pattern A
     instead of recommended Pattern C)
 3. **All machine configs in Git** under a future `k8s/talos-configs/`
   directory. Secrets are templated in at apply-time from 1Password / sops.
 ```mermaid
 flowchart LR
    BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
    BASE --> PATCH2[patch: cnode2]
    BASE --> PATCH3[patch: cnode3]
    BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
    PATCH1 --> APPLY1[talosctl apply]
    PATCH2 --> APPLY2[talosctl apply]
    PATCH3 --> APPLY3[talosctl apply]
    PATCHW --> APPLYW[talosctl apply]
 ```
 ### 2.4 Open question: cnode host spread
 If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
 cluster, tsys9 is brand-new hardware, single digit annual failure
 probability), the layout in §2.2 is fine.
 If not, alternative spread across 3 hosts:
 | cnode | Host | Boot disk |
 |-------|------|-----------|
 | cnode1 | tsys9 | local-lvm SSD |
 | cnode2 | tsys1 | local-lvm HDD |
 | cnode3 | tsys3 | local-lvm NVMe |
 tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
 [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
 would make an excellent etcd disk.
 **This 3-host spread survives any single host failure with quorum intact.
 Recommended.**
 ---
 ## 3. Network Topology
 ### 3.1 Zones
 ```mermaid
 flowchart TB
    subgraph INTERNET["Internet"]
        FIBER[Gigabit symmetric fiber]
    end
    subgraph RESLAN["Residence LAN 192.168.x.x/24"]
        subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
            CNODES[Cnodes 192.168.3.x]
            WNODES[Wnodes 192.168.3.x]
        end
        BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
        REG[Harbor registry<br/>192.168.3.x]
        STORAGE[NFS servers<br/>tsys4, tsys5]
    end
    subgraph TSNET["Tailscale 100.x.x.x/8"]
        OPS[Operator devices]
        KC[Keycloak<br/>via Cloudron prod]
    end
    FIBER --> BASTION
    BASTION <-. subnet route .-> CLUSTERNET
    OPS -->|TCP 50000 talos API| BASTION
    BASTION -->|LAN forward| CNODES
    CNODES -->|LAN mTLS| WNODES
    CNODES -->|OIDC HTTPS| KC
    WNODES -->|pull images| REG
    WNODES -->|bulk data IO| STORAGE
    CNODES -->|pull images| REG
 ```
 ### 3.2 Address plan (suggested)
 Reserve a small contiguous block in the residence LAN for cluster nodes:
 | Role | Range | Count |
 |------|-------|-------|
 | Cnodes | `192.168.3.31-33` | 3 |
 | Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
 | Bastion | existing `tailscale-router` | 1 |
 | Registry | `192.168.3.50` | 1 (Harbor) |
 **Static IPs are strongly recommended for cnodes** (etcd cluster membership
 is hostname-based; stable IPs make `talosctl` targeting simple). Workers
 can DHCP.
 ### 3.3 Firewall posture
 Each cnode/wnode has:
 - **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
  cnodes only), plus CNI ports (varies by CNI — see §4).
 - **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
 - **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
  firewall for these IPs. ITAR workloads must not be able to phone home.
 The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
 via the subnet route.
 ---
 ## 4. CNI: Cilium
 **Recommendation: Cilium** (eBPF-based CNI).
 | Property | Why it matters here |
 |----------|---------------------|
 | **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
 | **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
 | **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
 | **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
 | **Talos integration** | First-class. Talos docs document the install path. |
 Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
 enabled. Default-deny NetworkPolicy applied per namespace.
 ---
 ## 5. Identity and Trust
 ### 5.1 Trust flow
 ```mermaid
 sequenceDiagram
    autonumber
    participant Human as Operator
    participant TAIL as Tailscale
    participant BAST as Bastion
    participant TALOS as Talos API :50000
    participant KC as Keycloak (Cloudron)
    participant K8S as Kubernetes API :6443
    Human->>TAIL: Authenticate (device + SSO)
    TAIL-->>Human: Tailnet IP
    Human->>BAST: Reach bastion via tailnet
    BAST->>TALOS: Forward to LAN node :50000
    Human->>KC: OIDC login (browser)
    KC-->>Human: Bearer token (short-lived)
    Human->>TALOS: talosctl (mTLS with client cert)
    Human->>K8S: kubectl --oidc (Keycloak token)
    K8S->>KC: Validate token (introspection)
    KC-->>K8S: Valid + claims
    K8S-->>Human: Authorized response
 ```
 ### 5.2 Two distinct identity layers
 | Layer | Mechanism | Audience |
 |-------|-----------|----------|
 | **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
 | **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
 The **Talos secrets bundle** is the root of trust for the cluster. Lose it
 and you cannot operate the cluster; an attacker with it owns the cluster.
 Storage:
 1. **Primary:** 1Password (or equivalent) — operator-accessible.
 2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
 3. **NOT in Git.** Machine configs go in Git; secrets stay out.
 ### 5.3 Keycloak client configuration
 On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
 - **Authorization Code + PKCE flow** (no implicit, no password).
 - **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
  Omni/Rancher URLs if/when those are added.
 - **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
  `k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
 ---
 ## 6. Tenant Isolation (vcluster)
 ### 6.1 Why vcluster
 [vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
 plane** (API server, scheduler, controller-manager, etcd) inside a namespace
 of the host cluster. Tenant workloads run on the host's worker nodes but
 are isolated by:
 - Separate API server (tenant cannot see host cluster objects).
 - Separate RBAC and admission control.
 - Separate network policies (per-namespace).
 - Separate resource quotas.
 This aligns with the user's per-tenant plan from
 [`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
 ### 6.2 Tenant registry
 | Tenant | Compliance | Workload example | vcluster name |
 |--------|-----------|------------------|---------------|
 | RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
 | Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
 | Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
 | Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
 ```mermaid
 flowchart TB
    subgraph HOST["Talos host cluster"]
        CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
        subgraph NS["Host cluster namespaces"]
            NS_RR[ns: vc-rackrental]
            NS_SO[ns: vc-suborbital-open]
            NS_SI[ns: vc-suborbital-itar]
            NS_SLP[ns: vc-slp]
        end
    end
    subgraph VRR["vcluster: vc-rackrental"]
        API_RR[k8s API + etcd]
    end
    subgraph VSO["vcluster: vc-suborbital-open"]
        API_SO[k8s API + etcd]
    end
    subgraph VSI["vcluster: vc-suborbital-itar"]
        API_SI[k8s API + etcd]
    end
    subgraph VSLP["vcluster: vc-slp"]
        API_SLP[k8s API + etcd]
    end
    CP --> NS_RR & NS_SO & NS_SI & NS_SLP
    NS_RR --> API_RR
    NS_SO --> API_SO
    NS_SI --> API_SI
    NS_SLP --> API_SLP
    classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
    class NS_SI,API_SI itar
 ```
 ### 6.3 ITAR enforcement at host layer
 For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
 controls:
 - **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
  ITAR workloads (subset of wnodes, marked in node labels).
 - **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
  namespace. Allow only explicit destinations (registry, NFS for ITAR
  data tier, Keycloak).
 - **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
  `D3-itar` on tsys5) that no other tenant can mount.
 - **Audit:** Hubble flows + auditd on the host worker nodes capture all
  access to ITAR data.
 Rancher (or Sidero Omni) sits above this, presenting each tenant's
 vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
 access per tenant group claim.
 ---
 ## 7. Storage Integration
 Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
 | StorageClass | Provisioner | Backing | Speed | Use |
 |--------------|------------|---------|-------|-----|
 | `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
 | `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
 | `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
 ### 7.1 CSI driver notes
 - **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
  (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
  NFS server + base export path.
 - **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
  with one manifest. Uses wnode's kubelet root dir.
 ### 7.2 ITAR data isolation
 The ITAR tenant should target a dedicated NFS export, not shared
 `nfs-hdd`. Recommended:
 - Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
  `nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
  namespace via RBAC + NetworkPolicy.
 ---
 ## 8. Local Image Registry
 ### 8.1 Recommendation: Harbor on D3 SSD
 D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
 for a **Harbor** instance:
 | Property | Value |
 |----------|-------|
 | **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
 | **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
 | **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
 | **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
 ### 8.2 Pull-through cache benefit
 Cluster nodes have zero internet egress (per §3.3). Without a local cache,
 image pulls fail. With Harbor as a pull-through cache:
 ```mermaid
 sequenceDiagram
    WNODE->>HARBOR: docker pull nginx:1.25
    alt cache hit
        HARBOR-->>WNODE: layer bytes (LAN-speed)
    else cache miss
        HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
        DOCKERHUB-->>HARBOR: layer bytes
        HARBOR-->>WNODE: layer bytes (cached for next time)
    end
 ```
 Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
 machine in the cluster with container-registry internet egress, and that
 egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
 ghcr.io).
 ### 8.3 Supply-chain integrity (future)
 Harbor + Cosign lets you require that all images deployed to the ITAR
 tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
 control. Implementation deferred to a later session.
 ---
 ## 9. Bootstrap Procedure
 ### 9.1 One-time setup
 ```mermaid
 sequenceDiagram
    autonumber
    participant OP as Operator
    participant GIT as Git repo
    participant ONEPW as 1Password
    participant PBS as PBS (tsys4)
    OP->>GIT: Clone PFVCluster repo
    OP->>ONEPW: Generate Talos secrets bundle (offline)
    ONEPW-->>OP: secrets.yaml
    OP->>PBS: Backup secrets.yaml (encrypted)
    OP->>GIT: Write machine configs (no secrets)
 ```
 ### 9.2 Provision first cnode (bootstrap)
 ```mermaid
 sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C1 as cnode1
    participant ETCD as etcd (new)
    OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
    OP->>PX: qm start VMID
    C1->>C1: Boots Talos (no config yet, "maintenance mode")
    OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
    C1->>C1: Applies config, restarts services
    OP->>C1: talosctl bootstrap
    C1->>ETCD: Initialize single-node raft
    ETCD-->>C1: ready
    OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
    OP->>C1: talosctl etcd snapshot (initial backup → PBS)
 ```
 ### 9.3 Add second and third cnodes
 ```mermaid
 sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C2 as cnode2
    participant C3 as cnode3
    participant C1 as cnode1 (existing)
    OP->>PX: qm create + start cnode2 VM
    C2->>C2: Boots Talos maintenance mode
    OP->>C2: talosctl apply --patch cnode2.yaml
    C2->>C1: Join etcd cluster
    OP->>PX: qm create + start cnode3 VM
    C3->>C3: Boots Talos maintenance mode
    OP->>C3: talosctl apply --patch cnode3.yaml
    C3->>C1: Join etcd cluster
    Note over C1,C3: etcd now has 3/3 members → HA quorum
 ```
 ### 9.4 Post-bootstrap cluster configuration
 Once 3 cnodes are up and joined:
 1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
 2. **Install CSI drivers** — nfs-csi + local-path provisioner.
 3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
 4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
 5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
 6. **Apply default-deny NetworkPolicy** in all namespaces.
 7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
 8. **First etcd snapshot** + automated daily snapshot cron → PBS.
 ### 9.5 Add workers
 Workers are simpler (no etcd):
 ```mermaid
 sequenceDiagram
    OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
    WNODE->>WNODE: Boots maintenance mode
    OP->>WNODE: talosctl apply --patch worker-X.yaml
    WNODE->>C1: Kubelet registers with API server
    C1-->>WNODE: Approved (auto via bootstrap token)
    Note over WNODE: Joins cluster, becomes Ready
 ```
 ---
 ## 10. Disaster Recovery
 ### 10.1 Backup strategy
 | Artifact | Frequency | Storage | Tool |
 |----------|-----------|---------|------|
 | **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
 | **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
 | **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
 | **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
 | **Harbor metadata** | Daily | PBS | Harbor built-in backup |
 ### 10.2 Restore scenarios
 **Lost 1 cnode (e.g., tsys9 disk failure):**
 1. Provision new VM on tsys9 (or other host with local SSD).
 2. Apply cnode2 machine config patch.
 3. New cnode joins etcd, syncs state from survivors.
 4. Quorum was never lost (2 of 3 alive throughout).
 **Lost 2 cnodes simultaneously (quorum lost):**
 1. Use surviving cnode's etcd snapshot.
 2. Provision 3 new cnode VMs.
 3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
 4. Join other 2 cnodes.
 5. Workers reconnect automatically once API server is back.
 **Total cluster loss (all 3 cnodes):**
 1. Restore from latest PBS etcd snapshot.
 2. Provision new cnode VMs.
 3. `talosctl bootstrap --recover-from=snapshot.db`.
 4. Re-join workers.
 5. Verify tenant vclusters restored.
 ### 10.3 Recovery time objectives
 | Scenario | RTO | RPO |
 |----------|-----|-----|
 | Single cnode failure | < 30 min | 0 (no data loss) |
 | Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
 | Total cluster loss | < 4 hours | ≤ 24 hours |
 ---
 ## 11. Migration from Current State
 ### 11.1 Current state
 - 5 cnode VMs exist (Debian stock + Tailscale).
 - **No k3s deployed yet.** Cluster was never bootstrapped.
 - 6 wnode VMs exist (some stopped).
 - No workloads running in k8s.
 ### 11.2 Migration: clean cutover (not a migration)
 Since there is no etcd data and no workloads to preserve, the path is a
 **clean rebuild**:
 | Phase | Action | Risk |
 |-------|--------|------|
 | **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
 | **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
 | **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
 | **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
 | **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
 | **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
 ### 11.3 Open dependency: Friday hardware work
 Phases 1-2 require:
 - **tsys3 local-lvm available.** Per
  [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
  free NVMe local-lvm. Currently unused. **Ready.**
 - **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
 - **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
  [`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
  Harbor depends on D3 being available on tsys5.
 Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
 Harbor registry does.
 ---
 ## Appendix: Open questions for next session
 1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
 2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
   (tsys9, tsys1, tsys3).
 3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
 4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
   Defer until cluster is up.
 5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
   approval of `192.168.3.0/24` route advertisement.
 6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
   Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
   general capacity; tsys3 + tsys5 (local fast storage) reserved for
   non-ITAR HPC.
@@ -0,0 +1,188 @@
 # Distro Decision: Talos Linux vs k3s
 > **Recommendation: Talos Linux.**
 > The k3s-on-Debian plan was sound before the ITAR/classified requirement
 > entered scope. Once classified workloads are on the table, Talos's
 > immutable, API-only, measured-boot-capable posture is materially easier
 > to certify and defend.
 **Last updated:** 2026-07-28
 ---
 ## 1. Decision context
 | Factor | Constraint |
 |--------|-----------|
 | **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
 | **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
 | **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
 | **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
 | **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
 | **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
 ---
 ## 2. Head-to-head comparison
 ### 2.1 ITAR / classified posture
 | Property | Talos Linux | k3s on Debian |
 |----------|-------------|---------------|
 | **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
 | **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
 | **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
 | **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
 | **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
 | **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
 | **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
 | **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
 **Bottom line:** For classified workloads, an auditor's first question is
 "how do you prevent unauthorized changes to a node?" Talos's answer is
 "the OS is immutable and the only path is a signed API call." k3s's answer
 is"SSH is locked down and we scan with STIG." The first is structurally
 stronger; the second is operationally maintained.
 ### 2.2 Operational considerations
 | Property | Talos | k3s |
 |----------|-------|-----|
 | **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
 | **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
 | **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
 | **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
 | **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
 | **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
 | **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
 ### 2.3 Cost of choosing Talos over the existing k3s plan
 The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
 **no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
 - **No etcd data to migrate.** Clean cutover, not a migration.
 - **No workloads to drain.** The cluster is empty.
 - **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
  either way it's a `qm` script, not a stateful migration).
 - **Tailscale config shifts** from "installed via apt" to "Talos system
  extension." (Or, per our recommendation in
  [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
  nodes entirely** and onto the existing `tailscale-router` bastion as a
  subnet router. Cluster nodes become LAN-only.)
 **Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
 machine config YAML. The hardening investment already encoded in
 `provisioning/Modules/Security/` is **not wasted** — it still applies to
 every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
 the cnodes/wnodes move to Talos.
 ---
 ## 3. Tailscale compatibility (deep-dive)
 Tailscale on Talos is well-supported but introduces a configuration dimension
 worth being explicit about. Three patterns exist:
 ### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
 Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
 internet egress (to Tailscale DERP servers and for coordinate).
 - **Talos implementation:** add `siderolabs/tailscale` system extension to
  each machine config, configure `machine.network.interfaces`.
 - **Pros:** Operator can hit any node's Talos API from any Tailscale device.
 - **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
  finding (data exfiltration path).
 ### Pattern B — Tailscale on bastion only, SSH/API jump
 Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
 `tailscale-router` VM), then runs `talosctl` from the bastion.
 - **Pros:** Zero internet egress from cluster nodes.
 - **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
  action originates from the bastion (auditable but clunky).
 ### Pattern C — Tailscale subnet router on bastion (recommended)
 The existing `tailscale-router` VM advertises the cluster LAN subnet
 (e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
 Tailscale client transparently routes cluster-bound traffic through the
 bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
 "just works."
 - **Pros:**
  - Cluster nodes have **zero internet egress** (strongest ITAR posture).
  - Operator UX is unchanged from direct LAN access.
  - All access is mediated by Tailscale's identity + ACLs (already integrated
    with your env).
  - Audit trail lives in Tailscale + bastion logs.
 - **Cons:**
  - Bastion becomes a dependency for remote admin (LAN-local admin still
    works without it).
  - Must enable IP forwarding + subnet route approval in Tailscale ACLs.
 **Recommendation: Pattern C.** Documented in
 [`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
 ---
 ## 4. Recommendation
 **Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
 ### Justification
 1. **Compliance posture is structural, not operational.** "Immutable,
   API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
   is a property of how Debian is operated. The first is dramatically
   easier to argue to an ITAR counsel or classified accreditation officer
   (DSS, DCSA) than the second.
 2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
   3 cnodes with Talos is a `qm` script invocation, not a stateful
   migration. The sunk cost of "we planned k3s" is **zero deployed state**.
 3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
   `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
   learn but more error-prone to operate at HA.
 ### Acknowledged tradeoffs
 - **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
  replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
 - **No shell debugging.** When something breaks on a node, you cannot `ssh`
  in. Mitigation: `talosctl logs/support` produces a support bundle
  equivalent to a sosreport.
 - **Hardware/module surprises.** Talos ships a curated kernel. Anything
  beyond virtio + common NIC drivers needs a system extension. On Proxmox
  VMs this is **not expected to be a problem** — virtio is the path.
 - **Tailscale via system extension.** Adds one config dimension per node.
  Mitigated by Pattern C (above), which removes Tailscale from cluster
  nodes entirely.
 ### What we keep from the k3s mental model
 - **Single binary on each node** semantics (Talos is conceptually similar).
 - **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
  API. `kubectl`, `helm`, `kustomize` all work as-is.
 - **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
  distro-independent.
 ---
 ## 5. What we are NOT deciding here
 | Topic | Deferred to |
 |--------|-------------|
 | ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
 | HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
 | Per-tenant vcluster policy templates | Future session, post-bootstrap |
 | Solar-aware scale-out hosts | Future capacity planning session |
 | Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
 ---
 ## 6. Next step
 Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
 design, network topology, identity flow, and bootstrap procedure.
@@ -0,0 +1,44 @@
 # Kubernetes Architecture & Build Plan
 > **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
 > (which captured the storage/host analysis from the Proxmox audit).
 > This directory takes the next step: **which distro, how to build it,
 > how to operate it.**
 **Last updated:** 2026-07-28
 ## Documents in this directory
 | Document | Purpose |
 |----------|---------|
 | [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
 | [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
 ## TL;DR
 | Decision | Recommendation | Why |
 |----------|----------------|-----|
 | **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. |
 | **Runtime** | **containerd** | Talos/k3s default. |
 | **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. |
 | **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. |
 | **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. |
 | **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. |
 | **Multi-tenancy** | **vcluster** (per tenant) | Future work. |
 | **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. |
 | **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. |
 ## What this directory does NOT cover (deferred)
 - ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
 - HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
 - vcluster per-tenant policy templates.
 - Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
 These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
 ## Open question for the user
 1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
   [`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
   2-failure tolerance on the control plane, keep 5.
@@ -0,0 +1,485 @@
 # K8S.md -- Kubernetes Architecture Deep-Dive
 **Date:** 2026-07-27
 **Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
 Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment).
 **Status:** For discussion in a future session. No changes made.
 ---
 ## Table of Contents
 1. [Workload Profile](#1-workload-profile)
 2. [Current State](#2-current-state)
 3. [Target Architecture](#3-target-architecture)
 4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
 5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
 6. [Storage Class Design](#6-storage-class-design)
 7. [ETL/HPC Considerations](#7-etlhpc-considerations)
 8. [Migration Plan](#8-migration-plan)
 ---
 ## 1. Workload Profile
 This cluster runs **R&D and RackRental (containerlab) workloads** via
 Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
 Reston, VA running Cloudron.
 **Workload types expected:**
 | Type | Description | Storage need | RAM need | Examples |
 |------|------------|-------------|----------|---------|
 | **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
 | **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
 | **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
 **Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
 sequential I/O). HPC and containerlab need low-latency random I/O (local
 SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
 ---
 ## 2. Current State
 ### 2.1 pfv-k8s nodes and their storage
 | Node | Type | Host | Storage | Disk type | Status |
 |------|------|------|---------|-----------|--------|
 | cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
 | cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
 | cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
 | cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
 | cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
 | wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
 | wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
 ### 2.2 Problems
 1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
   etcd quorum lost.
 2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
 3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
 4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
   (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
 5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
 6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
 ---
 ## 3. Target Architecture
 ### 3.1 Design principles
 1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
   heavy workers.
 2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
   storage server failing.
 3. **Wnode boot disks on local storage** -- eliminates NFS latency for
   container runtime and kubelet.
 4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
 5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
 6. **One wnode per hypervisor host** -- maximize total cluster capacity.
 ### 3.2 Target node-host-storage matrix
 | Node | Type | Host | Boot disk | Data disk | Disk type |
 |------|------|------|-----------|-----------|-----------|
 | cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
 | cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
 | cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
 | cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
 | cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
 | wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
 | wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
 | wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
 | wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
 | wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
 | wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
 | wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
 ### 3.3 Storage server distribution after changes
 | Storage server | cnodes | wnodes (boot) | wnodes (data) |
 |---------------|--------|---------------|---------------|
 | tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
 | tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
 | tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
 | tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
 **Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
 NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
 than NFS-HDD, and the user has chosen not to install internal drives.
 **No single disk or server is a quorum-losing failure point.**
 ---
 ## 4. Control Plane (Cnodes)
 ### 4.1 Cnode sizing
 Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
 kubernetes control plane components (API server, scheduler, controller-manager).
 ### 4.2 Cnode host placement rationale
 | Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
 |------|--------|---------------|---------------|------------------------|
 | tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
 | tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
 | tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
 **tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
 ### 4.3 Cnode storage placement rationale
 The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
 | Disk | cnodes | Rationale |
 |------|--------|-----------|
 | D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
 | D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
 | S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
 | S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
 **If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
 **If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
 **If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
 tsys5. **Only 2 of 5 -- QUORUM LOST.**
 Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
 cnodes and only have 2 on tsys5. That loses quorum (need 3).
 **Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
 3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
 tsys4. Same problem inverted.
 The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
 3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
 holding 2 cnodes is safe (3 survive).
 **Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
 other.** After Friday's hardware work:
 - tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
 - tsys5 will have bond0 fixed + NVMe -- more reliable
 Either way, 3/2 split means one server failure could lose quorum. **To truly
 solve this, use a 3rd storage target.** Options:
 - Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
  independent failure domain)
 - Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
  no NFS dependency at all)
 **Best option: put cnode boot disks on local-lvm where available.** This
 eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
 its own host's local disk -- no shared dependency.
 | cnode | Host | **Recommended storage** | Type |
 |-------|------|------------------------|------|
 | cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
 | cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
 | cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
 | cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
 | cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
 With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
 down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
 survives.
 **This is the recommended approach.** Local storage for cnodes wherever
 possible. NFS only as fallback.
 ### 4.4 etcd performance on local vs NFS
 | Storage | Typical fsync latency | etcd commit latency | Impact |
 |---------|----------------------|--------------------|--------|
 | NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
 | NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
 | Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
 | Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
 | NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
 etcd is the heartbeat of the kubernetes control plane. Every API call, every
 pod schedule, every controller reconciliation involves an etcd write. Cutting
 etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
 responsive. **This is the single highest-impact change for k8s performance.**
 ---
 ## 5. Worker Nodes (Wnodes)
 ### 5.1 One wnode per hypervisor host
 | Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
 |------|-------|-----------|-----------|-----------|-----------|------|
 | tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
 | tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
 | tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
 | tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
 | tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
 | tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
 | tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
 ### 5.2 Why boot disks on local-lvm
 Current: all wnodes boot from NFS. Every container image pull, every kubelet
 log write, every ephemeral volume traverses the NFS network path.
 With local-lvm boot disks:
 - **Container image pulls** write to local disk (100-150 MB/s HDD, no network
  hop) instead of NFS-HDD (80-120 MB/s with network latency)
 - **kubelet logs** stay local (no NFS writes for log rotation)
 - **ephemeral storage** (emptyDir volumes) uses local disk by default
 - **NFS server failure does not kill the wnode** -- the VM stays running, only
  the data disk (if mounted) goes away
 ### 5.3 Wnode sizing guidance
 | Host | Recommended wnode config | Rationale |
 |------|------------------------|-----------|
 | tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
 | tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
 | tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
 | tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
 | tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
 | tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
 | tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
 ### 5.4 Tainting and labeling strategy
 Label wnodes by capability so the k8s scheduler can target them:
 ```yaml
 # Heavy RAM hosts (ETL/HPC)
 wnode-tsys6: workload=heavy, ram=128g
 wnode-tsys7: workload=heavy, ram=192g
 # NVMe host (I/O-intensive HPC)
 wnode-tsys5: workload=hpc, storage=nvme
 # SSD boot host (low-latency)
 wnode-tsys9: workload=light, storage=ssd
 # General workers
 wnode-tsys3: workload=general
 wnode-tsys2: workload=storage-fast, storage=nvme
 wnode-tsys1: workload=light
 ```
 Then use nodeSelector or nodeAffinity in job specs:
 ```yaml
 # Weather/GIS ETL job -- needs lots of RAM
 spec:
  nodeSelector:
    workload: heavy
 # Firmware build -- needs fast storage
 spec:
  nodeSelector:
    storage: nvme
 ```
 ---
 ## 6. Storage Class Design
 ### 6.1 Proposed StorageClasses
 | StorageClass | Provisioner | Where | Speed | Use case |
 |-------------|------------|-------|-------|----------|
 | `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
 | `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
 | `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
 ### 6.2 How this maps to wnode disk topology
 Each wnode has:
 - **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
  runtime. k8s `local-fast` StorageClass provisioner points here.
 - **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
  second block device or filesystem. k8s `nfs-hdd` provisioner points here.
 Inside k8s, pods request storage via PVC:
 ```yaml
 # ETL job: needs bulk storage for weather data
 apiVersion: v1
 kind: PersistentVolumeClaim
 spec:
  storageClassName: nfs-hdd
  accessModes: [ReadWriteMany]  # NFS allows RWX
  resources:
    requests:
      storage: 500Gi
 # HPC job: needs fast scratch
 spec:
  storageClassName: local-fast
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
 ```
 ### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
 **Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
 scratch space, with the exception of ultix-streaming which stays on T5-SSD.
 Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
 SIEM, etc.).
 The SSD NFS exports:
 - **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
  cache, ephemeral volumes). Currently 99% empty.
 - **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
  occupies 83 GB. Remaining 140 GB available for k8s use.
 **tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
 This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
 tsys5 SSD exports, bulk PVCs target either server.
 ### 6.4 NFS data distribution across storage servers
 To avoid re-creating the "everything on tsys4" problem, distribute NFS data
 disks across both servers:
 | wnode | Boot (local) | Bulk data (NFS) | NFS server |
 |-------|-------------|-----------------|------------|
 | wnode-tsys3 | local-lvm | S3 | tsys5 |
 | wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
 | wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
 | wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
 | wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
 This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
 ---
 ## 7. ETL/HPC Considerations
 ### 7.1 Weather/GIS ETL pipeline
 Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
 store results.
 | Stage | Storage class | Why |
 |-------|-------------|-----|
 | Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
 | Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
 | Store results | `nfs-hdd` | Large sequential writes. Persistent. |
 **Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
 wnode. ETL jobs use `/scratch` for intermediate processing and write final
 output to the NFS-mounted `/data`.
 ### 7.2 HPC workloads (hardware startup)
 Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
 | Workload | Best wnode | Why |
 |----------|-----------|-----|
 | RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
 | Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
 | Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
 | Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
 ### 7.3 RackRental/containerlab
 Rapid container deployment. Key need: fast container image pulls.
 This is where **local-lvm boot disks** shine. Currently, every container image
 pull writes through NFS to a spinning disk -- slow. With local-lvm, images
 cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
 On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
 ### 7.4 Data locality for ETL
 For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
 it on local-lvm of the heavy hosts:
 ```
 tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/  -- fastest cache tier
 tsys5 NVMe (local): /data/cache/gis/                 -- fastest cache tier
 ```
 **Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
 be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
 This avoids re-reading the same data from the same NFS export on every job
 if the data is already cached in the page cache.
 ---
 ## 8. Migration Plan
 **Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
 Datacenter Manager (PDM)** manages them collectively and supports VM migration
 between nodes. Storage migration can be done via the PDM/Proxmox UI rather
 than manual disk copies -- the destination node just needs access to the target
 storage (which all nodes have for NFS exports, and local storage can be
 migrated through the UI's "Storage Migrate" function).
 ### 8.1 Phase 1: Friday (after hardware work)
 After tsys5 cable + NVMe and tsys4 NIC + RAM:
 1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
 2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
   do not use for VM storage. Recreate on D2 or D5 NFS.
 3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
 ### 8.2 Phase 2: Cnode rebalance (maintenance window)
 These changes require creating new VMs on target hosts and migrating disks.
 Plan for a maintenance window with the k8s cluster briefly down.
 1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
 2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
 3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
 4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
 5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
 ### 8.3 Phase 3: Wnode local storage migration (maintenance window)
 1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
 2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
 3. wnode-tsys7 stays on NFS (same reason).
 4. Recreate wnode-tsys5 with boot disk on NVMe.
 5. Add data disks (NFS) as second SCSI devices where applicable.
 ### 8.4 Phase 4: tsys2 integration (when rebuilt)
 1. Install Proxmox on tsys2.
 2. Run `scripts/check.sh` to inventory.
 3. Run `scripts/apply-tunings.sh --apply`.
 4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
 5. Join to k8s cluster.
 ### 8.5 Phase 5: Critical VM relocation
 1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
 2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
 3. (No change to T5-SSD -- ultix-streaming stays.)
 ---
 ## Open questions for next session
 1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
   whether live migration is available (huge simplification) or we need manual
   disk migration. Check `pvecm status` on each host.
 2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
   nodes are joined/drain and how StorageClasses are configured.
 3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
 4. **Is there a container image registry mirror in the cluster?** Or do all
   pulls go to Docker Hub / external? A local registry on D3 SSD would speed
   up all pulls.
 5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
   whether jobs need shared (RWX) or exclusive (RWO) storage.
 6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
   how we label and taint nodes.
@@ -0,0 +1,857 @@
 # Proxmox Cluster Project Report
 **Date:** 2026-07-27 (re-audited)
 **Prepared by:** Performance Optimization Engagement
 **Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
 **Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
 `deploy-check.sh`. VM placements reflect live state after user's PDM
 migrations. This is ground truth.
 ---
 ## Table of Contents
 1. [Executive Summary](#1-executive-summary)
 2. [Host Fleet](#2-host-fleet)
 3. [Storage Architecture](#3-storage-architecture)
 4. [VM Fleet Inventory](#4-vm-fleet-inventory)
 5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
 6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
 7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
 8. [Role Alignment Audit](#8-role-alignment-audit)
 9. [Network Findings](#9-network-findings)
 10. [Recommendations](#10-recommendations)
 11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
 12. [Open Items](#12-open-items)
 ---
 ## 1. Executive Summary
 The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
 running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
 performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
 are blocked on physical hardware work scheduled for Friday.
 **Progress since initial audit:** The user has been actively rebalancing k8s
 nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
 73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
 needed for etcd quorum survival.
 The VM-layer assessment reveals:
 | # | Finding | Severity | Status |
 |---|---------|----------|--------|
 | 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
 | 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
 | 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
 | 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
 ---
 ## 2. Host Fleet
 ### 2.1 Inventory
 | Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
 |------|-------|-----------|-------|-----|-----------|-----------------|--------|
 | pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
 | pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
 | pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
 | pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
 | pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
 ### 2.2 Role taxonomy (per user directive)
 | Role | Hosts | Workload |
 |------|-------|----------|
 | **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
 | **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
 | **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
 **Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
 RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
 (tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
 run one per hypervisor host across the fleet to maximize total cluster capacity.
 pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
 Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
 running Cloudron -- not in this cluster.
 ---
 ## 3. Storage Architecture
 ### 3.1 NFS exports from tsys4 (primary storage server)
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
 | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
 | ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
 | D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
 Non-exported disks on tsys4:
 - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
 - sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
 - sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
 **D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
 tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
 eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 /sdc/sdd), so tsys5 is the better target. See section 3.2.
 ### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
 | S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
 | S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
 | S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
 | S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
 | T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
 | **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
 **tsys5 storage controllers (plenty of free ports):**
 - LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
  **5 free**
 - Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
 - Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
 - 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
 **Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
 wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
 (unused, 7 GB). S4 still 99% empty (435 GB free).
 T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
 **Friday additions:**
 1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
   USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
 2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
   boot disk and HPC scratch. The fastest tier in the fleet.
 After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
 NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
 natural home for latency-sensitive workloads and the k8s StorageClass design
 center.
 ### 3.3 Local storage tiers (per host, with utilization)
 | Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
 |------|-----------|-----------|-----|-------|------|------|-------------|
 | tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
 | **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
 | tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
 | tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
 | tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
 | **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
 | **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
 | **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
 | **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
 | **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
 **CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
 The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
 single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
 260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
 (USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
 **USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
 NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
 hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
 for VM storage.
 Additionally, both servers have completely empty internal drive bays:
 - **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
 drives.
 - **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
 DVD-ROM on port 5). No internal drives.
 This is a reliability risk beyond performance: the entire host OS boots
 from a consumer-grade portable USB drive not designed for 24/7 server use.
 **tsys3 correction:** Previously documented as HDD. Actually boots from a
 **Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
 fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
 **Critical observation: every k8s host has 0% used local-lvm.** tsys3
 (Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
 1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
 storage.
 **However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
 tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
 NFS-over-HDD and unsuitable for wnode boot disks.
 ### 3.4 Disk speed tiers summary
 | Tier | Where | Speed class | Best for |
 |------|-------|------------|----------|
 | **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
 | **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
 | **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
 | **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
 | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
 | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
 ### 3.5 Storage tier characterization per host
 | Host | Storage profile | Detail |
 |------|----------------|--------|
 | **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
 | **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
 | **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
 | **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
 | **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
 | **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
 | **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
 ### 3.6 All exports are single-disk with no redundancy
 Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
 mirror, no mdraid. A single disk failure takes down every VM whose disk lives
 on that export. This applies to **both storage servers** and to the **PBS
 backup target** (a single 4.5T SMR drive).
 ---
 ## 4. VM Fleet Inventory
 ### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
 #### tsys1 (Infrastructure) -- 11 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
 | 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
 | 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys3 (Kubernetes) -- 1 running VM
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
 #### tsys4 (Storage) -- 1 running VM
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
 #### tsys5 (Storage) -- 15 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
 | 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
 | 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
 | 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
 | 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
 | 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
 | 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
 #### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
 | 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
 | 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys7 (Kubernetes) -- 4 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
 | 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys9 (Infrastructure) -- 5 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
 | 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
 | 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
 ### 4.2 Storage concentration summary
 | Storage target | # of running VMs | % of fleet |
 |---------------|-----------------|------------|
 | **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
 | D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
 | local-nonprod (tsys5 local HDD) | 10 | 25% |
 | S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
 | T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
 | local-lvm (tsys4 local) | 1 | 3% |
 **27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
 If tsys4 goes offline, two-thirds of the fleet loses its storage.
 ---
 ## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
 ### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
 | VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
 |------|------|------------|---------|-----------|----------|
 | 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
 | 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
 | 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
 | 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
 | 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
 **Storage distribution:**
 | Storage server | cnodes | Quorum impact if it fails |
 |---------------|--------|--------------------------|
 | tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
 | tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
 **Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
 still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
 **Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
 tsys9) -- good host diversity.
 ### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
 | VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
 |------|------|------------|---------|-----------|-----|--------|----------|
 | 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
 | 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
 | 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
 | 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
 | 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
 | 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
 **One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
 stopped. wnode-tsys6 restarted.
 **Storage distribution:**
 | Storage server | wnodes | Notes |
 |---------------|--------|-------|
 | tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
 | tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
 ### 5.3 Summary: k8s node storage distribution
 | Storage server | cnodes | wnodes | Total k8s nodes |
 |---------------|--------|--------|-----------------|
 | **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
 | **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
 | Local SSD/NVMe | 0 | 0 | 0 (0%) |
 **Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
 cnodes on each storage server (60/40 or better) so either server failing
 leaves quorum intact.
 ### 5.4 Remaining cnode migration needed for etcd quorum survival
 To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
 be on tsys5:
 | Action | Effect |
 |--------|--------|
 | Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
 | Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
 | Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
 **Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
 PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
 ### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
 The k8s layer will be tackled soon. Key requirements from user:
 - **Platform:** vcluster + Rancher for multi-tenant management
 - **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
 - **Workload isolation (vcluster per tenant):**
  - RackRental workloads (containerlab network labs)
  - Suborbital ITAR (compliance-restricted)
  - Suborbital non-ITAR
  - Starting Line Productions customer workloads
 - **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
  will be brought online during peak solar production for burst capacity.
  These older cores/ram supplement the main fleet when power is abundant.
 - **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
  both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
  leftover host capacity) to large (28-32 GB, consuming most of a host).
 - **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
  full final audit will be performed at that time (post-NVMe install on tsys5,
  post-D3 SSD relocation, post-tsys4 NIC+RAM).
 ### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
 tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
 | VMID | Name | Storage |
 |------|------|---------|
 | 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
 | 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
 These are on local storage (good -- no NFS dependency) but on a single host's
 single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
 ---
 ## 6. Storage Redundancy Analysis
 ### 6.1 -01/-02 infrastructure pair audit
 Two -01/-02 pairs exist in the fleet:
 **Pair 1: pfv-netinfra (network infrastructure)**
 | Role | VMID | Host | Storage | NFS Server |
 |------|------|------|---------|-----------|
 | -01 | 103 | tsys1 | D5 | **tsys4** |
 | -02 | 904 | tsys9 | D2 | **tsys4** |
 **Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
 Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
 disks. The -02 half should be on an S2/S3/S4 export from tsys5.
 **Pair 2: tsys-ucs (Univention Corporate Server)**
 | Role | VMID | Host | Storage | NFS Server |
 |------|------|------|---------|-----------|
 | -01 | 108 | tsys1 | D2 | **tsys4** |
 | -02 | 902 | tsys9 | D5 | **tsys4** |
 **Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
 Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
 ### 6.2 Redundancy principle for paired VMs
 For any -01/-02 pair to survive a single storage server failure:
 ```
 -01 VM disk -> tsys4 NFS export (D2/D3/D5)
 -02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
 ```
 This ensures that losing either tsys4 or tsys5 takes down only one half of
 the pair. Currently, **both pairs fail this test** because both halves are on
 tsys4.
 ### 6.3 NFS server failure blast radius
 If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
 | Impact | Count |
 |--------|-------|
 | k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
 | k8s wnodes that lose storage | 4 of 5 |
 | Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
 | Total VMs that lose storage | **27 of 40 (68%)** |
 If **tsys5** goes offline:
 | Impact | Count |
 |--------|-------|
 | k8s cnodes that lose storage | 0 of 5 |
 | k8s wnodes that lose storage | 1 of 5 |
 | Total VMs that lose storage | 1 of 40 (3%) |
 **tsys4 is a massive blast-radius liability. tsys5 is barely used.**
 Rebalancing VM storage across both servers dramatically reduces risk.
 ---
 ## 7. Local SSD/NVMe Opportunity
 ### 7.1 Available fast tiers (currently unused by k8s)
 | Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
 |------|--------|------|------|--------------------|-----------------------|
 | tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
 | tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
 | tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
 | tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
 ### 7.2 Why local storage matters for k8s nodes
 Kubernetes nodes are latency-sensitive in two specific areas:
 1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
   On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
   client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
   latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
   This directly affects k8s API responsiveness and pod scheduling speed.
 2. **Container image pulls:** Worker nodes pull container images frequently.
   On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
   eliminates the network hop and reduces seek time. This matters most for
   RackRental/containerlab workloads that spin up containers rapidly.
 ### 7.3 Current waste: tsys9 local SSD
 tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
 **Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
 idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
 its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
 ### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
 The PCI NVMe being added to tsys5 will be the fastest storage tier in the
 fleet. Two placement options:
 **Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
 VMs that might need migration. Adds the NFS/network overhead back.
 **Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
 network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
 accessed from other hosts.
 **Recommendation:** Option B (local-only). k8s worker nodes do not need
 shared storage -- pods are ephemeral and reschedule on failure. The NVMe
 should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
 and used for local VM images.
 ---
 ## 8. Role Alignment Audit
 Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
 plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
 **correct by design** (keeps heavy hosts free for workers).
 ### 8.1 VMs that need to move
 | VMID | Name | Current host | Issue | Target |
 |------|------|-------------|-------|--------|
 | 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
 | 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
 ### 8.2 Host capacity for k8s nodes
 | Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
 |------|------|-------------------|-------------|-----------|----------|
 | tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
 | tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
 | tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
 | tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
 | tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
 | tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
 **tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
 free RAM respectively. They should be the primary targets for heavy worker
 nodes and ETL/HPC workloads.
 ### 8.3 tsys6 wnode-tsys6 is stopped
 VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
 NFS). This wnode should be restarted (or recreated on local-lvm) to restore
 cluster capacity.
 ---
 ## 9. Network Findings
 ### 9.1 tsys9 storage NIC is a USB dongle (new finding)
 Validating tsys9 revealed that its storage network interface
 (`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
 anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
 management; storage uses the USB adapter.
 **Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
 wobble, ESD, and USB controller resets. For an infrastructure host with 5
 VMs, this is a reliability risk.
 **Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
 PCIe NIC would eliminate this risk (same recommendation as tsys4).
 ### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
 | Host | Blocker | Staged fix |
 |------|---------|-----------|
 | tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
 | tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
 ### 9.3 LACP resolved on tsys6/tsys7
 tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
 The switch LACP hash change took effect after renegotiation. The 56-106K
 retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
 LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
 ### 9.4 NFS nconnect=4 + noatime confirmed active
 All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
 options. Each host maintains 4 TCP connections per NFS mount to each storage
 server. This was the Tier 0 tuning item from the performance optimization
 engagement and is confirmed working cluster-wide.
 ---
 ## 10. Recommendations
 **No changes have been made. These are assessment-only recommendations.**
 See `K8S.md` for the detailed kubernetes architecture deep-dive.
 ### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
 These are the most critical production VMs in the fleet. They must survive
 any single-point failure (host, storage server, or disk).
 **Design principle for -01/-02 HA pairs:**
 - Different hypervisors (already satisfied: tsys1 vs tsys9)
 - Different storage servers (currently FAILED: all on tsys4)
 - Prefer SSD for latency-sensitive services
 **Recommended placement:**
 | VM | Host | Storage | Tier | Free space | Rationale |
 |----|------|---------|------|-----------|-----------|
 | **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
 | **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
 | **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
 | **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
 **Failure survival matrix (all single-point failures):**
 | Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
 |---------|-------------|-------------|--------|--------|--------|
 | tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
 | tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
 | tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
 | tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
 | Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
 **Why this works:** Every row has at least one netinfra and one UCS alive.
 The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
 **D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
 Friday) is freed for latency-sensitive workloads that actually benefit from
 SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
 registry mirror. Not infrastructure VMs.
 ### 10.2 CRITICAL: Cnode (control plane) storage split
 **Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
 quorum (3 of 5 cnodes share D2).
 **Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
 tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
 | cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
 |-------|-------------|----------------|----------------|-------------------|-----------|
 | cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
 | cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
 | cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
 | cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
 | cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
 **Result after changes:**
 | Host | cnodes | Storage server |
 |------|--------|---------------|
 | tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
 | tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
 | tsys3 | cnode5 (S3) | tsys5 |
 - 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
  etcd keeps quorum (3 of 5 survive).
 - D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
  holds more than 2 cnodes.
 - Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
  at least 3 cnodes alive.
 - **tsys6 and tsys7 are fully freed** for heavy worker nodes.
 Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
 the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
 cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
 ### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
 **Target: one wnode per hypervisor host, using local storage where possible
 and NFS-HDD for bulk data.**
 | wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
 |-------|------|--------------------------|---------------------|-----------|
 | wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
 | wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
 | wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
 | wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
 | wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
 | wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
 | wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
 **Storage tiering strategy per wnode:**
 Each wnode gets two disk tiers mapped to k8s StorageClasses:
 1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
   images, ephemeral storage. This is where local SSD/NVMe shines -- container
   image pulls and layer extraction are seek-bound and benefit enormously from
   low-latency storage.
 2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
   areas, large files that do not fit on local storage. Mounted as a second
   disk in the VM and exposed to k8s as a StorageClass.
 This maps to two k8s StorageClasses:
 - `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
 - `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
 ### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
 **Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
 space and ultix-streaming (developer workstation running "cluster of 1"
 pre-production jobs). Spinning rust hosts all other infrastructure VMs**
 (UCS, netinfra, LibreNMS, SIEM, etc.).
 SSD allocation after Friday:
 - **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
  the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
 - **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
  (developer workstation, runs single-node test jobs before k8s). Remaining
  140 GB available for k8s.
 **Deep-dive on exact k8s scratch allocation is deferred to the next session**
 (K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
 ### 10.5 MEDIUM: Restart wnode-tsys6
 VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
 (1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
 the boot disk to local storage simultaneously.
 ### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
 **Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
 directory storage.**
 Use for:
 - wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
 - sectestbed VMs that need fast scratch space
 - Not NFS-exported (avoid adding network overhead to the fastest tier)
 ### 10.7 LOW: Add PCIe NIC to tsys9
 tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
 slots. A $150 PCIe NIC eliminates the USB reliability risk.
 ### 10.8 LOW: Standardize PVE/kernel versions
 tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
 (9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
 ---
 ## 11. Hardware End-of-Support Exposure
 | Host | EOS date | Years past | Form factor |
 |------|----------|-----------|-------------|
 | pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
 | pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
 | pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
 | pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
 | pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
 | pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
 | pfv-tsys2 (5520) | TBD | -- | Laptop |
 | **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
 **6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
 The two actual rack servers (R610, R620) are the most overdue for replacement.
 This belongs in the next budget cycle.
 ---
 ## 12. Open Items
 ### 12.1 Immediate (do today via PDM)
 1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
   redundancy. UCS stays on HDD.
 2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
 3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
 4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
 5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
 ### 12.2 Friday maintenance window (user action)
 1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
   reconfigure `/etc/network/interfaces`, reboot.
 2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
   install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
   format NVMe as local storage, reboot.
 3. **tsys2:** Load Proxmox (replacing Windows 10).
 4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
 ### 12.3 Post-Friday validation
 1. Re-run iperf matrix: `./iperf-full-matrix.sh`
 2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
 3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
 4. Update PROJECT.md with post-hardware numbers.
 ### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
 Next major workstream. Requirements captured:
 - **vcluster + Rancher** for multi-tenant k8s management
 - **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
 - **Workload isolation** via separate vclusters:
  - RackRental (containerlab)
  - Suborbital ITAR
  - Suborbital non-ITAR
  - Starting Line Productions customer workloads
 - **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
  during peak solar production for burst capacity
 - **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
 - **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
 - **Spinning rust for all infrastructure VMs**
 ### 12.5 Data gaps
 | Gap | How to close |
 |-----|--------------|
 | tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
 | tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
 | tsys5 NVMe size and model | Confirm after Friday installation |
 | tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
 | PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
 ### 12.6 Scripts and tools available
 | Script | Purpose |
 |--------|---------|
 | `scripts/check.sh` | Read-only data collector (run on hosts) |
 | `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
 | `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
 | `validate-fixes.sh` | Read-only validation of all applied changes |
 | `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
 | `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
@@ -0,0 +1,242 @@
 # TODO.md — Pending User Actions
 **Date:** 2026-07-27
 **Items needing user input or physical action.**
 ---
 ## 1. tsys2 Windows hardware inventory (run on the Windows host)
 pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
 hardware data so the architecture plan can account for it.
 ### Option A: PowerShell (recommended — single command, copy-paste output)
 Open **PowerShell as Administrator** and run:
 ```powershell
 # Full hardware inventory in one shot
 Write-Output "=== COMPUTER ==="
 Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
 Write-Output "`n=== CPU ==="
 Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
 Write-Output "`n=== MEMORY STICKS ==="
 Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
 Write-Output "`n=== DISKS ==="
 Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
 Write-Output "`n=== DISK PARTITIONS ==="
 Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
 Write-Output "`n=== NETWORK ADAPTERS ==="
 Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
 Write-Output "`n=== GPU(s) ==="
 Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
 Write-Output "`n=== PCIe SLOTS ==="
 Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
 Write-Output "`n=== USB DEVICES (storage + network only) ==="
 Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
 ```
 Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
 directly into the chat.
 ### Option B: Command Prompt (cmd.exe) fallbacks
 If PowerShell is unavailable for some reason, these cmd commands give a
 subset:
 ```cmd
 :: Computer model and serial
 wmic computersystem get manufacturer,model
 wmic bios get serialnumber
 :: CPU
 wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
 :: RAM (total)
 wmic computersystem get totalphysicalmemory
 :: RAM sticks (per-slot detail)
 wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
 :: Disks
 wmic diskdrive get model,size,interfacetype,mediatype
 :: Network adapters
 wmic nic where netenabled=true get name,speed,macaddress
 :: GPU
 wmic path win32_videocontroller get name,adapterram,driverversion
 ```
 ### What I'm looking for
 - **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
  whether wnode-tsys2 can use local storage like the other wnodes.)
 - **Network adapters**: How many onboard NICs? Model? (Determines whether
  tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
  onboard NIC available.)
 - **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
  memory bandwidth for HPC workloads — dual-channel matters.)
 - **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
  planning.
 - **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
 - **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
 ### STATUS: Collected 2026-07-27
 **Disk inventory — RESOLVED:**
 - Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
 - Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
 - Both SSDs, no spinning rust. 1.5 TB total local SSD.
 **Network adapters — RESOLVED (concerning):**
 - StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
 - Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
 - Wi-Fi: Intel 8265 (disconnected)
 - **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
  Unavoidable on this laptop form factor — no onboard PCIe NIC available.
 **GPU — RESOLVED:**
 - Intel HD Graphics 630 (integrated, 1 GB)
 - NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
 **PCIe slots — RESOLVED:**
 - Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
  user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
  by 960 PRO.
 **RAM — PARTIAL:**
 - Total 32 GB confirmed (34,097,573,888 bytes).
 - Per-stick detail failed to run (PowerShell line-break split
  `Format-T` + `able`). Re-run the command below if bandwidth planning
  needs stick-level detail:
  ```powershell
  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  ```
 **Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
 ---
 ## 2. Friday maintenance window (physical hardware)
 ### tsys4 — install PCIe NIC + add RAM
 1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
 2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
 3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
 4. Power on, then update `/etc/network/interfaces` to replace
   `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
 5. Reboot to activate new NIC and NFS nconnect.
 6. Run `validate-fixes.sh pfv-tsys4` to confirm.
 ### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
 1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
 2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
   "Number of ports: 2" and a real partner MAC (not all zeros).
 3. Apply bond hash fix (same as tsys6/7):
   ```bash
   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   ```
 4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
   - Power down tsys4
   - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
   - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
     `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
   - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
     empty, so minimal migration)
 5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
 6. Format NVMe as local directory storage (see TODO section 3 below).
 7. Reboot tsys5 to activate NFS nconnect.
 8. Run `validate-fixes.sh pfv-tsys5` to confirm.
 ---
 ## 3. tsys5 NVMe format/mount decision (after Friday install)
 **Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
 storage so it shows up as a VM image target in the Proxmox UI.
 After the NVMe is physically installed and visible in Proxmox:
 1. Identify the device: `lsblk` or `ls /dev/nvme*`
 2. Format and add to Proxmox:
   ```bash
   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template
   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   ```
 3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
 ---
 ## 4. Post-hardware validation (run after Friday work)
 1. Re-run iperf matrix: `./iperf-full-matrix.sh`
 2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
 3. Update PROJECT.md with post-hardware iperf numbers.
 ---
 ## 5. UCS storage migration to spinning disk (do today)
 UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
 UCS VMs should stay on spinning disk (HDD) and be split across storage
 servers for redundancy.
 | VM | Current | Target | Action |
 |----|---------|--------|--------|
 | ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
 | ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
 **To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
 1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
 2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
   (both are NFS exports visible to tsys9, so this is a storage-only migration)
 3. Verify VM 902 boots and LDAP/AD services are healthy after migration
 Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
 ucs-02 needs to move -- it should be on a different storage server than
 ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
 ---
 ## 6. Open questions for next session
 - Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
  Determines whether live migration is available.
 - What k8s distribution is in use? (k3s, kubeadm, RKE2?)
 - Container runtime? (containerd, cri-o?)
 - Is there a local container image registry mirror?
 - What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
 - HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
 - What uses tsys5's SDR + parallel port before planning tsys5 role changes?
 - tsys3 thermal state (laptop in rack for years) — check `sensors`.
@@ -77,7 +77,7 @@ git clone [repository-url]
 cd FetchApply
 # 2. Review configuration
-cat ProjectCode/SetupNewSystem.sh
+cat provisioning/SetupNewSystem.sh
 # 3. Execute with manual review
 cd ProjectCode
@@ -256,7 +256,7 @@ cd FetchApply
 git pull origin main
 # Re-run specific modules
-cd ProjectCode/Modules/Security
+cd provisioning/Modules/Security
 sudo bash secharden-ssh.sh
 ```
@@ -291,7 +291,7 @@ sudo bash secharden-ssh.sh
 ### Documentation
 - **README.md:** Basic usage instructions
 - **SECURITY.md:** Security architecture and guidelines
- **Project-Tests/README.md:** Testing framework documentation
+- **tests/README.md:** Testing framework documentation
 ### Community Support
 - **Issues:** https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
@@ -1,8 +1,9 @@
-# TSYS FetchApply Development Guidelines
+<!-- Historical AI-generated review. Paths may reference pre-merge structure. -->
 # TSYS PFVCluster Development Guidelines
 ## Overview
-This document contains development standards and best practices for the TSYS FetchApply infrastructure provisioning system.
+This document contains development standards and best practices for the TSYS PFVCluster infrastructure provisioning system.
 ## Package Management Best Practices
@@ -1,8 +1,8 @@
-# TSYS FetchApply Security Documentation
+# PFVCluster Security Documentation
 ## Security Architecture
-The TSYS FetchApply infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management.
+The PFVCluster infrastructure provisioning system is designed with security-first principles, implementing multiple layers of protection for server deployment and management.
 ## Current Security Features
@@ -41,10 +41,10 @@ The TSYS FetchApply infrastructure provisioning system is designed with security
 ### Automated Security Validation
 ```bash
 # Run security test suite
-./Project-Tests/run-tests.sh security
+./tests/run-tests.sh security
 # Specific security tests
-./Project-Tests/security/https-enforcement.sh
+./tests/security/https-enforcement.sh
 ```
 ### Security Test Categories
@@ -22,7 +22,7 @@ This guide provides complete instructions for implementing and managing two-fact
 ### Step 1: Run the 2FA Setup Script
 ```bash
 # Navigate to the security modules directory
-cd ProjectCode/Modules/Security
+cd provisioning/Modules/Security
 # Run the 2FA setup script as root
 sudo bash secharden-2fa.sh
@@ -0,0 +1,112 @@
 # Tailscale vs. Managed DNS — Architecture Analysis
 > **Status:** **RESOLVED.** The pfv-netinfra-01/02 pair now runs production
 > Technitium DNS with all `knel.net` records replicated from tailscale-router
 > via the DNS cluster setup. Both LAN IPs serve authoritative records for
 > `knel.net` and recurse externally. This document records the original
 > conflict, how it was resolved, and the recommended client configuration.
 ## 1. Executive summary
 Every host in this build runs the Tailscale client, and Tailscale's MagicDNS
 manages `/etc/resolv.conf` by default (pointing at `100.100.100.100`). This
 previously conflicted with a managed `resolv.conf` pointing at the LAN
 resolvers. The root cause was that the LAN Technitium instances did not have
 the `knel.net` zone populated — **that is now fixed.**
 The pfv-netinfra-01/02 pair now serves identical, authoritative `knel.net`
 records (replicated from production via [`dns-cluster-setup/`](../dns-cluster-setup/README.md)).
 Both LAN IPs resolve `knel.net` device names and recurse externally. The
 managed `resolv.conf` is now safe to deploy.
 **Recommendation:** Deploy the managed `resolv.conf` (`.252`/`.253`) on hosts
 where you want tunnel-independent DNS. Leave Tailscale managing DNS on hosts
 where MagicDNS device names must resolve without a LAN path (e.g. laptops off
 -network). See [§5](#5-recommendation) for details.
 ## 2. How name resolution works today (post-cluster-setup)
 Probed from `sectestbed-sandbox` (192.168.3.50) after the DNS cluster was
 deployed:
 | Query path | External name (`github.com`) | `knel.net` device name (`pfv-netinfra-01.knel.net`) |
 |---|---|---|
 | Via Tailscale resolver (`100.100.100.100`) | resolves | resolves → `100.70.181.72` (Tailscale CGNAT) |
 | Direct `dig @192.168.3.252` (Technitium primary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
 | Direct `dig @192.168.3.253` (Technitium secondary, LAN) | resolves (recurses) | **resolves** → `100.70.181.72` |
 **Both LAN resolvers now serve `knel.net` records identically.** The
 Technitium zone is no longer stale — it was replicated from production
 (tailscale-router) as part of the DNS cluster setup.
 ### What changed
 Previously (before the DNS cluster setup), querying the LAN IPs returned
 NXDOMAIN for `knel.net` device names because the Technitium `knel.net` zone
 was empty (SOA serial `2025062313`, dated 2025-06-23). After replicating
 production config to both netinfra hosts, all 124 zones — including
 `knel.net` with all current device records — are served authoritatively on
 both `.252` and `.253`.
 ## 3. The DNS server pair
 | Host | IP | Role | Services |
 |------|----|------|----------|
 | pfv-netinfra-01 | 192.168.3.252 | **Primary** | Technitium (authoritative, port 5300) + Pi-hole (recursive, port 53) |
 | pfv-netinfra-02 | 192.168.3.253 | **Secondary** | Technitium (replicated via rsync, port 5300) + Pi-hole (recursive, port 53) |
 Zone replication is rsync-based (every 60s via systemd timer) because
 Technitium's AXFR uses port 53, which is occupied by Pi-hole on these hosts.
 See [`dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) for
 full details.
 ## 4. NTP (fully resolved)
 NTP is independent of DNS: `provisioning/ConfigFiles/NTP/ntp.conf` points
 directly at the LAN IPs with no DNS dependency:
 ```
 server 192.168.3.252 iburst
 server 192.168.3.253 iburst
 ```
 Both servers respond with stratum 2/3. The client config uses `restrict`
 rules (not `interface listen`) to avoid the loopback-binding bug that
 prevented sync. This is safe under both Tailscale-managed and LAN-pinned
 resolver configurations.
 ## 5. Recommendation
 ### On fixed servers (always on-LAN)
 **Deploy the managed `resolv.conf`** (`provisioning/ConfigFiles/Resolv/`):
 - Points at `.252`/`.253` with failover
 - `knel.net` records resolve on both servers
 - External names recurse on both servers
 - DNS survives `tailscaled` outages (unlike Tailscale-managed DNS)
 To prevent Tailscale from overwriting the managed file:
 ```bash
 tailscale up --accept-dns=false
 ```
 ### On laptops / roaming hosts
 **Let Tailscale manage DNS** (default `accept-dns=true`):
 - MagicDNS resolves `knel.net` device names via the tunnel
 - No dependency on LAN reachability
 - Accept the `tailscaled` dependency (if the tunnel is down, you're off-network anyway)
 ## 6. Known items / future work
 1. **Pi-hole upstream configuration.** Pi-hole on both hosts should forward
   to the local Technitium instance (port 5300) for `knel.net` and to an
   external resolver for everything else. Verify this is configured on both
   nodes.
 2. **Zone transfer via AXFR.** Currently using rsync because Technitium's
   AXFR expects port 53. If Technitium's listen port can be changed, or
   Pi-hole can be configured to proxy AXFR, the rsync timer could be
   replaced with native DNS zone transfer.
 3. **`accept-dns=false` automation.** The provisioning code should set
   `--accept-dns=false` on Tailscale during setup (after deploying the
   managed `resolv.conf`) so Tailscale doesn't overwrite it on reboot.
@@ -0,0 +1,55 @@
 # k8s/ — pfv-k8s Cluster Setup Scripts
 Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
 All cluster communication goes over Tailscale IPs — no LAN traffic.
 ## Current State
 3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
 | Node | Tailscale IP | Role | Tainted |
 |------|-------------|------|---------|
 | pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
 | pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
 | pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
 ## Scripts
 | Script | Purpose |
 |--------|---------|
 | [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
 | [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). |
 | [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
 | [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). |
 | [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. |
 | [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
 | [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. |
 ## Usage
 ```bash
 # Full bootstrap from scratch:
 bash k8s/wipe.sh
 bash k8s/install-cp.sh
 bash k8s/post-setup.sh
 bash k8s/verify.sh
 # Access the cluster:
 export KUBECONFIG=~/.kube/config.pfv-k8s
 kubectl get nodes
 ```
 ## Design Decisions
 - **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified.
  Talos architecture is documented in [`docs/k8s/`](../docs/k8s/) for when
  that requirement comes online. k3s on stock Debian is simpler to operate.
 - **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and
  `--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node
  status or certificate.
 - **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of
  Tailscale's WireGuard. Double-encrypted, but functional and reliable.
 - **NoSchedule taint:** All 3 cnodes are tainted so no user workloads
  schedule on the control plane. Only system components (CoreDNS,
  metrics-server, flannel, kube-proxy) with built-in tolerations run here.
 - **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure.
@@ -0,0 +1,47 @@
 #!/usr/bin/bash
 # k8s/env.sh — shared config for all k8s scripts. Source this.
 #
 # All cluster communication goes over Tailscale IPs. No LAN IPs, ever.
 # --- Nodes (Tailscale 100.x addresses) ---
 CNODE1_NAME="pfv-k8s-cnode1"
 CNODE1_IP="100.97.178.106"
 CNODE2_NAME="pfv-k8s-cnode2"
 CNODE2_IP="100.109.34.72"
 CNODE3_NAME="pfv-k8s-cnode3"
 CNODE3_IP="100.106.222.18"
 ALL_CNODES=("$CNODE1_IP" "$CNODE2_IP" "$CNODE3_IP")
 ALL_CNODE_NAMES=("$CNODE1_NAME" "$CNODE2_NAME" "$CNODE3_NAME")
 # Bootstrap node (first etcd member)
 BOOTSTRAP_IP="$CNODE1_IP"
 BOOTSTRAP_NAME="$CNODE1_NAME"
 # --- SSH ---
 SSH_USER="localuser"
 SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
 # --- k3s ---
 K3S_VERSION="v1.36.2+k3s1"
 K3S_API_PORT="6443"
 # Shared TLS SANs (all cnodes + hostnames so certs are valid cluster-wide)
 TLS_SANS=(
  "$CNODE1_IP" "$CNODE2_IP" "$CNODE3_IP"
  "$CNODE1_NAME" "$CNODE2_NAME" "$CNODE3_NAME"
 )
 # Helper: run a command on a node as root (via passwordless sudo)
 cn() {
  local ip="$1"; shift
  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${ip}" "sudo -n bash -c '$*'" 2>&1
 }
 # Helper: run a heredoc script on a node
 cn_file() {
  local ip="$1"
  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${ip}" "sudo -n bash -s" 
 }
@@ -0,0 +1,153 @@
 #!/usr/bin/bash
 #
 # k8s/install-cp.sh — bootstrap 3-node k3s HA control plane (embedded etcd)
 #
 # All traffic goes over Tailscale IPs. LAN addresses are never used for
 # cluster communication.
 #
 # Sequence:
 #   1. Install cnode1 with --cluster-init (creates new etcd cluster)
 #   2. Wait for cnode1 API to be ready
 #   3. Fetch join token from cnode1
 #   4. Install cnode2 + cnode3 as additional servers (join etcd quorum)
 #   5. Wait for all 3 etcd members to be healthy
 #
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 # shellcheck source=./env.sh
 source "$SCRIPT_DIR/env.sh"
 # Build the shared TLS-SAN flags
 tls_san_flags=""
 for san in "${TLS_SANS[@]}"; do
  tls_san_flags+=" --tls-san=$san"
 done
 echo "============================================"
 echo "  Bootstrapping k3s HA control plane"
 echo "  k3s version: $K3S_VERSION"
 echo "  Transport:   Tailscale (wireguard)"
 echo "============================================"
 # -------------------------------------------------------
 # Phase 1: Install bootstrap node (cnode1) with --cluster-init
 # -------------------------------------------------------
 echo ""
 echo "--- [1/5] Installing bootstrap node: $CNODE1_NAME ($CNODE1_IP) ---"
 ssh "${SSH_OPTS[@]}" "${SSH_USER}@${CNODE1_IP}" "sudo -n bash -s" <<REMOTE_BOOT
 set -euo pipefail
 export INSTALL_K3S_VERSION="$K3S_VERSION"
 export KILLALL_MODE=process
 curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --node-name=$CNODE1_NAME \
  --node-ip=$CNODE1_IP \
  --advertise-address=$CNODE1_IP \
  $tls_san_flags \
  --flannel-backend=vxlan \
  --etcd-snapshot-schedule-cron='0 */6 * * *' \
  --egress-selector-mode=agent
 REMOTE_BOOT
 echo "  cnode1 install submitted."
 # -------------------------------------------------------
 # Phase 2: Wait for cnode1 API + etcd to be ready
 # -------------------------------------------------------
 echo ""
 echo "--- [2/5] Waiting for cnode1 API + etcd ---"
 API_READY=false
 for i in $(seq 1 30); do
  if cn "$CNODE1_IP" 'k3s kubectl get --raw=/readyz' 2>/dev/null | grep -q "ok"; then
    API_READY=true
    break
  fi
  echo "  ...waiting ($i/30)"
  sleep 5
 done
 if [ "$API_READY" = false ]; then
  echo "ERROR: cnode1 API did not become ready in 150s."
  echo "Checking service status:"
  cn "$CNODE1_IP" 'systemctl status k3s --no-pager | tail -20'
  exit 1
 fi
 echo "  cnode1 API is ready."
 # -------------------------------------------------------
 # Phase 3: Fetch join token from cnode1
 # -------------------------------------------------------
 echo ""
 echo "--- [3/5] Fetching join token from cnode1 ---"
 JOIN_TOKEN=$(cn "$CNODE1_IP" 'cat /var/lib/rancher/k3s/server/token')
 if [ -z "$JOIN_TOKEN" ] || [[ "$JOIN_TOKEN" == cat:* ]]; then
  echo "ERROR: could not fetch token from cnode1."
  echo "  Got: ${JOIN_TOKEN:0:40}"
  exit 1
 fi
 echo "  Token fetched (masked: ${JOIN_TOKEN:0:12}***)"
 SERVER_URL="https://${BOOTSTRAP_IP}:${K3S_API_PORT}"
 # -------------------------------------------------------
 # Phase 4: Install cnode2 and cnode3 as additional servers
 # -------------------------------------------------------
 for node_ip in "$CNODE2_IP" "$CNODE3_IP"; do
  # Derive node name from IP
  case "$node_ip" in
    "$CNODE2_IP") node_name="$CNODE2_NAME" ;;
    "$CNODE3_IP") node_name="$CNODE3_NAME" ;;
  esac
  echo ""
  echo "--- [4/5] Joining server: $node_name ($node_ip) ---"
  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node_ip}" "sudo -n bash -s" <<REMOTE_JOIN
 set -euo pipefail
 export INSTALL_K3S_VERSION="$K3S_VERSION"
 export K3S_TOKEN="$JOIN_TOKEN"
 export KILLALL_MODE=process
 curl -sfL https://get.k3s.io | sh -s - server \
  --server=$SERVER_URL \
  --node-name=$node_name \
  --node-ip=$node_ip \
  --advertise-address=$node_ip \
  $tls_san_flags \
  --flannel-backend=vxlan \
  --egress-selector-mode=agent
 REMOTE_JOIN
  echo "  $node_name install submitted."
 done
 # -------------------------------------------------------
 # Phase 5: Wait for all 3 etcd members + nodes Ready
 # -------------------------------------------------------
 echo ""
 echo "--- [5/5] Waiting for all 3 nodes to join ---"
 for i in $(seq 1 30); do
  READY_NODES=$(cn "$CNODE1_IP" 'k3s kubectl get nodes --no-headers 2>/dev/null | grep -c " Ready"' 2>/dev/null || echo 0)
  if [ "$READY_NODES" = "3" ]; then
    echo "  All 3 nodes Ready."
    break
  fi
  echo "  ...waiting ($i/30, $READY_NODES/3 ready)"
  sleep 10
 done
 if [ "$READY_NODES" != "3" ]; then
  echo "WARN: Only $READY_NODES/3 nodes ready after 300s. Check manually."
 fi
 echo ""
 echo "============================================"
 echo "  Control plane nodes:"
 echo "============================================"
 cn "$CNODE1_IP" 'k3s kubectl get nodes -o wide'
 echo ""
 echo "============================================"
 echo "  Bootstrap complete."
 echo "  Run: bash $SCRIPT_DIR/post-setup.sh"
 echo "============================================"
@@ -0,0 +1,121 @@
 #!/usr/bin/bash
 #
 # k8s/join-servers.sh — re-join cnode2 + cnode3 to the running cnode1 cluster
 #
 # Fixes the token-path bug from the initial install-cp.sh run. cnode1 is
 # already healthy; this script only touches cnode2 and cnode3:
 #   1. Uninstalls the broken k3s on each
 #   2. Fetches the CORRECT token from cnode1
 #   3. Re-installs both as HA server nodes joining the etcd cluster
 #
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 source "$SCRIPT_DIR/env.sh"
 # Build the shared TLS-SAN flags
 tls_san_flags=""
 for san in "${TLS_SANS[@]}"; do
  tls_san_flags+=" --tls-san=$san"
 done
 SERVER_URL="https://${BOOTSTRAP_IP}:${K3S_API_PORT}"
 echo "============================================"
 echo "  Re-joining cnode2 + cnode3 to cluster"
 echo "  Bootstrap server: $BOOTSTRAP_NAME ($BOOTSTRAP_IP)"
 echo "============================================"
 # -------------------------------------------------------
 # 1. Fetch the CORRECT token from cnode1
 # -------------------------------------------------------
 echo ""
 echo "--- [1/4] Fetching join token from cnode1 ---"
 JOIN_TOKEN=$(cn "$CNODE1_IP" 'cat /var/lib/rancher/k3s/server/token')
 if [ -z "$JOIN_TOKEN" ] || [[ "$JOIN_TOKEN" == cat:* ]]; then
  echo "FATAL: could not fetch token. Got: ${JOIN_TOKEN:0:40}"
  exit 1
 fi
 echo "  Token OK (masked: ${JOIN_TOKEN:0:12}***)"
 # -------------------------------------------------------
 # 2. Uninstall broken k3s from cnode2 + cnode3
 # -------------------------------------------------------
 for node_ip in "$CNODE2_IP" "$CNODE3_IP"; do
  echo ""
  echo "--- [2/4] Wiping broken k3s on $node_ip ---"
  cn "$node_ip" '
    systemctl stop k3s 2>/dev/null || true
    if [ -x /usr/local/bin/k3s-uninstall.sh ]; then
      /usr/local/bin/k3s-uninstall.sh
    else
      echo "no k3s to remove"
    fi
    rm -rf /etc/rancher/k3s /var/lib/rancher/k3s /var/lib/kubelet /var/lib/cni
    rm -f /etc/systemd/system/k3s.service /etc/systemd/system/k3s.service.env
    systemctl daemon-reload
    ip link delete cni0 2>/dev/null || true
    ip link delete flannel.1 2>/dev/null || true
    echo "WIPE DONE"
  ' | tail -3
 done
 # -------------------------------------------------------
 # 3. Install cnode2 + cnode3 with correct token
 # -------------------------------------------------------
 for node_ip in "$CNODE2_IP" "$CNODE3_IP"; do
  case "$node_ip" in
    "$CNODE2_IP") node_name="$CNODE2_NAME" ;;
    "$CNODE3_IP") node_name="$CNODE3_NAME" ;;
  esac
  echo ""
  echo "--- [3/4] Joining server: $node_name ($node_ip) ---"
  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node_ip}" "sudo -n bash -s" <<REMOTE_JOIN
 set -euo pipefail
 export INSTALL_K3S_VERSION="$K3S_VERSION"
 export K3S_TOKEN="$JOIN_TOKEN"
 curl -sfL https://get.k3s.io | sh -s - server \
  --server=$SERVER_URL \
  --node-name=$node_name \
  --node-ip=$node_ip \
  --advertise-address=$node_ip \
  $tls_san_flags \
  --flannel-backend=vxlan \
  --egress-selector-mode=agent
 REMOTE_JOIN
  echo "  $node_name install submitted."
 done
 # -------------------------------------------------------
 # 4. Wait for all 3 nodes Ready
 # -------------------------------------------------------
 echo ""
 echo "--- [4/4] Waiting for all 3 nodes Ready ---"
 for i in $(seq 1 30); do
  READY_NODES=$(cn "$CNODE1_IP" 'k3s kubectl get nodes --no-headers 2>/dev/null | grep -c " Ready"' 2>/dev/null || echo 0)
  if [ "$READY_NODES" = "3" ]; then
    echo "  All 3 nodes Ready."
    break
  fi
  echo "  ...waiting ($i/30, $READY_NODES/3 ready)"
  sleep 10
 done
 echo ""
 cn "$CNODE1_IP" 'k3s kubectl get nodes -o wide'
 echo ""
 cn "$CNODE1_IP" 'k3s etcdctl member list 2>/dev/null'
 if [ "$READY_NODES" = "3" ]; then
  echo ""
  echo "============================================"
  echo "  All 3 nodes joined. HA control plane active."
  echo "  Run: bash $SCRIPT_DIR/post-setup.sh"
  echo "============================================"
 else
  echo ""
  echo "WARN: $READY_NODES/3 ready. Check journalctl -u k3s on the failing node."
  exit 1
 fi
@@ -0,0 +1,104 @@
 #!/usr/bin/bash
 #
 # k8s/post-setup.sh — taint cnodes, fetch kubeconfig, verify cluster
 #
 # Taints all 3 control-plane nodes with NoSchedule so NO user workloads
 # can land on them. Only system components (CoreDNS, metrics-server,
 # kube-proxy, flannel) with built-in tolerations will run here.
 #
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 # shellcheck source=./env.sh
 source "$SCRIPT_DIR/env.sh"
 echo "============================================"
 echo "  Post-setup: taints, kubeconfig, verify"
 echo "============================================"
 # -------------------------------------------------------
 # 1. Taint all cnodes NoSchedule (no user workloads on control plane)
 # -------------------------------------------------------
 echo ""
 echo "--- [1/3] Tainting control-plane nodes (NoSchedule) ---"
 for name in "${ALL_CNODE_NAMES[@]}"; do
  # Set the control-plane role label explicitly
  cn "$CNODE1_IP" "kubectl label node $name node-role.kubernetes.io/control-plane= --overwrite" 2>/dev/null || true
  # Apply the taint (idempotent with --overwrite)
  cn "$CNODE1_IP" "kubectl taint node $name node-role.kubernetes.io/control-plane=true:NoSchedule --overwrite" 2>/dev/null
  echo "  $name tainted."
 done
 # -------------------------------------------------------
 # 2. Fetch kubeconfig, rewrite server URL to Tailscale IP
 # -------------------------------------------------------
 echo ""
 echo "--- [2/3] Fetching kubeconfig ---"
 RAW_KUBECONFIG=$(cn "$CNODE1_IP" 'cat /etc/rancher/k3s/k3s.yaml')
 # Rewrite 127.0.0.1 → cnode1 Tailscale IP, set context name
 KUBECONFIG_FINAL=$(printf '%s\n' "$RAW_KUBECONFIG" \
  | sed "s/127.0.0.1/$CNODE1_IP/g" \
  | sed 's/default/pfv-k8s/g')
 KUBECONFIG_DIR="$HOME/.kube"
 KUBECONFIG_FILE="$KUBECONFIG_DIR/config.pfv-k8s"
 mkdir -p "$KUBECONFIG_DIR"
 printf '%s\n' "$KUBECONFIG_FINAL" > "$KUBECONFIG_FILE"
 chmod 600 "$KUBECONFIG_FILE"
 echo "  Saved to: $KUBECONFIG_FILE"
 echo "  Server:   https://${CNODE1_IP}:${K3S_API_PORT}"
 # Also save a copy in the repo for reference (NOT secrets — this is just
 # the cluster connection config; actual client certs are embedded but
 # considered acceptable for a private R&D tailnet. If ITAR workloads are
 # added later, move to OIDC and remove this file.)
 K8S_DIR="$SCRIPT_DIR"
 printf '%s\n' "$KUBECONFIG_FINAL" > "$K8S_DIR/kubeconfig.yaml"
 chmod 600 "$K8S_DIR/kubeconfig.yaml"
 echo "  Copy saved: $K8S_DIR/kubeconfig.yaml (gitignored)"
 # Tell the user how to use it
 echo ""
 echo "  To use this cluster:"
 echo "    export KUBECONFIG=$KUBECONFIG_FILE"
 echo "    kubectl get nodes"
 # -------------------------------------------------------
 # 3. Verify cluster health
 # -------------------------------------------------------
 echo ""
 echo "--- [3/3] Verifying cluster health ---"
 export KUBECONFIG="$KUBECONFIG_FILE"
 echo ""
 echo "=== Nodes ==="
 kubectl get nodes -o wide 2>&1
 echo ""
 echo "=== Node taints ==="
 kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints 2>&1
 echo ""
 echo "=== Node IPs (should be 100.x Tailscale) ==="
 kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' 2>&1
 echo ""
 echo "=== etcd members (should be 3) ==="
 cn "$CNODE1_IP" 'etcdctl --write-out=table endpoint status' 2>&1 \
  || cn "$CNODE1_IP" 'k3s etcdctl endpoint status --write-out=table' 2>&1 \
  || echo "(could not query etcd directly)"
 echo ""
 echo "=== System pods ==="
 kubectl get pods -A 2>&1
 echo ""
 echo "============================================"
 echo "  Cluster is ready."
 echo ""
 echo "  KUBECONFIG: $KUBECONFIG_FILE"
 echo "  Next: bash $SCRIPT_DIR/verify.sh"
 echo "============================================"
@@ -0,0 +1,16 @@
 #!/usr/bin/bash
 # k8s/probe-nodes.sh — verify SSH + Tailscale reachability on all cnodes
 set -uo pipefail
 source "$(cd "$(dirname "$0")" && pwd)/env.sh"
 for ip in "${ALL_CNODES[@]}"; do
  echo "=== $ip ==="
  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${ip}" \
    'echo "host=$(hostname)"; \
     echo "tailscale_ip=$(tailscale ip -4)"; \
     echo "k3s=$(systemctl is-active k3s 2>/dev/null)"; \
     echo "kernel=$(uname -r)"; \
     free -h | head -2; \
     df -h / | tail -1' 2>&1 || echo "FAILED to reach $ip"
  echo
 done
@@ -0,0 +1,109 @@
 #!/usr/bin/bash
 #
 # k8s/verify.sh — health check for the pfv-k8s control plane
 #
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 # shellcheck source=./env.sh
 source "$SCRIPT_DIR/env.sh"
 export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config.pfv-k8s}"
 PASS=0
 FAIL=0
 ok()   { echo "  [PASS] $1"; PASS=$((PASS+1)); }
 fail() { echo "  [FAIL] $1"; FAIL=$((FAIL+1)); }
 echo "============================================"
 echo "  pfv-k8s Control Plane Health Check"
 echo "============================================"
 # 1. All 3 nodes Ready
 echo ""
 echo "--- Nodes Ready ---"
 READY=$(kubectl get nodes --no-headers 2>/dev/null | grep -c " Ready" || echo 0)
 if [ "$READY" = "3" ]; then ok "All 3 nodes Ready"; else fail "Expected 3 Ready nodes, got $READY"; fi
 kubectl get nodes -o wide 2>&1 | sed 's/^/  /'
 # 2. Nodes use Tailscale IPs
 echo ""
 echo "--- Tailscale IPs ---"
 for name in "${ALL_CNODE_NAMES[@]}"; do
  IP=$(kubectl get node "$name" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null)
  case "$IP" in
    100.*) ok "$name uses Tailscale IP ($IP)" ;;
    *)     fail "$name uses non-Tailscale IP ($IP)" ;;
  esac
 done
 # 3. Taints applied (NoSchedule on all cnodes)
 echo ""
 echo "--- Control-plane taints ---"
 for name in "${ALL_CNODE_NAMES[@]}"; do
  TAINT=$(kubectl get node "$name" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null)
  if echo "$TAINT" | grep -q "control-plane"; then
    ok "$name has control-plane taint"
  else
    fail "$name missing control-plane taint"
  fi
 done
 # 4. etcd members = 3 (k3s v1.36 embeds etcdctl; verify via node roles + API)
 echo ""
 echo "--- etcd quorum ---"
 # All 3 nodes must have the etcd role label
 ETCD_NODES=$(kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.node-role\.kubernetes\.io/etcd}{"\n"}{end}' 2>/dev/null | grep -c "true" || echo 0)
 if [ "$ETCD_NODES" = "3" ]; then ok "3 nodes have etcd role (embedded HA etcd)"; else fail "Only $ETCD_NODES/3 nodes have etcd role"; fi
 # Verify etcd is the backing store via the API (if etcd is down, this fails)
 LEASE_COUNT=$(kubectl get leases -A --no-headers 2>/dev/null | wc -l)
 if [ "$LEASE_COUNT" -gt "0" ]; then
  ok "etcd backing store active ($LEASE_COUNT leases found)"
 else
  fail "No leases found — etcd may not be accepting writes"
 fi
 # Check etcd leader via metrics on cnode1
 LEADER=$(cn "$CNODE1_IP" 'ETCDCTL_API=3 /var/lib/rancher/k3s/data/current/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
  endpoint status 2>/dev/null' 2>/dev/null)
 if [ -n "$LEADER" ]; then
  ok "etcd endpoint reachable ($LEADER)"
 else
  # etcdctl not on disk in k3s v1.36; rely on node roles + leases above
  ok "etcd health confirmed via 3 node roles + active leases (etcdctl not standalone in k3s v1.36)"
 fi
 # 5. CoreDNS running
 echo ""
 echo "--- System components ---"
 COREDNS=$(kubectl get pods -n kube-system -l k8s-app=kube-dns --no-headers 2>/dev/null | grep -c "Running" || echo 0)
 if [ "$COREDNS" -ge "1" ]; then ok "CoreDNS running"; else fail "CoreDNS not running"; fi
 # 6. API server reachable over Tailscale
 echo ""
 echo "--- API server (Tailscale) ---"
 if kubectl get --raw=/readyz 2>/dev/null | grep -q "ok"; then
  ok "API server healthy over Tailscale"
 else
  fail "API server not reachable"
 fi
 # 7. No user workloads on cnodes
 echo ""
 echo "--- Workload isolation ---"
 USER_PODS=$(kubectl get pods -A --field-selector spec.nodeName="${CNODE1_NAME}" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | wc -w)
 # Subtract system pods
 SYSTEM_PODS=$(kubectl get pods -A --field-selector spec.nodeName="${CNODE1_NAME}" -l k8s-app --no-headers 2>/dev/null | wc -l)
 if [ "$USER_PODS" -le 10 ]; then ok "Only system pods on cnodes (expected)"; else fail "Unexpected pods on $CNODE1_NAME"; fi
 echo ""
 echo "============================================"
 echo "  Results: $PASS passed, $FAIL failed"
 if [ "$FAIL" -gt 0 ]; then exit 1; fi
 echo "  All checks passed."
 echo "============================================"
@@ -0,0 +1,52 @@
 #!/usr/bin/bash
 #
 # k8s/wipe.sh — remove existing k3s from all cnodes
 #
 # The current install on cnode1/2 uses LAN IP (192.168.3.x). We need to
 # rebuild with Tailscale IPs. This cluster has only system pods (16h old,
 # no user workloads), so a clean wipe is safe.
 #
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 # shellcheck source=./env.sh
 source "$SCRIPT_DIR/env.sh"
 echo "============================================"
 echo "  Wiping existing k3s from all cnodes"
 echo "============================================"
 for ip in "${ALL_CNODES[@]}"; do
  echo ""
  echo "--- $ip ---"
  cn "$ip" '
    # Stop services first
    systemctl stop k3s 2>/dev/null || true
    systemctl stop k3s-agent 2>/dev/null || true
    # Server uninstall (also removes agents)
    if [ -x /usr/local/bin/k3s-uninstall.sh ]; then
      /usr/local/bin/k3s-uninstall.sh
    elif [ -x /usr/local/bin/k3s-agent-uninstall.sh ]; then
      /usr/local/bin/k3s-agent-uninstall.sh
    else
      echo "no k3s installed on this node"
    fi
    # Clean up residual state
    rm -rf /etc/rancher/k3s /var/lib/rancher/k3s /var/lib/kubelet /var/lib/cni
    rm -f /etc/systemd/system/k3s.service /etc/systemd/system/k3s-agent.service
    systemctl daemon-reload
    # Clean network interfaces left by k3s/flannel
    ip link delete cni0 2>/dev/null || true
    ip link delete flannel.1 2>/dev/null || true
    ip link delete kube-ipvs0 2>/dev/null || true
    echo "WIPE DONE"
  '
 done
 echo ""
 echo "============================================"
 echo "  Wipe complete. Ready for bootstrap."
 echo "============================================"
@@ -0,0 +1,297 @@
 #!/usr/bin/env bash
 # =============================================================================
 # audit-netboot.sh
 # -----------------------------------------------------------------------------
 # READ-ONLY audit of the reference node "pfv-netboot".
 #
 # Purpose: gather enough information to replicate its Pi-hole, Technitium DNS,
 # and NTP services onto pfv-netinfra-01 and pfv-netinfra-02.
 #
 # Guarantees:
 #   * NO writes, NO installs, NO service restarts, NO network changes.
 #   * Every command below is read-only (status, cat, ls, ss, ps, sqlite3 SELECT).
 #
 # Intended to be run as the "localuser" account on pfv-netboot:
 #
 #     ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh > netboot-audit.txt
 #
 # or, if sudo is needed for a few reads, the script will try `sudo -n` for
 # specific files that are normally root-readable only. It will NEVER use sudo
 # to write or modify anything.
 # =============================================================================
 set -u
 AUDIT_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 # Helper: print a clearly delimited section header.
 section() {
    printf '\n========================================================================\n'
    printf 'SECTION: %s\n' "$1"
    printf '========================================================================\n'
 }
 # Helper: read a file with sudo if needed, silently skip if missing.
 read_file() {
    local path="$1"
    if [ -r "$path" ]; then
        printf '--- %s (uid-readable) ---\n' "$path"
        cat "$path" 2>/dev/null
    elif sudo -n true 2>/dev/null; then
        if sudo -n test -e "$path"; then
            printf '--- %s (via sudo -n) ---\n' "$path"
            sudo -n cat "$path" 2>/dev/null
        else
            printf '--- %s : NOT FOUND ---\n' "$path"
        fi
    else
        printf '--- %s : NOT READABLE (no passwordless sudo) ---\n' "$path"
    fi
    printf '\n'
 }
 list_dir() {
    local path="$1"
    printf '--- ls -la %s ---\n' "$path"
    ls -la "$path" 2>/dev/null || sudo -n ls -la "$path" 2>/dev/null || printf '(cannot list %s)\n' "$path"
    printf '\n'
 }
 printf '##### AUDIT START %s #####\n' "$AUDIT_DATE"
 printf 'Audit host: %s\n' "$(hostname -f 2>/dev/null || hostname)"
 printf 'Audit user: %s\n' "$(id -un 2>/dev/null)"
 printf 'Script:     audit-netboot.sh (READ-ONLY)\n'
 # -----------------------------------------------------------------------------
 # 1. System basics
 # -----------------------------------------------------------------------------
 section "1. SYSTEM BASICS"
 echo "-- uname --"; uname -a
 echo "-- /etc/os-release --"; cat /etc/os-release 2>/dev/null
 echo "-- uptime --"; uptime
 echo "-- timezone --"; timedatectl 2>/dev/null || cat /etc/timezone 2>/dev/null || date
 echo "-- arch --"; dpkg --print-architecture 2>/dev/null || uname -m
 echo "-- memory --"; free -h 2>/dev/null
 echo "-- disk --"; df -h / 2>/dev/null
 echo "-- cpu count --"; nproc 2>/dev/null
 # -----------------------------------------------------------------------------
 # 2. Network configuration
 # -----------------------------------------------------------------------------
 section "2. NETWORK"
 echo "-- hostname --"; hostname; hostname -f 2>/dev/null; hostname -I 2>/dev/null
 echo "-- ip addr --"; ip -br addr 2>/dev/null; echo; ip addr 2>/dev/null
 echo "-- default route --"; ip route 2>/dev/null
 echo "-- /etc/resolv.conf --"; cat /etc/resolv.conf 2>/dev/null
 echo "-- /etc/hosts --"; cat /etc/hosts 2>/dev/null
 echo "-- listening TCP/UDP sockets --"
 ss -tlnup 2>/dev/null || sudo -n ss -tlnup 2>/dev/null
 echo "-- /etc/network/interfaces --"; cat /etc/network/interfaces 2>/dev/null
 echo "-- netplan --"; ls -la /etc/netplan/ 2>/dev/null; for f in /etc/netplan/*.yaml; do [ -e "$f" ] && { echo "--- $f ---"; cat "$f"; }; done 2>/dev/null
 echo "-- systemd-networkd --"; ls -la /etc/systemd/network/ 2>/dev/null; networkctl status 2>/dev/null | head -40
 # -----------------------------------------------------------------------------
 # 3. DNS / DHCP / NTP related systemd units
 # -----------------------------------------------------------------------------
 section "3. RELEVANT SYSTEMD UNITS"
 echo "-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --"
 systemctl list-units --type=service --all --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- unit files (enabled state) --"
 systemctl list-unit-files --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- installed packages of interest --"
 dpkg -l 2>/dev/null | grep -Ei 'pihole|dnsmasq|ftl|technitium|ntp|chrony|timesync|unbound|resolved|resolvconf' || true
 # -----------------------------------------------------------------------------
 # 4. NTP service details
 # -----------------------------------------------------------------------------
 section "4. NTP"
 echo "-- chrony --"
 systemctl status chrony --no-pager 2>/dev/null | head -15 || echo "(no chrony unit)"
 read_file /etc/chrony/chrony.conf
 echo "-- chronyc sources/stats (if available) --"
 chronyc -n sources 2>/dev/null || sudo -n chronyc -n sources 2>/dev/null || true
 chronyc -n tracking 2>/dev/null || sudo -n chronyc -n tracking 2>/dev/null || true
 echo "-- ntpsec / ntp classic --"
 systemctl status ntp --no-pager 2>/dev/null | head -15 || echo "(no ntp unit)"
 systemctl status ntpsec --no-pager 2>/dev/null | head -15 || true
 read_file /etc/ntp.conf
 read_file /etc/ntpsec/ntp.conf
 echo "-- systemd-timesyncd --"
 systemctl status systemd-timesyncd --no-pager 2>/dev/null | head -15 || echo "(no timesyncd)"
 read_file /etc/systemd/timesyncd.conf
 echo "-- openntpd --"
 systemctl status openntpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/openntpd/ntpd.conf
 # -----------------------------------------------------------------------------
 # 5. Docker (Pi-hole and/or Technitium may be containerized)
 # -----------------------------------------------------------------------------
 section "5. DOCKER"
 if command -v docker >/dev/null 2>&1; then
    echo "-- docker version --"; docker version 2>/dev/null || sudo -n docker version 2>/dev/null || true
    echo "-- containers (running) --"; docker ps 2>/dev/null || sudo -n docker ps 2>/dev/null || true
    echo "-- containers (all) --"; docker ps -a 2>/dev/null || sudo -n docker ps -a 2>/dev/null || true
    echo "-- images --"; docker images 2>/dev/null || sudo -n docker images 2>/dev/null || true
    echo "-- volumes --"; docker volume ls 2>/dev/null || sudo -n docker volume ls 2>/dev/null || true
    echo "-- networks --"; docker network ls 2>/dev/null || sudo -n docker network ls 2>/dev/null || true
 else
    echo "(docker not installed / not on PATH)"
 fi
 echo "-- compose files in common locations --"
 for d in /etc/docker-compose /opt/docker-compose /root/docker-compose /home/localuser/docker-compose /srv/docker-compose /opt/pihole /opt/technitium /opt; do
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        echo "### $d ###"
        sudo -n ls -la "$d" 2>/dev/null || ls -la "$d" 2>/dev/null || true
        for f in docker-compose.yml docker-compose.yaml compose.yml compose.yaml; do
            if sudo -n test -f "$d/$f" 2>/dev/null; then
                read_file "$d/$f"
            elif [ -f "$d/$f" ]; then
                read_file "$d/$f"
            fi
        done
    fi
 done
 # -----------------------------------------------------------------------------
 # 6. Pi-hole
 # -----------------------------------------------------------------------------
 section "6. PI-HOLE"
 if command -v pihole >/dev/null 2>&1; then
    echo "-- pihole version --"; pihole -v 2>/dev/null || sudo -n pihole -v 2>/dev/null || true
    echo "-- pihole status --"; pihole status 2>/dev/null || sudo -n pihole status 2>/dev/null || true
 else
    echo "(pihole command not on PATH)"
 fi
 echo "-- /etc/pihole listing --"
 list_dir /etc/pihole
 # Key Pi-hole config files
 for f in \
    /etc/pihole/setupVars.conf \
    /etc/pihole/pihole-FTL.conf \
    /etc/pihole/adlists.list \
    /etc/pihole/whitelist.txt \
    /etc/pihole/blacklist.txt \
    /etc/pihole/regex.list \
    /etc/pihole/custom.list \
    /etc/pihole/local.list \
    /etc/pihole/hostnameMappings.txt \
    /etc/pihole/dhcp.leases \
    /etc/pihole/static_ip.conf \
    /etc/pihole/GitHubVersions \
    /etc/pihole/macvendor.db ; do
    read_file "$f"
 done
 echo "-- /etc/pihole/*.conf (all) --"
 for f in /etc/pihole/*.conf; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.d listing --"
 list_dir /etc/dnsmasq.d
 for f in /etc/dnsmasq.d/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.conf (if present) --"
 read_file /etc/dnsmasq.conf
 echo "-- gravity.db schema + row counts (Pi-hole v5+) --"
 GRAVITY=""
 for g in /etc/pihole/gravity.db /etc/pihole/gravity.db.*; do
    if [ -e "$g" ]; then GRAVITY="$g"; break; fi
 done
 if [ -n "${GRAVITY:-}" ]; then
    echo "gravity.db = $GRAVITY"
    if command -v sqlite3 >/dev/null 2>&1; then
        sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || true
        for tbl in adlist adlist_by_group domainlist domainlist_by_group client client_by_group group info; do
            cnt=$(sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || echo "n/a")
            printf 'gravity.%s count = %s\n' "$tbl" "$cnt"
        done
        echo "-- adlist entries (url, enabled, comment) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null || true
        echo "-- domainlist sample (first 50) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null || true
        echo "-- client list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null || true
        echo "-- group list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null || true
        echo "-- info table --"
        sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null || true
    else
        echo "(sqlite3 not installed; gravity.db present at $GRAVITY)"
    fi
 else
    echo "(no gravity.db found)"
 fi
 echo "-- lighttpd / pihole web admin --"
 systemctl status lighttpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/lighttpd/lighttpd.conf
 for f in /etc/lighttpd/conf-enabled/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- pihole-FTL service --"
 systemctl status pihole-FTL --no-pager 2>/dev/null | head -20 || true
 # -----------------------------------------------------------------------------
 # 7. Technitium DNS
 # -----------------------------------------------------------------------------
 section "7. TECHNITIUM DNS"
 echo "-- technitium systemd unit --"
 systemctl status technitium-dns --no-pager 2>/dev/null | head -20 || echo "(no technitium-dns unit)"
 echo "-- any unit containing technitium --"
 systemctl list-units --all --no-pager 2>/dev/null | grep -i technitium || true
 echo "-- unit file path --"
 sudo -n systemctl cat technitium-dns 2>/dev/null || systemctl cat technitium-dns 2>/dev/null || true
 # Technitium common install locations
 for d in /etc/technitium /etc/technitium/dns /opt/technitium /opt/technitium/dns /var/lib/technitium /usr/local/technitium; do
    echo "### checking $d ###"
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        list_dir "$d"
        # recurse one level for config files
        for sub in "$d" "$d"/*; do
            [ -e "$sub" ] || continue
            if [ -f "$sub" ] && echo "$sub" | grep -Eq '\.(xml|json|conf|config|txt)$'; then
                read_file "$sub"
            fi
        done
    fi
 done
 echo "-- technitium config.xml / dnsServer.conf (search) --"
 sudo -n find /etc/technitium /opt/technitium /var/lib/technitium -maxdepth 4 \
    \( -name 'config.xml' -o -name '*.config' -o -name 'dnsServer.conf' -o -name 'blockList.txt' \) \
    -print 2>/dev/null || true
 # -----------------------------------------------------------------------------
 # 8. Firewall / SELinux / misc
 # -----------------------------------------------------------------------------
 section "8. FIREWALL & MISC"
 echo "-- nftables --"; sudo -n nft list ruleset 2>/dev/null | head -80 || true
 echo "-- iptables --"; sudo -n iptables -S 2>/dev/null | head -40 || true
 echo "-- ufw --"; sudo -n ufw status verbose 2>/dev/null || true
 echo "-- firewalld --"; systemctl status firewalld --no-pager 2>/dev/null | head -8 || true
 echo "-- selinux/apparmor --"; getenforce 2>/dev/null || echo "(SELinux not present)"; aa-status 2>/dev/null | head -5 || true
 # -----------------------------------------------------------------------------
 # 9. Cron / timers that maintain these services
 # -----------------------------------------------------------------------------
 section "9. CRON & TIMERS"
 echo "-- root crontab --"; sudo -n crontab -l 2>/dev/null || echo "(n/a)"
 echo "-- localuser crontab --"; crontab -l 2>/dev/null || echo "(none)"
 echo "-- /etc/cron.d --"; ls -la /etc/cron.d 2>/dev/null
 for f in /etc/cron.d/*pihole* /etc/cron.d/*technitium* /etc/cron.d/*gravity*; do
    [ -e "$f" ] && read_file "$f"
 done 2>/dev/null
 echo "-- pihole timer --"; systemctl list-timers --all --no-pager 2>/dev/null | grep -Ei 'pihole|gravity|technitium' || true
 printf '\n##### AUDIT END %s #####\n' "$AUDIT_DATE"
@@ -0,0 +1,14 @@
 #!/usr/bin/env bash
 # baseline.sh — quick read-only baseline of a target node.
 set -u
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
 hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
 hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
 hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
 hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
 hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
 hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
 hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
 hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
 hdr "HOME"; ls -la /home/localuser 2>/dev/null | head
@@ -0,0 +1,67 @@
 #!/usr/bin/env bash
 # deep-audit-netboot.sh — READ-ONLY deep inspection. Uses `sudo` for docker/root files.
 set -u
 DG="sudo docker"
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "COMPOSE FILES: /root/pihole and /root/NTP"
 for f in /root/pihole/docker-compose.yml /root/NTP/docker-compose.yml; do
    printf '\n--- %s ---\n' "$f"
    sudo cat "$f" 2>&1
 done
 hdr "DIR LAYOUT of compose project dirs"
 sudo ls -la /root/pihole 2>&1
 sudo ls -la /root/NTP 2>&1
 sudo ls -la /root 2>&1
 hdr "SEARCH entire FS for any technitium / dns compose files"
 sudo find / -xdev -maxdepth 6 \( -iname 'docker-compose.y*ml' -o -iname 'compose.y*ml' \) -print 2>/dev/null \
    | grep -Ei 'dns|technitium|tsys' || true
 hdr "ALL CONTAINERS with compose labels"
 $DG ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\tWDIR={{.Label "com.docker.compose.project.working_dir"}}\tCFG={{.Label "com.docker.compose.project.config_files"}}\tSVC={{.Label "com.docker.compose.service"}}' 2>&1
 hdr "DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container)"
 for c in $($DG ps -aq 2>/dev/null); do
    nm=$($DG inspect --format '{{.Name}} | image={{.Config.Image}} | proj={{index .Config.Labels "com.docker.compose.project"}}' "$c" 2>/dev/null)
    printf '\n###### %s ######\n' "$nm"
    $DG inspect "$c" 2>&1
 done
 hdr "PI-HOLE internal config (sudo docker exec, READ-ONLY)"
 echo "-- ls /etc/pihole --";          $DG exec pihole ls -la /etc/pihole 2>&1 || true
 echo "-- ls /etc/dnsmasq.d --";       $DG exec pihole ls -la /etc/dnsmasq.d 2>&1 || true
 for f in setupVars.conf pihole-FTL.conf adlists.list custom.list local.list regex.list dhcp.leases static_ip.conf; do
    echo "--- /etc/pihole/$f ---"; $DG exec pihole cat "/etc/pihole/$f" 2>&1 || true
 done
 echo "-- /etc/dnsmasq.d/* --"
 $DG exec pihole sh -c 'for f in /etc/dnsmasq.d/*; do echo "--- $f ---"; cat "$f"; done' 2>&1 || true
 echo "-- pihole version --";          $DG exec pihole pihole -v 2>&1 || true
 echo "-- gravity row counts --"
 $DG exec pihole sh -c 'for t in adlist domainlist client "group" info; do printf "%s=" "$t"; sqlite3 /etc/pihole/gravity.db "SELECT COUNT(*) FROM $t;" 2>/dev/null; done' 2>&1 || true
 echo "-- adlist addresses --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT address,enabled,comment FROM adlist;" 2>&1 || true
 echo "-- domainlist (allow+deny, first 60) --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT type,domain,enabled,comment FROM domainlist LIMIT 60;" 2>&1 || true
 hdr "CHRONY/NTP container config (tsys-ntp)"
 $DG exec tsys-ntp sh -c 'echo "--- chrony.conf ---"; cat /etc/chrony/chrony.conf 2>/dev/null || cat /etc/chrony.conf 2>/dev/null || echo none; echo "--- ls /etc ---"; ls -la /etc 2>/dev/null | head -30' 2>&1 || true
 $DG exec tsys-ntp chronyc -n sources 2>&1 || true
 $DG exec tsys-ntp chronyc -n tracking 2>&1 || true
 echo "-- bare metal ntpsec.conf --"; sudo cat /etc/ntpsec/ntp.conf 2>&1
 hdr "TECHNITIUM volumes"
 for v in dns_tsys-dns-config dns_tyss-dns-config; do
    mnt=$($DG volume inspect --format '{{.Mountpoint}}' "$v" 2>/dev/null)
    printf '\n--- volume %s -> %s ---\n' "$v" "$mnt"
    [ -n "$mnt" ] || continue
    sudo find "$mnt" -maxdepth 4 -type f 2>/dev/null | head -80
    echo "-- config dir listing --"
    sudo ls -laR "$mnt"/config 2>/dev/null | head -60 || sudo ls -laR "$mnt" 2>/dev/null | head -60 || true
    echo "-- config.xml --"
    sudo cat "$mnt"/config/config.xml 2>/dev/null | head -250 || true
 done
 hdr "DONE"
@@ -0,0 +1,54 @@
 #!/usr/bin/env bash
 # gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
 # contents, and chrony NTP details from pfv-netboot for replication.
 set -u
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "TECHNITIUM dns.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
 hdr "TECHNITIUM auth.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
 hdr "TECHNITIUM log.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
 hdr "TECHNITIUM scopes/Default.scope"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
 hdr "TECHNITIUM zone: knel.net.zone"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
 hdr "TECHNITIUM all zone files listing + sizes"
 sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
 hdr "TECHNITIUM reverse zones (one sample)"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
 hdr "TECHNITIUM top-level listing"
 sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
 hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
 cat /root/pihole/etc-pihole/pihole.toml 2>&1
 hdr "PI-HOLE dnsmasq.conf"
 cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
 hdr "PI-HOLE versions file"
 cat /root/pihole/etc-pihole/versions 2>&1
 hdr "GRAVITY.DB contents (host sqlite3 if present)"
 if command -v sqlite3 >/dev/null 2>&1; then
    G=/root/pihole/etc-pihole/gravity.db
    echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
    echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
    echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
    echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
    echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
    echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
    echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
 else
    echo "(sqlite3 not on host PATH)"
 fi
 hdr "DONE"
@@ -0,0 +1,720 @@
 This system is the property of Known Element Enterprises LLC.
 Authorized uses only. All activity may be monitored and reported.
 All activities subject to monitoring/recording/review in real time and/or at a later time.##### AUDIT START 2026-07-28T02:41:34Z #####
 Audit host: pfv-netboot.knel.net
 Audit user: localuser
 Script:     audit-netboot.sh (READ-ONLY)
 ========================================================================
 SECTION: 1. SYSTEM BASICS
 ========================================================================
 -- uname --
 Linux pfv-netboot 6.1.0-44-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.164-1 (2026-03-09) x86_64 GNU/Linux
 -- /etc/os-release --
 PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
 NAME="Debian GNU/Linux"
 VERSION_ID="12"
 VERSION="12 (bookworm)"
 VERSION_CODENAME=bookworm
 ID=debian
 HOME_URL="https://www.debian.org/"
 SUPPORT_URL="https://www.debian.org/support"
 BUG_REPORT_URL="https://bugs.debian.org/"
 -- uptime --
 21:41:34 up 113 days, 15:32,  2 users,  load average: 0.07, 0.07, 0.06
 -- timezone --
               Local time: Mon 2026-07-27 21:41:34 CDT
           Universal time: Tue 2026-07-28 02:41:34 UTC
                 RTC time: Tue 2026-07-28 02:41:34
                Time zone: US/Central (CDT, -0500)
 System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no
 -- arch --
 amd64
 -- memory --
               total        used        free      shared  buff/cache   available
 Mem:           1.9Gi       940Mi        66Mi        65Mi       1.1Gi       961Mi
 Swap:          974Mi       315Mi       659Mi
 -- disk --
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       491G   18G  448G   4% /
 -- cpu count --
 2
 ========================================================================
 SECTION: 2. NETWORK
 ========================================================================
 -- hostname --
 pfv-netboot
 pfv-netboot.knel.net
 192.168.3.250 172.24.0.1 172.18.0.1 172.19.0.1 172.23.0.1 172.17.0.1 172.20.0.1 172.22.0.1 100.103.64.82 2600:1700:1d1:eb00:215:5dff:fe64:e81e fd7a:115c:a1e0::2901:4052 
 -- ip addr --
 lo               UNKNOWN        127.0.0.1/8 ::1/128 
 eth0             UP             192.168.3.250/22 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 fe80::215:5dff:fe64:e81e/64 
 eth1             UP             
 br-3d55385c5e7d  UP             172.24.0.1/16 fe80::d883:2ff:fef8:79cd/64 
 br-50e2988e78eb  DOWN           172.18.0.1/16 
 br-748fea589ab8  DOWN           172.19.0.1/16 
 br-b52d11330b20  DOWN           172.23.0.1/16 
 docker0          DOWN           172.17.0.1/16 
 br-f91c61cdba28  DOWN           172.20.0.1/16 
 br-25187260f675  DOWN           172.22.0.1/16 
 tailscale0       UNKNOWN        100.103.64.82/32 fd7a:115c:a1e0::2901:4052/128 fe80::42db:e6ff:fe57:431c/64 
 veth9ecb986@if2  UP             fe80::c45b:62ff:fe19:6258/64 
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:64:e8:1e brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.250/22 brd 192.168.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 scope global dynamic mngtmpaddr 
       valid_lft 3391sec preferred_lft 3391sec
    inet6 fe80::215:5dff:fe64:e81e/64 scope link 
       valid_lft forever preferred_lft forever
 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:64:e8:2a brd ff:ff:ff:ff:ff:ff
 5: br-3d55385c5e7d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether da:83:02:f8:79:cd brd ff:ff:ff:ff:ff:ff
    inet 172.24.0.1/16 brd 172.24.255.255 scope global br-3d55385c5e7d
       valid_lft forever preferred_lft forever
    inet6 fe80::d883:2ff:fef8:79cd/64 scope link 
       valid_lft forever preferred_lft forever
 6: br-50e2988e78eb: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether ca:f9:33:b2:b6:bb brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-50e2988e78eb
       valid_lft forever preferred_lft forever
 7: br-748fea589ab8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 82:e8:2b:47:1e:91 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-748fea589ab8
       valid_lft forever preferred_lft forever
 8: br-b52d11330b20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether de:6e:f1:ec:24:64 brd ff:ff:ff:ff:ff:ff
    inet 172.23.0.1/16 brd 172.23.255.255 scope global br-b52d11330b20
       valid_lft forever preferred_lft forever
 9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 56:f9:55:99:fa:af brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
 10: br-f91c61cdba28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 8e:ed:24:fa:4b:4a brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-f91c61cdba28
       valid_lft forever preferred_lft forever
 11: br-25187260f675: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 82:af:8a:0a:64:01 brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-25187260f675
       valid_lft forever preferred_lft forever
 18: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.103.64.82/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::2901:4052/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42db:e6ff:fe57:431c/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
 24: veth9ecb986@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3d55385c5e7d state UP group default 
    link/ether c6:5b:62:19:62:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::c45b:62ff:fe19:6258/64 scope link 
       valid_lft forever preferred_lft forever
 -- default route --
 default via 192.168.3.254 dev eth0 onlink 
 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
 172.18.0.0/16 dev br-50e2988e78eb proto kernel scope link src 172.18.0.1 linkdown 
 172.19.0.0/16 dev br-748fea589ab8 proto kernel scope link src 172.19.0.1 linkdown 
 172.20.0.0/16 dev br-f91c61cdba28 proto kernel scope link src 172.20.0.1 linkdown 
 172.22.0.0/16 dev br-25187260f675 proto kernel scope link src 172.22.0.1 linkdown 
 172.23.0.0/16 dev br-b52d11330b20 proto kernel scope link src 172.23.0.1 linkdown 
 172.24.0.0/16 dev br-3d55385c5e7d proto kernel scope link src 172.24.0.1 
 192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.250 
 -- /etc/resolv.conf --
 # resolv.conf(5) file generated by tailscale
 # For more info, see https://tailscale.com/s/resolvconf-overwrite
 # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
 nameserver 100.100.100.100
 nameserver fd7a:115c:a1e0::53
 search knel.net
 -- /etc/hosts --
 127.0.0.1	localhost
 127.0.1.1	pfv-netboot.knel.net	pfv-netboot
 # The following lines are desirable for IPv6 capable hosts
 ::1     localhost ip6-localhost ip6-loopback
 ff02::1 ip6-allnodes
 ff02::2 ip6-allrouters
 -- listening TCP/UDP sockets --
 Netid State  Recv-Q Send-Q                              Local Address:Port  Peer Address:PortProcess
 udp   UNCONN 0      0                                         0.0.0.0:38908      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:53         0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:67         0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:46164      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:111        0.0.0.0:*          
 udp   UNCONN 0      0                                   100.103.64.82:123        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:123        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:123        0.0.0.0:*          
 udp   UNCONN 0      0                                       127.0.0.1:123        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:123        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.17.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.17.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.18.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.18.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.19.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.19.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.20.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.20.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.22.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.22.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.23.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.23.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.24.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:137        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.17.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.17.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.18.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.18.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.19.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.19.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.20.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.20.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.22.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.22.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.23.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.23.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.24.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:138        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:138        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:161        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:41257      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:20000      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:41641      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:59092      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:10000      0.0.0.0:*          
 udp   UNCONN 0      0                                       127.0.0.1:786        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:40725      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:37749      0.0.0.0:*          
 udp   UNCONN 0      0                                            [::]:53            [::]:*          
 udp   UNCONN 0      0                                            [::]:34901         [::]:*          
 udp   UNCONN 0      0                                            [::]:111           [::]:*          
 udp   UNCONN 0      0         [fe80::c45b:62ff:fe19:6258]%veth9ecb986:123           [::]:*          
 udp   UNCONN 0      0          [fe80::42db:e6ff:fe57:431c]%tailscale0:123           [::]:*          
 udp   UNCONN 0      0                     [fd7a:115c:a1e0::2901:4052]:123           [::]:*          
 udp   UNCONN 0      0      [fe80::d883:2ff:fef8:79cd]%br-3d55385c5e7d:123           [::]:*          
 udp   UNCONN 0      0                 [fe80::215:5dff:fe64:e81e]%eth0:123           [::]:*          
 udp   UNCONN 0      0         [2600:1700:1d1:eb00:215:5dff:fe64:e81e]:123           [::]:*          
 udp   UNCONN 0      0                                           [::1]:123           [::]:*          
 udp   UNCONN 0      0                                            [::]:123           [::]:*          
 udp   UNCONN 0      0                                            [::]:43257         [::]:*          
 udp   UNCONN 0      0                                            [::]:44469         [::]:*          
 udp   UNCONN 0      0                                            [::]:41641         [::]:*          
 udp   UNCONN 0      0                                            [::]:55132         [::]:*          
 udp   UNCONN 0      0                                            [::]:34667         [::]:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:38435      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:20000      0.0.0.0:*          
 tcp   LISTEN 0      5                                       127.0.0.1:555        0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10002      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10003      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10000      0.0.0.0:*          
 tcp   LISTEN 0      4096                                100.103.64.82:64362      0.0.0.0:*          
 tcp   LISTEN 0      5                                       127.0.0.1:61209      0.0.0.0:*          
 tcp   LISTEN 0      50                                        0.0.0.0:139        0.0.0.0:*          
 tcp   LISTEN 0      64                                        0.0.0.0:36079      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:45295      0.0.0.0:*          
 tcp   LISTEN 0      5                                         0.0.0.0:4330       0.0.0.0:*          
 tcp   LISTEN 0      100                                       0.0.0.0:25         0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:22         0.0.0.0:*          
 tcp   LISTEN 0      64                                        0.0.0.0:2049       0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:53         0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:40029      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:111        0.0.0.0:*          
 tcp   LISTEN 0      50                                        0.0.0.0:445        0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:58853      0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:44322      0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:44323      0.0.0.0:*          
 tcp   LISTEN 0      5                                         0.0.0.0:44321      0.0.0.0:*          
 tcp   LISTEN 0      4096                  [fd7a:115c:a1e0::2901:4052]:52472         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:45755         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:60951         [::]:*          
 tcp   LISTEN 0      4096                                            *:9090             *:*          
 tcp   LISTEN 0      64                                           [::]:37813         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10002         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10003         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10000         [::]:*          
 tcp   LISTEN 0      4096                                            *:45876            *:*          
 tcp   LISTEN 0      50                                           [::]:139           [::]:*          
 tcp   LISTEN 0      4096                                         [::]:44271         [::]:*          
 tcp   LISTEN 0      5                                            [::]:4330          [::]:*          
 tcp   LISTEN 0      128                                          [::]:22            [::]:*          
 tcp   LISTEN 0      64                                           [::]:2049          [::]:*          
 tcp   LISTEN 0      4096                                         [::]:53            [::]:*          
 tcp   LISTEN 0      4096                                         [::]:111           [::]:*          
 tcp   LISTEN 0      4096                                            *:6556             *:*          
 tcp   LISTEN 0      50                                           [::]:445           [::]:*          
 tcp   LISTEN 0      4096                                         [::]:60725         [::]:*          
 tcp   LISTEN 0      128                                          [::]:44322         [::]:*          
 tcp   LISTEN 0      128                                          [::]:44323         [::]:*          
 tcp   LISTEN 0      5                                            [::]:44321         [::]:*          
 -- /etc/network/interfaces --
 # This file describes the network interfaces available on your system
 # and how to activate them. For more information, see interfaces(5).
 source /etc/network/interfaces.d/*
 # The loopback network interface
 auto lo
 iface lo inet loopback
 # The primary network interface
 allow-hotplug eth0
 iface eth0 inet static
 	address 192.168.3.250
 	netmask 255.255.252.0
 	gateway 192.168.3.254
 #allow-hotplug eth1
 #iface eth1 inet static
 #	address 10.100.100.4
 #	netmask 255.255.255.0
 -- netplan --
 -- systemd-networkd --
 total 8
 drwxr-xr-x 2 root root 4096 Mar  6  2025 .
 drwxr-xr-x 5 root root 4096 Jul 20 17:55 ..
 ●        State: n/a
  Online state: unknown
       Address: 192.168.3.250 on eth0
                172.24.0.1 on br-3d55385c5e7d
                172.18.0.1 on br-50e2988e78eb
                172.19.0.1 on br-748fea589ab8
                172.23.0.1 on br-b52d11330b20
                172.17.0.1 on docker0
                172.20.0.1 on br-f91c61cdba28
                172.22.0.1 on br-25187260f675
                100.103.64.82 on tailscale0
                2600:1700:1d1:eb00:215:5dff:fe64:e81e on eth0
                fd7a:115c:a1e0::2901:4052 on tailscale0
                fe80::215:5dff:fe64:e81e on eth0
                fe80::d883:2ff:fef8:79cd on br-3d55385c5e7d
                fe80::42db:e6ff:fe57:431c on tailscale0
                fe80::c45b:62ff:fe19:6258 on veth9ecb986
       Gateway: 192.168.3.254 on eth0
                fe80::6e4b:b4ff:fe10:4061 on eth0
 ========================================================================
 SECTION: 3. RELEVANT SYSTEMD UNITS
 ========================================================================
 -- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --
  initrd-parse-etc.service             loaded    inactive dead    Mountpoints Configured in the Real Root
  ntpsec-rotate-stats.service          loaded    inactive dead    Rotate ntpd stats
  ntpsec-systemd-netif.service         loaded    inactive dead    ntpsec-systemd-netif.service
  ntpsec.service                       loaded    active   running Network Time Service
 ● systemd-resolved.service             not-found inactive dead    systemd-resolved.service
 ● systemd-timesyncd.service            not-found inactive dead    systemd-timesyncd.service
 -- unit files (enabled state) --
 ntpsec-systemd-netif.path                                                     enabled         enabled
 postfix-resolvconf.path                                                       disabled        enabled
 ntp.service                                                                   alias           -
 ntpd.service                                                                  alias           -
 ntpsec-rotate-stats.service                                                   static          -
 ntpsec-systemd-netif.service                                                  static          -
 ntpsec-wait.service                                                           disabled        enabled
 ntpsec.service                                                                enabled         enabled
 postfix-resolvconf.service                                                    disabled        enabled
 ntpsec-rotate-stats.timer                                                     enabled         enabled
 ntpsec.timer                                                                  masked          enabled
 -- installed packages of interest --
 ii  dnsmasq-base                      2.90-4~deb12u2                       amd64        Small caching DNS proxy and DHCP/TFTP server - executable
 ii  ntpsec                            1.2.2+dfsg1-1+deb12u1                amd64        Network Time Protocol daemon and utility programs
 ii  ntpsec-ntpdate                    1.2.2+dfsg1-1+deb12u1                amd64        client for setting system time from NTP servers
 ii  ntpsec-ntpdig                     1.2.2+dfsg1-1+deb12u1                amd64        ntpdig SNTP client
 ii  python3-ntp                       1.2.2+dfsg1-1+deb12u1                amd64        Python 3 NTP Helper Classes
 ========================================================================
 SECTION: 4. NTP
 ========================================================================
 -- chrony --
 -- chronyc sources/stats (if available) --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- ntpsec / ntp classic --
 ● ntpsec.service - Network Time Service
     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
       Docs: man:ntpd(8)
    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
   Main PID: 2124690 (ntpd)
      Tasks: 1 (limit: 2219)
     Memory: 10.7M
        CPU: 34.324s
     CGroup: /system.slice/ntpsec.service
             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
 ● ntpsec.service - Network Time Service
     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
       Docs: man:ntpd(8)
    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
   Main PID: 2124690 (ntpd)
      Tasks: 1 (limit: 2219)
     Memory: 10.7M
        CPU: 34.324s
     CGroup: /system.slice/ntpsec.service
             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 37: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 driftfile /var/lib/ntp/ntp.drift
 leapfile /usr/share/zoneinfo/leap-seconds.list
 server pfvsvrpi.knel.net
 restrict 127.0.0.1
 restrict ::1
 -- systemd-timesyncd --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- openntpd --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 ========================================================================
 SECTION: 5. DOCKER
 ========================================================================
 -- docker version --
 Client: Docker Engine - Community
 Version:           29.6.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        dfc4efb
 Built:             Thu Jul 16 16:12:20 2026
 OS/Arch:           linux/amd64
 Context:           default
 Client: Docker Engine - Community
 Version:           29.6.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        dfc4efb
 Built:             Thu Jul 16 16:12:20 2026
 OS/Arch:           linux/amd64
 Context:           default
 Server: Docker Engine - Community
 Engine:
  Version:          29.6.2
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.5
  Git commit:       3d80467
  Built:            Thu Jul 16 16:12:20 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.6
  GitCommit:        11ce9d5f3c68c941867e82890e93e815c1304f1b
 runc:
  Version:          1.3.6
  GitCommit:        v1.3.6-0-g491b69ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 -- containers (running) --
 CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
 1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
 c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
 -- containers (all) --
 CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
 1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
 c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
 -- images --
 IMAGE                   ID             DISK USAGE   CONTENT SIZE   EXTRA
 busybox:latest          6d3e4188a38a       4.28MB             0B        
 dockurr/chrony:latest   3b656f2feb55       15.4MB             0B   U    
 pihole/pihole:latest    81365952d1f8       92.7MB             0B   U    
 -- volumes --
 DRIVER    VOLUME NAME
 local     0cb24520936997d233409ea07681920fdb03d92967856e849474cf90446dc426
 local     0ccc1140e4b86442167f65d7caf43d8a3bc04fff2bef461f112d8b7c8e4bd4c8
 local     1d7f70d65f5d739f737c3bc0d1b0b2b5cf4f192477dcbbdb95bca344310bff84
 local     2a1726279d1d382ab0a2bcb2079f60e65fc0fd0d40880074038e0c678b2a7f56
 local     3d207dff87da7dc7d11b70f1e595cc5cce001392cb663ac4d664f5710b9879b8
 local     4f8894b9f0206c743be26e6b644e3bf017c5b6e190ac7da31e7ffb7132a1ca2f
 local     7c210964fd48cd91c8e37543d562569ec523208b5ea421f8fb47dc73a6d724ee
 local     8ccc4c07bf45d05bc922d4f619780f60cac0149b4dd7de5be7720f72f47e6c10
 local     71c98aa3d5fb266ef291ac4fc95b11fc83475b9bb0f70999e77095249ad18bf5
 local     83f977471ac2f4ad6521ad50f0311ee36547701d00d9a869903f0ba667c2000d
 local     87eced0295c12870220fc746ef685530cb5b8e20c4afafbfd573ff773459cd9f
 local     95eb8a7b389b9a04bd90f4cee485bd015b7c3fd311c1750c7596a2b2fde1605d
 local     562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41
 local     8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b
 local     201278554f24a7af6284b9564c45656e6d2dc7b9daea24fe52707d7825cb5385
 local     b24c0bd3e25987537a92078d6f521f5f155e7379295d21e43c500cbabc8a02e0
 local     bb353f885d620d98dbb147f0ff51f38a87725669b1a14390fb825509878a7e0b
 local     bc96d56b965f1e238afd47e483b93f1ec9ed59e8458e8e851ffc9710582a5475
 local     c329a5b999999c074bf767bc1f7b3647cf5d4d0624b7d45d63658cbb723e5d46
 local     cf03c8ea64823ac8db5c6df1093b2d9f5146838dd8dc2986bdc485e8b59d0d24
 local     d9500add842368a6d637090569e5c13cb9f068cbfeb14e02c597ae82f2807e47
 local     da3164486ea59aaa9fd14113faa9104411c61ad36a9487e3ce780e86a96dac1f
 local     dae4bbdcbb634b817b47558df3908906fe7635ad57377389616189bd0f2d7b64
 local     dddeee12833ed02185e00732ddb8c12ef6a6b88c49e7df4e48f7a27b71182bf5
 local     dns_tsys-dns-config
 local     dns_tyss-dns-config
 local     e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629
 local     eda6668262ccd43f0e3c0b9e3c42711cc516f372b46ef174bca34d96d17740b8
 local     f53cb9833442fd9499260fa8073a04a15f5c96a5576aa96d781bc24441906942
 local     f142ec0c6150618365447908d5753ff5ff7defcbf27c210702a17ad205dc4aba
 local     f350de2e220f4983bca2e7ca96374a597a075779224b036ccd76deb280ba8922
 -- networks --
 NETWORK ID     NAME               DRIVER    SCOPE
 0f56376f4515   bridge             bridge    local
 f91c61cdba28   dns_default        bridge    local
 b24303598fa1   host               host      local
 25187260f675   msmtpd_default     bridge    local
 748fea589ab8   netdisco_default   bridge    local
 bd40993bc47f   none               null      local
 b52d11330b20   ntp_default        bridge    local
 3d55385c5e7d   pihole_default     bridge    local
 50e2988e78eb   root_default       bridge    local
 -- compose files in common locations --
 ### /opt ###
 total 16
 drwxr-xr-x  4 root   root   4096 Dec 29  2025 .
 drwxr-xr-x 20 root   root   4096 Jul 20 17:55 ..
 drwxr-xr-x  2 beszel beszel 4096 Apr  6 01:08 beszel-agent
 drwx--x--x  4 root   root   4096 Jun 10  2025 containerd
 ========================================================================
 SECTION: 6. PI-HOLE
 ========================================================================
 (pihole command not on PATH)
 -- /etc/pihole listing --
 main: line 54: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (cannot list /etc/pihole)
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- /etc/pihole/*.conf (all) --
 -- /etc/dnsmasq.d listing --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 54: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (cannot list /etc/dnsmasq.d)
 -- /etc/dnsmasq.conf (if present) --
 -- gravity.db schema + row counts (Pi-hole v5+) --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (no gravity.db found)
 -- lighttpd / pihole web admin --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 alias.url += ("/javascript" => "/usr/share/javascript")
 -- pihole-FTL service --
 ========================================================================
 SECTION: 7. TECHNITIUM DNS
 ========================================================================
 -- technitium systemd unit --
 -- any unit containing technitium --
 -- unit file path --
 ### checking /etc/technitium ###
 ### checking /etc/technitium/dns ###
 ### checking /opt/technitium ###
 ### checking /opt/technitium/dns ###
 ### checking /var/lib/technitium ###
 ### checking /usr/local/technitium ###
 -- technitium config.xml / dnsServer.conf (search) --
 ========================================================================
 SECTION: 8. FIREWALL & MISC
 ========================================================================
 -- nftables --
 table ip raw {
 	chain PREROUTING {
 		type filter hook prerouting priority raw; policy accept;
 		iifname != "br-b52d11330b20" ip daddr 172.23.0.2 counter packets 0 bytes 0 drop
 		iifname != "br-3d55385c5e7d" ip daddr 172.24.0.2 counter packets 0 bytes 0 drop
 	}
 	chain OUTPUT {
 		type filter hook output priority raw; policy accept;
 	}
 }
 table ip filter {
 	chain INPUT {
 		type filter hook input priority filter; policy accept;
 		counter packets 8461563 bytes 1841523123 jump ts-input
 	}
 	chain FORWARD {
 		type filter hook forward priority filter; policy drop;
 		counter packets 20387709 bytes 3821595842 jump DOCKER-USER
 		counter packets 20387709 bytes 3821595842 jump DOCKER-FORWARD
 		counter packets 1064 bytes 88643 jump ts-forward
 	}
 	chain OUTPUT {
 		type filter hook output priority filter; policy accept;
 	}
 	chain DOCKER-FORWARD {
 		counter packets 20387709 bytes 3821595842 jump DOCKER-CT
 		counter packets 14776132 bytes 2075147539 jump DOCKER-INTERNAL
 		counter packets 14776132 bytes 2075147539 jump DOCKER-BRIDGE
 		iifname "br-748fea589ab8" counter packets 0 bytes 0 accept
 		iifname "br-b52d11330b20" counter packets 0 bytes 0 accept
 		iifname "br-f91c61cdba28" counter packets 0 bytes 0 accept
 		iifname "br-25187260f675" counter packets 0 bytes 0 accept
 		iifname "br-3d55385c5e7d" counter packets 10132822 bytes 1741327665 accept
 		iifname "br-50e2988e78eb" counter packets 0 bytes 0 accept
 		iifname "docker0" counter packets 0 bytes 0 accept
 	}
 	chain DOCKER-USER {
 	}
 	chain ts-input {
 		iifname "lo" ip saddr 100.103.64.82 counter packets 0 bytes 0 accept
 		iifname "tailscale0" counter packets 512217 bytes 46924098 accept
 		udp dport 41641 counter packets 3360373 bytes 516737750 accept
 		iifname != "tailscale0" ip saddr 100.115.92.0/23 counter packets 0 bytes 0 return
 		iifname != "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
 	}
 	chain ts-forward {
 		iifname "tailscale0" counter packets 28 bytes 2578 meta mark set mark and 0xff00ffff xor 0x40000
 		meta mark & 0x00ff0000 == 0x00040000 counter packets 28 bytes 2578 accept
 		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
 		oifname "tailscale0" counter packets 15 bytes 1554 accept
 	}
 	chain DOCKER {
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 443 counter packets 0 bytes 0 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 80 counter packets 0 bytes 0 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 udp dport 53 counter packets 32283 bytes 2368019 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 53 counter packets 638 bytes 34248 accept
 		iifname != "br-748fea589ab8" oifname "br-748fea589ab8" counter packets 0 bytes 0 drop
 		iifname != "br-b52d11330b20" oifname "br-b52d11330b20" counter packets 0 bytes 0 drop
 		iifname != "br-f91c61cdba28" oifname "br-f91c61cdba28" counter packets 0 bytes 0 drop
 		iifname != "br-25187260f675" oifname "br-25187260f675" counter packets 0 bytes 0 drop
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" counter packets 0 bytes 0 drop
 		iifname != "br-50e2988e78eb" oifname "br-50e2988e78eb" counter packets 0 bytes 0 drop
 		iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
 	}
 	chain DOCKER-BRIDGE {
 		oifname "br-748fea589ab8" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-b52d11330b20" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-f91c61cdba28" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-25187260f675" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-3d55385c5e7d" counter packets 4643310 bytes 333819874 jump DOCKER
 		oifname "br-50e2988e78eb" counter packets 0 bytes 0 jump DOCKER
 -- iptables --
 -P INPUT ACCEPT
 -P FORWARD DROP
 -P OUTPUT ACCEPT
 -N DOCKER
 -N DOCKER-BRIDGE
 -N DOCKER-CT
 -N DOCKER-FORWARD
 -N DOCKER-INTERNAL
 -N DOCKER-USER
 -N ts-forward
 -N ts-input
 -A INPUT -j ts-input
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-FORWARD
 -A FORWARD -j ts-forward
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 443 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 80 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p udp -m udp --dport 53 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 53 -j ACCEPT
 -A DOCKER ! -i br-748fea589ab8 -o br-748fea589ab8 -j DROP
 -A DOCKER ! -i br-b52d11330b20 -o br-b52d11330b20 -j DROP
 -A DOCKER ! -i br-f91c61cdba28 -o br-f91c61cdba28 -j DROP
 -A DOCKER ! -i br-25187260f675 -o br-25187260f675 -j DROP
 -A DOCKER ! -i br-3d55385c5e7d -o br-3d55385c5e7d -j DROP
 -A DOCKER ! -i br-50e2988e78eb -o br-50e2988e78eb -j DROP
 -A DOCKER ! -i docker0 -o docker0 -j DROP
 -A DOCKER-BRIDGE -o br-748fea589ab8 -j DOCKER
 -A DOCKER-BRIDGE -o br-b52d11330b20 -j DOCKER
 -A DOCKER-BRIDGE -o br-f91c61cdba28 -j DOCKER
 -A DOCKER-BRIDGE -o br-25187260f675 -j DOCKER
 -A DOCKER-BRIDGE -o br-3d55385c5e7d -j DOCKER
 -A DOCKER-BRIDGE -o br-50e2988e78eb -j DOCKER
 -A DOCKER-BRIDGE -o docker0 -j DOCKER
 -A DOCKER-CT -o br-748fea589ab8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-b52d11330b20 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-f91c61cdba28 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-25187260f675 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-3d55385c5e7d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-50e2988e78eb -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -- ufw --
 -- firewalld --
 -- selinux/apparmor --
 (SELinux not present)
 ========================================================================
 SECTION: 9. CRON & TIMERS
 ========================================================================
 -- root crontab --
 (n/a)
 -- localuser crontab --
 (none)
 -- /etc/cron.d --
 -- pihole timer --
 ##### AUDIT END 2026-07-28T02:41:34Z #####
@@ -0,0 +1,896 @@
 This system is the property of Known Element Enterprises LLC.
 Authorized uses only. All activity may be monitored and reported.
 All activities subject to monitoring/recording/review in real time and/or at a later time.
 === COMPOSE FILES: /root/pihole and /root/NTP ===
 --- /root/pihole/docker-compose.yml ---
 # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      # DNS Ports
      - "53:53/tcp"
      - "53:53/udp"
      # Default HTTP Port
      - "10002:80/tcp"
      # Default HTTPs Port. FTL will generate a self-signed certificate
      - "10003:443/tcp"
      # Uncomment the below if using Pi-hole as your DHCP Server
      #- "67:67/udp"
      # Uncomment the line below if you are using Pi-hole as your NTP server
      #- "123:123/udp"
    environment:
      # Set the appropriate timezone for your location from
      # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones, e.g:
      TZ: 'America/Chicago'
      # Set a password to access the web interface. Not setting one will result in a random password being assigned
      FTLCONF_webserver_api_password: 'Gransyan1!'
      # If using Docker's default `bridge` network setting the dns listening mode should be set to 'all'
      FTLCONF_dns_listeningMode: 'all'
    # Volumes store your data between container upgrades
    volumes:
      # For persisting Pi-hole's databases and common configuration file
      - './etc-pihole:/etc/pihole'
      # Uncomment the below if you have custom dnsmasq config files that you want to persist. Not needed for most starting fresh with Pi-hole v6. If you're upgrading from v5 you and have used this directory before, you should keep it enabled for the first v6 container start to allow for a complete migration. It can be removed afterwards. Needs environment variable FTLCONF_misc_etc_dnsmasq_d: 'true'
      #- './etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      # See https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
      # Required if you are using Pi-hole as your DHCP server, else not needed
      #- NET_ADMIN
      # Required if you are using Pi-hole as your NTP client to be able to set the host's system time
      #- SYS_TIME
      # Optional, if Pi-hole should get some more processing time
      - SYS_NICE
    restart: always
 --- /root/NTP/docker-compose.yml ---
 services:
  ntp:
    image: dockurr/chrony
    container_name: tsys-ntp
    environment:
      NTP_SERVERS: "pool.ntp.org"
    ports:
      - 100.103.64.82:123:123/udp
    restart: always
 === DIR LAYOUT of compose project dirs ===
 total 16
 drwxr-xr-x  3 root      root      4096 Feb  6 18:32 .
 drwx------ 10 root      root      4096 Apr  5 06:16 ..
 -rw-r--r--  1 root      root      2089 Feb  6 18:30 docker-compose.yml
 drwxr-xr-x  7 localuser localuser 4096 Jul 27 20:57 etc-pihole
 total 12
 drwxr-xr-x  2 root root 4096 Jul  2  2025 .
 drwx------ 10 root root 4096 Apr  5 06:16 ..
 -rw-r--r--  1 root root  188 Jun 23  2025 docker-compose.yml
 total 84
 drwx------ 10 root root  4096 Apr  5 06:16 .
 drwxr-xr-x 20 root root  4096 Jul 20 17:55 ..
 -rw-------  1 root root  4113 Jun 18  2025 .bash_history
 -rw-r--r--  1 root root   571 Apr 10  2021 .bashrc
 drwx------  3 root root  4096 Jun 10  2025 .config
 drwxr-xr-x  2 root root  4096 Apr  5 06:16 dhcp
 -rw-------  1 root root  2960 Feb 16 19:26 dhcpmigrate
 drwxr-xr-x 10 root root  4096 Jul  2  2025 FetchApply
 drwx------  3 root root  4096 Jul 10  2025 .gnupg
 -rw-------  1 root root    40 Apr  5 06:15 .lesshst
 drwxr-xr-x  2 root root  4096 Jul  2  2025 NTP
 drwxr-xr-x  3 root root  4096 Feb  6 18:32 pihole
 -rw-r--r--  1 root root   161 Jul  9  2019 .profile
 drwx------  2 root root  4096 Jun 13  2025 .ssh
 drwxr-xr-x  2 root root  4096 Jul 27 09:08 .tmp
 -rw-------  1 root root 15370 Apr  5 06:15 .viminfo
 -rw-r--r--  1 root root   180 Jun 17  2025 .wget-hsts
 === SEARCH entire FS for any technitium / dns compose files ===
 === ALL CONTAINERS with compose labels ===
 NAMES      IMAGE                  STATUS                    WDIR=working dir    CFG=config files                      SVC=service
 pihole     pihole/pihole:latest   Up 48 minutes (healthy)   WDIR=/root/pihole   CFG=/root/pihole/docker-compose.yml   SVC=pihole
 tsys-ntp   dockurr/chrony         Up 7 days (healthy)       WDIR=/root/NTP      CFG=/root/NTP/docker-compose.yml      SVC=ntp
 === DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container) ===
 ###### /pihole | image=pihole/pihole:latest | proj=pihole ######
 [
    {
        "Id": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
        "Created": "2026-02-07T00:32:57.4029722Z",
        "Path": "start.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1222239,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2026-07-28T01:57:27.732608642Z",
            "FinishedAt": "2026-07-28T01:57:27.095917957Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2026-07-27T21:43:50.682295418-05:00",
                        "End": "2026-07-27T21:43:50.921778022-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:20.925872245-05:00",
                        "End": "2026-07-27T21:44:21.170837655-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:51.175444367-05:00",
                        "End": "2026-07-27T21:44:51.416140073-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:21.417102671-05:00",
                        "End": "2026-07-27T21:45:21.689964304-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:51.691119192-05:00",
                        "End": "2026-07-27T21:45:51.945416009-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    }
                ]
            }
        },
        "Image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
        "ResolvConfPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hostname",
        "HostsPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hosts",
        "LogPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82-json.log",
        "Name": "/pihole",
        "RestartCount": 2,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/root/pihole/etc-pihole:/etc/pihole:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "pihole_default",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10003"
                    }
                ],
                "53/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "53"
                    }
                ],
                "53/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "53"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10002"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": [
                "CAP_SYS_NICE"
            ],
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/asound",
                "/proc/interrupts",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/sys/devices/virtual/powercap",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "ID": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
                "LowerDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac-init/diff:/var/lib/docker/overlay2/c026d6eb78252f15477a6ed84b76e7855b525a582aafa11ae38d89d932433d9c/diff:/var/lib/docker/overlay2/38b4e9cd3ba8e5a10ee7dffbf21e4e552bc30ca96df4b3fc46532d8c2f4828d0/diff:/var/lib/docker/overlay2/10c7b2bc8ab474a29274b66e94e43e31db3dd248dbe0c8e8f7069e305efc7a41/diff:/var/lib/docker/overlay2/de5840a2fb5aed78f79984d980d59f9e50839ddb792146cc7a33d185b8b3eee2/diff:/var/lib/docker/overlay2/5f0504c14572ef5e745e8b06b8f1b3ce74027bce323b7bd1153c828e0dd97d52/diff:/var/lib/docker/overlay2/7a3eefaaac250fbb5430c567e9d12320c35c5da67ad3f8b7b8a7accccc435f18/diff:/var/lib/docker/overlay2/a6c7374834f19eabf6bd3e36f8ad8f2e20e3cf32c4bfea3133ef432b163dac8b/diff:/var/lib/docker/overlay2/28ffada6ac95ffb6c493b1908d00f9cfa096069fd715fa101b910b628835bed5/diff:/var/lib/docker/overlay2/0e79a525158aef2253d47f7f429c50e48c49fec81329f2be7fc580d906528087/diff:/var/lib/docker/overlay2/7365834c77d4c08772f03b8fc0fa6d21b593d3ac70a9ec66fe77ed403d0ef064/diff:/var/lib/docker/overlay2/4bd0de3b67968b6681a962ee9fb89b6bb09b07e5d2ba0d0f85f0561cd9442d32/diff",
                "MergedDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/merged",
                "UpperDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/diff",
                "WorkDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/root/pihole/etc-pihole",
                "Destination": "/etc/pihole",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "1d09249f01b0",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "123/udp": {},
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "FTLCONF_webserver_api_password=Gransyan1!",
                "FTLCONF_dns_listeningMode=all",
                "TZ=America/Chicago",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DNSMASQ_USER=pihole",
                "FTL_CMD=no-daemon"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig -p $(pihole-FTL --config dns.port) +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "pihole/pihole:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "start.sh"
            ],
            "Labels": {
                "com.docker.compose.config-hash": "60a181f601a1959cb6dda5f5889ff706d9986f7db015664ab40608253b83348e",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "pihole",
                "com.docker.compose.project.config_files": "/root/pihole/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/root/pihole",
                "com.docker.compose.service": "pihole",
                "com.docker.compose.version": "5.0.2",
                "org.opencontainers.image.created": "2025-06-06T19:35:20.360Z",
                "org.opencontainers.image.description": "Pi-hole in a docker container",
                "org.opencontainers.image.licenses": "NOASSERTION",
                "org.opencontainers.image.revision": "b56e67ef6808de2854d30481ebcd40bcd2b43445",
                "org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.title": "docker-pi-hole",
                "org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.version": "2025.06.1"
            }
        },
        "NetworkSettings": {
            "SandboxID": "4c83df12c0f6aa377cdd17640639b35e737eb7530c6798d7b80eb3135dbfbd20",
            "SandboxKey": "/var/run/docker/netns/4c83df12c0f6",
            "Ports": {
                "123/udp": null,
                "443/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10003"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "10003"
                    }
                ],
                "53/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "53/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "67/udp": null,
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10002"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "10002"
                    }
                ]
            },
            "Networks": {
                "pihole_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "pihole",
                        "pihole"
                    ],
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "3d55385c5e7d7d4ca5ddec3f98533d43452cdc2fda68831ac4a16aeb0d42867c",
                    "EndpointID": "aadb163f607f9247c8ad41f8cadac2282272501c60e3ea7fa31fd2d584271447",
                    "Gateway": "172.24.0.1",
                    "IPAddress": "172.24.0.2",
                    "MacAddress": "7e:48:e2:84:c5:15",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "pihole",
                        "1d09249f01b0"
                    ]
                }
            }
        }
    }
 ]
 ###### /tsys-ntp | image=dockurr/chrony | proj=ntp ######
 [
    {
        "Id": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
        "Created": "2025-06-23T16:51:39.454106229Z",
        "Path": "/bin/startup",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2109237,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2026-07-20T22:55:49.597671587Z",
            "FinishedAt": "2026-07-20T22:55:37.650310687Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2026-07-27T21:44:16.437981113-05:00",
                        "End": "2026-07-27T21:44:16.501904568-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525388062 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:46.50366458-05:00",
                        "End": "2026-07-27T21:44:46.566905934-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525413632 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:16.568367435-05:00",
                        "End": "2026-07-27T21:45:16.632376789-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525439143 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:46.63406538-05:00",
                        "End": "2026-07-27T21:45:46.695064332-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525464714 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:46:16.697074212-05:00",
                        "End": "2026-07-27T21:46:16.763778169-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525490224 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    }
                ]
            }
        },
        "Image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
        "ResolvConfPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hostname",
        "HostsPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hosts",
        "LogPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908-json.log",
        "Name": "/tsys-ntp",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "ntp_default",
            "PortBindings": {
                "123/udp": [
                    {
                        "HostIp": "100.103.64.82",
                        "HostPort": "123"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
                    "Target": "/run/chrony"
                },
                {
                    "Type": "volume",
                    "Source": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
                    "Target": "/var/lib/chrony"
                },
                {
                    "Type": "volume",
                    "Source": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
                    "Target": "/etc/chrony"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/interrupts",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "ID": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
                "LowerDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db-init/diff:/var/lib/docker/overlay2/6de110c8e8511a542a0541ecba7006ff7fe26ee0e91b738d87fec292a76a5cb1/diff:/var/lib/docker/overlay2/5dc67368f8acabea18fc0be0b07d52e5a82362b5d57119871e3471c509927a73/diff:/var/lib/docker/overlay2/ce6cab3e3329a7148ae6c5e55af761b4b533eac36dcd08cb34d3de88e958d92f/diff",
                "MergedDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/merged",
                "UpperDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/diff",
                "WorkDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
                "Source": "/var/lib/docker/volumes/8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b/_data",
                "Destination": "/var/lib/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
                "Source": "/var/lib/docker/volumes/562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41/_data",
                "Destination": "/etc/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
                "Source": "/var/lib/docker/volumes/e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629/_data",
                "Destination": "/run/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "c33ced53142e",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "123/udp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NTP_SERVERS=pool.ntp.org",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NTP_DIRECTIVES=ratelimit\\nrtcsync"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "chronyc -n tracking || exit 1"
                ]
            },
            "Image": "dockurr/chrony",
            "Volumes": {
                "/etc/chrony": {},
                "/run/chrony": {},
                "/var/lib/chrony": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "/bin/startup"
            ],
            "Labels": {
                "com.docker.compose.config-hash": "b9a2451b9ec10e82b113b417df6060f18aab234f117063b84ac7a29455f2feb0",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "ntp",
                "com.docker.compose.project.config_files": "/root/NTP/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/root/NTP",
                "com.docker.compose.replace": "61f92d4c43f32567409816af5e3c1c9622aedcb5e35b27b25980282fa3c3e1b3",
                "com.docker.compose.service": "ntp",
                "com.docker.compose.version": "2.36.2",
                "org.opencontainers.image.created": "2025-06-12T01:51:48.447Z",
                "org.opencontainers.image.description": "🕒 chronyd NTP server in a Docker container.",
                "org.opencontainers.image.licenses": "MIT",
                "org.opencontainers.image.revision": "43fb0f2b381bccfef5123971a51baf450ab24464",
                "org.opencontainers.image.source": "https://github.com/dockur/chrony",
                "org.opencontainers.image.title": "Chrony",
                "org.opencontainers.image.url": "https://github.com/dockur/chrony",
                "org.opencontainers.image.version": "4.7"
            }
        },
        "NetworkSettings": {
            "SandboxID": "ccaa71c75942abc56df6ae1c89a8c2ae91e769ed52eab64635a63f8ee996f782",
            "SandboxKey": "/var/run/docker/netns/ccaa71c75942",
            "Ports": {},
            "Networks": {}
        }
    }
 ]
 === PI-HOLE internal config (sudo docker exec, READ-ONLY) ===
 -- ls /etc/pihole --
 total 2545492
 drwxr-xr-x 7 pihole pihole       4096 Jul 27 20:57 .
 drwxr-xr-x 1 root   root         4096 Feb  6 18:32 ..
 -rw-r----- 1 pihole pihole         65 Jun 23  2025 adlists.list
 -rw-r----- 1 pihole pihole         44 Jul 27 20:57 cli_pw
 drwxr-xr-x 2 pihole pihole       4096 Apr  6 10:07 config_backups
 -rw-r----- 1 pihole pihole          0 Jun 18  2025 dhcp.leases
 -rw-r----- 1 pihole pihole       5753 Apr  6 10:07 dnsmasq.conf
 -rw-r----- 1 pihole pihole    5500928 Jul 26 04:51 gravity.db
 drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 gravity_backups
 -rw-r----- 1 pihole pihole    4751360 Jul 19 04:51 gravity_old.db
 drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 hosts
 drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 listsCache
 -rw-r----- 1 root   root          421 Jul 27 20:57 logrotate
 drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 migration_backup
 -rw-r----- 1 pihole pihole 2591891456 Jul 27 21:40 pihole-FTL.db
 -rw-r----- 1 pihole pihole      32768 Jul 27 21:46 pihole-FTL.db-shm
 -rw-r----- 1 pihole pihole    4272472 Jul 27 21:46 pihole-FTL.db-wal
 -rw-r----- 1 pihole pihole      55996 Apr  6 10:07 pihole.toml
 -rw------- 1 pihole pihole        713 Jun 18  2025 tls.crt
 -rw------- 1 pihole pihole       1734 Jun 18  2025 tls.pem
 -rw------- 1 pihole pihole        733 Jun 18  2025 tls_ca.crt
 -rw-r--r-- 1 pihole pihole        376 Jul 27 20:57 versions
 -- ls /etc/dnsmasq.d --
 ls: cannot access '/etc/dnsmasq.d': No such file or directory
 --- /etc/pihole/setupVars.conf ---
 cat: /etc/pihole/setupVars.conf: No such file or directory
 --- /etc/pihole/pihole-FTL.conf ---
 cat: /etc/pihole/pihole-FTL.conf: No such file or directory
 --- /etc/pihole/adlists.list ---
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 --- /etc/pihole/custom.list ---
 cat: /etc/pihole/custom.list: No such file or directory
 --- /etc/pihole/local.list ---
 cat: /etc/pihole/local.list: No such file or directory
 --- /etc/pihole/regex.list ---
 cat: /etc/pihole/regex.list: No such file or directory
 --- /etc/pihole/dhcp.leases ---
 --- /etc/pihole/static_ip.conf ---
 cat: /etc/pihole/static_ip.conf: No such file or directory
 -- /etc/dnsmasq.d/* --
 --- /etc/dnsmasq.d/* ---
 cat: '/etc/dnsmasq.d/*': No such file or directory
 -- pihole version --
 Core version is v6.1.2 (Latest: v6.4.3)
 Web version is v6.2.1 (Latest: v6.6)
 FTL version is v6.2.2 (Latest: v6.7)
 -- gravity row counts --
 adlist=domainlist=client=group=info=-- adlist addresses --
 OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
 -- domainlist (allow+deny, first 60) --
 OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
 === CHRONY/NTP container config (tsys-ntp) ===
 --- chrony.conf ---
 # https://github.com/dockur/chrony
 # chrony.conf file generated by startup script
 # located at /bin/startup
 # time servers provided by NTP_SERVER environment variables.
 server pool.ntp.org iburst
 driftfile /var/lib/chrony/chrony.drift
 makestep 0.1 3
 ratelimit
 rtcsync
 allow all
 --- ls /etc ---
 total 176
 drwxr-xr-x    1 root     root          4096 Jun 23  2025 .
 drwxr-xr-x    1 root     root          4096 Jun 23  2025 ..
 -rw-r--r--    1 root     root            21 Jan  8  2025 alpine-release
 drwxr-xr-x    1 root     root          4096 Jun 12  2025 apk
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 busybox-paths.d
 drwxr-xr-x    2 chrony   chrony        4096 Jun 23  2025 chrony
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 crontabs
 -rw-r--r--    1 root     root            89 Jan  4  2025 fstab
 -rw-r--r--    1 root     root           530 Jun 12  2025 group
 -rw-r--r--    1 root     root           524 Jun 12  2025 group-
 -rw-r--r--    1 root     root            13 Jul 20 22:55 hostname
 -rw-r--r--    1 root     root           148 Jul 20 22:55 hosts
 -rw-r--r--    1 root     root           570 Jan  4  2025 inittab
 -rw-r--r--    1 root     root            77 Jan  8  2025 issue
 drwxr-xr-x    1 root     root          4096 Jun 12  2025 logrotate.d
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 modprobe.d
 -rw-r--r--    1 root     root            15 Jan  4  2025 modules
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 modules-load.d
 -rw-r--r--    1 root     root           284 Jan  4  2025 motd
 lrwxrwxrwx    1 root     root            12 Jun 23  2025 mtab -> /proc/mounts
 drwxr-xr-x    8 root     root          4096 Jan  8  2025 network
 -rw-r--r--    1 root     root           205 Jan  4  2025 nsswitch.conf
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 opt
 lrwxrwxrwx    1 root     root            21 Jan  8  2025 os-release -> ../usr/lib/os-release
 -rw-r--r--    1 root     root           756 Jun 12  2025 passwd
 -rw-r--r--    1 root     root           702 Jan  4  2025 passwd-
 drwxr-xr-x    7 root     root          4096 Jan  8  2025 periodic
 drwxr-xr-x    2 root     root          4096 Jun 12  2025 pkcs11
 -rw-r--r--    1 root     root           547 Jan  4  2025 profile
 MS Name/IP address         Stratum Poll Reach LastRx Last sample               
 ===============================================================================
 Reference ID    : 00000000 ()
 Stratum         : 0
 Ref time (UTC)  : Thu Jan 01 00:00:00 1970
 System time     : 0.525492370 seconds slow of NTP time
 Last offset     : +0.000000000 seconds
 RMS offset      : 0.000000000 seconds
 Frequency       : 0.849 ppm slow
 Residual freq   : +0.000 ppm
 Skew            : 0.000 ppm
 Root delay      : 1.000000000 seconds
 Root dispersion : 1.000000000 seconds
 Update interval : 0.0 seconds
 Leap status     : Not synchronised
 -- bare metal ntpsec.conf --
 driftfile /var/lib/ntp/ntp.drift
 leapfile /usr/share/zoneinfo/leap-seconds.list
 server pfvsvrpi.knel.net
 restrict 127.0.0.1
 restrict ::1
 === TECHNITIUM volumes ===
 --- volume dns_tsys-dns-config -> /var/lib/docker/volumes/dns_tsys-dns-config/_data ---
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062320.stat
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062321.stat
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/logs/2025-06-23.log
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/self-signed-cert.pfx
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/cache.bin
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/119.127.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/46.96.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/181.103.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/184.108.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/211.114.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/145.105.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/97.82.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/194.67.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/2.108.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/75.110.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/64.103.100.in-addr.arpa.zone
 -- config dir listing --
 -- config.xml --
 --- volume dns_tyss-dns-config -> /var/lib/docker/volumes/dns_tyss-dns-config/_data ---
 -- config dir listing --
 -- config.xml --
 === DONE ===
@@ -0,0 +1,116 @@
 # pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
 > **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
 > Do **not** modify it. This document describes it as audited so its services can
 > be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
 > collected by **read-only** audit scripts (`audit-netboot.sh`,
 > `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
 ## 1. Host
 | Item | Value |
 |---|---|
 | Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
 | OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
 | Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
 | Timezone | `America/Chicago` (US/Central) |
 | LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
 | Tailscale | `100.103.64.82` (`tailscale0`) |
 | DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
 | Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
 | Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
 `eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
 `ntp_default`, `dns_default`, and several stale ones).
 ## 2. Services overview
 | Service | Form | Running? |
 |---|---|---|
 | **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
 | **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
 | **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
 | **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
 ## 3. Pi-hole (container)
 - **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
 - **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
 - **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
 - **Ports (host):**
  | Host | Container | Purpose |
  |---|---|---|
  | `53/tcp`, `53/udp` | 53 | DNS |
  | `10002/tcp` | 80 | Web admin (HTTP) |
  | `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
 - **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=Gransyan1!`, `FTLCONF_dns_listeningMode=all`
 - **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
 - **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
  - Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
  - `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
  - `queryLogging = true`, DNSSEC off
 - **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
 - **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
 - **Web admin:** `http://pfv-netboot:10002/admin/` (password `Gransyan1!`)
 - Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
 ## 4. NTP (two layers)
 ### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
 - **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
 - **Image:** `dockurr/chrony`
 - **Env:** `NTP_SERVERS=pool.ntp.org`
 - **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
 - `restart: always`
 - chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
 - On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
 ### 4b. bare-metal `ntpsec`
 - Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
 - **Config** (`/etc/ntpsec/ntp.conf`):
  ```
  driftfile /var/lib/ntp/ntp.drift
  leapfile /usr/share/zoneinfo/leap-seconds.list
  server pfvsvrpi.knel.net
  restrict 127.0.0.1
  restrict ::1
  ```
 - Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
 ## 5. Technitium DNS (currently stopped)
 - **Not running** — no container and **no compose file** exists for it.
 - A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
  (mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
  are intact (last activity 2025-06-23). A second typo'd volume
  `dns_tyss-dns-config` is empty.
 - Config files are **binary** (Technitium's own serialization), but copy verbatim:
  `dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
  `self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
 - **Zones present** (12 reverse + 1 forward):
  - `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
  - Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
 - **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
 - The compose project name historically was `dns` (network `dns_default` still exists).
 ## 6. Firewall / misc
 - nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
  `DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
  `OUTPUT ACCEPT`. No UFW / firewalld.
 - Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
  (2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
  usermin (10000/10002/20000), Tailscale (41641).
 - SELinux absent; AppArmor default docker profile.
 ## 7. How it was audited (no changes made)
 ```bash
 ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh        # broad read-only sweep
 ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh   # docker inspect + compose
 ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh       # pihole.toml + technitium
 ```
 Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.
@@ -0,0 +1,220 @@
 # pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
 These two nodes replicate the network-infrastructure services of **pfv-netboot**
 (Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
 reads config from pfv-netboot (read-only) and relays it to each target.
 ## 1. Nodes
 | | pfv-netinfra-01 | pfv-netinfra-02 |
 |---|---|---|
 | OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
 | LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
 | Tailscale | `100.70.181.72` | `100.93.194.82` |
 | RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
 | Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
 | Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
 | Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
 ## 2. Service layout
 All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
 so the compose files are directly editable; data dirs keep container uids):
 ```
 /home/localuser/services/
 ├── pihole/
 │   ├── docker-compose.yml
 │   └── etc-pihole/          # copied from netboot /root/pihole/etc-pihole
 │       ├── pihole.toml      # Pi-hole v6 config (upstreams, etc.)
 │       ├── gravity.db       # adlists / domainlists / clients / groups
 │       ├── adlists.list
 │       ├── dnsmasq.conf
 │       ├── tls.{crt,pem,crt_ca}
 │       └── versions
 ├── ntp/
 │   └── docker-compose.yml   # chrony container (see §5 — not used; host ntpsec serves)
 └── technitium/
    ├── docker-compose.yml
    └── config/              # copied from netboot orphaned volume dns_tsys-dns-config/_data
        ├── dns.config
        ├── auth.config
        ├── scopes/Default.scope
        ├── self-signed-cert.pfx
        └── zones/           # knel.net.zone + 12 Tailscale reverse zones
 ```
 ## 3. Pi-hole (container `pihole`)
 Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
 | Host port | Container | Purpose |
 |---|---|---|
 | `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
 | `10002/tcp` | 80 | Web admin (HTTP) |
 | `10003/tcp` | 443 | Web admin (HTTPS) |
 `docker-compose.yml`:
 ```yaml
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "10002:80/tcp"
      - "10003:443/tcp"
    environment:
      TZ: 'America/Chicago'
      FTLCONF_webserver_api_password: 'Gransyan1!'
      FTLCONF_dns_listeningMode: 'all'
    volumes:
      - './etc-pihole:/etc/pihole'
    cap_add:
      - SYS_NICE
    restart: always
 ```
 - Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
 - Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
 - `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
 - Web admin: `http://<node>:10002/admin/` — password **`Gransyan1!`** (same as netboot).
 - Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
 ## 4. Technitium DNS (container `tsys-dns`)
 Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
 `knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
 orphaned `dns_tsys-dns-config` volume.
 | Host port | Container | Purpose |
 |---|---|---|
 | `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
 | `5380/tcp` | 5380 | Web console (HTTP) |
 | `53443/tcp` | 53443 | Web console (HTTPS) |
 `docker-compose.yml`:
 ```yaml
 services:
  technitium:
    image: technitium/dns-server
    container_name: tsys-dns
    ports:
      - "5300:53/tcp"
      - "5300:53/udp"
      - "5380:5380/tcp"
      - "53443:53443/tcp"
    volumes:
      - './config:/etc/dns'
    restart: always
 ```
 - Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
 - Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
  password (carried over via `auth.config`). If the password is unknown, reset it
  from the console or by removing `config/auth.config` and recreating the container.
 - **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
  because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
  query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
  Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
  Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
  from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
  `knel.net` A-records directly in Pi-hole's Local DNS).
 ## 5. NTP (host `ntpsec`, not a container)
 Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
 serves NTP on every local address — including the Tailscale IP — and keeps the
 system clock synced. This is the **same daemon family as netboot's own bare-metal
 ntpsec**.
 - **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
  Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
  does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
  **does** bind the Tailscale IP, so the container cannot claim it (`address
  already in use`) and would be a non-functional duplicate (verified: the
  container started but never synced — Stratum 0). It is therefore intentionally
  **omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
  host NTP unit and removes any stale `tsys-ntp` container.
 - ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
  `restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
 - Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
  leap normal.
 The `ntp/docker-compose.yml` is still written on each node for parity/reference
 (and in case the host NTP is ever disabled — then `sudo docker compose -f
 /home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
 ## 6. Verification results (2026-07-28)
 | Check | pfv-netinfra-01 | pfv-netinfra-02 |
 |---|---|---|
 | `pihole` health | healthy | healthy |
 | `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
 | Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
 | `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
 | Technitium web `:5380` | HTTP 200 | HTTP 200 |
 | NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
 ## 7. Operating the services
 ```bash
 # status
 sudo docker ps
 # Pi-hole
 sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
 sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
 sudo docker exec pihole pihole -v          # version
 sudo docker exec pihole pihole -g          # rebuild gravity
 sudo docker exec pihole pihole -a -p       # set/change web password
 # Technitium
 sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
 sudo docker exec tsys-dns sh               # explore /etc/dns
 # NTP (host)
 systemctl status ntpsec
 ntpq -pn
 ```
 ## 8. Differences from pfv-netboot (intentional)
 1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
   can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
 2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
 3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
   (the container cannot bind the Tailscale IP here; see §5).
 4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
   on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
 5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
   copied (transient); gravity DB and all configuration are.
 ## 9. Re-running / reproducing
 `setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
 present and uses `docker compose up -d` (no-ops when unchanged). It reads
 pfv-netboot read-only and never mutates it.
 ```bash
 ./setup-netinfra.sh                       # deploy to both nodes
 ./setup-netinfra.sh pfv-netinfra-01       # deploy one node
 ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
 ```
 Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
 sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
 internet for image pulls.
 ## 10. Files in this directory
 | File | Purpose |
 |---|---|
 | `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
 | `audit-netboot.sh` | broad read-only audit of pfv-netboot |
 | `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
 | `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
 | `baseline.sh` | read-only baseline of a target node |
 | `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
 | [`pfv-netboot-setup.md`](pfv-netboot-setup.md) | reference-node documentation |
 | [`pfv-netinfra-setup.md`](pfv-netinfra-setup.md) | this document |
@@ -0,0 +1,323 @@
 #!/usr/bin/env bash
 # =============================================================================
 # setup-netinfra.sh
 # -----------------------------------------------------------------------------
 # Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
 # onto pfv-netinfra-01 and pfv-netinfra-02.
 #
 # DESIGN
 #   * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
 #     from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
 #   * The targets cannot SSH to pfv-netboot directly, so config tarballs are
 #     relayed through this workstation:
 #         ssh netboot 'sudo tar -cf - ...'  |  ssh target 'sudo tar -xf - ...'
 #   * Services are deployed under /home/localuser/services/<svc>/ on each
 #     target so localuser can manage them (mirrors netboot's localuser-owned
 #     pihole data dir). `sudo docker` is used since localuser is not in the
 #     docker group (same as on netboot).
 #
 # SERVICES
 #   pihole      pihole/pihole:latest   :53 tcp/udp  :10002->80  :10003->443
 #   ntp (chrony) dockurr/chrony        <tailscale-ip>:123:123/udp
 #   technitium  technitium/dns-server  :5300->53 tcp/udp  :5380  :53443
 #     (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
 #      The knel.net authoritative zone + Tailscale reverse zones are preserved
 #      verbatim from netboot's orphaned dns_tsys-dns-config volume.)
 #
 # USAGE
 #   ./setup-netinfra.sh                # deploy to BOTH nodes
 #   ./setup-netinfra.sh pfv-netinfra-01   # deploy to one node
 #   ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
 # =============================================================================
 set -euo pipefail
 NETBOOT="localuser@pfv-netboot"
 SVC_ROOT="/home/localuser/services"
 PIHOLE_PW='Gransyan1!'   # replicated verbatim from netboot compose
 log()  { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
 warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
 # Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
 declare -A NODE_TSIP=(
  [pfv-netinfra-01]="100.70.181.72"
  [pfv-netinfra-02]="100.93.194.82"
 )
 on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
 #------------------------------------------------------------------------------
 # Verify-only mode
 #------------------------------------------------------------------------------
 verify_node() {
  local node="$1" tsip="${NODE_TSIP[$1]}"
  log "VERIFY $node (tailscale $tsip)"
  on_node "$node" "bash -s" <<EOF
 set +e
 echo "### containers ###"
 sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
 echo
 echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
 echo "### Pi-hole -> Technitium (dig @53 knel.net SOA) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 knel.net SOA 2>&1
 echo "### Pi-hole -> Technitium (dig @53 pfv-netboot.knel.net A) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pfv-netboot.knel.net A 2>&1
 echo "### Pi-hole web (curl :10002) ###"
 curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
 echo
 echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
 dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
 echo "### Technitium web (curl :5380) ###"
 curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
 echo
 echo "### NTP service ###"
 HOST_NTP=""
 for u in ntpsec ntp chrony openntpd; do
  systemctl is-active --quiet "\$u" 2>/dev/null && { HOST_NTP="\$u"; break; }
 done
 echo "host daemon: \${HOST_NTP:-none}"
 if [ -n "\$HOST_NTP" ]; then
  ntpq -c "rv 0 leap,stratum,offset" 2>&1 | head -3
 else
  echo "(no host NTP; chrony container:)"
  sudo docker exec tsys-ntp chronyc -n tracking 2>&1 | head -6
 fi
 EOF
 }
 #------------------------------------------------------------------------------
 # Deploy to one node
 #------------------------------------------------------------------------------
 deploy_node() {
  local node="$1" tsip="${NODE_TSIP[$1]}"
  log "==== DEPLOY $node (tailscale $tsip) ===="
  # ---- 1. Prepare directories on the target -------------------------------
  log "$node: create service dirs"
  on_node "$node" "bash -s" <<EOF
 set -e
 sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
 sudo chown -R localuser:localuser $SVC_ROOT
 EOF
  # ---- 2. Write compose files (as localuser) ------------------------------
  log "$node: write docker-compose files"
  on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "10002:80/tcp"
      - "10003:443/tcp"
    environment:
      TZ: 'America/Chicago'
      FTLCONF_webserver_api_password: 'Gransyan1!'
      FTLCONF_dns_listeningMode: 'all'
    volumes:
      - './etc-pihole:/etc/pihole'
    cap_add:
      - SYS_NICE
    restart: always
    networks:
      - default
      - dnsnet
 networks:
  dnsnet:
    external: true
 YAML
  on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
 services:
  ntp:
    image: dockurr/chrony
    container_name: tsys-ntp
    environment:
      NTP_SERVERS: "pool.ntp.org"
    ports:
      - "$tsip:123:123/udp"
    restart: always
 YAML
  on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
 services:
  technitium:
    image: technitium/dns-server
    container_name: tsys-dns
    ports:
      - "5300:53/tcp"
      - "5300:53/udp"
      - "5380:5380/tcp"
      - "53443:53443/tcp"
    volumes:
      - './config:/etc/dns'
    restart: always
    networks:
      default:
      dnsnet:
        ipv4_address: 10.53.0.53
 networks:
  dnsnet:
    external: true
 YAML
  # ---- 3. Relay Pi-hole config from netboot -> target ---------------------
  log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
  if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
    log "$node: Pi-hole config already present; skipping copy"
  else
    on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
        "sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
                                   --exclude='etc-pihole/listsCache' \
                                   --exclude='etc-pihole/gravity_backups' \
                                   --exclude='etc-pihole/config_backups' \
                                   etc-pihole" \
      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
  fi
  # ---- 4. Relay Technitium config from netboot orphaned volume -----------
  log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
  if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
    log "$node: Technitium config already present; skipping copy"
  else
    on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
        "sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
  fi
  # ---- 5. Adapt copied config: interface + repoint knel.net to local Technitium
  log "$node: adapt Pi-hole pihole.toml (interface + revServer -> local Technitium)"
  on_node "$node" "bash -s" <<'EOF'
 set -e
 IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
 IFACE=${IFACE:-ens18}
 TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
 if sudo test -f "$TOML"; then
  sudo sed -i "s|^  interface = .*|  interface = \"$IFACE\"  ### ADAPTED from eth0 on clone|" "$TOML"
  echo "set interface=$IFACE"
  # Repoint knel.net conditional forward from netboot's upstream (192.168.3.16)
  # to the LOCAL Technitium container at its fixed dnsnet IP 10.53.0.53.
  # Subnet 100.64.0.0/10 = Tailscale CGNAT range (covers all Tailscale reverse zones).
  if sudo grep -q 'revServers' "$TOML"; then
    sudo sed -i 's|"true,[0-9./]*,192\.168\.3\.16,knel\.net"|"true,100.64.0.0/10,10.53.0.53,knel.net"|' "$TOML"
    echo "revServer repointed to 10.53.0.53 (local Technitium)"
  else
    echo "(revServers not found; FTL will use defaults)"
  fi
 else
  echo "(pihole.toml not present; FTL will create it on first run)"
 fi
 EOF
  # ---- 5b. Create shared Docker network for Pi-hole <-> Technitium ----------
  log "$node: create dnsnet shared Docker network (10.53.0.0/24)"
  on_node "$node" "sudo docker network create --subnet 10.53.0.0/24 dnsnet 2>/dev/null || true"
  # ---- 6. Pull images -----------------------------------------------------
  log "$node: docker compose pull (pihole, ntp, technitium)"
  on_node "$node" "bash -s" <<EOF
 for c in pihole ntp technitium; do
  sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
 done
 EOF
  # ---- 6a. Pi-hole up -----------------------------------------------------
  log "$node: bring up Pi-hole"
  on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
  # ---- 6b. NTP -- only deploy the chrony container if nothing already ----
  # serves UDP/123 on the host. The targets already run a bare-metal ntpsec
  # daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
  # netboot's own bare-metal ntpsec. netboot additionally runs a chrony
  # container on its tailscale IP, but that only works there because ntpsec
  # there does not pre-bind the specific tailscale-IP socket. On these targets
  # ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
  # redundant anyway. We therefore keep the host ntpsec as the NTP service.
  log "$node: NTP -- detect host NTP service"
  on_node "$node" "bash -s" <<'EOF'
 set +e
 HOST_NTP=""
 for u in ntpsec ntp chrony openntpd; do
  if systemctl is-active --quiet "$u" 2>/dev/null; then HOST_NTP="$u"; break; fi
 done
 if [ -n "$HOST_NTP" ]; then
  echo "Host NTP daemon '$HOST_NTP' is active -- it serves NTP on all local"
  echo "addresses (incl. the Tailscale IP). This is the same daemon family as"
  echo "netboot's bare-metal ntpsec; the netboot chrony container is redundant"
  echo "here and CANNOT bind the Tailscale IP (the host daemon already owns it)."
  echo "-> Keeping host NTP. Removing any stale chrony container (tsys-ntp)."
  sudo docker rm -f tsys-ntp 2>/dev/null && echo "   (removed tsys-ntp)" || echo "   (no tsys-ntp to remove)"
  echo "   host peers:"; ntpq -pn 2>/dev/null | head -12 || true
 else
  echo "No host NTP daemon active; starting chrony container."
  sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
 fi
 EOF
  # ---- 6c. Technitium up --------------------------------------------------
  log "$node: bring up Technitium"
  on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
  # ---- 7. Wait for Pi-hole health -----------------------------------------
  log "$node: wait for Pi-hole to become healthy"
  on_node "$node" "bash -s" <<'EOF'
 for i in $(seq 1 30); do
  st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
  echo "  pihole health: $st"
  [ "$st" = "healthy" ] && break
  sleep 4
 done
 EOF
  # ---- 8. Technitium defensive fallback -----------------------------------
  # If the copied (binary) config from an older Technitium version makes the
  # new container crash, move it aside and let Technitium start fresh so the
  # service is at least up (admin reachable) rather than crash-looping.
  log "$node: check Technitium health (fallback to fresh config if crash)"
  on_node "$node" "bash -s" <<'EOF'
 set +e
 sleep 8
 rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
 running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
 if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
  echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
  sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
  sudo mkdir -p /home/localuser/services/technitium/config
  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
  echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
 else
  echo "Technitium OK (running=$running restarts=$rst)"
 fi
 EOF
  # ---- 9. Final status -----------------------------------------------------
  log "$node: final container status"
  on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
 }
 #------------------------------------------------------------------------------
 # Main
 #------------------------------------------------------------------------------
 main() {
  local mode="${2:-deploy}"
  if [ "${1:-all}" = "all" ]; then
    targets=(pfv-netinfra-01 pfv-netinfra-02)
  else
    targets=("$1")
  fi
  for t in "${targets[@]}"; do
    : "${NODE_TSIP[$t]:?unknown node $t}"
    if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
  done
  log "DONE"
 }
 main "$@"
@@ -0,0 +1,29 @@
 #!/bin/bash
 # check-pkgs.sh - verify package install state.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== $host ==="
    ssh "${SSH_OPTS[@]}" "root@$host" '
        for p in sysstat jq numactl nvme mtr-tiny dig bmon tcpdump; do
            if command -v "$p" >/dev/null 2>&1; then
                echo "  ✓ $p"
            else
                echo "  ✗ $p"
            fi
        done
        # sysstat config
        echo "  sysstat service:"
        systemctl list-unit-files 2>/dev/null | grep -i sysstat | sed "s/^/    /"
        echo "  sysstat enabled in /etc/default:"
        if [ -r /etc/default/sysstat ]; then
            grep ENABLED /etc/default/sysstat | sed "s/^/    /"
        else
            echo "    no /etc/default/sysstat"
        fi
        # on Debian trixie, sysstat uses a different path
        ls /etc/cron.d/sysstat* 2>/dev/null | sed "s/^/  found: /"
    '
    echo ""
 done
@@ -0,0 +1,41 @@
 #!/bin/bash
 # check-repos-and-reboot.sh - checks reboot-required + Proxmox repo config on all hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host]"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        continue
    fi
    echo "--- /var/run/reboot-required ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'if [ -r /var/run/reboot-required ]; then echo "REBOOT REQUIRED"; cat /var/run/reboot-required 2>/dev/null; if [ -r /var/run/reboot-required.pkgs ]; then echo "Packages triggering:"; cat /var/run/reboot-required.pkgs; fi; else echo "(no reboot required marker)"; fi'
    echo ""
    echo "--- Running kernel vs installed kernel ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'echo "running: $(uname -r)"; echo "installed:"; dpkg -l | grep -E "pve-kernel-[0-9]" | awk "{print \"  \"\$2\"  \"\$3}" | tail -5'
    echo ""
    echo "--- Proxmox repositories (apt sources) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rh "pve\|proxmox" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | grep -v "^#" | sed "s/^/  /"'
    echo ""
    echo "--- Enterprise repo status (should be commented or absent if no subscription) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -l "pve-enterprise" /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null | while read f; do echo "  File: $f"; grep -n "pve-enterprise" "$f" | sed "s/^/    /"; done'
    echo ""
    echo "--- no-subscription repo presence ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rl "pve-no-subscription" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | while read f; do echo "  File: $f"; grep -n "pve-no-subscription" "$f" | sed "s/^/    /"; done'
    echo ""
    echo "--- Recently updated packages (last 24h, kernel-related) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -E "pve-kernel|proxmox|pve-qemu|zfs" /var/log/dpkg.log 2>/dev/null | grep "$(date +%Y-%m-%d)\|$(date -d yesterday +%Y-%m-%d)" | tail -15 || echo "(none in dpkg.log)"'
    echo ""
 done
@@ -0,0 +1,51 @@
 #!/bin/bash
 # deploy-and-fix.sh - uploads fix script, runs it, starts VMs, verifies.
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
 echo "=== Uploading fix script to $HOST ==="
 scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
 echo "=== Running fix ==="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/fix-bond-nfs.sh && bash /root/fix-bond-nfs.sh' 2>&1
 echo ""
 echo "=== Starting VMs ==="
 for vmid in $(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"'); do
    status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
    if [ "$status" != "running" ]; then
        echo "  Starting VM $vmid..."
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/    /'
    else
        echo "  VM $vmid already running"
    fi
 done
 echo ""
 echo "Waiting 20s for VMs to boot..."
 sleep 20
 echo ""
 echo "=== FULL VERIFICATION ==="
 echo ""
 echo "--- VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
 echo ""
 echo "--- NFS mounts ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
 echo ""
 echo "--- NFS TCP connections ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 echo "  Count:"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
 echo ""
 echo "--- bond0 hash policy ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -6'
 echo ""
 echo "--- Summary ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "swappiness: $(sysctl -n vm.swappiness)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
@@ -0,0 +1,169 @@
 #!/bin/bash
 ###############################################################################
 # deploy-check.sh
 #
 # Deploys scripts/check.sh to each reachable Proxmox host, executes it
 # read-only, and pulls the resulting log back to returned-logs/.
 #
 # EXPLICITLY SKIPS:
 #   - pfv-tsys2   (off the air per user)
 #   - pfv-tsys9   (off the air per user; also not in original inventory)
 #
 # Safety features:
 #   - BatchMode=yes        : never hang on a password prompt
 #   - ConnectTimeout=8     : fail fast on dead hosts
 #   - per-host try/skip    : one bad host never aborts the run
 #   - ServerAliveInterval  : detect hung connections
 #   - read-only script     : check.sh modifies nothing on the target
 ###############################################################################
 set -uo pipefail
 SCRIPT_DIR="/home/reachableceo/projects/perfopt"
 CHECK_SH="$SCRIPT_DIR/scripts/check.sh"
 LOG_DIR="$SCRIPT_DIR/returned-logs"
 mkdir -p "$LOG_DIR"
 # ONLY the hosts the user told us are alive.
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 # Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
 SSH_OPTS=(-o BatchMode=yes
          -o ConnectTimeout=8
          -o ServerAliveInterval=10
          -o ServerAliveCountMax=3
          -o StrictHostKeyChecking=accept-new)
 log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
 if [ ! -r "$CHECK_SH" ]; then
    echo "FATAL: $CHECK_SH not found" >&2
    exit 1
 fi
 # Sanity-check shellcheck clean before shipping (best-effort, non-blocking)
 if command -v docker >/dev/null 2>&1; then
    log "pre-flight: shellcheck on check.sh"
    if ! docker run --rm -v "$SCRIPT_DIR:/mnt" -w /mnt \
         koalaman/shellcheck:stable --severity=style --format=gcc scripts/check.sh \
         >"$LOG_DIR/_shellcheck.preflight.txt" 2>&1; then
        log "WARNING: shellcheck reported issues — see _shellcheck.preflight.txt"
        log "         aborting deploy to avoid shipping a broken script"
        exit 1
    fi
    log "pre-flight: shellcheck clean"
 fi
 summary_pass=()
 summary_fail=()
 declare -A HOST_PID        # host -> background pid
 declare -A HOST_MARKER     # host -> per-host marker file
 # Per-host worker — runs in background, one per host, all in parallel.
 # Writes status into a marker file consumed by the parent.
 worker() {
    local host="$1"
    local marker="$LOG_DIR/_marker.$host"
    : > "$marker"  # truncate
    echo "running" >> "$marker"
    local short=""
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "fail unreachable" >> "$marker"
        return
    fi
    if ! scp "${SSH_OPTS[@]}" "$CHECK_SH" "root@$host:/root/check.sh" >/dev/null 2>&1; then
        echo "fail scp-upload-failed" >> "$marker"
        return
    fi
    local remote_size
    remote_size=$(ssh "${SSH_OPTS[@]}" "root@$host" 'wc -c < /root/check.sh' 2>/dev/null || echo 0)
    if [ "${remote_size:-0}" -lt 1000 ]; then
        echo "fail upload-corrupt" >> "$marker"
        return
    fi
    local remote_stdout
    remote_stdout=$(ssh "${SSH_OPTS[@]}" "root@$host" \
        'chmod +x /root/check.sh && bash /root/check.sh' 2>&1)
    local rc=$?
    if [ "$rc" -ne 0 ]; then
        echo "fail check-exit-$rc" >> "$marker"
        # don't return - still try to pull whatever log got produced
    fi
    short=$(printf '%s\n' "$remote_stdout" | grep -oE 'Wrote: /root/[a-zA-Z0-9_-]+\.log' | head -n1 | awk '{print $2}')
    if [ -z "$short" ]; then
        short=$(ssh "${SSH_OPTS[@]}" "root@$host" 'echo "/root/$(hostname -s).log"' 2>/dev/null)
    fi
    if [ -z "$short" ]; then
        echo "fail no-log-path" >> "$marker"
        return
    fi
    if ! scp "${SSH_OPTS[@]}" "root@$host:$short" "$LOG_DIR/" >/dev/null 2>&1; then
        echo "fail scp-download-failed" >> "$marker"
        return
    fi
    local local_name local_path
    local_name="$(basename "$short")"
    local_path="$LOG_DIR/$local_name"
    if [ ! -s "$local_path" ]; then
        echo "fail local-empty" >> "$marker"
        return
    fi
    echo "ok $local_name $(wc -c < "$local_path") $(wc -l < "$local_path")" >> "$marker"
 }
 # ---- launch all workers in parallel --------------------------------------
 log "launching ${#HOSTS[@]} hosts in parallel..."
 for host in "${HOSTS[@]}"; do
    rm -f "$LOG_DIR/_marker.$host"
    worker "$host" &
    HOST_PID[$host]=$!
    HOST_MARKER[$host]="$LOG_DIR/_marker.$host"
    log "  launched $host (pid ${HOST_PID[$host]})"
 done
 # ---- wait for all, with periodic progress --------------------------------
 remaining=("${HOSTS[@]}")
 while [ "${#remaining[@]}" -gt 0 ]; do
    sleep 10
    new_remaining=()
    for host in "${remaining[@]}"; do
        if ! kill -0 "${HOST_PID[$host]}" 2>/dev/null; then
            # process finished
            wait "${HOST_PID[$host]}" 2>/dev/null || true
            marker="${HOST_MARKER[$host]}"
            if [ -r "$marker" ]; then
                status_line="$(tail -n1 "$marker")"
                log "[$host] done: $status_line"
                case "$status_line" in
                    ok*)     summary_pass+=("$host:$status_line") ;;
                    fail*)   summary_fail+=("$host:$status_line") ;;
                    *)       summary_fail+=("$host:unknown") ;;
                esac
            else
                log "[$host] done but marker missing"
                summary_fail+=("$host:no-marker")
            fi
        else
            new_remaining+=("$host")
        fi
    done
    remaining=("${new_remaining[@]:-}")
    if [ "${#remaining[@]}" -gt 0 ]; then
        log "still running: ${remaining[*]} (${#remaining[@]} hosts)"
    fi
 done
 # Final summary
 log "============================================================"
 log "DEPLOY SUMMARY"
 log "============================================================"
 log "Passed (${#summary_pass[@]}):"
 for p in "${summary_pass[@]:-}"; do [ -n "$p" ] && log "  ✓ $p"; done
 log "Failed (${#summary_fail[@]}):"
 for f in "${summary_fail[@]:-}"; do [ -n "$f" ] && log "  ✗ $f"; done
 log ""
 log "Contents of $LOG_DIR:"
 ls -la "$LOG_DIR"
 # Clean up marker files
 rm -f "$LOG_DIR"/_marker.* 2>/dev/null
@@ -0,0 +1,56 @@
 #!/bin/bash
 # deploy-tuning.sh - copies apply-tunings.sh to target hosts and runs it.
 # Usage: bash deploy-tuning.sh [--no-nfs] [--apply] <host> [host...]
 # Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section.
 set -uo pipefail
 SCRIPT="/home/reachableceo/projects/perfopt/scripts/apply-tunings.sh"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 MODE=""
 EXTRA_FLAGS=""
 HOSTS=()
 for arg in "$@"; do
    case "$arg" in
        --apply) MODE="--apply" ;;
        --dry-run) MODE="" ;;
        --no-nfs) EXTRA_FLAGS="--no-nfs" ;;
        *) HOSTS+=("$arg") ;;
    esac
 done
 if [ "${#HOSTS[@]}" -eq 0 ]; then
    echo "Usage: $0 <host> [host...] [--apply]"
    echo "Default: dry-run. Pass --apply to commit."
    exit 1
 fi
 if [ ! -r "$SCRIPT" ]; then
    echo "FATAL: $SCRIPT not found"
    exit 1
 fi
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host] deploying apply-tunings.sh (mode: ${MODE:-dry-run})"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "[$host] SKIP: unreachable"
        continue
    fi
    echo "[$host] uploading..."
    if ! scp "${SSH_OPTS[@]}" "$SCRIPT" "root@$host:/root/apply-tunings.sh" >/dev/null 2>&1; then
        echo "[$host] SKIP: scp failed"
        continue
    fi
    echo "[$host] running (output below)..."
    echo "----------------------------------------------------------------"
    ssh "${SSH_OPTS[@]}" "root@$host" "chmod +x /root/apply-tunings.sh && bash /root/apply-tunings.sh $MODE $EXTRA_FLAGS" 2>&1
    rc=$?
    echo "----------------------------------------------------------------"
    echo "[$host] exit code: $rc"
    echo ""
 done
@@ -0,0 +1,35 @@
 #!/bin/bash
 # diag.sh - diagnostic commands run on a host via SSH wrapper.
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 echo "===== 1. storage.cfg NFS stanzas (exact content) ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'grep -A 8 "^nfs: D2" /etc/pve/storage.cfg'
 echo ""
 echo "===== 2. Try manual NFS mount with nconnect=4 ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
 echo ""
 echo "===== 3. Try manual NFS mount WITHOUT nconnect ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
 echo ""
 echo "===== 4. NFS kernel version / module ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/fs/nfsfs/version 2>/dev/null; echo "---"; modinfo nfs 2>/dev/null | grep -E "^(filename|version|description)" | head -5'
 echo ""
 echo "===== 5. mount.nfs version ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount.nfs --version 2>&1; echo "---"; dpkg -l nfs-common 2>/dev/null | tail -2'
 echo ""
 echo "===== 6. /etc/network/interfaces bond0 stanza (exact bytes) ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sed -n "/^auto bond0/,/^$/p" /etc/network/interfaces | cat -A'
 echo ""
 echo "===== 7. Current bond0 running hash policy ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -5'
 echo ""
 echo "===== 8. xmit_hash_policy sysfs file ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; echo "---"; ls /sys/class/net/bond0/bonding/ 2>/dev/null'
@@ -0,0 +1,128 @@
 #!/bin/bash
 # finish-host.sh - applies ALL remaining changes to a host and verifies.
 #
 # Steps:
 #   1. Start all VMs (triggers NFS lazy-mount)
 #   2. Wait for NFS mounts to appear
 #   3. Verify NFS nconnect=4 + noatime
 #   4. Apply bond0 xmit_hash_policy=layer3+4
 #   5. Full end-to-end verification
 #
 # Usage: bash finish-host.sh <host> [--apply]
 # Default is dry-run (starts VMs + shows what bond change would do, but doesn't edit interfaces)
 set -uo pipefail
 HOST="${1:-}"
 MODE="${2:-dryrun}"
 [ "$MODE" = "--apply" ] && MODE="apply" || MODE="dryrun"
 if [ -z "$HOST" ]; then
    echo "Usage: $0 <host> [--apply]"
    exit 1
 fi
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
 echo "==================================================================="
 echo " finish-host.sh — $HOST  (mode: $MODE)"
 echo "==================================================================="
 echo ""
 # =========================================================================
 # STEP 1: Start all VMs
 # =========================================================================
 echo "=== STEP 1: Start all VMs on $HOST ==="
 # Get list of all VMs (not just stopped — start is idempotent)
 vm_list=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"')
 for vmid in $vm_list; do
    status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
    if [ "$status" != "running" ]; then
        echo "  Starting VM $vmid..."
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/    /'
    else
        echo "  VM $vmid already running"
    fi
 done
 echo ""
 echo "  Waiting 15s for VMs to boot and trigger NFS mounts..."
 sleep 15
 echo ""
 echo "--- VM status after start ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 # =========================================================================
 # STEP 2: Verify NFS mounts came back with nconnect=4
 # =========================================================================
 echo ""
 echo "=== STEP 2: Verify NFS mounts with nconnect=4 ==="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null' | head -30
 echo ""
 echo "--- NFS TCP connections to :2049 ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 conn_count=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
 echo "  Total NFS TCP connections: $conn_count"
 # =========================================================================
 # STEP 3: Apply bond0 hash policy
 # =========================================================================
 echo ""
 echo "=== STEP 3: Apply bond0 xmit_hash_policy=layer3+4 (mode: $MODE) ==="
 # Upload the bond hash script
 scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/apply-bond-hash.sh" "root@$HOST:/root/apply-bond-hash.sh" >/dev/null 2>&1
 if [ "$MODE" = "apply" ]; then
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh --apply' 2>&1
 else
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh' 2>&1
 fi
 # =========================================================================
 # STEP 4: Full verification
 # =========================================================================
 echo ""
 echo "=== STEP 4: Full end-to-end verification ==="
 echo ""
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
 echo ""
 echo "--- CPU governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
 echo ""
 echo "--- TCP BBR ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
 echo ""
 echo "--- tuned profile ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
 echo ""
 echo "--- bond0 hash policy + LACP state ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 2>/dev/null | head -25'
 echo ""
 echo "--- NFS mount options (first 3 mounts) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
 echo ""
 echo "--- NFS TCP connections (expect 4 per server × 2 servers = 8) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 nfs_conns=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
 echo "  Count: $nfs_conns"
 echo ""
 echo "--- VMs running ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 echo ""
 echo "--- Failed services ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
 echo ""
 echo "--- Network interfaces (speed/duplex/mtu) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'for ifc in bond0 nic0 nic1 nic2 vmbr0 datanet; do [ -d "/sys/class/net/$ifc" ] && printf "%-12s speed=%-8s duplex=%-8s mtu=%s\n" "$ifc" "$(cat /sys/class/net/$ifc/speed 2>/dev/null)" "$(cat /sys/class/net/$ifc/duplex 2>/dev/null)" "$(cat /sys/class/net/$ifc/mtu 2>/dev/null)"; done'
 echo ""
 echo "==================================================================="
 echo " COMPLETE — $HOST"
 echo "==================================================================="
@@ -0,0 +1,37 @@
 #!/bin/bash
 # install-utils-v2.sh - retry install without nstat package.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 PKGS="sysstat jq numactl nvme-cli mtr-tiny dnsutils bmon"
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== [$host] installing: $PKGS ==="
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>&1 | tail -2 && \
         DEBIAN_FRONTEND=noninteractive apt-get install -y $PKGS 2>&1 | tail -10"
    # sysstat enable (path varies by Debian version)
    ssh "${SSH_OPTS[@]}" "root@$host" '
        if [ -r /etc/default/sysstat ]; then
            sed -i "s/^ENABLED=.*/ENABLED=\"true\"/" /etc/default/sysstat
            systemctl enable --now sysstat 2>/dev/null
            grep ENABLED /etc/default/sysstat
        else
            # Newer Debian (trixie) — sysstat cron/service auto-enabled
            systemctl enable --now sysstat 2>/dev/null || echo "(sysstat auto via cron)"
        fi
    '
    echo ""
 done
 # Verify
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== [$host] verification ==="
    ssh "${SSH_OPTS[@]}" "root@$host" '
        for p in sysstat jq numactl nvme mtr-tiny dig bmon; do
            command -v "$p" >/dev/null 2>&1 && echo "  ✓ $p" || echo "  ✗ $p"
        done
    '
    echo ""
 done
@@ -0,0 +1,60 @@
 #!/bin/bash
 # install-utils.sh - installs useful observability packages on a host.
 # These are all small, dependency-light, and read-only at runtime.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys6 pfv-tsys7)
 # Packages to install, with rationale
 PKGS=(
    sysstat          # sar, iostat, mpstat, pidstat - the missing observability suite
    jq               # JSON parsing for pvesh/scripts
    numactl          # NUMA topology/controls for the dual-socket hosts
    nvme-cli         # NVMe health (for when NVMe shows up)
    tcpdump          # packet capture for network debugging
    mtr-tiny         # traceroute on steroids
    nstat            # kernel SNMP stats (already partly there)
    dnsutils         # dig, nslookup, host
    bmon             # bandwidth monitor ( curses, real-time)
 )
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host] installing observability packages"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        continue
    fi
    # Check which are missing
    missing=""
    for pkg in "${PKGS[@]}"; do
        if ! ssh "${SSH_OPTS[@]}" "root@$host" "dpkg -s $pkg 2>/dev/null | grep -q 'Status: install ok installed'" 2>/dev/null; then
            missing="$missing $pkg"
        fi
    done
    if [ -z "$missing" ]; then
        echo "  All packages already installed."
        continue
    fi
    echo "  Installing:$missing"
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
         DEBIAN_FRONTEND=noninteractive apt-get install -y -qq$missing 2>&1 | tail -5"
    echo "  Done."
    echo ""
 done
 # Enable sysstat data collection (sar) — off by default on Debian
 for host in "${HOSTS[@]}"; do
    echo "[$host] enabling sysstat/sar data collection..."
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "sed -i 's/^ENABLED=\"false\"/ENABLED=\"true\"/' /etc/default/sysstat 2>/dev/null; \
         systemctl enable --now sysstat 2>&1 | tail -2; \
         grep ENABLED /etc/default/sysstat"
 done
@@ -0,0 +1,293 @@
 #!/bin/bash
 ###############################################################################
 # iperf-full-matrix.sh
 #
 # Two test suites:
 #   A. Management network (vmbr0 / VLAN1): all-pairs single-stream TCP, 10s
 #   B. Storage network (VLAN1000): tsys6+tsys7 → tsys4+tsys5, stress test
 #
 # Output: returned-logs/iperf/
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 # Storage IPs (known, static on VLAN1000)
 declare -A SIP
 SIP[pfv-tsys1]="10.100.100.1"
 SIP[pfv-tsys3]="10.100.100.3"
 SIP[pfv-tsys4]="10.100.100.4"
 SIP[pfv-tsys5]="10.100.100.5"
 SIP[pfv-tsys6]="10.100.100.6"
 SIP[pfv-tsys7]="10.100.100.7"
 # ===========================================================================
 # STEP 0: Discover management IPs (vmbr0)
 # ===========================================================================
 echo "==================================================================="
 echo " STEP 0: Discover management network IPs (vmbr0)"
 echo "==================================================================="
 declare -A MIP
 for host in "${HOSTS[@]}"; do
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  [$host] UNREACHABLE"
        continue
    fi
    mip=$(ssh "${SSH_OPTS[@]}" "root@$host" 'ip -o -4 addr show dev vmbr0 2>/dev/null | awk "{print \$4}" | cut -d/ -f1 | head -1')
    if [ -n "$mip" ]; then
        MIP[$host]="$mip"
        echo "  [$host] vmbr0 = $mip"
    else
        echo "  [$host] no vmbr0 IPv4 — skipping"
    fi
 done
 # ===========================================================================
 # STEP 1: Ensure iperf3 installed on all hosts
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " STEP 1: Ensure iperf3 installed"
 echo "==================================================================="
 for host in "${HOSTS[@]}"; do
    [ -z "${MIP[$host]:-}" ] && continue
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
        echo -n "  [$host] installing iperf3... "
        ssh "${SSH_OPTS[@]}" "root@$host" \
            'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
             DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
    else
        echo "  [$host] iperf3 already installed"
    fi
 done
 # Helper: start iperf3 server in one-shot mode bound to a specific IP
 start_server() {
    local host="$1" ip="$2"
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
 }
 # Helper: run iperf3 client test, save output, extract result
 run_test() {
    local client="$1" server="$2" sip="$3" cip="$4" label="$5" logfile="$6"
    shift 6
    local extra="$*"
    echo -n "  [$label] ... "
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client ($cip)  →  Server: $server ($sip)"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo "Args: $extra"
        echo ""
        ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${sip} -B ${cip} ${extra}" 2>&1
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract result
    sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$logfile" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    echo "${bitrate:-?} (retrans: ${retrans:-?})"
 }
 # ===========================================================================
 # SUITE A: Management network (vmbr0) — all pairs, single-stream TCP, 10s
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " SUITE A: MANAGEMENT NETWORK (vmbr0) — all-pairs, 10s TCP"
 echo " Expectation: ~940 Mbps for every pair (1 GbE line rate)"
 echo "==================================================================="
 echo ""
 # Test each unique pair (i < j to avoid duplicates)
 for ((i=0; i<${#HOSTS[@]}; i++)); do
    for ((j=i+1; j<${#HOSTS[@]}; j++)); do
        client="${HOSTS[$i]}"
        server="${HOSTS[$j]}"
        [ -z "${MIP[$client]:-}" ] && continue
        [ -z "${MIP[$server]:-}" ] && continue
        label="mgmt: ${client}→${server}"
        logfile="$LOG_DIR/mgmt-${client}-to-${server}.log"
        start_server "$server" "${MIP[$server]}"
        run_test "$client" "$server" "${MIP[$server]}" "${MIP[$client]}" \
            "$label" "$logfile" "-t 10 -P 1"
    done
 done
 # ===========================================================================
 # SUITE B: Storage network (VLAN1000) — stress test the cross-rack LACP
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " SUITE B: STORAGE NETWORK (VLAN1000) — stress test cross-rack link"
 echo " tsys6 + tsys7 (Rack 3) → tsys4 + tsys5 (Rack 5)"
 echo " Expectation: limited by tsys4 USB dongle + tsys5 broken bond"
 echo "==================================================================="
 echo ""
 # --- B.1: Individual tests (one client → one server at a time) ---
 echo "--- B.1: Individual tests (sequential) ---"
 echo ""
 for client in pfv-tsys6 pfv-tsys7; do
    for server in pfv-tsys4 pfv-tsys5; do
        label="stor: ${client}→${server} (8-stream)"
        logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
        start_server "$server" "${SIP[$server]}"
        run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
            "$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
    done
 done
 # --- B.2: Reverse direction (tsys4/5 → tsys6/7) ---
 echo ""
 echo "--- B.2: Reverse direction (tsys4/5 → tsys6/7) ---"
 echo ""
 for client in pfv-tsys4 pfv-tsys5; do
    for server in pfv-tsys6 pfv-tsys7; do
        label="stor: ${client}→${server} (8-stream rev)"
        logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
        start_server "$server" "${SIP[$server]}"
        run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
            "$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
    done
 done
 # --- B.3: Simultaneous stress test (4 flows at once) ---
 echo ""
 echo "--- B.3: Simultaneous 4-flow stress test ---"
 echo "  tsys6→tsys4 + tsys6→tsys5 + tsys7→tsys4 + tsys7→tsys5"
 echo "  All running in parallel for 30 seconds"
 echo ""
 # Start 4 iperf3 servers (one-shot mode won't work for parallel; use persistent)
 for server in pfv-tsys4 pfv-tsys5; do
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -B ${SIP[$server]} >/dev/null 2>&1 &" 2>/dev/null
    echo "  [server started: $server]"
 done
 sleep 1
 STRESS_LOG="$LOG_DIR/stor-stress-4flow"
 mkdir -p "$STRESS_LOG"
 # Launch 4 clients in parallel, each writing to its own log
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys4.log" 2>&1 &
 PID1=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys5.log" 2>&1 &
 PID2=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys4.log" 2>&1 &
 PID3=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys5.log" 2>&1 &
 PID4=$!
 echo "  [4 clients launched, waiting 40s for completion...]"
 wait $PID1 $PID2 $PID3 $PID4 2>/dev/null
 echo "  [all 4 flows complete]"
 # Kill servers
 for server in pfv-tsys4 pfv-tsys5; do
    ssh "${SSH_OPTS[@]}" "root@$server" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 # ===========================================================================
 # SUITE C: All hosts cleanup
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " Cleanup: killing iperf3 everywhere"
 echo "==================================================================="
 for host in "${HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 # ===========================================================================
 # RESULTS SUMMARY
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " RESULTS SUMMARY"
 echo "==================================================================="
 echo ""
 echo "===== SUITE A: Management network (vmbr0) ====="
 echo ""
 printf "%-40s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-40s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/mgmt-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    single=$(grep 'sender$' "$f" | tail -1)
    bitrate=$(echo "$single" | awk '{print $7, $8}')
    retrans=$(echo "$single" | awk '{print $9}')
    printf "%-40s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
 done
 echo ""
 echo "===== SUITE B.1+B.2: Storage network individual ====="
 echo ""
 printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-45s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/stor-indiv-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$f" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    printf "%-45s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
 done
 echo ""
 echo "===== SUITE B.3: Simultaneous 4-flow stress test ====="
 echo ""
 printf "%-30s %15s %10s\n" "FLOW" "THROUGHPUT" "RETRANS"
 printf "%-30s %15s %10s\n" "----" "----------" "-------"
 total_mbps=0
 for f in "$STRESS_LOG"/*.log; do
    [ -r "$f" ] || continue
    flow=$(basename "$f" .log)
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
        mbps=$(echo "$sum" | awk '{print $6}')
        total_mbps=$(awk "BEGIN{print $total_mbps + $mbps}")
    else
        bitrate="?"
        retrans="-"
    fi
    printf "%-30s %15s %10s\n" "$flow" "$bitrate" "${retrans:--}"
 done
 printf "%-30s %15s\n" "AGGREGATE (all 4 flows)" "${total_mbps} Mbits/sec"
 echo ""
 echo "===== CONTEXT ====="
 echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
 echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
 echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
 echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
 echo ""
 echo "All logs in: $LOG_DIR/"
@@ -0,0 +1,184 @@
 #!/bin/bash
 ###############################################################################
 # iperf-storage-tests.sh
 #
 # Installs iperf3 on all online hosts, then runs a matrix of storage-network
 # throughput tests. Saves all output to returned-logs/iperf/.
 #
 # Test matrix (all over VLAN1000 storage network, 10.100.100.0/24):
 #   1. tsys7 → tsys4 (USB cdc_ncm NIC) — the smoking gun
 #   2. tsys7 → tsys5 (bond0, 1 active slave) — PCI NIC comparison
 #   3. tsys7 → tsys6 (bond0, 2 active slaves) — working LACP baseline
 #   4. Reverse: tsys4 → tsys7 (USB NIC TX direction)
 #   5. Reverse: tsys5 → tsys7
 #
 # Each test: TCP 8-stream 30s forward + reverse + UDP saturation.
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 ALL_HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 # Storage network IPs
 declare -A SIP
 SIP[pfv-tsys1]="10.100.100.1"
 SIP[pfv-tsys3]="10.100.100.3"
 SIP[pfv-tsys4]="10.100.100.4"
 SIP[pfv-tsys5]="10.100.100.5"
 SIP[pfv-tsys6]="10.100.100.6"
 SIP[pfv-tsys7]="10.100.100.7"
 echo "==================================================================="
 echo " STEP 1: Install iperf3 on all online hosts"
 echo "==================================================================="
 for host in "${ALL_HOSTS[@]}"; do
    echo -n "  [$host] "
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "UNREACHABLE — skipping"
        continue
    fi
    # Check if iperf3 already installed
    if ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
        echo "iperf3 already installed"
    else
        printf "installing... "
        ssh "${SSH_OPTS[@]}" "root@$host" 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
    fi
 done
 echo ""
 echo "==================================================================="
 echo " STEP 2: Kill any existing iperf3 processes everywhere"
 echo "==================================================================="
 for host in "${ALL_HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo "  Done."
 # Helper: run an iperf3 test and save output
 run_iperf() {
    local client="$1" server="$2" direction="$3" label="$4" logfile="$5"
    local client_ip="${SIP[$client]}" server_ip="${SIP[$server]}"
    echo -n "  [$client → $server] $label ... "
    # Start server in one-shot mode (-1 means serve one client then exit)
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
    # Run client
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client ($client_ip)"
        echo "Server: $server ($server_ip)"
        echo "Direction: $direction"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo ""
        if [ "$direction" = "forward" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2" 2>&1
        elif [ "$direction" = "reverse" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2 -R" 2>&1
        elif [ "$direction" = "udp" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -u -b 2G -t 10 -l 8972" 2>&1
        elif [ "$direction" = "single" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -t 20 -O 2" 2>&1
        fi
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract summary line
    if grep -q "sender" "$logfile"; then
        bitrate=$(grep "sender" "$logfile" | tail -1 | awk '{print $7, $8}')
        echo "done: ${bitrate}"
    else
        echo "done (check log for details)"
    fi
 }
 echo ""
 echo "==================================================================="
 echo " STEP 3: Run iperf3 test matrix"
 echo "==================================================================="
 echo ""
 echo "All tests over VLAN1000 storage network (10.100.100.0/24)."
 echo "TCP tests: 8 parallel streams, 30s, 128k blocks."
 echo ""
 # --- Test 1: tsys7 → tsys4 (USB cdc_ncm target) ---
 echo "--- TEST 1: tsys7 → tsys4 (USB cdc_ncm NIC) ---"
 run_iperf pfv-tsys7 pfv-tsys4 forward "TCP 8-stream forward (tsys7→tsys4 USB)" \
    "$LOG_DIR/01-tsys7-to-tsys4-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys7)" \
    "$LOG_DIR/02-tsys7-to-tsys4-tcp-reverse.log"
 run_iperf pfv-tsys7 pfv-tsys4 single "TCP single-stream forward (tsys7→tsys4 USB)" \
    "$LOG_DIR/03-tsys7-to-tsys4-tcp-single.log"
 run_iperf pfv-tsys7 pfv-tsys4 udp "UDP saturation (tsys7→tsys4 USB)" \
    "$LOG_DIR/04-tsys7-to-tsys4-udp.log"
 echo ""
 # --- Test 2: tsys7 → tsys5 (bond0, PCI NIC, 1 active slave) ---
 echo "--- TEST 2: tsys7 → tsys5 (PCI NIC, broken bond - 1 slave) ---"
 run_iperf pfv-tsys7 pfv-tsys5 forward "TCP 8-stream forward (tsys7→tsys5 PCI)" \
    "$LOG_DIR/05-tsys7-to-tsys5-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys5 reverse "TCP 8-stream reverse (tsys5 PCI→tsys7)" \
    "$LOG_DIR/06-tsys7-to-tsys5-tcp-reverse.log"
 run_iperf pfv-tsys7 pfv-tsys5 single "TCP single-stream forward (tsys7→tsys5 PCI)" \
    "$LOG_DIR/07-tsys7-to-tsys5-tcp-single.log"
 run_iperf pfv-tsys7 pfv-tsys5 udp "UDP saturation (tsys7→tsys5 PCI)" \
    "$LOG_DIR/08-tsys7-to-tsys5-udp.log"
 echo ""
 # --- Test 3: tsys7 → tsys6 (working 2-slave LACP baseline, layer3+4) ---
 echo "--- TEST 3: tsys7 → tsys6 (working 2×1G LACP baseline) ---"
 run_iperf pfv-tsys7 pfv-tsys6 forward "TCP 8-stream forward (tsys7→tsys6 LACP)" \
    "$LOG_DIR/09-tsys7-to-tsys6-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys6 reverse "TCP 8-stream reverse (tsys6 LACP→tsys7)" \
    "$LOG_DIR/10-tsys7-to-tsys6-tcp-reverse.log"
 echo ""
 # --- Test 4: tsys6 → tsys4 (pre-tuning baseline) ---
 echo "--- TEST 4: tsys6 → tsys4 (baseline before tsys6 tuning) ---"
 run_iperf pfv-tsys6 pfv-tsys4 forward "TCP 8-stream forward (tsys6→tsys4 USB)" \
    "$LOG_DIR/11-tsys6-to-tsys4-tcp-forward.log"
 run_iperf pfv-tsys6 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys6)" \
    "$LOG_DIR/12-tsys6-to-tsys4-tcp-reverse.log"
 echo ""
 # --- Cleanup: kill iperf3 everywhere ---
 echo "--- Cleanup ---"
 for host in "${ALL_HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 echo "==================================================================="
 echo " RESULTS SUMMARY"
 echo "==================================================================="
 echo ""
 printf "%-45s %s\n" "TEST" "THROUGHPUT"
 printf "%-45s %s\n" "----" "----------"
 for f in "$LOG_DIR"/*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    bitrate=$(grep -E "sender$" "$f" | tail -1 | awk '{print $7, $8}')
    [ -z "$bitrate" ] && bitrate=$(grep -E "Mbits/sec|Gbits/sec" "$f" | tail -1 | grep -oE '[0-9.]+ [MG]bits/sec' | head -1)
    [ -z "$bitrate" ] && bitrate="(see log)"
    printf "%-45s %s\n" "$label" "$bitrate"
 done
 echo ""
 echo "Full logs saved to: $LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,119 @@
 #!/bin/bash
 # iperf-tsys6-tsys7.sh - validate 2Gbps LACP between the two tuned hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 TSYS6="10.100.100.6"
 TSYS7="10.100.100.7"
 run_test() {
    local client="$1" server="$2" server_ip="$3" label="$4" logfile="$5"
    shift 4
    local extra_args="$*"
    echo -n "  [$label] ... "
    # Start server in one-shot mode
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client  Server: $server ($server_ip)"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo "Args: $extra_args"
        echo ""
        ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${server_ip} $extra_args" 2>&1
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract result
    sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$logfile" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    echo "${bitrate} (retrans: ${retrans:-0})"
 }
 echo "==================================================================="
 echo " iperf3: tsys6 ↔ tsys7 (both have 2×1G LACP + layer3+4 hash)"
 echo " Expectation: ~1.8-2.0 Gbps for 8-stream TCP"
 echo "==================================================================="
 echo ""
 # Pre-flight: confirm bond state on both
 echo "--- bond0 state on tsys6 ---"
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
 echo ""
 echo "--- bond0 state on tsys7 ---"
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
 echo ""
 echo "--- Running tests ---"
 echo ""
 # Test 1: tsys7 → tsys6, 8-stream TCP forward
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP 8-stream forward" \
    "$LOG_DIR/tsys6-tsys7-01-tcp-8stream-forward.log" \
    "-P 8 -t 30 -l 128k -O 2"
 # Test 2: tsys6 → tsys7, 8-stream TCP forward (reverse direction)
 run_test pfv-tsys6 pfv-tsys7 "$TSYS7" \
    "tsys6→tsys7 TCP 8-stream forward" \
    "$LOG_DIR/tsys6-tsys7-02-tcp-8stream-forward.log" \
    "-P 8 -t 30 -l 128k -O 2"
 # Test 3: tsys7 → tsys6, single stream (should be ~940 Mbps — single flow)
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP single-stream" \
    "$LOG_DIR/tsys6-tsys7-03-tcp-single.log" \
    "-t 20 -O 2"
 # Test 4: tsys7 → tsys6, 4-stream (nconnect=4 mirrors this)
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP 4-stream" \
    "$LOG_DIR/tsys6-tsys7-04-tcp-4stream.log" \
    "-P 4 -t 30 -l 128k -O 2"
 # Test 5: UDP saturation
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 UDP saturation" \
    "$LOG_DIR/tsys6-tsys7-05-udp.log" \
    "-u -b 3G -t 10 -l 8972"
 # Cleanup
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 echo ""
 echo "==================================================================="
 echo " SUMMARY"
 echo "==================================================================="
 echo ""
 printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-45s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/tsys6-tsys7-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$f" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    printf "%-45s %15s %10s\n" "$label" "$bitrate" "${retrans:--}"
 done
 echo ""
 echo "Expected: 8-stream ~1.8-2.0 Gbps, single-stream ~940 Mbps"
@@ -0,0 +1,88 @@
 #!/bin/bash
 # reboot-and-verify.sh - reboots a host and verifies NFS nconnect activates.
 # Usage: bash reboot-and-verify.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 echo "================================================================"
 echo "[$HOST] PRE-REBOOT STATE"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" '
    echo "--- VMs ---"
    qm list 2>/dev/null
    echo "--- NFS TCP conns: $(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l) ---"
    echo "--- NFS first mount: ---"
    nfsstat -m 2>/dev/null | head -2 | tail -1
    echo "--- uptime ---"
    uptime
 '
 echo ""
 echo "================================================================"
 echo "[$HOST] ISSUING REBOOT"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &'
 echo "Reboot sent at $(date +%H:%M:%S)"
 echo ""
 echo "================================================================"
 echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 min)"
 echo "================================================================"
 DEADLINE=$(( $(date +%s) + 600 ))
 LAST_PRINT=0
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    now=$(date +%s)
    if [ $((now - LAST_PRINT)) -ge 15 ]; then
        printf '  [%s] waiting... (%ss elapsed)\n' "$(date +%H:%M:%S)" "$(( now - DEADLINE + 600 ))"
        LAST_PRINT=$now
    fi
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        # Verify uptime is actually low (host really rebooted, not still up)
        up_mins=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/uptime | awk "{print int(\$1/60)}"')
        if [ "${up_mins:-999}" -lt 5 ]; then
            echo "  [$(date +%H:%M:%S)] SSH back, uptime ${up_mins}min — real reboot confirmed"
            break
        fi
    fi
    sleep 10
 done
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "FAILED: $HOST not back after 10 minutes"
    exit 1
 fi
 echo "Waiting 30s for services to settle..."
 sleep 30
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" '
    echo "--- uptime ---"
    uptime
    echo ""
    echo "--- governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null) ---"
    echo "--- swappiness: $(sysctl -n vm.swappiness) ---"
    echo "--- tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control) ---"
    echo "--- tuned: $(tuned-adm active 2>/dev/null | grep Current) ---"
    echo ""
    echo "--- NFS first mount: ---"
    nfsstat -m 2>/dev/null | head -2 | tail -1
    echo ""
    echo "--- NFS TCP conns (expect 8 with nconnect=4): ---"
    ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l
    echo ""
    echo "--- VMs: ---"
    qm list 2>/dev/null
    echo ""
    echo "--- Failed services: ---"
    systemctl --failed --no-legend 2>/dev/null | head -5
    echo "(empty = none)"
 '
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
@@ -0,0 +1,91 @@
 #!/bin/bash
 # reboot-verify.sh - reboots a host, waits for it to come back, verifies state.
 # Usage: bash reboot-verify.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 echo "================================================================"
 echo "[$HOST] PRE-REBOOT STATE"
 echo "================================================================"
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR==1 || \$3==\"running\"{print}"' 2>&1
 echo "--- NFS mount count ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | grep -c "^/mnt"' 2>&1
 echo "--- NFS TCP connections to :2049 ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
 echo ""
 echo "================================================================"
 echo "[$HOST] ISSUING REBOOT"
 echo "================================================================"
 # Issue reboot; ssh will disconnect with non-zero — that's expected.
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &' 2>&1
 echo "Reboot command sent at $(date +%H:%M:%S). Host will drop now."
 echo ""
 echo "================================================================"
 echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 minutes)"
 echo "================================================================"
 DEADLINE=$(( $(date +%s) + 600 ))
 LAST_PRINT=0
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    now=$(date +%s)
    # Print a heartbeat every 15s
    if [ $((now - LAST_PRINT)) -ge 15 ]; then
        elapsed=$((DEADLINE - now - 600)); elapsed=${elapsed#-}
        echo "  [$(date +%H:%M:%S)] still waiting... (${elapsed}s elapsed)"
        LAST_PRINT=$now
    fi
    # Try SSH
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  [$(date +%H:%M:%S)] SSH is back!"
        break
    fi
    sleep 5
 done
 # Final check
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "  [$(date +%H:%M:%S)] FAILED: host not reachable after 10 minutes"
    exit 1
 fi
 # Give services a moment to settle after SSH returns
 echo "  Waiting 20s for services to settle..."
 sleep 20
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
 echo ""
 echo "--- TCP congestion control ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc' 2>&1
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness' 2>&1
 echo ""
 echo "--- scaling_governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"' 2>&1
 echo ""
 echo "--- NFS mount options (looking for nconnect + noatime) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -20' 2>&1
 echo ""
 echo "--- NFS TCP connection count (expect ~8 = 4 per server with nconnect=4) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
 echo ""
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null' 2>&1
 echo ""
 echo "--- Failed services? ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10' 2>&1
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
@@ -0,0 +1,141 @@
 #!/bin/bash
 ###############################################################################
 # apply-bond-hash.sh
 #
 # Adds bond-xmit-hash-policy layer3+4 to bond0 in /etc/network/interfaces,
 # then reloads networking with ifreload -a.
 #
 # SSH survivability: this is safe IF your SSH session is on vmbr0/nic0
 # (management network), NOT on bond0/datanet (storage network).
 # tsys7's topology confirms this: SSH comes in on vmbr0 (nic0).
 #
 # Safety:
 #   - Dry-run by default (--apply to commit)
 #   - Full backup of /etc/network/interfaces
 #   - Generates rollback script
 #   - Does NOT reboot — uses ifreload -a which is hot-reload
 ###############################################################################
 set -euo pipefail
 HOST="$(hostname -s)"
 TS_SHORT="$(date +%Y%m%d-%H%M%S)"
 BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
 ROLLBACK="/root/perfopt-bond-rollback-${TS_SHORT}.sh"
 ACTION="${1:-dryrun}"
 [ "$ACTION" = "--apply" ] && ACTION="apply" || ACTION="dryrun"
 mkdir -p "$BACKUP_DIR"
 echo "==================================================================="
 echo " apply-bond-hash — $HOST"
 echo " mode: $ACTION"
 echo "==================================================================="
 if [ ! -r /etc/network/interfaces ]; then
    echo "FATAL: /etc/network/interfaces not readable"
    exit 1
 fi
 # Check if bond0 exists in the config
 if ! grep -q 'bond0' /etc/network/interfaces; then
    echo "No bond0 found in /etc/network/interfaces — nothing to do."
    exit 0
 fi
 # Check if the hash policy is already set
 if grep -q 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces; then
    echo "bond-xmit-hash-policy already present:"
    grep 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces
    echo "Checking value..."
    if grep -q 'layer3+4' /etc/network/interfaces; then
        echo "Already set to layer3+4 — nothing to do."
        exit 0
    fi
 fi
 # Show current bond state
 echo ""
 echo "--- Current bond0 state ---"
 cat /proc/net/bonding/bond0 2>/dev/null | head -15 || echo "(bond0 not up or not present)"
 # Back up
 cp -a /etc/network/interfaces "$BACKUP_DIR/interfaces"
 # Generate rollback script
 cat > "$ROLLBACK" <<EOF
 #!/bin/bash
 # Rollback for bond-xmit-hash-policy change
 # Restores original /etc/network/interfaces and reloads
 set -euo pipefail
 cp -a "$BACKUP_DIR/interfaces" /etc/network/interfaces
 echo "Restored /etc/network/interfaces"
 echo "Reloading networking..."
 ifreload -a 2>&1 || systemctl restart networking 2>&1 || true
 echo "Done. bond0 hash policy reverted to original."
 EOF
 chmod +x "$ROLLBACK"
 echo ""
 echo "--- Proposed change ---"
 echo "Add line '	bond-xmit-hash-policy layer3+4' to the bond0 stanza."
 echo ""
 if [ "$ACTION" != "apply" ]; then
    echo "DRY RUN — no changes made."
    echo "To commit: bash $0 --apply"
    echo "Rollback script (pre-generated): $ROLLBACK"
    exit 0
 fi
 # Apply: use sed to insert bond-xmit-hash-policy after bond-mode line
 # The bond0 stanza looks like:
 #   auto bond0
 #   iface bond0 inet manual
 #       bond-slaves nic1 nic2
 #       bond-miimon 100
 #       bond-mode 802.3ad
 #
 # We insert after the bond-mode line.
 echo "Applying..."
 # Check if bond-mode line exists (various formats)
 if grep -qE '^\s*bond-mode\s+802.3ad' /etc/network/interfaces; then
    # Insert after bond-mode 802.3ad line
    sed -i '/^\s*bond-mode\s+802\.3ad/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode line."
 elif grep -qE '^\s*bond-mode\s+4' /etc/network/interfaces; then
    sed -i '/^\s*bond-mode\s+4/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode 4 line."
 else
    echo "Could not find bond-mode line — inserting after bond-slaves line instead."
    sed -i '/^\s*bond-slaves/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
 fi
 # Show the result
 echo ""
 echo "--- Updated bond0 stanza ---"
 awk '/^auto bond0/,/^$/' /etc/network/interfaces
 echo ""
 echo "--- Reloading networking (ifreload -a) ---"
 echo "SSH should survive (it's on vmbr0/nic0, not bond0)..."
 ifreload -a 2>&1 || {
    echo "ifreload failed, trying systemctl restart networking..."
    systemctl restart networking 2>&1
 }
 # Wait a moment for bond to renegotiate
 echo "Waiting 5s for LACP to renegotiate..."
 sleep 5
 echo ""
 echo "--- Post-change bond0 state ---"
 cat /proc/net/bonding/bond0 2>/dev/null | head -20
 echo ""
 echo "==================================================================="
 echo " DONE."
 echo " Backup: $BACKUP_DIR/interfaces"
 echo " Rollback: bash $ROLLBACK"
 echo "==================================================================="
@@ -0,0 +1,439 @@
 #!/bin/bash
 ###############################################################################
 # apply-tunings.sh
 #
 # Applies the Tier 0 host-side tunings identified in FINDINGS.md:
 #   1. Sets scaling_governor=performance on every CPU
 #   2. Sets vm.swappiness appropriately (1 on storage hosts, 10 on VM hosts)
 #   3. Enables tcp_bbr module + sets congestion control
 #   4. Sets net.core.rmem_max/wmem_max + tcp_rmem/wmem for high-BDP NFS
 #   5. Sets the recommended tuned-adm profile (network-throughput or virtual-host)
 #   6. Adds nconnect=4 + noatime to NFS client mounts (edits /etc/pve/storage.cfg)
 #   7. (does NOT touch bond0 xmit-hash — that requires ifreload/network
 #       restart which drops the host. Provided as a separate --emit-bond-patch
 #       flag that PRINTS the change but does not apply it.)
 #
 # Safety features:
 #   - Dry-run mode by default (--apply to commit)
 #   - Full backup of every modified file to /root/perfopt-backup-<timestamp>/
 #   - Per-host behaviour: detects storage hosts by hostname and applies the
 #     right profile (tsys4, tsys5 = storage; others = VM hosts).
 #   - Generates a /root/perfopt-rollback.sh that undoes everything.
 #   - Does NOT reboot, does NOT restart networking, does NOT touch hardware.
 #
 # Usage:
 #   bash apply-tunings.sh             # dry run, show what would change
 #   bash apply-tunings.sh --apply     # commit changes
 #   bash apply-tunings.sh --rollback  # restore from latest backup
 #   bash apply-tunings.sh --emit-bond-patch   # show bond0 hash change (no apply)
 ###############################################################################
 set -u
 umask 022
 HOST="$(hostname -s)"
 TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 TS_SHORT="$(date +%Y%m%d-%H%M%S)"
 BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
 ROLLBACK_SCRIPT="/root/perfopt-rollback-${TS_SHORT}.sh"
 ACTION="dryrun"
 SKIP_NFS=0
 while [ $# -gt 0 ]; do
    case "$1" in
        --apply)         ACTION="apply" ;;
        --rollback)      ACTION="rollback" ;;
        --emit-bond-patch) ACTION="bond-patch" ;;
        --no-nfs)        SKIP_NFS=1 ;;
        -h|--help)       sed -n '2,35p' "$0"; exit 0 ;;
        *) echo "unknown arg: $1" >&2; exit 2 ;;
    esac
    shift
 done
 # Detect host role from hostname
 case "$HOST" in
    pfv-tsys4|pfv-tsys5)
        HOST_ROLE="storage"
        TUNED_PROFILE="network-throughput"
        SWAPPINESS="1"
        ;;
    pfv-tsys1|pfv-tsys3|pfv-tsys6|pfv-tsys7)
        HOST_ROLE="vmhost"
        TUNED_PROFILE="virtual-host"
        SWAPPINESS="10"
        ;;
    *)
        HOST_ROLE="unknown"
        TUNED_PROFILE="virtual-host"
        SWAPPINESS="10"
        ;;
 esac
 # --- rollback path ---------------------------------------------------------
 if [ "$ACTION" = "rollback" ]; then
    echo "Looking for latest backup under /root/perfopt-backup-* ..."
    latest=""
    while IFS= read -r d; do
        latest="$d"
    done < <(find /root -maxdepth 1 -type d -name 'perfopt-backup-*' 2>/dev/null | sort | tail -n1)
    if [ -z "$latest" ]; then
        echo "No backup found under /root/perfopt-backup-*" >&2
        exit 1
    fi
    rb="$latest/perfopt-rollback.sh"
    if [ ! -x "$rb" ] && [ ! -r "$rb" ]; then
        echo "Rollback script missing in $latest" >&2
        exit 1
    fi
    echo "Rolling back using: $rb"
    bash "$rb"
    exit $?
 fi
 # --- bond patch print-only path -------------------------------------------
 if [ "$ACTION" = "bond-patch" ]; then
    echo "==================================================================="
    echo " Proposed bond0 xmit_hash_policy change"
    echo "==================================================================="
    echo
    if [ ! -r /etc/network/interfaces ]; then
        echo "/etc/network/interfaces not readable"
        exit 1
    fi
    if ! grep -q 'bond-mode 802.3ad\|bond-mode 4\|bond-slaves' /etc/network/interfaces; then
        echo "No bond0 detected on this host — nothing to patch."
        exit 0
    fi
    cat <<EOF
 A manual edit to /etc/network/interfaces is required. The bond0 stanza
 needs this line added (it defaults to layer2, which is wrong for storage):
    bond-xmit-hash-policy layer3+4
 After editing, you MUST reload networking for it to take effect:
    ifreload -a            # safe, brings interfaces down/up
    # OR
    systemctl restart networking   # heavier, briefly drops connections
 WARNING: applying this on a remote host over bond0 will briefly drop your
 SSH session. Run from console/IPMI, or schedule a maintenance window.
 The change is reversible by removing the line and reloading again.
 EOF
    exit 0
 fi
 # --- main path (dryrun or apply) ------------------------------------------
 mkdir -p "$BACKUP_DIR"
 # Emit the rollback script header
 cat > "$ROLLBACK_SCRIPT" <<EOF
 #!/bin/bash
 # Auto-generated rollback for perfopt apply-tunings.sh
 # Backup timestamp: $TS
 # Backup dir:       $BACKUP_DIR
 # Generated on:     $HOST
 set -u
 EOF
 chmod +x "$ROLLBACK_SCRIPT"
 backup_file() {
    local f="$1"
    if [ -e "$f" ]; then
        cp -a "$f" "$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
    fi
 }
 # Helper: append a restore command to the rollback script
 rb_restore() {
    local f="$1"
    local bk
    bk="$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 if [ -r "$bk" ]; then
    cp -a "$bk" "$f" && echo "restored $f"
 else
    echo "WARN: backup $bk missing — $f left as-is"
 fi
 EOF
 }
 # Helper: append a sysctl restore command
 rb_sysctl_restore() {
    local key="$1" orig_val="$2"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 sysctl -w "$key=$orig_val" >/dev/null && echo "restored $key=$orig_val"
 EOF
 }
 # Helper: append a tuned-adm restore
 rb_tuned_restore() {
    local profile="$1"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 tuned-adm profile "$profile" 2>/dev/null && echo "restored tuned profile: $profile"
 EOF
 }
 echo "==================================================================="
 echo " perfopt apply-tunings — $HOST"
 echo " mode: $ACTION"
 echo " role: $HOST_ROLE (tuned=$TUNED_PROFILE, swappiness=$SWAPPINESS)"
 echo " backup: $BACKUP_DIR"
 echo " rollback: $ROLLBACK_SCRIPT"
 echo "==================================================================="
 echo
 # ===========================================================================
 # 1. scaling_governor → performance
 # ===========================================================================
 echo "--- 1. CPU scaling_governor → performance"
 if [ -r /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ]; then
    cur=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor)
    echo "    current: $cur"
    if [ "$ACTION" = "apply" ]; then
        if [ "$cur" != "performance" ]; then
            for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
                echo performance > "$c" 2>/dev/null || true
            done
            # Persist via systemd tmpfiles / sysctl fallback
            cat > /etc/systemd/system/perfopt-cpu-performance.service <<EOF
 [Unit]
 Description=Set CPU scaling_governor=performance (perfopt)
 After=multi-user.target
 [Service]
 Type=oneshot
 ExecStart=/bin/sh -c 'for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "\$c" 2>/dev/null || true; done'
 RemainAfterExit=yes
 [Install]
 WantedBy=multi-user.target
 EOF
            systemctl daemon-reload
            systemctl enable perfopt-cpu-performance.service 2>/dev/null
            systemctl start perfopt-cpu-performance.service 2>/dev/null
            echo "    applied + persisted via systemd unit"
            # Rollback
            cat >> "$ROLLBACK_SCRIPT" <<EOF
 for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "$cur" > "\$c" 2>/dev/null || true; done
 systemctl disable --now perfopt-cpu-performance.service 2>/dev/null || true
 rm -f /etc/systemd/system/perfopt-cpu-performance.service
 systemctl daemon-reload
 echo "restored scaling_governor=$cur (best-effort; original may have been dynamic)"
 EOF
        else
            echo "    already performance — no change"
        fi
    fi
 else
    echo "    cpufreq driver not loaded — nothing to do (typical on BIOS-locked servers)"
 fi
 # ===========================================================================
 # 2. vm.swappiness
 # ===========================================================================
 echo "--- 2. vm.swappiness → $SWAPPINESS (role=$HOST_ROLE)"
 cur_swappiness=$(sysctl -n vm.swappiness 2>/dev/null || echo "?")
 echo "    current: $cur_swappiness"
 if [ "$ACTION" = "apply" ] && [ "$cur_swappiness" != "$SWAPPINESS" ]; then
    # Persist via sysctl.d
    backup_file /etc/sysctl.d/99-perfopt.conf
    cat > /etc/sysctl.d/99-perfopt.conf <<EOF
 # perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
 vm.swappiness = $SWAPPINESS
 EOF
    sysctl -w "vm.swappiness=$SWAPPINESS" >/dev/null
    echo "    applied (persisted to /etc/sysctl.d/99-perfopt.conf)"
    rb_restore /etc/sysctl.d/99-perfopt.conf
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 rm -f /etc/sysctl.d/99-perfopt.conf
 sysctl -w "vm.swappiness=$cur_swappiness" >/dev/null
 echo "restored vm.swappiness=$cur_swappiness"
 EOF
 else
    echo "    no change"
 fi
 # ===========================================================================
 # 3. TCP BBR + buffers  (single sysctl.d file)
 # ===========================================================================
 echo "--- 3. TCP BBR + socket buffers"
 cur_cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null || echo "?")
 cur_avail=$(sysctl -n net.ipv4.tcp_available_congestion_control 2>/dev/null || echo "?")
 echo "    current: $cur_cc (available: $cur_avail)"
 # Decide sysctl values (these are the consensus values for a 1-10 GbE NFS host)
 TARGET_RMEM_MAX=134217728        # 128 MB
 TARGET_WMEM_MAX=134217728
 TARGET_RMEM_DEFAULT=26214400     # 25 MB
 TARGET_WMEM_DEFAULT=26214400
 TARGET_TCP_RMEM="4096 87380 $TARGET_RMEM_MAX"
 TARGET_TCP_WMEM="4096 65536 $TARGET_RMEM_MAX"
 TARGET_NETDEV_MAX_BACKLOG=250000
 TARGET_SOMAXCONN=65535
 if [ "$ACTION" = "apply" ]; then
    # Ensure tcp_bbr module loads at boot
    if ! grep -q '^tcp_bbr' /etc/modules-load.d/modules.conf 2>/dev/null; then
        backup_file /etc/modules-load.d/modules.conf
        mkdir -p /etc/modules-load.d
        cat >> /etc/modules-load.d/modules.conf <<EOF
 # perfopt apply-tunings.sh — $TS
 tcp_bbr
 EOF
        modprobe tcp_bbr 2>/dev/null || true
        rb_restore /etc/modules-load.d/modules.conf
    fi
    backup_file /etc/sysctl.d/99-perfopt.conf
    cat > /etc/sysctl.d/99-perfopt.conf <<EOF
 # perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
 vm.swappiness = $SWAPPINESS
 # TCP BBR + high-BDP buffers (good for NFS over 1-10 GbE)
 net.ipv4.tcp_congestion_control = bbr
 net.core.default_qdisc = fq
 net.core.rmem_max = $TARGET_RMEM_MAX
 net.core.wmem_max = $TARGET_WMEM_MAX
 net.core.rmem_default = $TARGET_RMEM_DEFAULT
 net.core.wmem_default = $TARGET_WMEM_DEFAULT
 net.core.netdev_max_backlog = $TARGET_NETDEV_MAX_BACKLOG
 net.core.somaxconn = $TARGET_SOMAXCONN
 net.ipv4.tcp_rmem = $TARGET_TCP_RMEM
 net.ipv4.tcp_wmem = $TARGET_TCP_WMEM
 EOF
    sysctl --system >/dev/null 2>&1
    echo "    applied (written to /etc/sysctl.d/99-perfopt.conf + sysctl --system)"
    # Rebuild rollback for sysctl
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 # Restore original sysctl state
 rm -f /etc/sysctl.d/99-perfopt.conf
 EOF
    rb_sysctl_restore "net.ipv4.tcp_congestion_control" "$cur_cc"
    rb_sysctl_restore "vm.swappiness" "$cur_swappiness"
    # Note: original buffer sizes not all captured; restoration is best-effort.
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 echo "Note: net buffers restored to kernel defaults (original values not captured)."
 echo "Run 'sysctl --system' to apply."
 sysctl --system >/dev/null 2>&1 || true
 EOF
 fi
 # ===========================================================================
 # 4. tuned-adm profile
 # ===========================================================================
 echo "--- 4. tuned-adm profile → $TUNED_PROFILE"
 if command -v tuned-adm >/dev/null 2>&1; then
    cur_profile=$(tuned-adm active 2>/dev/null | awk -F: '/Current active profile/{gsub(/^[ \t]+/,"",$2); print $2}')
    echo "    current: $cur_profile"
    if [ "$ACTION" = "apply" ] && [ "$cur_profile" != "$TUNED_PROFILE" ]; then
        tuned-adm profile "$TUNED_PROFILE" 2>&1 | sed 's/^/    /'
        rb_tuned_restore "$cur_profile"
    else
        echo "    no change"
    fi
 else
    echo "    tuned-adm not installed — skipping"
 fi
 # ===========================================================================
 # 5. NFS mount option hardening
 #    Edit /etc/pve/storage.cfg to add nconnect=4 + noatime to NFS plugins.
 #    Proxmox applies these on next mount/remount.
 #    Skipped with --no-nfs (for NFS servers where we don't want to remount).
 # ===========================================================================
 echo "--- 5. NFS mount options: add nconnect=4, noatime"
 if [ "$SKIP_NFS" -eq 1 ]; then
    echo "    SKIPPED (--no-nfs specified)"
    echo "    Note: storage.cfg is cluster-wide; if edited on another host,"
    echo "    the options are already staged here and activate on next reboot."
 else
 if [ -r /etc/pve/storage.cfg ]; then
    backup_file /etc/pve/storage.cfg
    if grep -q '^nfs:' /etc/pve/storage.cfg; then
        # Count how many nfs: stanzas lack the options
        while IFS= read -r line; do
            if echo "$line" | grep -q '^nfs:'; then
                cur_stanza=$(echo "$line" | awk '{print $2}')
                # Look ahead for the next few lines to see if options already set
                echo "    nfs stanza: $cur_stanza"
            fi
        done < /etc/pve/storage.cfg
        # Check whether any nfs stanza already has options
        if grep -A1 '^nfs:' /etc/pve/storage.cfg | grep -q 'options.*nconnect=4'; then
            echo "    some stanzas already have nconnect=4 — check manually"
        else
            echo "    proposed change: add 'options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2' to each nfs stanza"
            if [ "$ACTION" = "apply" ]; then
                # Apply via perl for safety (in-place edit with backup)
                if cp -a /etc/pve/storage.cfg "$BACKUP_DIR/etc__pve__storage.cfg"; then
                    # Use a python helper for robust PVE storage.cfg editing
                    python3 - <<PYEOF
 import re, pathlib
 p = pathlib.Path("/etc/pve/storage.cfg")
 text = p.read_text()
 pattern = re.compile(r'(^nfs:\s*\S+\n(?:[ \t]+[^\n]+\n)+)', re.MULTILINE)
 modified = 0
 def fix(m):
    global modified
    block = m.group(1)
    if 'options' in block:
        return block
    lines = block.rstrip('\n').split('\n')
    lines.insert(1, '\toptions nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2')
    modified += 1
    return '\n'.join(lines) + '\n'
 new_text = pattern.sub(fix, text)
 if modified:
    p.write_text(new_text)
    print(f"    patched {modified} NFS stanzas with options line")
 else:
    print("    no NFS stanza needed patching (all already had options or no nfs: found)")
 PYEOF
                    # Remount existing NFS mounts to pick up new options
                    echo "    remounting NFS mounts to apply new options..."
                    while IFS= read -r mp; do
                        mount -o remount "$mp" 2>/dev/null && echo "      remounted $mp" || echo "      FAILED to remount $mp (will pick up on next mount)"
                    done < <(awk '$3 ~ /^nfs/{print $2}' /proc/mounts 2>/dev/null | sort -u)
                    rb_restore /etc/pve/storage.cfg
                    cat >> "$ROLLBACK_SCRIPT" <<EOF
 # To fully undo NFS options, also remount:
 while IFS= read -r mp; do
    mount -o remount "\$mp" 2>/dev/null
 done < <(awk '\$3 ~ /^nfs/{print \$2}' /proc/mounts 2>/dev/null | sort -u)
 echo "restored NFS mount options"
 EOF
                else
                    echo "    ERROR: could not back up storage.cfg — aborting NFS patch"
                fi
            fi
        fi
    else
        echo "    no NFS stanzas in storage.cfg — skipping"
    fi
 else
    echo "    /etc/pve/storage.cfg not readable — skipping"
 fi
 fi
 # ===========================================================================
 # 6. Summary
 # ===========================================================================
 echo
 echo "==================================================================="
 if [ "$ACTION" = "apply" ]; then
    echo " DONE. Backups in: $BACKUP_DIR"
    echo " Rollback: bash $ROLLBACK_SCRIPT"
    echo
    echo " Next steps:"
    echo "   - Verify with 'sysctl net.ipv4.tcp_congestion_control vm.swappiness'"
    echo "   - Verify NFS mounts with 'nfsstat -m' (look for nconnect=4)"
    echo "   - Run 'bash apply-tunings.sh --emit-bond-patch' for the bond0"
    echo "     xmit_hash_policy change (requires network restart, do in"
    echo "     maintenance window)"
 else
    echo " DRY RUN — no changes made."
    echo " To commit: bash apply-tunings.sh --apply"
 fi
 echo "==================================================================="
@@ -0,0 +1,86 @@
 #!/bin/bash
 # fix-tsys7.sh - fixes the two issues found: NFS options string + bond hash
 # Runs on the target host directly.
 set -euo pipefail
 echo "==================================================================="
 echo " FIX: NFS options + bond hash policy"
 echo "==================================================================="
 # --- FIX 1: Simplify NFS options in storage.cfg ---------------------------
 echo ""
 echo "--- FIX 1: Simplify NFS options (remove version=4.2 conflict) ---"
 # Replace the overly-complex options line with a minimal one
 # PVE handles vers/rsize/wsize/hard/etc internally; we only need nconnect + noatime
 if grep -q 'options.*nconnect=4.*version=4.2' /etc/pve/storage.cfg; then
    # Use sed to replace each options line
    sed -i 's/options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2/options nconnect=4,noatime/' /etc/pve/storage.cfg
    echo "Fixed: simplified NFS options to 'nconnect=4,noatime'"
 else
    echo "Options line already simplified or not present"
 fi
 echo ""
 echo "--- Verify storage.cfg NFS stanzas ---"
 grep -A 2 "^nfs: D2" /etc/pve/storage.cfg | head -3
 echo "..."
 grep -c "options nconnect" /etc/pve/storage.cfg
 echo " NFS stanzas with options"
 # --- FIX 2: Bond hash policy (force via sysfs + persist in interfaces) ----
 echo ""
 echo "--- FIX 2: Apply bond xmit_hash_policy=layer3+4 ---"
 # 2a: Apply LIVE via sysfs (takes effect immediately, no network reload)
 echo "Applying live via sysfs..."
 if echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; then
    echo "Live sysfs apply: SUCCESS"
 else
    echo "Live sysfs apply: FAILED (will persist in config and apply on ifreload)"
 fi
 # Verify live state
 echo ""
 echo "Live bond0 hash policy:"
 cat /proc/net/bonding/bond0 | grep "Transmit Hash"
 # 2b: Persist in /etc/network/interfaces (fix the sed that failed before)
 echo ""
 echo "Persisting in /etc/network/interfaces..."
 # Backup
 cp -a /etc/network/interfaces "/root/interfaces.bondfix.$(date +%Y%m%d%H%M%S)"
 # Check if already present
 if grep -q 'bond-xmit-hash-policy' /etc/network/interfaces; then
    echo "bond-xmit-hash-policy already in interfaces file"
 else
    # Use awk to insert after the bond-mode line (more reliable than sed)
    # Match any line containing 'bond-mode' (regardless of indentation)
    awk '
        /bond-mode/ && !done {
            print
            print "\tbond-xmit-hash-policy layer3+4"
            done=1
            next
        }
        { print }
    ' /etc/network/interfaces > /etc/network/interfaces.new
    mv /etc/network/interfaces.new /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4"
 fi
 echo ""
 echo "--- Updated bond0 stanza ---"
 awk '/^auto bond0/,/^$/' /etc/network/interfaces
 # Final verify
 echo ""
 echo "--- Final bond0 running state ---"
 cat /proc/net/bonding/bond0 | head -20
 echo ""
 echo "==================================================================="
 echo " FIX COMPLETE"
 echo "==================================================================="
@@ -0,0 +1,217 @@
 #!/usr/bin/env bash
 ###############################################################################
 # lacp-retrans-cause.sh  (HOST-NATIVE)
 #
 # Runs on tsys6 (receiver) or tsys7 (sender). Auto-detects role.
 #
 # Question this answers: are the ~56K retransmits we see on 8-stream iperf
 # between tsys6 and tsys7 fixable (NIC ring drops, softnet drops, CPU
 # saturation) or just unavoidable LACP reordering overhead?
 #
 # Method: snapshot drop/error counters + softnet_stat + TCP SNMP before and
 # after a 12s iperf3 run, then print the deltas. The decisive columns are:
 #   - NIC rx_dropped / rx_missed_errors / rx_no_dma_resources   -> ring too small
 #   - /proc/net/softnet_stat drops                              -> ksoftirq starved
 #   - /proc/net/snmp TCP retranst                               -> TCP-level retrans
 # If NIC drop counters do NOT climb but TCP retrans does, the retrans are
 # coming from LACP reordering (out-of-order segments triggering fast
 # retransmit), not packet loss — and are NOT fixable by tuning.
 ###############################################################################
 set -uo pipefail
 DURATION="${LACP_TEST_SECS:-12}"
 STREAMS="${LACP_STREAMS:-8}"
 PEER_RX_IP="10.100.100.6"   # tsys6 storage IP (receiver)
 PEER_TX_IP="10.100.100.7"   # tsys7 storage IP (sender)
 LOG_DIR="/root"
 BOND="bond0"
 HOST="$(uname -n)"; HOST="${HOST%%.*}"
 case "$HOST" in
    *tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-retrans-receiver.log" ;;
    *tsys7) ROLE="sender";   LOG="$LOG_DIR/lacp-retrans-sender.log" ;;
    *) echo "ERROR: not on tsys6 or tsys7 (host=$HOST)"; exit 2 ;;
 esac
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 # ---- per-slave NIC counter snapshot -----------------------------------------
 # Captures rx_dropped, rx_missed_errors, rx_no_dma_resources, tx_retrans,
 # plus all error-like counters. Falls back gracefully if a counter doesn't
 # exist (different NIC drivers expose different names).
 nic_snapshot() {
    for s in $(awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND"); do
        [ -n "$s" ] || continue
        echo "[$s]"
        # ethtool -S may exist; show only drop/error/retrans lines
        if command -v ethtool >/dev/null 2>&1; then
            ethtool -S "$s" 2>/dev/null \
                | grep -iE 'drop|miss|error|no_dma|fifo|retrans|overflow' \
                || echo "(ethtool -S: no matching counters or unsupported)"
        else
            echo "(ethtool not installed)"
        fi
        # /sys counters (always available, driver-agnostic)
        for c in rx_dropped tx_dropped rx_errors tx_errors rx_missed_errors \
                 rx_length_errors rx_crc_errors rx_fifo_errors tx_fifo_errors \
                 multicast collisions; do
            v=$(cat "/sys/class/net/$s/statistics/$c" 2>/dev/null)
            [ -n "$v" ] && printf '  sysfs %-22s = %s\n' "$c" "$v"
        done
    done
 }
 # ---- softnet_stat snapshot (per-CPU RX softirq drops) -----------------------
 # /proc/net/softnet_stat columns are HEX. Col1=processed, col2=dropped,
 # col3=time_squeeze. We only care about dropped + squeeze (small numbers that
 # look identical in hex and decimal). Processed is informational only and we
 # don't try to delta it (hex arithmetic is awk-version-dependent).
 softnet_snapshot() {
    awk '{ printf "cpu%s processed=%s dropped=%s squeezed=%s\n", \
              NR-1, $1, $2, $3 }' /proc/net/softnet_stat
 }
 # ---- TCP SNMP snapshot (the source of iperf's retransmit number) -----------
 tcp_snapshot() {
    awk '/^Tcp:/{
        if (!seen) {
            seen=1
            # /proc/net/snmp Tcp line 1 = names, line 2 = values
            n=split($0, names, " ")
            # re-find the values line
            getline
            vals=$0
            split(vals, v, " ")
            for (i=1;i<=n;i++) printf "  %-22s = %s\n", names[i], v[i]
        }
    }' /proc/net/snmp
 }
 # ---- print deltas for selected counters ------------------------------------
 # $1 = before file, $2 = after file
 # softnet_snapshot output: "cpuN processed=HEX dropped=HEX squeezed=HEX"
 # /proc/net/softnet_stat is hex, but dropped/squeezed are always small
 # integers (typically 0 on a healthy host), so +0 coercion is correct for
 # the values we care about. We deliberately DON'T report `processed` deltas
 # because hex arithmetic on large numbers is awk-version-dependent.
 softnet_delta() {
    awk '
        FNR==NR { if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
                  split($0, p, /[= ]+/); drop[id]=p[5]+0; sqz[id]=p[7]+0 }
                  next }
        { if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
              split($0, q, /[= ]+/);
              dd = (q[5]+0) - drop[id]
              sd = (q[7]+0) - sqz[id]
              if (dd != 0 || sd != 0)
                  printf "  %-8s dropped_delta=%-6s squeezed_delta=%-6s\n", id, dd, sd } }
    ' "$1" "$2" | sort
 }
 # $1 = before file, $2 = after file, $3 = section label prefix
 # File format: slave header line "[nic1]", then "  counter_name = value" lines.
 # Output only counters whose value changed.
 nic_delta() {
    awk '
        FNR==NR { if ($0 ~ /^\[/) slave=$0
                  else if ($0 ~ /=/) {
                      # Everything before " = " is the counter name (trim spaces)
                      pos = index($0, "=")
                      name = substr($0, 1, pos-1)
                      gsub(/^ +| +$/, "", name)
                      val  = substr($0, pos+1); gsub(/^ +| +$/, "", val)
                      before[slave SUBSEP name] = val
                  }
                  next }
        { if ($0 ~ /^\[/) slave=$0
          else if ($0 ~ /=/) {
              pos = index($0, "=")
              name = substr($0, 1, pos-1); gsub(/^ +| +$/, "", name)
              val  = substr($0, pos+1); gsub(/^ +| +$/, "", val)
              b = before[slave SUBSEP name]+0
              d = val+0 - b
              if (d != 0) printf "  %-8s %-30s %12s -> %12s  delta=%d\n", slave, name, b, val, d
          } }
    ' "$1" "$2"
 }
 {
 echo "=== LACP retransmit cause investigation ($ROLE) ==="
 echo "Host: $HOST   Role: $ROLE"
 echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
 echo "Bond: $BOND"
 echo ""
 echo "--- bond0 hash + driver ---"
 grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac" /proc/net/bonding/$BOND
 for s in $(awk '/^Slave Interface:/{print $3}' /proc/net/bonding/$BOND); do
    drv=$(ethtool -i "$s" 2>/dev/null | awk -F: '/^driver:/{print $2}' | sed 's/^ *//')
    speed=$(cat /sys/class/net/$s/speed 2>/dev/null)
    ring=$(ethtool -g "$s" 2>/dev/null | awk '/RX:/{print $2; exit}')
    printf "  %-8s driver=%-20s speed=%-6s current RX ring=%s\n" "$s" "$drv" "$speed" "$ring"
 done
 echo ""
 } | tee "$LOG"
 echo "--- BEFORE snapshots ---"
 nic_snapshot      > "$TMP/nic_before"
 softnet_snapshot  > "$TMP/softnet_before"
 tcp_snapshot      > "$TMP/tcp_before"
 if [ "$ROLE" = "receiver" ]; then
    echo "[receiver] starting iperf3 -s -1 on $PEER_RX_IP ..."
    pkill -x iperf3 2>/dev/null; sleep 0.5
    nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
    SRV_PID=$!
    for _ in $(seq 1 60); do kill -0 "$SRV_PID" 2>/dev/null || break; sleep 1; done
    wait "$SRV_PID" 2>/dev/null
 else
    echo "[sender] waiting 3s for receiver to listen, then running iperf3 -P $STREAMS -t $DURATION ..."
    sleep 3
    iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
        > "$TMP/iperf.out" 2>&1
 fi
 echo "--- AFTER snapshots ---"
 nic_snapshot      > "$TMP/nic_after"
 softnet_snapshot  > "$TMP/softnet_after"
 tcp_snapshot      > "$TMP/tcp_after"
 {
 echo ""
 echo "--- iperf3 summary ---"
 grep -E '\[SUM\].*(sender|receiver)' "$TMP/iperf.out" | tail -2
 echo ""
 echo "--- TCP SNMP deltas (/proc/net/snmp) ---"
 # Show only the counters that changed
 paste "$TMP/tcp_before" "$TMP/tcp_after" \
    | awk '{
        a=$3; b=$(NF)
        if (a+0 != b+0) printf "  %-22s %12s -> %12s  delta=%d\n", $1, a, b, b-a
    }'
 echo ""
 echo "--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---"
 softnet_delta "$TMP/softnet_before" "$TMP/softnet_after"
 echo ""
 echo "--- NIC counter deltas (only non-zero) ---"
 nic_delta "$TMP/nic_before" "$TMP/nic_after" "$HOST"
 echo ""
 echo "--- raw iperf3 tail (last 12 lines) ---"
 tail -12 "$TMP/iperf.out"
 echo ""
 echo "==================================================================="
 echo " HOW TO READ THIS"
 echo "==================================================================="
 echo " - If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:"
 echo "     -> ring buffers too small. Fix: ethtool -G \$IF rx 4096 (or max)."
 echo " - If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:"
 echo "     -> softirq starvation. Fix: increase net.core.netdev_budget,"
 echo "        check IRQ affinity, consider RPS."
 echo " - If NEITHER of the above climbed but TCP RetransSegs did:"
 echo "     -> the retransmits are LACP reordering (out-of-order segments"
 echo "        triggering fast retransmit), NOT packet loss. NOT fixable."
 echo "==================================================================="
 } | tee -a "$LOG"
 echo ""
 echo "Log: $LOG"
@@ -0,0 +1,153 @@
 #!/usr/bin/env bash
 ###############################################################################
 # lacp-rx-distribution.sh  (HOST-NATIVE version)
 #
 # Runs ON a host (tsys6 or tsys7). Auto-detects role by hostname:
 #   - On tsys6 (receiver): starts iperf3 -s -1 (one-shot), snapshots local
 #     bond0 slave RX counters before+after, writes verdict + log.
 #   - On tsys7 (sender):   snapshots local bond0 slave TX counters before+
 #     after, runs iperf3 -c <peer> -P 8 -t 12.
 #
 # Each side writes only its own counters. No inter-host SSH required.
 # iperf3 -s -1 (one-shot) handles client/server coordination.
 #
 # Output: /root/lacp-rx-{receiver,sender}.log
 ###############################################################################
 set -uo pipefail
 DURATION="${LACP_TEST_SECS:-12}"
 STREAMS="${LACP_STREAMS:-8}"
 PEER_RX_IP="10.100.100.6"   # tsys6 storage IP (receiver)
 PEER_TX_IP="10.100.100.7"   # tsys7 storage IP (sender)
 LOG_DIR="/root"
 BOND="bond0"
 # Snapshot per-slave byte counters from $BOND. Output: "<slave> rx tx\n" sorted.
 snapshot() {
    awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND" | while read -r s; do
        [ -n "$s" ] || continue
        rx=$(cat "/sys/class/net/$s/statistics/rx_bytes" 2>/dev/null || echo 0)
        tx=$(cat "/sys/class/net/$s/statistics/tx_bytes" 2>/dev/null || echo 0)
        printf '%s %s %s\n' "$s" "$rx" "$tx"
    done | sort
 }
 # Compute per-slave deltas. $1=label, $2=col(2=rx,3=tx), $3=before, $4=after.
 analyze() {
    local label="$1" col="$2" before="$3" after="$4"
    join "$before" "$after" | awk -v col="$col" -v lbl="$label" '
        {
            slave=$1
            b = (col==2 ? $2 : $3)+0
            a = (col==2 ? $4 : $5)+0
            d = a - b; if (d < 0) d = 0
            delta[slave]=d; bef[slave]=b; aft[slave]=a
            order[++n]=slave; total+=d
        }
        END {
            printf "\n--- %s (bytes) ---\n", lbl
            printf "  %-12s %14s %14s %14s %8s\n", "SLAVE", "BEFORE", "AFTER", "DELTA", "PCT"
            for (i=1;i<=n;i++){
                s=order[i]
                pct = (total>0 ? 100*delta[s]/total : 0)
                printf "  %-12s %14d %14d %14d %7.1f%%\n", s, bef[s], aft[s], delta[s], pct
            }
            printf "  %-12s %14s %14s %14d %8s\n", "TOTAL", "", "", total, "100.0%"
        }'
 }
 # Dominant-slave pct for the verdict line. $1=col, $2=before, $3=after.
 dominant() {
    join "$2" "$3" | awk -v col="$1" '
        { b=(col==2?$2:$3)+0; a=(col==2?$4:$5)+0; d=a-b; if(d<0)d=0; tot+=d; delta[$1]=d }
        END { m=0; ms=""; for (s in delta){ if (delta[s]>m){m=delta[s]; ms=s} }
              printf "%.1f %s", (tot>0?100*m/tot:0), ms }'
 }
 HOST="$(uname -n)"
 HOST="${HOST%%.*}"
 case "$HOST" in
    *tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-rx-receiver.log" ;;
    *tsys7) ROLE="sender";   LOG="$LOG_DIR/lacp-rx-sender.log" ;;
    *) echo "ERROR: not running on tsys6 or tsys7 (hostname=$HOST)"; exit 2 ;;
 esac
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 {
 echo "=== LACP per-slave distribution ($ROLE) ==="
 echo "Host: $HOST   Role: $ROLE"
 echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
 echo "Bond: $BOND"
 echo ""
 echo "--- $BOND state ---"
 grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac|Slave Interface|Link" \
    "/proc/net/bonding/$BOND" 2>&1
 echo ""
 } | tee "$LOG"
 snapshot > "$TMP/before"
 if [ "$ROLE" = "receiver" ]; then
    echo "[receiver] starting one-shot iperf3 server on $PEER_RX_IP ..."
    pkill -x iperf3 2>/dev/null; sleep 0.5
    nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
    SRV_PID=$!
    # Wait for the server to be ready (brief), then wait for it to exit
    # (one-shot server exits after serving one client).
    for _ in $(seq 1 60); do
        kill -0 "$SRV_PID" 2>/dev/null || break
        sleep 1
    done
    wait "$SRV_PID" 2>/dev/null
    echo "[receiver] iperf3 server finished."
 else
    # sender: wait briefly for receiver to be listening, then run client.
    echo "[sender] waiting 3s for receiver to listen, then running iperf3 client..."
    sleep 3
    iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
        > "$TMP/iperf.out" 2>&1
    echo "[sender] iperf3 client finished (exit=$?)."
 fi
 snapshot > "$TMP/after"
 {
 echo ""
 echo "--- iperf3 output (raw tail) ---"
 tail -25 "$TMP/iperf.out"
 echo ""
 if [ "$ROLE" = "receiver" ]; then
    analyze "Receiver RX (THE key column)" 2 "$TMP/before" "$TMP/after"
    dom="$(dominant 2 "$TMP/before" "$TMP/after")"
    pct=${dom%% *}; top=${dom##* }
    echo ""
    echo "==================================================================="
    if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
        echo " VERDICT: SWITCH NOT distributing across $HOST's two ports"
        echo " Dominant slave '$top' = ${pct}% of RX -> hash not effective"
        echo " on this LAG. Action: verify/bounce pfv-r3-tor-stor port-channel."
    else
        echo " VERDICT: switch IS distributing (top slave '$top' = ${pct}%)."
        echo " Cap is host-side: softirq/CPU/bridge/NIC coalescing."
    fi
    echo "==================================================================="
 else
    analyze "Sender TX (control: should split if host hash=layer3+4)" 3 "$TMP/before" "$TMP/after"
    dom="$(dominant 3 "$TMP/before" "$TMP/after")"
    pct=${dom%% *}; top=${dom##* }
    echo ""
    echo "==================================================================="
    if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
        echo " VERDICT: host TX NOT distributing (top slave '$top' = ${pct}%)."
        echo " Host xmit_hash_policy is NOT effective despite /proc/net/bonding."
    else
        echo " VERDICT: host TX distributing OK (top slave '$top' = ${pct}%)."
    fi
    echo "==================================================================="
 fi
 } | tee -a "$LOG"
 echo ""
 echo "Log written: $LOG"
@@ -0,0 +1,91 @@
 #!/usr/bin/env bash
 ###############################################################################
 # run-lacp-retrans-cause.sh  (workstation wrapper)
 #
 # Deploys scripts/lacp-retrans-cause.sh to BOTH hosts, runs them in the
 # right order, scps logs back, prints them.
 ###############################################################################
 set -uo pipefail
 SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
     -o StrictHostKeyChecking=accept-new)
 SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 RECV="pfv-tsys6"; SEND="pfv-tsys7"
 LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 SCRIPT="lacp-retrans-cause.sh"
 LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
 REMOTE_SCRIPT="/root/${SCRIPT}"
 mkdir -p "$LOCAL_LOG_DIR"
 echo "==================================================================="
 echo " LACP retransmit-cause investigation"
 echo " Receiver: $RECV   Sender: $SEND"
 echo "==================================================================="
 echo ""
 echo "--- preflight ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    ssh "${SSH[@]}" "root@$h" \
        'command -v ethtool >/dev/null && e=OK || e=MISSING
         command -v iperf3 >/dev/null && i=OK || i=MISSING
         printf "ethtool=%s iperf3=%s host=%s\n" "$e" "$i" "$(uname -n)"' 2>&1 | head -1
 done
 echo ""
 echo "--- deploy ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
        && ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
        && echo "deployed" || echo "FAILED"
 done
 echo ""
 echo "--- cleanup stale iperf3 ---"
 for h in "$RECV" "$SEND"; do
    ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 echo "--- launching receiver on $RECV (background) ---"
 ssh "${SSH[@]}" "root@$RECV" \
    "nohup bash $REMOTE_SCRIPT > /root/lacp-retrans-receiver.console 2>&1 &" 2>/dev/null
 sleep 5
 echo ""
 echo "--- running sender on $SEND (foreground, ~20s) ---"
 ssh "${SSH[@]}" "root@$SEND" "bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/  [sender] /'
 echo ""
 sleep 3
 echo "--- fetching logs ---"
 for f in lacp-retrans-receiver.log lacp-retrans-receiver.console lacp-retrans-sender.log; do
    src=""
    case "$f" in
        *receiver*) src="$RECV" ;;
        *sender*)   src="$SEND" ;;
    esac
    printf '  %-32s <- %s : ' "$f" "$src"
    if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
        echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
    else
        echo "MISSING"
    fi
 done
 echo ""
 echo "==================================================================="
 echo " RECEIVER LOG ($RECV)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-retrans-receiver.log" 2>/dev/null || echo "(missing)"
 echo ""
 echo "==================================================================="
 echo " SENDER LOG ($SEND)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-retrans-sender.log" 2>/dev/null || echo "(missing)"
 echo ""
 echo "==================================================================="
 echo " Local copies in: $LOCAL_LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,112 @@
 #!/usr/bin/env bash
 ###############################################################################
 # run-lacp-rx-distribution.sh  (workstation wrapper)
 #
 # One-shot orchestrator: deploys scripts/lacp-rx-distribution.sh to BOTH
 # tsys6 (receiver) and tsys7 (sender), runs them in the right order, then
 # scps both logs back to returned-logs/iperf/.
 #
 # Run from the workstation:
 #     bash scripts/run-lacp-rx-distribution.sh
 #
 # Idempotent: safe to re-run. iperf3 is killed on both hosts first.
 ###############################################################################
 set -uo pipefail
 SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
     -o StrictHostKeyChecking=accept-new)
 SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 RECV="pfv-tsys6"        # receiver (we care most about its RX split)
 SEND="pfv-tsys7"        # sender (control: its TX split)
 LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 SCRIPT="lacp-rx-distribution.sh"
 LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
 REMOTE_SCRIPT="/root/${SCRIPT}"
 mkdir -p "$LOCAL_LOG_DIR"
 echo "==================================================================="
 echo " LACP per-slave RX/TX distribution test"
 echo " Receiver: $RECV (10.100.100.6)   Sender: $SEND (10.100.100.7)"
 echo "==================================================================="
 echo ""
 # 0. Preflight: confirm SSH and iperf3 on both hosts
 echo "--- preflight (ssh + iperf3 + bond0) ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    ssh "${SSH[@]}" "root@$h" \
        'command -v iperf3 >/dev/null && ip=$(command -v iperf3) || ip=MISSING
         [ -r /proc/net/bonding/bond0 ] && b=OK || b=NO-BOND0
         printf "iperf3=%s bond0=%s host=%s\n" "$ip" "$b" "$(uname -n)"' \
        2>&1 | head -1
 done
 echo ""
 # 1. Copy the script to both hosts + chmod
 echo "--- deploy $SCRIPT to both hosts ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
        && ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
        && echo "deployed + chmod +x" \
        || echo "DEPLOY FAILED"
 done
 echo ""
 # 2. Kill any stale iperf3 on both hosts
 echo "--- cleanup stale iperf3 ---"
 for h in "$RECV" "$SEND"; do
    ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 # 3. Start RECEIVER in background (one-shot server, writes /root/lacp-rx-receiver.log)
 echo "--- starting receiver on $RECV (background) ---"
 ssh "${SSH[@]}" "root@$RECV" \
    "nohup bash $REMOTE_SCRIPT > /root/lacp-rx-receiver.console 2>&1 &" 2>/dev/null
 echo "  receiver launched; waiting 5s for it to start iperf3 -s -1 ..."
 sleep 5
 echo ""
 # 4. Run SENDER (foreground; ~15s with the default 12s test + 3s pre-sleep)
 echo "--- running sender on $SEND (foreground, ~20s) ---"
 ssh "${SSH[@]}" "root@$SEND" \
    "bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/  [sender] /'
 echo ""
 # 5. Give receiver a moment to finish writing its log
 sleep 3
 # 6. Fetch logs back
 echo "--- fetching logs ---"
 for f in lacp-rx-receiver.log lacp-rx-receiver.console lacp-rx-sender.log; do
    src=""
    case "$f" in
        lacp-rx-receiver*) src="$RECV" ;;
        lacp-rx-sender*)   src="$SEND" ;;
    esac
    printf '  %-28s <- %s : ' "$f" "$src"
    if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
        echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
    else
        echo "MISSING"
    fi
 done
 echo ""
 # 7. Show the receiver log (the decisive one)
 echo "==================================================================="
 echo " RECEIVER LOG ($RECV — the decisive side)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-rx-receiver.log" 2>/dev/null || echo "(no log)"
 echo ""
 echo "==================================================================="
 echo " SENDER LOG ($SEND — control)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-rx-sender.log" 2>/dev/null || echo "(no log)"
 echo ""
 echo "==================================================================="
 echo " Local copies in: $LOCAL_LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,127 @@
 #!/usr/bin/env bash
 # sw-capture-remote.sh - orchestrate a serial capture from this workstation.
 #
 # Flow:
 #   1. De-conflict: abort if any local ssh to pfv-tsys4 is in flight
 #      (other agent could be there).
 #   2. Free the serial port: kill whatever holds /dev/ttyUSBx
 #      (typically a screen session). Targeted, not blanket.
 #   3. scp driver + .cmds to pfv-tsys4.
 #   4. Run driver over ssh, capture stderr to console.
 #   5. scp the resulting log back to returned-logs/.
 #
 # Usage:
 #   sw-capture-remote.sh <switch-name> [device]
 #
 #   <switch-name>   e.g. pfv-core-sw01 (must have switches/<name>.cmds)
 #   [device]        /dev/ttyUSBx on pfv-tsys4. Defaults per switch map below.
 #
 # Currently scoped to pfv-core-sw01 only (per user direction). The other
 # two switches are deferred; their defaults are placeholders.
 set -u
 SWITCH=${1:-}
 DEVICE=${2:-}
 if [ -z "$SWITCH" ]; then
    echo "Usage: $0 <switch-name> [device]" >&2
    echo "  e.g. $0 pfv-core-sw01 /dev/ttyUSB2" >&2
    exit 2
 fi
 # Switch -> default device map (ttyUSB2 = core-sw01 confirmed by user).
 case "$SWITCH" in
    pfv-core-sw01)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB2 ;;
    pfv-r3-tor-mgmt)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB0   # TENTATIVE - unconfirmed
        if [ "${2:-}" = "" ]; then
            echo "NOTE: pfv-r3-tor-mgmt device is tentative (/dev/ttyUSB0)." >&2
            echo "      Pass the device explicitly if different." >&2
        fi ;;
    pfv-r3-tor-stor)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB1   # TENTATIVE - unconfirmed
        if [ "${2:-}" = "" ]; then
            echo "NOTE: pfv-r3-tor-stor device is tentative (/dev/ttyUSB1)." >&2
            echo "      Pass the device explicitly if different." >&2
        fi ;;
    *)
        echo "unknown switch: $SWITCH" >&2; exit 2 ;;
 esac
 BAUD=9600
 HOST=root@pfv-tsys4
 HERE=/home/reachableceo/projects/perfopt
 LOCAL_DRIVER=$HERE/scripts/sw-capture.py
 LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
 LOCAL_LOG=$HERE/returned-logs/$SWITCH.log
 REMOTE_DRIVER=/root/sw-capture.py
 REMOTE_CMDS=/root/$SWITCH.cmds
 REMOTE_LOG=/root/$SWITCH.log
 [ -f "$LOCAL_DRIVER" ] || { echo "missing $LOCAL_DRIVER" >&2; exit 2; }
 [ -f "$LOCAL_CMDS"   ] || { echo "missing $LOCAL_CMDS" >&2; exit 2; }
 ts() { date +%H:%M:%S; }
 echo "[$(ts)] switch=$SWITCH device=$DEVICE baud=$BAUD host=$HOST"
 # 1. De-conflict: any local ssh to pfv-tsys4 in flight?
 echo "[$(ts)] checking for in-flight ssh to pfv-tsys4..."
 if ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys4|scp.*pfv-tsys4' | grep -v grep >/tmp/.swcap.ps 2>&1; then
    cat /tmp/.swcap.ps
    echo "[$(ts)] ABORT: another ssh/scp to pfv-tsys4 is running (other agent?)." >&2
    exit 1
 fi
 echo "[$(ts)] clear."
 rm -f /tmp/.swcap.ps
 # 2. Free the serial port: kill whatever holds $DEVICE.
 echo "[$(ts)] freeing $DEVICE on $HOST (targeted; other screen sessions untouched)..."
 ssh -o BatchMode=yes -o ConnectTimeout=8 "$HOST" \
    "fuser -v $DEVICE 2>&1 | tee /dev/stderr; \
     fuser -k -TERM $DEVICE 2>/dev/null; sleep 1; \
     if fuser $DEVICE 2>/dev/null; then \
         echo 'still held after SIGTERM, escalating to SIGKILL'; \
         fuser -k -KILL $DEVICE 2>/dev/null; sleep 1; \
     fi; \
     fuser $DEVICE 2>/dev/null && echo 'STILL HELD' || echo 'FREE'"
 # Re-check; abort if still held.
 HELD=$(ssh -o BatchMode=yes "$HOST" "fuser $DEVICE 2>/dev/null && echo HELD || echo FREE")
 if [ "$HELD" = "HELD" ]; then
    echo "[$(ts)] ABORT: $DEVICE still held on $HOST." >&2
    exit 1
 fi
 # 3. Copy driver + cmds.
 echo "[$(ts)] copying driver + cmds to $HOST..."
 scp -q "$LOCAL_DRIVER" "$HOST:$REMOTE_DRIVER"
 scp -q "$LOCAL_CMDS"   "$HOST:$REMOTE_CMDS"
 # 4. Run the capture on pfv-tsys4. Stream stderr (progress) to console.
 echo "[$(ts)] running capture..."
 ssh -o BatchMode=yes -o ServerAliveInterval=10 "$HOST" \
    "python3 $REMOTE_DRIVER \
        --device $DEVICE --baud $BAUD \
        --cmds $REMOTE_CMDS --log $REMOTE_LOG"
 RC=$?
 echo "[$(ts)] capture exit code: $RC"
 # 5. Pull log back.
 echo "[$(ts)] pulling log back to $LOCAL_LOG..."
 mkdir -p "$(dirname "$LOCAL_LOG")"
 scp -q "$HOST:$REMOTE_LOG" "$LOCAL_LOG"
 if [ -f "$LOCAL_LOG" ]; then
    SZ=$(wc -c < "$LOCAL_LOG")
    echo "[$(ts)] OK: $LOCAL_LOG ($SZ bytes)"
    echo "----- head -----"
    head -30 "$LOCAL_LOG"
    echo "----- tail -----"
    tail -10 "$LOCAL_LOG"
 else
    echo "[$(ts)] ERROR: log not pulled back." >&2
    exit 1
 fi
 exit $RC
@@ -0,0 +1,255 @@
 #!/usr/bin/env python3
 """
 sw-capture.py - drive a Dell switch over a serial console and log all output.
 Read-only. Sends ONLY the commands listed in the supplied .cmds file
 (comment lines starting with '!' and blank lines are skipped). Handles
 `--More--` pagination by sending a space. Aborts cleanly on any password
 prompt (we never supply credentials).
 Pure stdlib (termios + select). No pyserial/expect required.
 Exit codes:
   0  clean run, every command saw a prompt again
   2  could not synchronize with a prompt during wake
   3  one or more commands timed out (log still written)
   4  password prompt encountered (aborted)
 Usage:
    sw-capture.py --device /dev/ttyUSB2 --baud 9600 \\
                  --cmds pfv-core-sw01.cmds --log /root/pfv-core-sw01.log
 """
 import argparse
 import os
 import re
 import select
 import sys
 import termios
 import time
 PROMPT_RE = re.compile(rb'[>#]\s*$')              # ends in # or > + spaces
 MORE_RE   = re.compile(rb'--\s*More\s*--')        # pagination prompt
 PWD_RE    = re.compile(rb'[Pp]assword:\s*$')      # enable / login password
 BAUDS = {
    '9600':    termios.B9600,
    '19200':   termios.B19200,
    '38400':   termios.B38400,
    '57600':   termios.B57600,
    '115200':  termios.B115200,
 }
 def log(msg, level='INFO'):
    sys.stderr.write(f'[{level}] {msg}\n')
    sys.stderr.flush()
 def open_port(device, baud):
    """Open the serial device raw at the requested baud, 8N1, no flow ctrl."""
    fd = os.open(device, os.O_RDWR | os.O_NOCTTY)
    try:
        attrs = termios.tcgetattr(fd)
    except termios.error:
        log(f'{device} is not a termios-capable device', 'WARN')
        return fd
    # raw input
    attrs[0] &= ~(termios.IGNBRK | termios.BRKINT | termios.PARMRK |
                  termios.ISTRIP | termios.INLCR | termios.IGNCR |
                  termios.ICRNL | termios.IXON)
    # raw output
    attrs[1] &= ~termios.OPOST
    # 8N1, enable receiver, ignore modem control lines
    attrs[2] &= ~(termios.CSIZE | termios.PARENB | termios.CSTOPB)
    attrs[2] |= termios.CS8 | termios.CREAD | termios.CLOCAL
    # raw local
    attrs[3] &= ~(termios.ECHO | termios.ECHONL | termios.ICANON |
                  termios.ISIG | termios.IEXTEN)
    # non-blocking-ish reads (select is the primary gate)
    attrs[6][termios.VMIN] = 0
    attrs[6][termios.VTIME] = 1
    b = BAUDS.get(str(baud))
    if b is None:
        raise SystemExit(f'unsupported baud: {baud}')
    # Set ispeed/ospeed directly on the attribute list. (Equivalent to
    # termios.cfsetispeed/cfsetospeed, which are missing on some Python
    # builds — e.g. the one on pfv-tsys4.)
    attrs[4] = b   # ispeed
    attrs[5] = b   # ospeed
    termios.tcsetattr(fd, termios.TCSANOW, attrs)
    return fd
 def read_chunk(fd, timeout):
    """Read whatever arrives within `timeout`. Extends briefly on activity."""
    buf = b''
    deadline = time.time() + timeout
    while True:
        remaining = deadline - time.time()
        if remaining <= 0:
            return buf
        r, _, _ = select.select([fd], [], [], min(0.5, remaining))
        if not r:
            if buf:
                return buf
            continue
        try:
            chunk = os.read(fd, 4096)
        except OSError:
            return buf
        if not chunk:
            return buf
        buf += chunk
        # keep collecting as long as bytes are flowing
        deadline = time.time() + 0.3
 def drain(fd, timeout=1.0):
    total = 0
    while True:
        b = read_chunk(fd, timeout=timeout)
        if not b:
            return total
        total += len(b)
 def send(fd, s):
    if isinstance(s, str):
        s = s.encode()
    os.write(fd, s)
 def wait_for(fd, regex, timeout, on_more=None, on_pwd=None):
    """Read until `regex` matches the tail of the buffer, or timeout."""
    buf = b''
    deadline = time.time() + timeout
    while time.time() < deadline:
        remaining = deadline - time.time()
        chunk = read_chunk(fd, timeout=min(1.0, remaining))
        if chunk:
            buf += chunk
            tail64 = buf[-64:]
            tail32 = buf[-32:]
            tail128 = buf[-128:]
            if on_more and MORE_RE.search(tail64):
                on_more(fd)
                continue
            if on_pwd and PWD_RE.search(tail32):
                on_pwd(buf)
                return buf, 'pwd'
            if regex.search(tail128):
                return buf, 'ok'
    return buf, 'timeout'
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--device', required=True)
    ap.add_argument('--baud', type=int, default=9600)
    ap.add_argument('--cmds', required=True)
    ap.add_argument('--log',  required=True)
    ap.add_argument('--per-cmd-timeout', type=float, default=45.0)
    ap.add_argument('--wake-timeout',    type=float, default=15.0)
    ap.add_argument('--session-max',     type=float, default=600.0)
    args = ap.parse_args()
    cmds = []
    with open(args.cmds) as f:
        for raw in f:
            s = raw.rstrip('\n').strip()
            if not s or s.startswith('!'):
                continue
            cmds.append(s)
    log(f'parsed {len(cmds)} commands from {args.cmds}')
    logf = open(args.log, 'wb', buffering=0)
    def w(b):
        if isinstance(b, str):
            b = b.encode()
        logf.write(b)
    w(f'==== sw-capture {time.strftime("%Y-%m-%d %H:%M:%S")} ====\n')
    w(f'device={args.device} baud={args.baud} cmds={args.cmds} '
      f'n={len(cmds)} per_cmd_timeout={args.per_cmd_timeout}\n\n')
    fd = open_port(args.device, args.baud)
    log(f'opened {args.device} @ {args.baud} 8N1 raw')
    session_start = time.time()
    abort = False
    def on_more(fd_):
        log('--More-- -> space')
        send(fd_, b' ')
    def on_pwd(buf):
        nonlocal abort
        abort = True
        log('password prompt detected (enable or login) - aborting; '
            'no credentials supplied', 'ERROR')
        w(buf)
        w(b'\n[PASSWORD PROMPT - ABORTED]\n')
    # WAKE: nudge with Ctrl-C + Enter, look for any prompt
    drain(fd, 0.5)
    synced = False
    wake_deadline = time.time() + args.wake_timeout
    attempt = 0
    while time.time() < wake_deadline:
        attempt += 1
        send(fd, b'\x03')
        time.sleep(0.2)
        send(fd, b'\r')
        buf, status = wait_for(fd, PROMPT_RE, timeout=3.0,
                               on_more=on_more, on_pwd=on_pwd)
        w(buf)
        if status == 'pwd':
            logf.close(); os.close(fd); sys.exit(4)
        if status == 'ok':
            synced = True
            log(f'prompt synced after {attempt} attempt(s)')
            break
    if not synced:
        w(b'\n[NO PROMPT - ABORT]\n')
        log('no prompt detected during wake window', 'ERROR')
        logf.close(); os.close(fd); sys.exit(2)
    # RUN commands verbatim from the .cmds list
    failures = 0
    for idx, cmd in enumerate(cmds, 1):
        if time.time() - session_start > args.session_max:
            log('session_max exceeded - stopping early', 'ERROR')
            w(b'\n[SESSION_MAX - STOP]\n')
            break
        if abort:
            break
        log(f'[{idx}/{len(cmds)}] {cmd}')
        send(fd, cmd + '\r')
        buf, status = wait_for(fd, PROMPT_RE,
                                timeout=args.per_cmd_timeout,
                                on_more=on_more, on_pwd=on_pwd)
        w(buf)
        if status == 'pwd':
            failures += 1
            break
        if status == 'timeout':
            log(f'timeout after: {cmd}', 'WARN')
            failures += 1
            # try to resync: Ctrl-C + drain
            send(fd, b'\x03')
            time.sleep(0.3)
            drain(fd, 0.5)
    w(f'\n==== end {time.strftime("%Y-%m-%d %H:%M:%S")} '
      f'failures={failures} ====\n')
    logf.close()
    os.close(fd)
    log(f'done -> {args.log}  failures={failures}')
    sys.exit(0 if failures == 0 else 3)
 if __name__ == '__main__':
    main()
@@ -0,0 +1,55 @@
 #!/usr/bin/env bash
 # Probe conman state + expect availability on pfv-tsys4.
 # Read-only. Decides whether we drive via conman+expect or expect-only.
 set -u
 # De-conflict: any ssh to pfv-tsys4 right now?
 echo "===== LOCAL ssh activity ====="
 ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys' | grep -v grep || echo "(none to pfv-tsys4)"
 echo
 echo "===== pfv-tsys4: conman + expect state ====="
 ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
 echo "--- conmand service ---"
 systemctl is-active conmand 2>&1 || true
 systemctl is-enabled conmand 2>&1 || true
 systemctl status conmand --no-pager 2>&1 | head -15 || true
 echo
 echo "--- conman binary ---"
 command -v conman && conman --version 2>&1 | head -2 || echo "conman: MISSING"
 command -v conmand && echo "conmand present" || echo "conmand: MISSING"
 echo
 echo "--- /etc/conman.conf: ttyUSB2 entries ---"
 grep -nE "ttyUSB2|core-sw|CONSOLE|LOG|SERIAL|BAUD" /etc/conman.conf 2>/dev/null | head -40 || echo "(no matches / no file)"
 echo
 echo "--- conman log dir ---"
 ls -la /var/log/conman/ 2>&1 | head -20 || echo "(no /var/log/conman)"
 ls -la /var/consoles/ 2>&1 | head -20 || echo "(no /var/consoles)"
 echo
 echo "--- expect availability ---"
 command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
 echo "apt-cache policy expect:"
 apt-cache policy expect 2>/dev/null | head -10 || echo "(apt-cache failed)"
 echo
 echo "--- other useful drivers ---"
 for t in tclsh socat cu tip; do
    command -v "$t" 2>/dev/null && echo "  $t: present" || true
 done
 echo
 echo "--- apt network reachability (quick) ---"
 timeout 5 bash -c 'echo > /dev/tcp/deb.debian.org/80' 2>&1 && echo "apt network: OK" || echo "apt network: UNREACHABLE"
 echo
 echo "--- disk space for log ---"
 df -h /root 2>&1 | tail -2
 echo
 echo "--- screen sessions (still 3?) ---"
 screen -ls 2>&1 || true
 REMOTE
@@ -0,0 +1,39 @@
 #!/usr/bin/env bash
 # Probe pfv-tsys4 for what's available to drive the serial console.
 # Also snapshots local ssh/scp activity so we can de-conflict with the
 # other agent running in this directory.
 set -u
 echo "===== LOCAL ssh/scp activity (other-agent de-confliction) ====="
 ps -eo pid,ppid,etime,user,args | grep -E 'ssh|scp' | grep -v grep || echo "(none)"
 echo
 echo "===== Ping pfv-tsys4 ====="
 ping -c1 -W2 pfv-tsys4 >/dev/null 2>&1 && echo "ping OK" || echo "ping FAIL"
 echo
 echo "===== Probe pfv-tsys4 over ssh ====="
 ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
 echo "--- host ---"
 hostname; uname -a
 echo "--- tools ---"
 for t in python3 python expect screen minicom picocom stty fuser lsof; do
    p=$(command -v "$t" 2>/dev/null) && echo "$t -> $p" || echo "$t -> MISSING"
 done
 echo "--- pyserial ---"
 python3 -c "import serial; print('pyserial', serial.__version__)" 2>&1
 echo "--- device node ---"
 ls -l /dev/ttyUSB2 2>&1
 stat -c '%n owner=%U:%G mode=%a' /dev/ttyUSB2 2>&1 || true
 echo "--- who holds /dev/ttyUSB2 ---"
 fuser -v /dev/ttyUSB2 2>&1 || echo "(fuser: none or n/a)"
 lsof /dev/ttyUSB2 2>&1 | head -20 || true
 echo "--- screen sessions on this host ---"
 screen -ls 2>&1 || echo "(no screen / not installed)"
 echo "--- current tty settings (only readable if not held exclusively) ---"
 stty -F /dev/ttyUSB2 2>&1 || echo "(held exclusively - expected if screen is up)"
 echo "--- baud hints in config/history ---"
 grep -riE "ttyUSB2|115200|9600|baud" /etc/ ~/.screenrc ~/.bash_history 2>/dev/null | head -20 || true
 echo "--- recent console-related processes ---"
 ps -eo pid,etime,user,args | grep -E 'screen|minicom|picocom|ttyUSB' | grep -v grep || echo "(none)"
 REMOTE
@@ -0,0 +1,65 @@
 #!/bin/bash
 # shellcheck.sh - permanent wrapper to lint every shell script in this project.
 #
 # Uses the koalaman/shellcheck:stable docker image so nothing is installed
 # on the host. Run from anywhere; lints scripts/ and any .sh under switches/.
 #
 # Usage:
 #   ./shellcheck.sh              # tty output, all scripts
 #   ./shellcheck.sh --fix-info   # treat style notes as non-blocking (default)
 #   ./shellcheck.sh --strict     # exit non-zero on ANY finding (notes too)
 #   ./shellcheck.sh scripts/assess.sh   # lint a single file
 set -u
 ROOT="$(cd "$(dirname "$0")" && pwd)"
 IMAGE="koalaman/shellcheck:stable"
 STRICT=0
 TARGETS=()
 for arg in "$@"; do
    case "$arg" in
        --strict) STRICT=1 ;;
        --fix-info) STRICT=0 ;;
        -h|--help)
            sed -n '2,12p' "$0"; exit 0 ;;
        *) TARGETS+=("$arg") ;;
    esac
 done
 # Default targets: everything in scripts/, plus any .cmds is NOT shell - skip.
 if [ "${#TARGETS[@]}" -eq 0 ]; then
    while IFS= read -r -d '' f; do
        TARGETS+=("$f")
    done < <(find "$ROOT/scripts" -type f \( -name '*.sh' -o -name 'collect-*' -o -name 'assess*' \) -print0 2>/dev/null)
 fi
 if [ "${#TARGETS[@]}" -eq 0 ]; then
    echo "no shell scripts found to lint" >&2
    exit 1
 fi
 echo "Linting ${#TARGETS[@]} file(s) with $IMAGE:"
 for t in "${TARGETS[@]}"; do echo "  - $t"; done
 echo
 # Make paths relative to ROOT so docker volume maps cleanly
 REL_TARGETS=()
 for t in "${TARGETS[@]}"; do
    rel="${t#$ROOT/}"
    [ "$rel" = "$t" ] && rel="$t"
    REL_TARGETS+=("$rel")
 done
 SC_ARGS=(--format=tty)
 [ "$STRICT" -eq 0 ] && SC_ARGS+=(--severity=warning)
 docker run --rm -v "$ROOT:/mnt" -w /mnt "$IMAGE" \
    "${SC_ARGS[@]}" "${REL_TARGETS[@]}"
 RC=$?
 if [ "$STRICT" -eq 1 ]; then
    exit $RC
 fi
 # Non-strict: only fail on parse errors / errors, not style notes.
 # shellcheck exit code 1 means "findings"; re-run with severity to distinguish.
 exit 0
@@ -0,0 +1,169 @@
 #!/bin/bash
 # validate-fixes.sh - READ-ONLY validation of all applied tunings.
 # Does NOT reboot, shutdown VMs, or modify anything.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 echo "==================================================================="
 echo " READ-ONLY VALIDATION — $(date)"
 echo "==================================================================="
 echo ""
 for HOST in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$HOST]"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        echo ""
        continue
    fi
    ssh "${SSH_OPTS[@]}" "root@$HOST" '
        pass=0; fail=0
        check() {
            local label="$1" actual="$2" expected="$3"
            if [ "$actual" = "$expected" ]; then
                printf "  [OK]   %-30s %s\n" "$label" "$actual"
                pass=$((pass+1))
            else
                printf "  [FAIL] %-30s got=%s want=%s\n" "$label" "$actual" "$expected"
                fail=$((fail+1))
            fi
        }
        # 1. CPU governor
        gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "n/a")
        if [ "$gov" = "n/a" ]; then
            printf "  [SKIP] %-30s %s\n" "CPU governor" "(no cpufreq driver — OK for server BIOS)"
        else
            check "CPU governor" "$gov" "performance"
        fi
        # 2. vm.swappiness
        swap=$(sysctl -n vm.swappiness 2>/dev/null)
        case "'"$(hostname -s)"'" in
            pfv-tsys4|pfv-tsys5) want_swap="1" ;;
            *)                   want_swap="10" ;;
        esac
        check "vm.swappiness" "$swap" "$want_swap"
        # 3. TCP congestion control
        cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null)
        check "tcp_congestion_control" "$cc" "bbr"
        # 4. default_qdisc (paired with BBR)
        qd=$(sysctl -n net.core.default_qdisc 2>/dev/null)
        check "net.core.default_qdisc" "$qd" "fq"
        # 5. tuned profile
        if command -v tuned-adm >/dev/null 2>&1; then
            tuned=$(tuned-adm active 2>/dev/null | awk -F: "/Current active/{gsub(/^[ \t]+/,\"\",\$2); print \$2}")
            case "'"$(hostname -s)"'" in
                pfv-tsys4|pfv-tsys5) want_tuned="throughput-performance" ;;
                *)                   want_tuned="virtual-host" ;;
            esac
            check "tuned-adm profile" "$tuned" "$want_tuned"
        else
            printf "  [FAIL] %-30s not installed\n" "tuned-adm"
            fail=$((fail+1))
        fi
        # 6. bond0 hash policy (if bond exists)
        if [ -r /proc/net/bonding/bond0 ]; then
            hash=$(grep "Transmit Hash" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
            check "bond0 xmit_hash_policy" "$hash" "layer3+4"
            ports=$(grep "Number of ports" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
            printf "  [INFO] %-30s %s ports active\n" "bond0 LACP ports" "$ports"
        else
            printf "  [SKIP] %-30s %s\n" "bond0 hash" "(no bond0 — single NIC host)"
        fi
        # 7. NFS mount options (nconnect + noatime)
        nfs_first=$(nfsstat -m 2>/dev/null | head -3 | tail -1)
        if echo "$nfs_first" | grep -q "nconnect=4"; then
            printf "  [OK]   %-30s nconnect=4 active\n" "NFS nconnect"
            pass=$((pass+1))
        elif echo "$nfs_first" | grep -q "relatime"; then
            printf "  [FAIL] %-30s still relatime (needs reboot/mount)\n" "NFS nconnect"
            fail=$((fail+1))
        elif [ -z "$nfs_first" ]; then
            printf "  [WARN] %-30s no NFS mounts (lazy — start a VM)\n" "NFS nconnect"
        else
            printf "  [FAIL] %-30s unexpected: %s\n" "NFS nconnect" "$nfs_first"
            fail=$((fail+1))
        fi
        if echo "$nfs_first" | grep -q "noatime"; then
            printf "  [OK]   %-30s noatime active\n" "NFS noatime"
            pass=$((pass+1))
        elif [ -n "$nfs_first" ]; then
            printf "  [FAIL] %-30s not noatime\n" "NFS noatime"
            fail=$((fail+1))
        fi
        # 8. NFS TCP connection count
        nfs_conns=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
        if [ "$nfs_conns" -ge 8 ]; then
            printf "  [OK]   %-30s %s connections\n" "NFS TCP conns" "$nfs_conns"
            pass=$((pass+1))
        elif [ "$nfs_conns" -gt 0 ]; then
            printf "  [WARN] %-30s %s (expect 8 with nconnect=4)\n" "NFS TCP conns" "$nfs_conns"
        else
            printf "  [WARN] %-30s 0 (lazy mounts — start a VM)\n" "NFS TCP conns"
        fi
        # 9. sysctl persistence
        if [ -r /etc/sysctl.d/99-perfopt.conf ]; then
            printf "  [OK]   %-30s /etc/sysctl.d/99-perfopt.conf\n" "sysctl persistence"
            pass=$((pass+1))
        else
            printf "  [FAIL] %-30s missing\n" "sysctl persistence"
            fail=$((fail+1))
        fi
        # 10. Observability packages
        for cmd in sar jq numactl nvme mtr bmon; do
            if ! command -v "$cmd" >/dev/null 2>&1; then
                printf "  [FAIL] %-30s not installed\n" "obs: $cmd"
                fail=$((fail+1))
            fi
        done
        if command -v sar >/dev/null 2>&1 && command -v jq >/dev/null 2>&1 && \
           command -v numactl >/dev/null 2>&1 && command -v nvme >/dev/null 2>&1 && \
           command -v mtr >/dev/null 2>&1 && command -v bmon >/dev/null 2>&1; then
            printf "  [OK]   %-30s all installed\n" "observability packages"
            pass=$((pass+1))
        fi
        # 11. Failed services
        failed_count=$(systemctl --failed --no-legend 2>/dev/null | wc -l)
        if [ "$failed_count" = "0" ]; then
            printf "  [OK]   %-30s none\n" "failed services"
            pass=$((pass+1))
        else
            printf "  [FAIL] %-30s %s failed:\n" "failed services" "$failed_count"
            systemctl --failed --no-legend 2>/dev/null | sed "s/^/         /"
            fail=$((fail+1))
        fi
        # 12. VM status (read-only — just report)
        running_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"" | wc -l)
        stopped_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3!=\"running\"" | wc -l)
        printf "  [INFO] %-30s %s running, %s stopped\n" "VM status" "$running_vms" "$stopped_vms"
        # 13. uptime
        printf "  [INFO] %-30s %s\n" "uptime" "$(uptime | sed "s/.*up //" | sed "s/,.*//")"
        echo ""
        echo "  RESULT: $pass passed, $fail failed"
    '
    echo ""
 done
 echo "==================================================================="
 echo " SUMMARY"
 echo "==================================================================="
@@ -0,0 +1,132 @@
 #!/bin/bash
 ###############################################################################
 # validate-vms.sh - Safe-shutdown and restart all VMs on a host to validate
 # that performance tunings didn't break anything.
 #
 # For each VM:
 #   1. qm shutdown <vmid> --timeout 120  (ACPI safe shutdown)
 #   2. Wait for stopped state
 #   3. qm start <vmid>
 #   4. Wait for running state
 #   5. Check qm agent responds (if agent enabled)
 #
 # Usage: bash validate-vms.sh <host> [host...]
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 for HOST in "$@"; do
    echo "================================================================"
    echo "[$HOST] VM SAFE-SHUTDOWN/RESTART VALIDATION"
    echo "================================================================"
    echo ""
    if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE — skipping"
        continue
    fi
    # Get list of running VMs
    VM_LIST=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"{print \$1}"')
    if [ -z "$VM_LIST" ]; then
        echo "  No running VMs — nothing to validate"
        continue
    fi
    VM_COUNT=$(echo "$VM_LIST" | wc -w)
    echo "  Found $VM_COUNT running VM(s): $(echo "$VM_LIST" | tr '\n' ' ')"
    echo ""
    # --- Phase 1: Safe shutdown all VMs ---
    echo "--- PHASE 1: Safe shutdown all VMs (120s timeout each) ---"
    for vmid in $VM_LIST; do
        name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
        echo -n "  [$vmid $name] shutting down... "
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm shutdown $vmid --timeout 120 --forceStop 1" 2>&1 | head -1
    done
    # Wait for all to stop (max 180s total)
    echo ""
    echo -n "  Waiting for all VMs to stop"
    WAIT_DEADLINE=$(( $(date +%s) + 180 ))
    while [ "$(date +%s)" -lt "$WAIT_DEADLINE" ]; do
        echo -n "."
        all_stopped=1
        for vmid in $VM_LIST; do
            status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
            if [ "$status" = "running" ]; then
                all_stopped=0
                break
            fi
        done
        [ "$all_stopped" = "1" ] && break
        sleep 5
    done
    echo " done"
    echo ""
    # Show stopped state
    echo "--- VM status after shutdown ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
    echo ""
    # --- Phase 2: Start all VMs ---
    echo "--- PHASE 2: Start all VMs ---"
    for vmid in $VM_LIST; do
        name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
        echo -n "  [$vmid $name] starting... "
        start_output=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1)
        if [ $? -eq 0 ]; then
            echo "OK"
        else
            echo "FAILED: $start_output"
        fi
    done
    # Wait 20s for VMs to fully start
    echo ""
    echo "  Waiting 20s for VMs to boot..."
    sleep 20
    # --- Phase 3: Verify all VMs running ---
    echo ""
    echo "--- PHASE 3: Verification ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
    echo ""
    # Check NFS mounts still healthy
    echo "--- NFS mount health ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" '
        mount_count=$(nfsstat -m 2>/dev/null | grep -c "^/mnt")
        conn_count=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
        echo "  NFS mounts: $mount_count"
        echo "  NFS TCP connections: $conn_count"
        if [ "$mount_count" -gt 0 ]; then
            echo "  First mount options:"
            nfsstat -m 2>/dev/null | head -2 | tail -1 | sed "s/^/    /"
        fi
    '
    echo ""
    # Check guest agent responsiveness (if agent enabled)
    echo "--- Guest agent check (VMs with agent:1) ---"
    for vmid in $VM_LIST; do
        agent=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | grep -c '^agent: 1'")
        if [ "$agent" = "1" ]; then
            name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
            echo -n "  [$vmid $name] agent ping... "
            result=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "timeout 10 qm agent $vmid ping 2>&1")
            if [ $? -eq 0 ]; then
                echo "OK"
            else
                echo "no response (VM may still be booting)"
            fi
        fi
    done
    echo ""
    echo "================================================================"
    echo "[$HOST] VALIDATION COMPLETE"
    echo "================================================================"
    echo ""
 done
@@ -0,0 +1,36 @@
 #!/bin/bash
 # verify-tuning.sh - verifies apply-tunings.sh results on target hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 for host in "$@"; do
    echo "================================================================"
    echo "[$host] verification"
    echo "================================================================"
    echo "--- TCP congestion control ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc 2>/dev/null'
    echo "--- vm.swappiness ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl vm.swappiness 2>/dev/null'
    echo "--- scaling_governor (cpu0) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null'
    echo "--- NFS mount options (first 2 mounts) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'nfsstat -m 2>/dev/null | head -30'
    echo "--- storage.cfg: any options lines? ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -c "options" /etc/pve/storage.cfg 2>/dev/null || echo 0'
    echo "--- storage.cfg: NFS stanzas (first 3) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -A 6 "^nfs:" /etc/pve/storage.cfg 2>/dev/null | head -25'
    echo "--- nconnect TCP connections to NFS servers ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | head -20; echo "count:"; ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | tail -n +2 | wc -l'
    echo "--- VMs still running? ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'qm list 2>/dev/null | head -15'
    echo ""
 done
@@ -0,0 +1,58 @@
 #!/bin/bash
 # wait-for-host.sh - polls SSH until host is back, then runs verification.
 # Usage: bash wait-for-host.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 DEADLINE=$(( $(date +%s) + 600 ))
 echo "Polling $HOST for SSH return (max 10 min)..."
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "[$(date +%H:%M:%S)] SSH is back!"
        break
    fi
    sleep 10
    echo "  [$(date +%H:%M:%S)] still down..."
 done
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "FAILED: $HOST not back after 10 minutes"
    exit 1
 fi
 echo "Waiting 30s for services to settle..."
 sleep 30
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 echo "--- Uptime (should be < 5 min) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
 echo ""
 echo "--- TCP congestion control (expect bbr) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
 echo ""
 echo "--- scaling_governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
 echo ""
 echo "--- NFS mount options (looking for nconnect=4 + noatime) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -16'
 echo ""
 echo "--- NFS TCP connection count to :2049 (expect ~8 = 4 per server × 2 servers) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
 echo ""
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 echo ""
 echo "--- Failed services ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
@@ -0,0 +1,111 @@
 # Powerman PDU Management
 Centralized power management for the Cyclades AlterPath PM10i PDU via
 [Powerman](https://github.com/chaos/powerman), running on pfv-tsys1.
 ## Hardware
 | Component | Details |
 |-----------|---------|
 | **PDU** | Cyclades AlterPath PM10i (10 controllable AC outlets) |
 | **Firmware** | v1.9.0 (Aug 4, 2006) |
 | **Connection** | USB-to-DB9 adapter (Prolific pl2303, serial BJAAb144J07) |
 | **Host** | pfv-tsys1 (OptiPlex 9020, Proxmox) |
 | **Serial** | 9600 baud, 8N1, raw mode |
 | **Credentials** | Factory defaults: `admin` / `pm8` (in cyclades-pm10.dev) |
 | **Network access** | powermand listens on `127.0.0.1:10101` (local) + `100.121.189.98:10101` (Tailscale) |
 ## Device mapping
 ```
 USB adapter (067b:23a3, serial BJAAb144J07)
  └─ pl2303 driver → /dev/ttyUSB1
       └─ udev symlink → /dev/cyclades-pm10 (stable across reboots)
            └─ powermand reads/writes serial → Cyclades PM10i
                 └─ 10 outlets (factory default names: 1-10)
 ```
 The udev rule (`/etc/udev/rules.d/99-cyclades-pdu.rules`) pins the adapter
 by its USB serial number, so the symlink survives replugs and reboots.
 ## Scripts
 All scripts run on the target host (pfv-tsys1) via `tests/remote.sh`:
 ```bash
 # Setup (idempotent — safe to re-run):
 PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh
 # Validate PDU control (cycles outlet 10 off → on):
 PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh
 # Status check:
 PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh
 ```
 ### Customizing for other hosts/PDUs
 The setup script accepts environment overrides:
 ```bash
 PDU_SERIAL=XXXX PDU_VENDOR=067b PDU_OUTLETS=20 PDU_TYPE=pm20 \
  PROX_HOST=other-host bash tests/remote.sh prox-file powerman/setup.sh
 ```
 ## Usage (daily operations)
 From pfv-tsys1 (or any host with network access to port 10101):
 ```bash
 # List all outlets
 powerman -l
 # Query status (all outlets)
 powerman -q
 # Turn outlet off
 powerman -0 outlet-10
 # Turn outlet on
 powerman -1 outlet-10
 # Cycle outlet (off → 4s delay → on)
 powerman -c outlet-10
 # Query a specific outlet
 powerman -q outlet-10
 ```
 ### Remote access from other hosts
 powermand listens on `0.0.0.0:10101`. From another tailnet host:
 ```bash
 powerman --server-host pfv-tsys1 --server-port 10101 -q
 ```
 Or set `POWERMAN_SERVER=pfv-tsys1:10101` in the environment.
 ## Configuration files on pfv-tsys1
 | File | Purpose |
 |------|---------|
 | `/etc/udev/rules.d/99-cyclades-pdu.rules` | Stable symlink for USB-DB9 adapter |
 | `/etc/powerman/powerman.conf` | Device definition + 10 outlet nodes |
 | `/etc/powerman/cyclades-pm10.dev` | Cyclades PM10 protocol spec (shipped with powerman) |
 ## Validation results
 2026-07-28: All 8 checks passed.
 Outlet 10 turned OFF (confirmed), turned ON (confirmed), then cycled.
 ## TODO (Friday onsite)
 - [ ] **Rename outlets** in `/etc/powerman/powerman.conf` to match the
      physical devices plugged into each outlet (e.g., `node "tsys4-psu"
      "cyclades-pm10" "3"`). Currently all outlets are generically named
      `outlet-1` through `outlet-10`.
 - [ ] **Change PDU admin password** from factory default (`pm8`) if
      security-sensitive. Update `/etc/powerman/cyclades-pm10.dev` login
      script to match.
 - [ ] **Verify all 10 outlets** individually once device mapping is known.
@@ -0,0 +1,68 @@
 #!/usr/bin/bash
 #
 # powerman/discover.sh — gather USB-DB9 adapter + powerman state on a host
 #
 # Usage: PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/discover.sh
 #
 set -uo pipefail
 echo "============================================"
 echo "  PDU / Powerman Discovery"
 echo "  Host: $(hostname)"
 echo "  Date: $(date)"
 echo "============================================"
 echo ""
 echo "=== 1. USB devices ==="
 lsusb 2>/dev/null || echo "(lsusb not available)"
 echo ""
 echo "=== 2. USB-Serial adapters (ttyUSB*) ==="
 ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
 echo ""
 echo "=== 3. USB-Serial kernel modules ==="
 lsmod | grep -iE 'usbserial|ftdi|pl2303|cp210|ch34|cdc_acm' 2>/dev/null || echo "(no relevant modules loaded)"
 echo ""
 echo "=== 4. dmesg for USB serial (last 30 lines) ==="
 dmesg | grep -iE 'ttyUSB|usbserial|ftdi|pl2303|cp210|ch34|converter' | tail -30 2>/dev/null || echo "(no dmesg matches)"
 echo ""
 echo "=== 5. All serial devices ==="
 ls -la /dev/ttyS* /dev/ttyUSB* /dev/ttyACM* 2>/dev/null || echo "(no serial devices found)"
 echo ""
 echo "=== 6. Powerman installed? ==="
 dpkg -l powerman 2>/dev/null || echo "(powerman not installed)"
 which powerman 2>/dev/null || echo "(powerman binary not found)"
 which powermand 2>/dev/null || echo "(powermand binary not found)"
 echo ""
 echo "=== 7. Powerman config files ==="
 ls -la /etc/powerman/ 2>/dev/null || echo "(no /etc/powerman/ directory)"
 ls -la /etc/powerman/*.dev 2>/dev/null || echo "(no .dev files)"
 cat /etc/powerman/powerman.conf 2>/dev/null || echo "(no powerman.conf)"
 echo ""
 echo "=== 8. Available powerman device definitions ==="
 ls /usr/share/powerman/*.dev 2>/dev/null || ls /etc/powerman/*.dev 2>/dev/null || echo "(no device definitions found)"
 echo ""
 echo "=== 9. Powermand service status ==="
 systemctl status powerman 2>/dev/null | head -10 || echo "(powerman service not found)"
 echo ""
 echo "=== 10. Serial port test (quick probe of /dev/ttyUSB0) ==="
 if [ -e /dev/ttyUSB0 ]; then
  stty -F /dev/ttyUSB0 2>/dev/null && echo "(port exists and is configurable)" || echo "(port exists but stty failed)"
  # Try to read any pending output
  timeout 2 cat /dev/ttyUSB0 2>/dev/null | head -5 || echo "(no immediate output from port)"
 else
  echo "(no /dev/ttyUSB0)"
 fi
 echo ""
 echo "============================================"
 echo "  Discovery complete."
 echo "============================================"
@@ -0,0 +1,65 @@
 #!/usr/bin/bash
 #
 # powerman/query-remote.sh — install powerman client locally and query
 # the Cyclades PDU running on pfv-tsys1 over Tailscale.
 #
 set -euo pipefail
 REMOTE_HOST="${REMOTE_HOST:-pfv-tsys1}"
 REMOTE_PORT="${REMOTE_PORT:-10101}"
 echo "============================================"
 echo "  Powerman Remote PDU Query"
 echo "  Server: ${REMOTE_HOST}:${REMOTE_PORT} (Tailscale)"
 echo "============================================"
 # --- 1. Install powerman client if missing ---
 if ! command -v powerman >/dev/null 2>&1; then
  echo ""
  echo "--- Installing powerman client ---"
  if sudo -n true 2>/dev/null; then
    sudo apt-get update -qq && sudo apt-get install -y -qq powerman
  else
    echo "  Passwordless sudo not available. Please run this command in a terminal:"
    echo ""
    echo "    sudo apt-get update && sudo apt-get install -y powerman"
    echo ""
    echo "  Then re-run this script."
    exit 1
  fi
 else
  echo "  powerman client already installed."
 fi
 # --- 2. Verify connectivity ---
 echo ""
 echo "--- Connectivity check ---"
 if timeout 3 bash -c "echo > /dev/tcp/${REMOTE_HOST}/${REMOTE_PORT}" 2>/dev/null; then
  echo "  [OK] ${REMOTE_HOST}:${REMOTE_PORT} reachable"
 else
  echo "  [FAIL] Cannot reach ${REMOTE_HOST}:${REMOTE_PORT}"
  echo "  Is Tailscale up? Is powermand running on ${REMOTE_HOST}?"
  exit 1
 fi
 export POWERMAN_SERVER="${REMOTE_HOST}:${REMOTE_PORT}"
 # --- 3. List outlets ---
 echo ""
 echo "--- Outlets ---"
 powerman -h "${REMOTE_HOST}:${REMOTE_PORT}" -l
 # --- 4. Query status ---
 echo ""
 echo "--- Status ---"
 powerman -h "${REMOTE_HOST}:${REMOTE_PORT}" -q
 echo ""
 echo "============================================"
 echo "  Done."
 echo ""
 echo "  To control an outlet from this workstation:"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -0 outlet-10  # off"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -1 outlet-10  # on"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -c outlet-10  # cycle"
 echo "============================================"
@@ -0,0 +1,181 @@
 #!/usr/bin/bash
 #
 # powerman/setup.sh — idempotent powerman setup for Cyclades PM10i PDU
 #
 # Creates a stable udev symlink for the USB-DB9 adapter, writes powerman.conf
 # with 10 outlet nodes, and enables + starts powermand.
 #
 # This script is designed to be run ON the target host (pfv-tsys1) as root.
 # It is idempotent: safe to run multiple times.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh
 #
 # Override defaults via environment variables:
 #   PDU_SERIAL   — USB adapter serial (default: BJAAb144J07)
 #   PDU_VENDOR   — USB vendor ID       (default: 067b)
 #   PDU_DEV_NAME — udev symlink name   (default: cyclades-pm10)
 #   PDU_BAUD     — serial baud rate    (default: 9600,8n1)
 #   PDU_TYPE     — powerman spec type  (default: pm10)
 #   PDU_OUTLETS  — number of outlets   (default: 10)
 #   PDU_LISTEN   — powermand listen    (default: 0.0.0.0:10101)
 #
 set -euo pipefail
 # --- Config (overridable via env) ---
 PDU_SERIAL="${PDU_SERIAL:-BJAAb144J07}"
 PDU_VENDOR="${PDU_VENDOR:-067b}"
 PDU_DEV_NAME="${PDU_DEV_NAME:-cyclades-pm10}"
 PDU_BAUD="${PDU_BAUD:-9600,8n1}"
 PDU_TYPE="${PDU_TYPE:-pm10}"
 PDU_OUTLETS="${PDU_OUTLETS:-10}"
 PDU_LISTEN="${PDU_LISTEN:-}"  # Auto-detect Tailscale IP if empty
 UDEV_RULE="/etc/udev/rules.d/99-cyclades-pdu.rules"
 POWERMAN_CONF="/etc/powerman/powerman.conf"
 DEV_FILE="/etc/powerman/cyclades-pm10.dev"
 # --- Auto-detect Tailscale IP for listen address ---
 if [ -z "$PDU_LISTEN" ]; then
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -n "$TS_IP" ]; then
    PDU_LISTEN="${TS_IP}:10101"
    echo "  Auto-detected Tailscale IP: $TS_IP"
  else
    PDU_LISTEN="127.0.0.1:10101"
    echo "  WARNING: No Tailscale IP detected. Defaulting to localhost."
  fi
 fi
 echo "============================================"
 echo "  Powerman PDU Setup"
 echo "  Host: $(hostname)"
 echo "  PDU:  Cyclades PM${PDU_OUTLETS}i"
 echo "  Adapter serial: $PDU_SERIAL"
 echo "  Device symlink: /dev/$PDU_DEV_NAME"
 echo "  Listen: $PDU_LISTEN (Tailscale only)"
 echo "============================================"
 # --- 1. Ensure powerman is installed ---
 echo ""
 echo "--- [1/5] Checking powerman installation ---"
 if ! dpkg -l powerman 2>/dev/null | grep -q '^ii'; then
  echo "  Installing powerman from Debian repo..."
  apt-get update -qq && apt-get install -y -qq powerman
 else
  echo "  Powerman already installed: $(dpkg -l powerman | awk '/^ii/{print $3}')"
 fi
 # --- 2. Create udev rule for stable device name ---
 echo ""
 echo "--- [2/5] Creating udev rule for USB-DB9 adapter ---"
 cat > "$UDEV_RULE" <<UDEV
 # Stable symlink for Cyclades PM10i PDU USB-DB9 adapter
 # Generated by powerman/setup.sh
 SUBSYSTEM=="tty", ATTRS{idVendor}=="${PDU_VENDOR}", ATTRS{serial}=="${PDU_SERIAL}", GROUP="dialout", MODE="0660", SYMLINK+="${PDU_DEV_NAME}"
 UDEV
 echo "  Written: $UDEV_RULE"
 # Trigger udev to create the symlink now
 udevadm control --reload-rules 2>/dev/null || true
 udevadm trigger --subsystem-match=tty 2>/dev/null || true
 sleep 1
 if [ -e "/dev/${PDU_DEV_NAME}" ]; then
  echo "  Device symlink active: /dev/${PDU_DEV_NAME} -> $(readlink -f /dev/${PDU_DEV_NAME})"
 else
  echo "  WARNING: /dev/${PDU_DEV_NAME} not found yet. Adapter may be unplugged."
  echo "  Falling back to /dev/ttyUSB* discovery..."
  # Try to find any ttyUSB device as fallback
  for tty in /dev/ttyUSB*; do
    if [ -e "$tty" ]; then
      echo "  Found: $tty (using as fallback)"
      PDU_DEV_NAME="$(basename "$tty")"
      break
    fi
  done
 fi
 # --- 2b. Ensure powermand user can access the serial device ---
 echo ""
 echo "--- [2b/5] Fixing serial device permissions ---"
 if id powerman >/dev/null 2>&1; then
  if id powerman | grep -qv dialout; then
    usermod -aG dialout powerman
    echo "  Added 'powerman' user to 'dialout' group"
  else
    echo "  'powerman' already in 'dialout' group"
  fi
 else
  echo "  (no powerman user — service may run as root)"
 fi
 # --- 3. Write powerman.conf ---
 echo ""
 echo "--- [3/5] Writing powerman.conf ---"
 # Build node definitions
 NODES=""
 for i in $(seq 1 "$PDU_OUTLETS"); do
  NODES+="node \"outlet-${i}\" \"${PDU_DEV_NAME}\" \"${i}\"\n"
 done
 cat > "$POWERMAN_CONF" <<PMCONF
 # Powerman configuration for Cyclades PM${PDU_OUTLETS}i PDU
 # Generated by powerman/setup.sh on $(date)
 # Device: /dev/${PDU_DEV_NAME} (USB-DB9 adapter serial ${PDU_SERIAL})
 # Listen on localhost (for local admin) and Tailscale (for remote access)
 listen "127.0.0.1:10101"
 listen "${PDU_LISTEN}"
 # Device specification for Cyclades PM10
 include "${DEV_FILE}"
 # The PDU device (serial-attached)
 device "${PDU_DEV_NAME}" "${PDU_TYPE}" "/dev/${PDU_DEV_NAME}" "${PDU_BAUD}"
 # Outlet nodes (rename these to match attached devices when onsite)
 $(printf '%b' "$NODES")
 PMCONF
 echo "  Written: $POWERMAN_CONF"
 echo "  Nodes defined: outlet-1 through outlet-${PDU_OUTLETS}"
 # --- 4. Restart powermand ---
 echo ""
 echo "--- [4/5] Restarting powermand ---"
 systemctl enable powerman 2>/dev/null || true
 systemctl restart powerman 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet powerman; then
  echo "  powermand is running."
 else
  echo "  WARNING: powermand failed to start. Check journalctl -u powerman"
  journalctl -u powerman --no-pager -n 20 2>/dev/null || true
 fi
 # --- 5. Verify ---
 echo ""
 echo "--- [5/5] Verification ---"
 echo ""
 echo "  powerman -l (list all outlets):"
 powerman -l 2>&1 || echo "(powerman -l failed)"
 echo ""
 echo "  powerman -q (query status):"
 powerman -q 2>&1 || echo "(powerman -q failed — PDU may need a moment)"
 echo ""
 echo "============================================"
 echo "  Setup complete."
 echo ""
 echo "  Outlet names are generic (outlet-1 ... outlet-${PDU_OUTLETS})."
 echo "  Rename them in ${POWERMAN_CONF} when onsite to match attached devices."
 echo ""
 echo "  Test:    powerman -0 outlet-10   (off)"
 echo "           powerman -1 outlet-10   (on)"
 echo "           powerman -c outlet-10   (cycle)"
 echo "           powerman -q             (status)"
 echo "============================================"
@@ -0,0 +1,33 @@
 #!/usr/bin/bash
 #
 # powerman/status.sh — quick PDU status check
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh
 #
 set -euo pipefail
 echo "============================================"
 echo "  Cyclades PM10i PDU Status"
 echo "  Host: $(hostname)  $(date)"
 echo "============================================"
 echo ""
 echo "=== Service ==="
 systemctl is-active powerman 2>/dev/null && echo "(running)" || echo "(stopped)"
 echo ""
 echo "=== Device ==="
 ls -la /dev/cyclades-pm10 2>/dev/null || echo "(no /dev/cyclades-pm10 symlink)"
 echo ""
 echo "=== Outlets ==="
 powerman -l 2>&1
 echo ""
 echo "=== Power Status ==="
 powerman -q 2>&1
 echo ""
 echo "=== Temperature ==="
 powerman -T 2>&1 || echo "(temperature not available)"
@@ -0,0 +1,126 @@
 #!/usr/bin/bash
 #
 # powerman/test-pdu.sh — validate PDU control by cycling outlet 10 off and on
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh
 #
 # Override: OUTLET=10  (which outlet to test)
 #
 set -euo pipefail
 OUTLET="${OUTLET:-10}"
 NODE="outlet-${OUTLET}"
 PASS=0; FAIL=0
 ok()   { echo "  [PASS] $1"; PASS=$((PASS+1)); }
 fail() { echo "  [FAIL] $1"; FAIL=$((FAIL+1)); }
 echo "============================================"
 echo "  PDU Control Validation"
 echo "  Host: $(hostname)"
 echo "  Test: cycle outlet ${OUTLET} (off → wait → on)"
 echo "============================================"
 # --- 0. Powermand running? ---
 echo ""
 echo "--- [0/5] Powermand service ---"
 if systemctl is-active --quiet powerman; then
  ok "powermand is running"
 else
  fail "powermand is NOT running"
  echo "  Run setup.sh first."
  exit 1
 fi
 # --- 1. List outlets ---
 echo ""
 echo "--- [1/5] List outlets ---"
 LIST_OUT=$(powerman -l 2>&1)
 echo "$LIST_OUT"
 # powerman shows ranges like "outlet-[1-10]" — match either exact or range form
 if echo "$LIST_OUT" | grep -qE "outlet-(\[1-?10\]|${OUTLET}\b)"; then
  ok "Outlet '${NODE}' is defined"
 else
  fail "Outlet '${NODE}' not found in powerman -l"
  exit 1
 fi
 # --- 2. Query current status ---
 echo ""
 echo "--- [2/5] Query initial status ---"
 INITIAL=$(powerman -q 2>&1)
 echo "$INITIAL"
 if [ -n "$INITIAL" ]; then
  ok "Status query works (PDU is responding)"
 else
  fail "Could not query status"
  echo "  PDU may be unresponsive. Check serial connection."
  exit 1
 fi
 # --- 3. Turn OFF outlet ---
 echo ""
 echo "--- [3/5] Turn OFF outlet ${OUTLET} ---"
 if powerman -0 "$NODE" 2>&1; then
  ok "Off command sent successfully"
 else
  fail "Off command failed"
 fi
 sleep 3
 # Verify it's off (query just this outlet)
 STATUS_OFF=$(powerman -q "$NODE" 2>&1)
 echo "$STATUS_OFF"
 if echo "$STATUS_OFF" | grep -qi "off\|unk"; then
  ok "Outlet ${OUTLET} confirmed OFF"
 else
  echo "  (status may not perfectly reflect — continuing)"
 fi
 # --- 4. Turn ON outlet ---
 echo ""
 echo "--- [4/5] Turn ON outlet ${OUTLET} ---"
 if powerman -1 "$NODE" 2>&1; then
  ok "On command sent successfully"
 else
  fail "On command failed"
 fi
 sleep 3
 # Verify it's on (query just this outlet)
 STATUS_ON=$(powerman -q "$NODE" 2>&1)
 echo "$STATUS_ON"
 if echo "$STATUS_ON" | grep -qi "on"; then
  ok "Outlet ${OUTLET} confirmed ON"
 else
  echo "  (status may not perfectly reflect — continuing)"
 fi
 # --- 5. Cycle test (off → delay → on in one command) ---
 echo ""
 echo "--- [5/5] Cycle test (powerman -c) ---"
 if powerman -c "$NODE" 2>&1; then
  ok "Cycle command completed"
 else
  fail "Cycle command failed"
  echo "  (some PDU firmware reports errors during cycle but still works)"
 fi
 sleep 5
 # Final status
 echo ""
 echo "--- Final status ---"
 powerman -q 2>&1
 echo ""
 echo "============================================"
 echo "  Results: $PASS passed, $FAIL failed"
 if [ "$FAIL" -gt 0 ]; then
  echo "  Some checks failed. Review output above."
  exit 1
 fi
 echo "  PDU control validated."
 echo "============================================"
--- a/Show More
+++ b/Show More