docs: unified README and AGENTS.md for merged repo

Replace the KNELServerBuild README with a unified PFVCluster README covering both provisioning and cluster ops. Update AGENTS.md to document the merged repo layout, key scripts, and project context. Consolidate all documentation under docs/. 🤖 Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush <crush@charm.land>
refactor: reorganize merged repo into clean directory structure
2026-07-28 11:25:16 -05:00 · 2026-07-28 11:24:39 -05:00 · 2026-07-28 11:23:22 -05:00 · 2026-07-28 05:16:01 -05:00 · 2026-07-27 22:04:05 -05:00 · 2026-07-27 21:46:48 -05:00
217 changed files with 124458 additions and 101 deletions
@@ -1,8 +1,27 @@
 # Crush internal state
 .crush/
 # OS/editor
 .DS_Store
 *.swp
 *.swo
 *~
 \#*\#
 .\#*
 # Python bytecode
 __pycache__/
 *.pyc
 # Temporary files
 *.tmp
 *.bak
 *.log.tmp
 # LOGFILENAME artifacts: the framework (Logging.sh + PrettyPrint.sh) appends
 # every print_info/print_error line to LOGFILENAME, defined as
 # "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$". Running any script that sources the
-# framework therefore drops a timestamped log file next to it. Ignore these
+# framework therefore drops a timestamped log file next to it.
 # everywhere in the repo.
 *.Monday-*
 *.Tuesday-*
 *.Wednesday-*
@@ -10,4 +29,7 @@
 *.Friday-*
 *.Saturday-*
 *.Sunday-*
 # Sensitive exports / runtime data
 dns-cluster-setup/.export/
 returned-logs/
@@ -1,76 +1,63 @@
 # Agent Guidelines
 This repo combines two formerly-separate projects:
 - **Server provisioning** (formerly KNELServerBuild): `provisioning/`, `tests/`,
  `vendor/`, `dns-cluster-setup/`
 - **Proxmox cluster ops** (formerly PFVCluster/perfopt): `perf/`, `netinfra/`,
  `switches/`, `returned-logs/`
 ## Repository Layout
 Knowing where things live prevents broken edits:
 - **Vendored framework**: `KNELShellFramework` lives at
-  `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`, **not** at
+  `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes
-  the repo root. Its includes are under `Framework-Includes/` there. Never
+  are under `Framework-Includes/` there. Never assume `./Framework-Includes`
-  assume `./Framework-Includes` exists relative to the repo root.
+  exists relative to the repo root.
- **Self-locating scripts**: All provisioning scripts derive their own
+- **Self-locating scripts**: All provisioning scripts derive their own location
-  location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must
+  via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
-  **never** depend on the current working directory or on `cd`/`realpath ..`
+  depend on the current working directory. Run from anywhere.
  chains. Run them from anywhere — `sudo bash SetupNewSystem.sh` works.
 - **Local config files are the source of truth**: Configs in
-  `ProjectCode/ConfigFiles/` are read with `cat`/`cp`. Do **not** re-introduce
+  `provisioning/ConfigFiles/` are read with `cat`/`cp`. Do NOT re-introduce
-  `curl ${DL_ROOT}/...` downloads from `dl.knownelement.com` — that CDN is
+  `curl ${DL_ROOT}/...` downloads — that CDN is deprecated.
-  deprecated for this repo.
+- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
 - **Path variables**: Scripts export `PROJECT_ROOT_PATH`, `CONFIGFILES_PATH`,
  `MODULES_PATH`, `SCRIPTS_PATH`, and `AGENTS_PATH` for locating repo content.
 - **Non-bash agents**: Some files under `ProjectCode/Agents/` carry a `.sh`
  extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
  checkers must skip these.
 - **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
  Datacenter Manager). SSH keys deployed to root on all hosts.
 - **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
  Use the wrapper scripts: `tests/remote.sh`, `dns-cluster-setup/remote-dns.sh`,
  or the `deploy-check.sh` / `deploy-tuning.sh` patterns.
 ## Git Commit Requirements
-When making changes to this repository, ALWAYS:
+1. **Commit atomically**: each logical change its own commit.
-
+2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`,
-1. **Commit atomically**: Each logical change should be its own commit
+   `docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`.
-2. **Use conventional commit format**:
+3. **Verbose messages**: title (50 chars max), blank line, body explaining
-   - `feat(scope): description` - New feature
+   WHAT and WHY, footer with attribution.
   - `fix(scope): description` - Bug fix
   - `docs: description` - Documentation changes
   - `refactor(scope): description` - Code refactoring
   - `test(scope): description` - Test additions/changes
   - `chore: description` - Maintenance tasks
 3. **Write verbose, beautifully formatted messages**:
   - Title line (50 chars max)
   - Blank line
   - Body explaining WHAT and WHY (not how)
   - Reference related files/issues
   - Include footer with attribution
 ## Example Commit
 ```
 feat(security-hardening): implement SCAP-STIG compliance logic
 Refactor apply script to implement comprehensive security hardening:
 - Add GRUB bootloader permission hardening (root:root, mode 0400)
 - Disable and remove autofs service per STIG requirements
 - Deploy modprobe configurations for kernel module blacklisting
 - Create STIG-compliant network protocol blacklist
 This ensures servers meet DoD security requirements for production
 deployment.
 🤖 Generated with [Crush](https://github.com/charmassociates/crush)
 Assisted-by: GLM-5 via Crush <crush@charm.land>
 ```
 ## Autonomous Git Workflow
-**Agents are authorized to commit AND push autonomously. Do not wait to be
+Agents are authorized to commit AND push autonomously. After each logical unit
-asked.** After each logical unit of work:
+of work: stage, commit, push to `origin/main`. Group changes so each commit is
 coherent on its own.
-1. Stage only the files belonging to that logical change.
+## Key scripts
 2. Commit with a conventional, well-formed message (see above).
 3. Push to `origin` (`git push`). The branch tracks `origin/main`.
 4. Repeat per logical unit.
-Group changes so each commit is coherent on its own (a reader should
+| Script | Purpose |
-understand the commit without seeing the others). Never batch unrelated
+|--------|---------|
-changes into one commit.
+| `provisioning/SetupNewSystem.sh` | Full server provisioning (packages, hardening, 2FA) |
 | `tests/vm-validation.sh` | End-to-end deploy + validate on sandbox VM |
 | `tests/run-tests.sh` | Project test suite (unit/security/validation) |
 | `dns-cluster-setup/setup.sh` | Technitium DNS cluster replication |
 | `perf/deploy-check.sh` | Deploy read-only data collector to hosts |
 | `perf/deploy-tuning.sh` | Deploy perf tunings to hosts |
 | `perf/validate-fixes.sh` | Validate applied tuning changes |
 | `perf/iperf-full-matrix.sh` | Full iperf throughput suite |
 ## Project context
 This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
 budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
 (PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
 `docs/PROJECT.md` for the comprehensive fleet report and `docs/TODO.md` for
 pending hardware work.
@@ -1,21 +1,70 @@
-# KNEL FetchApply
+# PFVCluster
-## Repo Issues
+Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
-https://projects.knownelement.com/project/reachableceo-vptechnicaloperations/timeline
+Combines server provisioning, Proxmox cluster operations, and DNS infrastructure.
-## Repo Discussion
+## Directory Structure
 https://community.turnsys.com/c/chieftechnologyandproductofficer/26
 ```
 provisioning/        Server provisioning (SetupNewSystem.sh, security hardening,
                     2FA, NTP/DNS config, SNMP, Dell OMSA)
 tests/               Test suite + VM validation harness
 dns-cluster-setup/   Technitium DNS cluster replication scripts
 perf/                Proxmox performance tuning, fleet audit, iperf, switch diagnostics
 netinfra/            pfv-netinfra-01/02 DNS/NTP setup + audit scripts
 switches/            Switch configuration captures
 docs/                All documentation (PROJECT.md, SECURITY.md, tailscale.md, etc.)
 vendor/              Vendored KNELShellFramework
 ```
-## Repo discription 
+## Quick Start
 Known Element Enterprises (the entity serving as the TSYS Group management company) (through it’s executive leader, the COO) provides core IT/back office systems/services/support on a hands off/fully delegated authortity basis to the CCO and the orgs/members.
-One of those functions is the provisoning of Linux servers. This repository is the Infrastructure As Code (IAC) repository for TSYS. 
+### Provision a new server
 ```bash
 sudo bash provisioning/SetupNewSystem.sh
 ```
 Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh),
 configures NTP/DNS/SNMP/syslog/postfix.
-In the future it will be used via FetchApply https://github.com/P5vc/fetch-apply
+### Validate provisioning on the sandbox VM
 ```bash
 VM_ID=6000 ./tests/vm-validation.sh all
 ```
 Snapshots, deploys, runs the test suite, auto-rolls back on failure.
-## Usage
+### Run the test suite
 ```bash
 ./tests/run-tests.sh all
 ```
-git clone this repo
+### Deploy DNS cluster setup
-cd FetchApply/ProjectCode
+```bash
-bash SetupNewSystem.sh
+cd dns-cluster-setup/
 ./setup.sh all
 ```
 ### Deploy perf tunings to hosts
 ```bash
 cd perf/
 ./deploy-check.sh    # read-only data collection
 ./deploy-tuning.sh   # apply sysctl/tuned/NFS tunings
 ```
 ## Key Documentation
 | Doc | Contents |
 |-----|----------|
 | `docs/PROJECT.md` | Comprehensive fleet report (7 hosts, VM inventory, storage) |
 | `docs/SECURITY.md` | Security architecture and hardening details |
 | `docs/tailscale.md` | Tailscale vs managed DNS analysis |
 | `docs/DEPLOYMENT.md` | Deployment procedures |
 | `docs/TODO.md` | Pending hardware work (tsys2/4/5) |
 | `dns-cluster-setup/README.md` | DNS cluster setup guide |
 ## Architecture
 - **Proxmox hosts**: 7 standalone PVE installs managed via PDM
 - **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
 - **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs)
 - **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
 - **Backups**: Proxmox Backup Server (PBS)
@@ -8,7 +8,7 @@
 #
 # WHY: one place to configure host aliases/users/keys, one place to audit,
 # and the command scanner only permits ssh when invoked indirectly via a
-# script. Mirrors the pattern of Project-Tests/remote.sh.
+# script. Mirrors the pattern of tests/remote.sh.
 #
 # HOSTS (override IPs via env if needed):
 #   tsrouter       tailscale-router.knel.net  (PRODUCTION — READ-ONLY here)
@@ -0,0 +1,485 @@
 # K8S.md -- Kubernetes Architecture Deep-Dive
 **Date:** 2026-07-27
 **Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
 Companion to `PROJECT.md` (which has the fleet-wide assessment).
 **Status:** For discussion in a future session. No changes made.
 ---
 ## Table of Contents
 1. [Workload Profile](#1-workload-profile)
 2. [Current State](#2-current-state)
 3. [Target Architecture](#3-target-architecture)
 4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
 5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
 6. [Storage Class Design](#6-storage-class-design)
 7. [ETL/HPC Considerations](#7-etlhpc-considerations)
 8. [Migration Plan](#8-migration-plan)
 ---
 ## 1. Workload Profile
 This cluster runs **R&D and RackRental (containerlab) workloads** via
 Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
 Reston, VA running Cloudron.
 **Workload types expected:**
 | Type | Description | Storage need | RAM need | Examples |
 |------|------------|-------------|----------|---------|
 | **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
 | **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
 | **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
 **Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
 sequential I/O). HPC and containerlab need low-latency random I/O (local
 SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
 ---
 ## 2. Current State
 ### 2.1 pfv-k8s nodes and their storage
 | Node | Type | Host | Storage | Disk type | Status |
 |------|------|------|---------|-----------|--------|
 | cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
 | cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
 | cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
 | cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
 | cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
 | wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
 | wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
 | wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
 ### 2.2 Problems
 1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
   etcd quorum lost.
 2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
 3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
 4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
   (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
 5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
 6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
 ---
 ## 3. Target Architecture
 ### 3.1 Design principles
 1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
   heavy workers.
 2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
   storage server failing.
 3. **Wnode boot disks on local storage** -- eliminates NFS latency for
   container runtime and kubelet.
 4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
 5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
 6. **One wnode per hypervisor host** -- maximize total cluster capacity.
 ### 3.2 Target node-host-storage matrix
 | Node | Type | Host | Boot disk | Data disk | Disk type |
 |------|------|------|-----------|-----------|-----------|
 | cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
 | cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
 | cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
 | cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
 | cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
 | wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
 | wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
 | wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
 | wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
 | wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
 | wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
 | wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
 ### 3.3 Storage server distribution after changes
 | Storage server | cnodes | wnodes (boot) | wnodes (data) |
 |---------------|--------|---------------|---------------|
 | tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
 | tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
 | tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
 | tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
 **Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
 NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
 than NFS-HDD, and the user has chosen not to install internal drives.
 **No single disk or server is a quorum-losing failure point.**
 ---
 ## 4. Control Plane (Cnodes)
 ### 4.1 Cnode sizing
 Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
 kubernetes control plane components (API server, scheduler, controller-manager).
 ### 4.2 Cnode host placement rationale
 | Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
 |------|--------|---------------|---------------|------------------------|
 | tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
 | tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
 | tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
 **tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
 ### 4.3 Cnode storage placement rationale
 The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
 | Disk | cnodes | Rationale |
 |------|--------|-----------|
 | D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
 | D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
 | S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
 | S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
 **If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
 **If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
 **If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
 tsys5. **Only 2 of 5 -- QUORUM LOST.**
 Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
 cnodes and only have 2 on tsys5. That loses quorum (need 3).
 **Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
 3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
 tsys4. Same problem inverted.
 The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
 3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
 holding 2 cnodes is safe (3 survive).
 **Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
 other.** After Friday's hardware work:
 - tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
 - tsys5 will have bond0 fixed + NVMe -- more reliable
 Either way, 3/2 split means one server failure could lose quorum. **To truly
 solve this, use a 3rd storage target.** Options:
 - Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
  independent failure domain)
 - Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
  no NFS dependency at all)
 **Best option: put cnode boot disks on local-lvm where available.** This
 eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
 its own host's local disk -- no shared dependency.
 | cnode | Host | **Recommended storage** | Type |
 |-------|------|------------------------|------|
 | cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
 | cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
 | cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
 | cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
 | cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
 With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
 down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
 survives.
 **This is the recommended approach.** Local storage for cnodes wherever
 possible. NFS only as fallback.
 ### 4.4 etcd performance on local vs NFS
 | Storage | Typical fsync latency | etcd commit latency | Impact |
 |---------|----------------------|--------------------|--------|
 | NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
 | NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
 | Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
 | Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
 | NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
 etcd is the heartbeat of the kubernetes control plane. Every API call, every
 pod schedule, every controller reconciliation involves an etcd write. Cutting
 etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
 responsive. **This is the single highest-impact change for k8s performance.**
 ---
 ## 5. Worker Nodes (Wnodes)
 ### 5.1 One wnode per hypervisor host
 | Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
 |------|-------|-----------|-----------|-----------|-----------|------|
 | tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
 | tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
 | tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
 | tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
 | tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
 | tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
 | tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
 ### 5.2 Why boot disks on local-lvm
 Current: all wnodes boot from NFS. Every container image pull, every kubelet
 log write, every ephemeral volume traverses the NFS network path.
 With local-lvm boot disks:
 - **Container image pulls** write to local disk (100-150 MB/s HDD, no network
  hop) instead of NFS-HDD (80-120 MB/s with network latency)
 - **kubelet logs** stay local (no NFS writes for log rotation)
 - **ephemeral storage** (emptyDir volumes) uses local disk by default
 - **NFS server failure does not kill the wnode** -- the VM stays running, only
  the data disk (if mounted) goes away
 ### 5.3 Wnode sizing guidance
 | Host | Recommended wnode config | Rationale |
 |------|------------------------|-----------|
 | tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
 | tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
 | tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
 | tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
 | tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
 | tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
 | tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
 ### 5.4 Tainting and labeling strategy
 Label wnodes by capability so the k8s scheduler can target them:
 ```yaml
 # Heavy RAM hosts (ETL/HPC)
 wnode-tsys6: workload=heavy, ram=128g
 wnode-tsys7: workload=heavy, ram=192g
 # NVMe host (I/O-intensive HPC)
 wnode-tsys5: workload=hpc, storage=nvme
 # SSD boot host (low-latency)
 wnode-tsys9: workload=light, storage=ssd
 # General workers
 wnode-tsys3: workload=general
 wnode-tsys2: workload=storage-fast, storage=nvme
 wnode-tsys1: workload=light
 ```
 Then use nodeSelector or nodeAffinity in job specs:
 ```yaml
 # Weather/GIS ETL job -- needs lots of RAM
 spec:
  nodeSelector:
    workload: heavy
 # Firmware build -- needs fast storage
 spec:
  nodeSelector:
    storage: nvme
 ```
 ---
 ## 6. Storage Class Design
 ### 6.1 Proposed StorageClasses
 | StorageClass | Provisioner | Where | Speed | Use case |
 |-------------|------------|-------|-------|----------|
 | `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
 | `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
 | `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
 ### 6.2 How this maps to wnode disk topology
 Each wnode has:
 - **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
  runtime. k8s `local-fast` StorageClass provisioner points here.
 - **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
  second block device or filesystem. k8s `nfs-hdd` provisioner points here.
 Inside k8s, pods request storage via PVC:
 ```yaml
 # ETL job: needs bulk storage for weather data
 apiVersion: v1
 kind: PersistentVolumeClaim
 spec:
  storageClassName: nfs-hdd
  accessModes: [ReadWriteMany]  # NFS allows RWX
  resources:
    requests:
      storage: 500Gi
 # HPC job: needs fast scratch
 spec:
  storageClassName: local-fast
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
 ```
 ### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
 **Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
 scratch space, with the exception of ultix-streaming which stays on T5-SSD.
 Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
 SIEM, etc.).
 The SSD NFS exports:
 - **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
  cache, ephemeral volumes). Currently 99% empty.
 - **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
  occupies 83 GB. Remaining 140 GB available for k8s use.
 **tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
 This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
 tsys5 SSD exports, bulk PVCs target either server.
 ### 6.4 NFS data distribution across storage servers
 To avoid re-creating the "everything on tsys4" problem, distribute NFS data
 disks across both servers:
 | wnode | Boot (local) | Bulk data (NFS) | NFS server |
 |-------|-------------|-----------------|------------|
 | wnode-tsys3 | local-lvm | S3 | tsys5 |
 | wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
 | wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
 | wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
 | wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
 This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
 ---
 ## 7. ETL/HPC Considerations
 ### 7.1 Weather/GIS ETL pipeline
 Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
 store results.
 | Stage | Storage class | Why |
 |-------|-------------|-----|
 | Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
 | Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
 | Store results | `nfs-hdd` | Large sequential writes. Persistent. |
 **Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
 wnode. ETL jobs use `/scratch` for intermediate processing and write final
 output to the NFS-mounted `/data`.
 ### 7.2 HPC workloads (hardware startup)
 Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
 | Workload | Best wnode | Why |
 |----------|-----------|-----|
 | RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
 | Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
 | Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
 | Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
 ### 7.3 RackRental/containerlab
 Rapid container deployment. Key need: fast container image pulls.
 This is where **local-lvm boot disks** shine. Currently, every container image
 pull writes through NFS to a spinning disk -- slow. With local-lvm, images
 cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
 On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
 ### 7.4 Data locality for ETL
 For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
 it on local-lvm of the heavy hosts:
 ```
 tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/  -- fastest cache tier
 tsys5 NVMe (local): /data/cache/gis/                 -- fastest cache tier
 ```
 **Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
 be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
 This avoids re-reading the same data from the same NFS export on every job
 if the data is already cached in the page cache.
 ---
 ## 8. Migration Plan
 **Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
 Datacenter Manager (PDM)** manages them collectively and supports VM migration
 between nodes. Storage migration can be done via the PDM/Proxmox UI rather
 than manual disk copies -- the destination node just needs access to the target
 storage (which all nodes have for NFS exports, and local storage can be
 migrated through the UI's "Storage Migrate" function).
 ### 8.1 Phase 1: Friday (after hardware work)
 After tsys5 cable + NVMe and tsys4 NIC + RAM:
 1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
 2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
   do not use for VM storage. Recreate on D2 or D5 NFS.
 3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
 ### 8.2 Phase 2: Cnode rebalance (maintenance window)
 These changes require creating new VMs on target hosts and migrating disks.
 Plan for a maintenance window with the k8s cluster briefly down.
 1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
 2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
 3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
 4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
 5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
 ### 8.3 Phase 3: Wnode local storage migration (maintenance window)
 1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
 2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
 3. wnode-tsys7 stays on NFS (same reason).
 4. Recreate wnode-tsys5 with boot disk on NVMe.
 5. Add data disks (NFS) as second SCSI devices where applicable.
 ### 8.4 Phase 4: tsys2 integration (when rebuilt)
 1. Install Proxmox on tsys2.
 2. Run `scripts/check.sh` to inventory.
 3. Run `scripts/apply-tunings.sh --apply`.
 4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
 5. Join to k8s cluster.
 ### 8.5 Phase 5: Critical VM relocation
 1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
 2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
 3. (No change to T5-SSD -- ultix-streaming stays.)
 ---
 ## Open questions for next session
 1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
   whether live migration is available (huge simplification) or we need manual
   disk migration. Check `pvecm status` on each host.
 2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
   nodes are joined/drain and how StorageClasses are configured.
 3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
 4. **Is there a container image registry mirror in the cluster?** Or do all
   pulls go to Docker Hub / external? A local registry on D3 SSD would speed
   up all pulls.
 5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
   whether jobs need shared (RWX) or exclusive (RWO) storage.
 6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
   how we label and taint nodes.
@@ -0,0 +1,857 @@
 # Proxmox Cluster Project Report
 **Date:** 2026-07-27 (re-audited)
 **Prepared by:** Performance Optimization Engagement
 **Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
 **Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
 `deploy-check.sh`. VM placements reflect live state after user's PDM
 migrations. This is ground truth.
 ---
 ## Table of Contents
 1. [Executive Summary](#1-executive-summary)
 2. [Host Fleet](#2-host-fleet)
 3. [Storage Architecture](#3-storage-architecture)
 4. [VM Fleet Inventory](#4-vm-fleet-inventory)
 5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
 6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
 7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
 8. [Role Alignment Audit](#8-role-alignment-audit)
 9. [Network Findings](#9-network-findings)
 10. [Recommendations](#10-recommendations)
 11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
 12. [Open Items](#12-open-items)
 ---
 ## 1. Executive Summary
 The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
 running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
 performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
 are blocked on physical hardware work scheduled for Friday.
 **Progress since initial audit:** The user has been actively rebalancing k8s
 nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
 73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
 needed for etcd quorum survival.
 The VM-layer assessment reveals:
 | # | Finding | Severity | Status |
 |---|---------|----------|--------|
 | 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
 | 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
 | 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
 | 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
 ---
 ## 2. Host Fleet
 ### 2.1 Inventory
 | Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
 |------|-------|-----------|-------|-----|-----------|-----------------|--------|
 | pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
 | pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
 | pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
 | pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
 | pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
 | pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
 ### 2.2 Role taxonomy (per user directive)
 | Role | Hosts | Workload |
 |------|-------|----------|
 | **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
 | **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
 | **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
 **Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
 RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
 (tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
 run one per hypervisor host across the fleet to maximize total cluster capacity.
 pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
 Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
 running Cloudron -- not in this cluster.
 ---
 ## 3. Storage Architecture
 ### 3.1 NFS exports from tsys4 (primary storage server)
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
 | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
 | ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
 | D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
 Non-exported disks on tsys4:
 - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
 - sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
 - sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
 **D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
 tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
 eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 /sdc/sdd), so tsys5 is the better target. See section 3.2.
 ### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
 | S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
 | S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
 | S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
 | S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
 | T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
 | **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
 **tsys5 storage controllers (plenty of free ports):**
 - LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
  **5 free**
 - Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
 - Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
 - 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
 **Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
 wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
 (unused, 7 GB). S4 still 99% empty (435 GB free).
 T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
 **Friday additions:**
 1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
   USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
 2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
   boot disk and HPC scratch. The fastest tier in the fleet.
 After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
 NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
 natural home for latency-sensitive workloads and the k8s StorageClass design
 center.
 ### 3.3 Local storage tiers (per host, with utilization)
 | Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
 |------|-----------|-----------|-----|-------|------|------|-------------|
 | tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
 | **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
 | tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
 | tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
 | tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
 | **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
 | **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
 | **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
 | **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
 | **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
 **CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
 The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
 single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
 260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
 (USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
 **USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
 NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
 hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
 for VM storage.
 Additionally, both servers have completely empty internal drive bays:
 - **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
 drives.
 - **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
 DVD-ROM on port 5). No internal drives.
 This is a reliability risk beyond performance: the entire host OS boots
 from a consumer-grade portable USB drive not designed for 24/7 server use.
 **tsys3 correction:** Previously documented as HDD. Actually boots from a
 **Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
 fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
 **Critical observation: every k8s host has 0% used local-lvm.** tsys3
 (Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
 1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
 storage.
 **However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
 tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
 NFS-over-HDD and unsuitable for wnode boot disks.
 ### 3.4 Disk speed tiers summary
 | Tier | Where | Speed class | Best for |
 |------|-------|------------|----------|
 | **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
 | **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
 | **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
 | **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
 | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
 | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
 ### 3.5 Storage tier characterization per host
 | Host | Storage profile | Detail |
 |------|----------------|--------|
 | **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
 | **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
 | **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
 | **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
 | **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
 | **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
 | **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
 ### 3.6 All exports are single-disk with no redundancy
 Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
 mirror, no mdraid. A single disk failure takes down every VM whose disk lives
 on that export. This applies to **both storage servers** and to the **PBS
 backup target** (a single 4.5T SMR drive).
 ---
 ## 4. VM Fleet Inventory
 ### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
 #### tsys1 (Infrastructure) -- 11 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
 | 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
 | 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys3 (Kubernetes) -- 1 running VM
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
 #### tsys4 (Storage) -- 1 running VM
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
 #### tsys5 (Storage) -- 15 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
 | 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
 | 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
 | 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
 | 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
 | 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
 | 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
 | 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
 #### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
 | 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
 | 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys7 (Kubernetes) -- 4 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
 | 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
 #### tsys9 (Infrastructure) -- 5 running VMs
 | VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
 |------|------|-------|----------|------|---------|------|
 | 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
 | 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
 | 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
 | 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
 | 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
 ### 4.2 Storage concentration summary
 | Storage target | # of running VMs | % of fleet |
 |---------------|-----------------|------------|
 | **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
 | D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
 | local-nonprod (tsys5 local HDD) | 10 | 25% |
 | S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
 | T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
 | local-lvm (tsys4 local) | 1 | 3% |
 **27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
 If tsys4 goes offline, two-thirds of the fleet loses its storage.
 ---
 ## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
 ### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
 | VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
 |------|------|------------|---------|-----------|----------|
 | 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
 | 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
 | 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
 | 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
 | 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
 **Storage distribution:**
 | Storage server | cnodes | Quorum impact if it fails |
 |---------------|--------|--------------------------|
 | tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
 | tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
 **Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
 still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
 **Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
 tsys9) -- good host diversity.
 ### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
 | VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
 |------|------|------------|---------|-----------|-----|--------|----------|
 | 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
 | 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
 | 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
 | 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
 | 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
 | 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
 **One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
 stopped. wnode-tsys6 restarted.
 **Storage distribution:**
 | Storage server | wnodes | Notes |
 |---------------|--------|-------|
 | tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
 | tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
 ### 5.3 Summary: k8s node storage distribution
 | Storage server | cnodes | wnodes | Total k8s nodes |
 |---------------|--------|--------|-----------------|
 | **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
 | **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
 | Local SSD/NVMe | 0 | 0 | 0 (0%) |
 **Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
 cnodes on each storage server (60/40 or better) so either server failing
 leaves quorum intact.
 ### 5.4 Remaining cnode migration needed for etcd quorum survival
 To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
 be on tsys5:
 | Action | Effect |
 |--------|--------|
 | Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
 | Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
 | Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
 **Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
 PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
 ### 5.5 Future k8s architecture (next session -- see K8S.md)
 The k8s layer will be tackled soon. Key requirements from user:
 - **Platform:** vcluster + Rancher for multi-tenant management
 - **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
 - **Workload isolation (vcluster per tenant):**
  - RackRental workloads (containerlab network labs)
  - Suborbital ITAR (compliance-restricted)
  - Suborbital non-ITAR
  - Starting Line Productions customer workloads
 - **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
  will be brought online during peak solar production for burst capacity.
  These older cores/ram supplement the main fleet when power is abundant.
 - **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
  both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
  leftover host capacity) to large (28-32 GB, consuming most of a host).
 - **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
  full final audit will be performed at that time (post-NVMe install on tsys5,
  post-D3 SSD relocation, post-tsys4 NIC+RAM).
 ### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
 tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
 | VMID | Name | Storage |
 |------|------|---------|
 | 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
 | 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
 These are on local storage (good -- no NFS dependency) but on a single host's
 single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
 ---
 ## 6. Storage Redundancy Analysis
 ### 6.1 -01/-02 infrastructure pair audit
 Two -01/-02 pairs exist in the fleet:
 **Pair 1: pfv-netinfra (network infrastructure)**
 | Role | VMID | Host | Storage | NFS Server |
 |------|------|------|---------|-----------|
 | -01 | 103 | tsys1 | D5 | **tsys4** |
 | -02 | 904 | tsys9 | D2 | **tsys4** |
 **Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
 Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
 disks. The -02 half should be on an S2/S3/S4 export from tsys5.
 **Pair 2: tsys-ucs (Univention Corporate Server)**
 | Role | VMID | Host | Storage | NFS Server |
 |------|------|------|---------|-----------|
 | -01 | 108 | tsys1 | D2 | **tsys4** |
 | -02 | 902 | tsys9 | D5 | **tsys4** |
 **Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
 Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
 ### 6.2 Redundancy principle for paired VMs
 For any -01/-02 pair to survive a single storage server failure:
 ```
 -01 VM disk -> tsys4 NFS export (D2/D3/D5)
 -02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
 ```
 This ensures that losing either tsys4 or tsys5 takes down only one half of
 the pair. Currently, **both pairs fail this test** because both halves are on
 tsys4.
 ### 6.3 NFS server failure blast radius
 If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
 | Impact | Count |
 |--------|-------|
 | k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
 | k8s wnodes that lose storage | 4 of 5 |
 | Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
 | Total VMs that lose storage | **27 of 40 (68%)** |
 If **tsys5** goes offline:
 | Impact | Count |
 |--------|-------|
 | k8s cnodes that lose storage | 0 of 5 |
 | k8s wnodes that lose storage | 1 of 5 |
 | Total VMs that lose storage | 1 of 40 (3%) |
 **tsys4 is a massive blast-radius liability. tsys5 is barely used.**
 Rebalancing VM storage across both servers dramatically reduces risk.
 ---
 ## 7. Local SSD/NVMe Opportunity
 ### 7.1 Available fast tiers (currently unused by k8s)
 | Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
 |------|--------|------|------|--------------------|-----------------------|
 | tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
 | tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
 | tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
 | tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
 ### 7.2 Why local storage matters for k8s nodes
 Kubernetes nodes are latency-sensitive in two specific areas:
 1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
   On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
   client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
   latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
   This directly affects k8s API responsiveness and pod scheduling speed.
 2. **Container image pulls:** Worker nodes pull container images frequently.
   On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
   eliminates the network hop and reduces seek time. This matters most for
   RackRental/containerlab workloads that spin up containers rapidly.
 ### 7.3 Current waste: tsys9 local SSD
 tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
 **Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
 idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
 its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
 ### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
 The PCI NVMe being added to tsys5 will be the fastest storage tier in the
 fleet. Two placement options:
 **Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
 VMs that might need migration. Adds the NFS/network overhead back.
 **Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
 network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
 accessed from other hosts.
 **Recommendation:** Option B (local-only). k8s worker nodes do not need
 shared storage -- pods are ephemeral and reschedule on failure. The NVMe
 should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
 and used for local VM images.
 ---
 ## 8. Role Alignment Audit
 Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
 plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
 **correct by design** (keeps heavy hosts free for workers).
 ### 8.1 VMs that need to move
 | VMID | Name | Current host | Issue | Target |
 |------|------|-------------|-------|--------|
 | 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
 | 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
 ### 8.2 Host capacity for k8s nodes
 | Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
 |------|------|-------------------|-------------|-----------|----------|
 | tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
 | tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
 | tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
 | tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
 | tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
 | tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
 **tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
 free RAM respectively. They should be the primary targets for heavy worker
 nodes and ETL/HPC workloads.
 ### 8.3 tsys6 wnode-tsys6 is stopped
 VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
 NFS). This wnode should be restarted (or recreated on local-lvm) to restore
 cluster capacity.
 ---
 ## 9. Network Findings
 ### 9.1 tsys9 storage NIC is a USB dongle (new finding)
 Validating tsys9 revealed that its storage network interface
 (`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
 anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
 management; storage uses the USB adapter.
 **Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
 wobble, ESD, and USB controller resets. For an infrastructure host with 5
 VMs, this is a reliability risk.
 **Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
 PCIe NIC would eliminate this risk (same recommendation as tsys4).
 ### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
 | Host | Blocker | Staged fix |
 |------|---------|-----------|
 | tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
 | tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
 ### 9.3 LACP resolved on tsys6/tsys7
 tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
 The switch LACP hash change took effect after renegotiation. The 56-106K
 retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
 LACP overhead. See `RESULTS.md` for the full analysis.
 ### 9.4 NFS nconnect=4 + noatime confirmed active
 All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
 options. Each host maintains 4 TCP connections per NFS mount to each storage
 server. This was the Tier 0 tuning item from the performance optimization
 engagement and is confirmed working cluster-wide.
 ---
 ## 10. Recommendations
 **No changes have been made. These are assessment-only recommendations.**
 See `K8S.md` for the detailed kubernetes architecture deep-dive.
 ### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
 These are the most critical production VMs in the fleet. They must survive
 any single-point failure (host, storage server, or disk).
 **Design principle for -01/-02 HA pairs:**
 - Different hypervisors (already satisfied: tsys1 vs tsys9)
 - Different storage servers (currently FAILED: all on tsys4)
 - Prefer SSD for latency-sensitive services
 **Recommended placement:**
 | VM | Host | Storage | Tier | Free space | Rationale |
 |----|------|---------|------|-----------|-----------|
 | **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
 | **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
 | **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
 | **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
 **Failure survival matrix (all single-point failures):**
 | Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
 |---------|-------------|-------------|--------|--------|--------|
 | tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
 | tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
 | tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
 | tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
 | Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
 **Why this works:** Every row has at least one netinfra and one UCS alive.
 The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
 **D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
 Friday) is freed for latency-sensitive workloads that actually benefit from
 SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
 registry mirror. Not infrastructure VMs.
 ### 10.2 CRITICAL: Cnode (control plane) storage split
 **Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
 quorum (3 of 5 cnodes share D2).
 **Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
 tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
 | cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
 |-------|-------------|----------------|----------------|-------------------|-----------|
 | cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
 | cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
 | cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
 | cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
 | cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
 **Result after changes:**
 | Host | cnodes | Storage server |
 |------|--------|---------------|
 | tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
 | tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
 | tsys3 | cnode5 (S3) | tsys5 |
 - 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
  etcd keeps quorum (3 of 5 survive).
 - D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
  holds more than 2 cnodes.
 - Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
  at least 3 cnodes alive.
 - **tsys6 and tsys7 are fully freed** for heavy worker nodes.
 Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
 the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
 cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
 ### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
 **Target: one wnode per hypervisor host, using local storage where possible
 and NFS-HDD for bulk data.**
 | wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
 |-------|------|--------------------------|---------------------|-----------|
 | wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
 | wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
 | wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
 | wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
 | wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
 | wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
 | wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
 **Storage tiering strategy per wnode:**
 Each wnode gets two disk tiers mapped to k8s StorageClasses:
 1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
   images, ephemeral storage. This is where local SSD/NVMe shines -- container
   image pulls and layer extraction are seek-bound and benefit enormously from
   low-latency storage.
 2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
   areas, large files that do not fit on local storage. Mounted as a second
   disk in the VM and exposed to k8s as a StorageClass.
 This maps to two k8s StorageClasses:
 - `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
 - `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
 ### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
 **Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
 space and ultix-streaming (developer workstation running "cluster of 1"
 pre-production jobs). Spinning rust hosts all other infrastructure VMs**
 (UCS, netinfra, LibreNMS, SIEM, etc.).
 SSD allocation after Friday:
 - **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
  the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
 - **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
  (developer workstation, runs single-node test jobs before k8s). Remaining
  140 GB available for k8s.
 **Deep-dive on exact k8s scratch allocation is deferred to the next session**
 (K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
 ### 10.5 MEDIUM: Restart wnode-tsys6
 VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
 (1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
 the boot disk to local storage simultaneously.
 ### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
 **Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
 directory storage.**
 Use for:
 - wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
 - sectestbed VMs that need fast scratch space
 - Not NFS-exported (avoid adding network overhead to the fastest tier)
 ### 10.7 LOW: Add PCIe NIC to tsys9
 tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
 slots. A $150 PCIe NIC eliminates the USB reliability risk.
 ### 10.8 LOW: Standardize PVE/kernel versions
 tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
 (9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
 ---
 ## 11. Hardware End-of-Support Exposure
 | Host | EOS date | Years past | Form factor |
 |------|----------|-----------|-------------|
 | pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
 | pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
 | pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
 | pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
 | pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
 | pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
 | pfv-tsys2 (5520) | TBD | -- | Laptop |
 | **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
 **6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
 The two actual rack servers (R610, R620) are the most overdue for replacement.
 This belongs in the next budget cycle.
 ---
 ## 12. Open Items
 ### 12.1 Immediate (do today via PDM)
 1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
   redundancy. UCS stays on HDD.
 2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
 3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
 4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
 5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
 ### 12.2 Friday maintenance window (user action)
 1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
   reconfigure `/etc/network/interfaces`, reboot.
 2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
   install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
   format NVMe as local storage, reboot.
 3. **tsys2:** Load Proxmox (replacing Windows 10).
 4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
 ### 12.3 Post-Friday validation
 1. Re-run iperf matrix: `./iperf-full-matrix.sh`
 2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
 3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
 4. Update PROJECT.md with post-hardware numbers.
 ### 12.4 Future: Kubernetes deep-dive (see K8S.md)
 Next major workstream. Requirements captured:
 - **vcluster + Rancher** for multi-tenant k8s management
 - **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
 - **Workload isolation** via separate vclusters:
  - RackRental (containerlab)
  - Suborbital ITAR
  - Suborbital non-ITAR
  - Starting Line Productions customer workloads
 - **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
  during peak solar production for burst capacity
 - **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
 - **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
 - **Spinning rust for all infrastructure VMs**
 ### 12.5 Data gaps
 | Gap | How to close |
 |-----|--------------|
 | tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
 | tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
 | tsys5 NVMe size and model | Confirm after Friday installation |
 | tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
 | PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
 ### 12.6 Scripts and tools available
 | Script | Purpose |
 |--------|---------|
 | `scripts/check.sh` | Read-only data collector (run on hosts) |
 | `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
 | `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
 | `validate-fixes.sh` | Read-only validation of all applied changes |
 | `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
 | `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
@@ -0,0 +1,242 @@
 # TODO.md — Pending User Actions
 **Date:** 2026-07-27
 **Items needing user input or physical action.**
 ---
 ## 1. tsys2 Windows hardware inventory (run on the Windows host)
 pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
 hardware data so the architecture plan can account for it.
 ### Option A: PowerShell (recommended — single command, copy-paste output)
 Open **PowerShell as Administrator** and run:
 ```powershell
 # Full hardware inventory in one shot
 Write-Output "=== COMPUTER ==="
 Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
 Write-Output "`n=== CPU ==="
 Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
 Write-Output "`n=== MEMORY STICKS ==="
 Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
 Write-Output "`n=== DISKS ==="
 Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
 Write-Output "`n=== DISK PARTITIONS ==="
 Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
 Write-Output "`n=== NETWORK ADAPTERS ==="
 Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
 Write-Output "`n=== GPU(s) ==="
 Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
 Write-Output "`n=== PCIe SLOTS ==="
 Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
 Write-Output "`n=== USB DEVICES (storage + network only) ==="
 Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
 ```
 Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
 directly into the chat.
 ### Option B: Command Prompt (cmd.exe) fallbacks
 If PowerShell is unavailable for some reason, these cmd commands give a
 subset:
 ```cmd
 :: Computer model and serial
 wmic computersystem get manufacturer,model
 wmic bios get serialnumber
 :: CPU
 wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
 :: RAM (total)
 wmic computersystem get totalphysicalmemory
 :: RAM sticks (per-slot detail)
 wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
 :: Disks
 wmic diskdrive get model,size,interfacetype,mediatype
 :: Network adapters
 wmic nic where netenabled=true get name,speed,macaddress
 :: GPU
 wmic path win32_videocontroller get name,adapterram,driverversion
 ```
 ### What I'm looking for
 - **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
  whether wnode-tsys2 can use local storage like the other wnodes.)
 - **Network adapters**: How many onboard NICs? Model? (Determines whether
  tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
  onboard NIC available.)
 - **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
  memory bandwidth for HPC workloads — dual-channel matters.)
 - **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
  planning.
 - **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
 - **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
 ### STATUS: Collected 2026-07-27
 **Disk inventory — RESOLVED:**
 - Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
 - Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
 - Both SSDs, no spinning rust. 1.5 TB total local SSD.
 **Network adapters — RESOLVED (concerning):**
 - StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
 - Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
 - Wi-Fi: Intel 8265 (disconnected)
 - **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
  Unavoidable on this laptop form factor — no onboard PCIe NIC available.
 **GPU — RESOLVED:**
 - Intel HD Graphics 630 (integrated, 1 GB)
 - NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
 **PCIe slots — RESOLVED:**
 - Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
  user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
  by 960 PRO.
 **RAM — PARTIAL:**
 - Total 32 GB confirmed (34,097,573,888 bytes).
 - Per-stick detail failed to run (PowerShell line-break split
  `Format-T` + `able`). Re-run the command below if bandwidth planning
  needs stick-level detail:
  ```powershell
  Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
  ```
 **Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
 ---
 ## 2. Friday maintenance window (physical hardware)
 ### tsys4 — install PCIe NIC + add RAM
 1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
 2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
 3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
 4. Power on, then update `/etc/network/interfaces` to replace
   `enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
 5. Reboot to activate new NIC and NFS nconnect.
 6. Run `validate-fixes.sh pfv-tsys4` to confirm.
 ### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
 1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
 2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
   "Number of ports: 2" and a real partner MAC (not all zeros).
 3. Apply bond hash fix (same as tsys6/7):
   ```bash
   echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
   ```
 4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
   - Power down tsys4
   - Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
   - Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
   - On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
   - Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
     `pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
   - Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
     empty, so minimal migration)
 5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
 6. Format NVMe as local directory storage (see TODO section 3 below).
 7. Reboot tsys5 to activate NFS nconnect.
 8. Run `validate-fixes.sh pfv-tsys5` to confirm.
 ---
 ## 3. tsys5 NVMe format/mount decision (after Friday install)
 **Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
 storage so it shows up as a VM image target in the Proxmox UI.
 After the NVMe is physically installed and visible in Proxmox:
 1. Identify the device: `lsblk` or `ls /dev/nvme*`
 2. Format and add to Proxmox:
   ```bash
   # Option A: LVM-thin (thin provisioning, snapshots)
   pvcreate /dev/nvme0n1
   vgcreate nvme-pool /dev/nvme0n1
   lvcreate -l 100%FREE -T nvme-pool/data
   # Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
   #   ID: nvme-local
   #   Volume Group: nvme-pool
   #   Thin Pool: data
   #   Content: Disk image, Container template
   # Option B: Directory (simpler, no thin provisioning)
   mkfs.ext4 /dev/nvme0n1
   mkdir -p /mnt/nvme
   mount /dev/nvme0n1 /mnt/nvme
   # Add to /etc/fstab for persistence
   # Then in Proxmox UI: Datacenter > Storage > Add > Directory
   #   ID: nvme-local
   #   Directory: /mnt/nvme
   #   Content: Disk image, Container template
   ```
 3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
 ---
 ## 4. Post-hardware validation (run after Friday work)
 1. Re-run iperf matrix: `./iperf-full-matrix.sh`
 2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
 3. Update PROJECT.md with post-hardware iperf numbers.
 ---
 ## 5. UCS storage migration to spinning disk (do today)
 UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
 UCS VMs should stay on spinning disk (HDD) and be split across storage
 servers for redundancy.
 | VM | Current | Target | Action |
 |----|---------|--------|--------|
 | ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
 | ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
 **To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
 1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
 2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
   (both are NFS exports visible to tsys9, so this is a storage-only migration)
 3. Verify VM 902 boots and LDAP/AD services are healthy after migration
 Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
 ucs-02 needs to move -- it should be on a different storage server than
 ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
 ---
 ## 6. Open questions for next session
 - Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
  Determines whether live migration is available.
 - What k8s distribution is in use? (k3s, kubeadm, RKE2?)
 - Container runtime? (containerd, cri-o?)
 - Is there a local container image registry mirror?
 - What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
 - HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
 - What uses tsys5's SDR + parallel port before planning tsys5 role changes?
 - tsys3 thermal state (laptop in rack for years) — check `sensors`.
@@ -0,0 +1,297 @@
 #!/usr/bin/env bash
 # =============================================================================
 # audit-netboot.sh
 # -----------------------------------------------------------------------------
 # READ-ONLY audit of the reference node "pfv-netboot".
 #
 # Purpose: gather enough information to replicate its Pi-hole, Technitium DNS,
 # and NTP services onto pfv-netinfra-01 and pfv-netinfra-02.
 #
 # Guarantees:
 #   * NO writes, NO installs, NO service restarts, NO network changes.
 #   * Every command below is read-only (status, cat, ls, ss, ps, sqlite3 SELECT).
 #
 # Intended to be run as the "localuser" account on pfv-netboot:
 #
 #     ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh > netboot-audit.txt
 #
 # or, if sudo is needed for a few reads, the script will try `sudo -n` for
 # specific files that are normally root-readable only. It will NEVER use sudo
 # to write or modify anything.
 # =============================================================================
 set -u
 AUDIT_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 # Helper: print a clearly delimited section header.
 section() {
    printf '\n========================================================================\n'
    printf 'SECTION: %s\n' "$1"
    printf '========================================================================\n'
 }
 # Helper: read a file with sudo if needed, silently skip if missing.
 read_file() {
    local path="$1"
    if [ -r "$path" ]; then
        printf '--- %s (uid-readable) ---\n' "$path"
        cat "$path" 2>/dev/null
    elif sudo -n true 2>/dev/null; then
        if sudo -n test -e "$path"; then
            printf '--- %s (via sudo -n) ---\n' "$path"
            sudo -n cat "$path" 2>/dev/null
        else
            printf '--- %s : NOT FOUND ---\n' "$path"
        fi
    else
        printf '--- %s : NOT READABLE (no passwordless sudo) ---\n' "$path"
    fi
    printf '\n'
 }
 list_dir() {
    local path="$1"
    printf '--- ls -la %s ---\n' "$path"
    ls -la "$path" 2>/dev/null || sudo -n ls -la "$path" 2>/dev/null || printf '(cannot list %s)\n' "$path"
    printf '\n'
 }
 printf '##### AUDIT START %s #####\n' "$AUDIT_DATE"
 printf 'Audit host: %s\n' "$(hostname -f 2>/dev/null || hostname)"
 printf 'Audit user: %s\n' "$(id -un 2>/dev/null)"
 printf 'Script:     audit-netboot.sh (READ-ONLY)\n'
 # -----------------------------------------------------------------------------
 # 1. System basics
 # -----------------------------------------------------------------------------
 section "1. SYSTEM BASICS"
 echo "-- uname --"; uname -a
 echo "-- /etc/os-release --"; cat /etc/os-release 2>/dev/null
 echo "-- uptime --"; uptime
 echo "-- timezone --"; timedatectl 2>/dev/null || cat /etc/timezone 2>/dev/null || date
 echo "-- arch --"; dpkg --print-architecture 2>/dev/null || uname -m
 echo "-- memory --"; free -h 2>/dev/null
 echo "-- disk --"; df -h / 2>/dev/null
 echo "-- cpu count --"; nproc 2>/dev/null
 # -----------------------------------------------------------------------------
 # 2. Network configuration
 # -----------------------------------------------------------------------------
 section "2. NETWORK"
 echo "-- hostname --"; hostname; hostname -f 2>/dev/null; hostname -I 2>/dev/null
 echo "-- ip addr --"; ip -br addr 2>/dev/null; echo; ip addr 2>/dev/null
 echo "-- default route --"; ip route 2>/dev/null
 echo "-- /etc/resolv.conf --"; cat /etc/resolv.conf 2>/dev/null
 echo "-- /etc/hosts --"; cat /etc/hosts 2>/dev/null
 echo "-- listening TCP/UDP sockets --"
 ss -tlnup 2>/dev/null || sudo -n ss -tlnup 2>/dev/null
 echo "-- /etc/network/interfaces --"; cat /etc/network/interfaces 2>/dev/null
 echo "-- netplan --"; ls -la /etc/netplan/ 2>/dev/null; for f in /etc/netplan/*.yaml; do [ -e "$f" ] && { echo "--- $f ---"; cat "$f"; }; done 2>/dev/null
 echo "-- systemd-networkd --"; ls -la /etc/systemd/network/ 2>/dev/null; networkctl status 2>/dev/null | head -40
 # -----------------------------------------------------------------------------
 # 3. DNS / DHCP / NTP related systemd units
 # -----------------------------------------------------------------------------
 section "3. RELEVANT SYSTEMD UNITS"
 echo "-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --"
 systemctl list-units --type=service --all --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- unit files (enabled state) --"
 systemctl list-unit-files --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- installed packages of interest --"
 dpkg -l 2>/dev/null | grep -Ei 'pihole|dnsmasq|ftl|technitium|ntp|chrony|timesync|unbound|resolved|resolvconf' || true
 # -----------------------------------------------------------------------------
 # 4. NTP service details
 # -----------------------------------------------------------------------------
 section "4. NTP"
 echo "-- chrony --"
 systemctl status chrony --no-pager 2>/dev/null | head -15 || echo "(no chrony unit)"
 read_file /etc/chrony/chrony.conf
 echo "-- chronyc sources/stats (if available) --"
 chronyc -n sources 2>/dev/null || sudo -n chronyc -n sources 2>/dev/null || true
 chronyc -n tracking 2>/dev/null || sudo -n chronyc -n tracking 2>/dev/null || true
 echo "-- ntpsec / ntp classic --"
 systemctl status ntp --no-pager 2>/dev/null | head -15 || echo "(no ntp unit)"
 systemctl status ntpsec --no-pager 2>/dev/null | head -15 || true
 read_file /etc/ntp.conf
 read_file /etc/ntpsec/ntp.conf
 echo "-- systemd-timesyncd --"
 systemctl status systemd-timesyncd --no-pager 2>/dev/null | head -15 || echo "(no timesyncd)"
 read_file /etc/systemd/timesyncd.conf
 echo "-- openntpd --"
 systemctl status openntpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/openntpd/ntpd.conf
 # -----------------------------------------------------------------------------
 # 5. Docker (Pi-hole and/or Technitium may be containerized)
 # -----------------------------------------------------------------------------
 section "5. DOCKER"
 if command -v docker >/dev/null 2>&1; then
    echo "-- docker version --"; docker version 2>/dev/null || sudo -n docker version 2>/dev/null || true
    echo "-- containers (running) --"; docker ps 2>/dev/null || sudo -n docker ps 2>/dev/null || true
    echo "-- containers (all) --"; docker ps -a 2>/dev/null || sudo -n docker ps -a 2>/dev/null || true
    echo "-- images --"; docker images 2>/dev/null || sudo -n docker images 2>/dev/null || true
    echo "-- volumes --"; docker volume ls 2>/dev/null || sudo -n docker volume ls 2>/dev/null || true
    echo "-- networks --"; docker network ls 2>/dev/null || sudo -n docker network ls 2>/dev/null || true
 else
    echo "(docker not installed / not on PATH)"
 fi
 echo "-- compose files in common locations --"
 for d in /etc/docker-compose /opt/docker-compose /root/docker-compose /home/localuser/docker-compose /srv/docker-compose /opt/pihole /opt/technitium /opt; do
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        echo "### $d ###"
        sudo -n ls -la "$d" 2>/dev/null || ls -la "$d" 2>/dev/null || true
        for f in docker-compose.yml docker-compose.yaml compose.yml compose.yaml; do
            if sudo -n test -f "$d/$f" 2>/dev/null; then
                read_file "$d/$f"
            elif [ -f "$d/$f" ]; then
                read_file "$d/$f"
            fi
        done
    fi
 done
 # -----------------------------------------------------------------------------
 # 6. Pi-hole
 # -----------------------------------------------------------------------------
 section "6. PI-HOLE"
 if command -v pihole >/dev/null 2>&1; then
    echo "-- pihole version --"; pihole -v 2>/dev/null || sudo -n pihole -v 2>/dev/null || true
    echo "-- pihole status --"; pihole status 2>/dev/null || sudo -n pihole status 2>/dev/null || true
 else
    echo "(pihole command not on PATH)"
 fi
 echo "-- /etc/pihole listing --"
 list_dir /etc/pihole
 # Key Pi-hole config files
 for f in \
    /etc/pihole/setupVars.conf \
    /etc/pihole/pihole-FTL.conf \
    /etc/pihole/adlists.list \
    /etc/pihole/whitelist.txt \
    /etc/pihole/blacklist.txt \
    /etc/pihole/regex.list \
    /etc/pihole/custom.list \
    /etc/pihole/local.list \
    /etc/pihole/hostnameMappings.txt \
    /etc/pihole/dhcp.leases \
    /etc/pihole/static_ip.conf \
    /etc/pihole/GitHubVersions \
    /etc/pihole/macvendor.db ; do
    read_file "$f"
 done
 echo "-- /etc/pihole/*.conf (all) --"
 for f in /etc/pihole/*.conf; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.d listing --"
 list_dir /etc/dnsmasq.d
 for f in /etc/dnsmasq.d/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.conf (if present) --"
 read_file /etc/dnsmasq.conf
 echo "-- gravity.db schema + row counts (Pi-hole v5+) --"
 GRAVITY=""
 for g in /etc/pihole/gravity.db /etc/pihole/gravity.db.*; do
    if [ -e "$g" ]; then GRAVITY="$g"; break; fi
 done
 if [ -n "${GRAVITY:-}" ]; then
    echo "gravity.db = $GRAVITY"
    if command -v sqlite3 >/dev/null 2>&1; then
        sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || true
        for tbl in adlist adlist_by_group domainlist domainlist_by_group client client_by_group group info; do
            cnt=$(sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || echo "n/a")
            printf 'gravity.%s count = %s\n' "$tbl" "$cnt"
        done
        echo "-- adlist entries (url, enabled, comment) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null || true
        echo "-- domainlist sample (first 50) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null || true
        echo "-- client list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null || true
        echo "-- group list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null || true
        echo "-- info table --"
        sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null || true
    else
        echo "(sqlite3 not installed; gravity.db present at $GRAVITY)"
    fi
 else
    echo "(no gravity.db found)"
 fi
 echo "-- lighttpd / pihole web admin --"
 systemctl status lighttpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/lighttpd/lighttpd.conf
 for f in /etc/lighttpd/conf-enabled/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- pihole-FTL service --"
 systemctl status pihole-FTL --no-pager 2>/dev/null | head -20 || true
 # -----------------------------------------------------------------------------
 # 7. Technitium DNS
 # -----------------------------------------------------------------------------
 section "7. TECHNITIUM DNS"
 echo "-- technitium systemd unit --"
 systemctl status technitium-dns --no-pager 2>/dev/null | head -20 || echo "(no technitium-dns unit)"
 echo "-- any unit containing technitium --"
 systemctl list-units --all --no-pager 2>/dev/null | grep -i technitium || true
 echo "-- unit file path --"
 sudo -n systemctl cat technitium-dns 2>/dev/null || systemctl cat technitium-dns 2>/dev/null || true
 # Technitium common install locations
 for d in /etc/technitium /etc/technitium/dns /opt/technitium /opt/technitium/dns /var/lib/technitium /usr/local/technitium; do
    echo "### checking $d ###"
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        list_dir "$d"
        # recurse one level for config files
        for sub in "$d" "$d"/*; do
            [ -e "$sub" ] || continue
            if [ -f "$sub" ] && echo "$sub" | grep -Eq '\.(xml|json|conf|config|txt)$'; then
                read_file "$sub"
            fi
        done
    fi
 done
 echo "-- technitium config.xml / dnsServer.conf (search) --"
 sudo -n find /etc/technitium /opt/technitium /var/lib/technitium -maxdepth 4 \
    \( -name 'config.xml' -o -name '*.config' -o -name 'dnsServer.conf' -o -name 'blockList.txt' \) \
    -print 2>/dev/null || true
 # -----------------------------------------------------------------------------
 # 8. Firewall / SELinux / misc
 # -----------------------------------------------------------------------------
 section "8. FIREWALL & MISC"
 echo "-- nftables --"; sudo -n nft list ruleset 2>/dev/null | head -80 || true
 echo "-- iptables --"; sudo -n iptables -S 2>/dev/null | head -40 || true
 echo "-- ufw --"; sudo -n ufw status verbose 2>/dev/null || true
 echo "-- firewalld --"; systemctl status firewalld --no-pager 2>/dev/null | head -8 || true
 echo "-- selinux/apparmor --"; getenforce 2>/dev/null || echo "(SELinux not present)"; aa-status 2>/dev/null | head -5 || true
 # -----------------------------------------------------------------------------
 # 9. Cron / timers that maintain these services
 # -----------------------------------------------------------------------------
 section "9. CRON & TIMERS"
 echo "-- root crontab --"; sudo -n crontab -l 2>/dev/null || echo "(n/a)"
 echo "-- localuser crontab --"; crontab -l 2>/dev/null || echo "(none)"
 echo "-- /etc/cron.d --"; ls -la /etc/cron.d 2>/dev/null
 for f in /etc/cron.d/*pihole* /etc/cron.d/*technitium* /etc/cron.d/*gravity*; do
    [ -e "$f" ] && read_file "$f"
 done 2>/dev/null
 echo "-- pihole timer --"; systemctl list-timers --all --no-pager 2>/dev/null | grep -Ei 'pihole|gravity|technitium' || true
 printf '\n##### AUDIT END %s #####\n' "$AUDIT_DATE"
@@ -0,0 +1,14 @@
 #!/usr/bin/env bash
 # baseline.sh — quick read-only baseline of a target node.
 set -u
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
 hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
 hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
 hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
 hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
 hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
 hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
 hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
 hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
 hdr "HOME"; ls -la /home/localuser 2>/dev/null | head
@@ -0,0 +1,67 @@
 #!/usr/bin/env bash
 # deep-audit-netboot.sh — READ-ONLY deep inspection. Uses `sudo` for docker/root files.
 set -u
 DG="sudo docker"
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "COMPOSE FILES: /root/pihole and /root/NTP"
 for f in /root/pihole/docker-compose.yml /root/NTP/docker-compose.yml; do
    printf '\n--- %s ---\n' "$f"
    sudo cat "$f" 2>&1
 done
 hdr "DIR LAYOUT of compose project dirs"
 sudo ls -la /root/pihole 2>&1
 sudo ls -la /root/NTP 2>&1
 sudo ls -la /root 2>&1
 hdr "SEARCH entire FS for any technitium / dns compose files"
 sudo find / -xdev -maxdepth 6 \( -iname 'docker-compose.y*ml' -o -iname 'compose.y*ml' \) -print 2>/dev/null \
    | grep -Ei 'dns|technitium|tsys' || true
 hdr "ALL CONTAINERS with compose labels"
 $DG ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\tWDIR={{.Label "com.docker.compose.project.working_dir"}}\tCFG={{.Label "com.docker.compose.project.config_files"}}\tSVC={{.Label "com.docker.compose.service"}}' 2>&1
 hdr "DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container)"
 for c in $($DG ps -aq 2>/dev/null); do
    nm=$($DG inspect --format '{{.Name}} | image={{.Config.Image}} | proj={{index .Config.Labels "com.docker.compose.project"}}' "$c" 2>/dev/null)
    printf '\n###### %s ######\n' "$nm"
    $DG inspect "$c" 2>&1
 done
 hdr "PI-HOLE internal config (sudo docker exec, READ-ONLY)"
 echo "-- ls /etc/pihole --";          $DG exec pihole ls -la /etc/pihole 2>&1 || true
 echo "-- ls /etc/dnsmasq.d --";       $DG exec pihole ls -la /etc/dnsmasq.d 2>&1 || true
 for f in setupVars.conf pihole-FTL.conf adlists.list custom.list local.list regex.list dhcp.leases static_ip.conf; do
    echo "--- /etc/pihole/$f ---"; $DG exec pihole cat "/etc/pihole/$f" 2>&1 || true
 done
 echo "-- /etc/dnsmasq.d/* --"
 $DG exec pihole sh -c 'for f in /etc/dnsmasq.d/*; do echo "--- $f ---"; cat "$f"; done' 2>&1 || true
 echo "-- pihole version --";          $DG exec pihole pihole -v 2>&1 || true
 echo "-- gravity row counts --"
 $DG exec pihole sh -c 'for t in adlist domainlist client "group" info; do printf "%s=" "$t"; sqlite3 /etc/pihole/gravity.db "SELECT COUNT(*) FROM $t;" 2>/dev/null; done' 2>&1 || true
 echo "-- adlist addresses --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT address,enabled,comment FROM adlist;" 2>&1 || true
 echo "-- domainlist (allow+deny, first 60) --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT type,domain,enabled,comment FROM domainlist LIMIT 60;" 2>&1 || true
 hdr "CHRONY/NTP container config (tsys-ntp)"
 $DG exec tsys-ntp sh -c 'echo "--- chrony.conf ---"; cat /etc/chrony/chrony.conf 2>/dev/null || cat /etc/chrony.conf 2>/dev/null || echo none; echo "--- ls /etc ---"; ls -la /etc 2>/dev/null | head -30' 2>&1 || true
 $DG exec tsys-ntp chronyc -n sources 2>&1 || true
 $DG exec tsys-ntp chronyc -n tracking 2>&1 || true
 echo "-- bare metal ntpsec.conf --"; sudo cat /etc/ntpsec/ntp.conf 2>&1
 hdr "TECHNITIUM volumes"
 for v in dns_tsys-dns-config dns_tyss-dns-config; do
    mnt=$($DG volume inspect --format '{{.Mountpoint}}' "$v" 2>/dev/null)
    printf '\n--- volume %s -> %s ---\n' "$v" "$mnt"
    [ -n "$mnt" ] || continue
    sudo find "$mnt" -maxdepth 4 -type f 2>/dev/null | head -80
    echo "-- config dir listing --"
    sudo ls -laR "$mnt"/config 2>/dev/null | head -60 || sudo ls -laR "$mnt" 2>/dev/null | head -60 || true
    echo "-- config.xml --"
    sudo cat "$mnt"/config/config.xml 2>/dev/null | head -250 || true
 done
 hdr "DONE"
@@ -0,0 +1,54 @@
 #!/usr/bin/env bash
 # gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
 # contents, and chrony NTP details from pfv-netboot for replication.
 set -u
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "TECHNITIUM dns.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
 hdr "TECHNITIUM auth.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
 hdr "TECHNITIUM log.config"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
 hdr "TECHNITIUM scopes/Default.scope"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
 hdr "TECHNITIUM zone: knel.net.zone"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
 hdr "TECHNITIUM all zone files listing + sizes"
 sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
 hdr "TECHNITIUM reverse zones (one sample)"
 sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
 hdr "TECHNITIUM top-level listing"
 sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
 hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
 cat /root/pihole/etc-pihole/pihole.toml 2>&1
 hdr "PI-HOLE dnsmasq.conf"
 cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
 hdr "PI-HOLE versions file"
 cat /root/pihole/etc-pihole/versions 2>&1
 hdr "GRAVITY.DB contents (host sqlite3 if present)"
 if command -v sqlite3 >/dev/null 2>&1; then
    G=/root/pihole/etc-pihole/gravity.db
    echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
    echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
    echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
    echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
    echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
    echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
    echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
 else
    echo "(sqlite3 not on host PATH)"
 fi
 hdr "DONE"
@@ -0,0 +1,720 @@
 This system is the property of Known Element Enterprises LLC.
 Authorized uses only. All activity may be monitored and reported.
 All activities subject to monitoring/recording/review in real time and/or at a later time.##### AUDIT START 2026-07-28T02:41:34Z #####
 Audit host: pfv-netboot.knel.net
 Audit user: localuser
 Script:     audit-netboot.sh (READ-ONLY)
 ========================================================================
 SECTION: 1. SYSTEM BASICS
 ========================================================================
 -- uname --
 Linux pfv-netboot 6.1.0-44-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.164-1 (2026-03-09) x86_64 GNU/Linux
 -- /etc/os-release --
 PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
 NAME="Debian GNU/Linux"
 VERSION_ID="12"
 VERSION="12 (bookworm)"
 VERSION_CODENAME=bookworm
 ID=debian
 HOME_URL="https://www.debian.org/"
 SUPPORT_URL="https://www.debian.org/support"
 BUG_REPORT_URL="https://bugs.debian.org/"
 -- uptime --
 21:41:34 up 113 days, 15:32,  2 users,  load average: 0.07, 0.07, 0.06
 -- timezone --
               Local time: Mon 2026-07-27 21:41:34 CDT
           Universal time: Tue 2026-07-28 02:41:34 UTC
                 RTC time: Tue 2026-07-28 02:41:34
                Time zone: US/Central (CDT, -0500)
 System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no
 -- arch --
 amd64
 -- memory --
               total        used        free      shared  buff/cache   available
 Mem:           1.9Gi       940Mi        66Mi        65Mi       1.1Gi       961Mi
 Swap:          974Mi       315Mi       659Mi
 -- disk --
 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda1       491G   18G  448G   4% /
 -- cpu count --
 2
 ========================================================================
 SECTION: 2. NETWORK
 ========================================================================
 -- hostname --
 pfv-netboot
 pfv-netboot.knel.net
 192.168.3.250 172.24.0.1 172.18.0.1 172.19.0.1 172.23.0.1 172.17.0.1 172.20.0.1 172.22.0.1 100.103.64.82 2600:1700:1d1:eb00:215:5dff:fe64:e81e fd7a:115c:a1e0::2901:4052 
 -- ip addr --
 lo               UNKNOWN        127.0.0.1/8 ::1/128 
 eth0             UP             192.168.3.250/22 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 fe80::215:5dff:fe64:e81e/64 
 eth1             UP             
 br-3d55385c5e7d  UP             172.24.0.1/16 fe80::d883:2ff:fef8:79cd/64 
 br-50e2988e78eb  DOWN           172.18.0.1/16 
 br-748fea589ab8  DOWN           172.19.0.1/16 
 br-b52d11330b20  DOWN           172.23.0.1/16 
 docker0          DOWN           172.17.0.1/16 
 br-f91c61cdba28  DOWN           172.20.0.1/16 
 br-25187260f675  DOWN           172.22.0.1/16 
 tailscale0       UNKNOWN        100.103.64.82/32 fd7a:115c:a1e0::2901:4052/128 fe80::42db:e6ff:fe57:431c/64 
 veth9ecb986@if2  UP             fe80::c45b:62ff:fe19:6258/64 
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:64:e8:1e brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.250/22 brd 192.168.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 scope global dynamic mngtmpaddr 
       valid_lft 3391sec preferred_lft 3391sec
    inet6 fe80::215:5dff:fe64:e81e/64 scope link 
       valid_lft forever preferred_lft forever
 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:64:e8:2a brd ff:ff:ff:ff:ff:ff
 5: br-3d55385c5e7d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether da:83:02:f8:79:cd brd ff:ff:ff:ff:ff:ff
    inet 172.24.0.1/16 brd 172.24.255.255 scope global br-3d55385c5e7d
       valid_lft forever preferred_lft forever
    inet6 fe80::d883:2ff:fef8:79cd/64 scope link 
       valid_lft forever preferred_lft forever
 6: br-50e2988e78eb: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether ca:f9:33:b2:b6:bb brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-50e2988e78eb
       valid_lft forever preferred_lft forever
 7: br-748fea589ab8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 82:e8:2b:47:1e:91 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-748fea589ab8
       valid_lft forever preferred_lft forever
 8: br-b52d11330b20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether de:6e:f1:ec:24:64 brd ff:ff:ff:ff:ff:ff
    inet 172.23.0.1/16 brd 172.23.255.255 scope global br-b52d11330b20
       valid_lft forever preferred_lft forever
 9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 56:f9:55:99:fa:af brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
 10: br-f91c61cdba28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 8e:ed:24:fa:4b:4a brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-f91c61cdba28
       valid_lft forever preferred_lft forever
 11: br-25187260f675: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 82:af:8a:0a:64:01 brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-25187260f675
       valid_lft forever preferred_lft forever
 18: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.103.64.82/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::2901:4052/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42db:e6ff:fe57:431c/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
 24: veth9ecb986@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3d55385c5e7d state UP group default 
    link/ether c6:5b:62:19:62:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::c45b:62ff:fe19:6258/64 scope link 
       valid_lft forever preferred_lft forever
 -- default route --
 default via 192.168.3.254 dev eth0 onlink 
 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
 172.18.0.0/16 dev br-50e2988e78eb proto kernel scope link src 172.18.0.1 linkdown 
 172.19.0.0/16 dev br-748fea589ab8 proto kernel scope link src 172.19.0.1 linkdown 
 172.20.0.0/16 dev br-f91c61cdba28 proto kernel scope link src 172.20.0.1 linkdown 
 172.22.0.0/16 dev br-25187260f675 proto kernel scope link src 172.22.0.1 linkdown 
 172.23.0.0/16 dev br-b52d11330b20 proto kernel scope link src 172.23.0.1 linkdown 
 172.24.0.0/16 dev br-3d55385c5e7d proto kernel scope link src 172.24.0.1 
 192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.250 
 -- /etc/resolv.conf --
 # resolv.conf(5) file generated by tailscale
 # For more info, see https://tailscale.com/s/resolvconf-overwrite
 # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
 nameserver 100.100.100.100
 nameserver fd7a:115c:a1e0::53
 search knel.net
 -- /etc/hosts --
 127.0.0.1	localhost
 127.0.1.1	pfv-netboot.knel.net	pfv-netboot
 # The following lines are desirable for IPv6 capable hosts
 ::1     localhost ip6-localhost ip6-loopback
 ff02::1 ip6-allnodes
 ff02::2 ip6-allrouters
 -- listening TCP/UDP sockets --
 Netid State  Recv-Q Send-Q                              Local Address:Port  Peer Address:PortProcess
 udp   UNCONN 0      0                                         0.0.0.0:38908      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:53         0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:67         0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:46164      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:111        0.0.0.0:*          
 udp   UNCONN 0      0                                   100.103.64.82:123        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:123        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:123        0.0.0.0:*          
 udp   UNCONN 0      0                                       127.0.0.1:123        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:123        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.17.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.17.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.18.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.18.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.19.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.19.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.20.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.20.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.22.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.22.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.23.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.23.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.24.255.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:137        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.255:137        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:137        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:137        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.17.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.17.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.18.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.18.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.19.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.19.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.20.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.20.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.22.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.22.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.23.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.23.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                  172.24.255.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                      172.24.0.1:138        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.255:138        0.0.0.0:*          
 udp   UNCONN 0      0                                   192.168.3.250:138        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:138        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:161        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:41257      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:20000      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:41641      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:59092      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:10000      0.0.0.0:*          
 udp   UNCONN 0      0                                       127.0.0.1:786        0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:40725      0.0.0.0:*          
 udp   UNCONN 0      0                                         0.0.0.0:37749      0.0.0.0:*          
 udp   UNCONN 0      0                                            [::]:53            [::]:*          
 udp   UNCONN 0      0                                            [::]:34901         [::]:*          
 udp   UNCONN 0      0                                            [::]:111           [::]:*          
 udp   UNCONN 0      0         [fe80::c45b:62ff:fe19:6258]%veth9ecb986:123           [::]:*          
 udp   UNCONN 0      0          [fe80::42db:e6ff:fe57:431c]%tailscale0:123           [::]:*          
 udp   UNCONN 0      0                     [fd7a:115c:a1e0::2901:4052]:123           [::]:*          
 udp   UNCONN 0      0      [fe80::d883:2ff:fef8:79cd]%br-3d55385c5e7d:123           [::]:*          
 udp   UNCONN 0      0                 [fe80::215:5dff:fe64:e81e]%eth0:123           [::]:*          
 udp   UNCONN 0      0         [2600:1700:1d1:eb00:215:5dff:fe64:e81e]:123           [::]:*          
 udp   UNCONN 0      0                                           [::1]:123           [::]:*          
 udp   UNCONN 0      0                                            [::]:123           [::]:*          
 udp   UNCONN 0      0                                            [::]:43257         [::]:*          
 udp   UNCONN 0      0                                            [::]:44469         [::]:*          
 udp   UNCONN 0      0                                            [::]:41641         [::]:*          
 udp   UNCONN 0      0                                            [::]:55132         [::]:*          
 udp   UNCONN 0      0                                            [::]:34667         [::]:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:38435      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:20000      0.0.0.0:*          
 tcp   LISTEN 0      5                                       127.0.0.1:555        0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10002      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10003      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:10000      0.0.0.0:*          
 tcp   LISTEN 0      4096                                100.103.64.82:64362      0.0.0.0:*          
 tcp   LISTEN 0      5                                       127.0.0.1:61209      0.0.0.0:*          
 tcp   LISTEN 0      50                                        0.0.0.0:139        0.0.0.0:*          
 tcp   LISTEN 0      64                                        0.0.0.0:36079      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:45295      0.0.0.0:*          
 tcp   LISTEN 0      5                                         0.0.0.0:4330       0.0.0.0:*          
 tcp   LISTEN 0      100                                       0.0.0.0:25         0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:22         0.0.0.0:*          
 tcp   LISTEN 0      64                                        0.0.0.0:2049       0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:53         0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:40029      0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:111        0.0.0.0:*          
 tcp   LISTEN 0      50                                        0.0.0.0:445        0.0.0.0:*          
 tcp   LISTEN 0      4096                                      0.0.0.0:58853      0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:44322      0.0.0.0:*          
 tcp   LISTEN 0      128                                       0.0.0.0:44323      0.0.0.0:*          
 tcp   LISTEN 0      5                                         0.0.0.0:44321      0.0.0.0:*          
 tcp   LISTEN 0      4096                  [fd7a:115c:a1e0::2901:4052]:52472         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:45755         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:60951         [::]:*          
 tcp   LISTEN 0      4096                                            *:9090             *:*          
 tcp   LISTEN 0      64                                           [::]:37813         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10002         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10003         [::]:*          
 tcp   LISTEN 0      4096                                         [::]:10000         [::]:*          
 tcp   LISTEN 0      4096                                            *:45876            *:*          
 tcp   LISTEN 0      50                                           [::]:139           [::]:*          
 tcp   LISTEN 0      4096                                         [::]:44271         [::]:*          
 tcp   LISTEN 0      5                                            [::]:4330          [::]:*          
 tcp   LISTEN 0      128                                          [::]:22            [::]:*          
 tcp   LISTEN 0      64                                           [::]:2049          [::]:*          
 tcp   LISTEN 0      4096                                         [::]:53            [::]:*          
 tcp   LISTEN 0      4096                                         [::]:111           [::]:*          
 tcp   LISTEN 0      4096                                            *:6556             *:*          
 tcp   LISTEN 0      50                                           [::]:445           [::]:*          
 tcp   LISTEN 0      4096                                         [::]:60725         [::]:*          
 tcp   LISTEN 0      128                                          [::]:44322         [::]:*          
 tcp   LISTEN 0      128                                          [::]:44323         [::]:*          
 tcp   LISTEN 0      5                                            [::]:44321         [::]:*          
 -- /etc/network/interfaces --
 # This file describes the network interfaces available on your system
 # and how to activate them. For more information, see interfaces(5).
 source /etc/network/interfaces.d/*
 # The loopback network interface
 auto lo
 iface lo inet loopback
 # The primary network interface
 allow-hotplug eth0
 iface eth0 inet static
 	address 192.168.3.250
 	netmask 255.255.252.0
 	gateway 192.168.3.254
 #allow-hotplug eth1
 #iface eth1 inet static
 #	address 10.100.100.4
 #	netmask 255.255.255.0
 -- netplan --
 -- systemd-networkd --
 total 8
 drwxr-xr-x 2 root root 4096 Mar  6  2025 .
 drwxr-xr-x 5 root root 4096 Jul 20 17:55 ..
 ●        State: n/a
  Online state: unknown
       Address: 192.168.3.250 on eth0
                172.24.0.1 on br-3d55385c5e7d
                172.18.0.1 on br-50e2988e78eb
                172.19.0.1 on br-748fea589ab8
                172.23.0.1 on br-b52d11330b20
                172.17.0.1 on docker0
                172.20.0.1 on br-f91c61cdba28
                172.22.0.1 on br-25187260f675
                100.103.64.82 on tailscale0
                2600:1700:1d1:eb00:215:5dff:fe64:e81e on eth0
                fd7a:115c:a1e0::2901:4052 on tailscale0
                fe80::215:5dff:fe64:e81e on eth0
                fe80::d883:2ff:fef8:79cd on br-3d55385c5e7d
                fe80::42db:e6ff:fe57:431c on tailscale0
                fe80::c45b:62ff:fe19:6258 on veth9ecb986
       Gateway: 192.168.3.254 on eth0
                fe80::6e4b:b4ff:fe10:4061 on eth0
 ========================================================================
 SECTION: 3. RELEVANT SYSTEMD UNITS
 ========================================================================
 -- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --
  initrd-parse-etc.service             loaded    inactive dead    Mountpoints Configured in the Real Root
  ntpsec-rotate-stats.service          loaded    inactive dead    Rotate ntpd stats
  ntpsec-systemd-netif.service         loaded    inactive dead    ntpsec-systemd-netif.service
  ntpsec.service                       loaded    active   running Network Time Service
 ● systemd-resolved.service             not-found inactive dead    systemd-resolved.service
 ● systemd-timesyncd.service            not-found inactive dead    systemd-timesyncd.service
 -- unit files (enabled state) --
 ntpsec-systemd-netif.path                                                     enabled         enabled
 postfix-resolvconf.path                                                       disabled        enabled
 ntp.service                                                                   alias           -
 ntpd.service                                                                  alias           -
 ntpsec-rotate-stats.service                                                   static          -
 ntpsec-systemd-netif.service                                                  static          -
 ntpsec-wait.service                                                           disabled        enabled
 ntpsec.service                                                                enabled         enabled
 postfix-resolvconf.service                                                    disabled        enabled
 ntpsec-rotate-stats.timer                                                     enabled         enabled
 ntpsec.timer                                                                  masked          enabled
 -- installed packages of interest --
 ii  dnsmasq-base                      2.90-4~deb12u2                       amd64        Small caching DNS proxy and DHCP/TFTP server - executable
 ii  ntpsec                            1.2.2+dfsg1-1+deb12u1                amd64        Network Time Protocol daemon and utility programs
 ii  ntpsec-ntpdate                    1.2.2+dfsg1-1+deb12u1                amd64        client for setting system time from NTP servers
 ii  ntpsec-ntpdig                     1.2.2+dfsg1-1+deb12u1                amd64        ntpdig SNTP client
 ii  python3-ntp                       1.2.2+dfsg1-1+deb12u1                amd64        Python 3 NTP Helper Classes
 ========================================================================
 SECTION: 4. NTP
 ========================================================================
 -- chrony --
 -- chronyc sources/stats (if available) --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- ntpsec / ntp classic --
 ● ntpsec.service - Network Time Service
     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
       Docs: man:ntpd(8)
    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
   Main PID: 2124690 (ntpd)
      Tasks: 1 (limit: 2219)
     Memory: 10.7M
        CPU: 34.324s
     CGroup: /system.slice/ntpsec.service
             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
 ● ntpsec.service - Network Time Service
     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
       Docs: man:ntpd(8)
    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
   Main PID: 2124690 (ntpd)
      Tasks: 1 (limit: 2219)
     Memory: 10.7M
        CPU: 34.324s
     CGroup: /system.slice/ntpsec.service
             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 37: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 driftfile /var/lib/ntp/ntp.drift
 leapfile /usr/share/zoneinfo/leap-seconds.list
 server pfvsvrpi.knel.net
 restrict 127.0.0.1
 restrict ::1
 -- systemd-timesyncd --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- openntpd --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 ========================================================================
 SECTION: 5. DOCKER
 ========================================================================
 -- docker version --
 Client: Docker Engine - Community
 Version:           29.6.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        dfc4efb
 Built:             Thu Jul 16 16:12:20 2026
 OS/Arch:           linux/amd64
 Context:           default
 Client: Docker Engine - Community
 Version:           29.6.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        dfc4efb
 Built:             Thu Jul 16 16:12:20 2026
 OS/Arch:           linux/amd64
 Context:           default
 Server: Docker Engine - Community
 Engine:
  Version:          29.6.2
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.5
  Git commit:       3d80467
  Built:            Thu Jul 16 16:12:20 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.6
  GitCommit:        11ce9d5f3c68c941867e82890e93e815c1304f1b
 runc:
  Version:          1.3.6
  GitCommit:        v1.3.6-0-g491b69ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 -- containers (running) --
 CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
 1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
 c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
 -- containers (all) --
 CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
 1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
 c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
 -- images --
 IMAGE                   ID             DISK USAGE   CONTENT SIZE   EXTRA
 busybox:latest          6d3e4188a38a       4.28MB             0B        
 dockurr/chrony:latest   3b656f2feb55       15.4MB             0B   U    
 pihole/pihole:latest    81365952d1f8       92.7MB             0B   U    
 -- volumes --
 DRIVER    VOLUME NAME
 local     0cb24520936997d233409ea07681920fdb03d92967856e849474cf90446dc426
 local     0ccc1140e4b86442167f65d7caf43d8a3bc04fff2bef461f112d8b7c8e4bd4c8
 local     1d7f70d65f5d739f737c3bc0d1b0b2b5cf4f192477dcbbdb95bca344310bff84
 local     2a1726279d1d382ab0a2bcb2079f60e65fc0fd0d40880074038e0c678b2a7f56
 local     3d207dff87da7dc7d11b70f1e595cc5cce001392cb663ac4d664f5710b9879b8
 local     4f8894b9f0206c743be26e6b644e3bf017c5b6e190ac7da31e7ffb7132a1ca2f
 local     7c210964fd48cd91c8e37543d562569ec523208b5ea421f8fb47dc73a6d724ee
 local     8ccc4c07bf45d05bc922d4f619780f60cac0149b4dd7de5be7720f72f47e6c10
 local     71c98aa3d5fb266ef291ac4fc95b11fc83475b9bb0f70999e77095249ad18bf5
 local     83f977471ac2f4ad6521ad50f0311ee36547701d00d9a869903f0ba667c2000d
 local     87eced0295c12870220fc746ef685530cb5b8e20c4afafbfd573ff773459cd9f
 local     95eb8a7b389b9a04bd90f4cee485bd015b7c3fd311c1750c7596a2b2fde1605d
 local     562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41
 local     8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b
 local     201278554f24a7af6284b9564c45656e6d2dc7b9daea24fe52707d7825cb5385
 local     b24c0bd3e25987537a92078d6f521f5f155e7379295d21e43c500cbabc8a02e0
 local     bb353f885d620d98dbb147f0ff51f38a87725669b1a14390fb825509878a7e0b
 local     bc96d56b965f1e238afd47e483b93f1ec9ed59e8458e8e851ffc9710582a5475
 local     c329a5b999999c074bf767bc1f7b3647cf5d4d0624b7d45d63658cbb723e5d46
 local     cf03c8ea64823ac8db5c6df1093b2d9f5146838dd8dc2986bdc485e8b59d0d24
 local     d9500add842368a6d637090569e5c13cb9f068cbfeb14e02c597ae82f2807e47
 local     da3164486ea59aaa9fd14113faa9104411c61ad36a9487e3ce780e86a96dac1f
 local     dae4bbdcbb634b817b47558df3908906fe7635ad57377389616189bd0f2d7b64
 local     dddeee12833ed02185e00732ddb8c12ef6a6b88c49e7df4e48f7a27b71182bf5
 local     dns_tsys-dns-config
 local     dns_tyss-dns-config
 local     e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629
 local     eda6668262ccd43f0e3c0b9e3c42711cc516f372b46ef174bca34d96d17740b8
 local     f53cb9833442fd9499260fa8073a04a15f5c96a5576aa96d781bc24441906942
 local     f142ec0c6150618365447908d5753ff5ff7defcbf27c210702a17ad205dc4aba
 local     f350de2e220f4983bca2e7ca96374a597a075779224b036ccd76deb280ba8922
 -- networks --
 NETWORK ID     NAME               DRIVER    SCOPE
 0f56376f4515   bridge             bridge    local
 f91c61cdba28   dns_default        bridge    local
 b24303598fa1   host               host      local
 25187260f675   msmtpd_default     bridge    local
 748fea589ab8   netdisco_default   bridge    local
 bd40993bc47f   none               null      local
 b52d11330b20   ntp_default        bridge    local
 3d55385c5e7d   pihole_default     bridge    local
 50e2988e78eb   root_default       bridge    local
 -- compose files in common locations --
 ### /opt ###
 total 16
 drwxr-xr-x  4 root   root   4096 Dec 29  2025 .
 drwxr-xr-x 20 root   root   4096 Jul 20 17:55 ..
 drwxr-xr-x  2 beszel beszel 4096 Apr  6 01:08 beszel-agent
 drwx--x--x  4 root   root   4096 Jun 10  2025 containerd
 ========================================================================
 SECTION: 6. PI-HOLE
 ========================================================================
 (pihole command not on PATH)
 -- /etc/pihole listing --
 main: line 54: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (cannot list /etc/pihole)
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 -- /etc/pihole/*.conf (all) --
 -- /etc/dnsmasq.d listing --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 main: line 54: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (cannot list /etc/dnsmasq.d)
 -- /etc/dnsmasq.conf (if present) --
 -- gravity.db schema + row counts (Pi-hole v5+) --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 (no gravity.db found)
 -- lighttpd / pihole web admin --
 main: line 44: printf: --: invalid option
 printf: usage: printf [-v var] format [arguments]
 alias.url += ("/javascript" => "/usr/share/javascript")
 -- pihole-FTL service --
 ========================================================================
 SECTION: 7. TECHNITIUM DNS
 ========================================================================
 -- technitium systemd unit --
 -- any unit containing technitium --
 -- unit file path --
 ### checking /etc/technitium ###
 ### checking /etc/technitium/dns ###
 ### checking /opt/technitium ###
 ### checking /opt/technitium/dns ###
 ### checking /var/lib/technitium ###
 ### checking /usr/local/technitium ###
 -- technitium config.xml / dnsServer.conf (search) --
 ========================================================================
 SECTION: 8. FIREWALL & MISC
 ========================================================================
 -- nftables --
 table ip raw {
 	chain PREROUTING {
 		type filter hook prerouting priority raw; policy accept;
 		iifname != "br-b52d11330b20" ip daddr 172.23.0.2 counter packets 0 bytes 0 drop
 		iifname != "br-3d55385c5e7d" ip daddr 172.24.0.2 counter packets 0 bytes 0 drop
 	}
 	chain OUTPUT {
 		type filter hook output priority raw; policy accept;
 	}
 }
 table ip filter {
 	chain INPUT {
 		type filter hook input priority filter; policy accept;
 		counter packets 8461563 bytes 1841523123 jump ts-input
 	}
 	chain FORWARD {
 		type filter hook forward priority filter; policy drop;
 		counter packets 20387709 bytes 3821595842 jump DOCKER-USER
 		counter packets 20387709 bytes 3821595842 jump DOCKER-FORWARD
 		counter packets 1064 bytes 88643 jump ts-forward
 	}
 	chain OUTPUT {
 		type filter hook output priority filter; policy accept;
 	}
 	chain DOCKER-FORWARD {
 		counter packets 20387709 bytes 3821595842 jump DOCKER-CT
 		counter packets 14776132 bytes 2075147539 jump DOCKER-INTERNAL
 		counter packets 14776132 bytes 2075147539 jump DOCKER-BRIDGE
 		iifname "br-748fea589ab8" counter packets 0 bytes 0 accept
 		iifname "br-b52d11330b20" counter packets 0 bytes 0 accept
 		iifname "br-f91c61cdba28" counter packets 0 bytes 0 accept
 		iifname "br-25187260f675" counter packets 0 bytes 0 accept
 		iifname "br-3d55385c5e7d" counter packets 10132822 bytes 1741327665 accept
 		iifname "br-50e2988e78eb" counter packets 0 bytes 0 accept
 		iifname "docker0" counter packets 0 bytes 0 accept
 	}
 	chain DOCKER-USER {
 	}
 	chain ts-input {
 		iifname "lo" ip saddr 100.103.64.82 counter packets 0 bytes 0 accept
 		iifname "tailscale0" counter packets 512217 bytes 46924098 accept
 		udp dport 41641 counter packets 3360373 bytes 516737750 accept
 		iifname != "tailscale0" ip saddr 100.115.92.0/23 counter packets 0 bytes 0 return
 		iifname != "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
 	}
 	chain ts-forward {
 		iifname "tailscale0" counter packets 28 bytes 2578 meta mark set mark and 0xff00ffff xor 0x40000
 		meta mark & 0x00ff0000 == 0x00040000 counter packets 28 bytes 2578 accept
 		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
 		oifname "tailscale0" counter packets 15 bytes 1554 accept
 	}
 	chain DOCKER {
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 443 counter packets 0 bytes 0 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 80 counter packets 0 bytes 0 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 udp dport 53 counter packets 32283 bytes 2368019 accept
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 53 counter packets 638 bytes 34248 accept
 		iifname != "br-748fea589ab8" oifname "br-748fea589ab8" counter packets 0 bytes 0 drop
 		iifname != "br-b52d11330b20" oifname "br-b52d11330b20" counter packets 0 bytes 0 drop
 		iifname != "br-f91c61cdba28" oifname "br-f91c61cdba28" counter packets 0 bytes 0 drop
 		iifname != "br-25187260f675" oifname "br-25187260f675" counter packets 0 bytes 0 drop
 		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" counter packets 0 bytes 0 drop
 		iifname != "br-50e2988e78eb" oifname "br-50e2988e78eb" counter packets 0 bytes 0 drop
 		iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
 	}
 	chain DOCKER-BRIDGE {
 		oifname "br-748fea589ab8" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-b52d11330b20" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-f91c61cdba28" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-25187260f675" counter packets 0 bytes 0 jump DOCKER
 		oifname "br-3d55385c5e7d" counter packets 4643310 bytes 333819874 jump DOCKER
 		oifname "br-50e2988e78eb" counter packets 0 bytes 0 jump DOCKER
 -- iptables --
 -P INPUT ACCEPT
 -P FORWARD DROP
 -P OUTPUT ACCEPT
 -N DOCKER
 -N DOCKER-BRIDGE
 -N DOCKER-CT
 -N DOCKER-FORWARD
 -N DOCKER-INTERNAL
 -N DOCKER-USER
 -N ts-forward
 -N ts-input
 -A INPUT -j ts-input
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-FORWARD
 -A FORWARD -j ts-forward
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 443 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 80 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p udp -m udp --dport 53 -j ACCEPT
 -A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 53 -j ACCEPT
 -A DOCKER ! -i br-748fea589ab8 -o br-748fea589ab8 -j DROP
 -A DOCKER ! -i br-b52d11330b20 -o br-b52d11330b20 -j DROP
 -A DOCKER ! -i br-f91c61cdba28 -o br-f91c61cdba28 -j DROP
 -A DOCKER ! -i br-25187260f675 -o br-25187260f675 -j DROP
 -A DOCKER ! -i br-3d55385c5e7d -o br-3d55385c5e7d -j DROP
 -A DOCKER ! -i br-50e2988e78eb -o br-50e2988e78eb -j DROP
 -A DOCKER ! -i docker0 -o docker0 -j DROP
 -A DOCKER-BRIDGE -o br-748fea589ab8 -j DOCKER
 -A DOCKER-BRIDGE -o br-b52d11330b20 -j DOCKER
 -A DOCKER-BRIDGE -o br-f91c61cdba28 -j DOCKER
 -A DOCKER-BRIDGE -o br-25187260f675 -j DOCKER
 -A DOCKER-BRIDGE -o br-3d55385c5e7d -j DOCKER
 -A DOCKER-BRIDGE -o br-50e2988e78eb -j DOCKER
 -A DOCKER-BRIDGE -o docker0 -j DOCKER
 -A DOCKER-CT -o br-748fea589ab8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-b52d11330b20 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-f91c61cdba28 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-25187260f675 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-3d55385c5e7d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o br-50e2988e78eb -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
 -- ufw --
 -- firewalld --
 -- selinux/apparmor --
 (SELinux not present)
 ========================================================================
 SECTION: 9. CRON & TIMERS
 ========================================================================
 -- root crontab --
 (n/a)
 -- localuser crontab --
 (none)
 -- /etc/cron.d --
 -- pihole timer --
 ##### AUDIT END 2026-07-28T02:41:34Z #####
@@ -0,0 +1,896 @@
 This system is the property of Known Element Enterprises LLC.
 Authorized uses only. All activity may be monitored and reported.
 All activities subject to monitoring/recording/review in real time and/or at a later time.
 === COMPOSE FILES: /root/pihole and /root/NTP ===
 --- /root/pihole/docker-compose.yml ---
 # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      # DNS Ports
      - "53:53/tcp"
      - "53:53/udp"
      # Default HTTP Port
      - "10002:80/tcp"
      # Default HTTPs Port. FTL will generate a self-signed certificate
      - "10003:443/tcp"
      # Uncomment the below if using Pi-hole as your DHCP Server
      #- "67:67/udp"
      # Uncomment the line below if you are using Pi-hole as your NTP server
      #- "123:123/udp"
    environment:
      # Set the appropriate timezone for your location from
      # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones, e.g:
      TZ: 'America/Chicago'
      # Set a password to access the web interface. Not setting one will result in a random password being assigned
      FTLCONF_webserver_api_password: 'Gransyan1!'
      # If using Docker's default `bridge` network setting the dns listening mode should be set to 'all'
      FTLCONF_dns_listeningMode: 'all'
    # Volumes store your data between container upgrades
    volumes:
      # For persisting Pi-hole's databases and common configuration file
      - './etc-pihole:/etc/pihole'
      # Uncomment the below if you have custom dnsmasq config files that you want to persist. Not needed for most starting fresh with Pi-hole v6. If you're upgrading from v5 you and have used this directory before, you should keep it enabled for the first v6 container start to allow for a complete migration. It can be removed afterwards. Needs environment variable FTLCONF_misc_etc_dnsmasq_d: 'true'
      #- './etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      # See https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
      # Required if you are using Pi-hole as your DHCP server, else not needed
      #- NET_ADMIN
      # Required if you are using Pi-hole as your NTP client to be able to set the host's system time
      #- SYS_TIME
      # Optional, if Pi-hole should get some more processing time
      - SYS_NICE
    restart: always
 --- /root/NTP/docker-compose.yml ---
 services:
  ntp:
    image: dockurr/chrony
    container_name: tsys-ntp
    environment:
      NTP_SERVERS: "pool.ntp.org"
    ports:
      - 100.103.64.82:123:123/udp
    restart: always
 === DIR LAYOUT of compose project dirs ===
 total 16
 drwxr-xr-x  3 root      root      4096 Feb  6 18:32 .
 drwx------ 10 root      root      4096 Apr  5 06:16 ..
 -rw-r--r--  1 root      root      2089 Feb  6 18:30 docker-compose.yml
 drwxr-xr-x  7 localuser localuser 4096 Jul 27 20:57 etc-pihole
 total 12
 drwxr-xr-x  2 root root 4096 Jul  2  2025 .
 drwx------ 10 root root 4096 Apr  5 06:16 ..
 -rw-r--r--  1 root root  188 Jun 23  2025 docker-compose.yml
 total 84
 drwx------ 10 root root  4096 Apr  5 06:16 .
 drwxr-xr-x 20 root root  4096 Jul 20 17:55 ..
 -rw-------  1 root root  4113 Jun 18  2025 .bash_history
 -rw-r--r--  1 root root   571 Apr 10  2021 .bashrc
 drwx------  3 root root  4096 Jun 10  2025 .config
 drwxr-xr-x  2 root root  4096 Apr  5 06:16 dhcp
 -rw-------  1 root root  2960 Feb 16 19:26 dhcpmigrate
 drwxr-xr-x 10 root root  4096 Jul  2  2025 FetchApply
 drwx------  3 root root  4096 Jul 10  2025 .gnupg
 -rw-------  1 root root    40 Apr  5 06:15 .lesshst
 drwxr-xr-x  2 root root  4096 Jul  2  2025 NTP
 drwxr-xr-x  3 root root  4096 Feb  6 18:32 pihole
 -rw-r--r--  1 root root   161 Jul  9  2019 .profile
 drwx------  2 root root  4096 Jun 13  2025 .ssh
 drwxr-xr-x  2 root root  4096 Jul 27 09:08 .tmp
 -rw-------  1 root root 15370 Apr  5 06:15 .viminfo
 -rw-r--r--  1 root root   180 Jun 17  2025 .wget-hsts
 === SEARCH entire FS for any technitium / dns compose files ===
 === ALL CONTAINERS with compose labels ===
 NAMES      IMAGE                  STATUS                    WDIR=working dir    CFG=config files                      SVC=service
 pihole     pihole/pihole:latest   Up 48 minutes (healthy)   WDIR=/root/pihole   CFG=/root/pihole/docker-compose.yml   SVC=pihole
 tsys-ntp   dockurr/chrony         Up 7 days (healthy)       WDIR=/root/NTP      CFG=/root/NTP/docker-compose.yml      SVC=ntp
 === DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container) ===
 ###### /pihole | image=pihole/pihole:latest | proj=pihole ######
 [
    {
        "Id": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
        "Created": "2026-02-07T00:32:57.4029722Z",
        "Path": "start.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1222239,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2026-07-28T01:57:27.732608642Z",
            "FinishedAt": "2026-07-28T01:57:27.095917957Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2026-07-27T21:43:50.682295418-05:00",
                        "End": "2026-07-27T21:43:50.921778022-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:20.925872245-05:00",
                        "End": "2026-07-27T21:44:21.170837655-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:51.175444367-05:00",
                        "End": "2026-07-27T21:44:51.416140073-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:21.417102671-05:00",
                        "End": "2026-07-27T21:45:21.689964304-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:51.691119192-05:00",
                        "End": "2026-07-27T21:45:51.945416009-05:00",
                        "ExitCode": 0,
                        "Output": "127.0.0.1\n"
                    }
                ]
            }
        },
        "Image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
        "ResolvConfPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hostname",
        "HostsPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hosts",
        "LogPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82-json.log",
        "Name": "/pihole",
        "RestartCount": 2,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/root/pihole/etc-pihole:/etc/pihole:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "pihole_default",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10003"
                    }
                ],
                "53/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "53"
                    }
                ],
                "53/udp": [
                    {
                        "HostIp": "",
                        "HostPort": "53"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10002"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": [
                "CAP_SYS_NICE"
            ],
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/asound",
                "/proc/interrupts",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/sys/devices/virtual/powercap",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "ID": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
                "LowerDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac-init/diff:/var/lib/docker/overlay2/c026d6eb78252f15477a6ed84b76e7855b525a582aafa11ae38d89d932433d9c/diff:/var/lib/docker/overlay2/38b4e9cd3ba8e5a10ee7dffbf21e4e552bc30ca96df4b3fc46532d8c2f4828d0/diff:/var/lib/docker/overlay2/10c7b2bc8ab474a29274b66e94e43e31db3dd248dbe0c8e8f7069e305efc7a41/diff:/var/lib/docker/overlay2/de5840a2fb5aed78f79984d980d59f9e50839ddb792146cc7a33d185b8b3eee2/diff:/var/lib/docker/overlay2/5f0504c14572ef5e745e8b06b8f1b3ce74027bce323b7bd1153c828e0dd97d52/diff:/var/lib/docker/overlay2/7a3eefaaac250fbb5430c567e9d12320c35c5da67ad3f8b7b8a7accccc435f18/diff:/var/lib/docker/overlay2/a6c7374834f19eabf6bd3e36f8ad8f2e20e3cf32c4bfea3133ef432b163dac8b/diff:/var/lib/docker/overlay2/28ffada6ac95ffb6c493b1908d00f9cfa096069fd715fa101b910b628835bed5/diff:/var/lib/docker/overlay2/0e79a525158aef2253d47f7f429c50e48c49fec81329f2be7fc580d906528087/diff:/var/lib/docker/overlay2/7365834c77d4c08772f03b8fc0fa6d21b593d3ac70a9ec66fe77ed403d0ef064/diff:/var/lib/docker/overlay2/4bd0de3b67968b6681a962ee9fb89b6bb09b07e5d2ba0d0f85f0561cd9442d32/diff",
                "MergedDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/merged",
                "UpperDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/diff",
                "WorkDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/root/pihole/etc-pihole",
                "Destination": "/etc/pihole",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "1d09249f01b0",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "123/udp": {},
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "FTLCONF_webserver_api_password=Gransyan1!",
                "FTLCONF_dns_listeningMode=all",
                "TZ=America/Chicago",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DNSMASQ_USER=pihole",
                "FTL_CMD=no-daemon"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig -p $(pihole-FTL --config dns.port) +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "pihole/pihole:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "start.sh"
            ],
            "Labels": {
                "com.docker.compose.config-hash": "60a181f601a1959cb6dda5f5889ff706d9986f7db015664ab40608253b83348e",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "pihole",
                "com.docker.compose.project.config_files": "/root/pihole/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/root/pihole",
                "com.docker.compose.service": "pihole",
                "com.docker.compose.version": "5.0.2",
                "org.opencontainers.image.created": "2025-06-06T19:35:20.360Z",
                "org.opencontainers.image.description": "Pi-hole in a docker container",
                "org.opencontainers.image.licenses": "NOASSERTION",
                "org.opencontainers.image.revision": "b56e67ef6808de2854d30481ebcd40bcd2b43445",
                "org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.title": "docker-pi-hole",
                "org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
                "org.opencontainers.image.version": "2025.06.1"
            }
        },
        "NetworkSettings": {
            "SandboxID": "4c83df12c0f6aa377cdd17640639b35e737eb7530c6798d7b80eb3135dbfbd20",
            "SandboxKey": "/var/run/docker/netns/4c83df12c0f6",
            "Ports": {
                "123/udp": null,
                "443/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10003"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "10003"
                    }
                ],
                "53/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "53/udp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "53"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "53"
                    }
                ],
                "67/udp": null,
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10002"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "10002"
                    }
                ]
            },
            "Networks": {
                "pihole_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "pihole",
                        "pihole"
                    ],
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "3d55385c5e7d7d4ca5ddec3f98533d43452cdc2fda68831ac4a16aeb0d42867c",
                    "EndpointID": "aadb163f607f9247c8ad41f8cadac2282272501c60e3ea7fa31fd2d584271447",
                    "Gateway": "172.24.0.1",
                    "IPAddress": "172.24.0.2",
                    "MacAddress": "7e:48:e2:84:c5:15",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "pihole",
                        "1d09249f01b0"
                    ]
                }
            }
        }
    }
 ]
 ###### /tsys-ntp | image=dockurr/chrony | proj=ntp ######
 [
    {
        "Id": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
        "Created": "2025-06-23T16:51:39.454106229Z",
        "Path": "/bin/startup",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2109237,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2026-07-20T22:55:49.597671587Z",
            "FinishedAt": "2026-07-20T22:55:37.650310687Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2026-07-27T21:44:16.437981113-05:00",
                        "End": "2026-07-27T21:44:16.501904568-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525388062 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:44:46.50366458-05:00",
                        "End": "2026-07-27T21:44:46.566905934-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525413632 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:16.568367435-05:00",
                        "End": "2026-07-27T21:45:16.632376789-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525439143 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:45:46.63406538-05:00",
                        "End": "2026-07-27T21:45:46.695064332-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525464714 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    },
                    {
                        "Start": "2026-07-27T21:46:16.697074212-05:00",
                        "End": "2026-07-27T21:46:16.763778169-05:00",
                        "ExitCode": 0,
                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525490224 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
                    }
                ]
            }
        },
        "Image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
        "ResolvConfPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hostname",
        "HostsPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hosts",
        "LogPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908-json.log",
        "Name": "/tsys-ntp",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "ntp_default",
            "PortBindings": {
                "123/udp": [
                    {
                        "HostIp": "100.103.64.82",
                        "HostPort": "123"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
                    "Target": "/run/chrony"
                },
                {
                    "Type": "volume",
                    "Source": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
                    "Target": "/var/lib/chrony"
                },
                {
                    "Type": "volume",
                    "Source": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
                    "Target": "/etc/chrony"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/interrupts",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "ID": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
                "LowerDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db-init/diff:/var/lib/docker/overlay2/6de110c8e8511a542a0541ecba7006ff7fe26ee0e91b738d87fec292a76a5cb1/diff:/var/lib/docker/overlay2/5dc67368f8acabea18fc0be0b07d52e5a82362b5d57119871e3471c509927a73/diff:/var/lib/docker/overlay2/ce6cab3e3329a7148ae6c5e55af761b4b533eac36dcd08cb34d3de88e958d92f/diff",
                "MergedDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/merged",
                "UpperDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/diff",
                "WorkDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
                "Source": "/var/lib/docker/volumes/8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b/_data",
                "Destination": "/var/lib/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
                "Source": "/var/lib/docker/volumes/562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41/_data",
                "Destination": "/etc/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
                "Source": "/var/lib/docker/volumes/e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629/_data",
                "Destination": "/run/chrony",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "c33ced53142e",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "123/udp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NTP_SERVERS=pool.ntp.org",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NTP_DIRECTIVES=ratelimit\\nrtcsync"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "chronyc -n tracking || exit 1"
                ]
            },
            "Image": "dockurr/chrony",
            "Volumes": {
                "/etc/chrony": {},
                "/run/chrony": {},
                "/var/lib/chrony": {}
            },
            "WorkingDir": "/",
            "Entrypoint": [
                "/bin/startup"
            ],
            "Labels": {
                "com.docker.compose.config-hash": "b9a2451b9ec10e82b113b417df6060f18aab234f117063b84ac7a29455f2feb0",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "ntp",
                "com.docker.compose.project.config_files": "/root/NTP/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/root/NTP",
                "com.docker.compose.replace": "61f92d4c43f32567409816af5e3c1c9622aedcb5e35b27b25980282fa3c3e1b3",
                "com.docker.compose.service": "ntp",
                "com.docker.compose.version": "2.36.2",
                "org.opencontainers.image.created": "2025-06-12T01:51:48.447Z",
                "org.opencontainers.image.description": "🕒 chronyd NTP server in a Docker container.",
                "org.opencontainers.image.licenses": "MIT",
                "org.opencontainers.image.revision": "43fb0f2b381bccfef5123971a51baf450ab24464",
                "org.opencontainers.image.source": "https://github.com/dockur/chrony",
                "org.opencontainers.image.title": "Chrony",
                "org.opencontainers.image.url": "https://github.com/dockur/chrony",
                "org.opencontainers.image.version": "4.7"
            }
        },
        "NetworkSettings": {
            "SandboxID": "ccaa71c75942abc56df6ae1c89a8c2ae91e769ed52eab64635a63f8ee996f782",
            "SandboxKey": "/var/run/docker/netns/ccaa71c75942",
            "Ports": {},
            "Networks": {}
        }
    }
 ]
 === PI-HOLE internal config (sudo docker exec, READ-ONLY) ===
 -- ls /etc/pihole --
 total 2545492
 drwxr-xr-x 7 pihole pihole       4096 Jul 27 20:57 .
 drwxr-xr-x 1 root   root         4096 Feb  6 18:32 ..
 -rw-r----- 1 pihole pihole         65 Jun 23  2025 adlists.list
 -rw-r----- 1 pihole pihole         44 Jul 27 20:57 cli_pw
 drwxr-xr-x 2 pihole pihole       4096 Apr  6 10:07 config_backups
 -rw-r----- 1 pihole pihole          0 Jun 18  2025 dhcp.leases
 -rw-r----- 1 pihole pihole       5753 Apr  6 10:07 dnsmasq.conf
 -rw-r----- 1 pihole pihole    5500928 Jul 26 04:51 gravity.db
 drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 gravity_backups
 -rw-r----- 1 pihole pihole    4751360 Jul 19 04:51 gravity_old.db
 drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 hosts
 drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 listsCache
 -rw-r----- 1 root   root          421 Jul 27 20:57 logrotate
 drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 migration_backup
 -rw-r----- 1 pihole pihole 2591891456 Jul 27 21:40 pihole-FTL.db
 -rw-r----- 1 pihole pihole      32768 Jul 27 21:46 pihole-FTL.db-shm
 -rw-r----- 1 pihole pihole    4272472 Jul 27 21:46 pihole-FTL.db-wal
 -rw-r----- 1 pihole pihole      55996 Apr  6 10:07 pihole.toml
 -rw------- 1 pihole pihole        713 Jun 18  2025 tls.crt
 -rw------- 1 pihole pihole       1734 Jun 18  2025 tls.pem
 -rw------- 1 pihole pihole        733 Jun 18  2025 tls_ca.crt
 -rw-r--r-- 1 pihole pihole        376 Jul 27 20:57 versions
 -- ls /etc/dnsmasq.d --
 ls: cannot access '/etc/dnsmasq.d': No such file or directory
 --- /etc/pihole/setupVars.conf ---
 cat: /etc/pihole/setupVars.conf: No such file or directory
 --- /etc/pihole/pihole-FTL.conf ---
 cat: /etc/pihole/pihole-FTL.conf: No such file or directory
 --- /etc/pihole/adlists.list ---
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 --- /etc/pihole/custom.list ---
 cat: /etc/pihole/custom.list: No such file or directory
 --- /etc/pihole/local.list ---
 cat: /etc/pihole/local.list: No such file or directory
 --- /etc/pihole/regex.list ---
 cat: /etc/pihole/regex.list: No such file or directory
 --- /etc/pihole/dhcp.leases ---
 --- /etc/pihole/static_ip.conf ---
 cat: /etc/pihole/static_ip.conf: No such file or directory
 -- /etc/dnsmasq.d/* --
 --- /etc/dnsmasq.d/* ---
 cat: '/etc/dnsmasq.d/*': No such file or directory
 -- pihole version --
 Core version is v6.1.2 (Latest: v6.4.3)
 Web version is v6.2.1 (Latest: v6.6)
 FTL version is v6.2.2 (Latest: v6.7)
 -- gravity row counts --
 adlist=domainlist=client=group=info=-- adlist addresses --
 OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
 -- domainlist (allow+deny, first 60) --
 OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
 === CHRONY/NTP container config (tsys-ntp) ===
 --- chrony.conf ---
 # https://github.com/dockur/chrony
 # chrony.conf file generated by startup script
 # located at /bin/startup
 # time servers provided by NTP_SERVER environment variables.
 server pool.ntp.org iburst
 driftfile /var/lib/chrony/chrony.drift
 makestep 0.1 3
 ratelimit
 rtcsync
 allow all
 --- ls /etc ---
 total 176
 drwxr-xr-x    1 root     root          4096 Jun 23  2025 .
 drwxr-xr-x    1 root     root          4096 Jun 23  2025 ..
 -rw-r--r--    1 root     root            21 Jan  8  2025 alpine-release
 drwxr-xr-x    1 root     root          4096 Jun 12  2025 apk
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 busybox-paths.d
 drwxr-xr-x    2 chrony   chrony        4096 Jun 23  2025 chrony
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 crontabs
 -rw-r--r--    1 root     root            89 Jan  4  2025 fstab
 -rw-r--r--    1 root     root           530 Jun 12  2025 group
 -rw-r--r--    1 root     root           524 Jun 12  2025 group-
 -rw-r--r--    1 root     root            13 Jul 20 22:55 hostname
 -rw-r--r--    1 root     root           148 Jul 20 22:55 hosts
 -rw-r--r--    1 root     root           570 Jan  4  2025 inittab
 -rw-r--r--    1 root     root            77 Jan  8  2025 issue
 drwxr-xr-x    1 root     root          4096 Jun 12  2025 logrotate.d
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 modprobe.d
 -rw-r--r--    1 root     root            15 Jan  4  2025 modules
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 modules-load.d
 -rw-r--r--    1 root     root           284 Jan  4  2025 motd
 lrwxrwxrwx    1 root     root            12 Jun 23  2025 mtab -> /proc/mounts
 drwxr-xr-x    8 root     root          4096 Jan  8  2025 network
 -rw-r--r--    1 root     root           205 Jan  4  2025 nsswitch.conf
 drwxr-xr-x    2 root     root          4096 Jan  8  2025 opt
 lrwxrwxrwx    1 root     root            21 Jan  8  2025 os-release -> ../usr/lib/os-release
 -rw-r--r--    1 root     root           756 Jun 12  2025 passwd
 -rw-r--r--    1 root     root           702 Jan  4  2025 passwd-
 drwxr-xr-x    7 root     root          4096 Jan  8  2025 periodic
 drwxr-xr-x    2 root     root          4096 Jun 12  2025 pkcs11
 -rw-r--r--    1 root     root           547 Jan  4  2025 profile
 MS Name/IP address         Stratum Poll Reach LastRx Last sample               
 ===============================================================================
 Reference ID    : 00000000 ()
 Stratum         : 0
 Ref time (UTC)  : Thu Jan 01 00:00:00 1970
 System time     : 0.525492370 seconds slow of NTP time
 Last offset     : +0.000000000 seconds
 RMS offset      : 0.000000000 seconds
 Frequency       : 0.849 ppm slow
 Residual freq   : +0.000 ppm
 Skew            : 0.000 ppm
 Root delay      : 1.000000000 seconds
 Root dispersion : 1.000000000 seconds
 Update interval : 0.0 seconds
 Leap status     : Not synchronised
 -- bare metal ntpsec.conf --
 driftfile /var/lib/ntp/ntp.drift
 leapfile /usr/share/zoneinfo/leap-seconds.list
 server pfvsvrpi.knel.net
 restrict 127.0.0.1
 restrict ::1
 === TECHNITIUM volumes ===
 --- volume dns_tsys-dns-config -> /var/lib/docker/volumes/dns_tsys-dns-config/_data ---
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062320.stat
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062321.stat
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/logs/2025-06-23.log
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/self-signed-cert.pfx
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/cache.bin
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/119.127.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/46.96.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/181.103.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/184.108.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/211.114.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/145.105.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/97.82.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/194.67.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/2.108.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/75.110.100.in-addr.arpa.zone
 /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/64.103.100.in-addr.arpa.zone
 -- config dir listing --
 -- config.xml --
 --- volume dns_tyss-dns-config -> /var/lib/docker/volumes/dns_tyss-dns-config/_data ---
 -- config dir listing --
 -- config.xml --
 === DONE ===
@@ -0,0 +1,116 @@
 # pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
 > **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
 > Do **not** modify it. This document describes it as audited so its services can
 > be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
 > collected by **read-only** audit scripts (`audit-netboot.sh`,
 > `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
 ## 1. Host
 | Item | Value |
 |---|---|
 | Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
 | OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
 | Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
 | Timezone | `America/Chicago` (US/Central) |
 | LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
 | Tailscale | `100.103.64.82` (`tailscale0`) |
 | DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
 | Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
 | Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
 `eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
 `ntp_default`, `dns_default`, and several stale ones).
 ## 2. Services overview
 | Service | Form | Running? |
 |---|---|---|
 | **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
 | **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
 | **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
 | **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
 ## 3. Pi-hole (container)
 - **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
 - **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
 - **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
 - **Ports (host):**
  | Host | Container | Purpose |
  |---|---|---|
  | `53/tcp`, `53/udp` | 53 | DNS |
  | `10002/tcp` | 80 | Web admin (HTTP) |
  | `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
 - **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=Gransyan1!`, `FTLCONF_dns_listeningMode=all`
 - **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
 - **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
  - Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
  - `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
  - `queryLogging = true`, DNSSEC off
 - **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
 - **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
 - **Web admin:** `http://pfv-netboot:10002/admin/` (password `Gransyan1!`)
 - Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
 ## 4. NTP (two layers)
 ### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
 - **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
 - **Image:** `dockurr/chrony`
 - **Env:** `NTP_SERVERS=pool.ntp.org`
 - **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
 - `restart: always`
 - chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
 - On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
 ### 4b. bare-metal `ntpsec`
 - Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
 - **Config** (`/etc/ntpsec/ntp.conf`):
  ```
  driftfile /var/lib/ntp/ntp.drift
  leapfile /usr/share/zoneinfo/leap-seconds.list
  server pfvsvrpi.knel.net
  restrict 127.0.0.1
  restrict ::1
  ```
 - Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
 ## 5. Technitium DNS (currently stopped)
 - **Not running** — no container and **no compose file** exists for it.
 - A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
  (mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
  are intact (last activity 2025-06-23). A second typo'd volume
  `dns_tyss-dns-config` is empty.
 - Config files are **binary** (Technitium's own serialization), but copy verbatim:
  `dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
  `self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
 - **Zones present** (12 reverse + 1 forward):
  - `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
  - Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
 - **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
 - The compose project name historically was `dns` (network `dns_default` still exists).
 ## 6. Firewall / misc
 - nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
  `DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
  `OUTPUT ACCEPT`. No UFW / firewalld.
 - Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
  (2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
  usermin (10000/10002/20000), Tailscale (41641).
 - SELinux absent; AppArmor default docker profile.
 ## 7. How it was audited (no changes made)
 ```bash
 ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh        # broad read-only sweep
 ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh   # docker inspect + compose
 ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh       # pihole.toml + technitium
 ```
 Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.
@@ -0,0 +1,220 @@
 # pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
 These two nodes replicate the network-infrastructure services of **pfv-netboot**
 (Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
 reads config from pfv-netboot (read-only) and relays it to each target.
 ## 1. Nodes
 | | pfv-netinfra-01 | pfv-netinfra-02 |
 |---|---|---|
 | OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
 | LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
 | Tailscale | `100.70.181.72` | `100.93.194.82` |
 | RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
 | Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
 | Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
 | Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
 ## 2. Service layout
 All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
 so the compose files are directly editable; data dirs keep container uids):
 ```
 /home/localuser/services/
 ├── pihole/
 │   ├── docker-compose.yml
 │   └── etc-pihole/          # copied from netboot /root/pihole/etc-pihole
 │       ├── pihole.toml      # Pi-hole v6 config (upstreams, etc.)
 │       ├── gravity.db       # adlists / domainlists / clients / groups
 │       ├── adlists.list
 │       ├── dnsmasq.conf
 │       ├── tls.{crt,pem,crt_ca}
 │       └── versions
 ├── ntp/
 │   └── docker-compose.yml   # chrony container (see §5 — not used; host ntpsec serves)
 └── technitium/
    ├── docker-compose.yml
    └── config/              # copied from netboot orphaned volume dns_tsys-dns-config/_data
        ├── dns.config
        ├── auth.config
        ├── scopes/Default.scope
        ├── self-signed-cert.pfx
        └── zones/           # knel.net.zone + 12 Tailscale reverse zones
 ```
 ## 3. Pi-hole (container `pihole`)
 Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
 | Host port | Container | Purpose |
 |---|---|---|
 | `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
 | `10002/tcp` | 80 | Web admin (HTTP) |
 | `10003/tcp` | 443 | Web admin (HTTPS) |
 `docker-compose.yml`:
 ```yaml
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "10002:80/tcp"
      - "10003:443/tcp"
    environment:
      TZ: 'America/Chicago'
      FTLCONF_webserver_api_password: 'Gransyan1!'
      FTLCONF_dns_listeningMode: 'all'
    volumes:
      - './etc-pihole:/etc/pihole'
    cap_add:
      - SYS_NICE
    restart: always
 ```
 - Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
 - Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
 - `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
 - Web admin: `http://<node>:10002/admin/` — password **`Gransyan1!`** (same as netboot).
 - Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
 ## 4. Technitium DNS (container `tsys-dns`)
 Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
 `knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
 orphaned `dns_tsys-dns-config` volume.
 | Host port | Container | Purpose |
 |---|---|---|
 | `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
 | `5380/tcp` | 5380 | Web console (HTTP) |
 | `53443/tcp` | 53443 | Web console (HTTPS) |
 `docker-compose.yml`:
 ```yaml
 services:
  technitium:
    image: technitium/dns-server
    container_name: tsys-dns
    ports:
      - "5300:53/tcp"
      - "5300:53/udp"
      - "5380:5380/tcp"
      - "53443:53443/tcp"
    volumes:
      - './config:/etc/dns'
    restart: always
 ```
 - Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
 - Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
  password (carried over via `auth.config`). If the password is unknown, reset it
  from the console or by removing `config/auth.config` and recreating the container.
 - **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
  because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
  query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
  Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
  Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
  from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
  `knel.net` A-records directly in Pi-hole's Local DNS).
 ## 5. NTP (host `ntpsec`, not a container)
 Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
 serves NTP on every local address — including the Tailscale IP — and keeps the
 system clock synced. This is the **same daemon family as netboot's own bare-metal
 ntpsec**.
 - **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
  Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
  does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
  **does** bind the Tailscale IP, so the container cannot claim it (`address
  already in use`) and would be a non-functional duplicate (verified: the
  container started but never synced — Stratum 0). It is therefore intentionally
  **omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
  host NTP unit and removes any stale `tsys-ntp` container.
 - ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
  `restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
 - Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
  leap normal.
 The `ntp/docker-compose.yml` is still written on each node for parity/reference
 (and in case the host NTP is ever disabled — then `sudo docker compose -f
 /home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
 ## 6. Verification results (2026-07-28)
 | Check | pfv-netinfra-01 | pfv-netinfra-02 |
 |---|---|---|
 | `pihole` health | healthy | healthy |
 | `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
 | Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
 | `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
 | Technitium web `:5380` | HTTP 200 | HTTP 200 |
 | NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
 ## 7. Operating the services
 ```bash
 # status
 sudo docker ps
 # Pi-hole
 sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
 sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
 sudo docker exec pihole pihole -v          # version
 sudo docker exec pihole pihole -g          # rebuild gravity
 sudo docker exec pihole pihole -a -p       # set/change web password
 # Technitium
 sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
 sudo docker exec tsys-dns sh               # explore /etc/dns
 # NTP (host)
 systemctl status ntpsec
 ntpq -pn
 ```
 ## 8. Differences from pfv-netboot (intentional)
 1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
   can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
 2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
 3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
   (the container cannot bind the Tailscale IP here; see §5).
 4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
   on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
 5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
   copied (transient); gravity DB and all configuration are.
 ## 9. Re-running / reproducing
 `setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
 present and uses `docker compose up -d` (no-ops when unchanged). It reads
 pfv-netboot read-only and never mutates it.
 ```bash
 ./setup-netinfra.sh                       # deploy to both nodes
 ./setup-netinfra.sh pfv-netinfra-01       # deploy one node
 ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
 ```
 Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
 sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
 internet for image pulls.
 ## 10. Files in this directory
 | File | Purpose |
 |---|---|
 | `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
 | `audit-netboot.sh` | broad read-only audit of pfv-netboot |
 | `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
 | `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
 | `baseline.sh` | read-only baseline of a target node |
 | `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
 | `pfv-netboot-setup.md` | reference-node documentation |
 | `pfv-netinfra-setup.md` | this document |
@@ -0,0 +1,323 @@
 #!/usr/bin/env bash
 # =============================================================================
 # setup-netinfra.sh
 # -----------------------------------------------------------------------------
 # Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
 # onto pfv-netinfra-01 and pfv-netinfra-02.
 #
 # DESIGN
 #   * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
 #     from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
 #   * The targets cannot SSH to pfv-netboot directly, so config tarballs are
 #     relayed through this workstation:
 #         ssh netboot 'sudo tar -cf - ...'  |  ssh target 'sudo tar -xf - ...'
 #   * Services are deployed under /home/localuser/services/<svc>/ on each
 #     target so localuser can manage them (mirrors netboot's localuser-owned
 #     pihole data dir). `sudo docker` is used since localuser is not in the
 #     docker group (same as on netboot).
 #
 # SERVICES
 #   pihole      pihole/pihole:latest   :53 tcp/udp  :10002->80  :10003->443
 #   ntp (chrony) dockurr/chrony        <tailscale-ip>:123:123/udp
 #   technitium  technitium/dns-server  :5300->53 tcp/udp  :5380  :53443
 #     (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
 #      The knel.net authoritative zone + Tailscale reverse zones are preserved
 #      verbatim from netboot's orphaned dns_tsys-dns-config volume.)
 #
 # USAGE
 #   ./setup-netinfra.sh                # deploy to BOTH nodes
 #   ./setup-netinfra.sh pfv-netinfra-01   # deploy to one node
 #   ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
 # =============================================================================
 set -euo pipefail
 NETBOOT="localuser@pfv-netboot"
 SVC_ROOT="/home/localuser/services"
 PIHOLE_PW='Gransyan1!'   # replicated verbatim from netboot compose
 log()  { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
 warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
 # Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
 declare -A NODE_TSIP=(
  [pfv-netinfra-01]="100.70.181.72"
  [pfv-netinfra-02]="100.93.194.82"
 )
 on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
 #------------------------------------------------------------------------------
 # Verify-only mode
 #------------------------------------------------------------------------------
 verify_node() {
  local node="$1" tsip="${NODE_TSIP[$1]}"
  log "VERIFY $node (tailscale $tsip)"
  on_node "$node" "bash -s" <<EOF
 set +e
 echo "### containers ###"
 sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
 echo
 echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
 echo "### Pi-hole -> Technitium (dig @53 knel.net SOA) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 knel.net SOA 2>&1
 echo "### Pi-hole -> Technitium (dig @53 pfv-netboot.knel.net A) ###"
 dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pfv-netboot.knel.net A 2>&1
 echo "### Pi-hole web (curl :10002) ###"
 curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
 echo
 echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
 dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
 echo "### Technitium web (curl :5380) ###"
 curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
 echo
 echo "### NTP service ###"
 HOST_NTP=""
 for u in ntpsec ntp chrony openntpd; do
  systemctl is-active --quiet "\$u" 2>/dev/null && { HOST_NTP="\$u"; break; }
 done
 echo "host daemon: \${HOST_NTP:-none}"
 if [ -n "\$HOST_NTP" ]; then
  ntpq -c "rv 0 leap,stratum,offset" 2>&1 | head -3
 else
  echo "(no host NTP; chrony container:)"
  sudo docker exec tsys-ntp chronyc -n tracking 2>&1 | head -6
 fi
 EOF
 }
 #------------------------------------------------------------------------------
 # Deploy to one node
 #------------------------------------------------------------------------------
 deploy_node() {
  local node="$1" tsip="${NODE_TSIP[$1]}"
  log "==== DEPLOY $node (tailscale $tsip) ===="
  # ---- 1. Prepare directories on the target -------------------------------
  log "$node: create service dirs"
  on_node "$node" "bash -s" <<EOF
 set -e
 sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
 sudo chown -R localuser:localuser $SVC_ROOT
 EOF
  # ---- 2. Write compose files (as localuser) ------------------------------
  log "$node: write docker-compose files"
  on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
 services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "10002:80/tcp"
      - "10003:443/tcp"
    environment:
      TZ: 'America/Chicago'
      FTLCONF_webserver_api_password: 'Gransyan1!'
      FTLCONF_dns_listeningMode: 'all'
    volumes:
      - './etc-pihole:/etc/pihole'
    cap_add:
      - SYS_NICE
    restart: always
    networks:
      - default
      - dnsnet
 networks:
  dnsnet:
    external: true
 YAML
  on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
 services:
  ntp:
    image: dockurr/chrony
    container_name: tsys-ntp
    environment:
      NTP_SERVERS: "pool.ntp.org"
    ports:
      - "$tsip:123:123/udp"
    restart: always
 YAML
  on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
 services:
  technitium:
    image: technitium/dns-server
    container_name: tsys-dns
    ports:
      - "5300:53/tcp"
      - "5300:53/udp"
      - "5380:5380/tcp"
      - "53443:53443/tcp"
    volumes:
      - './config:/etc/dns'
    restart: always
    networks:
      default:
      dnsnet:
        ipv4_address: 10.53.0.53
 networks:
  dnsnet:
    external: true
 YAML
  # ---- 3. Relay Pi-hole config from netboot -> target ---------------------
  log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
  if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
    log "$node: Pi-hole config already present; skipping copy"
  else
    on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
        "sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
                                   --exclude='etc-pihole/listsCache' \
                                   --exclude='etc-pihole/gravity_backups' \
                                   --exclude='etc-pihole/config_backups' \
                                   etc-pihole" \
      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
  fi
  # ---- 4. Relay Technitium config from netboot orphaned volume -----------
  log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
  if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
    log "$node: Technitium config already present; skipping copy"
  else
    on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
        "sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
  fi
  # ---- 5. Adapt copied config: interface + repoint knel.net to local Technitium
  log "$node: adapt Pi-hole pihole.toml (interface + revServer -> local Technitium)"
  on_node "$node" "bash -s" <<'EOF'
 set -e
 IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
 IFACE=${IFACE:-ens18}
 TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
 if sudo test -f "$TOML"; then
  sudo sed -i "s|^  interface = .*|  interface = \"$IFACE\"  ### ADAPTED from eth0 on clone|" "$TOML"
  echo "set interface=$IFACE"
  # Repoint knel.net conditional forward from netboot's upstream (192.168.3.16)
  # to the LOCAL Technitium container at its fixed dnsnet IP 10.53.0.53.
  # Subnet 100.64.0.0/10 = Tailscale CGNAT range (covers all Tailscale reverse zones).
  if sudo grep -q 'revServers' "$TOML"; then
    sudo sed -i 's|"true,[0-9./]*,192\.168\.3\.16,knel\.net"|"true,100.64.0.0/10,10.53.0.53,knel.net"|' "$TOML"
    echo "revServer repointed to 10.53.0.53 (local Technitium)"
  else
    echo "(revServers not found; FTL will use defaults)"
  fi
 else
  echo "(pihole.toml not present; FTL will create it on first run)"
 fi
 EOF
  # ---- 5b. Create shared Docker network for Pi-hole <-> Technitium ----------
  log "$node: create dnsnet shared Docker network (10.53.0.0/24)"
  on_node "$node" "sudo docker network create --subnet 10.53.0.0/24 dnsnet 2>/dev/null || true"
  # ---- 6. Pull images -----------------------------------------------------
  log "$node: docker compose pull (pihole, ntp, technitium)"
  on_node "$node" "bash -s" <<EOF
 for c in pihole ntp technitium; do
  sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
 done
 EOF
  # ---- 6a. Pi-hole up -----------------------------------------------------
  log "$node: bring up Pi-hole"
  on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
  # ---- 6b. NTP -- only deploy the chrony container if nothing already ----
  # serves UDP/123 on the host. The targets already run a bare-metal ntpsec
  # daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
  # netboot's own bare-metal ntpsec. netboot additionally runs a chrony
  # container on its tailscale IP, but that only works there because ntpsec
  # there does not pre-bind the specific tailscale-IP socket. On these targets
  # ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
  # redundant anyway. We therefore keep the host ntpsec as the NTP service.
  log "$node: NTP -- detect host NTP service"
  on_node "$node" "bash -s" <<'EOF'
 set +e
 HOST_NTP=""
 for u in ntpsec ntp chrony openntpd; do
  if systemctl is-active --quiet "$u" 2>/dev/null; then HOST_NTP="$u"; break; fi
 done
 if [ -n "$HOST_NTP" ]; then
  echo "Host NTP daemon '$HOST_NTP' is active -- it serves NTP on all local"
  echo "addresses (incl. the Tailscale IP). This is the same daemon family as"
  echo "netboot's bare-metal ntpsec; the netboot chrony container is redundant"
  echo "here and CANNOT bind the Tailscale IP (the host daemon already owns it)."
  echo "-> Keeping host NTP. Removing any stale chrony container (tsys-ntp)."
  sudo docker rm -f tsys-ntp 2>/dev/null && echo "   (removed tsys-ntp)" || echo "   (no tsys-ntp to remove)"
  echo "   host peers:"; ntpq -pn 2>/dev/null | head -12 || true
 else
  echo "No host NTP daemon active; starting chrony container."
  sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
 fi
 EOF
  # ---- 6c. Technitium up --------------------------------------------------
  log "$node: bring up Technitium"
  on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
  # ---- 7. Wait for Pi-hole health -----------------------------------------
  log "$node: wait for Pi-hole to become healthy"
  on_node "$node" "bash -s" <<'EOF'
 for i in $(seq 1 30); do
  st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
  echo "  pihole health: $st"
  [ "$st" = "healthy" ] && break
  sleep 4
 done
 EOF
  # ---- 8. Technitium defensive fallback -----------------------------------
  # If the copied (binary) config from an older Technitium version makes the
  # new container crash, move it aside and let Technitium start fresh so the
  # service is at least up (admin reachable) rather than crash-looping.
  log "$node: check Technitium health (fallback to fresh config if crash)"
  on_node "$node" "bash -s" <<'EOF'
 set +e
 sleep 8
 rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
 running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
 if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
  echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
  sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
  sudo mkdir -p /home/localuser/services/technitium/config
  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
  echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
 else
  echo "Technitium OK (running=$running restarts=$rst)"
 fi
 EOF
  # ---- 9. Final status -----------------------------------------------------
  log "$node: final container status"
  on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
 }
 #------------------------------------------------------------------------------
 # Main
 #------------------------------------------------------------------------------
 main() {
  local mode="${2:-deploy}"
  if [ "${1:-all}" = "all" ]; then
    targets=(pfv-netinfra-01 pfv-netinfra-02)
  else
    targets=("$1")
  fi
  for t in "${targets[@]}"; do
    : "${NODE_TSIP[$t]:?unknown node $t}"
    if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
  done
  log "DONE"
 }
 main "$@"
@@ -0,0 +1,29 @@
 #!/bin/bash
 # check-pkgs.sh - verify package install state.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== $host ==="
    ssh "${SSH_OPTS[@]}" "root@$host" '
        for p in sysstat jq numactl nvme mtr-tiny dig bmon tcpdump; do
            if command -v "$p" >/dev/null 2>&1; then
                echo "  ✓ $p"
            else
                echo "  ✗ $p"
            fi
        done
        # sysstat config
        echo "  sysstat service:"
        systemctl list-unit-files 2>/dev/null | grep -i sysstat | sed "s/^/    /"
        echo "  sysstat enabled in /etc/default:"
        if [ -r /etc/default/sysstat ]; then
            grep ENABLED /etc/default/sysstat | sed "s/^/    /"
        else
            echo "    no /etc/default/sysstat"
        fi
        # on Debian trixie, sysstat uses a different path
        ls /etc/cron.d/sysstat* 2>/dev/null | sed "s/^/  found: /"
    '
    echo ""
 done
@@ -0,0 +1,41 @@
 #!/bin/bash
 # check-repos-and-reboot.sh - checks reboot-required + Proxmox repo config on all hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host]"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        continue
    fi
    echo "--- /var/run/reboot-required ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'if [ -r /var/run/reboot-required ]; then echo "REBOOT REQUIRED"; cat /var/run/reboot-required 2>/dev/null; if [ -r /var/run/reboot-required.pkgs ]; then echo "Packages triggering:"; cat /var/run/reboot-required.pkgs; fi; else echo "(no reboot required marker)"; fi'
    echo ""
    echo "--- Running kernel vs installed kernel ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'echo "running: $(uname -r)"; echo "installed:"; dpkg -l | grep -E "pve-kernel-[0-9]" | awk "{print \"  \"\$2\"  \"\$3}" | tail -5'
    echo ""
    echo "--- Proxmox repositories (apt sources) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rh "pve\|proxmox" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | grep -v "^#" | sed "s/^/  /"'
    echo ""
    echo "--- Enterprise repo status (should be commented or absent if no subscription) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -l "pve-enterprise" /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null | while read f; do echo "  File: $f"; grep -n "pve-enterprise" "$f" | sed "s/^/    /"; done'
    echo ""
    echo "--- no-subscription repo presence ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -rl "pve-no-subscription" /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | while read f; do echo "  File: $f"; grep -n "pve-no-subscription" "$f" | sed "s/^/    /"; done'
    echo ""
    echo "--- Recently updated packages (last 24h, kernel-related) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -E "pve-kernel|proxmox|pve-qemu|zfs" /var/log/dpkg.log 2>/dev/null | grep "$(date +%Y-%m-%d)\|$(date -d yesterday +%Y-%m-%d)" | tail -15 || echo "(none in dpkg.log)"'
    echo ""
 done
@@ -0,0 +1,51 @@
 #!/bin/bash
 # deploy-and-fix.sh - uploads fix script, runs it, starts VMs, verifies.
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
 echo "=== Uploading fix script to $HOST ==="
 scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
 echo "=== Running fix ==="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/fix-bond-nfs.sh && bash /root/fix-bond-nfs.sh' 2>&1
 echo ""
 echo "=== Starting VMs ==="
 for vmid in $(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"'); do
    status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
    if [ "$status" != "running" ]; then
        echo "  Starting VM $vmid..."
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/    /'
    else
        echo "  VM $vmid already running"
    fi
 done
 echo ""
 echo "Waiting 20s for VMs to boot..."
 sleep 20
 echo ""
 echo "=== FULL VERIFICATION ==="
 echo ""
 echo "--- VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
 echo ""
 echo "--- NFS mounts ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
 echo ""
 echo "--- NFS TCP connections ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 echo "  Count:"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
 echo ""
 echo "--- bond0 hash policy ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -6'
 echo ""
 echo "--- Summary ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "swappiness: $(sysctl -n vm.swappiness)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null)"'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
@@ -0,0 +1,169 @@
 #!/bin/bash
 ###############################################################################
 # deploy-check.sh
 #
 # Deploys scripts/check.sh to each reachable Proxmox host, executes it
 # read-only, and pulls the resulting log back to returned-logs/.
 #
 # EXPLICITLY SKIPS:
 #   - pfv-tsys2   (off the air per user)
 #   - pfv-tsys9   (off the air per user; also not in original inventory)
 #
 # Safety features:
 #   - BatchMode=yes        : never hang on a password prompt
 #   - ConnectTimeout=8     : fail fast on dead hosts
 #   - per-host try/skip    : one bad host never aborts the run
 #   - ServerAliveInterval  : detect hung connections
 #   - read-only script     : check.sh modifies nothing on the target
 ###############################################################################
 set -uo pipefail
 SCRIPT_DIR="/home/reachableceo/projects/perfopt"
 CHECK_SH="$SCRIPT_DIR/scripts/check.sh"
 LOG_DIR="$SCRIPT_DIR/returned-logs"
 mkdir -p "$LOG_DIR"
 # ONLY the hosts the user told us are alive.
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 # Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
 SSH_OPTS=(-o BatchMode=yes
          -o ConnectTimeout=8
          -o ServerAliveInterval=10
          -o ServerAliveCountMax=3
          -o StrictHostKeyChecking=accept-new)
 log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; }
 if [ ! -r "$CHECK_SH" ]; then
    echo "FATAL: $CHECK_SH not found" >&2
    exit 1
 fi
 # Sanity-check shellcheck clean before shipping (best-effort, non-blocking)
 if command -v docker >/dev/null 2>&1; then
    log "pre-flight: shellcheck on check.sh"
    if ! docker run --rm -v "$SCRIPT_DIR:/mnt" -w /mnt \
         koalaman/shellcheck:stable --severity=style --format=gcc scripts/check.sh \
         >"$LOG_DIR/_shellcheck.preflight.txt" 2>&1; then
        log "WARNING: shellcheck reported issues — see _shellcheck.preflight.txt"
        log "         aborting deploy to avoid shipping a broken script"
        exit 1
    fi
    log "pre-flight: shellcheck clean"
 fi
 summary_pass=()
 summary_fail=()
 declare -A HOST_PID        # host -> background pid
 declare -A HOST_MARKER     # host -> per-host marker file
 # Per-host worker — runs in background, one per host, all in parallel.
 # Writes status into a marker file consumed by the parent.
 worker() {
    local host="$1"
    local marker="$LOG_DIR/_marker.$host"
    : > "$marker"  # truncate
    echo "running" >> "$marker"
    local short=""
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "fail unreachable" >> "$marker"
        return
    fi
    if ! scp "${SSH_OPTS[@]}" "$CHECK_SH" "root@$host:/root/check.sh" >/dev/null 2>&1; then
        echo "fail scp-upload-failed" >> "$marker"
        return
    fi
    local remote_size
    remote_size=$(ssh "${SSH_OPTS[@]}" "root@$host" 'wc -c < /root/check.sh' 2>/dev/null || echo 0)
    if [ "${remote_size:-0}" -lt 1000 ]; then
        echo "fail upload-corrupt" >> "$marker"
        return
    fi
    local remote_stdout
    remote_stdout=$(ssh "${SSH_OPTS[@]}" "root@$host" \
        'chmod +x /root/check.sh && bash /root/check.sh' 2>&1)
    local rc=$?
    if [ "$rc" -ne 0 ]; then
        echo "fail check-exit-$rc" >> "$marker"
        # don't return - still try to pull whatever log got produced
    fi
    short=$(printf '%s\n' "$remote_stdout" | grep -oE 'Wrote: /root/[a-zA-Z0-9_-]+\.log' | head -n1 | awk '{print $2}')
    if [ -z "$short" ]; then
        short=$(ssh "${SSH_OPTS[@]}" "root@$host" 'echo "/root/$(hostname -s).log"' 2>/dev/null)
    fi
    if [ -z "$short" ]; then
        echo "fail no-log-path" >> "$marker"
        return
    fi
    if ! scp "${SSH_OPTS[@]}" "root@$host:$short" "$LOG_DIR/" >/dev/null 2>&1; then
        echo "fail scp-download-failed" >> "$marker"
        return
    fi
    local local_name local_path
    local_name="$(basename "$short")"
    local_path="$LOG_DIR/$local_name"
    if [ ! -s "$local_path" ]; then
        echo "fail local-empty" >> "$marker"
        return
    fi
    echo "ok $local_name $(wc -c < "$local_path") $(wc -l < "$local_path")" >> "$marker"
 }
 # ---- launch all workers in parallel --------------------------------------
 log "launching ${#HOSTS[@]} hosts in parallel..."
 for host in "${HOSTS[@]}"; do
    rm -f "$LOG_DIR/_marker.$host"
    worker "$host" &
    HOST_PID[$host]=$!
    HOST_MARKER[$host]="$LOG_DIR/_marker.$host"
    log "  launched $host (pid ${HOST_PID[$host]})"
 done
 # ---- wait for all, with periodic progress --------------------------------
 remaining=("${HOSTS[@]}")
 while [ "${#remaining[@]}" -gt 0 ]; do
    sleep 10
    new_remaining=()
    for host in "${remaining[@]}"; do
        if ! kill -0 "${HOST_PID[$host]}" 2>/dev/null; then
            # process finished
            wait "${HOST_PID[$host]}" 2>/dev/null || true
            marker="${HOST_MARKER[$host]}"
            if [ -r "$marker" ]; then
                status_line="$(tail -n1 "$marker")"
                log "[$host] done: $status_line"
                case "$status_line" in
                    ok*)     summary_pass+=("$host:$status_line") ;;
                    fail*)   summary_fail+=("$host:$status_line") ;;
                    *)       summary_fail+=("$host:unknown") ;;
                esac
            else
                log "[$host] done but marker missing"
                summary_fail+=("$host:no-marker")
            fi
        else
            new_remaining+=("$host")
        fi
    done
    remaining=("${new_remaining[@]:-}")
    if [ "${#remaining[@]}" -gt 0 ]; then
        log "still running: ${remaining[*]} (${#remaining[@]} hosts)"
    fi
 done
 # Final summary
 log "============================================================"
 log "DEPLOY SUMMARY"
 log "============================================================"
 log "Passed (${#summary_pass[@]}):"
 for p in "${summary_pass[@]:-}"; do [ -n "$p" ] && log "  ✓ $p"; done
 log "Failed (${#summary_fail[@]}):"
 for f in "${summary_fail[@]:-}"; do [ -n "$f" ] && log "  ✗ $f"; done
 log ""
 log "Contents of $LOG_DIR:"
 ls -la "$LOG_DIR"
 # Clean up marker files
 rm -f "$LOG_DIR"/_marker.* 2>/dev/null
@@ -0,0 +1,56 @@
 #!/bin/bash
 # deploy-tuning.sh - copies apply-tunings.sh to target hosts and runs it.
 # Usage: bash deploy-tuning.sh [--no-nfs] [--apply] <host> [host...]
 # Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section.
 set -uo pipefail
 SCRIPT="/home/reachableceo/projects/perfopt/scripts/apply-tunings.sh"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 MODE=""
 EXTRA_FLAGS=""
 HOSTS=()
 for arg in "$@"; do
    case "$arg" in
        --apply) MODE="--apply" ;;
        --dry-run) MODE="" ;;
        --no-nfs) EXTRA_FLAGS="--no-nfs" ;;
        *) HOSTS+=("$arg") ;;
    esac
 done
 if [ "${#HOSTS[@]}" -eq 0 ]; then
    echo "Usage: $0 <host> [host...] [--apply]"
    echo "Default: dry-run. Pass --apply to commit."
    exit 1
 fi
 if [ ! -r "$SCRIPT" ]; then
    echo "FATAL: $SCRIPT not found"
    exit 1
 fi
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host] deploying apply-tunings.sh (mode: ${MODE:-dry-run})"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "[$host] SKIP: unreachable"
        continue
    fi
    echo "[$host] uploading..."
    if ! scp "${SSH_OPTS[@]}" "$SCRIPT" "root@$host:/root/apply-tunings.sh" >/dev/null 2>&1; then
        echo "[$host] SKIP: scp failed"
        continue
    fi
    echo "[$host] running (output below)..."
    echo "----------------------------------------------------------------"
    ssh "${SSH_OPTS[@]}" "root@$host" "chmod +x /root/apply-tunings.sh && bash /root/apply-tunings.sh $MODE $EXTRA_FLAGS" 2>&1
    rc=$?
    echo "----------------------------------------------------------------"
    echo "[$host] exit code: $rc"
    echo ""
 done
@@ -0,0 +1,35 @@
 #!/bin/bash
 # diag.sh - diagnostic commands run on a host via SSH wrapper.
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 echo "===== 1. storage.cfg NFS stanzas (exact content) ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'grep -A 8 "^nfs: D2" /etc/pve/storage.cfg'
 echo ""
 echo "===== 2. Try manual NFS mount with nconnect=4 ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
 echo ""
 echo "===== 3. Try manual NFS mount WITHOUT nconnect ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount -t nfs -o noatime,rsize=1048576,wsize=1048576,hard,proto=tcp pfv-tsys4-nfs-stor:/mnt/tsys4/D2 /mnt/pve/D2 2>&1; echo "exit=$?"'
 echo ""
 echo "===== 4. NFS kernel version / module ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/fs/nfsfs/version 2>/dev/null; echo "---"; modinfo nfs 2>/dev/null | grep -E "^(filename|version|description)" | head -5'
 echo ""
 echo "===== 5. mount.nfs version ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'mount.nfs --version 2>&1; echo "---"; dpkg -l nfs-common 2>/dev/null | tail -2'
 echo ""
 echo "===== 6. /etc/network/interfaces bond0 stanza (exact bytes) ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sed -n "/^auto bond0/,/^$/p" /etc/network/interfaces | cat -A'
 echo ""
 echo "===== 7. Current bond0 running hash policy ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -5'
 echo ""
 echo "===== 8. xmit_hash_policy sysfs file ====="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; echo "---"; ls /sys/class/net/bond0/bonding/ 2>/dev/null'
@@ -0,0 +1,128 @@
 #!/bin/bash
 # finish-host.sh - applies ALL remaining changes to a host and verifies.
 #
 # Steps:
 #   1. Start all VMs (triggers NFS lazy-mount)
 #   2. Wait for NFS mounts to appear
 #   3. Verify NFS nconnect=4 + noatime
 #   4. Apply bond0 xmit_hash_policy=layer3+4
 #   5. Full end-to-end verification
 #
 # Usage: bash finish-host.sh <host> [--apply]
 # Default is dry-run (starts VMs + shows what bond change would do, but doesn't edit interfaces)
 set -uo pipefail
 HOST="${1:-}"
 MODE="${2:-dryrun}"
 [ "$MODE" = "--apply" ] && MODE="apply" || MODE="dryrun"
 if [ -z "$HOST" ]; then
    echo "Usage: $0 <host> [--apply]"
    exit 1
 fi
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
 echo "==================================================================="
 echo " finish-host.sh — $HOST  (mode: $MODE)"
 echo "==================================================================="
 echo ""
 # =========================================================================
 # STEP 1: Start all VMs
 # =========================================================================
 echo "=== STEP 1: Start all VMs on $HOST ==="
 # Get list of all VMs (not just stopped — start is idempotent)
 vm_list=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"')
 for vmid in $vm_list; do
    status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
    if [ "$status" != "running" ]; then
        echo "  Starting VM $vmid..."
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/    /'
    else
        echo "  VM $vmid already running"
    fi
 done
 echo ""
 echo "  Waiting 15s for VMs to boot and trigger NFS mounts..."
 sleep 15
 echo ""
 echo "--- VM status after start ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 # =========================================================================
 # STEP 2: Verify NFS mounts came back with nconnect=4
 # =========================================================================
 echo ""
 echo "=== STEP 2: Verify NFS mounts with nconnect=4 ==="
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null' | head -30
 echo ""
 echo "--- NFS TCP connections to :2049 ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 conn_count=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
 echo "  Total NFS TCP connections: $conn_count"
 # =========================================================================
 # STEP 3: Apply bond0 hash policy
 # =========================================================================
 echo ""
 echo "=== STEP 3: Apply bond0 xmit_hash_policy=layer3+4 (mode: $MODE) ==="
 # Upload the bond hash script
 scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/apply-bond-hash.sh" "root@$HOST:/root/apply-bond-hash.sh" >/dev/null 2>&1
 if [ "$MODE" = "apply" ]; then
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh --apply' 2>&1
 else
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/apply-bond-hash.sh && bash /root/apply-bond-hash.sh' 2>&1
 fi
 # =========================================================================
 # STEP 4: Full verification
 # =========================================================================
 echo ""
 echo "=== STEP 4: Full end-to-end verification ==="
 echo ""
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
 echo ""
 echo "--- CPU governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
 echo ""
 echo "--- TCP BBR ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
 echo ""
 echo "--- tuned profile ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
 echo ""
 echo "--- bond0 hash policy + LACP state ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 2>/dev/null | head -25'
 echo ""
 echo "--- NFS mount options (first 3 mounts) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
 echo ""
 echo "--- NFS TCP connections (expect 4 per server × 2 servers = 8) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 nfs_conns=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l')
 echo "  Count: $nfs_conns"
 echo ""
 echo "--- VMs running ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 echo ""
 echo "--- Failed services ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
 echo ""
 echo "--- Network interfaces (speed/duplex/mtu) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'for ifc in bond0 nic0 nic1 nic2 vmbr0 datanet; do [ -d "/sys/class/net/$ifc" ] && printf "%-12s speed=%-8s duplex=%-8s mtu=%s\n" "$ifc" "$(cat /sys/class/net/$ifc/speed 2>/dev/null)" "$(cat /sys/class/net/$ifc/duplex 2>/dev/null)" "$(cat /sys/class/net/$ifc/mtu 2>/dev/null)"; done'
 echo ""
 echo "==================================================================="
 echo " COMPLETE — $HOST"
 echo "==================================================================="
@@ -0,0 +1,37 @@
 #!/bin/bash
 # install-utils-v2.sh - retry install without nstat package.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 PKGS="sysstat jq numactl nvme-cli mtr-tiny dnsutils bmon"
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== [$host] installing: $PKGS ==="
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "DEBIAN_FRONTEND=noninteractive apt-get update -qq 2>&1 | tail -2 && \
         DEBIAN_FRONTEND=noninteractive apt-get install -y $PKGS 2>&1 | tail -10"
    # sysstat enable (path varies by Debian version)
    ssh "${SSH_OPTS[@]}" "root@$host" '
        if [ -r /etc/default/sysstat ]; then
            sed -i "s/^ENABLED=.*/ENABLED=\"true\"/" /etc/default/sysstat
            systemctl enable --now sysstat 2>/dev/null
            grep ENABLED /etc/default/sysstat
        else
            # Newer Debian (trixie) — sysstat cron/service auto-enabled
            systemctl enable --now sysstat 2>/dev/null || echo "(sysstat auto via cron)"
        fi
    '
    echo ""
 done
 # Verify
 for host in pfv-tsys6 pfv-tsys7; do
    echo "=== [$host] verification ==="
    ssh "${SSH_OPTS[@]}" "root@$host" '
        for p in sysstat jq numactl nvme mtr-tiny dig bmon; do
            command -v "$p" >/dev/null 2>&1 && echo "  ✓ $p" || echo "  ✗ $p"
        done
    '
    echo ""
 done
@@ -0,0 +1,60 @@
 #!/bin/bash
 # install-utils.sh - installs useful observability packages on a host.
 # These are all small, dependency-light, and read-only at runtime.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys6 pfv-tsys7)
 # Packages to install, with rationale
 PKGS=(
    sysstat          # sar, iostat, mpstat, pidstat - the missing observability suite
    jq               # JSON parsing for pvesh/scripts
    numactl          # NUMA topology/controls for the dual-socket hosts
    nvme-cli         # NVMe health (for when NVMe shows up)
    tcpdump          # packet capture for network debugging
    mtr-tiny         # traceroute on steroids
    nstat            # kernel SNMP stats (already partly there)
    dnsutils         # dig, nslookup, host
    bmon             # bandwidth monitor ( curses, real-time)
 )
 for host in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$host] installing observability packages"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        continue
    fi
    # Check which are missing
    missing=""
    for pkg in "${PKGS[@]}"; do
        if ! ssh "${SSH_OPTS[@]}" "root@$host" "dpkg -s $pkg 2>/dev/null | grep -q 'Status: install ok installed'" 2>/dev/null; then
            missing="$missing $pkg"
        fi
    done
    if [ -z "$missing" ]; then
        echo "  All packages already installed."
        continue
    fi
    echo "  Installing:$missing"
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
         DEBIAN_FRONTEND=noninteractive apt-get install -y -qq$missing 2>&1 | tail -5"
    echo "  Done."
    echo ""
 done
 # Enable sysstat data collection (sar) — off by default on Debian
 for host in "${HOSTS[@]}"; do
    echo "[$host] enabling sysstat/sar data collection..."
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "sed -i 's/^ENABLED=\"false\"/ENABLED=\"true\"/' /etc/default/sysstat 2>/dev/null; \
         systemctl enable --now sysstat 2>&1 | tail -2; \
         grep ENABLED /etc/default/sysstat"
 done
@@ -0,0 +1,293 @@
 #!/bin/bash
 ###############################################################################
 # iperf-full-matrix.sh
 #
 # Two test suites:
 #   A. Management network (vmbr0 / VLAN1): all-pairs single-stream TCP, 10s
 #   B. Storage network (VLAN1000): tsys6+tsys7 → tsys4+tsys5, stress test
 #
 # Output: returned-logs/iperf/
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 # Storage IPs (known, static on VLAN1000)
 declare -A SIP
 SIP[pfv-tsys1]="10.100.100.1"
 SIP[pfv-tsys3]="10.100.100.3"
 SIP[pfv-tsys4]="10.100.100.4"
 SIP[pfv-tsys5]="10.100.100.5"
 SIP[pfv-tsys6]="10.100.100.6"
 SIP[pfv-tsys7]="10.100.100.7"
 # ===========================================================================
 # STEP 0: Discover management IPs (vmbr0)
 # ===========================================================================
 echo "==================================================================="
 echo " STEP 0: Discover management network IPs (vmbr0)"
 echo "==================================================================="
 declare -A MIP
 for host in "${HOSTS[@]}"; do
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "  [$host] UNREACHABLE"
        continue
    fi
    mip=$(ssh "${SSH_OPTS[@]}" "root@$host" 'ip -o -4 addr show dev vmbr0 2>/dev/null | awk "{print \$4}" | cut -d/ -f1 | head -1')
    if [ -n "$mip" ]; then
        MIP[$host]="$mip"
        echo "  [$host] vmbr0 = $mip"
    else
        echo "  [$host] no vmbr0 IPv4 — skipping"
    fi
 done
 # ===========================================================================
 # STEP 1: Ensure iperf3 installed on all hosts
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " STEP 1: Ensure iperf3 installed"
 echo "==================================================================="
 for host in "${HOSTS[@]}"; do
    [ -z "${MIP[$host]:-}" ] && continue
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
        echo -n "  [$host] installing iperf3... "
        ssh "${SSH_OPTS[@]}" "root@$host" \
            'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && \
             DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
    else
        echo "  [$host] iperf3 already installed"
    fi
 done
 # Helper: start iperf3 server in one-shot mode bound to a specific IP
 start_server() {
    local host="$1" ip="$2"
    ssh "${SSH_OPTS[@]}" "root@$host" \
        "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
 }
 # Helper: run iperf3 client test, save output, extract result
 run_test() {
    local client="$1" server="$2" sip="$3" cip="$4" label="$5" logfile="$6"
    shift 6
    local extra="$*"
    echo -n "  [$label] ... "
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client ($cip)  →  Server: $server ($sip)"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo "Args: $extra"
        echo ""
        ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${sip} -B ${cip} ${extra}" 2>&1
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract result
    sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$logfile" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    echo "${bitrate:-?} (retrans: ${retrans:-?})"
 }
 # ===========================================================================
 # SUITE A: Management network (vmbr0) — all pairs, single-stream TCP, 10s
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " SUITE A: MANAGEMENT NETWORK (vmbr0) — all-pairs, 10s TCP"
 echo " Expectation: ~940 Mbps for every pair (1 GbE line rate)"
 echo "==================================================================="
 echo ""
 # Test each unique pair (i < j to avoid duplicates)
 for ((i=0; i<${#HOSTS[@]}; i++)); do
    for ((j=i+1; j<${#HOSTS[@]}; j++)); do
        client="${HOSTS[$i]}"
        server="${HOSTS[$j]}"
        [ -z "${MIP[$client]:-}" ] && continue
        [ -z "${MIP[$server]:-}" ] && continue
        label="mgmt: ${client}→${server}"
        logfile="$LOG_DIR/mgmt-${client}-to-${server}.log"
        start_server "$server" "${MIP[$server]}"
        run_test "$client" "$server" "${MIP[$server]}" "${MIP[$client]}" \
            "$label" "$logfile" "-t 10 -P 1"
    done
 done
 # ===========================================================================
 # SUITE B: Storage network (VLAN1000) — stress test the cross-rack LACP
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " SUITE B: STORAGE NETWORK (VLAN1000) — stress test cross-rack link"
 echo " tsys6 + tsys7 (Rack 3) → tsys4 + tsys5 (Rack 5)"
 echo " Expectation: limited by tsys4 USB dongle + tsys5 broken bond"
 echo "==================================================================="
 echo ""
 # --- B.1: Individual tests (one client → one server at a time) ---
 echo "--- B.1: Individual tests (sequential) ---"
 echo ""
 for client in pfv-tsys6 pfv-tsys7; do
    for server in pfv-tsys4 pfv-tsys5; do
        label="stor: ${client}→${server} (8-stream)"
        logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
        start_server "$server" "${SIP[$server]}"
        run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
            "$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
    done
 done
 # --- B.2: Reverse direction (tsys4/5 → tsys6/7) ---
 echo ""
 echo "--- B.2: Reverse direction (tsys4/5 → tsys6/7) ---"
 echo ""
 for client in pfv-tsys4 pfv-tsys5; do
    for server in pfv-tsys6 pfv-tsys7; do
        label="stor: ${client}→${server} (8-stream rev)"
        logfile="$LOG_DIR/stor-indiv-${client}-to-${server}-8stream.log"
        start_server "$server" "${SIP[$server]}"
        run_test "$client" "$server" "${SIP[$server]}" "${SIP[$client]}" \
            "$label" "$logfile" "-P 8 -t 20 -l 128k -O 2"
    done
 done
 # --- B.3: Simultaneous stress test (4 flows at once) ---
 echo ""
 echo "--- B.3: Simultaneous 4-flow stress test ---"
 echo "  tsys6→tsys4 + tsys6→tsys5 + tsys7→tsys4 + tsys7→tsys5"
 echo "  All running in parallel for 30 seconds"
 echo ""
 # Start 4 iperf3 servers (one-shot mode won't work for parallel; use persistent)
 for server in pfv-tsys4 pfv-tsys5; do
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -B ${SIP[$server]} >/dev/null 2>&1 &" 2>/dev/null
    echo "  [server started: $server]"
 done
 sleep 1
 STRESS_LOG="$LOG_DIR/stor-stress-4flow"
 mkdir -p "$STRESS_LOG"
 # Launch 4 clients in parallel, each writing to its own log
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys4.log" 2>&1 &
 PID1=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys6]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys6-to-tsys5.log" 2>&1 &
 PID2=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys4]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys4.log" 2>&1 &
 PID3=$!
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" "iperf3 -c ${SIP[pfv-tsys5]} -B ${SIP[pfv-tsys7]} -P 4 -t 30 -l 128k -O 2" > "$STRESS_LOG/tsys7-to-tsys5.log" 2>&1 &
 PID4=$!
 echo "  [4 clients launched, waiting 40s for completion...]"
 wait $PID1 $PID2 $PID3 $PID4 2>/dev/null
 echo "  [all 4 flows complete]"
 # Kill servers
 for server in pfv-tsys4 pfv-tsys5; do
    ssh "${SSH_OPTS[@]}" "root@$server" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 # ===========================================================================
 # SUITE C: All hosts cleanup
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " Cleanup: killing iperf3 everywhere"
 echo "==================================================================="
 for host in "${HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 # ===========================================================================
 # RESULTS SUMMARY
 # ===========================================================================
 echo ""
 echo "==================================================================="
 echo " RESULTS SUMMARY"
 echo "==================================================================="
 echo ""
 echo "===== SUITE A: Management network (vmbr0) ====="
 echo ""
 printf "%-40s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-40s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/mgmt-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    single=$(grep 'sender$' "$f" | tail -1)
    bitrate=$(echo "$single" | awk '{print $7, $8}')
    retrans=$(echo "$single" | awk '{print $9}')
    printf "%-40s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
 done
 echo ""
 echo "===== SUITE B.1+B.2: Storage network individual ====="
 echo ""
 printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-45s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/stor-indiv-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$f" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    printf "%-45s %15s %10s\n" "$label" "${bitrate:-?}" "${retrans:--}"
 done
 echo ""
 echo "===== SUITE B.3: Simultaneous 4-flow stress test ====="
 echo ""
 printf "%-30s %15s %10s\n" "FLOW" "THROUGHPUT" "RETRANS"
 printf "%-30s %15s %10s\n" "----" "----------" "-------"
 total_mbps=0
 for f in "$STRESS_LOG"/*.log; do
    [ -r "$f" ] || continue
    flow=$(basename "$f" .log)
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
        mbps=$(echo "$sum" | awk '{print $6}')
        total_mbps=$(awk "BEGIN{print $total_mbps + $mbps}")
    else
        bitrate="?"
        retrans="-"
    fi
    printf "%-30s %15s %10s\n" "$flow" "$bitrate" "${retrans:--}"
 done
 printf "%-30s %15s\n" "AGGREGATE (all 4 flows)" "${total_mbps} Mbits/sec"
 echo ""
 echo "===== CONTEXT ====="
 echo "tsys4: USB cdc_ncm dongle (single 1G link, no bond)"
 echo "tsys5: bond0 broken (1 active slave, no LACP partner) — cable pending"
 echo "tsys6/7: working 2x1G LACP, layer3+4 hash (host side)"
 echo "Cross-rack: 4x1G LACP (pfv-r3-tor-stor → pfv-core-sw01)"
 echo ""
 echo "All logs in: $LOG_DIR/"
@@ -0,0 +1,184 @@
 #!/bin/bash
 ###############################################################################
 # iperf-storage-tests.sh
 #
 # Installs iperf3 on all online hosts, then runs a matrix of storage-network
 # throughput tests. Saves all output to returned-logs/iperf/.
 #
 # Test matrix (all over VLAN1000 storage network, 10.100.100.0/24):
 #   1. tsys7 → tsys4 (USB cdc_ncm NIC) — the smoking gun
 #   2. tsys7 → tsys5 (bond0, 1 active slave) — PCI NIC comparison
 #   3. tsys7 → tsys6 (bond0, 2 active slaves) — working LACP baseline
 #   4. Reverse: tsys4 → tsys7 (USB NIC TX direction)
 #   5. Reverse: tsys5 → tsys7
 #
 # Each test: TCP 8-stream 30s forward + reverse + UDP saturation.
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 ALL_HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
 # Storage network IPs
 declare -A SIP
 SIP[pfv-tsys1]="10.100.100.1"
 SIP[pfv-tsys3]="10.100.100.3"
 SIP[pfv-tsys4]="10.100.100.4"
 SIP[pfv-tsys5]="10.100.100.5"
 SIP[pfv-tsys6]="10.100.100.6"
 SIP[pfv-tsys7]="10.100.100.7"
 echo "==================================================================="
 echo " STEP 1: Install iperf3 on all online hosts"
 echo "==================================================================="
 for host in "${ALL_HOSTS[@]}"; do
    echo -n "  [$host] "
    if ! ssh "${SSH_OPTS[@]}" "root@$host" 'echo ok' >/dev/null 2>&1; then
        echo "UNREACHABLE — skipping"
        continue
    fi
    # Check if iperf3 already installed
    if ssh "${SSH_OPTS[@]}" "root@$host" 'command -v iperf3 >/dev/null 2>&1' 2>/dev/null; then
        echo "iperf3 already installed"
    else
        printf "installing... "
        ssh "${SSH_OPTS[@]}" "root@$host" 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq iperf3 >/dev/null 2>&1 && echo OK || echo FAILED'
    fi
 done
 echo ""
 echo "==================================================================="
 echo " STEP 2: Kill any existing iperf3 processes everywhere"
 echo "==================================================================="
 for host in "${ALL_HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo "  Done."
 # Helper: run an iperf3 test and save output
 run_iperf() {
    local client="$1" server="$2" direction="$3" label="$4" logfile="$5"
    local client_ip="${SIP[$client]}" server_ip="${SIP[$server]}"
    echo -n "  [$client → $server] $label ... "
    # Start server in one-shot mode (-1 means serve one client then exit)
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
    # Run client
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client ($client_ip)"
        echo "Server: $server ($server_ip)"
        echo "Direction: $direction"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo ""
        if [ "$direction" = "forward" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2" 2>&1
        elif [ "$direction" = "reverse" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -P 8 -t 30 -l 128k -O 2 -R" 2>&1
        elif [ "$direction" = "udp" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -u -b 2G -t 10 -l 8972" 2>&1
        elif [ "$direction" = "single" ]; then
            ssh "${SSH_OPTS[@]}" "root@$client" \
                "iperf3 -c ${server_ip} -B ${client_ip} -t 20 -O 2" 2>&1
        fi
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract summary line
    if grep -q "sender" "$logfile"; then
        bitrate=$(grep "sender" "$logfile" | tail -1 | awk '{print $7, $8}')
        echo "done: ${bitrate}"
    else
        echo "done (check log for details)"
    fi
 }
 echo ""
 echo "==================================================================="
 echo " STEP 3: Run iperf3 test matrix"
 echo "==================================================================="
 echo ""
 echo "All tests over VLAN1000 storage network (10.100.100.0/24)."
 echo "TCP tests: 8 parallel streams, 30s, 128k blocks."
 echo ""
 # --- Test 1: tsys7 → tsys4 (USB cdc_ncm target) ---
 echo "--- TEST 1: tsys7 → tsys4 (USB cdc_ncm NIC) ---"
 run_iperf pfv-tsys7 pfv-tsys4 forward "TCP 8-stream forward (tsys7→tsys4 USB)" \
    "$LOG_DIR/01-tsys7-to-tsys4-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys7)" \
    "$LOG_DIR/02-tsys7-to-tsys4-tcp-reverse.log"
 run_iperf pfv-tsys7 pfv-tsys4 single "TCP single-stream forward (tsys7→tsys4 USB)" \
    "$LOG_DIR/03-tsys7-to-tsys4-tcp-single.log"
 run_iperf pfv-tsys7 pfv-tsys4 udp "UDP saturation (tsys7→tsys4 USB)" \
    "$LOG_DIR/04-tsys7-to-tsys4-udp.log"
 echo ""
 # --- Test 2: tsys7 → tsys5 (bond0, PCI NIC, 1 active slave) ---
 echo "--- TEST 2: tsys7 → tsys5 (PCI NIC, broken bond - 1 slave) ---"
 run_iperf pfv-tsys7 pfv-tsys5 forward "TCP 8-stream forward (tsys7→tsys5 PCI)" \
    "$LOG_DIR/05-tsys7-to-tsys5-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys5 reverse "TCP 8-stream reverse (tsys5 PCI→tsys7)" \
    "$LOG_DIR/06-tsys7-to-tsys5-tcp-reverse.log"
 run_iperf pfv-tsys7 pfv-tsys5 single "TCP single-stream forward (tsys7→tsys5 PCI)" \
    "$LOG_DIR/07-tsys7-to-tsys5-tcp-single.log"
 run_iperf pfv-tsys7 pfv-tsys5 udp "UDP saturation (tsys7→tsys5 PCI)" \
    "$LOG_DIR/08-tsys7-to-tsys5-udp.log"
 echo ""
 # --- Test 3: tsys7 → tsys6 (working 2-slave LACP baseline, layer3+4) ---
 echo "--- TEST 3: tsys7 → tsys6 (working 2×1G LACP baseline) ---"
 run_iperf pfv-tsys7 pfv-tsys6 forward "TCP 8-stream forward (tsys7→tsys6 LACP)" \
    "$LOG_DIR/09-tsys7-to-tsys6-tcp-forward.log"
 run_iperf pfv-tsys7 pfv-tsys6 reverse "TCP 8-stream reverse (tsys6 LACP→tsys7)" \
    "$LOG_DIR/10-tsys7-to-tsys6-tcp-reverse.log"
 echo ""
 # --- Test 4: tsys6 → tsys4 (pre-tuning baseline) ---
 echo "--- TEST 4: tsys6 → tsys4 (baseline before tsys6 tuning) ---"
 run_iperf pfv-tsys6 pfv-tsys4 forward "TCP 8-stream forward (tsys6→tsys4 USB)" \
    "$LOG_DIR/11-tsys6-to-tsys4-tcp-forward.log"
 run_iperf pfv-tsys6 pfv-tsys4 reverse "TCP 8-stream reverse (tsys4 USB→tsys6)" \
    "$LOG_DIR/12-tsys6-to-tsys4-tcp-reverse.log"
 echo ""
 # --- Cleanup: kill iperf3 everywhere ---
 echo "--- Cleanup ---"
 for host in "${ALL_HOSTS[@]}"; do
    ssh "${SSH_OPTS[@]}" "root@$host" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 echo "==================================================================="
 echo " RESULTS SUMMARY"
 echo "==================================================================="
 echo ""
 printf "%-45s %s\n" "TEST" "THROUGHPUT"
 printf "%-45s %s\n" "----" "----------"
 for f in "$LOG_DIR"/*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    bitrate=$(grep -E "sender$" "$f" | tail -1 | awk '{print $7, $8}')
    [ -z "$bitrate" ] && bitrate=$(grep -E "Mbits/sec|Gbits/sec" "$f" | tail -1 | grep -oE '[0-9.]+ [MG]bits/sec' | head -1)
    [ -z "$bitrate" ] && bitrate="(see log)"
    printf "%-45s %s\n" "$label" "$bitrate"
 done
 echo ""
 echo "Full logs saved to: $LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,119 @@
 #!/bin/bash
 # iperf-tsys6-tsys7.sh - validate 2Gbps LACP between the two tuned hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 mkdir -p "$LOG_DIR"
 TSYS6="10.100.100.6"
 TSYS7="10.100.100.7"
 run_test() {
    local client="$1" server="$2" server_ip="$3" label="$4" logfile="$5"
    shift 4
    local extra_args="$*"
    echo -n "  [$label] ... "
    # Start server in one-shot mode
    ssh "${SSH_OPTS[@]}" "root@$server" "pkill -x iperf3 2>/dev/null; nohup iperf3 -s -1 -B ${server_ip} >/dev/null 2>&1 &" 2>/dev/null
    sleep 1
    {
        echo "=== iperf3: $label ==="
        echo "Client: $client  Server: $server ($server_ip)"
        echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
        echo "Args: $extra_args"
        echo ""
        ssh "${SSH_OPTS[@]}" "root@$client" "iperf3 -c ${server_ip} $extra_args" 2>&1
        echo ""
        echo "=== END ==="
    } > "$logfile" 2>&1
    # Extract result
    sum=$(grep '\[SUM\].*sender$' "$logfile" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$logfile" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    echo "${bitrate} (retrans: ${retrans:-0})"
 }
 echo "==================================================================="
 echo " iperf3: tsys6 ↔ tsys7 (both have 2×1G LACP + layer3+4 hash)"
 echo " Expectation: ~1.8-2.0 Gbps for 8-stream TCP"
 echo "==================================================================="
 echo ""
 # Pre-flight: confirm bond state on both
 echo "--- bond0 state on tsys6 ---"
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
 echo ""
 echo "--- bond0 state on tsys7 ---"
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'grep -E "Transmit Hash|Number of ports|Bonding Mode" /proc/net/bonding/bond0'
 echo ""
 echo "--- Running tests ---"
 echo ""
 # Test 1: tsys7 → tsys6, 8-stream TCP forward
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP 8-stream forward" \
    "$LOG_DIR/tsys6-tsys7-01-tcp-8stream-forward.log" \
    "-P 8 -t 30 -l 128k -O 2"
 # Test 2: tsys6 → tsys7, 8-stream TCP forward (reverse direction)
 run_test pfv-tsys6 pfv-tsys7 "$TSYS7" \
    "tsys6→tsys7 TCP 8-stream forward" \
    "$LOG_DIR/tsys6-tsys7-02-tcp-8stream-forward.log" \
    "-P 8 -t 30 -l 128k -O 2"
 # Test 3: tsys7 → tsys6, single stream (should be ~940 Mbps — single flow)
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP single-stream" \
    "$LOG_DIR/tsys6-tsys7-03-tcp-single.log" \
    "-t 20 -O 2"
 # Test 4: tsys7 → tsys6, 4-stream (nconnect=4 mirrors this)
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 TCP 4-stream" \
    "$LOG_DIR/tsys6-tsys7-04-tcp-4stream.log" \
    "-P 4 -t 30 -l 128k -O 2"
 # Test 5: UDP saturation
 run_test pfv-tsys7 pfv-tsys6 "$TSYS6" \
    "tsys7→tsys6 UDP saturation" \
    "$LOG_DIR/tsys6-tsys7-05-udp.log" \
    "-u -b 3G -t 10 -l 8972"
 # Cleanup
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys6" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 ssh "${SSH_OPTS[@]}" "root@pfv-tsys7" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 echo ""
 echo "==================================================================="
 echo " SUMMARY"
 echo "==================================================================="
 echo ""
 printf "%-45s %15s %10s\n" "TEST" "THROUGHPUT" "RETRANS"
 printf "%-45s %15s %10s\n" "----" "----------" "-------"
 for f in "$LOG_DIR"/tsys6-tsys7-*.log; do
    [ -r "$f" ] || continue
    label=$(head -1 "$f" | sed 's/^=== iperf3: //; s/ ===$//')
    sum=$(grep '\[SUM\].*sender$' "$f" | tail -1)
    if [ -n "$sum" ]; then
        bitrate=$(echo "$sum" | awk '{print $6, $7}')
        retrans=$(echo "$sum" | awk '{print $8}')
    else
        single=$(grep 'sender$' "$f" | tail -1)
        bitrate=$(echo "$single" | awk '{print $7, $8}')
        retrans=$(echo "$single" | awk '{print $9}')
    fi
    printf "%-45s %15s %10s\n" "$label" "$bitrate" "${retrans:--}"
 done
 echo ""
 echo "Expected: 8-stream ~1.8-2.0 Gbps, single-stream ~940 Mbps"
@@ -0,0 +1,88 @@
 #!/bin/bash
 # reboot-and-verify.sh - reboots a host and verifies NFS nconnect activates.
 # Usage: bash reboot-and-verify.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 echo "================================================================"
 echo "[$HOST] PRE-REBOOT STATE"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" '
    echo "--- VMs ---"
    qm list 2>/dev/null
    echo "--- NFS TCP conns: $(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l) ---"
    echo "--- NFS first mount: ---"
    nfsstat -m 2>/dev/null | head -2 | tail -1
    echo "--- uptime ---"
    uptime
 '
 echo ""
 echo "================================================================"
 echo "[$HOST] ISSUING REBOOT"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &'
 echo "Reboot sent at $(date +%H:%M:%S)"
 echo ""
 echo "================================================================"
 echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 min)"
 echo "================================================================"
 DEADLINE=$(( $(date +%s) + 600 ))
 LAST_PRINT=0
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    now=$(date +%s)
    if [ $((now - LAST_PRINT)) -ge 15 ]; then
        printf '  [%s] waiting... (%ss elapsed)\n' "$(date +%H:%M:%S)" "$(( now - DEADLINE + 600 ))"
        LAST_PRINT=$now
    fi
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        # Verify uptime is actually low (host really rebooted, not still up)
        up_mins=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/uptime | awk "{print int(\$1/60)}"')
        if [ "${up_mins:-999}" -lt 5 ]; then
            echo "  [$(date +%H:%M:%S)] SSH back, uptime ${up_mins}min — real reboot confirmed"
            break
        fi
    fi
    sleep 10
 done
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "FAILED: $HOST not back after 10 minutes"
    exit 1
 fi
 echo "Waiting 30s for services to settle..."
 sleep 30
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 ssh "${SSH_OPTS[@]}" "root@$HOST" '
    echo "--- uptime ---"
    uptime
    echo ""
    echo "--- governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null) ---"
    echo "--- swappiness: $(sysctl -n vm.swappiness) ---"
    echo "--- tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control) ---"
    echo "--- tuned: $(tuned-adm active 2>/dev/null | grep Current) ---"
    echo ""
    echo "--- NFS first mount: ---"
    nfsstat -m 2>/dev/null | head -2 | tail -1
    echo ""
    echo "--- NFS TCP conns (expect 8 with nconnect=4): ---"
    ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l
    echo ""
    echo "--- VMs: ---"
    qm list 2>/dev/null
    echo ""
    echo "--- Failed services: ---"
    systemctl --failed --no-legend 2>/dev/null | head -5
    echo "(empty = none)"
 '
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
@@ -0,0 +1,91 @@
 #!/bin/bash
 # reboot-verify.sh - reboots a host, waits for it to come back, verifies state.
 # Usage: bash reboot-verify.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 echo "================================================================"
 echo "[$HOST] PRE-REBOOT STATE"
 echo "================================================================"
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR==1 || \$3==\"running\"{print}"' 2>&1
 echo "--- NFS mount count ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | grep -c "^/mnt"' 2>&1
 echo "--- NFS TCP connections to :2049 ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
 echo ""
 echo "================================================================"
 echo "[$HOST] ISSUING REBOOT"
 echo "================================================================"
 # Issue reboot; ssh will disconnect with non-zero — that's expected.
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nohup sh -c "(sleep 2; systemctl reboot)" >/dev/null 2>&1 &' 2>&1
 echo "Reboot command sent at $(date +%H:%M:%S). Host will drop now."
 echo ""
 echo "================================================================"
 echo "[$HOST] WAITING FOR SSH TO RETURN (max 10 minutes)"
 echo "================================================================"
 DEADLINE=$(( $(date +%s) + 600 ))
 LAST_PRINT=0
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    now=$(date +%s)
    # Print a heartbeat every 15s
    if [ $((now - LAST_PRINT)) -ge 15 ]; then
        elapsed=$((DEADLINE - now - 600)); elapsed=${elapsed#-}
        echo "  [$(date +%H:%M:%S)] still waiting... (${elapsed}s elapsed)"
        LAST_PRINT=$now
    fi
    # Try SSH
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  [$(date +%H:%M:%S)] SSH is back!"
        break
    fi
    sleep 5
 done
 # Final check
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "  [$(date +%H:%M:%S)] FAILED: host not reachable after 10 minutes"
    exit 1
 fi
 # Give services a moment to settle after SSH returns
 echo "  Waiting 20s for services to settle..."
 sleep 20
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 echo "--- Uptime ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime' 2>&1
 echo ""
 echo "--- TCP congestion control ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc' 2>&1
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness' 2>&1
 echo ""
 echo "--- scaling_governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"' 2>&1
 echo ""
 echo "--- NFS mount options (looking for nconnect + noatime) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -20' 2>&1
 echo ""
 echo "--- NFS TCP connection count (expect ~8 = 4 per server with nconnect=4) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l' 2>&1
 echo ""
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null' 2>&1
 echo ""
 echo "--- Failed services? ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10' 2>&1
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
@@ -0,0 +1,141 @@
 #!/bin/bash
 ###############################################################################
 # apply-bond-hash.sh
 #
 # Adds bond-xmit-hash-policy layer3+4 to bond0 in /etc/network/interfaces,
 # then reloads networking with ifreload -a.
 #
 # SSH survivability: this is safe IF your SSH session is on vmbr0/nic0
 # (management network), NOT on bond0/datanet (storage network).
 # tsys7's topology confirms this: SSH comes in on vmbr0 (nic0).
 #
 # Safety:
 #   - Dry-run by default (--apply to commit)
 #   - Full backup of /etc/network/interfaces
 #   - Generates rollback script
 #   - Does NOT reboot — uses ifreload -a which is hot-reload
 ###############################################################################
 set -euo pipefail
 HOST="$(hostname -s)"
 TS_SHORT="$(date +%Y%m%d-%H%M%S)"
 BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
 ROLLBACK="/root/perfopt-bond-rollback-${TS_SHORT}.sh"
 ACTION="${1:-dryrun}"
 [ "$ACTION" = "--apply" ] && ACTION="apply" || ACTION="dryrun"
 mkdir -p "$BACKUP_DIR"
 echo "==================================================================="
 echo " apply-bond-hash — $HOST"
 echo " mode: $ACTION"
 echo "==================================================================="
 if [ ! -r /etc/network/interfaces ]; then
    echo "FATAL: /etc/network/interfaces not readable"
    exit 1
 fi
 # Check if bond0 exists in the config
 if ! grep -q 'bond0' /etc/network/interfaces; then
    echo "No bond0 found in /etc/network/interfaces — nothing to do."
    exit 0
 fi
 # Check if the hash policy is already set
 if grep -q 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces; then
    echo "bond-xmit-hash-policy already present:"
    grep 'bond-xmit-hash-policy\|xmit_hash_policy' /etc/network/interfaces
    echo "Checking value..."
    if grep -q 'layer3+4' /etc/network/interfaces; then
        echo "Already set to layer3+4 — nothing to do."
        exit 0
    fi
 fi
 # Show current bond state
 echo ""
 echo "--- Current bond0 state ---"
 cat /proc/net/bonding/bond0 2>/dev/null | head -15 || echo "(bond0 not up or not present)"
 # Back up
 cp -a /etc/network/interfaces "$BACKUP_DIR/interfaces"
 # Generate rollback script
 cat > "$ROLLBACK" <<EOF
 #!/bin/bash
 # Rollback for bond-xmit-hash-policy change
 # Restores original /etc/network/interfaces and reloads
 set -euo pipefail
 cp -a "$BACKUP_DIR/interfaces" /etc/network/interfaces
 echo "Restored /etc/network/interfaces"
 echo "Reloading networking..."
 ifreload -a 2>&1 || systemctl restart networking 2>&1 || true
 echo "Done. bond0 hash policy reverted to original."
 EOF
 chmod +x "$ROLLBACK"
 echo ""
 echo "--- Proposed change ---"
 echo "Add line '	bond-xmit-hash-policy layer3+4' to the bond0 stanza."
 echo ""
 if [ "$ACTION" != "apply" ]; then
    echo "DRY RUN — no changes made."
    echo "To commit: bash $0 --apply"
    echo "Rollback script (pre-generated): $ROLLBACK"
    exit 0
 fi
 # Apply: use sed to insert bond-xmit-hash-policy after bond-mode line
 # The bond0 stanza looks like:
 #   auto bond0
 #   iface bond0 inet manual
 #       bond-slaves nic1 nic2
 #       bond-miimon 100
 #       bond-mode 802.3ad
 #
 # We insert after the bond-mode line.
 echo "Applying..."
 # Check if bond-mode line exists (various formats)
 if grep -qE '^\s*bond-mode\s+802.3ad' /etc/network/interfaces; then
    # Insert after bond-mode 802.3ad line
    sed -i '/^\s*bond-mode\s+802\.3ad/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode line."
 elif grep -qE '^\s*bond-mode\s+4' /etc/network/interfaces; then
    sed -i '/^\s*bond-mode\s+4/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4 after bond-mode 4 line."
 else
    echo "Could not find bond-mode line — inserting after bond-slaves line instead."
    sed -i '/^\s*bond-slaves/a\\tbond-xmit-hash-policy layer3+4' /etc/network/interfaces
 fi
 # Show the result
 echo ""
 echo "--- Updated bond0 stanza ---"
 awk '/^auto bond0/,/^$/' /etc/network/interfaces
 echo ""
 echo "--- Reloading networking (ifreload -a) ---"
 echo "SSH should survive (it's on vmbr0/nic0, not bond0)..."
 ifreload -a 2>&1 || {
    echo "ifreload failed, trying systemctl restart networking..."
    systemctl restart networking 2>&1
 }
 # Wait a moment for bond to renegotiate
 echo "Waiting 5s for LACP to renegotiate..."
 sleep 5
 echo ""
 echo "--- Post-change bond0 state ---"
 cat /proc/net/bonding/bond0 2>/dev/null | head -20
 echo ""
 echo "==================================================================="
 echo " DONE."
 echo " Backup: $BACKUP_DIR/interfaces"
 echo " Rollback: bash $ROLLBACK"
 echo "==================================================================="
@@ -0,0 +1,439 @@
 #!/bin/bash
 ###############################################################################
 # apply-tunings.sh
 #
 # Applies the Tier 0 host-side tunings identified in FINDINGS.md:
 #   1. Sets scaling_governor=performance on every CPU
 #   2. Sets vm.swappiness appropriately (1 on storage hosts, 10 on VM hosts)
 #   3. Enables tcp_bbr module + sets congestion control
 #   4. Sets net.core.rmem_max/wmem_max + tcp_rmem/wmem for high-BDP NFS
 #   5. Sets the recommended tuned-adm profile (network-throughput or virtual-host)
 #   6. Adds nconnect=4 + noatime to NFS client mounts (edits /etc/pve/storage.cfg)
 #   7. (does NOT touch bond0 xmit-hash — that requires ifreload/network
 #       restart which drops the host. Provided as a separate --emit-bond-patch
 #       flag that PRINTS the change but does not apply it.)
 #
 # Safety features:
 #   - Dry-run mode by default (--apply to commit)
 #   - Full backup of every modified file to /root/perfopt-backup-<timestamp>/
 #   - Per-host behaviour: detects storage hosts by hostname and applies the
 #     right profile (tsys4, tsys5 = storage; others = VM hosts).
 #   - Generates a /root/perfopt-rollback.sh that undoes everything.
 #   - Does NOT reboot, does NOT restart networking, does NOT touch hardware.
 #
 # Usage:
 #   bash apply-tunings.sh             # dry run, show what would change
 #   bash apply-tunings.sh --apply     # commit changes
 #   bash apply-tunings.sh --rollback  # restore from latest backup
 #   bash apply-tunings.sh --emit-bond-patch   # show bond0 hash change (no apply)
 ###############################################################################
 set -u
 umask 022
 HOST="$(hostname -s)"
 TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 TS_SHORT="$(date +%Y%m%d-%H%M%S)"
 BACKUP_DIR="/root/perfopt-backup-${TS_SHORT}"
 ROLLBACK_SCRIPT="/root/perfopt-rollback-${TS_SHORT}.sh"
 ACTION="dryrun"
 SKIP_NFS=0
 while [ $# -gt 0 ]; do
    case "$1" in
        --apply)         ACTION="apply" ;;
        --rollback)      ACTION="rollback" ;;
        --emit-bond-patch) ACTION="bond-patch" ;;
        --no-nfs)        SKIP_NFS=1 ;;
        -h|--help)       sed -n '2,35p' "$0"; exit 0 ;;
        *) echo "unknown arg: $1" >&2; exit 2 ;;
    esac
    shift
 done
 # Detect host role from hostname
 case "$HOST" in
    pfv-tsys4|pfv-tsys5)
        HOST_ROLE="storage"
        TUNED_PROFILE="network-throughput"
        SWAPPINESS="1"
        ;;
    pfv-tsys1|pfv-tsys3|pfv-tsys6|pfv-tsys7)
        HOST_ROLE="vmhost"
        TUNED_PROFILE="virtual-host"
        SWAPPINESS="10"
        ;;
    *)
        HOST_ROLE="unknown"
        TUNED_PROFILE="virtual-host"
        SWAPPINESS="10"
        ;;
 esac
 # --- rollback path ---------------------------------------------------------
 if [ "$ACTION" = "rollback" ]; then
    echo "Looking for latest backup under /root/perfopt-backup-* ..."
    latest=""
    while IFS= read -r d; do
        latest="$d"
    done < <(find /root -maxdepth 1 -type d -name 'perfopt-backup-*' 2>/dev/null | sort | tail -n1)
    if [ -z "$latest" ]; then
        echo "No backup found under /root/perfopt-backup-*" >&2
        exit 1
    fi
    rb="$latest/perfopt-rollback.sh"
    if [ ! -x "$rb" ] && [ ! -r "$rb" ]; then
        echo "Rollback script missing in $latest" >&2
        exit 1
    fi
    echo "Rolling back using: $rb"
    bash "$rb"
    exit $?
 fi
 # --- bond patch print-only path -------------------------------------------
 if [ "$ACTION" = "bond-patch" ]; then
    echo "==================================================================="
    echo " Proposed bond0 xmit_hash_policy change"
    echo "==================================================================="
    echo
    if [ ! -r /etc/network/interfaces ]; then
        echo "/etc/network/interfaces not readable"
        exit 1
    fi
    if ! grep -q 'bond-mode 802.3ad\|bond-mode 4\|bond-slaves' /etc/network/interfaces; then
        echo "No bond0 detected on this host — nothing to patch."
        exit 0
    fi
    cat <<EOF
 A manual edit to /etc/network/interfaces is required. The bond0 stanza
 needs this line added (it defaults to layer2, which is wrong for storage):
    bond-xmit-hash-policy layer3+4
 After editing, you MUST reload networking for it to take effect:
    ifreload -a            # safe, brings interfaces down/up
    # OR
    systemctl restart networking   # heavier, briefly drops connections
 WARNING: applying this on a remote host over bond0 will briefly drop your
 SSH session. Run from console/IPMI, or schedule a maintenance window.
 The change is reversible by removing the line and reloading again.
 EOF
    exit 0
 fi
 # --- main path (dryrun or apply) ------------------------------------------
 mkdir -p "$BACKUP_DIR"
 # Emit the rollback script header
 cat > "$ROLLBACK_SCRIPT" <<EOF
 #!/bin/bash
 # Auto-generated rollback for perfopt apply-tunings.sh
 # Backup timestamp: $TS
 # Backup dir:       $BACKUP_DIR
 # Generated on:     $HOST
 set -u
 EOF
 chmod +x "$ROLLBACK_SCRIPT"
 backup_file() {
    local f="$1"
    if [ -e "$f" ]; then
        cp -a "$f" "$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
    fi
 }
 # Helper: append a restore command to the rollback script
 rb_restore() {
    local f="$1"
    local bk
    bk="$BACKUP_DIR/$(echo "$f" | sed 's|^/||; s|/|__|g')"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 if [ -r "$bk" ]; then
    cp -a "$bk" "$f" && echo "restored $f"
 else
    echo "WARN: backup $bk missing — $f left as-is"
 fi
 EOF
 }
 # Helper: append a sysctl restore command
 rb_sysctl_restore() {
    local key="$1" orig_val="$2"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 sysctl -w "$key=$orig_val" >/dev/null && echo "restored $key=$orig_val"
 EOF
 }
 # Helper: append a tuned-adm restore
 rb_tuned_restore() {
    local profile="$1"
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 tuned-adm profile "$profile" 2>/dev/null && echo "restored tuned profile: $profile"
 EOF
 }
 echo "==================================================================="
 echo " perfopt apply-tunings — $HOST"
 echo " mode: $ACTION"
 echo " role: $HOST_ROLE (tuned=$TUNED_PROFILE, swappiness=$SWAPPINESS)"
 echo " backup: $BACKUP_DIR"
 echo " rollback: $ROLLBACK_SCRIPT"
 echo "==================================================================="
 echo
 # ===========================================================================
 # 1. scaling_governor → performance
 # ===========================================================================
 echo "--- 1. CPU scaling_governor → performance"
 if [ -r /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ]; then
    cur=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor)
    echo "    current: $cur"
    if [ "$ACTION" = "apply" ]; then
        if [ "$cur" != "performance" ]; then
            for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
                echo performance > "$c" 2>/dev/null || true
            done
            # Persist via systemd tmpfiles / sysctl fallback
            cat > /etc/systemd/system/perfopt-cpu-performance.service <<EOF
 [Unit]
 Description=Set CPU scaling_governor=performance (perfopt)
 After=multi-user.target
 [Service]
 Type=oneshot
 ExecStart=/bin/sh -c 'for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "\$c" 2>/dev/null || true; done'
 RemainAfterExit=yes
 [Install]
 WantedBy=multi-user.target
 EOF
            systemctl daemon-reload
            systemctl enable perfopt-cpu-performance.service 2>/dev/null
            systemctl start perfopt-cpu-performance.service 2>/dev/null
            echo "    applied + persisted via systemd unit"
            # Rollback
            cat >> "$ROLLBACK_SCRIPT" <<EOF
 for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "$cur" > "\$c" 2>/dev/null || true; done
 systemctl disable --now perfopt-cpu-performance.service 2>/dev/null || true
 rm -f /etc/systemd/system/perfopt-cpu-performance.service
 systemctl daemon-reload
 echo "restored scaling_governor=$cur (best-effort; original may have been dynamic)"
 EOF
        else
            echo "    already performance — no change"
        fi
    fi
 else
    echo "    cpufreq driver not loaded — nothing to do (typical on BIOS-locked servers)"
 fi
 # ===========================================================================
 # 2. vm.swappiness
 # ===========================================================================
 echo "--- 2. vm.swappiness → $SWAPPINESS (role=$HOST_ROLE)"
 cur_swappiness=$(sysctl -n vm.swappiness 2>/dev/null || echo "?")
 echo "    current: $cur_swappiness"
 if [ "$ACTION" = "apply" ] && [ "$cur_swappiness" != "$SWAPPINESS" ]; then
    # Persist via sysctl.d
    backup_file /etc/sysctl.d/99-perfopt.conf
    cat > /etc/sysctl.d/99-perfopt.conf <<EOF
 # perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
 vm.swappiness = $SWAPPINESS
 EOF
    sysctl -w "vm.swappiness=$SWAPPINESS" >/dev/null
    echo "    applied (persisted to /etc/sysctl.d/99-perfopt.conf)"
    rb_restore /etc/sysctl.d/99-perfopt.conf
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 rm -f /etc/sysctl.d/99-perfopt.conf
 sysctl -w "vm.swappiness=$cur_swappiness" >/dev/null
 echo "restored vm.swappiness=$cur_swappiness"
 EOF
 else
    echo "    no change"
 fi
 # ===========================================================================
 # 3. TCP BBR + buffers  (single sysctl.d file)
 # ===========================================================================
 echo "--- 3. TCP BBR + socket buffers"
 cur_cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null || echo "?")
 cur_avail=$(sysctl -n net.ipv4.tcp_available_congestion_control 2>/dev/null || echo "?")
 echo "    current: $cur_cc (available: $cur_avail)"
 # Decide sysctl values (these are the consensus values for a 1-10 GbE NFS host)
 TARGET_RMEM_MAX=134217728        # 128 MB
 TARGET_WMEM_MAX=134217728
 TARGET_RMEM_DEFAULT=26214400     # 25 MB
 TARGET_WMEM_DEFAULT=26214400
 TARGET_TCP_RMEM="4096 87380 $TARGET_RMEM_MAX"
 TARGET_TCP_WMEM="4096 65536 $TARGET_RMEM_MAX"
 TARGET_NETDEV_MAX_BACKLOG=250000
 TARGET_SOMAXCONN=65535
 if [ "$ACTION" = "apply" ]; then
    # Ensure tcp_bbr module loads at boot
    if ! grep -q '^tcp_bbr' /etc/modules-load.d/modules.conf 2>/dev/null; then
        backup_file /etc/modules-load.d/modules.conf
        mkdir -p /etc/modules-load.d
        cat >> /etc/modules-load.d/modules.conf <<EOF
 # perfopt apply-tunings.sh — $TS
 tcp_bbr
 EOF
        modprobe tcp_bbr 2>/dev/null || true
        rb_restore /etc/modules-load.d/modules.conf
    fi
    backup_file /etc/sysctl.d/99-perfopt.conf
    cat > /etc/sysctl.d/99-perfopt.conf <<EOF
 # perfopt apply-tunings.sh — $TS — host=$HOST role=$HOST_ROLE
 vm.swappiness = $SWAPPINESS
 # TCP BBR + high-BDP buffers (good for NFS over 1-10 GbE)
 net.ipv4.tcp_congestion_control = bbr
 net.core.default_qdisc = fq
 net.core.rmem_max = $TARGET_RMEM_MAX
 net.core.wmem_max = $TARGET_WMEM_MAX
 net.core.rmem_default = $TARGET_RMEM_DEFAULT
 net.core.wmem_default = $TARGET_WMEM_DEFAULT
 net.core.netdev_max_backlog = $TARGET_NETDEV_MAX_BACKLOG
 net.core.somaxconn = $TARGET_SOMAXCONN
 net.ipv4.tcp_rmem = $TARGET_TCP_RMEM
 net.ipv4.tcp_wmem = $TARGET_TCP_WMEM
 EOF
    sysctl --system >/dev/null 2>&1
    echo "    applied (written to /etc/sysctl.d/99-perfopt.conf + sysctl --system)"
    # Rebuild rollback for sysctl
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 # Restore original sysctl state
 rm -f /etc/sysctl.d/99-perfopt.conf
 EOF
    rb_sysctl_restore "net.ipv4.tcp_congestion_control" "$cur_cc"
    rb_sysctl_restore "vm.swappiness" "$cur_swappiness"
    # Note: original buffer sizes not all captured; restoration is best-effort.
    cat >> "$ROLLBACK_SCRIPT" <<EOF
 echo "Note: net buffers restored to kernel defaults (original values not captured)."
 echo "Run 'sysctl --system' to apply."
 sysctl --system >/dev/null 2>&1 || true
 EOF
 fi
 # ===========================================================================
 # 4. tuned-adm profile
 # ===========================================================================
 echo "--- 4. tuned-adm profile → $TUNED_PROFILE"
 if command -v tuned-adm >/dev/null 2>&1; then
    cur_profile=$(tuned-adm active 2>/dev/null | awk -F: '/Current active profile/{gsub(/^[ \t]+/,"",$2); print $2}')
    echo "    current: $cur_profile"
    if [ "$ACTION" = "apply" ] && [ "$cur_profile" != "$TUNED_PROFILE" ]; then
        tuned-adm profile "$TUNED_PROFILE" 2>&1 | sed 's/^/    /'
        rb_tuned_restore "$cur_profile"
    else
        echo "    no change"
    fi
 else
    echo "    tuned-adm not installed — skipping"
 fi
 # ===========================================================================
 # 5. NFS mount option hardening
 #    Edit /etc/pve/storage.cfg to add nconnect=4 + noatime to NFS plugins.
 #    Proxmox applies these on next mount/remount.
 #    Skipped with --no-nfs (for NFS servers where we don't want to remount).
 # ===========================================================================
 echo "--- 5. NFS mount options: add nconnect=4, noatime"
 if [ "$SKIP_NFS" -eq 1 ]; then
    echo "    SKIPPED (--no-nfs specified)"
    echo "    Note: storage.cfg is cluster-wide; if edited on another host,"
    echo "    the options are already staged here and activate on next reboot."
 else
 if [ -r /etc/pve/storage.cfg ]; then
    backup_file /etc/pve/storage.cfg
    if grep -q '^nfs:' /etc/pve/storage.cfg; then
        # Count how many nfs: stanzas lack the options
        while IFS= read -r line; do
            if echo "$line" | grep -q '^nfs:'; then
                cur_stanza=$(echo "$line" | awk '{print $2}')
                # Look ahead for the next few lines to see if options already set
                echo "    nfs stanza: $cur_stanza"
            fi
        done < /etc/pve/storage.cfg
        # Check whether any nfs stanza already has options
        if grep -A1 '^nfs:' /etc/pve/storage.cfg | grep -q 'options.*nconnect=4'; then
            echo "    some stanzas already have nconnect=4 — check manually"
        else
            echo "    proposed change: add 'options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2' to each nfs stanza"
            if [ "$ACTION" = "apply" ]; then
                # Apply via perl for safety (in-place edit with backup)
                if cp -a /etc/pve/storage.cfg "$BACKUP_DIR/etc__pve__storage.cfg"; then
                    # Use a python helper for robust PVE storage.cfg editing
                    python3 - <<PYEOF
 import re, pathlib
 p = pathlib.Path("/etc/pve/storage.cfg")
 text = p.read_text()
 pattern = re.compile(r'(^nfs:\s*\S+\n(?:[ \t]+[^\n]+\n)+)', re.MULTILINE)
 modified = 0
 def fix(m):
    global modified
    block = m.group(1)
    if 'options' in block:
        return block
    lines = block.rstrip('\n').split('\n')
    lines.insert(1, '\toptions nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2')
    modified += 1
    return '\n'.join(lines) + '\n'
 new_text = pattern.sub(fix, text)
 if modified:
    p.write_text(new_text)
    print(f"    patched {modified} NFS stanzas with options line")
 else:
    print("    no NFS stanza needed patching (all already had options or no nfs: found)")
 PYEOF
                    # Remount existing NFS mounts to pick up new options
                    echo "    remounting NFS mounts to apply new options..."
                    while IFS= read -r mp; do
                        mount -o remount "$mp" 2>/dev/null && echo "      remounted $mp" || echo "      FAILED to remount $mp (will pick up on next mount)"
                    done < <(awk '$3 ~ /^nfs/{print $2}' /proc/mounts 2>/dev/null | sort -u)
                    rb_restore /etc/pve/storage.cfg
                    cat >> "$ROLLBACK_SCRIPT" <<EOF
 # To fully undo NFS options, also remount:
 while IFS= read -r mp; do
    mount -o remount "\$mp" 2>/dev/null
 done < <(awk '\$3 ~ /^nfs/{print \$2}' /proc/mounts 2>/dev/null | sort -u)
 echo "restored NFS mount options"
 EOF
                else
                    echo "    ERROR: could not back up storage.cfg — aborting NFS patch"
                fi
            fi
        fi
    else
        echo "    no NFS stanzas in storage.cfg — skipping"
    fi
 else
    echo "    /etc/pve/storage.cfg not readable — skipping"
 fi
 fi
 # ===========================================================================
 # 6. Summary
 # ===========================================================================
 echo
 echo "==================================================================="
 if [ "$ACTION" = "apply" ]; then
    echo " DONE. Backups in: $BACKUP_DIR"
    echo " Rollback: bash $ROLLBACK_SCRIPT"
    echo
    echo " Next steps:"
    echo "   - Verify with 'sysctl net.ipv4.tcp_congestion_control vm.swappiness'"
    echo "   - Verify NFS mounts with 'nfsstat -m' (look for nconnect=4)"
    echo "   - Run 'bash apply-tunings.sh --emit-bond-patch' for the bond0"
    echo "     xmit_hash_policy change (requires network restart, do in"
    echo "     maintenance window)"
 else
    echo " DRY RUN — no changes made."
    echo " To commit: bash apply-tunings.sh --apply"
 fi
 echo "==================================================================="
@@ -0,0 +1,86 @@
 #!/bin/bash
 # fix-tsys7.sh - fixes the two issues found: NFS options string + bond hash
 # Runs on the target host directly.
 set -euo pipefail
 echo "==================================================================="
 echo " FIX: NFS options + bond hash policy"
 echo "==================================================================="
 # --- FIX 1: Simplify NFS options in storage.cfg ---------------------------
 echo ""
 echo "--- FIX 1: Simplify NFS options (remove version=4.2 conflict) ---"
 # Replace the overly-complex options line with a minimal one
 # PVE handles vers/rsize/wsize/hard/etc internally; we only need nconnect + noatime
 if grep -q 'options.*nconnect=4.*version=4.2' /etc/pve/storage.cfg; then
    # Use sed to replace each options line
    sed -i 's/options nconnect=4,noatime,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,version=4.2/options nconnect=4,noatime/' /etc/pve/storage.cfg
    echo "Fixed: simplified NFS options to 'nconnect=4,noatime'"
 else
    echo "Options line already simplified or not present"
 fi
 echo ""
 echo "--- Verify storage.cfg NFS stanzas ---"
 grep -A 2 "^nfs: D2" /etc/pve/storage.cfg | head -3
 echo "..."
 grep -c "options nconnect" /etc/pve/storage.cfg
 echo " NFS stanzas with options"
 # --- FIX 2: Bond hash policy (force via sysfs + persist in interfaces) ----
 echo ""
 echo "--- FIX 2: Apply bond xmit_hash_policy=layer3+4 ---"
 # 2a: Apply LIVE via sysfs (takes effect immediately, no network reload)
 echo "Applying live via sysfs..."
 if echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy 2>/dev/null; then
    echo "Live sysfs apply: SUCCESS"
 else
    echo "Live sysfs apply: FAILED (will persist in config and apply on ifreload)"
 fi
 # Verify live state
 echo ""
 echo "Live bond0 hash policy:"
 cat /proc/net/bonding/bond0 | grep "Transmit Hash"
 # 2b: Persist in /etc/network/interfaces (fix the sed that failed before)
 echo ""
 echo "Persisting in /etc/network/interfaces..."
 # Backup
 cp -a /etc/network/interfaces "/root/interfaces.bondfix.$(date +%Y%m%d%H%M%S)"
 # Check if already present
 if grep -q 'bond-xmit-hash-policy' /etc/network/interfaces; then
    echo "bond-xmit-hash-policy already in interfaces file"
 else
    # Use awk to insert after the bond-mode line (more reliable than sed)
    # Match any line containing 'bond-mode' (regardless of indentation)
    awk '
        /bond-mode/ && !done {
            print
            print "\tbond-xmit-hash-policy layer3+4"
            done=1
            next
        }
        { print }
    ' /etc/network/interfaces > /etc/network/interfaces.new
    mv /etc/network/interfaces.new /etc/network/interfaces
    echo "Inserted bond-xmit-hash-policy layer3+4"
 fi
 echo ""
 echo "--- Updated bond0 stanza ---"
 awk '/^auto bond0/,/^$/' /etc/network/interfaces
 # Final verify
 echo ""
 echo "--- Final bond0 running state ---"
 cat /proc/net/bonding/bond0 | head -20
 echo ""
 echo "==================================================================="
 echo " FIX COMPLETE"
 echo "==================================================================="
@@ -0,0 +1,217 @@
 #!/usr/bin/env bash
 ###############################################################################
 # lacp-retrans-cause.sh  (HOST-NATIVE)
 #
 # Runs on tsys6 (receiver) or tsys7 (sender). Auto-detects role.
 #
 # Question this answers: are the ~56K retransmits we see on 8-stream iperf
 # between tsys6 and tsys7 fixable (NIC ring drops, softnet drops, CPU
 # saturation) or just unavoidable LACP reordering overhead?
 #
 # Method: snapshot drop/error counters + softnet_stat + TCP SNMP before and
 # after a 12s iperf3 run, then print the deltas. The decisive columns are:
 #   - NIC rx_dropped / rx_missed_errors / rx_no_dma_resources   -> ring too small
 #   - /proc/net/softnet_stat drops                              -> ksoftirq starved
 #   - /proc/net/snmp TCP retranst                               -> TCP-level retrans
 # If NIC drop counters do NOT climb but TCP retrans does, the retrans are
 # coming from LACP reordering (out-of-order segments triggering fast
 # retransmit), not packet loss — and are NOT fixable by tuning.
 ###############################################################################
 set -uo pipefail
 DURATION="${LACP_TEST_SECS:-12}"
 STREAMS="${LACP_STREAMS:-8}"
 PEER_RX_IP="10.100.100.6"   # tsys6 storage IP (receiver)
 PEER_TX_IP="10.100.100.7"   # tsys7 storage IP (sender)
 LOG_DIR="/root"
 BOND="bond0"
 HOST="$(uname -n)"; HOST="${HOST%%.*}"
 case "$HOST" in
    *tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-retrans-receiver.log" ;;
    *tsys7) ROLE="sender";   LOG="$LOG_DIR/lacp-retrans-sender.log" ;;
    *) echo "ERROR: not on tsys6 or tsys7 (host=$HOST)"; exit 2 ;;
 esac
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 # ---- per-slave NIC counter snapshot -----------------------------------------
 # Captures rx_dropped, rx_missed_errors, rx_no_dma_resources, tx_retrans,
 # plus all error-like counters. Falls back gracefully if a counter doesn't
 # exist (different NIC drivers expose different names).
 nic_snapshot() {
    for s in $(awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND"); do
        [ -n "$s" ] || continue
        echo "[$s]"
        # ethtool -S may exist; show only drop/error/retrans lines
        if command -v ethtool >/dev/null 2>&1; then
            ethtool -S "$s" 2>/dev/null \
                | grep -iE 'drop|miss|error|no_dma|fifo|retrans|overflow' \
                || echo "(ethtool -S: no matching counters or unsupported)"
        else
            echo "(ethtool not installed)"
        fi
        # /sys counters (always available, driver-agnostic)
        for c in rx_dropped tx_dropped rx_errors tx_errors rx_missed_errors \
                 rx_length_errors rx_crc_errors rx_fifo_errors tx_fifo_errors \
                 multicast collisions; do
            v=$(cat "/sys/class/net/$s/statistics/$c" 2>/dev/null)
            [ -n "$v" ] && printf '  sysfs %-22s = %s\n' "$c" "$v"
        done
    done
 }
 # ---- softnet_stat snapshot (per-CPU RX softirq drops) -----------------------
 # /proc/net/softnet_stat columns are HEX. Col1=processed, col2=dropped,
 # col3=time_squeeze. We only care about dropped + squeeze (small numbers that
 # look identical in hex and decimal). Processed is informational only and we
 # don't try to delta it (hex arithmetic is awk-version-dependent).
 softnet_snapshot() {
    awk '{ printf "cpu%s processed=%s dropped=%s squeezed=%s\n", \
              NR-1, $1, $2, $3 }' /proc/net/softnet_stat
 }
 # ---- TCP SNMP snapshot (the source of iperf's retransmit number) -----------
 tcp_snapshot() {
    awk '/^Tcp:/{
        if (!seen) {
            seen=1
            # /proc/net/snmp Tcp line 1 = names, line 2 = values
            n=split($0, names, " ")
            # re-find the values line
            getline
            vals=$0
            split(vals, v, " ")
            for (i=1;i<=n;i++) printf "  %-22s = %s\n", names[i], v[i]
        }
    }' /proc/net/snmp
 }
 # ---- print deltas for selected counters ------------------------------------
 # $1 = before file, $2 = after file
 # softnet_snapshot output: "cpuN processed=HEX dropped=HEX squeezed=HEX"
 # /proc/net/softnet_stat is hex, but dropped/squeezed are always small
 # integers (typically 0 on a healthy host), so +0 coercion is correct for
 # the values we care about. We deliberately DON'T report `processed` deltas
 # because hex arithmetic on large numbers is awk-version-dependent.
 softnet_delta() {
    awk '
        FNR==NR { if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
                  split($0, p, /[= ]+/); drop[id]=p[5]+0; sqz[id]=p[7]+0 }
                  next }
        { if (match($0,/cpu[0-9]+/)) { id=substr($0,RSTART,RLENGTH);
              split($0, q, /[= ]+/);
              dd = (q[5]+0) - drop[id]
              sd = (q[7]+0) - sqz[id]
              if (dd != 0 || sd != 0)
                  printf "  %-8s dropped_delta=%-6s squeezed_delta=%-6s\n", id, dd, sd } }
    ' "$1" "$2" | sort
 }
 # $1 = before file, $2 = after file, $3 = section label prefix
 # File format: slave header line "[nic1]", then "  counter_name = value" lines.
 # Output only counters whose value changed.
 nic_delta() {
    awk '
        FNR==NR { if ($0 ~ /^\[/) slave=$0
                  else if ($0 ~ /=/) {
                      # Everything before " = " is the counter name (trim spaces)
                      pos = index($0, "=")
                      name = substr($0, 1, pos-1)
                      gsub(/^ +| +$/, "", name)
                      val  = substr($0, pos+1); gsub(/^ +| +$/, "", val)
                      before[slave SUBSEP name] = val
                  }
                  next }
        { if ($0 ~ /^\[/) slave=$0
          else if ($0 ~ /=/) {
              pos = index($0, "=")
              name = substr($0, 1, pos-1); gsub(/^ +| +$/, "", name)
              val  = substr($0, pos+1); gsub(/^ +| +$/, "", val)
              b = before[slave SUBSEP name]+0
              d = val+0 - b
              if (d != 0) printf "  %-8s %-30s %12s -> %12s  delta=%d\n", slave, name, b, val, d
          } }
    ' "$1" "$2"
 }
 {
 echo "=== LACP retransmit cause investigation ($ROLE) ==="
 echo "Host: $HOST   Role: $ROLE"
 echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
 echo "Bond: $BOND"
 echo ""
 echo "--- bond0 hash + driver ---"
 grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac" /proc/net/bonding/$BOND
 for s in $(awk '/^Slave Interface:/{print $3}' /proc/net/bonding/$BOND); do
    drv=$(ethtool -i "$s" 2>/dev/null | awk -F: '/^driver:/{print $2}' | sed 's/^ *//')
    speed=$(cat /sys/class/net/$s/speed 2>/dev/null)
    ring=$(ethtool -g "$s" 2>/dev/null | awk '/RX:/{print $2; exit}')
    printf "  %-8s driver=%-20s speed=%-6s current RX ring=%s\n" "$s" "$drv" "$speed" "$ring"
 done
 echo ""
 } | tee "$LOG"
 echo "--- BEFORE snapshots ---"
 nic_snapshot      > "$TMP/nic_before"
 softnet_snapshot  > "$TMP/softnet_before"
 tcp_snapshot      > "$TMP/tcp_before"
 if [ "$ROLE" = "receiver" ]; then
    echo "[receiver] starting iperf3 -s -1 on $PEER_RX_IP ..."
    pkill -x iperf3 2>/dev/null; sleep 0.5
    nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
    SRV_PID=$!
    for _ in $(seq 1 60); do kill -0 "$SRV_PID" 2>/dev/null || break; sleep 1; done
    wait "$SRV_PID" 2>/dev/null
 else
    echo "[sender] waiting 3s for receiver to listen, then running iperf3 -P $STREAMS -t $DURATION ..."
    sleep 3
    iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
        > "$TMP/iperf.out" 2>&1
 fi
 echo "--- AFTER snapshots ---"
 nic_snapshot      > "$TMP/nic_after"
 softnet_snapshot  > "$TMP/softnet_after"
 tcp_snapshot      > "$TMP/tcp_after"
 {
 echo ""
 echo "--- iperf3 summary ---"
 grep -E '\[SUM\].*(sender|receiver)' "$TMP/iperf.out" | tail -2
 echo ""
 echo "--- TCP SNMP deltas (/proc/net/snmp) ---"
 # Show only the counters that changed
 paste "$TMP/tcp_before" "$TMP/tcp_after" \
    | awk '{
        a=$3; b=$(NF)
        if (a+0 != b+0) printf "  %-22s %12s -> %12s  delta=%d\n", $1, a, b, b-a
    }'
 echo ""
 echo "--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---"
 softnet_delta "$TMP/softnet_before" "$TMP/softnet_after"
 echo ""
 echo "--- NIC counter deltas (only non-zero) ---"
 nic_delta "$TMP/nic_before" "$TMP/nic_after" "$HOST"
 echo ""
 echo "--- raw iperf3 tail (last 12 lines) ---"
 tail -12 "$TMP/iperf.out"
 echo ""
 echo "==================================================================="
 echo " HOW TO READ THIS"
 echo "==================================================================="
 echo " - If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:"
 echo "     -> ring buffers too small. Fix: ethtool -G \$IF rx 4096 (or max)."
 echo " - If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:"
 echo "     -> softirq starvation. Fix: increase net.core.netdev_budget,"
 echo "        check IRQ affinity, consider RPS."
 echo " - If NEITHER of the above climbed but TCP RetransSegs did:"
 echo "     -> the retransmits are LACP reordering (out-of-order segments"
 echo "        triggering fast retransmit), NOT packet loss. NOT fixable."
 echo "==================================================================="
 } | tee -a "$LOG"
 echo ""
 echo "Log: $LOG"
@@ -0,0 +1,153 @@
 #!/usr/bin/env bash
 ###############################################################################
 # lacp-rx-distribution.sh  (HOST-NATIVE version)
 #
 # Runs ON a host (tsys6 or tsys7). Auto-detects role by hostname:
 #   - On tsys6 (receiver): starts iperf3 -s -1 (one-shot), snapshots local
 #     bond0 slave RX counters before+after, writes verdict + log.
 #   - On tsys7 (sender):   snapshots local bond0 slave TX counters before+
 #     after, runs iperf3 -c <peer> -P 8 -t 12.
 #
 # Each side writes only its own counters. No inter-host SSH required.
 # iperf3 -s -1 (one-shot) handles client/server coordination.
 #
 # Output: /root/lacp-rx-{receiver,sender}.log
 ###############################################################################
 set -uo pipefail
 DURATION="${LACP_TEST_SECS:-12}"
 STREAMS="${LACP_STREAMS:-8}"
 PEER_RX_IP="10.100.100.6"   # tsys6 storage IP (receiver)
 PEER_TX_IP="10.100.100.7"   # tsys7 storage IP (sender)
 LOG_DIR="/root"
 BOND="bond0"
 # Snapshot per-slave byte counters from $BOND. Output: "<slave> rx tx\n" sorted.
 snapshot() {
    awk '/^Slave Interface:/{print $3}' "/proc/net/bonding/$BOND" | while read -r s; do
        [ -n "$s" ] || continue
        rx=$(cat "/sys/class/net/$s/statistics/rx_bytes" 2>/dev/null || echo 0)
        tx=$(cat "/sys/class/net/$s/statistics/tx_bytes" 2>/dev/null || echo 0)
        printf '%s %s %s\n' "$s" "$rx" "$tx"
    done | sort
 }
 # Compute per-slave deltas. $1=label, $2=col(2=rx,3=tx), $3=before, $4=after.
 analyze() {
    local label="$1" col="$2" before="$3" after="$4"
    join "$before" "$after" | awk -v col="$col" -v lbl="$label" '
        {
            slave=$1
            b = (col==2 ? $2 : $3)+0
            a = (col==2 ? $4 : $5)+0
            d = a - b; if (d < 0) d = 0
            delta[slave]=d; bef[slave]=b; aft[slave]=a
            order[++n]=slave; total+=d
        }
        END {
            printf "\n--- %s (bytes) ---\n", lbl
            printf "  %-12s %14s %14s %14s %8s\n", "SLAVE", "BEFORE", "AFTER", "DELTA", "PCT"
            for (i=1;i<=n;i++){
                s=order[i]
                pct = (total>0 ? 100*delta[s]/total : 0)
                printf "  %-12s %14d %14d %14d %7.1f%%\n", s, bef[s], aft[s], delta[s], pct
            }
            printf "  %-12s %14s %14s %14d %8s\n", "TOTAL", "", "", total, "100.0%"
        }'
 }
 # Dominant-slave pct for the verdict line. $1=col, $2=before, $3=after.
 dominant() {
    join "$2" "$3" | awk -v col="$1" '
        { b=(col==2?$2:$3)+0; a=(col==2?$4:$5)+0; d=a-b; if(d<0)d=0; tot+=d; delta[$1]=d }
        END { m=0; ms=""; for (s in delta){ if (delta[s]>m){m=delta[s]; ms=s} }
              printf "%.1f %s", (tot>0?100*m/tot:0), ms }'
 }
 HOST="$(uname -n)"
 HOST="${HOST%%.*}"
 case "$HOST" in
    *tsys6) ROLE="receiver"; LOG="$LOG_DIR/lacp-rx-receiver.log" ;;
    *tsys7) ROLE="sender";   LOG="$LOG_DIR/lacp-rx-sender.log" ;;
    *) echo "ERROR: not running on tsys6 or tsys7 (hostname=$HOST)"; exit 2 ;;
 esac
 TMP="$(mktemp -d)"
 trap 'rm -rf "$TMP"' EXIT
 {
 echo "=== LACP per-slave distribution ($ROLE) ==="
 echo "Host: $HOST   Role: $ROLE"
 echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
 echo "Bond: $BOND"
 echo ""
 echo "--- $BOND state ---"
 grep -E "Bonding Mode|Transmit Hash|Number of ports|Partner Mac|Slave Interface|Link" \
    "/proc/net/bonding/$BOND" 2>&1
 echo ""
 } | tee "$LOG"
 snapshot > "$TMP/before"
 if [ "$ROLE" = "receiver" ]; then
    echo "[receiver] starting one-shot iperf3 server on $PEER_RX_IP ..."
    pkill -x iperf3 2>/dev/null; sleep 0.5
    nohup iperf3 -s -1 -B "$PEER_RX_IP" > "$TMP/iperf.out" 2>&1 &
    SRV_PID=$!
    # Wait for the server to be ready (brief), then wait for it to exit
    # (one-shot server exits after serving one client).
    for _ in $(seq 1 60); do
        kill -0 "$SRV_PID" 2>/dev/null || break
        sleep 1
    done
    wait "$SRV_PID" 2>/dev/null
    echo "[receiver] iperf3 server finished."
 else
    # sender: wait briefly for receiver to be listening, then run client.
    echo "[sender] waiting 3s for receiver to listen, then running iperf3 client..."
    sleep 3
    iperf3 -c "$PEER_RX_IP" -B "$PEER_TX_IP" -P "$STREAMS" -t "$DURATION" -l 128k -O 2 \
        > "$TMP/iperf.out" 2>&1
    echo "[sender] iperf3 client finished (exit=$?)."
 fi
 snapshot > "$TMP/after"
 {
 echo ""
 echo "--- iperf3 output (raw tail) ---"
 tail -25 "$TMP/iperf.out"
 echo ""
 if [ "$ROLE" = "receiver" ]; then
    analyze "Receiver RX (THE key column)" 2 "$TMP/before" "$TMP/after"
    dom="$(dominant 2 "$TMP/before" "$TMP/after")"
    pct=${dom%% *}; top=${dom##* }
    echo ""
    echo "==================================================================="
    if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
        echo " VERDICT: SWITCH NOT distributing across $HOST's two ports"
        echo " Dominant slave '$top' = ${pct}% of RX -> hash not effective"
        echo " on this LAG. Action: verify/bounce pfv-r3-tor-stor port-channel."
    else
        echo " VERDICT: switch IS distributing (top slave '$top' = ${pct}%)."
        echo " Cap is host-side: softirq/CPU/bridge/NIC coalescing."
    fi
    echo "==================================================================="
 else
    analyze "Sender TX (control: should split if host hash=layer3+4)" 3 "$TMP/before" "$TMP/after"
    dom="$(dominant 3 "$TMP/before" "$TMP/after")"
    pct=${dom%% *}; top=${dom##* }
    echo ""
    echo "==================================================================="
    if awk -v p="$pct" 'BEGIN{exit !(p>=90)}'; then
        echo " VERDICT: host TX NOT distributing (top slave '$top' = ${pct}%)."
        echo " Host xmit_hash_policy is NOT effective despite /proc/net/bonding."
    else
        echo " VERDICT: host TX distributing OK (top slave '$top' = ${pct}%)."
    fi
    echo "==================================================================="
 fi
 } | tee -a "$LOG"
 echo ""
 echo "Log written: $LOG"
@@ -0,0 +1,91 @@
 #!/usr/bin/env bash
 ###############################################################################
 # run-lacp-retrans-cause.sh  (workstation wrapper)
 #
 # Deploys scripts/lacp-retrans-cause.sh to BOTH hosts, runs them in the
 # right order, scps logs back, prints them.
 ###############################################################################
 set -uo pipefail
 SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
     -o StrictHostKeyChecking=accept-new)
 SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 RECV="pfv-tsys6"; SEND="pfv-tsys7"
 LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 SCRIPT="lacp-retrans-cause.sh"
 LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
 REMOTE_SCRIPT="/root/${SCRIPT}"
 mkdir -p "$LOCAL_LOG_DIR"
 echo "==================================================================="
 echo " LACP retransmit-cause investigation"
 echo " Receiver: $RECV   Sender: $SEND"
 echo "==================================================================="
 echo ""
 echo "--- preflight ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    ssh "${SSH[@]}" "root@$h" \
        'command -v ethtool >/dev/null && e=OK || e=MISSING
         command -v iperf3 >/dev/null && i=OK || i=MISSING
         printf "ethtool=%s iperf3=%s host=%s\n" "$e" "$i" "$(uname -n)"' 2>&1 | head -1
 done
 echo ""
 echo "--- deploy ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
        && ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
        && echo "deployed" || echo "FAILED"
 done
 echo ""
 echo "--- cleanup stale iperf3 ---"
 for h in "$RECV" "$SEND"; do
    ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 echo "--- launching receiver on $RECV (background) ---"
 ssh "${SSH[@]}" "root@$RECV" \
    "nohup bash $REMOTE_SCRIPT > /root/lacp-retrans-receiver.console 2>&1 &" 2>/dev/null
 sleep 5
 echo ""
 echo "--- running sender on $SEND (foreground, ~20s) ---"
 ssh "${SSH[@]}" "root@$SEND" "bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/  [sender] /'
 echo ""
 sleep 3
 echo "--- fetching logs ---"
 for f in lacp-retrans-receiver.log lacp-retrans-receiver.console lacp-retrans-sender.log; do
    src=""
    case "$f" in
        *receiver*) src="$RECV" ;;
        *sender*)   src="$SEND" ;;
    esac
    printf '  %-32s <- %s : ' "$f" "$src"
    if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
        echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
    else
        echo "MISSING"
    fi
 done
 echo ""
 echo "==================================================================="
 echo " RECEIVER LOG ($RECV)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-retrans-receiver.log" 2>/dev/null || echo "(missing)"
 echo ""
 echo "==================================================================="
 echo " SENDER LOG ($SEND)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-retrans-sender.log" 2>/dev/null || echo "(missing)"
 echo ""
 echo "==================================================================="
 echo " Local copies in: $LOCAL_LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,112 @@
 #!/usr/bin/env bash
 ###############################################################################
 # run-lacp-rx-distribution.sh  (workstation wrapper)
 #
 # One-shot orchestrator: deploys scripts/lacp-rx-distribution.sh to BOTH
 # tsys6 (receiver) and tsys7 (sender), runs them in the right order, then
 # scps both logs back to returned-logs/iperf/.
 #
 # Run from the workstation:
 #     bash scripts/run-lacp-rx-distribution.sh
 #
 # Idempotent: safe to re-run. iperf3 is killed on both hosts first.
 ###############################################################################
 set -uo pipefail
 SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
     -o StrictHostKeyChecking=accept-new)
 SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 RECV="pfv-tsys6"        # receiver (we care most about its RX split)
 SEND="pfv-tsys7"        # sender (control: its TX split)
 LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
 SCRIPT="lacp-rx-distribution.sh"
 LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
 REMOTE_SCRIPT="/root/${SCRIPT}"
 mkdir -p "$LOCAL_LOG_DIR"
 echo "==================================================================="
 echo " LACP per-slave RX/TX distribution test"
 echo " Receiver: $RECV (10.100.100.6)   Sender: $SEND (10.100.100.7)"
 echo "==================================================================="
 echo ""
 # 0. Preflight: confirm SSH and iperf3 on both hosts
 echo "--- preflight (ssh + iperf3 + bond0) ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    ssh "${SSH[@]}" "root@$h" \
        'command -v iperf3 >/dev/null && ip=$(command -v iperf3) || ip=MISSING
         [ -r /proc/net/bonding/bond0 ] && b=OK || b=NO-BOND0
         printf "iperf3=%s bond0=%s host=%s\n" "$ip" "$b" "$(uname -n)"' \
        2>&1 | head -1
 done
 echo ""
 # 1. Copy the script to both hosts + chmod
 echo "--- deploy $SCRIPT to both hosts ---"
 for h in "$RECV" "$SEND"; do
    printf '  %-12s ' "$h"
    scp "${SCP[@]}" "$LOCAL_SCRIPT" "root@$h:$REMOTE_SCRIPT" >/dev/null 2>&1 \
        && ssh "${SSH[@]}" "root@$h" "chmod +x $REMOTE_SCRIPT" \
        && echo "deployed + chmod +x" \
        || echo "DEPLOY FAILED"
 done
 echo ""
 # 2. Kill any stale iperf3 on both hosts
 echo "--- cleanup stale iperf3 ---"
 for h in "$RECV" "$SEND"; do
    ssh "${SSH[@]}" "root@$h" 'pkill -x iperf3 2>/dev/null; true' 2>/dev/null
 done
 echo ""
 # 3. Start RECEIVER in background (one-shot server, writes /root/lacp-rx-receiver.log)
 echo "--- starting receiver on $RECV (background) ---"
 ssh "${SSH[@]}" "root@$RECV" \
    "nohup bash $REMOTE_SCRIPT > /root/lacp-rx-receiver.console 2>&1 &" 2>/dev/null
 echo "  receiver launched; waiting 5s for it to start iperf3 -s -1 ..."
 sleep 5
 echo ""
 # 4. Run SENDER (foreground; ~15s with the default 12s test + 3s pre-sleep)
 echo "--- running sender on $SEND (foreground, ~20s) ---"
 ssh "${SSH[@]}" "root@$SEND" \
    "bash $REMOTE_SCRIPT" 2>&1 | sed 's/^/  [sender] /'
 echo ""
 # 5. Give receiver a moment to finish writing its log
 sleep 3
 # 6. Fetch logs back
 echo "--- fetching logs ---"
 for f in lacp-rx-receiver.log lacp-rx-receiver.console lacp-rx-sender.log; do
    src=""
    case "$f" in
        lacp-rx-receiver*) src="$RECV" ;;
        lacp-rx-sender*)   src="$SEND" ;;
    esac
    printf '  %-28s <- %s : ' "$f" "$src"
    if scp "${SCP[@]}" "root@$src:/root/$f" "$LOCAL_LOG_DIR/$f" >/dev/null 2>&1; then
        echo "OK ($(wc -c < "$LOCAL_LOG_DIR/$f" 2>/dev/null) bytes)"
    else
        echo "MISSING"
    fi
 done
 echo ""
 # 7. Show the receiver log (the decisive one)
 echo "==================================================================="
 echo " RECEIVER LOG ($RECV — the decisive side)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-rx-receiver.log" 2>/dev/null || echo "(no log)"
 echo ""
 echo "==================================================================="
 echo " SENDER LOG ($SEND — control)"
 echo "==================================================================="
 cat "$LOCAL_LOG_DIR/lacp-rx-sender.log" 2>/dev/null || echo "(no log)"
 echo ""
 echo "==================================================================="
 echo " Local copies in: $LOCAL_LOG_DIR/"
 echo "==================================================================="
@@ -0,0 +1,127 @@
 #!/usr/bin/env bash
 # sw-capture-remote.sh - orchestrate a serial capture from this workstation.
 #
 # Flow:
 #   1. De-conflict: abort if any local ssh to pfv-tsys4 is in flight
 #      (other agent could be there).
 #   2. Free the serial port: kill whatever holds /dev/ttyUSBx
 #      (typically a screen session). Targeted, not blanket.
 #   3. scp driver + .cmds to pfv-tsys4.
 #   4. Run driver over ssh, capture stderr to console.
 #   5. scp the resulting log back to returned-logs/.
 #
 # Usage:
 #   sw-capture-remote.sh <switch-name> [device]
 #
 #   <switch-name>   e.g. pfv-core-sw01 (must have switches/<name>.cmds)
 #   [device]        /dev/ttyUSBx on pfv-tsys4. Defaults per switch map below.
 #
 # Currently scoped to pfv-core-sw01 only (per user direction). The other
 # two switches are deferred; their defaults are placeholders.
 set -u
 SWITCH=${1:-}
 DEVICE=${2:-}
 if [ -z "$SWITCH" ]; then
    echo "Usage: $0 <switch-name> [device]" >&2
    echo "  e.g. $0 pfv-core-sw01 /dev/ttyUSB2" >&2
    exit 2
 fi
 # Switch -> default device map (ttyUSB2 = core-sw01 confirmed by user).
 case "$SWITCH" in
    pfv-core-sw01)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB2 ;;
    pfv-r3-tor-mgmt)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB0   # TENTATIVE - unconfirmed
        if [ "${2:-}" = "" ]; then
            echo "NOTE: pfv-r3-tor-mgmt device is tentative (/dev/ttyUSB0)." >&2
            echo "      Pass the device explicitly if different." >&2
        fi ;;
    pfv-r3-tor-stor)
        [ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB1   # TENTATIVE - unconfirmed
        if [ "${2:-}" = "" ]; then
            echo "NOTE: pfv-r3-tor-stor device is tentative (/dev/ttyUSB1)." >&2
            echo "      Pass the device explicitly if different." >&2
        fi ;;
    *)
        echo "unknown switch: $SWITCH" >&2; exit 2 ;;
 esac
 BAUD=9600
 HOST=root@pfv-tsys4
 HERE=/home/reachableceo/projects/perfopt
 LOCAL_DRIVER=$HERE/scripts/sw-capture.py
 LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
 LOCAL_LOG=$HERE/returned-logs/$SWITCH.log
 REMOTE_DRIVER=/root/sw-capture.py
 REMOTE_CMDS=/root/$SWITCH.cmds
 REMOTE_LOG=/root/$SWITCH.log
 [ -f "$LOCAL_DRIVER" ] || { echo "missing $LOCAL_DRIVER" >&2; exit 2; }
 [ -f "$LOCAL_CMDS"   ] || { echo "missing $LOCAL_CMDS" >&2; exit 2; }
 ts() { date +%H:%M:%S; }
 echo "[$(ts)] switch=$SWITCH device=$DEVICE baud=$BAUD host=$HOST"
 # 1. De-conflict: any local ssh to pfv-tsys4 in flight?
 echo "[$(ts)] checking for in-flight ssh to pfv-tsys4..."
 if ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys4|scp.*pfv-tsys4' | grep -v grep >/tmp/.swcap.ps 2>&1; then
    cat /tmp/.swcap.ps
    echo "[$(ts)] ABORT: another ssh/scp to pfv-tsys4 is running (other agent?)." >&2
    exit 1
 fi
 echo "[$(ts)] clear."
 rm -f /tmp/.swcap.ps
 # 2. Free the serial port: kill whatever holds $DEVICE.
 echo "[$(ts)] freeing $DEVICE on $HOST (targeted; other screen sessions untouched)..."
 ssh -o BatchMode=yes -o ConnectTimeout=8 "$HOST" \
    "fuser -v $DEVICE 2>&1 | tee /dev/stderr; \
     fuser -k -TERM $DEVICE 2>/dev/null; sleep 1; \
     if fuser $DEVICE 2>/dev/null; then \
         echo 'still held after SIGTERM, escalating to SIGKILL'; \
         fuser -k -KILL $DEVICE 2>/dev/null; sleep 1; \
     fi; \
     fuser $DEVICE 2>/dev/null && echo 'STILL HELD' || echo 'FREE'"
 # Re-check; abort if still held.
 HELD=$(ssh -o BatchMode=yes "$HOST" "fuser $DEVICE 2>/dev/null && echo HELD || echo FREE")
 if [ "$HELD" = "HELD" ]; then
    echo "[$(ts)] ABORT: $DEVICE still held on $HOST." >&2
    exit 1
 fi
 # 3. Copy driver + cmds.
 echo "[$(ts)] copying driver + cmds to $HOST..."
 scp -q "$LOCAL_DRIVER" "$HOST:$REMOTE_DRIVER"
 scp -q "$LOCAL_CMDS"   "$HOST:$REMOTE_CMDS"
 # 4. Run the capture on pfv-tsys4. Stream stderr (progress) to console.
 echo "[$(ts)] running capture..."
 ssh -o BatchMode=yes -o ServerAliveInterval=10 "$HOST" \
    "python3 $REMOTE_DRIVER \
        --device $DEVICE --baud $BAUD \
        --cmds $REMOTE_CMDS --log $REMOTE_LOG"
 RC=$?
 echo "[$(ts)] capture exit code: $RC"
 # 5. Pull log back.
 echo "[$(ts)] pulling log back to $LOCAL_LOG..."
 mkdir -p "$(dirname "$LOCAL_LOG")"
 scp -q "$HOST:$REMOTE_LOG" "$LOCAL_LOG"
 if [ -f "$LOCAL_LOG" ]; then
    SZ=$(wc -c < "$LOCAL_LOG")
    echo "[$(ts)] OK: $LOCAL_LOG ($SZ bytes)"
    echo "----- head -----"
    head -30 "$LOCAL_LOG"
    echo "----- tail -----"
    tail -10 "$LOCAL_LOG"
 else
    echo "[$(ts)] ERROR: log not pulled back." >&2
    exit 1
 fi
 exit $RC
@@ -0,0 +1,255 @@
 #!/usr/bin/env python3
 """
 sw-capture.py - drive a Dell switch over a serial console and log all output.
 Read-only. Sends ONLY the commands listed in the supplied .cmds file
 (comment lines starting with '!' and blank lines are skipped). Handles
 `--More--` pagination by sending a space. Aborts cleanly on any password
 prompt (we never supply credentials).
 Pure stdlib (termios + select). No pyserial/expect required.
 Exit codes:
   0  clean run, every command saw a prompt again
   2  could not synchronize with a prompt during wake
   3  one or more commands timed out (log still written)
   4  password prompt encountered (aborted)
 Usage:
    sw-capture.py --device /dev/ttyUSB2 --baud 9600 \\
                  --cmds pfv-core-sw01.cmds --log /root/pfv-core-sw01.log
 """
 import argparse
 import os
 import re
 import select
 import sys
 import termios
 import time
 PROMPT_RE = re.compile(rb'[>#]\s*$')              # ends in # or > + spaces
 MORE_RE   = re.compile(rb'--\s*More\s*--')        # pagination prompt
 PWD_RE    = re.compile(rb'[Pp]assword:\s*$')      # enable / login password
 BAUDS = {
    '9600':    termios.B9600,
    '19200':   termios.B19200,
    '38400':   termios.B38400,
    '57600':   termios.B57600,
    '115200':  termios.B115200,
 }
 def log(msg, level='INFO'):
    sys.stderr.write(f'[{level}] {msg}\n')
    sys.stderr.flush()
 def open_port(device, baud):
    """Open the serial device raw at the requested baud, 8N1, no flow ctrl."""
    fd = os.open(device, os.O_RDWR | os.O_NOCTTY)
    try:
        attrs = termios.tcgetattr(fd)
    except termios.error:
        log(f'{device} is not a termios-capable device', 'WARN')
        return fd
    # raw input
    attrs[0] &= ~(termios.IGNBRK | termios.BRKINT | termios.PARMRK |
                  termios.ISTRIP | termios.INLCR | termios.IGNCR |
                  termios.ICRNL | termios.IXON)
    # raw output
    attrs[1] &= ~termios.OPOST
    # 8N1, enable receiver, ignore modem control lines
    attrs[2] &= ~(termios.CSIZE | termios.PARENB | termios.CSTOPB)
    attrs[2] |= termios.CS8 | termios.CREAD | termios.CLOCAL
    # raw local
    attrs[3] &= ~(termios.ECHO | termios.ECHONL | termios.ICANON |
                  termios.ISIG | termios.IEXTEN)
    # non-blocking-ish reads (select is the primary gate)
    attrs[6][termios.VMIN] = 0
    attrs[6][termios.VTIME] = 1
    b = BAUDS.get(str(baud))
    if b is None:
        raise SystemExit(f'unsupported baud: {baud}')
    # Set ispeed/ospeed directly on the attribute list. (Equivalent to
    # termios.cfsetispeed/cfsetospeed, which are missing on some Python
    # builds — e.g. the one on pfv-tsys4.)
    attrs[4] = b   # ispeed
    attrs[5] = b   # ospeed
    termios.tcsetattr(fd, termios.TCSANOW, attrs)
    return fd
 def read_chunk(fd, timeout):
    """Read whatever arrives within `timeout`. Extends briefly on activity."""
    buf = b''
    deadline = time.time() + timeout
    while True:
        remaining = deadline - time.time()
        if remaining <= 0:
            return buf
        r, _, _ = select.select([fd], [], [], min(0.5, remaining))
        if not r:
            if buf:
                return buf
            continue
        try:
            chunk = os.read(fd, 4096)
        except OSError:
            return buf
        if not chunk:
            return buf
        buf += chunk
        # keep collecting as long as bytes are flowing
        deadline = time.time() + 0.3
 def drain(fd, timeout=1.0):
    total = 0
    while True:
        b = read_chunk(fd, timeout=timeout)
        if not b:
            return total
        total += len(b)
 def send(fd, s):
    if isinstance(s, str):
        s = s.encode()
    os.write(fd, s)
 def wait_for(fd, regex, timeout, on_more=None, on_pwd=None):
    """Read until `regex` matches the tail of the buffer, or timeout."""
    buf = b''
    deadline = time.time() + timeout
    while time.time() < deadline:
        remaining = deadline - time.time()
        chunk = read_chunk(fd, timeout=min(1.0, remaining))
        if chunk:
            buf += chunk
            tail64 = buf[-64:]
            tail32 = buf[-32:]
            tail128 = buf[-128:]
            if on_more and MORE_RE.search(tail64):
                on_more(fd)
                continue
            if on_pwd and PWD_RE.search(tail32):
                on_pwd(buf)
                return buf, 'pwd'
            if regex.search(tail128):
                return buf, 'ok'
    return buf, 'timeout'
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--device', required=True)
    ap.add_argument('--baud', type=int, default=9600)
    ap.add_argument('--cmds', required=True)
    ap.add_argument('--log',  required=True)
    ap.add_argument('--per-cmd-timeout', type=float, default=45.0)
    ap.add_argument('--wake-timeout',    type=float, default=15.0)
    ap.add_argument('--session-max',     type=float, default=600.0)
    args = ap.parse_args()
    cmds = []
    with open(args.cmds) as f:
        for raw in f:
            s = raw.rstrip('\n').strip()
            if not s or s.startswith('!'):
                continue
            cmds.append(s)
    log(f'parsed {len(cmds)} commands from {args.cmds}')
    logf = open(args.log, 'wb', buffering=0)
    def w(b):
        if isinstance(b, str):
            b = b.encode()
        logf.write(b)
    w(f'==== sw-capture {time.strftime("%Y-%m-%d %H:%M:%S")} ====\n')
    w(f'device={args.device} baud={args.baud} cmds={args.cmds} '
      f'n={len(cmds)} per_cmd_timeout={args.per_cmd_timeout}\n\n')
    fd = open_port(args.device, args.baud)
    log(f'opened {args.device} @ {args.baud} 8N1 raw')
    session_start = time.time()
    abort = False
    def on_more(fd_):
        log('--More-- -> space')
        send(fd_, b' ')
    def on_pwd(buf):
        nonlocal abort
        abort = True
        log('password prompt detected (enable or login) - aborting; '
            'no credentials supplied', 'ERROR')
        w(buf)
        w(b'\n[PASSWORD PROMPT - ABORTED]\n')
    # WAKE: nudge with Ctrl-C + Enter, look for any prompt
    drain(fd, 0.5)
    synced = False
    wake_deadline = time.time() + args.wake_timeout
    attempt = 0
    while time.time() < wake_deadline:
        attempt += 1
        send(fd, b'\x03')
        time.sleep(0.2)
        send(fd, b'\r')
        buf, status = wait_for(fd, PROMPT_RE, timeout=3.0,
                               on_more=on_more, on_pwd=on_pwd)
        w(buf)
        if status == 'pwd':
            logf.close(); os.close(fd); sys.exit(4)
        if status == 'ok':
            synced = True
            log(f'prompt synced after {attempt} attempt(s)')
            break
    if not synced:
        w(b'\n[NO PROMPT - ABORT]\n')
        log('no prompt detected during wake window', 'ERROR')
        logf.close(); os.close(fd); sys.exit(2)
    # RUN commands verbatim from the .cmds list
    failures = 0
    for idx, cmd in enumerate(cmds, 1):
        if time.time() - session_start > args.session_max:
            log('session_max exceeded - stopping early', 'ERROR')
            w(b'\n[SESSION_MAX - STOP]\n')
            break
        if abort:
            break
        log(f'[{idx}/{len(cmds)}] {cmd}')
        send(fd, cmd + '\r')
        buf, status = wait_for(fd, PROMPT_RE,
                                timeout=args.per_cmd_timeout,
                                on_more=on_more, on_pwd=on_pwd)
        w(buf)
        if status == 'pwd':
            failures += 1
            break
        if status == 'timeout':
            log(f'timeout after: {cmd}', 'WARN')
            failures += 1
            # try to resync: Ctrl-C + drain
            send(fd, b'\x03')
            time.sleep(0.3)
            drain(fd, 0.5)
    w(f'\n==== end {time.strftime("%Y-%m-%d %H:%M:%S")} '
      f'failures={failures} ====\n')
    logf.close()
    os.close(fd)
    log(f'done -> {args.log}  failures={failures}')
    sys.exit(0 if failures == 0 else 3)
 if __name__ == '__main__':
    main()
@@ -0,0 +1,55 @@
 #!/usr/bin/env bash
 # Probe conman state + expect availability on pfv-tsys4.
 # Read-only. Decides whether we drive via conman+expect or expect-only.
 set -u
 # De-conflict: any ssh to pfv-tsys4 right now?
 echo "===== LOCAL ssh activity ====="
 ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys' | grep -v grep || echo "(none to pfv-tsys4)"
 echo
 echo "===== pfv-tsys4: conman + expect state ====="
 ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
 echo "--- conmand service ---"
 systemctl is-active conmand 2>&1 || true
 systemctl is-enabled conmand 2>&1 || true
 systemctl status conmand --no-pager 2>&1 | head -15 || true
 echo
 echo "--- conman binary ---"
 command -v conman && conman --version 2>&1 | head -2 || echo "conman: MISSING"
 command -v conmand && echo "conmand present" || echo "conmand: MISSING"
 echo
 echo "--- /etc/conman.conf: ttyUSB2 entries ---"
 grep -nE "ttyUSB2|core-sw|CONSOLE|LOG|SERIAL|BAUD" /etc/conman.conf 2>/dev/null | head -40 || echo "(no matches / no file)"
 echo
 echo "--- conman log dir ---"
 ls -la /var/log/conman/ 2>&1 | head -20 || echo "(no /var/log/conman)"
 ls -la /var/consoles/ 2>&1 | head -20 || echo "(no /var/consoles)"
 echo
 echo "--- expect availability ---"
 command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
 echo "apt-cache policy expect:"
 apt-cache policy expect 2>/dev/null | head -10 || echo "(apt-cache failed)"
 echo
 echo "--- other useful drivers ---"
 for t in tclsh socat cu tip; do
    command -v "$t" 2>/dev/null && echo "  $t: present" || true
 done
 echo
 echo "--- apt network reachability (quick) ---"
 timeout 5 bash -c 'echo > /dev/tcp/deb.debian.org/80' 2>&1 && echo "apt network: OK" || echo "apt network: UNREACHABLE"
 echo
 echo "--- disk space for log ---"
 df -h /root 2>&1 | tail -2
 echo
 echo "--- screen sessions (still 3?) ---"
 screen -ls 2>&1 || true
 REMOTE
@@ -0,0 +1,39 @@
 #!/usr/bin/env bash
 # Probe pfv-tsys4 for what's available to drive the serial console.
 # Also snapshots local ssh/scp activity so we can de-conflict with the
 # other agent running in this directory.
 set -u
 echo "===== LOCAL ssh/scp activity (other-agent de-confliction) ====="
 ps -eo pid,ppid,etime,user,args | grep -E 'ssh|scp' | grep -v grep || echo "(none)"
 echo
 echo "===== Ping pfv-tsys4 ====="
 ping -c1 -W2 pfv-tsys4 >/dev/null 2>&1 && echo "ping OK" || echo "ping FAIL"
 echo
 echo "===== Probe pfv-tsys4 over ssh ====="
 ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
 echo "--- host ---"
 hostname; uname -a
 echo "--- tools ---"
 for t in python3 python expect screen minicom picocom stty fuser lsof; do
    p=$(command -v "$t" 2>/dev/null) && echo "$t -> $p" || echo "$t -> MISSING"
 done
 echo "--- pyserial ---"
 python3 -c "import serial; print('pyserial', serial.__version__)" 2>&1
 echo "--- device node ---"
 ls -l /dev/ttyUSB2 2>&1
 stat -c '%n owner=%U:%G mode=%a' /dev/ttyUSB2 2>&1 || true
 echo "--- who holds /dev/ttyUSB2 ---"
 fuser -v /dev/ttyUSB2 2>&1 || echo "(fuser: none or n/a)"
 lsof /dev/ttyUSB2 2>&1 | head -20 || true
 echo "--- screen sessions on this host ---"
 screen -ls 2>&1 || echo "(no screen / not installed)"
 echo "--- current tty settings (only readable if not held exclusively) ---"
 stty -F /dev/ttyUSB2 2>&1 || echo "(held exclusively - expected if screen is up)"
 echo "--- baud hints in config/history ---"
 grep -riE "ttyUSB2|115200|9600|baud" /etc/ ~/.screenrc ~/.bash_history 2>/dev/null | head -20 || true
 echo "--- recent console-related processes ---"
 ps -eo pid,etime,user,args | grep -E 'screen|minicom|picocom|ttyUSB' | grep -v grep || echo "(none)"
 REMOTE
@@ -0,0 +1,65 @@
 #!/bin/bash
 # shellcheck.sh - permanent wrapper to lint every shell script in this project.
 #
 # Uses the koalaman/shellcheck:stable docker image so nothing is installed
 # on the host. Run from anywhere; lints scripts/ and any .sh under switches/.
 #
 # Usage:
 #   ./shellcheck.sh              # tty output, all scripts
 #   ./shellcheck.sh --fix-info   # treat style notes as non-blocking (default)
 #   ./shellcheck.sh --strict     # exit non-zero on ANY finding (notes too)
 #   ./shellcheck.sh scripts/assess.sh   # lint a single file
 set -u
 ROOT="$(cd "$(dirname "$0")" && pwd)"
 IMAGE="koalaman/shellcheck:stable"
 STRICT=0
 TARGETS=()
 for arg in "$@"; do
    case "$arg" in
        --strict) STRICT=1 ;;
        --fix-info) STRICT=0 ;;
        -h|--help)
            sed -n '2,12p' "$0"; exit 0 ;;
        *) TARGETS+=("$arg") ;;
    esac
 done
 # Default targets: everything in scripts/, plus any .cmds is NOT shell - skip.
 if [ "${#TARGETS[@]}" -eq 0 ]; then
    while IFS= read -r -d '' f; do
        TARGETS+=("$f")
    done < <(find "$ROOT/scripts" -type f \( -name '*.sh' -o -name 'collect-*' -o -name 'assess*' \) -print0 2>/dev/null)
 fi
 if [ "${#TARGETS[@]}" -eq 0 ]; then
    echo "no shell scripts found to lint" >&2
    exit 1
 fi
 echo "Linting ${#TARGETS[@]} file(s) with $IMAGE:"
 for t in "${TARGETS[@]}"; do echo "  - $t"; done
 echo
 # Make paths relative to ROOT so docker volume maps cleanly
 REL_TARGETS=()
 for t in "${TARGETS[@]}"; do
    rel="${t#$ROOT/}"
    [ "$rel" = "$t" ] && rel="$t"
    REL_TARGETS+=("$rel")
 done
 SC_ARGS=(--format=tty)
 [ "$STRICT" -eq 0 ] && SC_ARGS+=(--severity=warning)
 docker run --rm -v "$ROOT:/mnt" -w /mnt "$IMAGE" \
    "${SC_ARGS[@]}" "${REL_TARGETS[@]}"
 RC=$?
 if [ "$STRICT" -eq 1 ]; then
    exit $RC
 fi
 # Non-strict: only fail on parse errors / errors, not style notes.
 # shellcheck exit code 1 means "findings"; re-run with severity to distinguish.
 exit 0
@@ -0,0 +1,169 @@
 #!/bin/bash
 # validate-fixes.sh - READ-ONLY validation of all applied tunings.
 # Does NOT reboot, shutdown VMs, or modify anything.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 echo "==================================================================="
 echo " READ-ONLY VALIDATION — $(date)"
 echo "==================================================================="
 echo ""
 for HOST in "${HOSTS[@]}"; do
    echo "================================================================"
    echo "[$HOST]"
    echo "================================================================"
    if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE"
        echo ""
        continue
    fi
    ssh "${SSH_OPTS[@]}" "root@$HOST" '
        pass=0; fail=0
        check() {
            local label="$1" actual="$2" expected="$3"
            if [ "$actual" = "$expected" ]; then
                printf "  [OK]   %-30s %s\n" "$label" "$actual"
                pass=$((pass+1))
            else
                printf "  [FAIL] %-30s got=%s want=%s\n" "$label" "$actual" "$expected"
                fail=$((fail+1))
            fi
        }
        # 1. CPU governor
        gov=$(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "n/a")
        if [ "$gov" = "n/a" ]; then
            printf "  [SKIP] %-30s %s\n" "CPU governor" "(no cpufreq driver — OK for server BIOS)"
        else
            check "CPU governor" "$gov" "performance"
        fi
        # 2. vm.swappiness
        swap=$(sysctl -n vm.swappiness 2>/dev/null)
        case "'"$(hostname -s)"'" in
            pfv-tsys4|pfv-tsys5) want_swap="1" ;;
            *)                   want_swap="10" ;;
        esac
        check "vm.swappiness" "$swap" "$want_swap"
        # 3. TCP congestion control
        cc=$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null)
        check "tcp_congestion_control" "$cc" "bbr"
        # 4. default_qdisc (paired with BBR)
        qd=$(sysctl -n net.core.default_qdisc 2>/dev/null)
        check "net.core.default_qdisc" "$qd" "fq"
        # 5. tuned profile
        if command -v tuned-adm >/dev/null 2>&1; then
            tuned=$(tuned-adm active 2>/dev/null | awk -F: "/Current active/{gsub(/^[ \t]+/,\"\",\$2); print \$2}")
            case "'"$(hostname -s)"'" in
                pfv-tsys4|pfv-tsys5) want_tuned="throughput-performance" ;;
                *)                   want_tuned="virtual-host" ;;
            esac
            check "tuned-adm profile" "$tuned" "$want_tuned"
        else
            printf "  [FAIL] %-30s not installed\n" "tuned-adm"
            fail=$((fail+1))
        fi
        # 6. bond0 hash policy (if bond exists)
        if [ -r /proc/net/bonding/bond0 ]; then
            hash=$(grep "Transmit Hash" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
            check "bond0 xmit_hash_policy" "$hash" "layer3+4"
            ports=$(grep "Number of ports" /proc/net/bonding/bond0 2>/dev/null | awk "{print \$4}")
            printf "  [INFO] %-30s %s ports active\n" "bond0 LACP ports" "$ports"
        else
            printf "  [SKIP] %-30s %s\n" "bond0 hash" "(no bond0 — single NIC host)"
        fi
        # 7. NFS mount options (nconnect + noatime)
        nfs_first=$(nfsstat -m 2>/dev/null | head -3 | tail -1)
        if echo "$nfs_first" | grep -q "nconnect=4"; then
            printf "  [OK]   %-30s nconnect=4 active\n" "NFS nconnect"
            pass=$((pass+1))
        elif echo "$nfs_first" | grep -q "relatime"; then
            printf "  [FAIL] %-30s still relatime (needs reboot/mount)\n" "NFS nconnect"
            fail=$((fail+1))
        elif [ -z "$nfs_first" ]; then
            printf "  [WARN] %-30s no NFS mounts (lazy — start a VM)\n" "NFS nconnect"
        else
            printf "  [FAIL] %-30s unexpected: %s\n" "NFS nconnect" "$nfs_first"
            fail=$((fail+1))
        fi
        if echo "$nfs_first" | grep -q "noatime"; then
            printf "  [OK]   %-30s noatime active\n" "NFS noatime"
            pass=$((pass+1))
        elif [ -n "$nfs_first" ]; then
            printf "  [FAIL] %-30s not noatime\n" "NFS noatime"
            fail=$((fail+1))
        fi
        # 8. NFS TCP connection count
        nfs_conns=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
        if [ "$nfs_conns" -ge 8 ]; then
            printf "  [OK]   %-30s %s connections\n" "NFS TCP conns" "$nfs_conns"
            pass=$((pass+1))
        elif [ "$nfs_conns" -gt 0 ]; then
            printf "  [WARN] %-30s %s (expect 8 with nconnect=4)\n" "NFS TCP conns" "$nfs_conns"
        else
            printf "  [WARN] %-30s 0 (lazy mounts — start a VM)\n" "NFS TCP conns"
        fi
        # 9. sysctl persistence
        if [ -r /etc/sysctl.d/99-perfopt.conf ]; then
            printf "  [OK]   %-30s /etc/sysctl.d/99-perfopt.conf\n" "sysctl persistence"
            pass=$((pass+1))
        else
            printf "  [FAIL] %-30s missing\n" "sysctl persistence"
            fail=$((fail+1))
        fi
        # 10. Observability packages
        for cmd in sar jq numactl nvme mtr bmon; do
            if ! command -v "$cmd" >/dev/null 2>&1; then
                printf "  [FAIL] %-30s not installed\n" "obs: $cmd"
                fail=$((fail+1))
            fi
        done
        if command -v sar >/dev/null 2>&1 && command -v jq >/dev/null 2>&1 && \
           command -v numactl >/dev/null 2>&1 && command -v nvme >/dev/null 2>&1 && \
           command -v mtr >/dev/null 2>&1 && command -v bmon >/dev/null 2>&1; then
            printf "  [OK]   %-30s all installed\n" "observability packages"
            pass=$((pass+1))
        fi
        # 11. Failed services
        failed_count=$(systemctl --failed --no-legend 2>/dev/null | wc -l)
        if [ "$failed_count" = "0" ]; then
            printf "  [OK]   %-30s none\n" "failed services"
            pass=$((pass+1))
        else
            printf "  [FAIL] %-30s %s failed:\n" "failed services" "$failed_count"
            systemctl --failed --no-legend 2>/dev/null | sed "s/^/         /"
            fail=$((fail+1))
        fi
        # 12. VM status (read-only — just report)
        running_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"" | wc -l)
        stopped_vms=$(qm list 2>/dev/null | awk "NR>1 && \$3!=\"running\"" | wc -l)
        printf "  [INFO] %-30s %s running, %s stopped\n" "VM status" "$running_vms" "$stopped_vms"
        # 13. uptime
        printf "  [INFO] %-30s %s\n" "uptime" "$(uptime | sed "s/.*up //" | sed "s/,.*//")"
        echo ""
        echo "  RESULT: $pass passed, $fail failed"
    '
    echo ""
 done
 echo "==================================================================="
 echo " SUMMARY"
 echo "==================================================================="
@@ -0,0 +1,132 @@
 #!/bin/bash
 ###############################################################################
 # validate-vms.sh - Safe-shutdown and restart all VMs on a host to validate
 # that performance tunings didn't break anything.
 #
 # For each VM:
 #   1. qm shutdown <vmid> --timeout 120  (ACPI safe shutdown)
 #   2. Wait for stopped state
 #   3. qm start <vmid>
 #   4. Wait for running state
 #   5. Check qm agent responds (if agent enabled)
 #
 # Usage: bash validate-vms.sh <host> [host...]
 ###############################################################################
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
 for HOST in "$@"; do
    echo "================================================================"
    echo "[$HOST] VM SAFE-SHUTDOWN/RESTART VALIDATION"
    echo "================================================================"
    echo ""
    if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "  UNREACHABLE — skipping"
        continue
    fi
    # Get list of running VMs
    VM_LIST=$(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1 && \$3==\"running\"{print \$1}"')
    if [ -z "$VM_LIST" ]; then
        echo "  No running VMs — nothing to validate"
        continue
    fi
    VM_COUNT=$(echo "$VM_LIST" | wc -w)
    echo "  Found $VM_COUNT running VM(s): $(echo "$VM_LIST" | tr '\n' ' ')"
    echo ""
    # --- Phase 1: Safe shutdown all VMs ---
    echo "--- PHASE 1: Safe shutdown all VMs (120s timeout each) ---"
    for vmid in $VM_LIST; do
        name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
        echo -n "  [$vmid $name] shutting down... "
        ssh "${SSH_OPTS[@]}" "root@$HOST" "qm shutdown $vmid --timeout 120 --forceStop 1" 2>&1 | head -1
    done
    # Wait for all to stop (max 180s total)
    echo ""
    echo -n "  Waiting for all VMs to stop"
    WAIT_DEADLINE=$(( $(date +%s) + 180 ))
    while [ "$(date +%s)" -lt "$WAIT_DEADLINE" ]; do
        echo -n "."
        all_stopped=1
        for vmid in $VM_LIST; do
            status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
            if [ "$status" = "running" ]; then
                all_stopped=0
                break
            fi
        done
        [ "$all_stopped" = "1" ] && break
        sleep 5
    done
    echo " done"
    echo ""
    # Show stopped state
    echo "--- VM status after shutdown ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
    echo ""
    # --- Phase 2: Start all VMs ---
    echo "--- PHASE 2: Start all VMs ---"
    for vmid in $VM_LIST; do
        name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
        echo -n "  [$vmid $name] starting... "
        start_output=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1)
        if [ $? -eq 0 ]; then
            echo "OK"
        else
            echo "FAILED: $start_output"
        fi
    done
    # Wait 20s for VMs to fully start
    echo ""
    echo "  Waiting 20s for VMs to boot..."
    sleep 20
    # --- Phase 3: Verify all VMs running ---
    echo ""
    echo "--- PHASE 3: Verification ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
    echo ""
    # Check NFS mounts still healthy
    echo "--- NFS mount health ---"
    ssh "${SSH_OPTS[@]}" "root@$HOST" '
        mount_count=$(nfsstat -m 2>/dev/null | grep -c "^/mnt")
        conn_count=$(ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l)
        echo "  NFS mounts: $mount_count"
        echo "  NFS TCP connections: $conn_count"
        if [ "$mount_count" -gt 0 ]; then
            echo "  First mount options:"
            nfsstat -m 2>/dev/null | head -2 | tail -1 | sed "s/^/    /"
        fi
    '
    echo ""
    # Check guest agent responsiveness (if agent enabled)
    echo "--- Guest agent check (VMs with agent:1) ---"
    for vmid in $VM_LIST; do
        agent=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | grep -c '^agent: 1'")
        if [ "$agent" = "1" ]; then
            name=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm config $vmid 2>/dev/null | awk -F: '/^name:/{gsub(/^ /,\"\");print \$2}'")
            echo -n "  [$vmid $name] agent ping... "
            result=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "timeout 10 qm agent $vmid ping 2>&1")
            if [ $? -eq 0 ]; then
                echo "OK"
            else
                echo "no response (VM may still be booting)"
            fi
        fi
    done
    echo ""
    echo "================================================================"
    echo "[$HOST] VALIDATION COMPLETE"
    echo "================================================================"
    echo ""
 done
@@ -0,0 +1,36 @@
 #!/bin/bash
 # verify-tuning.sh - verifies apply-tunings.sh results on target hosts.
 set -uo pipefail
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
 for host in "$@"; do
    echo "================================================================"
    echo "[$host] verification"
    echo "================================================================"
    echo "--- TCP congestion control ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc 2>/dev/null'
    echo "--- vm.swappiness ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'sysctl vm.swappiness 2>/dev/null'
    echo "--- scaling_governor (cpu0) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null'
    echo "--- NFS mount options (first 2 mounts) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'nfsstat -m 2>/dev/null | head -30'
    echo "--- storage.cfg: any options lines? ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -c "options" /etc/pve/storage.cfg 2>/dev/null || echo 0'
    echo "--- storage.cfg: NFS stanzas (first 3) ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'grep -A 6 "^nfs:" /etc/pve/storage.cfg 2>/dev/null | head -25'
    echo "--- nconnect TCP connections to NFS servers ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | head -20; echo "count:"; ss -tn state established "( dport = :nfs or sport = :nfs )" 2>/dev/null | tail -n +2 | wc -l'
    echo "--- VMs still running? ---"
    ssh "${SSH_OPTS[@]}" "root@$host" 'qm list 2>/dev/null | head -15'
    echo ""
 done
@@ -0,0 +1,58 @@
 #!/bin/bash
 # wait-for-host.sh - polls SSH until host is back, then runs verification.
 # Usage: bash wait-for-host.sh <host>
 set -uo pipefail
 HOST="$1"
 SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=5 -o ServerAliveInterval=5 -o StrictHostKeyChecking=accept-new)
 DEADLINE=$(( $(date +%s) + 600 ))
 echo "Polling $HOST for SSH return (max 10 min)..."
 while [ "$(date +%s)" -lt "$DEADLINE" ]; do
    if ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
        echo "[$(date +%H:%M:%S)] SSH is back!"
        break
    fi
    sleep 10
    echo "  [$(date +%H:%M:%S)] still down..."
 done
 if ! ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo ok' >/dev/null 2>&1; then
    echo "FAILED: $HOST not back after 10 minutes"
    exit 1
 fi
 echo "Waiting 30s for services to settle..."
 sleep 30
 echo ""
 echo "================================================================"
 echo "[$HOST] POST-REBOOT VERIFICATION"
 echo "================================================================"
 echo "--- Uptime (should be < 5 min) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'uptime'
 echo ""
 echo "--- TCP congestion control (expect bbr) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc'
 echo ""
 echo "--- vm.swappiness ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'sysctl vm.swappiness'
 echo ""
 echo "--- scaling_governor ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "(no cpufreq driver)"'
 echo ""
 echo "--- NFS mount options (looking for nconnect=4 + noatime) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -16'
 echo ""
 echo "--- NFS TCP connection count to :2049 (expect ~8 = 4 per server × 2 servers) ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
 echo ""
 echo "--- Running VMs ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null'
 echo ""
 echo "--- Failed services ---"
 ssh "${SSH_OPTS[@]}" "root@$HOST" 'systemctl --failed --no-legend 2>/dev/null | head -10'
 echo ""
 echo "================================================================"
 echo "[$HOST] DONE"
 echo "================================================================"
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
mrcharles	8e5b9558fe	docs: unified README and AGENTS.md for merged repo Replace the KNELServerBuild README with a unified PFVCluster README covering both provisioning and cluster ops. Update AGENTS.md to document the merged repo layout, key scripts, and project context. Consolidate all documentation under docs/. 🤖 Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush <crush@charm.land>	2026-07-28 11:25:16 -05:00
mrcharles	66e7843f27	refactor: reorganize merged repo into clean directory structure Reorganize the merged KNELServerBuild + PFVCluster repo: provisioning/ server provisioning (was ProjectCode/ + Project-Includes/ + Project-ConfigFiles/) tests/ test suite (was Project-Tests/) perf/ Proxmox perf scripts (was top-level *.sh + scripts/) docs/ all documentation (was ProjectDocs/ + PROJECT.md + K8S.md + TODO.md) dns-cluster-setup/ Technitium DNS cluster (unchanged) netinfra/ netinfra audit scripts (unchanged) switches/ switch configs (unchanged) vendor/ vendored KNELShellFramework (unchanged) Update all internal path references from old directory names (ProjectCode/, Project-Includes/, Project-Tests/) to the new ones (provisioning/, tests/) across all scripts. 🤖 Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush <crush@charm.land>	2026-07-28 11:24:39 -05:00
mrcharles	c14b48f39e	merge: combine KNELServerBuild into PFVCluster Merge the KNELServerBuild repository (server provisioning, security hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox cluster ops, performance tuning, fleet audit). Both histories are preserved via --allow-unrelated-histories. The two repos had no source-file collisions; only AGENTS.md and .gitignore conflicted (both resolved by merging content from both). Directory reorganization and doc gardening will follow in subsequent commits. 🤖 Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush <crush@charm.land>	2026-07-28 11:23:22 -05:00
mrcharles	18d57ea4fb	feat: wire Pi-hole to forward knel.net to local Technitium + document setup Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and Technitium containers so Pi-hole can conditionally forward knel.net and Tailscale-reverse queries to the local authoritative Technitium instance (10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds end-to-end documentation for both the reference node (pfv-netboot) and the replicated nodes (pfv-netinfra-01/02). 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-28 05:16:01 -05:00
mrcharles	89469ff028	feat: full re-audit of all 7 hosts with fresh ground truth Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state after the user's PDM migrations: Cnode movements since last audit: - cnode1: tsys1 -> tsys9 - cnode2: tsys6 -> tsys7 - cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5) Wnode changes: - wnode-tsys1 (102): new VM on S2, stopped - wnode-tsys3: RAM bumped 20 -> 28 GB - wnode-tsys6: now running (was stopped) - wnode-tsys9: storage moved S3 -> S2 Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5). Still need 2 more cnode moves for etcd quorum survival. Updated executive summary, k8s distribution tables, storage utilization, and open items with the fresh data. Captured future k8s requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation (RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out with PowerEdge 19xx/2950 systems. Added tsys9 to deploy-check.sh host list.	2026-07-27 22:04:05 -05:00
mrcharles	48cb6842c6	docs: lock in storage philosophy and PDM migration capability Storage philosophy (user directive): - NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation running "cluster of 1" pre-prod jobs before full k8s deployment) - Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM) Clarified that hosts are standalone but managed via Proxmox Datacenter Manager (PDM), which supports VM migration between nodes through the UI -- eliminating the need for manual disk copies in the migration plan. Updated all migration steps to reference PDM storage migrate instead of manual cp commands. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 21:46:48 -05:00
mrcharles	6814374bac	fix: keep UCS on spinning disk, repurpose D3 SSD for k8s/CI workloads UCS (LDAP/AD) does not need SSD. Corrected placement: - ucs-01 stays on D2 (tsys4 HDD) -- no change needed - ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server redundancy only, not for speed D3 SSD (moving to tsys5 Friday) is now freed for workloads that actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a container image registry mirror. Added storage tier characterization per host (tsys4 = bulk/spinning only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added UCS migration TODO item for today. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:51:30 -05:00
mrcharles	019ab468bf	feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday Deep-dive on tsys4/5 storage controllers revealed: - tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0 - tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0 Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB bottleneck and consolidates all fast storage on tsys5: NVMe (local) + D3 SSD + T5-SSD + 4 HDD exports. Updated storage architecture, speed tiers, critical VM placement, failure survival matrix, StorageClass design, and Friday plan. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:50:16 -05:00
mrcharles	ea2048dc87	fix: correct storage bus types after deep-dive on tsys6/7 Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not internal SATA/SAS as previously assumed. Both R610 and R620 only have EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but no drives installed; tsys6's SAS controller is disabled in BIOS. Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable for VM boot disks. Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously documented as HDD). Its 349 GB local-lvm is the fastest existing local storage in the fleet. Updated speed tier table, wnode placement tables, storage distribution matrices, migration plan, and caching strategy across PROJECT.md and K8S.md. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:45:06 -05:00
mrcharles	db575080c9	docs: integrate tsys2 hardware findings from Windows inventory tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on this laptop form factor, no PCIe NIC expansion possible. Updated storage tiers, wnode placement, and k8s labeling to reflect that tsys2 is the fastest-storage worker node. Marked TODO.md items as resolved. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:39:59 -05:00
mrcharles	11ae08f7e6	docs: add TODO.md with tsys2 Windows inventory commands Adds the PowerShell and cmd.exe commands the user needs to run on the Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe details before rebuilding it as Proxmox. Also documents the Friday maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open questions for the next session. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:35:33 -05:00
mrcharles	a791baa085	chore: ignore python bytecode and document git workflow Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode file. Add version control instructions to AGENTS.md for future sessions. 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:31:49 -05:00
mrcharles	422999bf3c	chore: initialize repo with full project state Performance optimization engagement for a 7-host Proxmox R&D cluster. Captures the accumulated work across host tuning, network analysis, fleet assessment, and kubernetes architecture planning. Contents: - Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS nconnect, tuned profiles -- complete on 5 of 7 hosts - Validation + benchmarking scripts: iperf matrix, bond/NFS fixes - Collected host data (returned-logs/): check.sh output from all 7 hosts + iperf results, including newly-validated pfv-tsys9 - AGENTS.md: operating context for AI agents - PROJECT.md: board-ready fleet assessment with VM placement and storage redundancy analysis (40 VMs across 7 hosts) - K8S.md: kubernetes architecture deep-dive covering cnode/wnode distribution, StorageClass design, and ETL/HPC workload planning 💘 Generated with Crush Assisted-by: Crush:glm-5.2	2026-07-27 11:31:29 -05:00