feat: full re-audit of all 7 hosts with fresh ground truth

Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:

Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)

Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2

Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.

Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.

Added tsys9 to deploy-check.sh host list.

AGENTS.md

@@ -51,22 +51,21 @@ Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
 workers). Wnodes: one per hypervisor host across the fleet. Production lives
 on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
 
-## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
+## Critical VM-layer findings (re-audited 2026-07-27 21:50)
 
-1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
+1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
-   k8s cluster dead. etcd quorum lost.
+   more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
-2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
+   actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
-   if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
+2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
-   (tsys4 SSD), ucs-02 to tsys9 local SSD.
+   netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
-3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
+3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
-   unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
+   (Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
-4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
+4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
-   NFS export. Should host ucs-01 (LDAP/AD latency benefit).
+   Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
-5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
+5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
-   used. VMs should be rebalanced to reduce tsys4 blast radius.
+   workstation). Spinning rust = all infrastructure VMs.
-6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
+6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
-   Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
+   VM migration between nodes is done through PDM UI, not manual disk copy.
-   hosts boots from USB 2.0 -- user accepts this risk.
 
 ## Storage network IPs (VLAN1000, 10.100.100.0/24)
 

PROJECT.md

@@ -1,9 +1,13 @@
 # Proxmox Cluster Project Report
 
-**Date:** 2026-07-27
+**Date:** 2026-07-27 (re-audited)
 **Prepared by:** Performance Optimization Engagement
 **Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
 
+**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
+`deploy-check.sh`. VM placements reflect live state after user's PDM
+migrations. This is ground truth.
+
 ---
 
 ## Table of Contents
@@ -26,22 +30,23 @@
 ## 1. Executive Summary
 
 The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
-running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side
+running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
-performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned
+performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
-profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked
+are blocked on physical hardware work scheduled for Friday.
-on physical hardware work scheduled for Friday.
 
-The VM-layer assessment reveals three structural issues:
+**Progress since initial audit:** The user has been actively rebalancing k8s
+nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
+73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
+needed for etcd quorum survival.
 
-| # | Finding | Severity | Fix Cost |
+The VM-layer assessment reveals:
-|---|---------|----------|----------|
-| 1 | **Every pfv-k8s node stores its disk on tsys4 NFS** (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. | **CRITICAL** | $0 (rebalance) |
-| 2 | **No k8s node uses SSD or NVMe.** All boot from NFS on spinning disks, adding latency to every etcd write and container pull. | **HIGH** | $0 (move to local SSD where available) |
-| 3 | **Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only.** tsys4 failure takes down both halves of each pair. | **HIGH** | $0 (move -02 halves to tsys5 NFS) |
 
-Additionally, tsys9 was validated this session and revealed two new findings:
+| # | Finding | Severity | Status |
-its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has
+|---|---------|----------|--------|
-a 250 GB local SSD that no VM is using.
+| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
+| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
+| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
+| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
 
 ---
 
@@ -87,7 +92,7 @@ running Cloudron -- not in this cluster.
 |--------|-----------|------|-----|-------|------|------|------|
 | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
 | ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
-| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
+| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
 
 Non-exported disks on tsys4:
 - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
@@ -103,11 +108,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
-| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% |
+| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
-| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** |
+| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
-| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** |
+| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
 | S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
-| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% |
+| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
 | **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
 
 **tsys5 storage controllers (plenty of free ports):**
@@ -117,9 +122,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 - Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
 - 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
 
-**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
+**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
-has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to
+wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
-reduce tsys4's blast radius.
+(unused, 7 GB). S4 still 99% empty (435 GB free).
+
+T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
 
 **Friday additions:**
 1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
@@ -312,57 +319,98 @@ If tsys4 goes offline, two-thirds of the fleet loses its storage.
 
 ---
 
-## 5. Kubernetes Node Distribution
+## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
 
-### 5.1 pfv-k8s cnode (control plane) placement
+### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
 
-| VMID | Name | Hypervisor | Storage | Disk Type | Host Role Compliant? |
+| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
-|------|------|------------|---------|-----------|---------------------|
+|------|------|------------|---------|-----------|----------|
-| 107 | cnode1 | tsys1 | D5 (tsys4) | HDD | **NO** (infra host) |
+| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
-| 106 | cnode3 | tsys1 | D2 (tsys4) | HDD | **NO** (infra host) |
+| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
-| 603 | cnode2 | tsys6 | D2 (tsys4) | HDD | YES |
+| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
-| 601 | cnode4 | tsys6 | D2 (tsys4) | HDD | YES |
+| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
-| 602 | cnode5 | tsys6 | D5 (tsys4) | HDD | YES |
+| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
 
-**Findings:**
+**Storage distribution:**
-- 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
-- **All 5 cnodes store disks on tsys4 NFS.** Zero use tsys5 storage.
-- 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
-- All cnodes on spinning HDDs. No SSD/NVMe use.
-- **etcd quorum risk:** if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their
-  disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is
-  LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes
-  inoperable.
 
-### 5.2 pfv-k8s wnode (worker) placement
+| Storage server | cnodes | Quorum impact if it fails |
+|---------------|--------|--------------------------|
+| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
+| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
 
-| VMID | Name | Hypervisor | Storage | Disk Type | Status | Host Role Compliant? |
+**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
-|------|------|------------|---------|-----------|--------|---------------------|
+still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
-| 313 | wnode-tsys3 | tsys3 | D5 (tsys4) | HDD | running | YES |
-| 509 | wnode-tsys5 | tsys5 | D2 (tsys4) | HDD | running | **NO** (storage host) |
-| 100 | wnode-tsys6 | tsys6 | D5 (tsys4) | HDD | **STOPPED** | YES (but offline) |
-| 701 | wnode-tsys7 | tsys7 | D5 (tsys4) | HDD | running | YES |
-| 905 | wnode-tsys9 | tsys9 | S3 (tsys5) | HDD | running | **NO** (infra host) |
 
-**Findings:**
+**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
-- 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
+tsys9) -- good host diversity.
-- wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) --
+
-  it depends on tsys4 for its disk despite running on tsys5.
+### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
-- wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
+
-- wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it
+| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
-  is the ONLY k8s node using tsys5 storage (which is good for redundancy).
+|------|------|------------|---------|-----------|-----|--------|----------|
-- All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.
+| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
+| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
+| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
+| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
+| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
+| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
+
+**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
+stopped. wnode-tsys6 restarted.
+
+**Storage distribution:**
+
+| Storage server | wnodes | Notes |
+|---------------|--------|-------|
+| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
+| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
 
 ### 5.3 Summary: k8s node storage distribution
 
 | Storage server | cnodes | wnodes | Total k8s nodes |
 |---------------|--------|--------|-----------------|
-| **tsys4 NFS** | **5 (100%)** | **4 (80%)** | **9 (90%)** |
+| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
-| tsys5 NFS | 0 | 1 (20%) | 1 (10%) |
+| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
 | Local SSD/NVMe | 0 | 0 | 0 (0%) |
 
-**90% of the kubernetes cluster's storage depends on a single host (tsys4).**
+**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
-This is the single most critical risk in the VM layer.
+cnodes on each storage server (60/40 or better) so either server failing
+leaves quorum intact.
+
+### 5.4 Remaining cnode migration needed for etcd quorum survival
+
+To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
+be on tsys5:
+
+| Action | Effect |
+|--------|--------|
+| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
+| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
+| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
+
+**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
+PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
+
+### 5.5 Future k8s architecture (next session -- see K8S.md)
+
+The k8s layer will be tackled soon. Key requirements from user:
+
+- **Platform:** vcluster + Rancher for multi-tenant management
+- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
+- **Workload isolation (vcluster per tenant):**
+  - RackRental workloads (containerlab network labs)
+  - Suborbital ITAR (compliance-restricted)
+  - Suborbital non-ITAR
+  - Starting Line Productions customer workloads
+- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
+  will be brought online during peak solar production for burst capacity.
+  These older cores/ram supplement the main fleet when power is abundant.
+- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
+  both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
+  leftover host capacity) to large (28-32 GB, consuming most of a host).
+- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
+  full final audit will be performed at that time (post-NVMe install on tsys5,
+  post-D3 SSD relocation, post-tsys4 NIC+RAM).
 
 ### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
 
@@ -745,37 +793,65 @@ This belongs in the next budget cycle.
 
 ## 12. Open Items
 
-### 12.1 Friday maintenance window (user action)
+### 12.1 Immediate (do today via PDM)
 
-1. **tsys5:** Plug 2nd ethernet cable into storage NIC, verify bond0 recovers,
+1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
-   apply layer3+4 hash, reboot to activate NFS nconnect.
+   redundancy. UCS stays on HDD.
-2. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
+2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
+3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
+4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
+5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
+
+### 12.2 Friday maintenance window (user action)
+
+1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
    reconfigure `/etc/network/interfaces`, reboot.
-3. **tsys5:** Install PCI NVMe drive, format and mount (recommend local
+2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
-   directory storage for VM images).
+   install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
+   format NVMe as local storage, reboot.
+3. **tsys2:** Load Proxmox (replacing Windows 10).
+4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
 
-### 12.2 Post-hardware validation
+### 12.3 Post-Friday validation
 
-1. Re-run `iperf-full-matrix.sh` to confirm tsys4/tsys5 network recovery.
+1. Re-run iperf matrix: `./iperf-full-matrix.sh`
-2. Run `validate-fixes.sh` on tsys4 and tsys5.
+2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
+3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
+4. Update PROJECT.md with post-hardware numbers.
 
-### 12.3 Data gaps
+### 12.4 Future: Kubernetes deep-dive (see K8S.md)
+
+Next major workstream. Requirements captured:
+- **vcluster + Rancher** for multi-tenant k8s management
+- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
+- **Workload isolation** via separate vclusters:
+  - RackRental (containerlab)
+  - Suborbital ITAR
+  - Suborbital non-ITAR
+  - Starting Line Productions customer workloads
+- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
+  during peak solar production for burst capacity
+- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
+- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
+- **Spinning rust for all infrastructure VMs**
+
+### 12.5 Data gaps
 
 | Gap | How to close |
 |-----|--------------|
-| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR before planning tsys5 role changes |
+| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
-| tsys2 post-rebuild baseline | Run `check.sh` once it is converted to Proxmox |
+| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
 | tsys5 NVMe size and model | Confirm after Friday installation |
-| Whether hosts form a Proxmox cluster or are standalone | Check `pvecm status` on each host |
 | tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
+| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
 
-### 12.4 Scripts and tools available
+### 12.6 Scripts and tools available
 
 | Script | Purpose |
 |--------|---------|
 | `scripts/check.sh` | Read-only data collector (run on hosts) |
 | `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
-| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash (live) |
+| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
 | `validate-fixes.sh` | Read-only validation of all applied changes |
 | `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
-| `deploy-check.sh` | Deploy check.sh to all hosts via SSH |
+| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |

deploy-check.sh

@@ -24,7 +24,7 @@ LOG_DIR="$SCRIPT_DIR/returned-logs"
 mkdir -p "$LOG_DIR"
 
 # ONLY the hosts the user told us are alive.
-HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
+HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 
 # Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
 SSH_OPTS=(-o BatchMode=yes

netinfra/baseline.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# baseline.sh — quick read-only baseline of a target node.
+set -u
+hdr() { printf '\n=== %s ===\n' "$1"; }
+hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
+hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
+hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
+hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
+hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
+hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
+hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
+hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
+hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
+hdr "HOME"; ls -la /home/localuser 2>/dev/null | head

netinfra/gather-configs.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
+# contents, and chrony NTP details from pfv-netboot for replication.
+set -u
+hdr() { printf '\n=== %s ===\n' "$1"; }
+
+hdr "TECHNITIUM dns.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
+
+hdr "TECHNITIUM auth.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
+
+hdr "TECHNITIUM log.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
+
+hdr "TECHNITIUM scopes/Default.scope"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
+
+hdr "TECHNITIUM zone: knel.net.zone"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
+
+hdr "TECHNITIUM all zone files listing + sizes"
+sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
+
+hdr "TECHNITIUM reverse zones (one sample)"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
+
+hdr "TECHNITIUM top-level listing"
+sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
+
+hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
+cat /root/pihole/etc-pihole/pihole.toml 2>&1
+
+hdr "PI-HOLE dnsmasq.conf"
+cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
+
+hdr "PI-HOLE versions file"
+cat /root/pihole/etc-pihole/versions 2>&1
+
+hdr "GRAVITY.DB contents (host sqlite3 if present)"
+if command -v sqlite3 >/dev/null 2>&1; then
+    G=/root/pihole/etc-pihole/gravity.db
+    echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
+    echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
+    echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
+    echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
+    echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
+    echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
+    echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
+else
+    echo "(sqlite3 not on host PATH)"
+fi
+
+hdr "DONE"

netinfra/setup-netinfra.sh

@@ -0,0 +1,283 @@
+#!/usr/bin/env bash
+# =============================================================================
+# setup-netinfra.sh
+# -----------------------------------------------------------------------------
+# Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
+# onto pfv-netinfra-01 and pfv-netinfra-02.
+#
+# DESIGN
+#   * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
+#     from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
+#   * The targets cannot SSH to pfv-netboot directly, so config tarballs are
+#     relayed through this workstation:
+#         ssh netboot 'sudo tar -cf - ...'  |  ssh target 'sudo tar -xf - ...'
+#   * Services are deployed under /home/localuser/services/<svc>/ on each
+#     target so localuser can manage them (mirrors netboot's localuser-owned
+#     pihole data dir). `sudo docker` is used since localuser is not in the
+#     docker group (same as on netboot).
+#
+# SERVICES
+#   pihole      pihole/pihole:latest   :53 tcp/udp  :10002->80  :10003->443
+#   ntp (chrony) dockurr/chrony        <tailscale-ip>:123:123/udp
+#   technitium  technitium/dns-server  :5300->53 tcp/udp  :5380  :53443
+#     (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
+#      The knel.net authoritative zone + Tailscale reverse zones are preserved
+#      verbatim from netboot's orphaned dns_tsys-dns-config volume.)
+#
+# USAGE
+#   ./setup-netinfra.sh                # deploy to BOTH nodes
+#   ./setup-netinfra.sh pfv-netinfra-01   # deploy to one node
+#   ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
+# =============================================================================
+
+set -euo pipefail
+
+NETBOOT="localuser@pfv-netboot"
+SVC_ROOT="/home/localuser/services"
+PIHOLE_PW='Gransyan1!'   # replicated verbatim from netboot compose
+
+log()  { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
+warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
+
+# Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
+declare -A NODE_TSIP=(
+  [pfv-netinfra-01]="100.70.181.72"
+  [pfv-netinfra-02]="100.93.194.82"
+)
+
+on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
+
+#------------------------------------------------------------------------------
+# Verify-only mode
+#------------------------------------------------------------------------------
+verify_node() {
+  local node="$1" tsip="${NODE_TSIP[$1]}"
+  log "VERIFY $node (tailscale $tsip)"
+  on_node "$node" "bash -s" <<EOF
+set +e
+echo "### containers ###"
+sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
+echo
+echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
+dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
+echo "### Pi-hole web (curl :10002) ###"
+curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
+echo
+echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
+dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
+echo "### Technitium web (curl :5380) ###"
+curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
+echo
+echo "### NTP on tailscale IP (ntpdig -p 1) ###"
+timeout 5 ntpdig -t1 -c1 -p1 $tsip 2>&1 || echo "(ntpdig not available or no response)"
+echo "### chrony container sources ###"
+sudo docker exec tsys-ntp chronyc -n sources 2>&1 | head -8
+EOF
+}
+
+#------------------------------------------------------------------------------
+# Deploy to one node
+#------------------------------------------------------------------------------
+deploy_node() {
+  local node="$1" tsip="${NODE_TSIP[$1]}"
+  log "==== DEPLOY $node (tailscale $tsip) ===="
+
+  # ---- 1. Prepare directories on the target -------------------------------
+  log "$node: create service dirs"
+  on_node "$node" "bash -s" <<EOF
+set -e
+sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
+sudo chown -R localuser:localuser $SVC_ROOT
+EOF
+
+  # ---- 2. Write compose files (as localuser) ------------------------------
+  log "$node: write docker-compose files"
+  on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
+services:
+  pihole:
+    container_name: pihole
+    image: pihole/pihole:latest
+    hostname: pihole
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: 'Gransyan1!'
+      FTLCONF_dns_listeningMode: 'all'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+    cap_add:
+      - SYS_NICE
+    restart: always
+YAML
+
+  on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
+services:
+  ntp:
+    image: dockurr/chrony
+    container_name: tsys-ntp
+    environment:
+      NTP_SERVERS: "pool.ntp.org"
+    ports:
+      - "$tsip:123:123/udp"
+    restart: always
+YAML
+
+  on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
+services:
+  technitium:
+    image: technitium/dns-server
+    container_name: tsys-dns
+    ports:
+      - "5300:53/tcp"
+      - "5300:53/udp"
+      - "5380:5380/tcp"
+      - "53443:53443/tcp"
+    volumes:
+      - './config:/etc/dns'
+    restart: always
+YAML
+
+  # ---- 3. Relay Pi-hole config from netboot -> target ---------------------
+  log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
+  if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
+    log "$node: Pi-hole config already present; skipping copy"
+  else
+    on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
+    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
+        "sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
+                                   --exclude='etc-pihole/listsCache' \
+                                   --exclude='etc-pihole/gravity_backups' \
+                                   --exclude='etc-pihole/config_backups' \
+                                   etc-pihole" \
+      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
+  fi
+
+  # ---- 4. Relay Technitium config from netboot orphaned volume -----------
+  log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
+  if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
+    log "$node: Technitium config already present; skipping copy"
+  else
+    on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
+    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
+        "sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
+      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
+  fi
+
+  # ---- 5. Adapt copied config: Pi-hole interface name ---------------------
+  log "$node: adapt Pi-hole pihole.toml interface name to actual iface"
+  on_node "$node" "bash -s" <<'EOF'
+set -e
+IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
+IFACE=${IFACE:-ens18}
+TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
+if sudo test -f "$TOML"; then
+  sudo sed -i "s|^  interface = .*|  interface = \"$IFACE\"  ### ADAPTED from eth0 on clone|" "$TOML"
+  echo "set interface=$IFACE"
+else
+  echo "(pihole.toml not present; FTL will create it on first run)"
+fi
+# Strip netboot-specific primary upstream 192.168.3.16? -> keep, it is reachable on LAN.
+EOF
+
+  # ---- 6. Pull images -----------------------------------------------------
+  log "$node: docker compose pull (pihole, ntp, technitium)"
+  on_node "$node" "bash -s" <<EOF
+for c in pihole ntp technitium; do
+  sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
+done
+EOF
+
+  # ---- 6a. Pi-hole up -----------------------------------------------------
+  log "$node: bring up Pi-hole"
+  on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
+
+  # ---- 6b. NTP -- only deploy the chrony container if nothing already ----
+  # serves UDP/123 on the host. The targets already run a bare-metal ntpsec
+  # daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
+  # netboot's own bare-metal ntpsec. netboot additionally runs a chrony
+  # container on its tailscale IP, but that only works there because ntpsec
+  # there does not pre-bind the specific tailscale-IP socket. On these targets
+  # ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
+  # redundant anyway. We therefore keep the host ntpsec as the NTP service.
+  log "$node: NTP -- check whether host already serves UDP/123"
+  on_node "$node" "bash -s" <<'EOF'
+set +e
+HOST_NTP=$(sudo ss -lun 2>/dev/null | awk '$5 ~ /:123$/ {print}' | head -1)
+if [ -n "$HOST_NTP" ]; then
+  echo "Host already serves UDP/123 ($HOST_NTP); host daemon:"
+  for u in ntpsec ntp chrony openntpd systemd-timesyncd; do
+    systemctl is-active "$u" 2>/dev/null | grep -q active && echo "  -> $u active"
+  done
+  echo "Skipping chrony container (host NTP provides the service)."
+  echo "Removing any stale tsys-ntp container..."
+  sudo docker rm -f tsys-ntp 2>/dev/null || true
+else
+  echo "Nothing serving UDP/123; starting chrony container."
+  sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
+fi
+EOF
+
+  # ---- 6c. Technitium up --------------------------------------------------
+  log "$node: bring up Technitium"
+  on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
+
+  # ---- 7. Wait for Pi-hole health -----------------------------------------
+  log "$node: wait for Pi-hole to become healthy"
+  on_node "$node" "bash -s" <<'EOF'
+for i in $(seq 1 30); do
+  st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
+  echo "  pihole health: $st"
+  [ "$st" = "healthy" ] && break
+  sleep 4
+done
+EOF
+
+  # ---- 8. Technitium defensive fallback -----------------------------------
+  # If the copied (binary) config from an older Technitium version makes the
+  # new container crash, move it aside and let Technitium start fresh so the
+  # service is at least up (admin reachable) rather than crash-looping.
+  log "$node: check Technitium health (fallback to fresh config if crash)"
+  on_node "$node" "bash -s" <<'EOF'
+set +e
+sleep 8
+rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
+running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
+if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
+  echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
+  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
+  sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
+  sudo mkdir -p /home/localuser/services/technitium/config
+  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
+  echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
+else
+  echo "Technitium OK (running=$running restarts=$rst)"
+fi
+EOF
+
+  # ---- 9. Final status -----------------------------------------------------
+  log "$node: final container status"
+  on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
+}
+
+#------------------------------------------------------------------------------
+# Main
+#------------------------------------------------------------------------------
+main() {
+  local mode="${2:-deploy}"
+  if [ "${1:-all}" = "all" ]; then
+    targets=(pfv-netinfra-01 pfv-netinfra-02)
+  else
+    targets=("$1")
+  fi
+  for t in "${targets[@]}"; do
+    : "${NODE_TSIP[$t]:?unknown node $t}"
+    if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
+  done
+  log "DONE"
+}
+
+main "$@"

returned-logs/_marker.pfv-tsys1

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys1.log 911607 17324
+ok pfv-tsys1.log 951792 16094

returned-logs/_marker.pfv-tsys3

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys3.log 725429 10091
+ok pfv-tsys3.log 750211 11660

returned-logs/_marker.pfv-tsys4

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys4.log 603525 11030
+ok pfv-tsys4.log 617476 11115

returned-logs/_marker.pfv-tsys5

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys5.log 1047724 17559
+ok pfv-tsys5.log 1064227 18745

returned-logs/_marker.pfv-tsys6

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys6.log 743495 14563
+ok pfv-tsys6.log 907424 16997

returned-logs/_marker.pfv-tsys7

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys7.log 896348 17341
+ok pfv-tsys7.log 1114764 18714

returned-logs/_marker.pfv-tsys9

@@ -0,0 +1,2 @@
+running
+ok pfv-tsys9.log 793781 14230