feat: wire Pi-hole to forward knel.net to local Technitium + document setup

Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

AGENTS.md

@@ -51,22 +51,21 @@ Cnodes weighted to tsys1/9 (lightweight hosts, keep heavy hosts free for
 workers). Wnodes: one per hypervisor host across the fleet. Production lives
 on a VPS in Reston VA (Cloudron) -- this cluster is R&D only.
 
-## Critical VM-layer findings (see PROJECT.md + K8S.md for full detail)
+## Critical VM-layer findings (re-audited 2026-07-27 21:50)
 
-1. **ALL 10 pfv-k8s nodes store disks on tsys4 NFS.** tsys4 failure = entire
+1. **4 of 5 cnodes still on tsys4 NFS** (cnode5 moved to tsys5 S2). Need 2
-   k8s cluster dead. etcd quorum lost.
+   more moves (cnode3→S3, cnode4→S2) for etcd quorum survival. User has been
-2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS only.** Both halves die
+   actively rebalancing via PDM -- storage distribution improved 90/10 to 73/27.
-   if tsys4 goes down. Recommended: netinfra-02 to S3 (tsys5), ucs-01 to D3
+2. **Both -01/-02 pairs (netinfra, UCS) on tsys4 NFS.** TODO today: move
-   (tsys4 SSD), ucs-02 to tsys9 local SSD.
+   netinfra-02 to S3, ucs-02 to S2 (both tsys5 HDD).
-3. **No k8s node uses SSD or NVMe.** All on NFS-over-HDD. tsys9 has 136 GB
+3. **No k8s node uses SSD/NVMe yet.** tsys3 has 349 GB unused local NVMe
-   unused local SSD; tsys3 has 349 GB unused local **NVMe** (Samsung PM961).
+   (Samsung PM961), tsys9 has 136 GB local SSD. Deferred to k8s session.
-4. **D3 SSD (tsys4) is 99% empty (445 GB free).** Nobody using the only SSD
+4. **tsys6/7 local-lvm is USB 2.0 portable HDD** (~30 MB/s). Slower than NFS.
-   NFS export. Should host ucs-01 (LDAP/AD latency benefit).
+   Do NOT use for VM storage. User accepts OS-on-USB for these hosts.
-5. **S2/S3/S4 (tsys5) are all 99% empty.** Secondary storage server barely
+5. **Storage philosophy:** NVMe/SSD = k8s scratch + ultix-streaming (dev
-   used. VMs should be rebalanced to reduce tsys4 blast radius.
+   workstation). Spinning rust = all infrastructure VMs.
-6. **tsys6/7 local-lvm is USB 2.0 portable HDD** (WD My Passport, ~30 MB/s).
+6. **Hosts are standalone, managed via PDM** (Proxmox Datacenter Manager).
-   Slower than NFS-HDD. Do NOT use for VM storage. Entire Proxmox OS on these
+   VM migration between nodes is done through PDM UI, not manual disk copy.
-   hosts boots from USB 2.0 -- user accepts this risk.
 
 ## Storage network IPs (VLAN1000, 10.100.100.0/24)
 
@@ -112,9 +111,13 @@ use `deploy-check.sh` / `deploy-tuning.sh` wrapper patterns instead.
    `/etc/network/interfaces` with awk (sed fails on tab-indented stanza).
 4. Always shellcheck before shipping: `docker run --rm -v "$PWD:/mnt"
    koalaman/shellcheck:stable --severity=style scripts/*.sh`
-5. The hosts appear to be **standalone Proxmox installs**, not a shared
+5. The hosts are **standalone Proxmox installs** (not a pvecm cluster), but
-   cluster (storage.cfg differs per host). VMs are pinned to their host.
+   managed collectively via **Proxmox Datacenter Manager (PDM)**. PDM supports
-   Moving VMs requires manual disk migration, not live migration.
+   VM migration between nodes via the UI. NFS exports are visible to all nodes;
+   local storage migration is done through PDM's "Storage Migrate" function.
+6. **Storage philosophy:** NVMe/SSD is for k8s worker scratch + ultix-streaming
+   (developer workstation for "cluster of 1" pre-prod jobs). Spinning rust
+   hosts all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM).
 
 ## Friday walkthrough (user action)
 

K8S.md

@@ -328,16 +328,20 @@ spec:
 
 ### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
 
-The SSD NFS exports are a middle tier. **Both consolidated on tsys5** after
+**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
-the Friday D3 relocation:
+scratch space, with the exception of ultix-streaming which stays on T5-SSD.
-- **D3 (tsys5 SAS, 445 GB free):** CI/CD artifacts, container image cache,
+Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
-  UCS/LDAP. Currently 99% empty. Fastest NFS tier after the move off USB.
+SIEM, etc.).
-- **T5-SSD (tsys5 SAS, 140 GB free):** SIEM databases, monitoring data.
+
-  Currently 35% used by ultix-streaming.
+The SSD NFS exports:
+- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
+  cache, ephemeral volumes). Currently 99% empty.
+- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
+  occupies 83 GB. Remaining 140 GB available for k8s use.
 
 **tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
-This simplifies the StorageClass design -- latency-sensitive PVCs target
+This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
-tsys5 exports, bulk PVCs target either server.
+tsys5 SSD exports, bulk PVCs target either server.
 
 ### 6.4 NFS data distribution across storage servers
 
@@ -413,6 +417,13 @@ if the data is already cached in the page cache.
 
 ## 8. Migration Plan
 
+**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
+Datacenter Manager (PDM)** manages them collectively and supports VM migration
+between nodes. Storage migration can be done via the PDM/Proxmox UI rather
+than manual disk copies -- the destination node just needs access to the target
+storage (which all nodes have for NFS exports, and local storage can be
+migrated through the UI's "Storage Migrate" function).
+
 ### 8.1 Phase 1: Friday (after hardware work)
 
 After tsys5 cable + NVMe and tsys4 NIC + RAM:
@@ -451,9 +462,9 @@ Plan for a maintenance window with the k8s cluster briefly down.
 
 ### 8.5 Phase 5: Critical VM relocation
 
-1. Move netinfra-02 (VM 904) from D2 to S3.
+1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
-2. Move ucs-01 (VM 108) from D2 to D3 (SSD).
+2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
-3. Move ucs-02 (VM 902) from D5 to tsys9 local-lvm (SSD).
+3. (No change to T5-SSD -- ultix-streaming stays.)
 
 ---
 

PROJECT.md

@@ -1,9 +1,13 @@
 # Proxmox Cluster Project Report
 
-**Date:** 2026-07-27
+**Date:** 2026-07-27 (re-audited)
 **Prepared by:** Performance Optimization Engagement
 **Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
 
+**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
+`deploy-check.sh`. VM placements reflect live state after user's PDM
+migrations. This is ground truth.
+
 ---
 
 ## Table of Contents
@@ -26,22 +30,23 @@
 ## 1. Executive Summary
 
 The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
-running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side
+running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
-performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned
+performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
-profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked
+are blocked on physical hardware work scheduled for Friday.
-on physical hardware work scheduled for Friday.
 
-The VM-layer assessment reveals three structural issues:
+**Progress since initial audit:** The user has been actively rebalancing k8s
+nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
+73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
+needed for etcd quorum survival.
 
-| # | Finding | Severity | Fix Cost |
+The VM-layer assessment reveals:
-|---|---------|----------|----------|
-| 1 | **Every pfv-k8s node stores its disk on tsys4 NFS** (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. | **CRITICAL** | $0 (rebalance) |
-| 2 | **No k8s node uses SSD or NVMe.** All boot from NFS on spinning disks, adding latency to every etcd write and container pull. | **HIGH** | $0 (move to local SSD where available) |
-| 3 | **Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only.** tsys4 failure takes down both halves of each pair. | **HIGH** | $0 (move -02 halves to tsys5 NFS) |
 
-Additionally, tsys9 was validated this session and revealed two new findings:
+| # | Finding | Severity | Status |
-its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has
+|---|---------|----------|--------|
-a 250 GB local SSD that no VM is using.
+| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
+| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
+| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
+| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
 
 ---
 
@@ -87,7 +92,7 @@ running Cloudron -- not in this cluster.
 |--------|-----------|------|-----|-------|------|------|------|
 | D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
 | ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
-| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
+| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
 
 Non-exported disks on tsys4:
 - sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
@@ -103,11 +108,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 
 | Export | Disk model | Type | Bus | Total | Used | Free | Use% |
 |--------|-----------|------|-----|-------|------|------|------|
-| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% |
+| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
-| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** |
+| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
-| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** |
+| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
 | S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
-| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% |
+| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
 | **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
 
 **tsys5 storage controllers (plenty of free ports):**
@@ -117,9 +122,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
 - Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
 - 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
 
-**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
+**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
-has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to
+wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
-reduce tsys4's blast radius.
+(unused, 7 GB). S4 still 99% empty (435 GB free).
+
+T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
 
 **Friday additions:**
 1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
@@ -187,7 +194,7 @@ NFS-over-HDD and unsuitable for wnode boot disks.
 |------|-------|------------|----------|
 | **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
 | **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
-| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs |
+| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
 | **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
 | **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
 | **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
@@ -312,57 +319,98 @@ If tsys4 goes offline, two-thirds of the fleet loses its storage.
 
 ---
 
-## 5. Kubernetes Node Distribution
+## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
 
-### 5.1 pfv-k8s cnode (control plane) placement
+### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
 
-| VMID | Name | Hypervisor | Storage | Disk Type | Host Role Compliant? |
+| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
-|------|------|------------|---------|-----------|---------------------|
+|------|------|------------|---------|-----------|----------|
-| 107 | cnode1 | tsys1 | D5 (tsys4) | HDD | **NO** (infra host) |
+| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
-| 106 | cnode3 | tsys1 | D2 (tsys4) | HDD | **NO** (infra host) |
+| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
-| 603 | cnode2 | tsys6 | D2 (tsys4) | HDD | YES |
+| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
-| 601 | cnode4 | tsys6 | D2 (tsys4) | HDD | YES |
+| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
-| 602 | cnode5 | tsys6 | D5 (tsys4) | HDD | YES |
+| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
 
-**Findings:**
+**Storage distribution:**
-- 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
-- **All 5 cnodes store disks on tsys4 NFS.** Zero use tsys5 storage.
-- 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
-- All cnodes on spinning HDDs. No SSD/NVMe use.
-- **etcd quorum risk:** if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their
-  disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is
-  LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes
-  inoperable.
 
-### 5.2 pfv-k8s wnode (worker) placement
+| Storage server | cnodes | Quorum impact if it fails |
+|---------------|--------|--------------------------|
+| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
+| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
 
-| VMID | Name | Hypervisor | Storage | Disk Type | Status | Host Role Compliant? |
+**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
-|------|------|------------|---------|-----------|--------|---------------------|
+still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
-| 313 | wnode-tsys3 | tsys3 | D5 (tsys4) | HDD | running | YES |
-| 509 | wnode-tsys5 | tsys5 | D2 (tsys4) | HDD | running | **NO** (storage host) |
-| 100 | wnode-tsys6 | tsys6 | D5 (tsys4) | HDD | **STOPPED** | YES (but offline) |
-| 701 | wnode-tsys7 | tsys7 | D5 (tsys4) | HDD | running | YES |
-| 905 | wnode-tsys9 | tsys9 | S3 (tsys5) | HDD | running | **NO** (infra host) |
 
-**Findings:**
+**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
-- 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
+tsys9) -- good host diversity.
-- wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) --
+
-  it depends on tsys4 for its disk despite running on tsys5.
+### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
-- wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
+
-- wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it
+| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
-  is the ONLY k8s node using tsys5 storage (which is good for redundancy).
+|------|------|------------|---------|-----------|-----|--------|----------|
-- All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.
+| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
+| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
+| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
+| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
+| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
+| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
+
+**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
+stopped. wnode-tsys6 restarted.
+
+**Storage distribution:**
+
+| Storage server | wnodes | Notes |
+|---------------|--------|-------|
+| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
+| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
 
 ### 5.3 Summary: k8s node storage distribution
 
 | Storage server | cnodes | wnodes | Total k8s nodes |
 |---------------|--------|--------|-----------------|
-| **tsys4 NFS** | **5 (100%)** | **4 (80%)** | **9 (90%)** |
+| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
-| tsys5 NFS | 0 | 1 (20%) | 1 (10%) |
+| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
 | Local SSD/NVMe | 0 | 0 | 0 (0%) |
 
-**90% of the kubernetes cluster's storage depends on a single host (tsys4).**
+**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
-This is the single most critical risk in the VM layer.
+cnodes on each storage server (60/40 or better) so either server failing
+leaves quorum intact.
+
+### 5.4 Remaining cnode migration needed for etcd quorum survival
+
+To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
+be on tsys5:
+
+| Action | Effect |
+|--------|--------|
+| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
+| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
+| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
+
+**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
+PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
+
+### 5.5 Future k8s architecture (next session -- see K8S.md)
+
+The k8s layer will be tackled soon. Key requirements from user:
+
+- **Platform:** vcluster + Rancher for multi-tenant management
+- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
+- **Workload isolation (vcluster per tenant):**
+  - RackRental workloads (containerlab network labs)
+  - Suborbital ITAR (compliance-restricted)
+  - Suborbital non-ITAR
+  - Starting Line Productions customer workloads
+- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
+  will be brought online during peak solar production for burst capacity.
+  These older cores/ram supplement the main fleet when power is abundant.
+- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
+  both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
+  leftover host capacity) to large (28-32 GB, consuming most of a host).
+- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
+  full final audit will be performed at that time (post-NVMe install on tsys5,
+  post-D3 SSD relocation, post-tsys4 NIC+RAM).
 
 ### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
 
@@ -679,19 +727,22 @@ This maps to two k8s StorageClasses:
 - `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
 - `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
 
-### 10.4 HIGH: Use D3 SSD and T5-SSD for latency-sensitive VMs
+### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
 
-Currently wasted:
+**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
-- **D3 (tsys4 SK hynix SSD, 445 GB free, 0% used)** -- nobody using it
+space and ultix-streaming (developer workstation running "cluster of 1"
-- **T5-SSD (tsys5 Samsung 860 PRO, 140 GB free)** -- only ultix-streaming
+pre-production jobs). Spinning rust hosts all other infrastructure VMs**
+(UCS, netinfra, LibreNMS, SIEM, etc.).
 
-Recommended uses for D3 (after moving ucs-01 there, 32 GB):
+SSD allocation after Friday:
-- Remaining D3 space (413 GB): sectestbed-k8s-cnode, sectestbed-k8s-wnode
+- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
-  (move from local-nonprod HDD to NFS-SSD for faster test runs)
+  the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
+- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
+  (developer workstation, runs single-node test jobs before k8s). Remaining
+  140 GB available for k8s.
 
-Recommended uses for T5-SSD (after moving nothing new, ultix-streaming uses 83 GB):
+**Deep-dive on exact k8s scratch allocation is deferred to the next session**
-- Remaining T5-SSD space (140 GB): tsys-siem (VM 901, 132 GB disk currently
+(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
-  on D2 HDD) would benefit from SSD for SIEM database writes.
 
 ### 10.5 MEDIUM: Restart wnode-tsys6
 
@@ -742,37 +793,65 @@ This belongs in the next budget cycle.
 
 ## 12. Open Items
 
-### 12.1 Friday maintenance window (user action)
+### 12.1 Immediate (do today via PDM)
 
-1. **tsys5:** Plug 2nd ethernet cable into storage NIC, verify bond0 recovers,
+1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
-   apply layer3+4 hash, reboot to activate NFS nconnect.
+   redundancy. UCS stays on HDD.
-2. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
+2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
+3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
+4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
+5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
+
+### 12.2 Friday maintenance window (user action)
+
+1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
    reconfigure `/etc/network/interfaces`, reboot.
-3. **tsys5:** Install PCI NVMe drive, format and mount (recommend local
+2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
-   directory storage for VM images).
+   install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
+   format NVMe as local storage, reboot.
+3. **tsys2:** Load Proxmox (replacing Windows 10).
+4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
 
-### 12.2 Post-hardware validation
+### 12.3 Post-Friday validation
 
-1. Re-run `iperf-full-matrix.sh` to confirm tsys4/tsys5 network recovery.
+1. Re-run iperf matrix: `./iperf-full-matrix.sh`
-2. Run `validate-fixes.sh` on tsys4 and tsys5.
+2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
+3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
+4. Update PROJECT.md with post-hardware numbers.
 
-### 12.3 Data gaps
+### 12.4 Future: Kubernetes deep-dive (see K8S.md)
+
+Next major workstream. Requirements captured:
+- **vcluster + Rancher** for multi-tenant k8s management
+- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
+- **Workload isolation** via separate vclusters:
+  - RackRental (containerlab)
+  - Suborbital ITAR
+  - Suborbital non-ITAR
+  - Starting Line Productions customer workloads
+- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
+  during peak solar production for burst capacity
+- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
+- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
+- **Spinning rust for all infrastructure VMs**
+
+### 12.5 Data gaps
 
 | Gap | How to close |
 |-----|--------------|
-| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR before planning tsys5 role changes |
+| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
-| tsys2 post-rebuild baseline | Run `check.sh` once it is converted to Proxmox |
+| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
 | tsys5 NVMe size and model | Confirm after Friday installation |
-| Whether hosts form a Proxmox cluster or are standalone | Check `pvecm status` on each host |
 | tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
+| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
 
-### 12.4 Scripts and tools available
+### 12.6 Scripts and tools available
 
 | Script | Purpose |
 |--------|---------|
 | `scripts/check.sh` | Read-only data collector (run on hosts) |
 | `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
-| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash (live) |
+| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
 | `validate-fixes.sh` | Read-only validation of all applied changes |
 | `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
-| `deploy-check.sh` | Deploy check.sh to all hosts via SSH |
+| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |

TODO.md

@@ -217,12 +217,11 @@ servers for redundancy.
 | ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
 | ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
 
-**To migrate ucs-02 to S2:**
+**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
-1. Shut down VM 902 on tsys9
+1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
-2. Copy disk: `cp /mnt/pve/D5/902/vm-902-disk-0.qcow2 /mnt/pve/S2/902/`
+2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
-3. Update VM 902 config: change `scsi0` from `D5:902/...` to `S2:902/...`
+   (both are NFS exports visible to tsys9, so this is a storage-only migration)
-4. Start VM 902, verify it boots and LDAP/AD services are healthy
+3. Verify VM 902 boots and LDAP/AD services are healthy after migration
-5. Remove old disk from D5
 
 Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
 ucs-02 needs to move -- it should be on a different storage server than

deploy-check.sh

@@ -24,7 +24,7 @@ LOG_DIR="$SCRIPT_DIR/returned-logs"
 mkdir -p "$LOG_DIR"
 
 # ONLY the hosts the user told us are alive.
-HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
+HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9)
 
 # Common ssh options: non-interactive, fail-fast, no host-key prompt blocking.
 SSH_OPTS=(-o BatchMode=yes

netinfra/audit-netboot.sh

@@ -0,0 +1,297 @@
+#!/usr/bin/env bash
+# =============================================================================
+# audit-netboot.sh
+# -----------------------------------------------------------------------------
+# READ-ONLY audit of the reference node "pfv-netboot".
+#
+# Purpose: gather enough information to replicate its Pi-hole, Technitium DNS,
+# and NTP services onto pfv-netinfra-01 and pfv-netinfra-02.
+#
+# Guarantees:
+#   * NO writes, NO installs, NO service restarts, NO network changes.
+#   * Every command below is read-only (status, cat, ls, ss, ps, sqlite3 SELECT).
+#
+# Intended to be run as the "localuser" account on pfv-netboot:
+#
+#     ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh > netboot-audit.txt
+#
+# or, if sudo is needed for a few reads, the script will try `sudo -n` for
+# specific files that are normally root-readable only. It will NEVER use sudo
+# to write or modify anything.
+# =============================================================================
+
+set -u
+AUDIT_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+
+# Helper: print a clearly delimited section header.
+section() {
+    printf '\n========================================================================\n'
+    printf 'SECTION: %s\n' "$1"
+    printf '========================================================================\n'
+}
+
+# Helper: read a file with sudo if needed, silently skip if missing.
+read_file() {
+    local path="$1"
+    if [ -r "$path" ]; then
+        printf '--- %s (uid-readable) ---\n' "$path"
+        cat "$path" 2>/dev/null
+    elif sudo -n true 2>/dev/null; then
+        if sudo -n test -e "$path"; then
+            printf '--- %s (via sudo -n) ---\n' "$path"
+            sudo -n cat "$path" 2>/dev/null
+        else
+            printf '--- %s : NOT FOUND ---\n' "$path"
+        fi
+    else
+        printf '--- %s : NOT READABLE (no passwordless sudo) ---\n' "$path"
+    fi
+    printf '\n'
+}
+
+list_dir() {
+    local path="$1"
+    printf '--- ls -la %s ---\n' "$path"
+    ls -la "$path" 2>/dev/null || sudo -n ls -la "$path" 2>/dev/null || printf '(cannot list %s)\n' "$path"
+    printf '\n'
+}
+
+printf '##### AUDIT START %s #####\n' "$AUDIT_DATE"
+printf 'Audit host: %s\n' "$(hostname -f 2>/dev/null || hostname)"
+printf 'Audit user: %s\n' "$(id -un 2>/dev/null)"
+printf 'Script:     audit-netboot.sh (READ-ONLY)\n'
+
+# -----------------------------------------------------------------------------
+# 1. System basics
+# -----------------------------------------------------------------------------
+section "1. SYSTEM BASICS"
+echo "-- uname --"; uname -a
+echo "-- /etc/os-release --"; cat /etc/os-release 2>/dev/null
+echo "-- uptime --"; uptime
+echo "-- timezone --"; timedatectl 2>/dev/null || cat /etc/timezone 2>/dev/null || date
+echo "-- arch --"; dpkg --print-architecture 2>/dev/null || uname -m
+echo "-- memory --"; free -h 2>/dev/null
+echo "-- disk --"; df -h / 2>/dev/null
+echo "-- cpu count --"; nproc 2>/dev/null
+
+# -----------------------------------------------------------------------------
+# 2. Network configuration
+# -----------------------------------------------------------------------------
+section "2. NETWORK"
+echo "-- hostname --"; hostname; hostname -f 2>/dev/null; hostname -I 2>/dev/null
+echo "-- ip addr --"; ip -br addr 2>/dev/null; echo; ip addr 2>/dev/null
+echo "-- default route --"; ip route 2>/dev/null
+echo "-- /etc/resolv.conf --"; cat /etc/resolv.conf 2>/dev/null
+echo "-- /etc/hosts --"; cat /etc/hosts 2>/dev/null
+echo "-- listening TCP/UDP sockets --"
+ss -tlnup 2>/dev/null || sudo -n ss -tlnup 2>/dev/null
+echo "-- /etc/network/interfaces --"; cat /etc/network/interfaces 2>/dev/null
+echo "-- netplan --"; ls -la /etc/netplan/ 2>/dev/null; for f in /etc/netplan/*.yaml; do [ -e "$f" ] && { echo "--- $f ---"; cat "$f"; }; done 2>/dev/null
+echo "-- systemd-networkd --"; ls -la /etc/systemd/network/ 2>/dev/null; networkctl status 2>/dev/null | head -40
+
+# -----------------------------------------------------------------------------
+# 3. DNS / DHCP / NTP related systemd units
+# -----------------------------------------------------------------------------
+section "3. RELEVANT SYSTEMD UNITS"
+echo "-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --"
+systemctl list-units --type=service --all --no-pager 2>/dev/null \
+    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
+echo "-- unit files (enabled state) --"
+systemctl list-unit-files --no-pager 2>/dev/null \
+    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
+echo "-- installed packages of interest --"
+dpkg -l 2>/dev/null | grep -Ei 'pihole|dnsmasq|ftl|technitium|ntp|chrony|timesync|unbound|resolved|resolvconf' || true
+
+# -----------------------------------------------------------------------------
+# 4. NTP service details
+# -----------------------------------------------------------------------------
+section "4. NTP"
+echo "-- chrony --"
+systemctl status chrony --no-pager 2>/dev/null | head -15 || echo "(no chrony unit)"
+read_file /etc/chrony/chrony.conf
+echo "-- chronyc sources/stats (if available) --"
+chronyc -n sources 2>/dev/null || sudo -n chronyc -n sources 2>/dev/null || true
+chronyc -n tracking 2>/dev/null || sudo -n chronyc -n tracking 2>/dev/null || true
+
+echo "-- ntpsec / ntp classic --"
+systemctl status ntp --no-pager 2>/dev/null | head -15 || echo "(no ntp unit)"
+systemctl status ntpsec --no-pager 2>/dev/null | head -15 || true
+read_file /etc/ntp.conf
+read_file /etc/ntpsec/ntp.conf
+
+echo "-- systemd-timesyncd --"
+systemctl status systemd-timesyncd --no-pager 2>/dev/null | head -15 || echo "(no timesyncd)"
+read_file /etc/systemd/timesyncd.conf
+
+echo "-- openntpd --"
+systemctl status openntpd --no-pager 2>/dev/null | head -15 || true
+read_file /etc/openntpd/ntpd.conf
+
+# -----------------------------------------------------------------------------
+# 5. Docker (Pi-hole and/or Technitium may be containerized)
+# -----------------------------------------------------------------------------
+section "5. DOCKER"
+if command -v docker >/dev/null 2>&1; then
+    echo "-- docker version --"; docker version 2>/dev/null || sudo -n docker version 2>/dev/null || true
+    echo "-- containers (running) --"; docker ps 2>/dev/null || sudo -n docker ps 2>/dev/null || true
+    echo "-- containers (all) --"; docker ps -a 2>/dev/null || sudo -n docker ps -a 2>/dev/null || true
+    echo "-- images --"; docker images 2>/dev/null || sudo -n docker images 2>/dev/null || true
+    echo "-- volumes --"; docker volume ls 2>/dev/null || sudo -n docker volume ls 2>/dev/null || true
+    echo "-- networks --"; docker network ls 2>/dev/null || sudo -n docker network ls 2>/dev/null || true
+else
+    echo "(docker not installed / not on PATH)"
+fi
+echo "-- compose files in common locations --"
+for d in /etc/docker-compose /opt/docker-compose /root/docker-compose /home/localuser/docker-compose /srv/docker-compose /opt/pihole /opt/technitium /opt; do
+    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
+        echo "### $d ###"
+        sudo -n ls -la "$d" 2>/dev/null || ls -la "$d" 2>/dev/null || true
+        for f in docker-compose.yml docker-compose.yaml compose.yml compose.yaml; do
+            if sudo -n test -f "$d/$f" 2>/dev/null; then
+                read_file "$d/$f"
+            elif [ -f "$d/$f" ]; then
+                read_file "$d/$f"
+            fi
+        done
+    fi
+done
+
+# -----------------------------------------------------------------------------
+# 6. Pi-hole
+# -----------------------------------------------------------------------------
+section "6. PI-HOLE"
+if command -v pihole >/dev/null 2>&1; then
+    echo "-- pihole version --"; pihole -v 2>/dev/null || sudo -n pihole -v 2>/dev/null || true
+    echo "-- pihole status --"; pihole status 2>/dev/null || sudo -n pihole status 2>/dev/null || true
+else
+    echo "(pihole command not on PATH)"
+fi
+
+echo "-- /etc/pihole listing --"
+list_dir /etc/pihole
+
+# Key Pi-hole config files
+for f in \
+    /etc/pihole/setupVars.conf \
+    /etc/pihole/pihole-FTL.conf \
+    /etc/pihole/adlists.list \
+    /etc/pihole/whitelist.txt \
+    /etc/pihole/blacklist.txt \
+    /etc/pihole/regex.list \
+    /etc/pihole/custom.list \
+    /etc/pihole/local.list \
+    /etc/pihole/hostnameMappings.txt \
+    /etc/pihole/dhcp.leases \
+    /etc/pihole/static_ip.conf \
+    /etc/pihole/GitHubVersions \
+    /etc/pihole/macvendor.db ; do
+    read_file "$f"
+done
+
+echo "-- /etc/pihole/*.conf (all) --"
+for f in /etc/pihole/*.conf; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
+
+echo "-- /etc/dnsmasq.d listing --"
+list_dir /etc/dnsmasq.d
+for f in /etc/dnsmasq.d/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
+
+echo "-- /etc/dnsmasq.conf (if present) --"
+read_file /etc/dnsmasq.conf
+
+echo "-- gravity.db schema + row counts (Pi-hole v5+) --"
+GRAVITY=""
+for g in /etc/pihole/gravity.db /etc/pihole/gravity.db.*; do
+    if [ -e "$g" ]; then GRAVITY="$g"; break; fi
+done
+if [ -n "${GRAVITY:-}" ]; then
+    echo "gravity.db = $GRAVITY"
+    if command -v sqlite3 >/dev/null 2>&1; then
+        sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || true
+        for tbl in adlist adlist_by_group domainlist domainlist_by_group client client_by_group group info; do
+            cnt=$(sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || echo "n/a")
+            printf 'gravity.%s count = %s\n' "$tbl" "$cnt"
+        done
+        echo "-- adlist entries (url, enabled, comment) --"
+        sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null \
+            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null || true
+        echo "-- domainlist sample (first 50) --"
+        sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null \
+            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null || true
+        echo "-- client list --"
+        sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null \
+            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null || true
+        echo "-- group list --"
+        sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null \
+            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null || true
+        echo "-- info table --"
+        sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null \
+            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null || true
+    else
+        echo "(sqlite3 not installed; gravity.db present at $GRAVITY)"
+    fi
+else
+    echo "(no gravity.db found)"
+fi
+
+echo "-- lighttpd / pihole web admin --"
+systemctl status lighttpd --no-pager 2>/dev/null | head -15 || true
+read_file /etc/lighttpd/lighttpd.conf
+for f in /etc/lighttpd/conf-enabled/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
+
+echo "-- pihole-FTL service --"
+systemctl status pihole-FTL --no-pager 2>/dev/null | head -20 || true
+
+# -----------------------------------------------------------------------------
+# 7. Technitium DNS
+# -----------------------------------------------------------------------------
+section "7. TECHNITIUM DNS"
+echo "-- technitium systemd unit --"
+systemctl status technitium-dns --no-pager 2>/dev/null | head -20 || echo "(no technitium-dns unit)"
+echo "-- any unit containing technitium --"
+systemctl list-units --all --no-pager 2>/dev/null | grep -i technitium || true
+echo "-- unit file path --"
+sudo -n systemctl cat technitium-dns 2>/dev/null || systemctl cat technitium-dns 2>/dev/null || true
+
+# Technitium common install locations
+for d in /etc/technitium /etc/technitium/dns /opt/technitium /opt/technitium/dns /var/lib/technitium /usr/local/technitium; do
+    echo "### checking $d ###"
+    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
+        list_dir "$d"
+        # recurse one level for config files
+        for sub in "$d" "$d"/*; do
+            [ -e "$sub" ] || continue
+            if [ -f "$sub" ] && echo "$sub" | grep -Eq '\.(xml|json|conf|config|txt)$'; then
+                read_file "$sub"
+            fi
+        done
+    fi
+done
+
+echo "-- technitium config.xml / dnsServer.conf (search) --"
+sudo -n find /etc/technitium /opt/technitium /var/lib/technitium -maxdepth 4 \
+    \( -name 'config.xml' -o -name '*.config' -o -name 'dnsServer.conf' -o -name 'blockList.txt' \) \
+    -print 2>/dev/null || true
+
+# -----------------------------------------------------------------------------
+# 8. Firewall / SELinux / misc
+# -----------------------------------------------------------------------------
+section "8. FIREWALL & MISC"
+echo "-- nftables --"; sudo -n nft list ruleset 2>/dev/null | head -80 || true
+echo "-- iptables --"; sudo -n iptables -S 2>/dev/null | head -40 || true
+echo "-- ufw --"; sudo -n ufw status verbose 2>/dev/null || true
+echo "-- firewalld --"; systemctl status firewalld --no-pager 2>/dev/null | head -8 || true
+echo "-- selinux/apparmor --"; getenforce 2>/dev/null || echo "(SELinux not present)"; aa-status 2>/dev/null | head -5 || true
+
+# -----------------------------------------------------------------------------
+# 9. Cron / timers that maintain these services
+# -----------------------------------------------------------------------------
+section "9. CRON & TIMERS"
+echo "-- root crontab --"; sudo -n crontab -l 2>/dev/null || echo "(n/a)"
+echo "-- localuser crontab --"; crontab -l 2>/dev/null || echo "(none)"
+echo "-- /etc/cron.d --"; ls -la /etc/cron.d 2>/dev/null
+for f in /etc/cron.d/*pihole* /etc/cron.d/*technitium* /etc/cron.d/*gravity*; do
+    [ -e "$f" ] && read_file "$f"
+done 2>/dev/null
+echo "-- pihole timer --"; systemctl list-timers --all --no-pager 2>/dev/null | grep -Ei 'pihole|gravity|technitium' || true
+
+printf '\n##### AUDIT END %s #####\n' "$AUDIT_DATE"

netinfra/baseline.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# baseline.sh — quick read-only baseline of a target node.
+set -u
+hdr() { printf '\n=== %s ===\n' "$1"; }
+hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
+hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
+hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
+hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
+hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
+hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
+hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
+hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
+hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
+hdr "HOME"; ls -la /home/localuser 2>/dev/null | head

netinfra/deep-audit-netboot.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# deep-audit-netboot.sh — READ-ONLY deep inspection. Uses `sudo` for docker/root files.
+set -u
+DG="sudo docker"
+
+hdr() { printf '\n=== %s ===\n' "$1"; }
+
+hdr "COMPOSE FILES: /root/pihole and /root/NTP"
+for f in /root/pihole/docker-compose.yml /root/NTP/docker-compose.yml; do
+    printf '\n--- %s ---\n' "$f"
+    sudo cat "$f" 2>&1
+done
+
+hdr "DIR LAYOUT of compose project dirs"
+sudo ls -la /root/pihole 2>&1
+sudo ls -la /root/NTP 2>&1
+sudo ls -la /root 2>&1
+
+hdr "SEARCH entire FS for any technitium / dns compose files"
+sudo find / -xdev -maxdepth 6 \( -iname 'docker-compose.y*ml' -o -iname 'compose.y*ml' \) -print 2>/dev/null \
+    | grep -Ei 'dns|technitium|tsys' || true
+
+hdr "ALL CONTAINERS with compose labels"
+$DG ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\tWDIR={{.Label "com.docker.compose.project.working_dir"}}\tCFG={{.Label "com.docker.compose.project.config_files"}}\tSVC={{.Label "com.docker.compose.service"}}' 2>&1
+
+hdr "DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container)"
+for c in $($DG ps -aq 2>/dev/null); do
+    nm=$($DG inspect --format '{{.Name}} | image={{.Config.Image}} | proj={{index .Config.Labels "com.docker.compose.project"}}' "$c" 2>/dev/null)
+    printf '\n###### %s ######\n' "$nm"
+    $DG inspect "$c" 2>&1
+done
+
+hdr "PI-HOLE internal config (sudo docker exec, READ-ONLY)"
+echo "-- ls /etc/pihole --";          $DG exec pihole ls -la /etc/pihole 2>&1 || true
+echo "-- ls /etc/dnsmasq.d --";       $DG exec pihole ls -la /etc/dnsmasq.d 2>&1 || true
+for f in setupVars.conf pihole-FTL.conf adlists.list custom.list local.list regex.list dhcp.leases static_ip.conf; do
+    echo "--- /etc/pihole/$f ---"; $DG exec pihole cat "/etc/pihole/$f" 2>&1 || true
+done
+echo "-- /etc/dnsmasq.d/* --"
+$DG exec pihole sh -c 'for f in /etc/dnsmasq.d/*; do echo "--- $f ---"; cat "$f"; done' 2>&1 || true
+echo "-- pihole version --";          $DG exec pihole pihole -v 2>&1 || true
+echo "-- gravity row counts --"
+$DG exec pihole sh -c 'for t in adlist domainlist client "group" info; do printf "%s=" "$t"; sqlite3 /etc/pihole/gravity.db "SELECT COUNT(*) FROM $t;" 2>/dev/null; done' 2>&1 || true
+echo "-- adlist addresses --"
+$DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT address,enabled,comment FROM adlist;" 2>&1 || true
+echo "-- domainlist (allow+deny, first 60) --"
+$DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT type,domain,enabled,comment FROM domainlist LIMIT 60;" 2>&1 || true
+
+hdr "CHRONY/NTP container config (tsys-ntp)"
+$DG exec tsys-ntp sh -c 'echo "--- chrony.conf ---"; cat /etc/chrony/chrony.conf 2>/dev/null || cat /etc/chrony.conf 2>/dev/null || echo none; echo "--- ls /etc ---"; ls -la /etc 2>/dev/null | head -30' 2>&1 || true
+$DG exec tsys-ntp chronyc -n sources 2>&1 || true
+$DG exec tsys-ntp chronyc -n tracking 2>&1 || true
+echo "-- bare metal ntpsec.conf --"; sudo cat /etc/ntpsec/ntp.conf 2>&1
+
+hdr "TECHNITIUM volumes"
+for v in dns_tsys-dns-config dns_tyss-dns-config; do
+    mnt=$($DG volume inspect --format '{{.Mountpoint}}' "$v" 2>/dev/null)
+    printf '\n--- volume %s -> %s ---\n' "$v" "$mnt"
+    [ -n "$mnt" ] || continue
+    sudo find "$mnt" -maxdepth 4 -type f 2>/dev/null | head -80
+    echo "-- config dir listing --"
+    sudo ls -laR "$mnt"/config 2>/dev/null | head -60 || sudo ls -laR "$mnt" 2>/dev/null | head -60 || true
+    echo "-- config.xml --"
+    sudo cat "$mnt"/config/config.xml 2>/dev/null | head -250 || true
+done
+
+hdr "DONE"

netinfra/gather-configs.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# gather-configs.sh — READ-ONLY. Pull Technitium configs, pihole.toml, gravity.db
+# contents, and chrony NTP details from pfv-netboot for replication.
+set -u
+hdr() { printf '\n=== %s ===\n' "$1"; }
+
+hdr "TECHNITIUM dns.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config 2>&1
+
+hdr "TECHNITIUM auth.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config 2>&1
+
+hdr "TECHNITIUM log.config"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config 2>&1
+
+hdr "TECHNITIUM scopes/Default.scope"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope 2>&1
+
+hdr "TECHNITIUM zone: knel.net.zone"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone 2>&1
+
+hdr "TECHNITIUM all zone files listing + sizes"
+sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/ 2>&1
+
+hdr "TECHNITIUM reverse zones (one sample)"
+sudo cat /var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone 2>&1
+
+hdr "TECHNITIUM top-level listing"
+sudo ls -la /var/lib/docker/volumes/dns_tsys-dns-config/_data/ 2>&1
+
+hdr "PI-HOLE pihole.toml (v6 config) from host bind mount"
+cat /root/pihole/etc-pihole/pihole.toml 2>&1
+
+hdr "PI-HOLE dnsmasq.conf"
+cat /root/pihole/etc-pihole/dnsmasq.conf 2>&1
+
+hdr "PI-HOLE versions file"
+cat /root/pihole/etc-pihole/versions 2>&1
+
+hdr "GRAVITY.DB contents (host sqlite3 if present)"
+if command -v sqlite3 >/dev/null 2>&1; then
+    G=/root/pihole/etc-pihole/gravity.db
+    echo "-- adlist --"; sqlite3 -readonly "$G" "SELECT address,enabled,comment,DATE(date_added,'unixepoch'),DATE(date_updated,'unixepoch') FROM adlist;" 2>&1
+    echo "-- adlist count --"; sqlite3 -readonly "$G" "SELECT COUNT(*) FROM adlist;" 2>&1
+    echo "-- domainlist count by type --"; sqlite3 -readonly "$G" "SELECT type,COUNT(*) FROM domainlist GROUP BY type;" 2>&1
+    echo "-- domainlist (allow=0/allow_exact, deny=1/deny_exact, etc.) first 80 --"; sqlite3 -readonly "$G" "SELECT type,domain,enabled,comment FROM domainlist LIMIT 80;" 2>&1
+    echo -- client --"; sqlite3 -readonly "$G" "SELECT ip,comment FROM client;" 2>&1
+    echo "-- group --"; sqlite3 -readonly "$G" "SELECT id,name,enabled,comment FROM 'group';" 2>&1
+    echo "-- info --"; sqlite3 -readonly "$G" "SELECT * FROM info;" 2>&1
+else
+    echo "(sqlite3 not on host PATH)"
+fi
+
+hdr "DONE"

netinfra/netboot-audit.txt

@@ -0,0 +1,720 @@
+This system is the property of Known Element Enterprises LLC.
+
+Authorized uses only. All activity may be monitored and reported.
+
+All activities subject to monitoring/recording/review in real time and/or at a later time.##### AUDIT START 2026-07-28T02:41:34Z #####
+Audit host: pfv-netboot.knel.net
+Audit user: localuser
+Script:     audit-netboot.sh (READ-ONLY)
+
+========================================================================
+SECTION: 1. SYSTEM BASICS
+========================================================================
+-- uname --
+Linux pfv-netboot 6.1.0-44-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.164-1 (2026-03-09) x86_64 GNU/Linux
+-- /etc/os-release --
+PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
+NAME="Debian GNU/Linux"
+VERSION_ID="12"
+VERSION="12 (bookworm)"
+VERSION_CODENAME=bookworm
+ID=debian
+HOME_URL="https://www.debian.org/"
+SUPPORT_URL="https://www.debian.org/support"
+BUG_REPORT_URL="https://bugs.debian.org/"
+-- uptime --
+ 21:41:34 up 113 days, 15:32,  2 users,  load average: 0.07, 0.07, 0.06
+-- timezone --
+               Local time: Mon 2026-07-27 21:41:34 CDT
+           Universal time: Tue 2026-07-28 02:41:34 UTC
+                 RTC time: Tue 2026-07-28 02:41:34
+                Time zone: US/Central (CDT, -0500)
+System clock synchronized: yes
+              NTP service: n/a
+          RTC in local TZ: no
+-- arch --
+amd64
+-- memory --
+               total        used        free      shared  buff/cache   available
+Mem:           1.9Gi       940Mi        66Mi        65Mi       1.1Gi       961Mi
+Swap:          974Mi       315Mi       659Mi
+-- disk --
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/sda1       491G   18G  448G   4% /
+-- cpu count --
+2
+
+========================================================================
+SECTION: 2. NETWORK
+========================================================================
+-- hostname --
+pfv-netboot
+pfv-netboot.knel.net
+192.168.3.250 172.24.0.1 172.18.0.1 172.19.0.1 172.23.0.1 172.17.0.1 172.20.0.1 172.22.0.1 100.103.64.82 2600:1700:1d1:eb00:215:5dff:fe64:e81e fd7a:115c:a1e0::2901:4052 
+-- ip addr --
+lo               UNKNOWN        127.0.0.1/8 ::1/128 
+eth0             UP             192.168.3.250/22 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 fe80::215:5dff:fe64:e81e/64 
+eth1             UP             
+br-3d55385c5e7d  UP             172.24.0.1/16 fe80::d883:2ff:fef8:79cd/64 
+br-50e2988e78eb  DOWN           172.18.0.1/16 
+br-748fea589ab8  DOWN           172.19.0.1/16 
+br-b52d11330b20  DOWN           172.23.0.1/16 
+docker0          DOWN           172.17.0.1/16 
+br-f91c61cdba28  DOWN           172.20.0.1/16 
+br-25187260f675  DOWN           172.22.0.1/16 
+tailscale0       UNKNOWN        100.103.64.82/32 fd7a:115c:a1e0::2901:4052/128 fe80::42db:e6ff:fe57:431c/64 
+veth9ecb986@if2  UP             fe80::c45b:62ff:fe19:6258/64 
+
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host noprefixroute 
+       valid_lft forever preferred_lft forever
+2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
+    link/ether 00:15:5d:64:e8:1e brd ff:ff:ff:ff:ff:ff
+    inet 192.168.3.250/22 brd 192.168.3.255 scope global eth0
+       valid_lft forever preferred_lft forever
+    inet6 2600:1700:1d1:eb00:215:5dff:fe64:e81e/64 scope global dynamic mngtmpaddr 
+       valid_lft 3391sec preferred_lft 3391sec
+    inet6 fe80::215:5dff:fe64:e81e/64 scope link 
+       valid_lft forever preferred_lft forever
+3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
+    link/ether 00:15:5d:64:e8:2a brd ff:ff:ff:ff:ff:ff
+5: br-3d55385c5e7d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
+    link/ether da:83:02:f8:79:cd brd ff:ff:ff:ff:ff:ff
+    inet 172.24.0.1/16 brd 172.24.255.255 scope global br-3d55385c5e7d
+       valid_lft forever preferred_lft forever
+    inet6 fe80::d883:2ff:fef8:79cd/64 scope link 
+       valid_lft forever preferred_lft forever
+6: br-50e2988e78eb: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether ca:f9:33:b2:b6:bb brd ff:ff:ff:ff:ff:ff
+    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-50e2988e78eb
+       valid_lft forever preferred_lft forever
+7: br-748fea589ab8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether 82:e8:2b:47:1e:91 brd ff:ff:ff:ff:ff:ff
+    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-748fea589ab8
+       valid_lft forever preferred_lft forever
+8: br-b52d11330b20: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether de:6e:f1:ec:24:64 brd ff:ff:ff:ff:ff:ff
+    inet 172.23.0.1/16 brd 172.23.255.255 scope global br-b52d11330b20
+       valid_lft forever preferred_lft forever
+9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether 56:f9:55:99:fa:af brd ff:ff:ff:ff:ff:ff
+    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
+       valid_lft forever preferred_lft forever
+10: br-f91c61cdba28: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether 8e:ed:24:fa:4b:4a brd ff:ff:ff:ff:ff:ff
+    inet 172.20.0.1/16 brd 172.20.255.255 scope global br-f91c61cdba28
+       valid_lft forever preferred_lft forever
+11: br-25187260f675: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+    link/ether 82:af:8a:0a:64:01 brd ff:ff:ff:ff:ff:ff
+    inet 172.22.0.1/16 brd 172.22.255.255 scope global br-25187260f675
+       valid_lft forever preferred_lft forever
+18: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
+    link/none 
+    inet 100.103.64.82/32 scope global tailscale0
+       valid_lft forever preferred_lft forever
+    inet6 fd7a:115c:a1e0::2901:4052/128 scope global 
+       valid_lft forever preferred_lft forever
+    inet6 fe80::42db:e6ff:fe57:431c/64 scope link stable-privacy 
+       valid_lft forever preferred_lft forever
+24: veth9ecb986@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3d55385c5e7d state UP group default 
+    link/ether c6:5b:62:19:62:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
+    inet6 fe80::c45b:62ff:fe19:6258/64 scope link 
+       valid_lft forever preferred_lft forever
+-- default route --
+default via 192.168.3.254 dev eth0 onlink 
+172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
+172.18.0.0/16 dev br-50e2988e78eb proto kernel scope link src 172.18.0.1 linkdown 
+172.19.0.0/16 dev br-748fea589ab8 proto kernel scope link src 172.19.0.1 linkdown 
+172.20.0.0/16 dev br-f91c61cdba28 proto kernel scope link src 172.20.0.1 linkdown 
+172.22.0.0/16 dev br-25187260f675 proto kernel scope link src 172.22.0.1 linkdown 
+172.23.0.0/16 dev br-b52d11330b20 proto kernel scope link src 172.23.0.1 linkdown 
+172.24.0.0/16 dev br-3d55385c5e7d proto kernel scope link src 172.24.0.1 
+192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.250 
+-- /etc/resolv.conf --
+# resolv.conf(5) file generated by tailscale
+# For more info, see https://tailscale.com/s/resolvconf-overwrite
+# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
+
+nameserver 100.100.100.100
+nameserver fd7a:115c:a1e0::53
+search knel.net
+-- /etc/hosts --
+127.0.0.1	localhost
+127.0.1.1	pfv-netboot.knel.net	pfv-netboot
+
+# The following lines are desirable for IPv6 capable hosts
+::1     localhost ip6-localhost ip6-loopback
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+-- listening TCP/UDP sockets --
+Netid State  Recv-Q Send-Q                              Local Address:Port  Peer Address:PortProcess
+udp   UNCONN 0      0                                         0.0.0.0:38908      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:53         0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:67         0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:46164      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:111        0.0.0.0:*          
+udp   UNCONN 0      0                                   100.103.64.82:123        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.24.0.1:123        0.0.0.0:*          
+udp   UNCONN 0      0                                   192.168.3.250:123        0.0.0.0:*          
+udp   UNCONN 0      0                                       127.0.0.1:123        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:123        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.17.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.17.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.18.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.18.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.19.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.19.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.20.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.20.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.22.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.22.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.23.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.23.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.24.255.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.24.0.1:137        0.0.0.0:*          
+udp   UNCONN 0      0                                   192.168.3.255:137        0.0.0.0:*          
+udp   UNCONN 0      0                                   192.168.3.250:137        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:137        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.17.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.17.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.18.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.18.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.19.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.19.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.20.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.20.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.22.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.22.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.23.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.23.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                  172.24.255.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                      172.24.0.1:138        0.0.0.0:*          
+udp   UNCONN 0      0                                   192.168.3.255:138        0.0.0.0:*          
+udp   UNCONN 0      0                                   192.168.3.250:138        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:138        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:161        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:41257      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:20000      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:41641      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:59092      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:10000      0.0.0.0:*          
+udp   UNCONN 0      0                                       127.0.0.1:786        0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:40725      0.0.0.0:*          
+udp   UNCONN 0      0                                         0.0.0.0:37749      0.0.0.0:*          
+udp   UNCONN 0      0                                            [::]:53            [::]:*          
+udp   UNCONN 0      0                                            [::]:34901         [::]:*          
+udp   UNCONN 0      0                                            [::]:111           [::]:*          
+udp   UNCONN 0      0         [fe80::c45b:62ff:fe19:6258]%veth9ecb986:123           [::]:*          
+udp   UNCONN 0      0          [fe80::42db:e6ff:fe57:431c]%tailscale0:123           [::]:*          
+udp   UNCONN 0      0                     [fd7a:115c:a1e0::2901:4052]:123           [::]:*          
+udp   UNCONN 0      0      [fe80::d883:2ff:fef8:79cd]%br-3d55385c5e7d:123           [::]:*          
+udp   UNCONN 0      0                 [fe80::215:5dff:fe64:e81e]%eth0:123           [::]:*          
+udp   UNCONN 0      0         [2600:1700:1d1:eb00:215:5dff:fe64:e81e]:123           [::]:*          
+udp   UNCONN 0      0                                           [::1]:123           [::]:*          
+udp   UNCONN 0      0                                            [::]:123           [::]:*          
+udp   UNCONN 0      0                                            [::]:43257         [::]:*          
+udp   UNCONN 0      0                                            [::]:44469         [::]:*          
+udp   UNCONN 0      0                                            [::]:41641         [::]:*          
+udp   UNCONN 0      0                                            [::]:55132         [::]:*          
+udp   UNCONN 0      0                                            [::]:34667         [::]:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:38435      0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:20000      0.0.0.0:*          
+tcp   LISTEN 0      5                                       127.0.0.1:555        0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:10002      0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:10003      0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:10000      0.0.0.0:*          
+tcp   LISTEN 0      4096                                100.103.64.82:64362      0.0.0.0:*          
+tcp   LISTEN 0      5                                       127.0.0.1:61209      0.0.0.0:*          
+tcp   LISTEN 0      50                                        0.0.0.0:139        0.0.0.0:*          
+tcp   LISTEN 0      64                                        0.0.0.0:36079      0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:45295      0.0.0.0:*          
+tcp   LISTEN 0      5                                         0.0.0.0:4330       0.0.0.0:*          
+tcp   LISTEN 0      100                                       0.0.0.0:25         0.0.0.0:*          
+tcp   LISTEN 0      128                                       0.0.0.0:22         0.0.0.0:*          
+tcp   LISTEN 0      64                                        0.0.0.0:2049       0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:53         0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:40029      0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:111        0.0.0.0:*          
+tcp   LISTEN 0      50                                        0.0.0.0:445        0.0.0.0:*          
+tcp   LISTEN 0      4096                                      0.0.0.0:58853      0.0.0.0:*          
+tcp   LISTEN 0      128                                       0.0.0.0:44322      0.0.0.0:*          
+tcp   LISTEN 0      128                                       0.0.0.0:44323      0.0.0.0:*          
+tcp   LISTEN 0      5                                         0.0.0.0:44321      0.0.0.0:*          
+tcp   LISTEN 0      4096                  [fd7a:115c:a1e0::2901:4052]:52472         [::]:*          
+tcp   LISTEN 0      4096                                         [::]:45755         [::]:*          
+tcp   LISTEN 0      4096                                         [::]:60951         [::]:*          
+tcp   LISTEN 0      4096                                            *:9090             *:*          
+tcp   LISTEN 0      64                                           [::]:37813         [::]:*          
+tcp   LISTEN 0      4096                                         [::]:10002         [::]:*          
+tcp   LISTEN 0      4096                                         [::]:10003         [::]:*          
+tcp   LISTEN 0      4096                                         [::]:10000         [::]:*          
+tcp   LISTEN 0      4096                                            *:45876            *:*          
+tcp   LISTEN 0      50                                           [::]:139           [::]:*          
+tcp   LISTEN 0      4096                                         [::]:44271         [::]:*          
+tcp   LISTEN 0      5                                            [::]:4330          [::]:*          
+tcp   LISTEN 0      128                                          [::]:22            [::]:*          
+tcp   LISTEN 0      64                                           [::]:2049          [::]:*          
+tcp   LISTEN 0      4096                                         [::]:53            [::]:*          
+tcp   LISTEN 0      4096                                         [::]:111           [::]:*          
+tcp   LISTEN 0      4096                                            *:6556             *:*          
+tcp   LISTEN 0      50                                           [::]:445           [::]:*          
+tcp   LISTEN 0      4096                                         [::]:60725         [::]:*          
+tcp   LISTEN 0      128                                          [::]:44322         [::]:*          
+tcp   LISTEN 0      128                                          [::]:44323         [::]:*          
+tcp   LISTEN 0      5                                            [::]:44321         [::]:*          
+-- /etc/network/interfaces --
+# This file describes the network interfaces available on your system
+# and how to activate them. For more information, see interfaces(5).
+
+source /etc/network/interfaces.d/*
+
+# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+allow-hotplug eth0
+iface eth0 inet static
+	address 192.168.3.250
+	netmask 255.255.252.0
+	gateway 192.168.3.254
+
+#allow-hotplug eth1
+#iface eth1 inet static
+#	address 10.100.100.4
+#	netmask 255.255.255.0
+-- netplan --
+-- systemd-networkd --
+total 8
+drwxr-xr-x 2 root root 4096 Mar  6  2025 .
+drwxr-xr-x 5 root root 4096 Jul 20 17:55 ..
+●        State: n/a
+  Online state: unknown
+       Address: 192.168.3.250 on eth0
+                172.24.0.1 on br-3d55385c5e7d
+                172.18.0.1 on br-50e2988e78eb
+                172.19.0.1 on br-748fea589ab8
+                172.23.0.1 on br-b52d11330b20
+                172.17.0.1 on docker0
+                172.20.0.1 on br-f91c61cdba28
+                172.22.0.1 on br-25187260f675
+                100.103.64.82 on tailscale0
+                2600:1700:1d1:eb00:215:5dff:fe64:e81e on eth0
+                fd7a:115c:a1e0::2901:4052 on tailscale0
+                fe80::215:5dff:fe64:e81e on eth0
+                fe80::d883:2ff:fef8:79cd on br-3d55385c5e7d
+                fe80::42db:e6ff:fe57:431c on tailscale0
+                fe80::c45b:62ff:fe19:6258 on veth9ecb986
+       Gateway: 192.168.3.254 on eth0
+                fe80::6e4b:b4ff:fe10:4061 on eth0
+
+========================================================================
+SECTION: 3. RELEVANT SYSTEMD UNITS
+========================================================================
+-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --
+  initrd-parse-etc.service             loaded    inactive dead    Mountpoints Configured in the Real Root
+  ntpsec-rotate-stats.service          loaded    inactive dead    Rotate ntpd stats
+  ntpsec-systemd-netif.service         loaded    inactive dead    ntpsec-systemd-netif.service
+  ntpsec.service                       loaded    active   running Network Time Service
+● systemd-resolved.service             not-found inactive dead    systemd-resolved.service
+● systemd-timesyncd.service            not-found inactive dead    systemd-timesyncd.service
+-- unit files (enabled state) --
+ntpsec-systemd-netif.path                                                     enabled         enabled
+postfix-resolvconf.path                                                       disabled        enabled
+ntp.service                                                                   alias           -
+ntpd.service                                                                  alias           -
+ntpsec-rotate-stats.service                                                   static          -
+ntpsec-systemd-netif.service                                                  static          -
+ntpsec-wait.service                                                           disabled        enabled
+ntpsec.service                                                                enabled         enabled
+postfix-resolvconf.service                                                    disabled        enabled
+ntpsec-rotate-stats.timer                                                     enabled         enabled
+ntpsec.timer                                                                  masked          enabled
+-- installed packages of interest --
+ii  dnsmasq-base                      2.90-4~deb12u2                       amd64        Small caching DNS proxy and DHCP/TFTP server - executable
+ii  ntpsec                            1.2.2+dfsg1-1+deb12u1                amd64        Network Time Protocol daemon and utility programs
+ii  ntpsec-ntpdate                    1.2.2+dfsg1-1+deb12u1                amd64        client for setting system time from NTP servers
+ii  ntpsec-ntpdig                     1.2.2+dfsg1-1+deb12u1                amd64        ntpdig SNTP client
+ii  python3-ntp                       1.2.2+dfsg1-1+deb12u1                amd64        Python 3 NTP Helper Classes
+
+========================================================================
+SECTION: 4. NTP
+========================================================================
+-- chrony --
+
+-- chronyc sources/stats (if available) --
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+-- ntpsec / ntp classic --
+● ntpsec.service - Network Time Service
+     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
+     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
+       Docs: man:ntpd(8)
+    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
+   Main PID: 2124690 (ntpd)
+      Tasks: 1 (limit: 2219)
+     Memory: 10.7M
+        CPU: 34.324s
+     CGroup: /system.slice/ntpsec.service
+             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
+● ntpsec.service - Network Time Service
+     Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
+     Active: active (running) since Mon 2026-07-20 17:56:32 CDT; 1 week 0 days ago
+       Docs: man:ntpd(8)
+    Process: 2124687 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
+   Main PID: 2124690 (ntpd)
+      Tasks: 1 (limit: 2219)
+     Memory: 10.7M
+        CPU: 34.324s
+     CGroup: /system.slice/ntpsec.service
+             └─2124690 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+main: line 37: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+driftfile /var/lib/ntp/ntp.drift
+leapfile /usr/share/zoneinfo/leap-seconds.list
+server pfvsvrpi.knel.net
+restrict 127.0.0.1
+restrict ::1
+
+-- systemd-timesyncd --
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+-- openntpd --
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+========================================================================
+SECTION: 5. DOCKER
+========================================================================
+-- docker version --
+Client: Docker Engine - Community
+ Version:           29.6.2
+ API version:       1.55
+ Go version:        go1.26.5
+ Git commit:        dfc4efb
+ Built:             Thu Jul 16 16:12:20 2026
+ OS/Arch:           linux/amd64
+ Context:           default
+Client: Docker Engine - Community
+ Version:           29.6.2
+ API version:       1.55
+ Go version:        go1.26.5
+ Git commit:        dfc4efb
+ Built:             Thu Jul 16 16:12:20 2026
+ OS/Arch:           linux/amd64
+ Context:           default
+
+Server: Docker Engine - Community
+ Engine:
+  Version:          29.6.2
+  API version:      1.55 (minimum version 1.40)
+  Go version:       go1.26.5
+  Git commit:       3d80467
+  Built:            Thu Jul 16 16:12:20 2026
+  OS/Arch:          linux/amd64
+  Experimental:     false
+ containerd:
+  Version:          v2.2.6
+  GitCommit:        11ce9d5f3c68c941867e82890e93e815c1304f1b
+ runc:
+  Version:          1.3.6
+  GitCommit:        v1.3.6-0-g491b69ba
+ docker-init:
+  Version:          0.19.0
+  GitCommit:        de40ad0
+-- containers (running) --
+CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
+1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
+c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
+-- containers (all) --
+CONTAINER ID   IMAGE                  COMMAND          CREATED         STATUS                    PORTS                                                                                                                                                                               NAMES
+1d09249f01b0   pihole/pihole:latest   "start.sh"       5 months ago    Up 44 minutes (healthy)   67/udp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, [::]:53->53/tcp, [::]:53->53/udp, 123/udp, 0.0.0.0:10002->80/tcp, [::]:10002->80/tcp, 0.0.0.0:10003->443/tcp, [::]:10003->443/tcp   pihole
+c33ced53142e   dockurr/chrony         "/bin/startup"   13 months ago   Up 7 days (healthy)                                                                                                                                                                                           tsys-ntp
+-- images --
+IMAGE                   ID             DISK USAGE   CONTENT SIZE   EXTRA
+busybox:latest          6d3e4188a38a       4.28MB             0B        
+dockurr/chrony:latest   3b656f2feb55       15.4MB             0B   U    
+pihole/pihole:latest    81365952d1f8       92.7MB             0B   U    
+-- volumes --
+DRIVER    VOLUME NAME
+local     0cb24520936997d233409ea07681920fdb03d92967856e849474cf90446dc426
+local     0ccc1140e4b86442167f65d7caf43d8a3bc04fff2bef461f112d8b7c8e4bd4c8
+local     1d7f70d65f5d739f737c3bc0d1b0b2b5cf4f192477dcbbdb95bca344310bff84
+local     2a1726279d1d382ab0a2bcb2079f60e65fc0fd0d40880074038e0c678b2a7f56
+local     3d207dff87da7dc7d11b70f1e595cc5cce001392cb663ac4d664f5710b9879b8
+local     4f8894b9f0206c743be26e6b644e3bf017c5b6e190ac7da31e7ffb7132a1ca2f
+local     7c210964fd48cd91c8e37543d562569ec523208b5ea421f8fb47dc73a6d724ee
+local     8ccc4c07bf45d05bc922d4f619780f60cac0149b4dd7de5be7720f72f47e6c10
+local     71c98aa3d5fb266ef291ac4fc95b11fc83475b9bb0f70999e77095249ad18bf5
+local     83f977471ac2f4ad6521ad50f0311ee36547701d00d9a869903f0ba667c2000d
+local     87eced0295c12870220fc746ef685530cb5b8e20c4afafbfd573ff773459cd9f
+local     95eb8a7b389b9a04bd90f4cee485bd015b7c3fd311c1750c7596a2b2fde1605d
+local     562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41
+local     8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b
+local     201278554f24a7af6284b9564c45656e6d2dc7b9daea24fe52707d7825cb5385
+local     b24c0bd3e25987537a92078d6f521f5f155e7379295d21e43c500cbabc8a02e0
+local     bb353f885d620d98dbb147f0ff51f38a87725669b1a14390fb825509878a7e0b
+local     bc96d56b965f1e238afd47e483b93f1ec9ed59e8458e8e851ffc9710582a5475
+local     c329a5b999999c074bf767bc1f7b3647cf5d4d0624b7d45d63658cbb723e5d46
+local     cf03c8ea64823ac8db5c6df1093b2d9f5146838dd8dc2986bdc485e8b59d0d24
+local     d9500add842368a6d637090569e5c13cb9f068cbfeb14e02c597ae82f2807e47
+local     da3164486ea59aaa9fd14113faa9104411c61ad36a9487e3ce780e86a96dac1f
+local     dae4bbdcbb634b817b47558df3908906fe7635ad57377389616189bd0f2d7b64
+local     dddeee12833ed02185e00732ddb8c12ef6a6b88c49e7df4e48f7a27b71182bf5
+local     dns_tsys-dns-config
+local     dns_tyss-dns-config
+local     e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629
+local     eda6668262ccd43f0e3c0b9e3c42711cc516f372b46ef174bca34d96d17740b8
+local     f53cb9833442fd9499260fa8073a04a15f5c96a5576aa96d781bc24441906942
+local     f142ec0c6150618365447908d5753ff5ff7defcbf27c210702a17ad205dc4aba
+local     f350de2e220f4983bca2e7ca96374a597a075779224b036ccd76deb280ba8922
+-- networks --
+NETWORK ID     NAME               DRIVER    SCOPE
+0f56376f4515   bridge             bridge    local
+f91c61cdba28   dns_default        bridge    local
+b24303598fa1   host               host      local
+25187260f675   msmtpd_default     bridge    local
+748fea589ab8   netdisco_default   bridge    local
+bd40993bc47f   none               null      local
+b52d11330b20   ntp_default        bridge    local
+3d55385c5e7d   pihole_default     bridge    local
+50e2988e78eb   root_default       bridge    local
+-- compose files in common locations --
+### /opt ###
+total 16
+drwxr-xr-x  4 root   root   4096 Dec 29  2025 .
+drwxr-xr-x 20 root   root   4096 Jul 20 17:55 ..
+drwxr-xr-x  2 beszel beszel 4096 Apr  6 01:08 beszel-agent
+drwx--x--x  4 root   root   4096 Jun 10  2025 containerd
+
+========================================================================
+SECTION: 6. PI-HOLE
+========================================================================
+(pihole command not on PATH)
+-- /etc/pihole listing --
+main: line 54: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+(cannot list /etc/pihole)
+
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+
+-- /etc/pihole/*.conf (all) --
+-- /etc/dnsmasq.d listing --
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+main: line 54: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+(cannot list /etc/dnsmasq.d)
+
+-- /etc/dnsmasq.conf (if present) --
+
+-- gravity.db schema + row counts (Pi-hole v5+) --
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+(no gravity.db found)
+-- lighttpd / pihole web admin --
+
+main: line 44: printf: --: invalid option
+printf: usage: printf [-v var] format [arguments]
+alias.url += ("/javascript" => "/usr/share/javascript")
+
+-- pihole-FTL service --
+
+========================================================================
+SECTION: 7. TECHNITIUM DNS
+========================================================================
+-- technitium systemd unit --
+-- any unit containing technitium --
+-- unit file path --
+### checking /etc/technitium ###
+### checking /etc/technitium/dns ###
+### checking /opt/technitium ###
+### checking /opt/technitium/dns ###
+### checking /var/lib/technitium ###
+### checking /usr/local/technitium ###
+-- technitium config.xml / dnsServer.conf (search) --
+
+========================================================================
+SECTION: 8. FIREWALL & MISC
+========================================================================
+-- nftables --
+table ip raw {
+	chain PREROUTING {
+		type filter hook prerouting priority raw; policy accept;
+		iifname != "br-b52d11330b20" ip daddr 172.23.0.2 counter packets 0 bytes 0 drop
+		iifname != "br-3d55385c5e7d" ip daddr 172.24.0.2 counter packets 0 bytes 0 drop
+	}
+
+	chain OUTPUT {
+		type filter hook output priority raw; policy accept;
+	}
+}
+table ip filter {
+	chain INPUT {
+		type filter hook input priority filter; policy accept;
+		counter packets 8461563 bytes 1841523123 jump ts-input
+	}
+
+	chain FORWARD {
+		type filter hook forward priority filter; policy drop;
+		counter packets 20387709 bytes 3821595842 jump DOCKER-USER
+		counter packets 20387709 bytes 3821595842 jump DOCKER-FORWARD
+		counter packets 1064 bytes 88643 jump ts-forward
+	}
+
+	chain OUTPUT {
+		type filter hook output priority filter; policy accept;
+	}
+
+	chain DOCKER-FORWARD {
+		counter packets 20387709 bytes 3821595842 jump DOCKER-CT
+		counter packets 14776132 bytes 2075147539 jump DOCKER-INTERNAL
+		counter packets 14776132 bytes 2075147539 jump DOCKER-BRIDGE
+		iifname "br-748fea589ab8" counter packets 0 bytes 0 accept
+		iifname "br-b52d11330b20" counter packets 0 bytes 0 accept
+		iifname "br-f91c61cdba28" counter packets 0 bytes 0 accept
+		iifname "br-25187260f675" counter packets 0 bytes 0 accept
+		iifname "br-3d55385c5e7d" counter packets 10132822 bytes 1741327665 accept
+		iifname "br-50e2988e78eb" counter packets 0 bytes 0 accept
+		iifname "docker0" counter packets 0 bytes 0 accept
+	}
+
+	chain DOCKER-USER {
+	}
+
+	chain ts-input {
+		iifname "lo" ip saddr 100.103.64.82 counter packets 0 bytes 0 accept
+		iifname "tailscale0" counter packets 512217 bytes 46924098 accept
+		udp dport 41641 counter packets 3360373 bytes 516737750 accept
+		iifname != "tailscale0" ip saddr 100.115.92.0/23 counter packets 0 bytes 0 return
+		iifname != "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
+	}
+
+	chain ts-forward {
+		iifname "tailscale0" counter packets 28 bytes 2578 meta mark set mark and 0xff00ffff xor 0x40000
+		meta mark & 0x00ff0000 == 0x00040000 counter packets 28 bytes 2578 accept
+		oifname "tailscale0" ip saddr 100.64.0.0/10 counter packets 0 bytes 0 drop
+		oifname "tailscale0" counter packets 15 bytes 1554 accept
+	}
+
+	chain DOCKER {
+		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 443 counter packets 0 bytes 0 accept
+		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 80 counter packets 0 bytes 0 accept
+		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 udp dport 53 counter packets 32283 bytes 2368019 accept
+		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" ip daddr 172.24.0.2 tcp dport 53 counter packets 638 bytes 34248 accept
+		iifname != "br-748fea589ab8" oifname "br-748fea589ab8" counter packets 0 bytes 0 drop
+		iifname != "br-b52d11330b20" oifname "br-b52d11330b20" counter packets 0 bytes 0 drop
+		iifname != "br-f91c61cdba28" oifname "br-f91c61cdba28" counter packets 0 bytes 0 drop
+		iifname != "br-25187260f675" oifname "br-25187260f675" counter packets 0 bytes 0 drop
+		iifname != "br-3d55385c5e7d" oifname "br-3d55385c5e7d" counter packets 0 bytes 0 drop
+		iifname != "br-50e2988e78eb" oifname "br-50e2988e78eb" counter packets 0 bytes 0 drop
+		iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
+	}
+
+	chain DOCKER-BRIDGE {
+		oifname "br-748fea589ab8" counter packets 0 bytes 0 jump DOCKER
+		oifname "br-b52d11330b20" counter packets 0 bytes 0 jump DOCKER
+		oifname "br-f91c61cdba28" counter packets 0 bytes 0 jump DOCKER
+		oifname "br-25187260f675" counter packets 0 bytes 0 jump DOCKER
+		oifname "br-3d55385c5e7d" counter packets 4643310 bytes 333819874 jump DOCKER
+		oifname "br-50e2988e78eb" counter packets 0 bytes 0 jump DOCKER
+-- iptables --
+-P INPUT ACCEPT
+-P FORWARD DROP
+-P OUTPUT ACCEPT
+-N DOCKER
+-N DOCKER-BRIDGE
+-N DOCKER-CT
+-N DOCKER-FORWARD
+-N DOCKER-INTERNAL
+-N DOCKER-USER
+-N ts-forward
+-N ts-input
+-A INPUT -j ts-input
+-A FORWARD -j DOCKER-USER
+-A FORWARD -j DOCKER-FORWARD
+-A FORWARD -j ts-forward
+-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 443 -j ACCEPT
+-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 80 -j ACCEPT
+-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p udp -m udp --dport 53 -j ACCEPT
+-A DOCKER -d 172.24.0.2/32 ! -i br-3d55385c5e7d -o br-3d55385c5e7d -p tcp -m tcp --dport 53 -j ACCEPT
+-A DOCKER ! -i br-748fea589ab8 -o br-748fea589ab8 -j DROP
+-A DOCKER ! -i br-b52d11330b20 -o br-b52d11330b20 -j DROP
+-A DOCKER ! -i br-f91c61cdba28 -o br-f91c61cdba28 -j DROP
+-A DOCKER ! -i br-25187260f675 -o br-25187260f675 -j DROP
+-A DOCKER ! -i br-3d55385c5e7d -o br-3d55385c5e7d -j DROP
+-A DOCKER ! -i br-50e2988e78eb -o br-50e2988e78eb -j DROP
+-A DOCKER ! -i docker0 -o docker0 -j DROP
+-A DOCKER-BRIDGE -o br-748fea589ab8 -j DOCKER
+-A DOCKER-BRIDGE -o br-b52d11330b20 -j DOCKER
+-A DOCKER-BRIDGE -o br-f91c61cdba28 -j DOCKER
+-A DOCKER-BRIDGE -o br-25187260f675 -j DOCKER
+-A DOCKER-BRIDGE -o br-3d55385c5e7d -j DOCKER
+-A DOCKER-BRIDGE -o br-50e2988e78eb -j DOCKER
+-A DOCKER-BRIDGE -o docker0 -j DOCKER
+-A DOCKER-CT -o br-748fea589ab8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o br-b52d11330b20 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o br-f91c61cdba28 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o br-25187260f675 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o br-3d55385c5e7d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o br-50e2988e78eb -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
+-- ufw --
+-- firewalld --
+-- selinux/apparmor --
+(SELinux not present)
+
+========================================================================
+SECTION: 9. CRON & TIMERS
+========================================================================
+-- root crontab --
+(n/a)
+-- localuser crontab --
+(none)
+-- /etc/cron.d --
+-- pihole timer --
+
+##### AUDIT END 2026-07-28T02:41:34Z #####

netinfra/netboot-deep-audit.txt

@@ -0,0 +1,896 @@
+This system is the property of Known Element Enterprises LLC.
+
+Authorized uses only. All activity may be monitored and reported.
+
+All activities subject to monitoring/recording/review in real time and/or at a later time.
+=== COMPOSE FILES: /root/pihole and /root/NTP ===
+
+--- /root/pihole/docker-compose.yml ---
+# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
+services:
+  pihole:
+    container_name: pihole
+    image: pihole/pihole:latest
+    ports:
+      # DNS Ports
+      - "53:53/tcp"
+      - "53:53/udp"
+      # Default HTTP Port
+      - "10002:80/tcp"
+      # Default HTTPs Port. FTL will generate a self-signed certificate
+      - "10003:443/tcp"
+      # Uncomment the below if using Pi-hole as your DHCP Server
+      #- "67:67/udp"
+      # Uncomment the line below if you are using Pi-hole as your NTP server
+      #- "123:123/udp"
+    environment:
+      # Set the appropriate timezone for your location from
+      # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones, e.g:
+      TZ: 'America/Chicago'
+      # Set a password to access the web interface. Not setting one will result in a random password being assigned
+      FTLCONF_webserver_api_password: 'Gransyan1!'
+      # If using Docker's default `bridge` network setting the dns listening mode should be set to 'all'
+      FTLCONF_dns_listeningMode: 'all'
+    # Volumes store your data between container upgrades
+    volumes:
+      # For persisting Pi-hole's databases and common configuration file
+      - './etc-pihole:/etc/pihole'
+      # Uncomment the below if you have custom dnsmasq config files that you want to persist. Not needed for most starting fresh with Pi-hole v6. If you're upgrading from v5 you and have used this directory before, you should keep it enabled for the first v6 container start to allow for a complete migration. It can be removed afterwards. Needs environment variable FTLCONF_misc_etc_dnsmasq_d: 'true'
+      #- './etc-dnsmasq.d:/etc/dnsmasq.d'
+    cap_add:
+      # See https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
+      # Required if you are using Pi-hole as your DHCP server, else not needed
+      #- NET_ADMIN
+      # Required if you are using Pi-hole as your NTP client to be able to set the host's system time
+      #- SYS_TIME
+      # Optional, if Pi-hole should get some more processing time
+      - SYS_NICE
+    restart: always
+
+--- /root/NTP/docker-compose.yml ---
+services:
+  ntp:
+    image: dockurr/chrony
+    container_name: tsys-ntp
+    environment:
+      NTP_SERVERS: "pool.ntp.org"
+    ports:
+      - 100.103.64.82:123:123/udp
+    restart: always
+
+=== DIR LAYOUT of compose project dirs ===
+total 16
+drwxr-xr-x  3 root      root      4096 Feb  6 18:32 .
+drwx------ 10 root      root      4096 Apr  5 06:16 ..
+-rw-r--r--  1 root      root      2089 Feb  6 18:30 docker-compose.yml
+drwxr-xr-x  7 localuser localuser 4096 Jul 27 20:57 etc-pihole
+total 12
+drwxr-xr-x  2 root root 4096 Jul  2  2025 .
+drwx------ 10 root root 4096 Apr  5 06:16 ..
+-rw-r--r--  1 root root  188 Jun 23  2025 docker-compose.yml
+total 84
+drwx------ 10 root root  4096 Apr  5 06:16 .
+drwxr-xr-x 20 root root  4096 Jul 20 17:55 ..
+-rw-------  1 root root  4113 Jun 18  2025 .bash_history
+-rw-r--r--  1 root root   571 Apr 10  2021 .bashrc
+drwx------  3 root root  4096 Jun 10  2025 .config
+drwxr-xr-x  2 root root  4096 Apr  5 06:16 dhcp
+-rw-------  1 root root  2960 Feb 16 19:26 dhcpmigrate
+drwxr-xr-x 10 root root  4096 Jul  2  2025 FetchApply
+drwx------  3 root root  4096 Jul 10  2025 .gnupg
+-rw-------  1 root root    40 Apr  5 06:15 .lesshst
+drwxr-xr-x  2 root root  4096 Jul  2  2025 NTP
+drwxr-xr-x  3 root root  4096 Feb  6 18:32 pihole
+-rw-r--r--  1 root root   161 Jul  9  2019 .profile
+drwx------  2 root root  4096 Jun 13  2025 .ssh
+drwxr-xr-x  2 root root  4096 Jul 27 09:08 .tmp
+-rw-------  1 root root 15370 Apr  5 06:15 .viminfo
+-rw-r--r--  1 root root   180 Jun 17  2025 .wget-hsts
+
+=== SEARCH entire FS for any technitium / dns compose files ===
+
+=== ALL CONTAINERS with compose labels ===
+NAMES      IMAGE                  STATUS                    WDIR=working dir    CFG=config files                      SVC=service
+pihole     pihole/pihole:latest   Up 48 minutes (healthy)   WDIR=/root/pihole   CFG=/root/pihole/docker-compose.yml   SVC=pihole
+tsys-ntp   dockurr/chrony         Up 7 days (healthy)       WDIR=/root/NTP      CFG=/root/NTP/docker-compose.yml      SVC=ntp
+
+=== DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container) ===
+
+###### /pihole | image=pihole/pihole:latest | proj=pihole ######
+[
+    {
+        "Id": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
+        "Created": "2026-02-07T00:32:57.4029722Z",
+        "Path": "start.sh",
+        "Args": [],
+        "State": {
+            "Status": "running",
+            "Running": true,
+            "Paused": false,
+            "Restarting": false,
+            "OOMKilled": false,
+            "Dead": false,
+            "Pid": 1222239,
+            "ExitCode": 0,
+            "Error": "",
+            "StartedAt": "2026-07-28T01:57:27.732608642Z",
+            "FinishedAt": "2026-07-28T01:57:27.095917957Z",
+            "Health": {
+                "Status": "healthy",
+                "FailingStreak": 0,
+                "Log": [
+                    {
+                        "Start": "2026-07-27T21:43:50.682295418-05:00",
+                        "End": "2026-07-27T21:43:50.921778022-05:00",
+                        "ExitCode": 0,
+                        "Output": "127.0.0.1\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:44:20.925872245-05:00",
+                        "End": "2026-07-27T21:44:21.170837655-05:00",
+                        "ExitCode": 0,
+                        "Output": "127.0.0.1\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:44:51.175444367-05:00",
+                        "End": "2026-07-27T21:44:51.416140073-05:00",
+                        "ExitCode": 0,
+                        "Output": "127.0.0.1\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:45:21.417102671-05:00",
+                        "End": "2026-07-27T21:45:21.689964304-05:00",
+                        "ExitCode": 0,
+                        "Output": "127.0.0.1\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:45:51.691119192-05:00",
+                        "End": "2026-07-27T21:45:51.945416009-05:00",
+                        "ExitCode": 0,
+                        "Output": "127.0.0.1\n"
+                    }
+                ]
+            }
+        },
+        "Image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
+        "ResolvConfPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/resolv.conf",
+        "HostnamePath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hostname",
+        "HostsPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/hosts",
+        "LogPath": "/var/lib/docker/containers/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82/1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82-json.log",
+        "Name": "/pihole",
+        "RestartCount": 2,
+        "Driver": "overlay2",
+        "Platform": "linux",
+        "MountLabel": "",
+        "ProcessLabel": "",
+        "AppArmorProfile": "docker-default",
+        "ExecIDs": null,
+        "HostConfig": {
+            "Binds": [
+                "/root/pihole/etc-pihole:/etc/pihole:rw"
+            ],
+            "ContainerIDFile": "",
+            "LogConfig": {
+                "Type": "json-file",
+                "Config": {}
+            },
+            "NetworkMode": "pihole_default",
+            "PortBindings": {
+                "443/tcp": [
+                    {
+                        "HostIp": "",
+                        "HostPort": "10003"
+                    }
+                ],
+                "53/tcp": [
+                    {
+                        "HostIp": "",
+                        "HostPort": "53"
+                    }
+                ],
+                "53/udp": [
+                    {
+                        "HostIp": "",
+                        "HostPort": "53"
+                    }
+                ],
+                "80/tcp": [
+                    {
+                        "HostIp": "",
+                        "HostPort": "10002"
+                    }
+                ]
+            },
+            "RestartPolicy": {
+                "Name": "always",
+                "MaximumRetryCount": 0
+            },
+            "AutoRemove": false,
+            "VolumeDriver": "",
+            "VolumesFrom": null,
+            "ConsoleSize": [
+                0,
+                0
+            ],
+            "CapAdd": [
+                "CAP_SYS_NICE"
+            ],
+            "CapDrop": null,
+            "CgroupnsMode": "private",
+            "Dns": [],
+            "DnsOptions": [],
+            "DnsSearch": [],
+            "ExtraHosts": [],
+            "GroupAdd": null,
+            "IpcMode": "private",
+            "Cgroup": "",
+            "Links": null,
+            "OomScoreAdj": 0,
+            "PidMode": "",
+            "Privileged": false,
+            "PublishAllPorts": false,
+            "ReadonlyRootfs": false,
+            "SecurityOpt": null,
+            "UTSMode": "",
+            "UsernsMode": "",
+            "ShmSize": 67108864,
+            "Runtime": "runc",
+            "Isolation": "",
+            "CpuShares": 0,
+            "Memory": 0,
+            "NanoCpus": 0,
+            "CgroupParent": "",
+            "BlkioWeight": 0,
+            "BlkioWeightDevice": null,
+            "BlkioDeviceReadBps": null,
+            "BlkioDeviceWriteBps": null,
+            "BlkioDeviceReadIOps": null,
+            "BlkioDeviceWriteIOps": null,
+            "CpuPeriod": 0,
+            "CpuQuota": 0,
+            "CpuRealtimePeriod": 0,
+            "CpuRealtimeRuntime": 0,
+            "CpusetCpus": "",
+            "CpusetMems": "",
+            "Devices": null,
+            "DeviceCgroupRules": null,
+            "DeviceRequests": null,
+            "MemoryReservation": 0,
+            "MemorySwap": 0,
+            "MemorySwappiness": null,
+            "OomKillDisable": null,
+            "PidsLimit": null,
+            "Ulimits": null,
+            "CpuCount": 0,
+            "CpuPercent": 0,
+            "IOMaximumIOps": 0,
+            "IOMaximumBandwidth": 0,
+            "MaskedPaths": [
+                "/proc/acpi",
+                "/proc/asound",
+                "/proc/interrupts",
+                "/proc/kcore",
+                "/proc/keys",
+                "/proc/latency_stats",
+                "/proc/sched_debug",
+                "/proc/scsi",
+                "/proc/timer_list",
+                "/proc/timer_stats",
+                "/sys/devices/virtual/powercap",
+                "/sys/firmware"
+            ],
+            "ReadonlyPaths": [
+                "/proc/bus",
+                "/proc/fs",
+                "/proc/irq",
+                "/proc/sys",
+                "/proc/sysrq-trigger"
+            ]
+        },
+        "GraphDriver": {
+            "Data": {
+                "ID": "1d09249f01b08b0ffea241791a11cb8f4b1f06e0a9e258f23d6a483e0b6cfa82",
+                "LowerDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac-init/diff:/var/lib/docker/overlay2/c026d6eb78252f15477a6ed84b76e7855b525a582aafa11ae38d89d932433d9c/diff:/var/lib/docker/overlay2/38b4e9cd3ba8e5a10ee7dffbf21e4e552bc30ca96df4b3fc46532d8c2f4828d0/diff:/var/lib/docker/overlay2/10c7b2bc8ab474a29274b66e94e43e31db3dd248dbe0c8e8f7069e305efc7a41/diff:/var/lib/docker/overlay2/de5840a2fb5aed78f79984d980d59f9e50839ddb792146cc7a33d185b8b3eee2/diff:/var/lib/docker/overlay2/5f0504c14572ef5e745e8b06b8f1b3ce74027bce323b7bd1153c828e0dd97d52/diff:/var/lib/docker/overlay2/7a3eefaaac250fbb5430c567e9d12320c35c5da67ad3f8b7b8a7accccc435f18/diff:/var/lib/docker/overlay2/a6c7374834f19eabf6bd3e36f8ad8f2e20e3cf32c4bfea3133ef432b163dac8b/diff:/var/lib/docker/overlay2/28ffada6ac95ffb6c493b1908d00f9cfa096069fd715fa101b910b628835bed5/diff:/var/lib/docker/overlay2/0e79a525158aef2253d47f7f429c50e48c49fec81329f2be7fc580d906528087/diff:/var/lib/docker/overlay2/7365834c77d4c08772f03b8fc0fa6d21b593d3ac70a9ec66fe77ed403d0ef064/diff:/var/lib/docker/overlay2/4bd0de3b67968b6681a962ee9fb89b6bb09b07e5d2ba0d0f85f0561cd9442d32/diff",
+                "MergedDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/merged",
+                "UpperDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/diff",
+                "WorkDir": "/var/lib/docker/overlay2/69d0a802515624c1ced9a0ae9e09ed76f94b3077d3d5628914ccee60941a51ac/work"
+            },
+            "Name": "overlay2"
+        },
+        "Mounts": [
+            {
+                "Type": "bind",
+                "Source": "/root/pihole/etc-pihole",
+                "Destination": "/etc/pihole",
+                "Mode": "rw",
+                "RW": true,
+                "Propagation": "rprivate"
+            }
+        ],
+        "Config": {
+            "Hostname": "1d09249f01b0",
+            "Domainname": "",
+            "User": "",
+            "AttachStdin": false,
+            "AttachStdout": true,
+            "AttachStderr": true,
+            "ExposedPorts": {
+                "123/udp": {},
+                "443/tcp": {},
+                "53/tcp": {},
+                "53/udp": {},
+                "67/udp": {},
+                "80/tcp": {}
+            },
+            "Tty": false,
+            "OpenStdin": false,
+            "StdinOnce": false,
+            "Env": [
+                "FTLCONF_webserver_api_password=Gransyan1!",
+                "FTLCONF_dns_listeningMode=all",
+                "TZ=America/Chicago",
+                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
+                "DNSMASQ_USER=pihole",
+                "FTL_CMD=no-daemon"
+            ],
+            "Cmd": null,
+            "Healthcheck": {
+                "Test": [
+                    "CMD-SHELL",
+                    "dig -p $(pihole-FTL --config dns.port) +short +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
+                ]
+            },
+            "Image": "pihole/pihole:latest",
+            "Volumes": null,
+            "WorkingDir": "/",
+            "Entrypoint": [
+                "start.sh"
+            ],
+            "Labels": {
+                "com.docker.compose.config-hash": "60a181f601a1959cb6dda5f5889ff706d9986f7db015664ab40608253b83348e",
+                "com.docker.compose.container-number": "1",
+                "com.docker.compose.depends_on": "",
+                "com.docker.compose.image": "sha256:81365952d1f82f114cea23ef5a2961c73da61ec5dc0e066ee2f625b1eb98854b",
+                "com.docker.compose.oneoff": "False",
+                "com.docker.compose.project": "pihole",
+                "com.docker.compose.project.config_files": "/root/pihole/docker-compose.yml",
+                "com.docker.compose.project.working_dir": "/root/pihole",
+                "com.docker.compose.service": "pihole",
+                "com.docker.compose.version": "5.0.2",
+                "org.opencontainers.image.created": "2025-06-06T19:35:20.360Z",
+                "org.opencontainers.image.description": "Pi-hole in a docker container",
+                "org.opencontainers.image.licenses": "NOASSERTION",
+                "org.opencontainers.image.revision": "b56e67ef6808de2854d30481ebcd40bcd2b43445",
+                "org.opencontainers.image.source": "https://github.com/pi-hole/docker-pi-hole",
+                "org.opencontainers.image.title": "docker-pi-hole",
+                "org.opencontainers.image.url": "https://github.com/pi-hole/docker-pi-hole",
+                "org.opencontainers.image.version": "2025.06.1"
+            }
+        },
+        "NetworkSettings": {
+            "SandboxID": "4c83df12c0f6aa377cdd17640639b35e737eb7530c6798d7b80eb3135dbfbd20",
+            "SandboxKey": "/var/run/docker/netns/4c83df12c0f6",
+            "Ports": {
+                "123/udp": null,
+                "443/tcp": [
+                    {
+                        "HostIp": "0.0.0.0",
+                        "HostPort": "10003"
+                    },
+                    {
+                        "HostIp": "::",
+                        "HostPort": "10003"
+                    }
+                ],
+                "53/tcp": [
+                    {
+                        "HostIp": "0.0.0.0",
+                        "HostPort": "53"
+                    },
+                    {
+                        "HostIp": "::",
+                        "HostPort": "53"
+                    }
+                ],
+                "53/udp": [
+                    {
+                        "HostIp": "0.0.0.0",
+                        "HostPort": "53"
+                    },
+                    {
+                        "HostIp": "::",
+                        "HostPort": "53"
+                    }
+                ],
+                "67/udp": null,
+                "80/tcp": [
+                    {
+                        "HostIp": "0.0.0.0",
+                        "HostPort": "10002"
+                    },
+                    {
+                        "HostIp": "::",
+                        "HostPort": "10002"
+                    }
+                ]
+            },
+            "Networks": {
+                "pihole_default": {
+                    "IPAMConfig": null,
+                    "Links": null,
+                    "Aliases": [
+                        "pihole",
+                        "pihole"
+                    ],
+                    "DriverOpts": null,
+                    "GwPriority": 0,
+                    "NetworkID": "3d55385c5e7d7d4ca5ddec3f98533d43452cdc2fda68831ac4a16aeb0d42867c",
+                    "EndpointID": "aadb163f607f9247c8ad41f8cadac2282272501c60e3ea7fa31fd2d584271447",
+                    "Gateway": "172.24.0.1",
+                    "IPAddress": "172.24.0.2",
+                    "MacAddress": "7e:48:e2:84:c5:15",
+                    "IPPrefixLen": 16,
+                    "IPv6Gateway": "",
+                    "GlobalIPv6Address": "",
+                    "GlobalIPv6PrefixLen": 0,
+                    "DNSNames": [
+                        "pihole",
+                        "1d09249f01b0"
+                    ]
+                }
+            }
+        }
+    }
+]
+
+###### /tsys-ntp | image=dockurr/chrony | proj=ntp ######
+[
+    {
+        "Id": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
+        "Created": "2025-06-23T16:51:39.454106229Z",
+        "Path": "/bin/startup",
+        "Args": [],
+        "State": {
+            "Status": "running",
+            "Running": true,
+            "Paused": false,
+            "Restarting": false,
+            "OOMKilled": false,
+            "Dead": false,
+            "Pid": 2109237,
+            "ExitCode": 0,
+            "Error": "",
+            "StartedAt": "2026-07-20T22:55:49.597671587Z",
+            "FinishedAt": "2026-07-20T22:55:37.650310687Z",
+            "Health": {
+                "Status": "healthy",
+                "FailingStreak": 0,
+                "Log": [
+                    {
+                        "Start": "2026-07-27T21:44:16.437981113-05:00",
+                        "End": "2026-07-27T21:44:16.501904568-05:00",
+                        "ExitCode": 0,
+                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525388062 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:44:46.50366458-05:00",
+                        "End": "2026-07-27T21:44:46.566905934-05:00",
+                        "ExitCode": 0,
+                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525413632 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:45:16.568367435-05:00",
+                        "End": "2026-07-27T21:45:16.632376789-05:00",
+                        "ExitCode": 0,
+                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525439143 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:45:46.63406538-05:00",
+                        "End": "2026-07-27T21:45:46.695064332-05:00",
+                        "ExitCode": 0,
+                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525464714 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
+                    },
+                    {
+                        "Start": "2026-07-27T21:46:16.697074212-05:00",
+                        "End": "2026-07-27T21:46:16.763778169-05:00",
+                        "ExitCode": 0,
+                        "Output": "Reference ID    : 00000000 ()\nStratum         : 0\nRef time (UTC)  : Thu Jan 01 00:00:00 1970\nSystem time     : 0.525490224 seconds slow of NTP time\nLast offset     : +0.000000000 seconds\nRMS offset      : 0.000000000 seconds\nFrequency       : 0.849 ppm slow\nResidual freq   : +0.000 ppm\nSkew            : 0.000 ppm\nRoot delay      : 1.000000000 seconds\nRoot dispersion : 1.000000000 seconds\nUpdate interval : 0.0 seconds\nLeap status     : Not synchronised\n"
+                    }
+                ]
+            }
+        },
+        "Image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
+        "ResolvConfPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/resolv.conf",
+        "HostnamePath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hostname",
+        "HostsPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/hosts",
+        "LogPath": "/var/lib/docker/containers/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908/c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908-json.log",
+        "Name": "/tsys-ntp",
+        "RestartCount": 0,
+        "Driver": "overlay2",
+        "Platform": "linux",
+        "MountLabel": "",
+        "ProcessLabel": "",
+        "AppArmorProfile": "docker-default",
+        "ExecIDs": null,
+        "HostConfig": {
+            "Binds": null,
+            "ContainerIDFile": "",
+            "LogConfig": {
+                "Type": "json-file",
+                "Config": {}
+            },
+            "NetworkMode": "ntp_default",
+            "PortBindings": {
+                "123/udp": [
+                    {
+                        "HostIp": "100.103.64.82",
+                        "HostPort": "123"
+                    }
+                ]
+            },
+            "RestartPolicy": {
+                "Name": "always",
+                "MaximumRetryCount": 0
+            },
+            "AutoRemove": false,
+            "VolumeDriver": "",
+            "VolumesFrom": null,
+            "ConsoleSize": [
+                0,
+                0
+            ],
+            "CapAdd": null,
+            "CapDrop": null,
+            "CgroupnsMode": "private",
+            "Dns": [],
+            "DnsOptions": [],
+            "DnsSearch": [],
+            "ExtraHosts": [],
+            "GroupAdd": null,
+            "IpcMode": "private",
+            "Cgroup": "",
+            "Links": null,
+            "OomScoreAdj": 0,
+            "PidMode": "",
+            "Privileged": false,
+            "PublishAllPorts": false,
+            "ReadonlyRootfs": false,
+            "SecurityOpt": null,
+            "UTSMode": "",
+            "UsernsMode": "",
+            "ShmSize": 67108864,
+            "Runtime": "runc",
+            "Isolation": "",
+            "CpuShares": 0,
+            "Memory": 0,
+            "NanoCpus": 0,
+            "CgroupParent": "",
+            "BlkioWeight": 0,
+            "BlkioWeightDevice": null,
+            "BlkioDeviceReadBps": null,
+            "BlkioDeviceWriteBps": null,
+            "BlkioDeviceReadIOps": null,
+            "BlkioDeviceWriteIOps": null,
+            "CpuPeriod": 0,
+            "CpuQuota": 0,
+            "CpuRealtimePeriod": 0,
+            "CpuRealtimeRuntime": 0,
+            "CpusetCpus": "",
+            "CpusetMems": "",
+            "Devices": null,
+            "DeviceCgroupRules": null,
+            "DeviceRequests": null,
+            "MemoryReservation": 0,
+            "MemorySwap": 0,
+            "MemorySwappiness": null,
+            "OomKillDisable": null,
+            "PidsLimit": null,
+            "Ulimits": null,
+            "CpuCount": 0,
+            "CpuPercent": 0,
+            "IOMaximumIOps": 0,
+            "IOMaximumBandwidth": 0,
+            "Mounts": [
+                {
+                    "Type": "volume",
+                    "Source": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
+                    "Target": "/run/chrony"
+                },
+                {
+                    "Type": "volume",
+                    "Source": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
+                    "Target": "/var/lib/chrony"
+                },
+                {
+                    "Type": "volume",
+                    "Source": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
+                    "Target": "/etc/chrony"
+                }
+            ],
+            "MaskedPaths": [
+                "/proc/asound",
+                "/proc/acpi",
+                "/proc/interrupts",
+                "/proc/kcore",
+                "/proc/keys",
+                "/proc/latency_stats",
+                "/proc/timer_list",
+                "/proc/timer_stats",
+                "/proc/sched_debug",
+                "/proc/scsi",
+                "/sys/firmware",
+                "/sys/devices/virtual/powercap"
+            ],
+            "ReadonlyPaths": [
+                "/proc/bus",
+                "/proc/fs",
+                "/proc/irq",
+                "/proc/sys",
+                "/proc/sysrq-trigger"
+            ]
+        },
+        "GraphDriver": {
+            "Data": {
+                "ID": "c33ced53142e99743b52fb71e834543017f4312360cfef1fe7fd6857b0676908",
+                "LowerDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db-init/diff:/var/lib/docker/overlay2/6de110c8e8511a542a0541ecba7006ff7fe26ee0e91b738d87fec292a76a5cb1/diff:/var/lib/docker/overlay2/5dc67368f8acabea18fc0be0b07d52e5a82362b5d57119871e3471c509927a73/diff:/var/lib/docker/overlay2/ce6cab3e3329a7148ae6c5e55af761b4b533eac36dcd08cb34d3de88e958d92f/diff",
+                "MergedDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/merged",
+                "UpperDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/diff",
+                "WorkDir": "/var/lib/docker/overlay2/40b6c14e2ca0bdc9b2624ca63e7e7b4ad588098c00e53ebf18b4aac5a132c6db/work"
+            },
+            "Name": "overlay2"
+        },
+        "Mounts": [
+            {
+                "Type": "volume",
+                "Name": "8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b",
+                "Source": "/var/lib/docker/volumes/8974fb973f2be9c05c1ec6548ffaad0b1caa302972f7567c3fb3a97d59b9076b/_data",
+                "Destination": "/var/lib/chrony",
+                "Driver": "local",
+                "Mode": "z",
+                "RW": true,
+                "Propagation": ""
+            },
+            {
+                "Type": "volume",
+                "Name": "562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41",
+                "Source": "/var/lib/docker/volumes/562da84ce678f8d8b9cc45665099f2562beb4a41c860b5951ef8b88a8f682e41/_data",
+                "Destination": "/etc/chrony",
+                "Driver": "local",
+                "Mode": "z",
+                "RW": true,
+                "Propagation": ""
+            },
+            {
+                "Type": "volume",
+                "Name": "e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629",
+                "Source": "/var/lib/docker/volumes/e054c42f042095e22be91432ca87e524fc0ebbc447a800c263c389577e190629/_data",
+                "Destination": "/run/chrony",
+                "Driver": "local",
+                "Mode": "z",
+                "RW": true,
+                "Propagation": ""
+            }
+        ],
+        "Config": {
+            "Hostname": "c33ced53142e",
+            "Domainname": "",
+            "User": "",
+            "AttachStdin": false,
+            "AttachStdout": true,
+            "AttachStderr": true,
+            "ExposedPorts": {
+                "123/udp": {}
+            },
+            "Tty": false,
+            "OpenStdin": false,
+            "StdinOnce": false,
+            "Env": [
+                "NTP_SERVERS=pool.ntp.org",
+                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
+                "NTP_DIRECTIVES=ratelimit\\nrtcsync"
+            ],
+            "Cmd": null,
+            "Healthcheck": {
+                "Test": [
+                    "CMD-SHELL",
+                    "chronyc -n tracking || exit 1"
+                ]
+            },
+            "Image": "dockurr/chrony",
+            "Volumes": {
+                "/etc/chrony": {},
+                "/run/chrony": {},
+                "/var/lib/chrony": {}
+            },
+            "WorkingDir": "/",
+            "Entrypoint": [
+                "/bin/startup"
+            ],
+            "Labels": {
+                "com.docker.compose.config-hash": "b9a2451b9ec10e82b113b417df6060f18aab234f117063b84ac7a29455f2feb0",
+                "com.docker.compose.container-number": "1",
+                "com.docker.compose.depends_on": "",
+                "com.docker.compose.image": "sha256:3b656f2feb55b967a6773337da2f982bc489a463066e2a4d3e82d04e53f13fbc",
+                "com.docker.compose.oneoff": "False",
+                "com.docker.compose.project": "ntp",
+                "com.docker.compose.project.config_files": "/root/NTP/docker-compose.yml",
+                "com.docker.compose.project.working_dir": "/root/NTP",
+                "com.docker.compose.replace": "61f92d4c43f32567409816af5e3c1c9622aedcb5e35b27b25980282fa3c3e1b3",
+                "com.docker.compose.service": "ntp",
+                "com.docker.compose.version": "2.36.2",
+                "org.opencontainers.image.created": "2025-06-12T01:51:48.447Z",
+                "org.opencontainers.image.description": "🕒 chronyd NTP server in a Docker container.",
+                "org.opencontainers.image.licenses": "MIT",
+                "org.opencontainers.image.revision": "43fb0f2b381bccfef5123971a51baf450ab24464",
+                "org.opencontainers.image.source": "https://github.com/dockur/chrony",
+                "org.opencontainers.image.title": "Chrony",
+                "org.opencontainers.image.url": "https://github.com/dockur/chrony",
+                "org.opencontainers.image.version": "4.7"
+            }
+        },
+        "NetworkSettings": {
+            "SandboxID": "ccaa71c75942abc56df6ae1c89a8c2ae91e769ed52eab64635a63f8ee996f782",
+            "SandboxKey": "/var/run/docker/netns/ccaa71c75942",
+            "Ports": {},
+            "Networks": {}
+        }
+    }
+]
+
+=== PI-HOLE internal config (sudo docker exec, READ-ONLY) ===
+-- ls /etc/pihole --
+total 2545492
+drwxr-xr-x 7 pihole pihole       4096 Jul 27 20:57 .
+drwxr-xr-x 1 root   root         4096 Feb  6 18:32 ..
+-rw-r----- 1 pihole pihole         65 Jun 23  2025 adlists.list
+-rw-r----- 1 pihole pihole         44 Jul 27 20:57 cli_pw
+drwxr-xr-x 2 pihole pihole       4096 Apr  6 10:07 config_backups
+-rw-r----- 1 pihole pihole          0 Jun 18  2025 dhcp.leases
+-rw-r----- 1 pihole pihole       5753 Apr  6 10:07 dnsmasq.conf
+-rw-r----- 1 pihole pihole    5500928 Jul 26 04:51 gravity.db
+drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 gravity_backups
+-rw-r----- 1 pihole pihole    4751360 Jul 19 04:51 gravity_old.db
+drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 hosts
+drwxr-xr-x 2 pihole pihole       4096 Jul 26 04:51 listsCache
+-rw-r----- 1 root   root          421 Jul 27 20:57 logrotate
+drwxr-xr-x 2 pihole pihole       4096 Jun 18  2025 migration_backup
+-rw-r----- 1 pihole pihole 2591891456 Jul 27 21:40 pihole-FTL.db
+-rw-r----- 1 pihole pihole      32768 Jul 27 21:46 pihole-FTL.db-shm
+-rw-r----- 1 pihole pihole    4272472 Jul 27 21:46 pihole-FTL.db-wal
+-rw-r----- 1 pihole pihole      55996 Apr  6 10:07 pihole.toml
+-rw------- 1 pihole pihole        713 Jun 18  2025 tls.crt
+-rw------- 1 pihole pihole       1734 Jun 18  2025 tls.pem
+-rw------- 1 pihole pihole        733 Jun 18  2025 tls_ca.crt
+-rw-r--r-- 1 pihole pihole        376 Jul 27 20:57 versions
+-- ls /etc/dnsmasq.d --
+ls: cannot access '/etc/dnsmasq.d': No such file or directory
+--- /etc/pihole/setupVars.conf ---
+cat: /etc/pihole/setupVars.conf: No such file or directory
+--- /etc/pihole/pihole-FTL.conf ---
+cat: /etc/pihole/pihole-FTL.conf: No such file or directory
+--- /etc/pihole/adlists.list ---
+https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+--- /etc/pihole/custom.list ---
+cat: /etc/pihole/custom.list: No such file or directory
+--- /etc/pihole/local.list ---
+cat: /etc/pihole/local.list: No such file or directory
+--- /etc/pihole/regex.list ---
+cat: /etc/pihole/regex.list: No such file or directory
+--- /etc/pihole/dhcp.leases ---
+--- /etc/pihole/static_ip.conf ---
+cat: /etc/pihole/static_ip.conf: No such file or directory
+-- /etc/dnsmasq.d/* --
+--- /etc/dnsmasq.d/* ---
+cat: '/etc/dnsmasq.d/*': No such file or directory
+-- pihole version --
+Core version is v6.1.2 (Latest: v6.4.3)
+Web version is v6.2.1 (Latest: v6.6)
+FTL version is v6.2.2 (Latest: v6.7)
+-- gravity row counts --
+adlist=domainlist=client=group=info=-- adlist addresses --
+OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
+-- domainlist (allow+deny, first 60) --
+OCI runtime exec failed: exec failed: unable to start container process: exec: "sqlite3": executable file not found in $PATH
+
+=== CHRONY/NTP container config (tsys-ntp) ===
+--- chrony.conf ---
+# https://github.com/dockur/chrony
+
+# chrony.conf file generated by startup script
+# located at /bin/startup
+
+# time servers provided by NTP_SERVER environment variables.
+server pool.ntp.org iburst
+
+driftfile /var/lib/chrony/chrony.drift
+makestep 0.1 3
+ratelimit
+rtcsync
+
+allow all
+--- ls /etc ---
+total 176
+drwxr-xr-x    1 root     root          4096 Jun 23  2025 .
+drwxr-xr-x    1 root     root          4096 Jun 23  2025 ..
+-rw-r--r--    1 root     root            21 Jan  8  2025 alpine-release
+drwxr-xr-x    1 root     root          4096 Jun 12  2025 apk
+drwxr-xr-x    2 root     root          4096 Jan  8  2025 busybox-paths.d
+drwxr-xr-x    2 chrony   chrony        4096 Jun 23  2025 chrony
+drwxr-xr-x    2 root     root          4096 Jan  8  2025 crontabs
+-rw-r--r--    1 root     root            89 Jan  4  2025 fstab
+-rw-r--r--    1 root     root           530 Jun 12  2025 group
+-rw-r--r--    1 root     root           524 Jun 12  2025 group-
+-rw-r--r--    1 root     root            13 Jul 20 22:55 hostname
+-rw-r--r--    1 root     root           148 Jul 20 22:55 hosts
+-rw-r--r--    1 root     root           570 Jan  4  2025 inittab
+-rw-r--r--    1 root     root            77 Jan  8  2025 issue
+drwxr-xr-x    1 root     root          4096 Jun 12  2025 logrotate.d
+drwxr-xr-x    2 root     root          4096 Jan  8  2025 modprobe.d
+-rw-r--r--    1 root     root            15 Jan  4  2025 modules
+drwxr-xr-x    2 root     root          4096 Jan  8  2025 modules-load.d
+-rw-r--r--    1 root     root           284 Jan  4  2025 motd
+lrwxrwxrwx    1 root     root            12 Jun 23  2025 mtab -> /proc/mounts
+drwxr-xr-x    8 root     root          4096 Jan  8  2025 network
+-rw-r--r--    1 root     root           205 Jan  4  2025 nsswitch.conf
+drwxr-xr-x    2 root     root          4096 Jan  8  2025 opt
+lrwxrwxrwx    1 root     root            21 Jan  8  2025 os-release -> ../usr/lib/os-release
+-rw-r--r--    1 root     root           756 Jun 12  2025 passwd
+-rw-r--r--    1 root     root           702 Jan  4  2025 passwd-
+drwxr-xr-x    7 root     root          4096 Jan  8  2025 periodic
+drwxr-xr-x    2 root     root          4096 Jun 12  2025 pkcs11
+-rw-r--r--    1 root     root           547 Jan  4  2025 profile
+MS Name/IP address         Stratum Poll Reach LastRx Last sample               
+===============================================================================
+Reference ID    : 00000000 ()
+Stratum         : 0
+Ref time (UTC)  : Thu Jan 01 00:00:00 1970
+System time     : 0.525492370 seconds slow of NTP time
+Last offset     : +0.000000000 seconds
+RMS offset      : 0.000000000 seconds
+Frequency       : 0.849 ppm slow
+Residual freq   : +0.000 ppm
+Skew            : 0.000 ppm
+Root delay      : 1.000000000 seconds
+Root dispersion : 1.000000000 seconds
+Update interval : 0.0 seconds
+Leap status     : Not synchronised
+-- bare metal ntpsec.conf --
+driftfile /var/lib/ntp/ntp.drift
+leapfile /usr/share/zoneinfo/leap-seconds.list
+server pfvsvrpi.knel.net
+restrict 127.0.0.1
+restrict ::1
+
+=== TECHNITIUM volumes ===
+
+--- volume dns_tsys-dns-config -> /var/lib/docker/volumes/dns_tsys-dns-config/_data ---
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062320.stat
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/stats/2025062321.stat
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/dns.config
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/auth.config
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/logs/2025-06-23.log
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/scopes/Default.scope
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/log.config
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/self-signed-cert.pfx
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/cache.bin
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/199.86.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/knel.net.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/119.127.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/46.96.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/181.103.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/184.108.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/211.114.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/145.105.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/97.82.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/194.67.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/2.108.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/75.110.100.in-addr.arpa.zone
+/var/lib/docker/volumes/dns_tsys-dns-config/_data/zones/64.103.100.in-addr.arpa.zone
+-- config dir listing --
+-- config.xml --
+
+--- volume dns_tyss-dns-config -> /var/lib/docker/volumes/dns_tyss-dns-config/_data ---
+-- config dir listing --
+-- config.xml --
+
+=== DONE ===

netinfra/pfv-netboot-setup.md

@@ -0,0 +1,116 @@
+# pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
+
+> **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
+> Do **not** modify it. This document describes it as audited so its services can
+> be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
+> collected by **read-only** audit scripts (`audit-netboot.sh`,
+> `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
+
+## 1. Host
+
+| Item | Value |
+|---|---|
+| Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
+| OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
+| Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
+| Timezone | `America/Chicago` (US/Central) |
+| LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
+| Tailscale | `100.103.64.82` (`tailscale0`) |
+| DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
+| Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
+| Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
+
+`eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
+`ntp_default`, `dns_default`, and several stale ones).
+
+## 2. Services overview
+
+| Service | Form | Running? |
+|---|---|---|
+| **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
+| **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
+| **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
+| **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
+
+## 3. Pi-hole (container)
+
+- **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
+- **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
+- **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
+- **Ports (host):**
+
+  | Host | Container | Purpose |
+  |---|---|---|
+  | `53/tcp`, `53/udp` | 53 | DNS |
+  | `10002/tcp` | 80 | Web admin (HTTP) |
+  | `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
+
+- **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=Gransyan1!`, `FTLCONF_dns_listeningMode=all`
+- **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
+- **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
+  - Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
+  - `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
+  - `queryLogging = true`, DNSSEC off
+- **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
+- **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
+- **Web admin:** `http://pfv-netboot:10002/admin/` (password `Gransyan1!`)
+- Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
+
+## 4. NTP (two layers)
+
+### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
+- **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
+- **Image:** `dockurr/chrony`
+- **Env:** `NTP_SERVERS=pool.ntp.org`
+- **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
+- `restart: always`
+- chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
+- On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
+
+### 4b. bare-metal `ntpsec`
+- Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
+- **Config** (`/etc/ntpsec/ntp.conf`):
+  ```
+  driftfile /var/lib/ntp/ntp.drift
+  leapfile /usr/share/zoneinfo/leap-seconds.list
+  server pfvsvrpi.knel.net
+  restrict 127.0.0.1
+  restrict ::1
+  ```
+- Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
+
+## 5. Technitium DNS (currently stopped)
+
+- **Not running** — no container and **no compose file** exists for it.
+- A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
+  (mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
+  are intact (last activity 2025-06-23). A second typo'd volume
+  `dns_tyss-dns-config` is empty.
+- Config files are **binary** (Technitium's own serialization), but copy verbatim:
+  `dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
+  `self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
+- **Zones present** (12 reverse + 1 forward):
+  - `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
+  - Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
+- **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
+- The compose project name historically was `dns` (network `dns_default` still exists).
+
+## 6. Firewall / misc
+
+- nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
+  `DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
+  `OUTPUT ACCEPT`. No UFW / firewalld.
+- Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
+  (2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
+  usermin (10000/10002/20000), Tailscale (41641).
+- SELinux absent; AppArmor default docker profile.
+
+## 7. How it was audited (no changes made)
+
+```bash
+ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh        # broad read-only sweep
+ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh   # docker inspect + compose
+ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh       # pihole.toml + technitium
+```
+
+Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.

netinfra/pfv-netinfra-setup.md

@@ -0,0 +1,220 @@
+# pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
+
+These two nodes replicate the network-infrastructure services of **pfv-netboot**
+(Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
+reads config from pfv-netboot (read-only) and relays it to each target.
+
+## 1. Nodes
+
+| | pfv-netinfra-01 | pfv-netinfra-02 |
+|---|---|---|
+| OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
+| LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
+| Tailscale | `100.70.181.72` | `100.93.194.82` |
+| RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
+| Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
+| Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
+| Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
+
+## 2. Service layout
+
+All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
+so the compose files are directly editable; data dirs keep container uids):
+
+```
+/home/localuser/services/
+├── pihole/
+│   ├── docker-compose.yml
+│   └── etc-pihole/          # copied from netboot /root/pihole/etc-pihole
+│       ├── pihole.toml      # Pi-hole v6 config (upstreams, etc.)
+│       ├── gravity.db       # adlists / domainlists / clients / groups
+│       ├── adlists.list
+│       ├── dnsmasq.conf
+│       ├── tls.{crt,pem,crt_ca}
+│       └── versions
+├── ntp/
+│   └── docker-compose.yml   # chrony container (see §5 — not used; host ntpsec serves)
+└── technitium/
+    ├── docker-compose.yml
+    └── config/              # copied from netboot orphaned volume dns_tsys-dns-config/_data
+        ├── dns.config
+        ├── auth.config
+        ├── scopes/Default.scope
+        ├── self-signed-cert.pfx
+        └── zones/           # knel.net.zone + 12 Tailscale reverse zones
+```
+
+## 3. Pi-hole (container `pihole`)
+
+Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
+
+| Host port | Container | Purpose |
+|---|---|---|
+| `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
+| `10002/tcp` | 80 | Web admin (HTTP) |
+| `10003/tcp` | 443 | Web admin (HTTPS) |
+
+`docker-compose.yml`:
+```yaml
+services:
+  pihole:
+    container_name: pihole
+    image: pihole/pihole:latest
+    hostname: pihole
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: 'Gransyan1!'
+      FTLCONF_dns_listeningMode: 'all'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+    cap_add:
+      - SYS_NICE
+    restart: always
+```
+- Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
+- Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
+- `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
+- Web admin: `http://<node>:10002/admin/` — password **`Gransyan1!`** (same as netboot).
+- Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
+
+## 4. Technitium DNS (container `tsys-dns`)
+
+Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
+`knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
+orphaned `dns_tsys-dns-config` volume.
+
+| Host port | Container | Purpose |
+|---|---|---|
+| `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
+| `5380/tcp` | 5380 | Web console (HTTP) |
+| `53443/tcp` | 53443 | Web console (HTTPS) |
+
+`docker-compose.yml`:
+```yaml
+services:
+  technitium:
+    image: technitium/dns-server
+    container_name: tsys-dns
+    ports:
+      - "5300:53/tcp"
+      - "5300:53/udp"
+      - "5380:5380/tcp"
+      - "53443:53443/tcp"
+    volumes:
+      - './config:/etc/dns'
+    restart: always
+```
+- Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
+- Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
+  password (carried over via `auth.config`). If the password is unknown, reset it
+  from the console or by removing `config/auth.config` and recreating the container.
+- **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
+  because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
+  query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
+  Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
+  Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
+  from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
+  `knel.net` A-records directly in Pi-hole's Local DNS).
+
+## 5. NTP (host `ntpsec`, not a container)
+
+Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
+serves NTP on every local address — including the Tailscale IP — and keeps the
+system clock synced. This is the **same daemon family as netboot's own bare-metal
+ntpsec**.
+
+- **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
+  Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
+  does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
+  **does** bind the Tailscale IP, so the container cannot claim it (`address
+  already in use`) and would be a non-functional duplicate (verified: the
+  container started but never synced — Stratum 0). It is therefore intentionally
+  **omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
+  host NTP unit and removes any stale `tsys-ntp` container.
+- ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
+  `restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
+- Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
+  leap normal.
+
+The `ntp/docker-compose.yml` is still written on each node for parity/reference
+(and in case the host NTP is ever disabled — then `sudo docker compose -f
+/home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
+
+## 6. Verification results (2026-07-28)
+
+| Check | pfv-netinfra-01 | pfv-netinfra-02 |
+|---|---|---|
+| `pihole` health | healthy | healthy |
+| `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
+| Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
+| `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
+| Technitium web `:5380` | HTTP 200 | HTTP 200 |
+| NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
+
+## 7. Operating the services
+
+```bash
+# status
+sudo docker ps
+
+# Pi-hole
+sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
+sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
+sudo docker exec pihole pihole -v          # version
+sudo docker exec pihole pihole -g          # rebuild gravity
+sudo docker exec pihole pihole -a -p       # set/change web password
+
+# Technitium
+sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
+sudo docker exec tsys-dns sh               # explore /etc/dns
+
+# NTP (host)
+systemctl status ntpsec
+ntpq -pn
+```
+
+## 8. Differences from pfv-netboot (intentional)
+
+1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
+   can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
+2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
+3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
+   (the container cannot bind the Tailscale IP here; see §5).
+4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
+   on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
+5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
+   copied (transient); gravity DB and all configuration are.
+
+## 9. Re-running / reproducing
+
+`setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
+present and uses `docker compose up -d` (no-ops when unchanged). It reads
+pfv-netboot read-only and never mutates it.
+
+```bash
+./setup-netinfra.sh                       # deploy to both nodes
+./setup-netinfra.sh pfv-netinfra-01       # deploy one node
+./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
+```
+
+Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
+sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
+internet for image pulls.
+
+## 10. Files in this directory
+
+| File | Purpose |
+|---|---|
+| `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
+| `audit-netboot.sh` | broad read-only audit of pfv-netboot |
+| `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
+| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
+| `baseline.sh` | read-only baseline of a target node |
+| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
+| `pfv-netboot-setup.md` | reference-node documentation |
+| `pfv-netinfra-setup.md` | this document |

netinfra/setup-netinfra.sh

@@ -0,0 +1,323 @@
+#!/usr/bin/env bash
+# =============================================================================
+# setup-netinfra.sh
+# -----------------------------------------------------------------------------
+# Replicate pfv-netboot's network services (Pi-hole, Technitium DNS, NTP)
+# onto pfv-netinfra-01 and pfv-netinfra-02.
+#
+# DESIGN
+#   * pfv-netboot is REFERENCE ONLY -- this script NEVER mutates it. All reads
+#     from it are via `ssh localuser@pfv-netboot 'sudo ...'` (read-only cmds).
+#   * The targets cannot SSH to pfv-netboot directly, so config tarballs are
+#     relayed through this workstation:
+#         ssh netboot 'sudo tar -cf - ...'  |  ssh target 'sudo tar -xf - ...'
+#   * Services are deployed under /home/localuser/services/<svc>/ on each
+#     target so localuser can manage them (mirrors netboot's localuser-owned
+#     pihole data dir). `sudo docker` is used since localuser is not in the
+#     docker group (same as on netboot).
+#
+# SERVICES
+#   pihole      pihole/pihole:latest   :53 tcp/udp  :10002->80  :10003->443
+#   ntp (chrony) dockurr/chrony        <tailscale-ip>:123:123/udp
+#   technitium  technitium/dns-server  :5300->53 tcp/udp  :5380  :53443
+#     (Technitium DNS is remapped off :53 to avoid clashing with Pi-hole.
+#      The knel.net authoritative zone + Tailscale reverse zones are preserved
+#      verbatim from netboot's orphaned dns_tsys-dns-config volume.)
+#
+# USAGE
+#   ./setup-netinfra.sh                # deploy to BOTH nodes
+#   ./setup-netinfra.sh pfv-netinfra-01   # deploy to one node
+#   ./setup-netinfra.sh pfv-netinfra-01 verify   # verify only
+# =============================================================================
+
+set -euo pipefail
+
+NETBOOT="localuser@pfv-netboot"
+SVC_ROOT="/home/localuser/services"
+PIHOLE_PW='Gransyan1!'   # replicated verbatim from netboot compose
+
+log()  { printf '\n\033[1;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
+warn() { printf '\n\033[1;33m[WARN %s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
+
+# Per-node parameters. (LAN iface is auto-detected at deploy time as a fallback.)
+declare -A NODE_TSIP=(
+  [pfv-netinfra-01]="100.70.181.72"
+  [pfv-netinfra-02]="100.93.194.82"
+)
+
+on_node() { ssh -o StrictHostKeyChecking=no "localuser@$1" "$2"; }
+
+#------------------------------------------------------------------------------
+# Verify-only mode
+#------------------------------------------------------------------------------
+verify_node() {
+  local node="$1" tsip="${NODE_TSIP[$1]}"
+  log "VERIFY $node (tailscale $tsip)"
+  on_node "$node" "bash -s" <<EOF
+set +e
+echo "### containers ###"
+sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' 2>&1
+echo
+echo "### Pi-hole DNS (dig @127.0.0.1:53 pi.hole) ###"
+dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pi.hole 2>&1
+echo "### Pi-hole -> Technitium (dig @53 knel.net SOA) ###"
+dig +time=3 +tries=1 +short @127.0.0.1 -p 53 knel.net SOA 2>&1
+echo "### Pi-hole -> Technitium (dig @53 pfv-netboot.knel.net A) ###"
+dig +time=3 +tries=1 +short @127.0.0.1 -p 53 pfv-netboot.knel.net A 2>&1
+echo "### Pi-hole web (curl :10002) ###"
+curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:10002/admin/ 2>&1
+echo
+echo "### Technitium DNS (dig @127.0.0.1:5300 knel.net SOA) ###"
+dig +time=3 +tries=1 @127.0.0.1 -p 5300 knel.net SOA +short 2>&1
+echo "### Technitium web (curl :5380) ###"
+curl -sk -o /dev/null -w 'http=%{http_code}\n' http://127.0.0.1:5380/ 2>&1
+echo
+echo "### NTP service ###"
+HOST_NTP=""
+for u in ntpsec ntp chrony openntpd; do
+  systemctl is-active --quiet "\$u" 2>/dev/null && { HOST_NTP="\$u"; break; }
+done
+echo "host daemon: \${HOST_NTP:-none}"
+if [ -n "\$HOST_NTP" ]; then
+  ntpq -c "rv 0 leap,stratum,offset" 2>&1 | head -3
+else
+  echo "(no host NTP; chrony container:)"
+  sudo docker exec tsys-ntp chronyc -n tracking 2>&1 | head -6
+fi
+EOF
+}
+
+#------------------------------------------------------------------------------
+# Deploy to one node
+#------------------------------------------------------------------------------
+deploy_node() {
+  local node="$1" tsip="${NODE_TSIP[$1]}"
+  log "==== DEPLOY $node (tailscale $tsip) ===="
+
+  # ---- 1. Prepare directories on the target -------------------------------
+  log "$node: create service dirs"
+  on_node "$node" "bash -s" <<EOF
+set -e
+sudo mkdir -p $SVC_ROOT/pihole $SVC_ROOT/ntp $SVC_ROOT/technitium
+sudo chown -R localuser:localuser $SVC_ROOT
+EOF
+
+  # ---- 2. Write compose files (as localuser) ------------------------------
+  log "$node: write docker-compose files"
+  on_node "$node" "cat > $SVC_ROOT/pihole/docker-compose.yml" <<'YAML'
+services:
+  pihole:
+    container_name: pihole
+    image: pihole/pihole:latest
+    hostname: pihole
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: 'Gransyan1!'
+      FTLCONF_dns_listeningMode: 'all'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+    cap_add:
+      - SYS_NICE
+    restart: always
+    networks:
+      - default
+      - dnsnet
+networks:
+  dnsnet:
+    external: true
+YAML
+
+  on_node "$node" "cat > $SVC_ROOT/ntp/docker-compose.yml" <<YAML
+services:
+  ntp:
+    image: dockurr/chrony
+    container_name: tsys-ntp
+    environment:
+      NTP_SERVERS: "pool.ntp.org"
+    ports:
+      - "$tsip:123:123/udp"
+    restart: always
+YAML
+
+  on_node "$node" "cat > $SVC_ROOT/technitium/docker-compose.yml" <<'YAML'
+services:
+  technitium:
+    image: technitium/dns-server
+    container_name: tsys-dns
+    ports:
+      - "5300:53/tcp"
+      - "5300:53/udp"
+      - "5380:5380/tcp"
+      - "53443:53443/tcp"
+    volumes:
+      - './config:/etc/dns'
+    restart: always
+    networks:
+      default:
+      dnsnet:
+        ipv4_address: 10.53.0.53
+networks:
+  dnsnet:
+    external: true
+YAML
+
+  # ---- 3. Relay Pi-hole config from netboot -> target ---------------------
+  log "$node: copy Pi-hole /etc/pihole from netboot (excluding query logs)"
+  if on_node "$node" "test -f $SVC_ROOT/pihole/etc-pihole/gravity.db"; then
+    log "$node: Pi-hole config already present; skipping copy"
+  else
+    on_node "$node" "sudo rm -rf $SVC_ROOT/pihole/etc-pihole"
+    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
+        "sudo tar -cf - -C /root/pihole --exclude='etc-pihole/pihole-FTL.db*' \
+                                   --exclude='etc-pihole/listsCache' \
+                                   --exclude='etc-pihole/gravity_backups' \
+                                   --exclude='etc-pihole/config_backups' \
+                                   etc-pihole" \
+      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/pihole"
+  fi
+
+  # ---- 4. Relay Technitium config from netboot orphaned volume -----------
+  log "$node: copy Technitium config from netboot (orphaned dns_tsys-dns-config volume)"
+  if on_node "$node" "test -f $SVC_ROOT/technitium/config/dns.config"; then
+    log "$node: Technitium config already present; skipping copy"
+  else
+    on_node "$node" "sudo rm -rf $SVC_ROOT/technitium/config"
+    ssh -o StrictHostKeyChecking=no "$NETBOOT" \
+        "sudo tar -cf - -C /var/lib/docker/volumes/dns_tsys-dns-config _data" \
+      | on_node "$node" "sudo tar -xf - -C $SVC_ROOT/technitium && sudo mv $SVC_ROOT/technitium/_data $SVC_ROOT/technitium/config"
+  fi
+
+  # ---- 5. Adapt copied config: interface + repoint knel.net to local Technitium
+  log "$node: adapt Pi-hole pihole.toml (interface + revServer -> local Technitium)"
+  on_node "$node" "bash -s" <<'EOF'
+set -e
+IFACE=$(ip -o -4 route show to default 2>/dev/null | awk '{print $5; exit}')
+IFACE=${IFACE:-ens18}
+TOML=/home/localuser/services/pihole/etc-pihole/pihole.toml
+if sudo test -f "$TOML"; then
+  sudo sed -i "s|^  interface = .*|  interface = \"$IFACE\"  ### ADAPTED from eth0 on clone|" "$TOML"
+  echo "set interface=$IFACE"
+  # Repoint knel.net conditional forward from netboot's upstream (192.168.3.16)
+  # to the LOCAL Technitium container at its fixed dnsnet IP 10.53.0.53.
+  # Subnet 100.64.0.0/10 = Tailscale CGNAT range (covers all Tailscale reverse zones).
+  if sudo grep -q 'revServers' "$TOML"; then
+    sudo sed -i 's|"true,[0-9./]*,192\.168\.3\.16,knel\.net"|"true,100.64.0.0/10,10.53.0.53,knel.net"|' "$TOML"
+    echo "revServer repointed to 10.53.0.53 (local Technitium)"
+  else
+    echo "(revServers not found; FTL will use defaults)"
+  fi
+else
+  echo "(pihole.toml not present; FTL will create it on first run)"
+fi
+EOF
+
+  # ---- 5b. Create shared Docker network for Pi-hole <-> Technitium ----------
+  log "$node: create dnsnet shared Docker network (10.53.0.0/24)"
+  on_node "$node" "sudo docker network create --subnet 10.53.0.0/24 dnsnet 2>/dev/null || true"
+
+  # ---- 6. Pull images -----------------------------------------------------
+  log "$node: docker compose pull (pihole, ntp, technitium)"
+  on_node "$node" "bash -s" <<EOF
+for c in pihole ntp technitium; do
+  sudo docker compose -f $SVC_ROOT/\$c/docker-compose.yml pull || echo "(pull \$c failed, continuing)"
+done
+EOF
+
+  # ---- 6a. Pi-hole up -----------------------------------------------------
+  log "$node: bring up Pi-hole"
+  on_node "$node" "sudo docker compose -f $SVC_ROOT/pihole/docker-compose.yml up -d"
+
+  # ---- 6b. NTP -- only deploy the chrony container if nothing already ----
+  # serves UDP/123 on the host. The targets already run a bare-metal ntpsec
+  # daemon (stratum-2, synced) on 0.0.0.0:123 -- the SAME service family as
+  # netboot's own bare-metal ntpsec. netboot additionally runs a chrony
+  # container on its tailscale IP, but that only works there because ntpsec
+  # there does not pre-bind the specific tailscale-IP socket. On these targets
+  # ntpsec DOES bind the tailscale IP, so the container cannot claim it and is
+  # redundant anyway. We therefore keep the host ntpsec as the NTP service.
+  log "$node: NTP -- detect host NTP service"
+  on_node "$node" "bash -s" <<'EOF'
+set +e
+HOST_NTP=""
+for u in ntpsec ntp chrony openntpd; do
+  if systemctl is-active --quiet "$u" 2>/dev/null; then HOST_NTP="$u"; break; fi
+done
+if [ -n "$HOST_NTP" ]; then
+  echo "Host NTP daemon '$HOST_NTP' is active -- it serves NTP on all local"
+  echo "addresses (incl. the Tailscale IP). This is the same daemon family as"
+  echo "netboot's bare-metal ntpsec; the netboot chrony container is redundant"
+  echo "here and CANNOT bind the Tailscale IP (the host daemon already owns it)."
+  echo "-> Keeping host NTP. Removing any stale chrony container (tsys-ntp)."
+  sudo docker rm -f tsys-ntp 2>/dev/null && echo "   (removed tsys-ntp)" || echo "   (no tsys-ntp to remove)"
+  echo "   host peers:"; ntpq -pn 2>/dev/null | head -12 || true
+else
+  echo "No host NTP daemon active; starting chrony container."
+  sudo docker compose -f /home/localuser/services/ntp/docker-compose.yml up -d
+fi
+EOF
+
+  # ---- 6c. Technitium up --------------------------------------------------
+  log "$node: bring up Technitium"
+  on_node "$node" "sudo docker compose -f $SVC_ROOT/technitium/docker-compose.yml up -d"
+
+  # ---- 7. Wait for Pi-hole health -----------------------------------------
+  log "$node: wait for Pi-hole to become healthy"
+  on_node "$node" "bash -s" <<'EOF'
+for i in $(seq 1 30); do
+  st=$(sudo docker inspect --format '{{.State.Health.Status}}' pihole 2>/dev/null || echo none)
+  echo "  pihole health: $st"
+  [ "$st" = "healthy" ] && break
+  sleep 4
+done
+EOF
+
+  # ---- 8. Technitium defensive fallback -----------------------------------
+  # If the copied (binary) config from an older Technitium version makes the
+  # new container crash, move it aside and let Technitium start fresh so the
+  # service is at least up (admin reachable) rather than crash-looping.
+  log "$node: check Technitium health (fallback to fresh config if crash)"
+  on_node "$node" "bash -s" <<'EOF'
+set +e
+sleep 8
+rst=$(sudo docker inspect --format '{{.RestartCount}}' tsys-dns 2>/dev/null || echo 0)
+running=$(sudo docker inspect --format '{{.State.Running}}' tsys-dns 2>/dev/null || echo false)
+if [ "$running" != "true" ] || [ "$rst" -ge 4 ]; then
+  echo "Technitium unhealthy (running=$running restarts=$rst); quarantining copied config"
+  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml stop
+  sudo mv /home/localuser/services/technitium/config /home/localuser/services/technitium/config.quarantine.$(date +%s)
+  sudo mkdir -p /home/localuser/services/technitium/config
+  sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml up -d
+  echo "Technitium restarted with fresh config (old config saved as config.quarantine.*)"
+else
+  echo "Technitium OK (running=$running restarts=$rst)"
+fi
+EOF
+
+  # ---- 9. Final status -----------------------------------------------------
+  log "$node: final container status"
+  on_node "$node" "sudo docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
+}
+
+#------------------------------------------------------------------------------
+# Main
+#------------------------------------------------------------------------------
+main() {
+  local mode="${2:-deploy}"
+  if [ "${1:-all}" = "all" ]; then
+    targets=(pfv-netinfra-01 pfv-netinfra-02)
+  else
+    targets=("$1")
+  fi
+  for t in "${targets[@]}"; do
+    : "${NODE_TSIP[$t]:?unknown node $t}"
+    if [ "$mode" = "verify" ]; then verify_node "$t"; else deploy_node "$t"; fi
+  done
+  log "DONE"
+}
+
+main "$@"

returned-logs/_marker.pfv-tsys1

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys1.log 911607 17324
+ok pfv-tsys1.log 951792 16094

returned-logs/_marker.pfv-tsys3

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys3.log 725429 10091
+ok pfv-tsys3.log 750211 11660

returned-logs/_marker.pfv-tsys4

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys4.log 603525 11030
+ok pfv-tsys4.log 617476 11115

returned-logs/_marker.pfv-tsys5

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys5.log 1047724 17559
+ok pfv-tsys5.log 1064227 18745

returned-logs/_marker.pfv-tsys6

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys6.log 743495 14563
+ok pfv-tsys6.log 907424 16997

returned-logs/_marker.pfv-tsys7

@@ -1,2 +1,2 @@
 running
-ok pfv-tsys7.log 896348 17341
+ok pfv-tsys7.log 1114764 18714

returned-logs/_marker.pfv-tsys9

@@ -0,0 +1,2 @@
+running
+ok pfv-tsys9.log 793781 14230