refactor: split domain tooling into dedicated KNEL repos [#769]

siem->KNEL/siem, netinfra->KNEL/netinfra, ca->KNEL/ca, oam->KNEL/monitoring, cmdb->KNEL/inventory, dcinfra->KNEL/facilities, proxmox/perf->KNEL/perf; unit tests moved with their code. AGENTS.md layout/paths repointed. Full history retained here. Map: Redmine #769, Discourse t/331. https://projects.knownelement.com/issues/769#note-4152
2026-09-03 21:49:50 -05:00
parent ebd876b085
commit dba54b294f
342 changed files with 0 additions and 25503 deletions
@@ -1,18 +0,0 @@
 # WORKING.md — Active Session Tracker
 Agent work only. The human decides when it's done.
 A commit is blocked while any task below remains unchecked.
 ## Current Tasks — 2026-09-02 full session (all verified; details in Redmine #682/#683)
 - [x] hap-bridge retired; unsupported fixed; ecobee multicast-reachable; daemon tests (babd6d2)
 - [x] iDRAC SNMP both live; DRAC temps → 5 HA entities °F; extend shellchecked (#625 @95%)
 - [x] Govee configured E2E (key+2FA+MQTT); zero cloud devices = BLE-only (#620 @60%)
 - [x] Dashboards: Server-Room (views fix + rack-labeled temps), Health v2 (+Heart view), Garage (+PDU), Art Room (#618 @85%)
 - [x] Fleet snmp package: unknown-render + state_class fixes (2026.8 add-time entity drop) — 22 entities live
 - [x] Garage PDU → tsys-librenms relay → HA (A + W); PDU deleted from LibreNMS (authorized)
 - [x] Alerts: DRACs in 104°F trigger; tsys9 dead refs dropped; sensor-watchdog automation; test-fire sent (UAT pending)
 - [x] VM ingestion verified; Grafana started + datasource live; NUT add-on disabled; watchdog re-enabled
 - [x] Masters #682/#683 cut + reparented; #628 plan v4 final (PBS + drive rotation); #684 PBS ticket
 - [x] tsys9 upsd.users synced (founder-approved) — UPS hardware absent (founder)
 - [x] Field lessons + audit #298 (#50/#53/#55/#60) + notes on #344/#618/#620/#621/#625/#627/#628/#439/#465/#682/#630
@@ -1,36 +0,0 @@
 # ca/ — PFV fleet Certificate Authority tooling [#697]
 > Design + runbook (canonical): https://community.turnsys.com/t/320
 > Root of trust for: LDAPS, RADIUS (#476), OPNsense, k8s, iDRAC/OME,
 > Wazuh/syslog TLS. HSM-backed root ceremony comes later (#697 Nitrokey).
 ## Architecture (v1, software)
 - **Root CA**: RSA-4096, 10y, OFFLINE — lives only in `/root/ca-root/` on
  tsys-ca (moved to Nitrokey at the #697 ceremony). Never leaves the box;
  signs only the intermediate.
 - **Intermediate CA**: RSA-4096, 5y, on tsys-ca at `/etc/ssl/tsys-ca/` —
  signs all leaf certs. pathlen:0.
 - **Leaves**: RSA-2048, ≤825 days, SAN-based (serverAuth + clientAuth).
 - **No CRL/OCSP in v1** (fleet-internal); revocation = re-issue + intermediate
  pinning. CRL endpoint deferred to the GLPI/ITSM pass.
 - **Compat stance**: RSA+SHA-256 only — old Dell/iDRAC-era clients.
 ## Files
 | File | Purpose |
 |------|---------|
 | `ca-init.sh` | Initialize root + intermediate dirs ON THE CA HOST (keys never leave tsys-ca) |
 | `issue-cert.sh` | Issue a leaf cert: `issue-cert.sh <common-name> "DNS:a,DNS:b,IP:x"` |
 | `selftest.sh` | Full throwaway loop in /tmp — init, issue, chain-verify (TDD gate) |
 ## Usage (on tsys-ca, as root)
 ```bash
 bash ca-init.sh /root/ca-root /etc/ssl/tsys-ca        # once
 bash issue-cert.sh tsys-wazuh.knel.net "DNS:tsys-wazuh.knel.net,DNS:tsys-siem.knel.net"
 ```
 Certs/CSRs land in the intermediate dir's `certs/` + `csr/`. Private keys
 stay on tsys-ca (0600); nothing in this directory is ever committed with
 key material.
@@ -1,46 +0,0 @@
 #!/bin/bash
 #
 # artifact-mirror.sh — pinned artifact mirror on the tsys-ca webroot [#758]
 #
 # Run ON tsys-ca as root (via tests/remote.sh vm-file):
 #   VM_IP=tsys-ca VM_USER=root bash tests/remote.sh vm-file ca/artifact-mirror.sh
 #
 # Publishes under http://tsys-ca.knel.net/ca/dist/:
 #   dist/wazuh-agent/<ver>/wazuh-agent_<ver>_<arch>.deb   (amd64, arm64, armhf)
 #   dist/wazuh-agent/<ver>/SHA256SUMS
 # and refreshes the top-level MANIFEST.sha256.
 #
 # Fleet rule: agents install from THIS mirror (DNS name), never from the
 # public internet — deploys stop depending on upstream availability and
 # get a stable, pinned, hash-verifiable source. Integrity = SHA256SUMS +
 # the root CA fingerprint recorded on Redmine #697.
 #
 set -euo pipefail
 WWW="${WWW:-/var/www/html/ca}"
 WZ_VER="${WZ_VER:-4.14.7-1}"
 UPSTREAM="https://packages.wazuh.com/4.x/apt/pool/main/w/wazuh-agent"
 DEST="$WWW/dist/wazuh-agent/$WZ_VER"
 mkdir -p "$DEST"
 cd "$DEST"
 for arch in amd64 arm64 armhf; do
  f="wazuh-agent_${WZ_VER}_${arch}.deb"
  if [ -s "$f" ]; then
    echo "have $f"
  else
    echo "fetching $f"
    curl -fSs --retry 2 --max-time 240 -o "$f" "$UPSTREAM/$f"
  fi
 done
 # SHA256SUMS for this directory (regenerated every run — cheap)
 sha256sum -- *.deb > SHA256SUMS.tmp && mv SHA256SUMS.tmp SHA256SUMS
 cat SHA256SUMS
 # Top-level manifest rolls up the whole published tree
 cd "$WWW"
 find . -type f ! -name MANIFEST.sha256 -print0 | sort -z \
  | xargs -0 sha256sum > MANIFEST.sha256.tmp && mv MANIFEST.sha256.tmp MANIFEST.sha256
 echo "top-level MANIFEST.sha256 refreshed ($(wc -l < MANIFEST.sha256) files)"
@@ -1,50 +0,0 @@
 #!/bin/bash
 #
 # ca-init.sh — initialize the fleet CA on the CA host [#697]
 # Usage: ca-init.sh <root-dir> <intermediate-dir>
 # Creates: root CA (RSA-4096, 10y, offline dir) + intermediate (RSA-4096, 5y)
 # signing with the root. Keys are generated locally; nothing leaves the host.
 #
 set -euo pipefail
 ROOT_DIR="${1:?usage: ca-init.sh <root-dir> <intermediate-dir>}"
 INT_DIR="${2:?usage: ca-init.sh <root-dir> <intermediate-dir>}"
 ROOT_KEY="$ROOT_DIR/root.key"
 ROOT_CRT="$ROOT_DIR/root.crt"
 INT_KEY="$INT_DIR/intermediate.key"
 INT_CSR="$INT_DIR/intermediate.csr"
 INT_CRT="$INT_DIR/intermediate.crt"
 if [ -s "$ROOT_CRT" ]; then
  echo "root already exists at $ROOT_CRT — refusing to overwrite" >&2
  exit 1
 fi
 mkdir -p "$ROOT_DIR" "$INT_DIR/certs" "$INT_DIR/csr"
 chmod 700 "$ROOT_DIR"
 # --- Root CA (offline; signs only the intermediate) ---
 openssl genrsa -out "$ROOT_KEY" 4096 2>/dev/null
 chmod 400 "$ROOT_KEY"
 openssl req -x509 -new -key "$ROOT_KEY" -sha256 -days 3650 \
  -out "$ROOT_CRT" \
  -subj "/C=US/ST=Texas/O=Known Element Enterprises/OU=TechOps/CN=PFV Fleet Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "subjectKeyIdentifier=hash"
 echo "root CA: $ROOT_CRT ($(openssl x509 -in "$ROOT_CRT" -noout -subject))"
 # --- Intermediate CA (signs leaves) ---
 openssl genrsa -out "$INT_KEY" 4096 2>/dev/null
 chmod 400 "$INT_KEY"
 openssl req -new -key "$INT_KEY" -out "$INT_CSR" -sha256 \
  -subj "/C=US/ST=Texas/O=Known Element Enterprises/OU=TechOps/CN=PFV Fleet Intermediate CA"
 openssl x509 -req -in "$INT_CSR" -CA "$ROOT_CRT" -CAkey "$ROOT_KEY" \
  -CAcreateserial -days 1825 -sha256 -out "$INT_CRT" \
  -extfile <(printf 'basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid:always')
 echo "intermediate: $INT_CRT"
 # --- Chain file for distribution to TLS servers ---
 cat "$INT_CRT" "$ROOT_CRT" > "$INT_DIR/ca-chain.crt"
 openssl verify -CAfile "$ROOT_CRT" "$INT_CRT"
 echo "init complete: intermediate ready at $INT_DIR"
@@ -1,36 +0,0 @@
 #!/bin/bash
 #
 # issue-cert.sh — issue a leaf certificate from the fleet intermediate [#697]
 # Usage: issue-cert.sh <common-name> "SAN list"   e.g.
 #   issue-cert.sh tsys-wazuh.knel.net "DNS:tsys-wazuh.knel.net,DNS:tsys-siem.knel.net"
 # Requires: intermediate initialized at /etc/ssl/tsys-ca (ca-init.sh).
 # Output: <certs>/<cn>.crt (leaf+chain) + <certs>/<cn>.key (0600) on this host.
 #
 set -euo pipefail
 CN="${1:?usage: issue-cert.sh <common-name> \"SAN list\"}"
 SAN="${2:?usage: issue-cert.sh <common-name> \"SAN list\"}"
 INT_DIR="${INT_DIR:-/etc/ssl/tsys-ca}"
 INT_KEY="$INT_DIR/intermediate.key"
 INT_CRT="$INT_DIR/intermediate.crt"
 CHAIN="$INT_DIR/ca-chain.crt"
 [ -s "$INT_CRT" ] || { echo "intermediate missing at $INT_CRT — run ca-init.sh first" >&2; exit 1; }
 [ -s "$INT_KEY" ] || { echo "intermediate key missing" >&2; exit 1; }
 KEY="$INT_DIR/certs/$CN.key"
 CSR="$INT_DIR/csr/$CN.csr"
 CRT="$INT_DIR/certs/$CN.crt"
 [ -e "$CRT" ] && { echo "cert already exists: $CRT (revoke/rename first)" >&2; exit 1; }
 openssl genrsa -out "$KEY" 2048 2>/dev/null
 chmod 600 "$KEY"
 openssl req -new -key "$KEY" -out "$CSR" -sha256 \
  -subj "/C=US/ST=Texas/O=Known Element Enterprises/OU=TechOps/CN=$CN"
 openssl x509 -req -in "$CSR" -CA "$INT_CRT" -CAkey "$INT_KEY" \
  -CAcreateserial -days 825 -sha256 -out "$CRT" \
  -extfile <(printf 'basicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=%s\nauthorityKeyIdentifier=keyid,issuer\n' "$SAN")
 cat "$CRT" "$CHAIN" > "$INT_DIR/certs/$CN.fullchain"
 openssl verify -CAfile "$CHAIN" "$CRT"
 echo "issued: $CRT (+ .key, .csr, .fullchain)"
@@ -1,17 +0,0 @@
 #!/bin/bash
 #
 # selftest.sh — throwaway end-to-end CA loop in /tmp [#697]
 # TDD gate for the CA tooling: init -> issue -> verify chain -> negative check.
 #
 set -euo pipefail
 T="$(mktemp -d)"
 trap 'rm -rf "$T"' EXIT
 bash "$(dirname "$0")/ca-init.sh" "$T/root" "$T/int" > /dev/null
 [ -s "$T/root/root.key" ] && [ -s "$T/int/intermediate.crt" ] || { echo "FAIL: init" ; exit 1; }
 INT_DIR="$T/int" bash "$(dirname "$0")/issue-cert.sh" test.knel.net "DNS:test.knel.net,IP:10.0.0.1" > /dev/null
 openssl verify -CAfile "$T/root/root.crt" "$T/int/intermediate.crt" > /dev/null
 openssl x509 -in "$T/int/certs/test.knel.net.crt" -noout -ext subjectAltName | grep -q "test.knel.net"
 openssl x509 -in "$T/int/certs/test.knel.net.fullchain" -noout | head -1 > /dev/null
 echo "SELFTEST PASS: root->intermediate->leaf chain verified with SANs"
@@ -1,79 +0,0 @@
 #!/usr/bin/bash
 #
 # from-inventory.sh — convert the Discourse #307 system inventory (raw
 # markdown of its wiki post) into the CMDB seed CSV [#705]
 #
 # Input:  raw markdown of https://community.turnsys.com/t/307 (post 385),
 #         fetched e.g. via the discourse-cli raw-API escape hatch
 # Output: cmdb/seed/systems.csv (normalized CI rows; v0 schema)
 #
 # Re-run whenever #307 changes; the CSV is the GLPI import candidate, the
 # markdown topic stays the human-readable inventory.
 #
 set -euo pipefail
 if [ $# -ne 1 ] || [ ! -r "$1" ]; then
  echo "usage: $0 <inv307-raw.md  >systems.csv" >&2
  exit 2
 fi
 echo 'category,ci_name,host_node,vmid,specs,ts_ip,extra_ip,tuned,access,extra,status,source_section'
 awk -F'|' '
  function trim(s) { gsub(/^[ \t]+|[ \t]+$/, "", s); return s }
  function strip(s) { gsub(/"/, "", s); return trim(s) }
  function csv(s) { gsub(/"/, "", s); return "\"" trim(s)"\"" }
  # canon(s) — apply the section-7 VM-name -> Tailscale/DNS-name map so
  # ci_name always uses the canonical fleet name [#705]
  function canon(s) { s = strip(s); return (s in map) ? map[s] : s }
  /^## / {
    sec = substr($0, 4)
    sub(/\(.*/, "", sec)
    gsub(/^[ \t]+|[ \t]+$/, "", sec)
    next
  }
  /^\|/ {
    # pass 1 (same file read twice): capture only the section-7 rename map
    if (FNR == NR) {
      if (sec ~ /^7\./) {
        for (i = 1; i <= NF; i++) f[i] = trim($i)
        if (f[2] == "" || f[2] ~ /^-+$/ || f[2] == "VM Name") next
        key = f[2]; sub(/[ (].*/, "", key)
        map[key] = trim(f[3])
      }
      next
    }
    if (sec ~ /^7\./) next
    if (sec !~ /^[1-6]\./) next
    for (i = 1; i <= NF; i++) f[i] = trim($i)
    if (f[2] == "" || f[2] ~ /^-+$/) next
    if (f[2] == "Host" || f[2] == "VMID" || f[2] == "Name" || f[2] == "DNS Name") next
    status = (index($0, "RETIRED") > 0) ? "retired" : "active"
    if (sec ~ /^1\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "hypervisor", csv(canon(f[2])), "", "",
        csv(f[3] "; CPU " f[4] "; RAM " f[5]), csv(f[6]), csv(f[8]),
        csv(f[9]), csv(f[10]), csv("mgmt " f[7]), status, csv("307 s1")
    else if (sec ~ /^2\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "vm-prod", csv(canon(f[3])), csv(f[4]), csv(f[2]),
        csv("CPU " f[5] "; RAM " f[6] "; TS? " f[7]), csv(f[8]), "",
        csv(f[10]), csv(f[9]), "", status, csv("307 s2")
    else if (sec ~ /^3\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "vm-sectestbed", csv(canon(f[3])), "pfv-tsys5", csv(f[2]), "", csv(f[5]),
        "", csv(f[7]), csv(f[6]), "", status, csv("307 s3")
    else if (sec ~ /^4\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "vm-preprod", csv(canon(f[3])), "pfv-tsys5", csv(f[2]), "", csv(f[5]),
        "", csv(f[7]), csv(f[6]), "", status, csv("307 s4")
    else if (sec ~ /^5\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "physical-sbc", csv(canon(f[2])), "", "", "", csv(f[3]), "",
        "", csv(f[5]), csv("dns " f[4]), status, csv("307 s5")
    else if (sec ~ /^6\./)
      printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        "stale-dns-record", csv(f[2]), "", "", "", "", "",
        "", "", "", "retired", csv("307 s6: " f[4])
  }
 ' "$1" "$1"
@@ -1,169 +0,0 @@
 #!/bin/bash
 #
 # software-catalog.sh — seed the GLPI software catalog [#705]
 #
 # Usage: software-catalog.sh [--dry-run]
 # Needs: ~/.creds/glpi.env (GLPI_URL, GLPI_APP_TOKEN, GLPI_USER_TOKEN)
 #        ~/.creds/wazuh.env optional (WAZUH_API_PASS) — for live agent installs
 #
 # Creates Software assets + SoftwareVersions and links installations to
 # Computer CIs. Idempotent at all three levels (software, version, install).
 # Versions are recorded ONLY where verified; unknowns say so — GLPI Agent
 # auto-inventory (proposed) is the durable fix for version drift.
 #
 set -euo pipefail
 DRY=0
 [ "${1:-}" = "--dry-run" ] && DRY=1
 CREDS="${HOME}/.creds/glpi.env"
 # shellcheck disable=SC1090
 . "$CREDS"
 : "${GLPI_URL:?}" "${GLPI_APP_TOKEN:?}" "${GLPI_USER_TOKEN:?}"
 API="$GLPI_URL/apirest.php"
 api() {
  local method="$1" path="$2" body="${3:-}"
  if [ -n "$body" ]; then
    curl -sS --max-time 30 -X "$method" -H "App-Token: $GLPI_APP_TOKEN" \
      -H "Session-Token: $SESSION" -H "Content-Type: application/json" \
      -d "$body" "$API/$path"
  else
    curl -sS --max-time 30 -X "$method" -H "App-Token: $GLPI_APP_TOKEN" \
      -H "Session-Token: $SESSION" "$API/$path"
  fi
 }
 SESSION=$(curl -sS --max-time 20 -H "Content-Type: application/json" \
  -H "App-Token: $GLPI_APP_TOKEN" \
  -H "Authorization: user_token $GLPI_USER_TOKEN" \
  "$API/initSession" | jq -re '.session_token')
 echo "— session ok"
 # software_id <name> — existing id or empty
 sw_id() {
  api GET "Software?searchText%5Bname%5D=$(printf %s "$1" | sed 's/ /%20/g')" \
    | jq -r --arg n "$1" '[.[] | select(.name==$n)][0].id // empty'
 }
 # version_id <swid> <version> — GLPI ignores softwares_id as a search param,
 # so pull the full list and filter client-side (parent AND name)
 ver_id() {
  api GET "SoftwareVersion?range=0-999" \
    | jq -r --arg s "$1" --arg v "$2" \
      '[.[] | select((.softwares_id|tostring)==$s and .name==$v)][0].id // empty'
 }
 # computer_id <ci-name>
 pc_id() {
  api GET "Computer?searchText%5Bname%5D=$(printf %s "$1" | sed 's/ /%20/g')" \
    | jq -r --arg n "$1" '[.[] | select(.name==$n)][0].id // empty'
 }
 # install_exists <verid> <pcid>
 install_exists() {
  api GET "Item_SoftwareVersion?range=0-4999" \
    | jq -e --arg v "$1" --arg p "$2" \
      '[.[] | select(.softwareversions_id==($v|tonumber) and .items_id==($p|tonumber) and .itemtype=="Computer")] | length > 0' \
    >/dev/null 2>&1
 }
 # ensure <swname> <version> <targets-comma-list|-> — '-' = catalog only
 ensure() {
  local sw="$1" ver="$2" targets="$3" sid vid cid
  sid=$(sw_id "$sw")
  if [ -z "$sid" ]; then
    if [ "$DRY" = 1 ]; then echo "  DRY: software '$sw'"; sid=0; else
      sid=$(api POST Software "{\"input\":{\"name\":\"$sw\",\"manufacturers_id\":0,\"comment\":\"seeded by software-catalog.sh (#705)\"}}" \
        | jq -r '.id // empty')
      echo "  + software '$sw' (id=$sid)"
    fi
  else
    echo "  = software '$sw' (id=$sid)"
  fi
  [ "$sid" = 0 ] && [ "$DRY" = 1 ] && return 0
  vid=$(ver_id "$sid" "$ver")
  if [ -z "$vid" ]; then
    if [ "$DRY" = 1 ]; then echo "  DRY: version '$ver'"; else
      vid=$(api POST SoftwareVersion "{\"input\":{\"softwares_id\":\"$sid\",\"name\":\"$ver\",\"state\":0}}" \
        | jq -r '.id // empty')
      echo "  + version '$ver' (id=$vid)"
    fi
  else
    echo "  = version '$ver' (id=$vid)"
  fi
  [ -z "$vid" ] && return 0
  [ "$targets" = "-" ] && return 0
  local IFS=','
  for t in $targets; do
    cid=$(pc_id "$t")
    if [ -z "$cid" ]; then echo "    ! no CI named '$t' (skipped)"; continue; fi
    if install_exists "$vid" "$cid"; then echo "    = installed on $t"; continue; fi
    if [ "$DRY" = 1 ]; then echo "    DRY: install on $t"; else
      api POST Item_SoftwareVersion \
        "{\"input\":{\"itemtype\":\"Computer\",\"items_id\":\"$cid\",\"softwareversions_id\":\"$vid\"}}" \
        | jq -re 'if type=="array" or .id then empty else . end' >/dev/null 2>&1 \
        && echo "    ! install POST failed on $t" || echo "    + installed on $t"
    fi
  done
 }
 echo "— catalog rows"
 # name | version | targets (comma list, '-' = catalog only)
 while IFS='|' read -r sw ver targets; do
  [ -z "$sw" ] && continue
  ensure "$(echo "$sw" | xargs)" "$(echo "$ver" | xargs)" "$(echo "$targets" | xargs)"
 done <<EOF
 Proxmox VE|9.2.5|pfv-tsys1,pfv-tsys3,pfv-tsys4,pfv-tsys5,pfv-tsys6,pfv-tsys7,pfv-tsys9
 Wazuh Manager|4.14.7|tsys-siem
 Wazuh Indexer|4.14.7|tsys-siem
 Wazuh Dashboard|4.14.7|tsys-siem
 Wazuh Agent|4.14.7-1|-
 Docker Engine|unknown (GLPI Agent pending)|pfv-netinfra-01,pfv-netinfra-02,tsys-librenms,tsys-cloudron
 Pi-hole FTL (v6)|unknown (GLPI Agent pending)|pfv-netinfra-01,pfv-netinfra-02
 Technitium DNS Server|unknown (GLPI Agent pending)|pfv-netinfra-01,pfv-netinfra-02
 ISC dhcpd|unknown (GLPI Agent pending)|pfv-netinfra-01,pfv-netinfra-02
 nginx|unknown (GLPI Agent pending)|tsys-ca,tsys-librenms
 LibreNMS|unknown (GLPI Agent pending)|tsys-librenms
 NetDisco|unknown (GLPI Agent pending)|tsys-librenms
 Oxidized|unknown (GLPI Agent pending)|tsys-librenms
 SmokePing|unknown (GLPI Agent pending)|tsys-librenms
 unpoller|unknown (GLPI Agent pending)|tsys-librenms
 NUT (Network UPS Tools)|unknown (GLPI Agent pending)|-
 Proxmox Backup Server|4.x (dev, VM 5104)|-
 k3s|unknown (k8s lane)|-
 Uptime Kuma|unknown (Cloudron app)|tsys-cloudron
 GLPI|unknown (Cloudron app)|tsys-cloudron
 Vaultwarden|unknown (Cloudron app)|tsys-cloudron
 EOF
 # Wazuh Agent installs: from the LIVE manager (ground truth), CI matched by name
 if [ -r "${HOME}/.creds/wazuh.env" ] && [ "${SKIP_AGENTS:-0}" != 1 ]; then
  # shellcheck disable=SC1090
  . "${HOME}/.creds/wazuh.env"
  echo "— wazuh agent installs (live manager)"
  sid=$(sw_id "Wazuh Agent")
  [ -z "$sid" ] && { echo "  ! Wazuh Agent software missing"; exit 0; }
  vid=$(ver_id "$sid" "4.14.7-1")
  [ -z "$vid" ] && { echo "  ! version 4.14.7-1 missing"; exit 0; }
  TOKEN=$(curl -sS --max-time 20 -u "wazuh:${WAZUH_API_PASS}" -k \
    -X POST "https://tsys-wazuh.knel.net:55000/security/user/authenticate?raw=true")
  curl -sS --max-time 30 -k -H "Authorization: Bearer $TOKEN" \
    "https://tsys-wazuh.knel.net:55000/agents?status=active&limit=500" \
    | jq -r '.data.affected_items[].name' > /tmp/wz-agents.txt
  while IFS= read -r a; do
    [ "$a" = "000" ] && continue
    cid=$(pc_id "$a")
    if [ -z "$cid" ]; then echo "  ! agent '$a' has no CI (skipped)"; continue; fi
    if install_exists "$vid" "$cid"; then echo "  = $a"; continue; fi
    if [ "$DRY" = 1 ]; then echo "  DRY: install on $a"; else
      api POST Item_SoftwareVersion \
        "{\"input\":{\"itemtype\":\"Computer\",\"items_id\":\"$cid\",\"softwareversions_id\":\"$vid\"}}" >/dev/null \
        && echo "  + $a"
    fi
  done < /tmp/wz-agents.txt
 fi
 echo "— done"
@@ -1,76 +0,0 @@
 category,ci_name,host_node,vmid,specs,ts_ip,extra_ip,tuned,access,extra,status,source_section
 hypervisor,"pfv-tsys1",,,"OptiPlex 9020; CPU 8c; RAM 31Gi","100.121.189.98","10.100.100.1","virtual-host","root","mgmt 192.168.3.11",active,"307 s1"
 hypervisor,"pfv-tsys3",,,"Precision 7510; CPU 8c; RAM 31Gi","100.66.186.10","10.100.100.3","virtual-host","root","mgmt 192.168.2.5",active,"307 s1"
 hypervisor,"pfv-tsys4",,,"Precision T1700; CPU 8c; RAM 15Gi","100.70.77.93","10.100.100.4","virtual-host","root","mgmt 192.168.3.251",active,"307 s1"
 hypervisor,"pfv-tsys5",,,"Precision T7500; CPU 8c; RAM 94Gi","100.87.135.12","10.100.100.5","virtual-host","root","mgmt 192.168.3.250",active,"307 s1"
 hypervisor,"pfv-tsys6",,,"PowerEdge R610; CPU 16c; RAM 125Gi","100.73.35.111","(bond)","virtual-host","root","mgmt 192.168.3.169",active,"307 s1"
 hypervisor,"pfv-tsys7",,,"PowerEdge R620; CPU 24c; RAM 188Gi","100.110.146.27","10.100.100.7","virtual-host","root","mgmt 192.168.0.250",active,"307 s1"
 hypervisor,"pfv-tsys9",,,"OptiPlex 7080; CPU 12c; RAM 23Gi","100.101.158.76","10.100.100.9","virtual-host","root","mgmt 192.168.3.58",active,"307 s1"
 vm-prod,"pfv-bms","tsys1","100","CPU 4c; RAM 6GB; TS? Yes","100.67.108.125",,"-","SSH 22222 (LAN: pfv-bms-lan.knel.net)",,active,"307 s2"
 vm-prod,"tsys-ca","tsys1","101","CPU 2c; RAM 4GB; TS? No (DNS: 100.102.96.24)","-",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-cnode1","tsys1","102","CPU 2c; RAM 4GB; TS? Yes","100.125.134.53",,"network-latency","localuser",,active,"307 s2"
 vm-prod,"pfv-netinfra-01","tsys1","103","CPU 2c; RAM 4GB; TS? Yes","100.70.181.72",,"network-latency","localuser",,active,"307 s2"
 vm-prod,"tsys-librenms","tsys1","104","CPU 2c; RAM 4GB; TS? No (DNS: 100.86.204.77)","-",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-proxmox-datacenter","tsys1","105","CPU 2c; RAM 2GB; TS? No (DNS: 100.125.183.68)","-",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-ucs-01","tsys1","108","CPU 2c; RAM 6GB; TS? No (DNS: 100.109.13.110)","-",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-wnode-tsys3","tsys3","313","CPU 8c; RAM 28GB; TS? Yes","100.126.9.112",,"-","localuser",,active,"307 s2"
 vm-prod,"pfv-k8s-wnode-tsys5","tsys5","500","CPU 2c; RAM 12GB; TS? Yes","100.122.252.116",,"-","localuser",,retired,"307 s2"
 vm-prod,"devbox-cloudron","tsys5","501","CPU 2c; RAM 4GB; TS? Yes","100.119.72.25",,"-","NO",,active,"307 s2"
 vm-prod,"hfnoc-uisp-preprod","tsys5","515","CPU 2c; RAM 2GB; TS? No","-",,"-","NO",,active,"307 s2"
 vm-prod,"ultix-streaming","tsys5","5111","CPU 4c; RAM 25GB; TS? Yes","100.101.187.119",,"-","NO (do-not-reboot)",,active,"307 s2"
 vm-prod,"ultix-offstage","tsys5","5112","CPU 4c; RAM 22GB; TS? Yes","100.70.119.59",,"-","localuser (do-not-reboot)",,retired,"307 s2"
 vm-prod,"tsys-awx","tsys6","600","CPU 2c; RAM 12GB; TS? No (DNS: 100.91.39.53)","-",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-wnode-tsys6","tsys6","601","CPU 2c; RAM 98GB; TS? Yes","100.83.49.75",,"-","localuser",,active,"307 s2"
 vm-prod,"pfv-rr-middleware-02","tsys6","602","CPU 2c; RAM 6GB; TS? Yes","100.93.47.4",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-cnode3","tsys6","603","CPU 4c; RAM 4GB; TS? Yes","100.106.222.18",,"network-latency","localuser",,active,"307 s2"
 vm-prod,"tsys-proxmox-mailgw-01","tsys6","604","CPU 2c; RAM 4GB; TS? No (DNS: 100.68.129.71)","-",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-wnode-tsys7","tsys7","701","CPU 2c; RAM 98GB; TS? Yes","100.119.240.11",,"-","localuser",,active,"307 s2"
 vm-prod,"hfnoc-uisp","tsys7","702","CPU 2c; RAM 8GB; TS? Yes","100.94.188.89",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-rr-middleware-01","tsys7","703","CPU 2c; RAM 6GB; TS? Yes","100.106.54.59",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-k8s-cnode2","tsys7","705","CPU 4c; RAM 4GB; TS? Yes","100.109.34.72",,"network-latency","localuser",,active,"307 s2"
 vm-prod,"kali-rd","tsys7","706","CPU 2c; RAM 6GB; TS? Yes","100.105.136.23",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-siem","tsys7","707","CPU 2c; RAM 8GB; TS? No (DNS: 100.72.35.113)","-",,"-","NO",,active,"307 s2"
 vm-prod,"kali-tsys","tsys7","708","CPU 4c; RAM 6GB; TS? Yes","100.82.30.115",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-voip","tsys7","709","CPU 4c; RAM 4GB; TS? No (DNS: 100.83.126.67)","-",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-umbrel","tsys7","710","CPU 2c; RAM 26GB; TS? No (DNS: 100.66.182.14)","-",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-proxmox-mailgw-02","tsys7","711","CPU 2c; RAM 4GB; TS? No (DNS: 100.126.29.88)","-",,"-","NO",,active,"307 s2"
 vm-prod,"tsys-ucs-02","tsys9","902","CPU 2c; RAM 4GB; TS? No (DNS: 100.68.10.17)","-",,"-","NO",,active,"307 s2"
 vm-prod,"pfv-netinfra-02","tsys9","904","CPU 2c; RAM 4GB; TS? Yes","100.71.171.20",,"network-latency","localuser",,active,"307 s2"
 vm-prod,"pfv-k8s-wnode-tsys9","tsys9","905","CPU 4c; RAM 14GB; TS? Yes","100.95.201.66",,"-","localuser",,active,"307 s2"
 vm-sectestbed,"sectestbed-sandbox",pfv-tsys5,"5000",,"100.64.20.60",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-siem",pfv-tsys5,"5101",,"100.108.121.18",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-proxmox-pve",pfv-tsys5,"5102",,"100.80.72.71",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-proxmox-datacenter",pfv-tsys5,"5103",,"100.94.1.34",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-proxmox-pbs",pfv-tsys5,"5104",,"100.127.238.29",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-awx",pfv-tsys5,"5105",,"100.64.56.24",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-k8s-cnode",pfv-tsys5,"5106",,"100.68.155.111",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-k8s-wnode",pfv-tsys5,"5107",,"100.107.110.89",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-librenms",pfv-tsys5,"5108",,"100.92.94.87",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-netinfra",pfv-tsys5,"5109",,"100.100.220.117",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-cloudron",pfv-tsys5,"51011",,"100.97.140.105",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-hfnoc-uisp",pfv-tsys5,"51012",,"100.101.168.80",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-rancherplatform",pfv-tsys5,"51013",,"100.88.171.10",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-proxmox-mailgw",pfv-tsys5,"51014",,"100.117.24.21",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-ca",pfv-tsys5,"51015",,"100.113.245.124",,"-","NO",,active,"307 s3"
 vm-sectestbed,"sectestbed-voip",pfv-tsys5,"51016",,"100.86.176.105",,"-","NO",,active,"307 s3"
 vm-preprod,"preprod-awx",pfv-tsys5,"53100",,"100.77.216.36",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-siem",pfv-tsys5,"53101",,"100.98.162.14",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-rancherplatform",pfv-tsys5,"53102",,"100.126.231.121",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-proxmox-mailgw",pfv-tsys5,"53103",,"100.114.9.49",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-ca",pfv-tsys5,"53104",,"100.94.119.5",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-proxmox-datacenter",pfv-tsys5,"53105",,"100.101.250.10",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-librenms",pfv-tsys5,"53106",,"100.79.52.34",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-voip",pfv-tsys5,"53107",,"100.109.99.109",,"-","NO",,active,"307 s4"
 vm-preprod,"preprod-cloudron",pfv-tsys5,"53108",,"100.95.69.89",,"-","NO",,active,"307 s4"
 physical-sbc,"pfvsvrpi",,,,"100.91.151.113",,,"Raspberry Pi","dns pfvsvrpi.knel.net",active,"307 s5"
 physical-sbc,"pfv-jetson-nano-1",,,,"100.82.230.119",,,"NVIDIA Jetson","dns pfv-jetson-nano-1.knel.net",active,"307 s5"
 physical-sbc,"subopi-dev-3",,,,"100.64.231.65",,,"SBC (Pi?)","dns subopi-dev-3.knel.net",active,"307 s5"
 physical-sbc,"subopi-dev-4",,,,"-",,,"SBC","dns subopi-dev-4.knel.net (100.65.224.85)",active,"307 s5"
 physical-sbc,"subopi3",,,,"-",,,"SBC","dns subopi3.knel.net (100.93.17.77)",active,"307 s5"
 physical-sbc,"stlpc-artroom",,,,"100.120.77.113",,,"Linux PC","dns stlpc-artroom.knel.net",active,"307 s5"
 physical-sbc,"stlpc-garage",,,,"100.72.192.22",,,"Linux PC","dns stlpc-garage.knel.net",active,"307 s5"
 physical-sbc,"stlp-3dscanner",,,,"100.125.14.37",,,"OFFLINE 184d","dns stlp-3dscanner.knel.net",active,"307 s5"
 physical-sbc,"tailscale-router",,,,"-",,,"**RETIRED 2026-09-02** — subnet-router role replaced by pfv-netinfra-01/02; host removed from tailnet; no DNS records remain","dns ~~tailscale-router.knel.net (100.103.48.57)~~",retired,"307 s5"
 physical-sbc,"netbird",,,,"100.123.45.23",,,"NetBird VPN","dns -",active,"307 s5"
 physical-sbc,"pfv-proxmox-backup-server",,,,"100.114.81.107",,,"PBS","dns pfv-proxmox-backup-server.knel.net",active,"307 s5"
 physical-sbc,"stlpc-bizoffice",,,,"100.96.130.53",,,"WINDOWS (not Linux)","dns stlpc-bizoffice.knel.net",active,"307 s5"
@@ -1,104 +0,0 @@
 #!/bin/bash
 #
 # to-glpi.sh — seed GLPI from the CMDB CSV + create the scoped agent user [#705]
 #
 # Prereqs: ~/.creds/glpi.env with GLPI_URL, GLPI_APP_TOKEN, and a VALID
 #          GLPI_USER_TOKEN (glpi admin). Run cmdb/seed/from-inventory.sh first
 #          to produce systems.csv.
 #
 # Usage:
 #   to-glpi.sh [--dry-run] [systems.csv]
 #
 # What it does:
 #   1. initSession as the admin user token.
 #   2. Creates local user 'cmdb' (Read-Only profile, random password +
 #      random api_token) — the scoped agent identity per Charles's ruling.
 #      Writes ~/.creds/glpi-agent.env (0600). Skipped if the user exists.
 #   3. Imports systems.csv rows as GLPI Computers (batch; skips names that
 #      already exist). comment carries category/specs/ts_ip provenance.
 #
 set -euo pipefail
 DRY=0
 [ "${1:-}" = "--dry-run" ] && { DRY=1; shift; }
 CSV="${1:-$(dirname "$0")/systems.csv}"
 [ -r "$CSV" ] || { echo "no CSV at $CSV" >&2; exit 2; }
 CREDS="${HOME}/.creds/glpi.env"
 # shellcheck disable=SC1090
 . "$CREDS"
 : "${GLPI_URL:?}" "${GLPI_APP_TOKEN:?}" "${GLPI_USER_TOKEN:?}"
 API="$GLPI_URL/apirest.php"
 hdr() { printf 'Content-Type: application/json\nApp-Token: %s\nSession-Token: %s\n' "$GLPI_APP_TOKEN" "$SESSION"; }
 echo "— initSession"
 SESSION=$(curl -sS --max-time 20 -H "Content-Type: application/json" \
  -H "App-Token: $GLPI_APP_TOKEN" \
  -H "Authorization: user_token $GLPI_USER_TOKEN" \
  "$API/initSession" | jq -re '.session_token')
 echo "  session ok (${#SESSION} chars)"
 # --- 2. scoped agent user -------------------------------------------------
 if curl -sS -H "$(hdr)" "$API/User?searchText%5Bname%5D=cmdb" | jq -e 'if type=="object" then (.totalcount > 0) else (length > 0) end' >/dev/null; then
  echo "— user 'cmdb' already exists (skipping create)"
 else
  AGENT_PASS=$(head -c 24 /dev/urandom | base64 | tr -d '/+=')
  AGENT_TOKEN=$(head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n')
  if [ "$DRY" = "1" ]; then
    echo "— DRY: would create user cmdb"
  else
    UID_JSON=$(curl -sS -X POST -H "$(hdr)" -d '{"input":{"name":"cmdb","realname":"CMDB agent (core-IT)","password":"'"$AGENT_PASS"'","api_token":"'"$AGENT_TOKEN"'"}}' "$API/User")
    NEWUID=$(echo "$UID_JSON" | jq -r '.id // .users_id // empty')
    echo "  created users_id=$NEWUID"
    PROF_ID=$(curl -sS -H "$(hdr)" "$API/Profile?range=0-50" | jq -r '.[] | select(.name=="Read-Only") | .id' | head -1)
    [ -n "$PROF_ID" ] && curl -sS -X POST -H "$(hdr)" \
      -d '{"input":{"users_id":"'"$NEWUID"'","profiles_id":"'"$PROF_ID"'","entities_id":0,"is_recursive":1}}' \
      "$API/Profile_User" > /dev/null && echo "  profile Read-Only ($PROF_ID) @ root entity"
    umask 077
    printf 'GLPI_URL=%s\nGLPI_APP_TOKEN=%s\nGLPI_USER_TOKEN=%s\n' \
      "$GLPI_URL" "$GLPI_APP_TOKEN" "$AGENT_TOKEN" > "${HOME}/.creds/glpi-agent.env"
    echo "  agent creds written to ~/.creds/glpi-agent.env (0600)"
  fi
 fi
 # --- 3. seed import -------------------------------------------------------
 echo "— existing Computers"
 EXIST=$(curl -sS -H "$(hdr)" "$API/Computer?range=0-999&is_deleted=0" | jq -r '[.[].name] | join("\n")')
 # CSV -> JSON objects (pure jq; header names become keys)
 jq -Rn '
  def strip: gsub("^\"";"") | gsub("\"$";"");
  (input | split(",") | map(strip)) as $keys |
  [ inputs | split(",") | map(strip) as $v |
    reduce range(0; ($keys | length)) as $i
      ({}; . + {($keys[$i]): ($v[$i] // "")}) ]
 ' "$CSV" > /tmp/glpi-rows.json
 # GLPI Computer input objects; skip names already present
 jq -c --arg existing "$EXIST" '
  [ .[] | select(.ci_name != "") | select(.ci_name as $n | ($existing | split("\n")) | index($n) | not)]
 ' /tmp/glpi-rows.json > /tmp/glpi-batch.json
 # Map to GLPI Computer fields
 jq -c '[ .[] | {name: .ci_name,
  comment: ((.category // "") + " | " + (.specs // "") + " | ts=" + (.ts_ip // "") +
            " | " + (.extra // "") + " | src=" + (.source_section // "") +
            " | status=" + (.status // ""))} ]' /tmp/glpi-batch.json > /tmp/glpi-input.json
 mv /tmp/glpi-input.json /tmp/glpi-batch.json
 TOTAL=$(jq 'length' /tmp/glpi-batch.json)
 skipped=$(( $(wc -l < "$CSV") - 1 - TOTAL ))
 echo "— import: $TOTAL to create, $skipped already present"
 if [ "$DRY" = "1" ]; then echo "— DRY: no changes made"; exit 0; fi
 # Batch-create in chunks of 25 (API limits)
 SPLIT=25; i=0
 while [ "$i" -lt "$TOTAL" ]; do
  jq ".[$i:$((i+SPLIT))]" /tmp/glpi-batch.json > /tmp/glpi-chunk.json
  curl -sS -X POST -H "$(hdr)" \
    -d "$(jq -c '{input: .}' /tmp/glpi-chunk.json)" "$API/Computer" \
    | jq -r 'if type=="array" then "  +\(length) created" else "  ERR: \(tostring)" end'
  i=$((i+SPLIT))
 done
 echo "done — verify count via GET /Computer?range=0-200"
@@ -1,10 +0,0 @@
 # dcinfra/console/README.md
 > **Documentation moved to Discourse — the canonical source of truth.**
 >
 > **Serial console management (ser2net + conman)**
 >
 > **Read it here:** https://community.turnsys.com/t/301
 >
 > *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
 > Do not update content here — edit the Discourse wiki topic instead.*
@@ -1,103 +0,0 @@
 #!/usr/bin/bash
 # shellcheck disable=SC2010,SC2012 # diagnostic script; ls|grep/ls -la on sysfs & log dirs is intentional for human-readable output
 #
 # console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4
 #
 # Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
 #
 # This script is strictly read-only. No writes to the system.
 #
 set -uo pipefail
 echo "============================================"
 echo "  Console Setup Discovery"
 echo "  Host: $(hostname)"
 echo "  Date: $(date)"
 echo "  READ-ONLY"
 echo "============================================"
 echo ""
 echo "=== 1. USB devices ==="
 lsusb 2>/dev/null || echo "(lsusb not available)"
 echo ""
 echo "=== 2. All ttyUSB* devices (with major/minor) ==="
 ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
 echo ""
 echo "=== 3. USB-serial driver bindings ==="
 echo "-- pl2303 --"
 ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- cp210x --"
 ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ftdi_sio --"
 ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ch341 --"
 ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo ""
 echo "=== 4. USB serial adapter details (vendor/model/serial per port) ==="
 for tty in /dev/ttyUSB*; do
  [ -e "$tty" ] || continue
  echo "--- $tty ---"
  udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/  /'
 done
 echo ""
 echo "=== 5. Existing /root/conmap ==="
 if [ -f /root/conmap ]; then
  cat /root/conmap
 else
  echo "(no /root/conmap)"
 fi
 ls -la /root/conmap* 2>/dev/null
 echo ""
 echo "=== 6. Screen sessions (running) ==="
 screen -ls 2>&1 || echo "(screen not running or not installed)"
 echo ""
 echo "=== 7. Existing screen wrappers/scripts in /root ==="
 ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)"
 echo ""
 echo "=== 8. ser2net ==="
 which ser2net 2>/dev/null || echo "(ser2net not installed)"
 dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)"
 cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)"
 systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)"
 echo ""
 echo "=== 9. conman ==="
 which conman 2>/dev/null || echo "(conman not installed)"
 which conmand 2>/dev/null || echo "(conmand not installed)"
 dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)"
 echo "--- /etc/conman.conf (console lines only) ---"
 grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)"
 echo "--- conmand service ---"
 systemctl is-active conmand 2>/dev/null || echo "(conmand not running)"
 systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)"
 echo ""
 echo "=== 10. Existing console logs ==="
 ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)"
 ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)"
 echo ""
 echo "=== 11. udev rules for ttyUSB ==="
 grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)"
 grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true
 echo ""
 echo "=== 12. expect availability ==="
 command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
 command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed"
 echo ""
 echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ==="
 ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)"
 echo ""
 echo "============================================"
 echo "  Discovery complete (read-only)."
 echo "============================================"
@@ -1,254 +0,0 @@
 #!/usr/bin/bash
 #
 # console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf
 #
 # Reads console/mapping.txt (the source of truth) and generates all three
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
 #      to a named symlink like /dev/consoles/pfv-r5-core-01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing
 #
 # Run this script ON the target host. It writes to:
 #   /etc/udev/rules.d/99-console-ports.rules
 #   /etc/ser2net.yaml
 #   /etc/conman/console-consoles.conf  (included by /etc/conman.conf)
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh
 #
 # Environment overrides:
 #   MAPPING_FILE  — path to mapping.txt (default: auto-detect next to this script)
 #   TS_IP         — Tailscale IP to bind ser2net on (default: auto-detect)
 #   CONMAN_LOGDIR — conman log directory (default: /var/log/conman)
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}"
 CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}"
 UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
 SER2NET_CONF="/etc/ser2net.yaml"
 CONMAN_CONF="/etc/conman.conf"
 echo "============================================"
 echo "  Console Config Generator"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- Locate mapping file ---
 # When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong.
 # Search common locations.
 if [ ! -f "$MAPPING_FILE" ]; then
  for candidate in \
    "/root/console/mapping.txt" \
    "/tmp/mapping.txt" \
    "$(dirname "$0")/mapping.txt"; do
    if [ -f "$candidate" ]; then
      MAPPING_FILE="$candidate"
      break
    fi
  done
 fi
 if [ ! -f "$MAPPING_FILE" ]; then
  echo "FATAL: mapping file not found. Tried: $MAPPING_FILE"
  echo "Copy mapping.txt to the target host first."
  exit 1
 fi
 echo "  Mapping file: $MAPPING_FILE"
 # --- Auto-detect Tailscale IP ---
 if [ -z "${TS_IP:-}" ]; then
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -z "$TS_IP" ]; then
    echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually."
    exit 1
  fi
 fi
 echo "  Tailscale IP: $TS_IP"
 echo "  ser2net will bind to: $TS_IP"
 # --- Parse mapping file (skip comments and blank lines) ---
 echo ""
 echo "--- Parsing mapping file ---"
 ENTRIES=()
 while IFS= read -r line; do
  # Skip comments and blank lines
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"  # trim whitespace
  [ -z "$line" ] && continue
  ENTRIES+=("$line")
  echo "  $line"
 done < "$MAPPING_FILE"
 if [ "${#ENTRIES[@]}" -eq 0 ]; then
  echo "FATAL: no entries found in mapping file."
  exit 1
 fi
 echo ""
 echo "  ${#ENTRIES[@]} console ports configured."
 # ============================================================
 # 1. Generate udev rules
 # ============================================================
 echo ""
 echo "--- [1/3] Generating udev rules: $UDEV_RULES ---"
 cat > "$UDEV_RULES" <<'UDEV_HEADER'
 # Stable symlinks for USB-DB9 console adapters
 # Generated by console/generate-config.sh
 # DO NOT EDIT — edit mapping.txt and re-run generate-config.sh
 #
 # These rules pin each adapter to a named symlink based on its physical
 # USB port path (ID_PATH), which is stable across reboots regardless of
 # enumeration order. This is the fix for the "USB adapters shift on reboot"
 # problem.
 #
 # To find the ID_PATH for a device:
 #   udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH
 UDEV_HEADER
 for entry in "${ENTRIES[@]}"; do
  IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
  # Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4"
  # The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0"
  # We match on the substring to be portable across PCI bus changes.
  {
    echo ""
    echo "# $name (TCP $tcp_port): $comment"
    echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"consoles/$name\""
  } >> "$UDEV_RULES"
 done
 echo "  Written: $UDEV_RULES"
 echo "  Symlinks: /dev/consoles/<name> for each device"
 # ============================================================
 # 2. Generate ser2net.yaml
 # ============================================================
 echo ""
 echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---"
 # Backup existing config if not already backed up
 if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then
  cp "$SER2NET_CONF" "${SER2NET_CONF}.orig"
  echo "  Backed up original to ${SER2NET_CONF}.orig"
 fi
 {
  echo "%YAML 1.1"
  echo "---"
  echo "# ser2net configuration for pfv-tsys4 console ports"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "#"
  echo "# All ports use telnet(rfc2217) accepter so conman and telnet clients"
  echo "# negotiate proper telnet binary mode — this prevents CR stripping"
  printf '%s\n' "# and stair-stepping on devices that send \\n\\r (LF+CR) line endings."
  echo "# Ports bound to Tailscale IP ($TS_IP) for secure remote access."
  echo "#"
  echo "# Direct telnet:  telnet $TS_IP 2001"
  echo "# Via conman:     conman -f <name>"
  echo ""
  printf '%s\n' "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    # ser2net connection block — telnet(rfc2217) accepter so conman and
    # telnet clients negotiate proper telnet binary mode. This prevents
    # CR stripping that occurs with raw TCP + conman's telnet NVT.
    echo "connection: &con${tcp_port}"
    echo "    accepter: telnet(rfc2217),tcp,${TS_IP},${tcp_port}"
    echo "    enable: on"
    echo "    options:"
    echo "      banner: *banner"
    echo "      kickolduser: true"
    echo "      telnet-brk-on-sync: true"
    echo "    connector: serialdev,"
    echo "              /dev/consoles/${name},"
    echo "              ${baud},local"
    echo ""
  done
 } > "$SER2NET_CONF"
 echo "  Written: $SER2NET_CONF"
 echo "  ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)"
 # ============================================================
 # 3. Write conman console entries directly into conman.conf
 # ============================================================
 # conman 0.3.x does NOT support the 'include' directive, so we write
 # CONSOLE entries directly into /etc/conman.conf between idempotent markers.
 echo ""
 echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---"
 # Ensure logdir exists
 mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true
 # Ensure LOGDIR is set in conman.conf (server-level directive for log file paths)
 if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
  # Insert near the top, after the first SERVER directives
  sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF"
  echo "  Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
 fi
 # Ensure loopback=off so conmand is reachable over Tailscale (not localhost-only)
 if ! grep -qiE '^\s*server\s+loopback\s*=' "$CONMAN_CONF" 2>/dev/null; then
  sed -i "/^server logdir/a server loopback=off" "$CONMAN_CONF"
  echo "  Added server loopback=off to $CONMAN_CONF (enables remote access)"
 fi
 # Remove any previous auto-generated block (between markers)
 # Then append the new block
 MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
 MARKER_END="# END PFV CONSOLE DEFINITIONS"
 # Strip old block if present
 if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then
  sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF"
  echo "  Removed previous console definitions."
 fi
 # Append new block
 {
  echo ""
  echo "$MARKER_BEGIN"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "# Each console connects to a ser2net TCP port via telnet protocol."
  echo "# ser2net uses telnet(rfc2217) accepter so binary mode is negotiated"
  echo "# and CR/LF translation is handled correctly by the telnet NVT layer."
  echo "# Access: conman -f <name>"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
  done
  echo "$MARKER_END"
 } >> "$CONMAN_CONF"
 CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0)
 echo "  Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF"
 # ============================================================
 # Summary
 # ============================================================
 echo ""
 echo "============================================"
 echo "  Configuration generated successfully."
 echo ""
 echo "  Files written:"
 echo "    $UDEV_RULES       ($(wc -l < "$UDEV_RULES") lines)"
 echo "    $SER2NET_CONF     ($(wc -l < "$SER2NET_CONF") lines)"
 echo "    $CONMAN_CONF      (CONSOLE entries appended between markers)"
 echo ""
 echo "  Next steps:"
 echo "    1. Reload udev:   udevadm control --reload-rules && udevadm trigger"
 echo "    2. Restart ser2net: systemctl restart ser2net"
 echo "    3. Start conman:  systemctl enable --now conmand"
 echo "    4. Or run:       bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
 echo "============================================"
@@ -1,29 +0,0 @@
 # console/mapping.txt — Source of Truth for console port assignments
 #
 # Format: <tcp_port>|<name>|<id_path_substring>|<baud>|<comment>
 #
 # Delimiter is | (pipe) because ID_PATH values contain colons.
 #
 # - tcp_port:     TCP port ser2net listens on (also the conman console name suffix)
 # - name:         Device name (used for /dev/console/<name> symlink, conman console name)
 # - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH`
 #                      These are STABLE across reboots as long as adapters aren't moved
 #                      to different physical USB ports.
 # - baud:         Serial baud rate (9600n81 = 9600 8N1, no flow control)
 # - comment:      Free-form description
 #
 # To RE-MAP after physically moving an adapter:
 #   1. Run: bash console/discover.sh   (find the new ID_PATH for the device)
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
 2001|pfv-r5-core-01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (rack 5 core switch)
 2002|pfv-r3-tor-mgmt-01|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
 2003|pfv-r3-tor-stor-01|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
 2005|pfv-r2-tor-01|usb-0:1.6.3.3.3|9600n81|Rack 2 TOR switch
 2006|pfv-r6-mgmt-01|usb-0:1.5.4.1|9600n81|Rack 6 management switch
 # 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch (dead, removed)
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
@@ -1,62 +0,0 @@
 #!/usr/bin/bash
 #
 # console/query-remote.sh — install conman client and connect to a console
 # on pfv-tsys4 over Tailscale.
 #
 # Usage:
 #   bash console/query-remote.sh                    # list consoles
 #   bash console/query-remote.sh pfv-r5-core-01      # connect to a console
 #
 set -euo pipefail
 REMOTE_HOST="${REMOTE_HOST:-pfv-tsys4}"
 REMOTE_PORT="${REMOTE_PORT:-7890}"
 echo "============================================"
 echo "  Conman Remote Console Access"
 echo "  Server: ${REMOTE_HOST}:${REMOTE_PORT} (Tailscale)"
 echo "============================================"
 # --- 1. Install conman client if missing ---
 if ! command -v conman >/dev/null 2>&1; then
  echo ""
  echo "--- Installing conman client ---"
  if sudo -n true 2>/dev/null; then
    sudo apt-get update -qq && sudo apt-get install -y -qq conman
  else
    echo "  Passwordless sudo not available. Please run:"
    echo "    sudo apt-get update && sudo apt-get install -y conman"
    echo "  Then re-run this script."
    exit 1
  fi
 else
  echo "  conman client already installed."
 fi
 # --- 2. Verify connectivity ---
 echo ""
 echo "--- Connectivity check ---"
 if timeout 3 bash -c "echo > /dev/tcp/${REMOTE_HOST}/${REMOTE_PORT}" 2>/dev/null; then
  echo "  [OK] ${REMOTE_HOST}:${REMOTE_PORT} reachable"
 else
  echo "  [FAIL] Cannot reach ${REMOTE_HOST}:${REMOTE_PORT}"
  echo "  Is Tailscale up? Is conmand running on ${REMOTE_HOST}?"
  exit 1
 fi
 # --- 3. List or connect ---
 CONSOLE="${1:-}"
 if [ -z "$CONSOLE" ]; then
  echo ""
  echo "--- Available consoles ---"
  conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
  echo ""
  echo "To connect: bash $0 <console-name>"
  echo "  e.g: bash $0 pfv-r5-core-01"
 else
  echo ""
  echo "--- Connecting to: $CONSOLE ---"
  echo "  Escape sequence: &. (to disconnect)"
  echo ""
  conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -f "$CONSOLE"
 fi
@@ -1,217 +0,0 @@
 #!/usr/bin/bash
 # shellcheck disable=SC2010 # diagnostic; ls|grep on /dev listing is intentional
 #
 # console/setup.sh — deploy console management on pfv-tsys4
 #
 # Orchestrates the full setup:
 #   1. Ensures ser2net + conman are installed
 #   2. Copies mapping.txt to the target host (if running remotely)
 #   3. Runs generate-config.sh to produce udev rules + ser2net.yaml + conman.conf
 #   4. Reloads udev, creates /dev/consoles/ symlinks
 #   5. Restarts ser2net (TCP ports on Tailscale IP)
 #   6. Enables + starts conmand (logging + multiplexing)
 #   7. Verifies
 #
 # This script is IDEMPOTENT — safe to run multiple times.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/setup.sh
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 echo "============================================"
 echo "  Console Management Setup"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- 1. Install dependencies ---
 echo ""
 echo "--- [1/7] Checking dependencies ---"
 NEED_INSTALL=()
 dpkg -l ser2net 2>/dev/null | grep -q '^ii' && echo "  ser2net: installed" || NEED_INSTALL+=(ser2net)
 dpkg -l conman  2>/dev/null | grep -q '^ii' && echo "  conman: installed"  || NEED_INSTALL+=(conman)
 if [ "${#NEED_INSTALL[@]}" -gt 0 ]; then
  echo "  Installing: ${NEED_INSTALL[*]}"
  apt-get update -qq
  apt-get install -y -qq "${NEED_INSTALL[@]}"
 else
  echo "  All dependencies present."
 fi
 # --- 2. Ensure mapping file is available ---
 echo ""
 echo "--- [2/7] Locating mapping file ---"
 MAPPING_FILE=""
 for candidate in \
  "$SCRIPT_DIR/mapping.txt" \
  "$(dirname "$0")/mapping.txt" \
  "/root/console/mapping.txt" \
  "/tmp/mapping.txt"; do
  if [ -f "$candidate" ]; then
    MAPPING_FILE="$candidate"
    break
  fi
 done
 if [ -z "$MAPPING_FILE" ]; then
  echo "FATAL: mapping.txt not found. Copy it to the target host."
  exit 1
 fi
 echo "  Using: $MAPPING_FILE"
 # --- 3. Generate configs ---
 echo ""
 echo "--- [3/7] Generating configs ---"
 export MAPPING_FILE
 bash "$(dirname "$0")/generate-config.sh" 2>&1 || bash "$SCRIPT_DIR/generate-config.sh" 2>&1 || {
  echo "FATAL: generate-config.sh failed."
  exit 1
 }
 # --- 4. Reload udev + create symlinks ---
 echo ""
 echo "--- [4/7] Reloading udev rules ---"
 udevadm control --reload-rules
 # Try trigger first (works on some systems)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty" ] && udevadm trigger --action=add "$tty" 2>/dev/null || true
 done
 # Also try writing to uevent (forces udev reprocessing)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty/uevent" ] && echo "add" > "$tty/uevent" 2>/dev/null || true
 done
 sleep 2
 # FALLBACK: if udev symlinks don't exist (common when devices are already
 # discovered — udev trigger doesn't always re-create symlinks for existing
 # devices), create them manually by matching ID_PATH. The udev rules will
 # handle future boots/hotplugs automatically.
 if [ ! -d /dev/consoles ] || [ -z "$(ls /dev/consoles/ 2>/dev/null)" ]; then
  echo "  udev trigger didn't create symlinks. Creating manually..."
  mkdir -p /dev/consoles
  while IFS= read -r line; do
    line="${line%%#*}"
    line="$(echo "$line" | xargs)"
    [ -z "$line" ] && continue
    IFS='|' read -r _ name id_path _ _ <<< "$line"
    # Find the ttyUSB whose ID_PATH contains the mapping's id_path substring
    for tty in /dev/ttyUSB*; do
      [ -e "$tty" ] || continue
      DEV_IDPATH=$(udevadm info -q property -n "$tty" 2>/dev/null | grep ^ID_PATH= | cut -d= -f2)
      if echo "$DEV_IDPATH" | grep -q "$id_path"; then
        ln -sf "$tty" "/dev/consoles/$name"
        echo "    ln -s $tty -> /dev/consoles/$name"
        break
      fi
    done
  done < "$MAPPING_FILE"
 fi
 echo "  Stable symlinks:"
 ls -la /dev/consoles/ 2>/dev/null | grep -v '^total\|^d' | sed 's/^/    /' || echo "    (none created)"
 # Verify each symlink resolves
 echo ""
 echo "  Symlink verification:"
 while IFS= read -r line; do
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"
  [ -z "$line" ] && continue
  IFS='|' read -r _ name id_path _ _ <<< "$line"
  if [ -e "/dev/consoles/$name" ]; then
    TARGET=$(readlink -f "/dev/consoles/$name")
    echo "    [OK] /dev/consoles/$name -> $TARGET"
  else
    echo "    [MISSING] /dev/consoles/$name (adapter unplugged or ID_PATH changed)"
  fi
 done < "$MAPPING_FILE"
 # --- 5. Restart ser2net ---
 echo ""
 echo "--- [5/7] Restarting ser2net ---"
 systemctl enable ser2net
 systemctl restart ser2net
 sleep 2
 if systemctl is-active --quiet ser2net; then
  echo "  ser2net is running (telnet rfc2217 accepters)."
  TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1")
  echo "  Listening ports:"
  ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/    /'
 else
  echo "  WARNING: ser2net failed to start. Checking journal..."
  journalctl -u ser2net --no-pager -n 20
 fi
 # --- 6. Enable + start conmand ---
 echo ""
 echo "--- [6/7] Starting conmand ---"
 # conman package on Debian may not ship a systemd unit. Create one if missing.
 if ! systemctl cat conmand >/dev/null 2>&1; then
  echo "  No systemd unit for conmand — creating one..."
  cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT'
 [Unit]
 Description=ConMan (Console Manager)
 After=network.target ser2net.service
 Requires=ser2net.service
 [Service]
 Type=forking
 ExecStart=/usr/sbin/conmand -c /etc/conman.conf
 Restart=on-failure
 RestartSec=5
 [Install]
 WantedBy=multi-user.target
 CONMAND_UNIT
  systemctl daemon-reload
  echo "  Created /etc/systemd/system/conmand.service"
 fi
 # Kill any manually-started conmand first
 pkill -x conmand 2>/dev/null || true
 sleep 1
 systemctl enable conmand 2>/dev/null || true
 systemctl restart conmand 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet conmand; then
  echo "  conmand is running."
  echo "  Consoles:"
  conman -q 2>&1 | sed 's/^/    /' || true
 else
  echo "  WARNING: conmand failed to start. Checking journal..."
  journalctl -u conmand --no-pager -n 20 2>/dev/null || true
  # Try manual start as fallback
  echo "  Attempting manual start..."
  /usr/sbin/conmand -c /etc/conman.conf 2>&1 || true
 fi
 # --- 7. Summary ---
 echo ""
 echo "--- [7/7] Setup complete ---"
 echo ""
 echo "  ser2net + conman architecture (telnet rfc2217):"
 echo "    ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
 echo "    conman connects via telnet for logging + multiplexing"
 echo ""
 echo "  Connect from any Tailscale workstation:"
 echo "    conman -d pfv-tsys4:7890 -f pfv-r5-core-01"
 echo "    conman -d pfv-tsys4:7890 -q              # list consoles"
 echo ""
 echo "  Direct telnet (emergency, conflicts with conman):"
 echo "    ssh pfv-tsys4 'systemctl stop conmand'"
 echo "    telnet pfv-tsys4 2001"
 echo "    ssh pfv-tsys4 'systemctl start conmand'"
 echo ""
 echo "  To regenerate after changing mapping.txt:"
 echo "    bash generate-config.sh"
 echo "    udevadm trigger"
 echo "    systemctl restart ser2net conmand"
 echo "============================================"
@@ -1,89 +0,0 @@
 #!/usr/bin/bash
 # shellcheck disable=SC2012,SC2001 # diagnostic script; ls -la listings and sed line-prefixing are intentional
 #
 # console/validate-conman.sh — verify conman can actually reach devices via
 # ser2net TCP ports and is capturing log output to files.
 #
 # This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device
 #
 set -uo pipefail
 TS_IP=$(tailscale ip -4)
 LOGDIR="/var/log/conman"
 echo "============================================"
 echo "  Conman Data Path + Log Validation"
 echo "  Host: $(hostname)   TS IP: $TS_IP"
 echo "============================================"
 echo ""
 echo "--- 1. conman.conf log settings ---"
 grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo "  (no explicit logdir — defaults to /var/log/conman)"
 echo "  Log dir: $LOGDIR"
 ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo "  ($LOGDIR does not exist yet)"
 echo ""
 echo "--- 2. CONSOLE entries: each has a log= directive? ---"
 # Extract the auto-generated block and check each CONSOLE line has log=
 sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do
  name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p')
  if echo "$line" | grep -q 'log='; then
    logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p')
    echo "  [OK] $name → log=$logfile"
  else
    echo "  [FAIL] $name has NO log= directive"
  fi
 done
 echo ""
 echo "--- 3. Trigger log capture: connect to each console briefly ---"
 # conman -e changes the escape char. We use -j (join, read-only) with a timeout.
 # Actually, conman doesn't have a built-in "connect for N seconds" — but conmand
 # connects to each device ON STARTUP and keeps the connection open for logging.
 # The log files should already be created. Let's check timestamps.
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
 for name in pfv-r5-core-01 pfv-r3-tor-mgmt-01 pfv-r3-tor-stor-01 pfv-rrinfra-rtr pfv-r2-tor-01 pfv-r6-mgmt-01; do
  logfile="$LOGDIR/${name}.log"
  if [ -f "$logfile" ]; then
    SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
    MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1)
    echo "  [OK] $logfile ($SIZE bytes, modified $MTIME)"
  else
    echo "  [MISSING] $logfile — conmand may not be writing yet"
  fi
 done
 echo ""
 echo "--- 5. conmand connection status (journal) ---"
 # conmand logs connection attempts/errors to syslog
 journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo "  (no relevant journal entries)"
 echo ""
 echo "--- 6. Verify ser2net is proxying data (telnet rfc2217) ---"
 echo "  Probing TCP $TS_IP:2007 for data..."
 RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
 if [ -n "$RESPONSE" ]; then
  echo "  [OK] Data flowing through ser2net TCP 2007:"
  echo "$RESPONSE" | sed 's/^/    /'
 else
  echo "  (no immediate response — device may need more interaction)"
 fi
 echo ""
 echo "--- 7. Check if conmand has open connections to ser2net ports ---"
 CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
 if [ -n "$CONMAND_PID" ]; then
  echo "  conmand PID: $CONMAND_PID"
  echo "  Open connections to ser2net (expect 7 to 100.x:200X):"
  ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/    /'
  COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
  echo "  Total conmand→ser2net connections: $COUNT (expect 7)"
 else
  echo "  [FAIL] conmand not running"
 fi
 echo ""
 echo "============================================"
@@ -1,34 +0,0 @@
 #!/bin/bash
 # drac-extend: expose iDRAC IPMI temperature SDRs via snmpd extend [#625]
 #
 # Deployed to /usr/local/bin/drac-extend on pfv-tsys6 / pfv-tsys7.
 # Creds live in /etc/snmp/drac-env (0600 root): DRAC_HOST, IPMI_USER, IPMI_PASS.
 # Output: "<label> <celsius>" per enabled sensor, e.g. "inlet_temp 22".
 # Consumed by Home Assistant SNMP sensors via value_template regex.
 #
 # Why extend and not the DRAC's own SNMP agent: iDRAC6/7 hardware MIB tables
 # are unpopulated on this firmware generation (verified 2026-09-02), while
 # IPMI SDR works. [#625]
 set -euo pipefail
 ENV_FILE="${DRAC_ENV:-/etc/snmp/drac-env}"
 if [ ! -r "$ENV_FILE" ]; then
    echo "drac-extend: missing $ENV_FILE" >&2
    exit 1
 fi
 # shellcheck disable=SC1090  # env file is host-local by design (creds)
 . "$ENV_FILE"
 : "${DRAC_HOST:?DRAC_HOST unset}" "${IPMI_USER:?IPMI_USER unset}" "${IPMI_PASS:?IPMI_PASS unset}"
 export IPMI_PASSWORD="${IPMI_PASS}"
 ipmitool -I lanplus -E -H "$DRAC_HOST" -U "$IPMI_USER" \
    sdr type temperature 2>/dev/null |
    awk -F'|' '
        $3 ~ /ok/ && $5 ~ /degrees C/ {
            label = tolower($1)
            gsub(/[^a-z0-9]+/, "_", label)
            gsub(/^_+|_+$/, "", label)
            value = $5
            gsub(/[^0-9]/, "", value)
            print label, value
        }'
@@ -1,10 +0,0 @@
 # dcinfra/powerman/README.md
 > **Documentation moved to Discourse — the canonical source of truth.**
 >
 > **Cyclades PM10i PDU management via powerman**
 >
 > **Read it here:** https://community.turnsys.com/t/301
 >
 > *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
 > Do not update content here — edit the Discourse wiki topic instead.*
@@ -1,68 +0,0 @@
 #!/usr/bin/bash
 #
 # powerman/discover.sh — gather USB-DB9 adapter + powerman state on a host
 #
 # Usage: PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/discover.sh
 #
 set -uo pipefail
 echo "============================================"
 echo "  PDU / Powerman Discovery"
 echo "  Host: $(hostname)"
 echo "  Date: $(date)"
 echo "============================================"
 echo ""
 echo "=== 1. USB devices ==="
 lsusb 2>/dev/null || echo "(lsusb not available)"
 echo ""
 echo "=== 2. USB-Serial adapters (ttyUSB*) ==="
 ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
 echo ""
 echo "=== 3. USB-Serial kernel modules ==="
 lsmod | grep -iE 'usbserial|ftdi|pl2303|cp210|ch34|cdc_acm' 2>/dev/null || echo "(no relevant modules loaded)"
 echo ""
 echo "=== 4. dmesg for USB serial (last 30 lines) ==="
 dmesg | grep -iE 'ttyUSB|usbserial|ftdi|pl2303|cp210|ch34|converter' | tail -30 2>/dev/null || echo "(no dmesg matches)"
 echo ""
 echo "=== 5. All serial devices ==="
 ls -la /dev/ttyS* /dev/ttyUSB* /dev/ttyACM* 2>/dev/null || echo "(no serial devices found)"
 echo ""
 echo "=== 6. Powerman installed? ==="
 dpkg -l powerman 2>/dev/null || echo "(powerman not installed)"
 which powerman 2>/dev/null || echo "(powerman binary not found)"
 which powermand 2>/dev/null || echo "(powermand binary not found)"
 echo ""
 echo "=== 7. Powerman config files ==="
 ls -la /etc/powerman/ 2>/dev/null || echo "(no /etc/powerman/ directory)"
 ls -la /etc/powerman/*.dev 2>/dev/null || echo "(no .dev files)"
 cat /etc/powerman/powerman.conf 2>/dev/null || echo "(no powerman.conf)"
 echo ""
 echo "=== 8. Available powerman device definitions ==="
 ls /usr/share/powerman/*.dev 2>/dev/null || ls /etc/powerman/*.dev 2>/dev/null || echo "(no device definitions found)"
 echo ""
 echo "=== 9. Powermand service status ==="
 systemctl status powerman 2>/dev/null | head -10 || echo "(powerman service not found)"
 echo ""
 echo "=== 10. Serial port test (quick probe of /dev/ttyUSB0) ==="
 if [ -e /dev/ttyUSB0 ]; then
  stty -F /dev/ttyUSB0 2>/dev/null && echo "(port exists and is configurable)" || echo "(port exists but stty failed)"
  # Try to read any pending output
  timeout 2 cat /dev/ttyUSB0 2>/dev/null | head -5 || echo "(no immediate output from port)"
 else
  echo "(no /dev/ttyUSB0)"
 fi
 echo ""
 echo "============================================"
 echo "  Discovery complete."
 echo "============================================"
@@ -1,41 +0,0 @@
 #!/usr/bin/bash
 # powerman/identify-outlets.sh — flash each PDU outlet sequentially for physical tracing
 #
 # Run this from the workstation. It flashes each outlet one at a time so you
 # can walk the rack and see which device's LED blinks. Write down the mapping,
 # then run rename-outlets.sh with that mapping.
 #
 # Usage:
 #   bash dcinfra/powerman/identify-outlets.sh
 #
 # On Friday: run this, walk the rack, note which outlet → which device.
 set -uo pipefail
 PROX_HOST="${PROX_HOST:-pfv-tsys1}"
 REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
 echo "PDU Outlet Identification — Flash Sequence"
 echo "============================================"
 echo "Each outlet will flash for 5 seconds. Walk the rack and note the device."
 echo "Press Enter to start..."
 read -r
 for i in $(seq 1 10); do
  echo "--- Outlet $i: FLASHING (5s) ---"
  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -f outlet-$i" </dev/null 2>/dev/null
  sleep 5
  PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox "powerman -u outlet-$i" </dev/null 2>/dev/null
  echo "    Outlet $i → ? (write it down)"
  echo ""
  [ "$i" -lt 10 ] && { echo "Press Enter for next outlet..."; read -r; }
 done
 echo "============================================"
 echo "Done. Now create your mapping file and run:"
 echo "  bash dcinfra/powerman/rename-outlets.sh"
 echo ""
 echo "Format: outlet-number:new-name (one per line)"
 echo "Example:"
 echo "  1:pfv-tsys1"
 echo "  2:pfv-tsys3"
 echo "  ..."
 echo "============================================"
@@ -1,65 +0,0 @@
 #!/usr/bin/bash
 #
 # powerman/query-remote.sh — install powerman client locally and query
 # the Cyclades PDU running on pfv-tsys1 over Tailscale.
 #
 set -euo pipefail
 REMOTE_HOST="${REMOTE_HOST:-pfv-tsys1}"
 REMOTE_PORT="${REMOTE_PORT:-10101}"
 echo "============================================"
 echo "  Powerman Remote PDU Query"
 echo "  Server: ${REMOTE_HOST}:${REMOTE_PORT} (Tailscale)"
 echo "============================================"
 # --- 1. Install powerman client if missing ---
 if ! command -v powerman >/dev/null 2>&1; then
  echo ""
  echo "--- Installing powerman client ---"
  if sudo -n true 2>/dev/null; then
    sudo apt-get update -qq && sudo apt-get install -y -qq powerman
  else
    echo "  Passwordless sudo not available. Please run this command in a terminal:"
    echo ""
    echo "    sudo apt-get update && sudo apt-get install -y powerman"
    echo ""
    echo "  Then re-run this script."
    exit 1
  fi
 else
  echo "  powerman client already installed."
 fi
 # --- 2. Verify connectivity ---
 echo ""
 echo "--- Connectivity check ---"
 if timeout 3 bash -c "echo > /dev/tcp/${REMOTE_HOST}/${REMOTE_PORT}" 2>/dev/null; then
  echo "  [OK] ${REMOTE_HOST}:${REMOTE_PORT} reachable"
 else
  echo "  [FAIL] Cannot reach ${REMOTE_HOST}:${REMOTE_PORT}"
  echo "  Is Tailscale up? Is powermand running on ${REMOTE_HOST}?"
  exit 1
 fi
 export POWERMAN_SERVER="${REMOTE_HOST}:${REMOTE_PORT}"
 # --- 3. List outlets ---
 echo ""
 echo "--- Outlets ---"
 powerman -h "${REMOTE_HOST}:${REMOTE_PORT}" -l
 # --- 4. Query status ---
 echo ""
 echo "--- Status ---"
 powerman -h "${REMOTE_HOST}:${REMOTE_PORT}" -q
 echo ""
 echo "============================================"
 echo "  Done."
 echo ""
 echo "  To control an outlet from this workstation:"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -0 outlet-10  # off"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -1 outlet-10  # on"
 echo "    powerman -h ${REMOTE_HOST}:${REMOTE_PORT} -c outlet-10  # cycle"
 echo "============================================"
@@ -1,52 +0,0 @@
 #!/usr/bin/bash
 # powerman/rename-outlets.sh — rename PDU outlets in powerman.conf
 #
 # Takes a mapping file (outlet-number:new-name, one per line) and rewrites
 # the node entries in /etc/powerman/powerman.conf on pfv-tsys1, then
 # restarts powermand.
 #
 # Usage:
 #   bash dcinfra/powerman/rename-outlets.sh <mapping-file>
 #
 # Example mapping file:
 #   1:pfv-tsys1
 #   2:pfv-tsys3
 #   3:pfv-tsys4
 #   ...
 set -euo pipefail
 PROX_HOST="${PROX_HOST:-pfv-tsys1}"
 REMOTE_SH="$(cd "$(dirname "$0")/../.." && pwd)/tests/remote.sh"
 MAP_FILE="${1:-}"
 if [ -z "$MAP_FILE" ] || [ ! -f "$MAP_FILE" ]; then
  echo "Usage: $0 <mapping-file>"
  echo "  Format: outlet-number:new-name (one per line)"
  echo "  Run identify-outlets.sh first to get the mapping."
  exit 1
 fi
 # Build the new node lines
 NODE_LINES=""
 while IFS=: read -r num name; do
  [ -z "$num" ] && continue
  NODE_LINES+="node \"$name\" \"cyclades-pm10\" \"$num\""$'\n'
 done < "$MAP_FILE"
 # Send to tsys1: backup conf, write new node section, restart powermand
 PROX_HOST="$PROX_HOST" bash "$REMOTE_SH" prox-file - <<REMOTE_SCRIPT
 set -euo pipefail
 cp /etc/powerman/powerman.conf /etc/powerman/powerman.conf.bak.\$(date +%Y%m%d-%H%M%S)
 # Strip existing node lines and append new ones
 grep -v '^node "outlet-' /etc/powerman/powerman.conf > /tmp/powerman.conf.new
 cat >> /tmp/powerman.conf.new <<'NODES'
 $(echo -n "$NODE_LINES")
 NODES
 mv /tmp/powerman.conf.new /etc/powerman/powerman.conf
 systemctl restart powerman
 sleep 1
 powerman -l
 REMOTE_SCRIPT
 echo "PDU outlets renamed. Verify with: PROX_HOST=$PROX_HOST bash $REMOTE_SH prox 'powerman -q'"
@@ -1,181 +0,0 @@
 #!/usr/bin/bash
 #
 # powerman/setup.sh — idempotent powerman setup for Cyclades PM10i PDU
 #
 # Creates a stable udev symlink for the USB-DB9 adapter, writes powerman.conf
 # with 10 outlet nodes, and enables + starts powermand.
 #
 # This script is designed to be run ON the target host (pfv-tsys1) as root.
 # It is idempotent: safe to run multiple times.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/setup.sh
 #
 # Override defaults via environment variables:
 #   PDU_SERIAL   — USB adapter serial (default: BJAAb144J07)
 #   PDU_VENDOR   — USB vendor ID       (default: 067b)
 #   PDU_DEV_NAME — udev symlink name   (default: cyclades-pm10)
 #   PDU_BAUD     — serial baud rate    (default: 9600,8n1)
 #   PDU_TYPE     — powerman spec type  (default: pm10)
 #   PDU_OUTLETS  — number of outlets   (default: 10)
 #   PDU_LISTEN   — powermand listen    (default: 0.0.0.0:10101)
 #
 set -euo pipefail
 # --- Config (overridable via env) ---
 PDU_SERIAL="${PDU_SERIAL:-BJAAb144J07}"
 PDU_VENDOR="${PDU_VENDOR:-067b}"
 PDU_DEV_NAME="${PDU_DEV_NAME:-cyclades-pm10}"
 PDU_BAUD="${PDU_BAUD:-9600,8n1}"
 PDU_TYPE="${PDU_TYPE:-pm10}"
 PDU_OUTLETS="${PDU_OUTLETS:-10}"
 PDU_LISTEN="${PDU_LISTEN:-}"  # Auto-detect Tailscale IP if empty
 UDEV_RULE="/etc/udev/rules.d/99-cyclades-pdu.rules"
 POWERMAN_CONF="/etc/powerman/powerman.conf"
 DEV_FILE="/etc/powerman/cyclades-pm10.dev"
 # --- Auto-detect Tailscale IP for listen address ---
 if [ -z "$PDU_LISTEN" ]; then
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -n "$TS_IP" ]; then
    PDU_LISTEN="${TS_IP}:10101"
    echo "  Auto-detected Tailscale IP: $TS_IP"
  else
    PDU_LISTEN="127.0.0.1:10101"
    echo "  WARNING: No Tailscale IP detected. Defaulting to localhost."
  fi
 fi
 echo "============================================"
 echo "  Powerman PDU Setup"
 echo "  Host: $(hostname)"
 echo "  PDU:  Cyclades PM${PDU_OUTLETS}i"
 echo "  Adapter serial: $PDU_SERIAL"
 echo "  Device symlink: /dev/$PDU_DEV_NAME"
 echo "  Listen: $PDU_LISTEN (Tailscale only)"
 echo "============================================"
 # --- 1. Ensure powerman is installed ---
 echo ""
 echo "--- [1/5] Checking powerman installation ---"
 if ! dpkg -l powerman 2>/dev/null | grep -q '^ii'; then
  echo "  Installing powerman from Debian repo..."
  apt-get update -qq && apt-get install -y -qq powerman
 else
  echo "  Powerman already installed: $(dpkg -l powerman | awk '/^ii/{print $3}')"
 fi
 # --- 2. Create udev rule for stable device name ---
 echo ""
 echo "--- [2/5] Creating udev rule for USB-DB9 adapter ---"
 cat > "$UDEV_RULE" <<UDEV
 # Stable symlink for Cyclades PM10i PDU USB-DB9 adapter
 # Generated by powerman/setup.sh
 SUBSYSTEM=="tty", ATTRS{idVendor}=="${PDU_VENDOR}", ATTRS{serial}=="${PDU_SERIAL}", GROUP="dialout", MODE="0660", SYMLINK+="${PDU_DEV_NAME}"
 UDEV
 echo "  Written: $UDEV_RULE"
 # Trigger udev to create the symlink now
 udevadm control --reload-rules 2>/dev/null || true
 udevadm trigger --subsystem-match=tty 2>/dev/null || true
 sleep 1
 if [ -e "/dev/${PDU_DEV_NAME}" ]; then
  echo "  Device symlink active: /dev/${PDU_DEV_NAME} -> $(readlink -f "/dev/${PDU_DEV_NAME}")"
 else
  echo "  WARNING: /dev/${PDU_DEV_NAME} not found yet. Adapter may be unplugged."
  echo "  Falling back to /dev/ttyUSB* discovery..."
  # Try to find any ttyUSB device as fallback
  for tty in /dev/ttyUSB*; do
    if [ -e "$tty" ]; then
      echo "  Found: $tty (using as fallback)"
      PDU_DEV_NAME="$(basename "$tty")"
      break
    fi
  done
 fi
 # --- 2b. Ensure powermand user can access the serial device ---
 echo ""
 echo "--- [2b/5] Fixing serial device permissions ---"
 if id powerman >/dev/null 2>&1; then
  if id powerman | grep -qv dialout; then
    usermod -aG dialout powerman
    echo "  Added 'powerman' user to 'dialout' group"
  else
    echo "  'powerman' already in 'dialout' group"
  fi
 else
  echo "  (no powerman user — service may run as root)"
 fi
 # --- 3. Write powerman.conf ---
 echo ""
 echo "--- [3/5] Writing powerman.conf ---"
 # Build node definitions
 NODES=""
 for i in $(seq 1 "$PDU_OUTLETS"); do
  NODES+="node \"outlet-${i}\" \"${PDU_DEV_NAME}\" \"${i}\"\n"
 done
 cat > "$POWERMAN_CONF" <<PMCONF
 # Powerman configuration for Cyclades PM${PDU_OUTLETS}i PDU
 # Generated by powerman/setup.sh on $(date)
 # Device: /dev/${PDU_DEV_NAME} (USB-DB9 adapter serial ${PDU_SERIAL})
 # Listen on localhost (for local admin) and Tailscale (for remote access)
 listen "127.0.0.1:10101"
 listen "${PDU_LISTEN}"
 # Device specification for Cyclades PM10
 include "${DEV_FILE}"
 # The PDU device (serial-attached)
 device "${PDU_DEV_NAME}" "${PDU_TYPE}" "/dev/${PDU_DEV_NAME}" "${PDU_BAUD}"
 # Outlet nodes (rename these to match attached devices when onsite)
 $(printf '%b' "$NODES")
 PMCONF
 echo "  Written: $POWERMAN_CONF"
 echo "  Nodes defined: outlet-1 through outlet-${PDU_OUTLETS}"
 # --- 4. Restart powermand ---
 echo ""
 echo "--- [4/5] Restarting powermand ---"
 systemctl enable powerman 2>/dev/null || true
 systemctl restart powerman 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet powerman; then
  echo "  powermand is running."
 else
  echo "  WARNING: powermand failed to start. Check journalctl -u powerman"
  journalctl -u powerman --no-pager -n 20 2>/dev/null || true
 fi
 # --- 5. Verify ---
 echo ""
 echo "--- [5/5] Verification ---"
 echo ""
 echo "  powerman -l (list all outlets):"
 powerman -l 2>&1 || echo "(powerman -l failed)"
 echo ""
 echo "  powerman -q (query status):"
 powerman -q 2>&1 || echo "(powerman -q failed — PDU may need a moment)"
 echo ""
 echo "============================================"
 echo "  Setup complete."
 echo ""
 echo "  Outlet names are generic (outlet-1 ... outlet-${PDU_OUTLETS})."
 echo "  Rename them in ${POWERMAN_CONF} when onsite to match attached devices."
 echo ""
 echo "  Test:    powerman -0 outlet-10   (off)"
 echo "           powerman -1 outlet-10   (on)"
 echo "           powerman -c outlet-10   (cycle)"
 echo "           powerman -q             (status)"
 echo "============================================"
@@ -1,33 +0,0 @@
 #!/usr/bin/bash
 #
 # powerman/status.sh — quick PDU status check
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/status.sh
 #
 set -euo pipefail
 echo "============================================"
 echo "  Cyclades PM10i PDU Status"
 echo "  Host: $(hostname)  $(date)"
 echo "============================================"
 echo ""
 echo "=== Service ==="
 systemctl is-active powerman 2>/dev/null && echo "(running)" || echo "(stopped)"
 echo ""
 echo "=== Device ==="
 ls -la /dev/cyclades-pm10 2>/dev/null || echo "(no /dev/cyclades-pm10 symlink)"
 echo ""
 echo "=== Outlets ==="
 powerman -l 2>&1
 echo ""
 echo "=== Power Status ==="
 powerman -q 2>&1
 echo ""
 echo "=== Temperature ==="
 powerman -T 2>&1 || echo "(temperature not available)"
@@ -1,126 +0,0 @@
 #!/usr/bin/bash
 #
 # powerman/test-pdu.sh — validate PDU control by cycling outlet 10 off and on
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file powerman/test-pdu.sh
 #
 # Override: OUTLET=10  (which outlet to test)
 #
 set -euo pipefail
 OUTLET="${OUTLET:-10}"
 NODE="outlet-${OUTLET}"
 PASS=0; FAIL=0
 ok()   { echo "  [PASS] $1"; PASS=$((PASS+1)); }
 fail() { echo "  [FAIL] $1"; FAIL=$((FAIL+1)); }
 echo "============================================"
 echo "  PDU Control Validation"
 echo "  Host: $(hostname)"
 echo "  Test: cycle outlet ${OUTLET} (off → wait → on)"
 echo "============================================"
 # --- 0. Powermand running? ---
 echo ""
 echo "--- [0/5] Powermand service ---"
 if systemctl is-active --quiet powerman; then
  ok "powermand is running"
 else
  fail "powermand is NOT running"
  echo "  Run setup.sh first."
  exit 1
 fi
 # --- 1. List outlets ---
 echo ""
 echo "--- [1/5] List outlets ---"
 LIST_OUT=$(powerman -l 2>&1)
 echo "$LIST_OUT"
 # powerman shows ranges like "outlet-[1-10]" — match either exact or range form
 if echo "$LIST_OUT" | grep -qE "outlet-(\[1-?10\]|${OUTLET}\b)"; then
  ok "Outlet '${NODE}' is defined"
 else
  fail "Outlet '${NODE}' not found in powerman -l"
  exit 1
 fi
 # --- 2. Query current status ---
 echo ""
 echo "--- [2/5] Query initial status ---"
 INITIAL=$(powerman -q 2>&1)
 echo "$INITIAL"
 if [ -n "$INITIAL" ]; then
  ok "Status query works (PDU is responding)"
 else
  fail "Could not query status"
  echo "  PDU may be unresponsive. Check serial connection."
  exit 1
 fi
 # --- 3. Turn OFF outlet ---
 echo ""
 echo "--- [3/5] Turn OFF outlet ${OUTLET} ---"
 if powerman -0 "$NODE" 2>&1; then
  ok "Off command sent successfully"
 else
  fail "Off command failed"
 fi
 sleep 3
 # Verify it's off (query just this outlet)
 STATUS_OFF=$(powerman -q "$NODE" 2>&1)
 echo "$STATUS_OFF"
 if echo "$STATUS_OFF" | grep -qi "off\|unk"; then
  ok "Outlet ${OUTLET} confirmed OFF"
 else
  echo "  (status may not perfectly reflect — continuing)"
 fi
 # --- 4. Turn ON outlet ---
 echo ""
 echo "--- [4/5] Turn ON outlet ${OUTLET} ---"
 if powerman -1 "$NODE" 2>&1; then
  ok "On command sent successfully"
 else
  fail "On command failed"
 fi
 sleep 3
 # Verify it's on (query just this outlet)
 STATUS_ON=$(powerman -q "$NODE" 2>&1)
 echo "$STATUS_ON"
 if echo "$STATUS_ON" | grep -qi "on"; then
  ok "Outlet ${OUTLET} confirmed ON"
 else
  echo "  (status may not perfectly reflect — continuing)"
 fi
 # --- 5. Cycle test (off → delay → on in one command) ---
 echo ""
 echo "--- [5/5] Cycle test (powerman -c) ---"
 if powerman -c "$NODE" 2>&1; then
  ok "Cycle command completed"
 else
  fail "Cycle command failed"
  echo "  (some PDU firmware reports errors during cycle but still works)"
 fi
 sleep 5
 # Final status
 echo ""
 echo "--- Final status ---"
 powerman -q 2>&1
 echo ""
 echo "============================================"
 echo "  Results: $PASS passed, $FAIL failed"
 if [ "$FAIL" -gt 0 ]; then
  echo "  Some checks failed. Review output above."
  exit 1
 fi
 echo "  PDU control validated."
 echo "============================================"
@@ -1,20 +0,0 @@
 #!/bin/bash
 # garage-pdu-relay: expose APC AP7830 metered output current via snmpd extend.
 #
 # Relay pattern (founder ruling 2026-09-02): the garage PDU speaks SNMPv1
 # only, walks die on its old AOS, and its ACL admits the LibreNMS VM vantage.
 # This relay runs ON tsys-librenms; HA polls the relay via the fleet snmpd
 # extend (same rails as drac-extend). Output: "output_amps <x.y>".
 set -euo pipefail
 PDU_HOST="${PDU_HOST:-pfv-garage-pdu-1.knel.net}"
 COMMUNITY="${PDU_COMMUNITY:-kn3lmgmt}"
 OID=".1.3.6.1.4.1.318.1.1.12.2.3.1.1.2.1"
 raw=$(snmpget -v1 -c "$COMMUNITY" -On -Ov "$PDU_HOST" "$OID" 2>/dev/null \
    | grep -oE '[0-9]+$') || true
 if [ -z "${raw:-}" ]; then
    echo "output_amps unavailable"
    exit 0
 fi
 awk -v t="$raw" 'BEGIN { printf "output_amps %.1f\n", t/10 }'
@@ -1,146 +0,0 @@
 #!/bin/bash
 # Deploy lm-sensors extend (native sensors + TEMPer) + snmpd on a host [#341/#439]
 #
 # Run ON the target host via the remote.sh chokepoint (bash -s). Copy the
 # wrapper over first (vm-copy / prox-copy). Idempotent: safe to re-run.
 #
 # Native lm-sensors (CPU/board temps, fans) is installed on every target.
 # The TEMPer USB layer (venv + temperusb + udev rule) is installed ONLY when
 # a 0c45:7401 probe is attached.
 #
 # Usage (from repo root):
 #   VM_IP=stlpc-artroom VM_USER=root bash tests/remote.sh vm-copy dcinfra/sensors/temper/lmsensors-extend.sh /tmp/lmsensors-extend.sh
 #   VM_IP=stlpc-artroom VM_USER=root bash tests/remote.sh vm-file dcinfra/sensors/temper/deploy.sh
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-copy dcinfra/sensors/temper/lmsensors-extend.sh /tmp/lmsensors-extend.sh
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file dcinfra/sensors/temper/deploy.sh
 #
 # snmpd is a targeted exception: sensor hosts + tsys-* Proxmox hosts
 # (founder-directed 2026-08-27), not fleet-wide.
 set -euo pipefail
 COMMUNITY="${SNMP_COMMUNITY:-kn3lmgmt}"
 VENV_DIR="${VENV_DIR:-/opt/temper-venv}"
 WRAP_SRC="${WRAP_SRC:-/tmp/lmsensors-extend.sh}"
 WRAP_DST="/usr/local/bin/lmsensors-extend"
 WRAP_DST_OLD="/usr/local/bin/temper-lmsensors"
 SNMPD_CONF="/etc/snmp/snmpd.conf"
 UDEV_RULE="/etc/udev/rules.d/99-temper.rules"
 MARK_BEGIN="# lmsensors-extend managed block begin"
 MARK_END="# lmsensors-extend managed block end"
 OLD_MARK_BEGIN="# temper-lmsensors managed block begin"
 OLD_MARK_END="# temper-lmsensors managed block end"
 # Source allowlist (founder ruling 2026-08-27: bind Tailscale-only where
 # possible; HA is a Tailscale APP connector so its polls source from its LAN
 # IP and must be ACL'd explicitly — no broad subnet grants).
 #   192.168.3.176   tsys-librenms LAN (poller)
 #   100.86.204.77   tsys-librenms Tailscale
 #   192.168.3.12    pfv-bms / Home Assistant LAN (app connector source)
 #   100.67.108.125  pfv-bms Tailscale
 #   100.101.187.119 admin workstation Tailscale
 #   152.53.37.179 + 2a0a:4cc0:2000:686::a496:3bff:fe35:610a  Cloudron VPS
 #                  (future Uptime Kuma SNMP polling)
 SNMP_SOURCES="${SNMP_SOURCES:-192.168.3.176 192.168.3.252 192.168.3.253 100.70.181.72 100.71.171.20 100.86.204.77 192.168.3.12 100.67.108.125 100.101.187.119 152.53.37.179 2a0a:4cc0:2000:686:a496:3bff:fe35:610a}"
 log() { echo "[deploy] $*"; }
 has_probe() {
    command -v lsusb >/dev/null 2>&1 && lsusb | grep -qi '0c45:7401'
 }
 host_lan_ip() {
    # first 192.168.* address (mgmt/user LAN; excludes 127/10.100 storage/dockers)
    ip -4 addr show 2>/dev/null | grep -oP 'inet \K192\.168\.[0-9.]+' | head -1
 }
 host_ts_ip() {
    command -v tailscale >/dev/null 2>&1 && tailscale ip -4 2>/dev/null | head -1
 }
 [[ $EUID -eq 0 ]] || { echo "must run as root" >&2; exit 1; }
 [[ -f "$WRAP_SRC" ]] || { echo "wrapper not found at $WRAP_SRC (copy it first)" >&2; exit 1; }
 # --- native lm-sensors layer (all hosts) ---
 if ! dpkg -s lm-sensors >/dev/null 2>&1; then
    log "installing lm-sensors"
    DEBIAN_FRONTEND=noninteractive apt-get update -qq
    DEBIAN_FRONTEND=noninteractive apt-get install -y -qq lm-sensors
 fi
 if [[ -z "$(sensors 2>/dev/null)" ]]; then
    log "no sensor readings yet; trying coretemp/k10temp modules"
    modprobe coretemp 2>/dev/null || true
    modprobe k10temp 2>/dev/null || true
 fi
 log "native sensors: $(sensors 2>/dev/null | grep -c '°C') temp readings"
 # --- TEMPer USB layer (probe-conditional) ---
 if has_probe; then
    log "TEMPer probe present: installing venv layer"
    if ! dpkg -s python3-venv >/dev/null 2>&1; then
        log "installing python3-venv"
        DEBIAN_FRONTEND=noninteractive apt-get update -qq
        DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-venv
    fi
    if [[ ! -x "$VENV_DIR/bin/pip" ]]; then
        [[ -d "$VENV_DIR" ]] && { log "removing broken venv $VENV_DIR"; rm -rf "$VENV_DIR"; }
        log "creating venv $VENV_DIR"
        python3 -m venv "$VENV_DIR"
    fi
    log "ensuring temperusb in venv"
    "$VENV_DIR/bin/pip" install --quiet --disable-pip-version-check temperusb
    # TEMPer USB dongle must be readable by the unprivileged snmpd user
    # (extend runs as Debian-snmp; pyusb needs the raw usb device node).
    log "installing udev rule for 0c45:7401"
    cat >"$UDEV_RULE" <<'EOF'
 SUBSYSTEM=="usb", ATTRS{idVendor}=="0c45", ATTRS{idProduct}=="7401", MODE="0666"
 EOF
    udevadm control --reload-rules
    udevadm trigger
 else
    log "no TEMPer probe: skipping venv/udev layer"
 fi
 log "installing wrapper to $WRAP_DST"
 install -m 0755 "$WRAP_SRC" "$WRAP_DST"
 rm -f "$WRAP_DST_OLD"
 if ! dpkg -s snmpd >/dev/null 2>&1 || ! command -v snmpget >/dev/null 2>&1; then
    log "installing snmpd + snmp client"
    DEBIAN_FRONTEND=noninteractive apt-get update -qq
    DEBIAN_FRONTEND=noninteractive apt-get install -y -qq snmpd snmp
 fi
 log "configuring snmpd (community=$COMMUNITY, scoped sources)"
 LAN_IP="$(host_lan_ip)"
 TS_IP="$(host_ts_ip)"
 AGENT_ADDRS=""
 [[ -n "$LAN_IP" ]] && AGENT_ADDRS="udp:${LAN_IP}:161"
 [[ -n "$TS_IP" ]] && AGENT_ADDRS="${AGENT_ADDRS:+$AGENT_ADDRS,}udp:${TS_IP}:161"
 [[ -n "$AGENT_ADDRS" ]] || { echo "no LAN/Tailscale address found; refusing to bind 0.0.0.0" >&2; exit 1; }
 sed -i "/^$OLD_MARK_BEGIN\$/,/^$OLD_MARK_END\$/d" "$SNMPD_CONF"
 sed -i "/^$MARK_BEGIN\$/,/^$MARK_END\$/d" "$SNMPD_CONF"
 sed -i 's/^agentaddress/#agentaddress/' "$SNMPD_CONF"
 {
    echo "$MARK_BEGIN"
    echo "agentaddress $AGENT_ADDRS"
    for src in $SNMP_SOURCES; do
        if [[ "$src" == *:* ]]; then echo "rocommunity6 $COMMUNITY $src"; else echo "rocommunity $COMMUNITY $src"; fi
    done
    echo "extend lmsensors $WRAP_DST"
    echo "extend lmsensors_n $WRAP_DST -n"
    echo "$MARK_END"
 } >>"$SNMPD_CONF"
 log "agentaddress: $AGENT_ADDRS"
 log "restarting snmpd"
 systemctl enable --now snmpd >/dev/null
 systemctl restart snmpd
 log "wrapper output:"
 "$WRAP_DST"
 log "snmp extend self-test (expect lm-sensors block below):"
 sleep 1
 snmpget -v2c -c "$COMMUNITY" -Ovq localhost \
    .1.3.6.1.4.1.8072.1.3.2.3.1.2.9.108.109.115.101.110.115.111.114.115 2>/dev/null || \
    log "WARN: snmpget self-test failed (check snmpd logs)"
 log "done"
@@ -1,196 +0,0 @@
 #!/bin/bash
 # lm-sensors extend for snmpd: native sensors + TEMPer USB block [#341/#439]
 #
 # Deployed to /usr/local/bin/lmsensors-extend on sensor hosts by deploy.sh.
 # Output = lm-sensors format, consumed by Home Assistant's SNMP integration
 # (nsExtendOutputFull."lmsensors") and, if enabled later, LibreNMS's
 # lm-sensors app.
 #
 # Layers (each optional, at least one required):
 #   1. native /usr/bin/sensors output (CPU/board temps, fans) — lm-sensors pkg
 #   2. TEMPer USB probe (temperusb in venv /opt/temper-venv) — probe-optional
 #      (venv-on-host is founder-approved for sensor hosts, incl. Proxmox)
 set -euo pipefail
 VENV_PY="${VENV_PY:-/opt/temper-venv/bin/python}"
 NATIVE_SENSORS_BIN="${NATIVE_SENSORS_BIN:-/usr/bin/sensors}"
 SYSFS_THERMAL_ROOT="${SYSFS_THERMAL_ROOT:-/sys/class/thermal}"
 CHIP_NAME="${CHIP_NAME:-temper-usb-1}"
 # Render lm-sensors format block from internal/external Celsius values.
 # Empty/absent external renders internal only. Non-numeric input is rejected.
 format_block() {
    local internal="${1:-}" external="${2:-}"
    local num_re='^-?[0-9]+(\.[0-9]+)?$'
    [[ "$internal" =~ $num_re ]] || {
        echo "error: internal temp not numeric: '$internal'" >&2
        return 1
    }
    local signed
    printf '%s\nAdapter: USB adapter\n' "$CHIP_NAME"
    signed="${internal#+}"; [[ "$signed" == -* ]] || signed="+$signed"
    printf 'Internal:  %s°C\n' "$signed"
    if [[ -n "$external" && "$external" =~ $num_re ]]; then
        signed="${external#+}"; [[ "$signed" == -* ]] || signed="+$signed"
        printf 'External:  %s°C\n' "$signed"
    fi
 }
 # True when a TEMPer USB dongle (0c45:7401) is attached and the venv exists.
 # Numeric mode: print ONE Celsius float for snmpd extend lmsensors_n [#618].
 # Preference: TEMPer external > TEMPer internal > first native/sysfs temp
 # reading. Consumed by HA's SNMP config-entry integration (device_class
 # temperature; HA renders C in the instance display unit).
 numeric_pick() {
    local internal="${1:-}" external="${2:-}" native="${3:-}" pick
    local num_re='^-?[0-9]+(\.[0-9]+)?$'
    for pick in "$external" "$internal"; do
        pick="${pick#+}"
        [[ "$pick" =~ $num_re ]] && { printf '%s\n' "$pick"; return 0; }
    done
    if [[ -n "$native" ]]; then
        pick="$(grep -oE "\-?[0-9]+(\.[0-9]+)?°C" <<<"$native" | head -1)"; pick="${pick%°C}"
        [[ "$pick" =~ $num_re ]] && { printf '%s\n' "$pick"; return 0; }
    fi
    return 1
 }
 numeric_main() {
    local internal external line
    if has_temper_probe; then
        if read -r internal external <<<"$(read_temper 2>/dev/null)"; then
            numeric_pick "${internal:-}" "${external:-}" "" && return 0
        fi
    fi
    line="$(run_native_sensors | grep -m1 '°C')"
    [[ -n "$line" ]] && numeric_pick "" "" "$line" && return 0
    line="$(read_sysfs_thermal 2>/dev/null | head -1 | awk '{printf "+%.1f°C", $2/1000}')"
    [[ -n "$line" ]] && numeric_pick "" "" "$line" && return 0
    echo "error: numeric mode found no sensor source" >&2
    return 1
 }
 has_temper_probe() {
    command -v lsusb >/dev/null 2>&1 || return 1
    [[ -x "$VENV_PY" ]] || return 1
    lsusb | grep -qi '0c45:7401'
 }
 # Read the TEMPer probe via temperusb; prints "<internal> <external>" in C.
 # Sensor 0 = internal, sensor 1 = external probe (if attached).
 read_temper() {
    "$VENV_PY" - <<'PY'
 from temperusb import TemperHandler
 internal = external = ""
 for device in TemperHandler().get_devices():
    try:
        temps = device.get_temperatures(sensors=[0, 1])
    except (ValueError, KeyError):
        temps = device.get_temperatures()
    for index, slot in ((0, "internal"), (1, "external")):
        reading = temps.get(index)
        if reading and reading.get("temperature_c") is not None:
            value = f"{reading['temperature_c']:.1f}"
            if slot == "internal":
                internal = value
            else:
                external = value
 print(f"{internal} {external}".strip())
 PY
 }
 run_native_sensors() {
    [[ -x "$NATIVE_SENSORS_BIN" ]] || return 1
    "$NATIVE_SENSORS_BIN" 2>/dev/null
 }
 # SBC fallback (Raspberry Pi et al.): lm-sensors often reports nothing; SoC
 # temperatures live in /sys/class/thermal/thermal_zone*/{type,temp} (mC).
 read_sysfs_thermal() {
    [[ -d "$SYSFS_THERMAL_ROOT" ]] || return 1
    local z type mc found=0
    for z in "$SYSFS_THERMAL_ROOT"/thermal_zone[0-9]*; do
        [[ -r "$z/type" && -r "$z/temp" ]] || continue
        type="$(<"$z/type")" mc="$(<"$z/temp")"
        [[ "$mc" =~ ^-?[0-9]+$ ]] || continue
        printf '%s %s\n' "$type" "$mc"
        found=1
    done
    ((found)) || return 1
 }
 # Render lm-sensors format block from read_sysfs_thermal lines "type millicelsius".
 format_sysfs_block() {
    local input="${1:-}" type mc signed n=0
    [[ -n "$input" ]] || return 1
    printf 'soc-thermal-virtual-0\nAdapter: Virtual device\n'
    while read -r type mc; do
        ((++n))
        signed="$(awk -v mc="$mc" 'BEGIN { printf "%.1f", mc / 1000 }')"
        signed="${signed#+}"
        [[ "$signed" == -* ]] || signed="+$signed"
        printf 'temp%d:        %s°C\n' "$n" "$signed"
    done <<<"$input"
 }
 # Labeled per-zone temps (e.g. Jetson GPU-therm) [#736]. Hosts opt in by
 # listing zone types (one per line) in /etc/snmp/lmsensors-zone-types; no
 # snmpd.conf change or restart needed (extend re-execs per poll). Each type
 # renders its own chip block so HA templates can anchor on the chip name.
 ZONE_TYPES_FILE="${ZONE_TYPES_FILE:-/etc/snmp/lmsensors-zone-types}"
 format_labeled_zones() {
    local z type mc want signed
    [[ -r "$ZONE_TYPES_FILE" ]] || return 1
    while IFS= read -r want; do
        want="${want%%#*}"; want="${want//[[:space:]]/}"
        [[ -n "$want" ]] || continue
        for z in "$SYSFS_THERMAL_ROOT"/thermal_zone[0-9]*; do
            [[ -r "$z/type" && -r "$z/temp" ]] || continue
            [[ "$(<"$z/type")" == "$want" ]] || continue
            mc="$(<"$z/temp")"
            [[ "$mc" =~ ^-?[0-9]+$ ]] || continue
            signed="$(awk -v mc="$mc" 'BEGIN { printf "%.1f", mc / 1000 }')"
            signed="${signed#+}"
            [[ "$signed" == -* ]] || signed="+$signed"
            printf 'zone-%s-virtual-0\nAdapter: Virtual device\ntemp1:        %s°C\n\n' "$want" "$signed"
        done
    done <"$ZONE_TYPES_FILE"
    return 0
 }
 main() {
    local out="" block internal external native="" sysinput
    if native="$(run_native_sensors)" && [[ -n "$native" ]]; then
        out+="$native"$'\n'
    else
        native=""
    fi
    if has_temper_probe; then
        if block="$(read_temper 2>/dev/null)" \
           && read -r internal external <<<"$block" \
           && block="$(format_block "${internal:-}" "${external:-}")"; then
            out+="$block"$'\n'
        else
            echo "warn: TEMPer probe present but read failed" >&2
        fi
    fi
    if [[ -z "$native" ]] \
       && sysinput="$(read_sysfs_thermal 2>/dev/null)" \
       && block="$(format_sysfs_block "$sysinput")"; then
        out+="$block"$'\n'
    fi
    if block="$(format_labeled_zones)" && [[ -n "$block" ]]; then
        out+="$block"$'\n'
    fi
    [[ -n "$out" ]] || {
        echo "error: no sensor sources available (lm-sensors pkg? TEMPer venv? sysfs thermal?)" >&2
        return 1
    }
    printf '%s' "$out"
 }
 if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
    if [[ "${1:-}" == "-n" ]]; then numeric_main; else main; fi
 fi
@@ -1,10 +0,0 @@
 # dcinfra/ups/README.md
 > **Documentation moved to Discourse — the canonical source of truth.**
 >
 > **UPS management (NUT) for APC Smart-UPS C 1500**
 >
 > **Read it here:** https://community.turnsys.com/t/301
 >
 > *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
 > Do not update content here — edit the Discourse wiki topic instead.*
@@ -1,86 +0,0 @@
 #!/usr/bin/bash
 #
 # ups/discover.sh — probe USB UPS units and NUT state on the local host
 #
 # Read-only. Prints everything needed to configure NUT. No changes made.
 #
 # Usage (run ON the target host via remote.sh):
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/discover.sh
 #
 set -uo pipefail
 echo "======================================================"
 echo "  UPS / NUT Discovery on $(hostname)"
 echo "======================================================"
 # --- 1. USB UPS devices ------------------------------------------------------
 echo ""
 echo "--- [1/5] USB UPS devices (lsusb) ---"
 lsusb 2>/dev/null | grep -iE "UPS|American Power|Tripp|APC" || echo "  (no UPS devices found in lsusb)"
 echo ""
 echo "--- [2/5] UPS detail (vendor/product/serial/model) ---"
 # Common UPS vendor IDs: 051d (APC), 09ae (Tripp Lite), 0463 (Eaton),
 # 06da (MGE), 0764 (Cyber Power)
 for vid in 051d 09ae 0463 06da 0764; do
  while read -r bus dev pid; do
    [ -n "$bus" ] || continue
    echo "  --- $bus:$dev ($vid:$pid) ---"
    lsusb -v -s "${bus}:${dev}" 2>/dev/null \
      | grep -iE "iManufacturer|iProduct|iSerial|bcdDevice" \
      | sed 's/^/    /'
  done < <(lsusb 2>/dev/null | awk -v v="$vid" '$0~v{split($2,a,":"); split($4,b,":"); print a[1], b[1], $6}')
 done
 # --- 2. sysfs paths (for udev rules) -----------------------------------------
 echo ""
 echo "--- [3/5] sysfs device paths + serials ---"
 for d in /sys/bus/usb/devices/*; do
  man=$(cat "$d/manufacturer" 2>/dev/null)
  prod=$(cat "$d/product" 2>/dev/null)
  ser=$(cat "$d/serial" 2>/dev/null)
  vid=$(cat "$d/idVendor" 2>/dev/null)
  pid=$(cat "$d/idProduct" 2>/dev/null)
  if echo "$man $prod" | grep -qiE "apc|tripp|power conversion|ups|eaton|mge|cyber power"; then
    # Resolve stable ID_PATH for udev pinning
    path=$(udevadm info -q property -p "$d" 2>/dev/null | awk -F= '/^ID_PATH=/{print $2}')
    echo "  $d"
    echo "    vendor=$vid product=$pid"
    echo "    manufacturer=$man"
    echo "    product=$prod"
    echo "    serial=$ser"
    echo "    ID_PATH=$path"
  fi
 done
 # --- 3. HID device nodes -----------------------------------------------------
 echo ""
 echo "--- [4/5] HID device nodes ---"
 ls -la /dev/hidraw* /dev/usb/hiddev* 2>/dev/null || echo "  (no hidraw/hiddev nodes)"
 # --- 4. NUT install state ----------------------------------------------------
 echo ""
 echo "--- [5/5] NUT install + service state ---"
 if dpkg -l nut-server nut-client 2>/dev/null | grep -q '^ii'; then
  echo "  NUT installed:"
  dpkg -l nut-server nut-client 2>/dev/null | awk '/^ii/{print "    "$2" "$3}'
 else
  echo "  NUT not installed (apt: nut-server nut-client)"
 fi
 echo ""
 echo "  Services:"
 for svc in nut-driver nut-server nut-monitor; do
  printf "    %-14s " "$svc:"
  systemctl is-active "$svc" 2>/dev/null || true
 done
 echo ""
 echo "  Existing config:"
 # shellcheck disable=SC2012 # ls -la is intentional for human-readable listing
 ls -la /etc/nut/ 2>/dev/null | sed 's/^/    /' || echo "    (no /etc/nut)"
 echo ""
 echo "======================================================"
 echo "  Discovery complete."
 echo "======================================================"
@@ -1,134 +0,0 @@
 #!/usr/bin/env python3
 """
 ha-nut-setup.py — Add the Home Assistant NUT integration via REST config-flow API.
 Stdlib-only (no pip). Idempotent: skips if a NUT config entry already exists.
 Env:
  HA_HOST   (default pfv-bms.knel.net)
  HA_PORT   (default 8123)
  HA_TOKEN  (long-lived access token)
  NUT_HOST  (default 100.121.189.98)
  NUT_PORT  (default 3493)
  NUT_USER  (default homeassistant)
  NUT_PASS  (required)
  NUT_UPS   (default apc-smartups-c1500)
 """
 import os, json, sys, time, urllib.request, urllib.error
 HA_HOST  = os.environ.get("HA_HOST", "pfv-bms.knel.net")
 HA_PORT  = int(os.environ.get("HA_PORT", "8123"))
 TOKEN    = os.environ["HA_TOKEN"]
 NUT_HOST = os.environ.get("NUT_HOST", "192.168.3.11")
 NUT_PORT = int(os.environ.get("NUT_PORT", "3493"))
 NUT_USER = os.environ.get("NUT_USER", "homeassistant")
 NUT_PASS = os.environ["NUT_PASS"]
 NUT_UPS  = os.environ.get("NUT_UPS", "apc-smartups-c1500")
 BASE     = f"http://{HA_HOST}:{HA_PORT}"
 def api(method, path, data=None):
    body = json.dumps(data).encode() if data else None
    req = urllib.request.Request(
        f"{BASE}/api{path}", data=body, method=method,
        headers={"Authorization": f"Bearer {TOKEN}",
                 "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            return json.loads(r.read())
    except urllib.error.HTTPError as e:
        raw = e.read().decode()
        try:
            return json.loads(raw)
        except Exception:
            return {"_http_error": e.code, "_raw": raw[:300]}
    except Exception as e:
        return {"_error": str(e)}
 # ── verify token ──
 cfg = api("GET", "/config")
 if "_http_error" in cfg or "_error" in cfg:
    print(f"Cannot reach HA or token invalid: {cfg}"); sys.exit(1)
 print(f"HA {cfg.get('version')} — token valid")
 # ── check existing entries (idempotent) ──
 entries = api("GET", "/config/config_entries/entry")
 existing = [e for e in entries if e.get("domain") == "nut"]
 if existing:
    for e in existing:
        print(f"NUT already configured: {e.get('title')} "
              f"(data={json.dumps(e.get('data', {}))})")
    print("Skipping — delete it in HA UI first if you want to re-run.")
    sys.exit(0)
 print("No existing NUT entry. Starting config flow.")
 # ── initiate flow ──
 flow = api("POST", "/config/config_entries/flow", {"handler": "nut"})
 if "flow_id" not in flow:
    print(f"Flow init failed: {json.dumps(flow)}"); sys.exit(1)
 fid = flow["flow_id"]
 print(f"Flow started: step={flow.get('step_id')} "
      f"fields={[f.get('name') for f in flow.get('data_schema', [])]}")
 # ── submit connection details ──
 creds = {"host": NUT_HOST, "port": NUT_PORT,
         "username": NUT_USER, "password": NUT_PASS}
 flow = api("POST", f"/config/config_entries/flow/{fid}", creds)
 if flow.get("errors"):
    print(f"Validation errors: {flow['errors']}"); sys.exit(1)
 print(f"After submit: type={flow.get('type')} step={flow.get('step_id')}")
 # ── handle follow-up steps (UPS selection etc.) ──
 while flow.get("type") == "form":
    step = flow.get("step_id", "?")
    schema = flow.get("data_schema", [])
    print(f"Step '{step}': fields={[f.get('name') for f in schema]}")
    for f in schema:
        opts = f.get("options") or f.get("values")
        if opts:
            print(f"  {f.get('name')} options: {opts}")
    submission = {}
    for f in schema:
        nm = f.get("name")
        ftype = f.get("type", "")
        if ftype == "multi_select":
            opts = f.get("options", [])
            vals = [o[0] if isinstance(o, list) else o for o in opts]
            submission[nm] = [NUT_UPS] if NUT_UPS in vals else vals[:1]
        elif nm in creds:
            submission[nm] = creds[nm]
        elif "default" in f:
            submission[nm] = f["default"]
        elif ftype == "select":
            opts = f.get("options", [])
            vals = [o[0] if isinstance(o, list) else o for o in opts]
            submission[nm] = NUT_UPS if NUT_UPS in vals else (vals[0] if vals else "")
    fid = flow.get("flow_id", fid)
    flow = api("POST", f"/config/config_entries/flow/{fid}", submission)
    if flow.get("errors"):
        print(f"Validation errors: {flow['errors']}"); sys.exit(1)
    print(f"  -> type={flow.get('type')} step={flow.get('step_id')}")
 # ── result ──
 if flow.get("type") == "create_entry":
    print(f"\nNUT integration created: {flow.get('title')}")
 elif flow.get("type") == "abort":
    print(f"\nFlow aborted: {flow.get('reason')}"); sys.exit(1)
 else:
    print(f"\nFinal state: {flow.get('type')} — {json.dumps(flow)[:200]}")
 # ── verify sensors ──
 print("\nWaiting 10s for entities ...")
 time.sleep(10)
 states = api("GET", "/states")
 ups = [s for s in states
       if "apc_smartups" in s["entity_id"].lower()
       or "sensor.ups_" in s["entity_id"].lower()]
 if ups:
    print(f"Found {len(ups)} UPS sensors:")
    for e in sorted(ups, key=lambda x: x["entity_id"]):
        st = e.get("state", "?")
        unit = e.get("attributes", {}).get("unit_of_measurement", "")
        name = e.get("attributes", {}).get("friendly_name", "")
        print(f"  {e['entity_id']:55s} {st:>8} {unit:4s} {name}")
 else:
    print("No UPS sensors yet (may still be initialising — check HA UI).")
@@ -1,65 +0,0 @@
 #!/usr/bin/env bash
 #
 # setup-ha-nut.sh — Add the Home Assistant NUT integration via REST API.
 #
 # Idempotent: skips if a NUT entry already exists. Reads secrets from
 # ~/.config/pfvcluster/ (ha-token, nut-password) or env vars.
 #
 # Usage:
 #   bash ups/setup-ha-nut.sh
 #
 # Env overrides:
 #   HA_TOKEN     HA long-lived access token
 #   NUT_PASS     NUT upsd password for the homeassistant user
 #   HA_HOST      HA host (default pfv-bms.knel.net)
 #   NUT_HOST     upsd host (default 192.168.3.11 — LAN, see README)
 #   NUT_PORT     upsd port (default 3493)
 #   NUT_USER     upsd user (default homeassistant)
 #   NUT_UPS      UPS name (default apc-smartups-c1500)
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 CONF_DIR="${PFV_CONF_DIR:-$HOME/.config/pfvcluster}"
 # --- HA token ---
 HA_TOKEN="${HA_TOKEN:-}"
 if [[ -z "$HA_TOKEN" ]]; then
  TOKEN_FILE="$CONF_DIR/ha-token"
  if [[ -f "$TOKEN_FILE" ]]; then
    HA_TOKEN="$(head -1 "$TOKEN_FILE" | tr -d '[:space:]')"
  else
    echo "error: no HA token. Set \$HA_TOKEN or create $TOKEN_FILE" >&2
    echo "  (HA → Profile → Long-Lived Access Tokens → Create Token)" >&2
    exit 1
  fi
 fi
 # --- NUT password ---
 NUT_PASS="${NUT_PASS:-}"
 if [[ -z "$NUT_PASS" ]]; then
  PASS_FILE="$CONF_DIR/nut-password"
  if [[ -f "$PASS_FILE" ]]; then
    NUT_PASS="$(head -1 "$PASS_FILE" | tr -d '[:space:]')"
  else
    echo "error: no NUT password. Set \$NUT_PASS or create $PASS_FILE" >&2
    echo "  (value is in /etc/nut/upsd.users on the NUT host)" >&2
    exit 1
  fi
 fi
 # --- connection params (override for your own kit) ---
 export HA_TOKEN
 export NUT_PASS
 export HA_HOST="${HA_HOST:-pfv-bms.knel.net}"
 export HA_PORT="${HA_PORT:-8123}"
 export NUT_HOST="${NUT_HOST:-192.168.3.11}"
 export NUT_PORT="${NUT_PORT:-3493}"
 export NUT_USER="${NUT_USER:-homeassistant}"
 export NUT_UPS="${NUT_UPS:-apc-smartups-c1500}"
 echo "HA:   ${HA_HOST}:${HA_PORT}"
 echo "NUT:  ${NUT_USER}@${NUT_HOST}:${NUT_PORT} (UPS: ${NUT_UPS})"
 echo ""
 exec python3 "$SCRIPT_DIR/ha-nut-setup.py"
@@ -1,298 +0,0 @@
 #!/usr/bin/bash
 #
 # ups/setup.sh — idempotent Network UPS Tools (NUT) setup on pfv-tsys1
 #
 # Installs NUT, configures two USB HID UPS units (APC + Tripp Lite) pinned by
 # USB serial, runs upsd as a network server for Home Assistant polling, and
 # runs upsmon locally so the hypervisor can shut down gracefully on battery.
 #
 # Designed to run ON the target host (pfv-tsys1) as root, idempotent.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/setup.sh
 #
 # Overrides (defaults suit pfv-tsys1):
 #   APC_SERIAL         APC UPS USB serial            (default AS1213210423)
 #  APC_VID/APC_PID     APC vendor/product ID          (default 051d / 0003)
 #   APC_NAME           NUT section name for APC       (default apc-smartups-c1500)
 #  TRIPP_SERIAL        Tripp Lite UPS USB serial      (default 2352CVLSM871900694)
 #  TRIPP_VID/TRIPP_PID Tripp Lite vendor/product      (default 09ae / 3016)
 #  TRIPP_NAME          NUT section name for Tripp     (default tripp-lite-ups)
 #  TRIPP_SUBDRIVER     Forced HID subdriver for Tripp (default "TrippLite HID 0.85")
 #  TRIPP_ENABLED       Set to 0 to disable Tripp Lite (default 1)
 #  NUT_LISTEN_IPS      space-separated upsd LISTEN IPs (default: auto Tailscale + 127.0.0.1)
 #  HA_USER             upsd username for HA           (default homeassistant)
 #  HA_PASSWORD         upsd password for HA           (default: reuse or generate)
 #  MON_USER            upsd username for local upsmon (default monuser)
 #  MON_PASSWORD        upsd password for upsmon       (default: reuse or generate)
 #
 set -euo pipefail
 # --- Config (overridable via env) ---
 APC_SERIAL="${APC_SERIAL:-AS1213210423}"
 APC_VID="${APC_VID:-051d}"
 APC_PID="${APC_PID:-0003}"
 APC_NAME="${APC_NAME:-apc-smartups-c1500}"
 TRIPP_SERIAL="${TRIPP_SERIAL:-2352CVLSM871900694}"
 TRIPP_VID="${TRIPP_VID:-09ae}"
 TRIPP_PID="${TRIPP_PID:-3016}"
 TRIPP_NAME="${TRIPP_NAME:-tripp-lite-ups}"
 TRIPP_SUBDRIVER="${TRIPP_SUBDRIVER:-TrippLite HID 0.85}"
 TRIPP_ENABLED="${TRIPP_ENABLED:-1}"
 HA_USER="${HA_USER:-homeassistant}"
 MON_USER="${MON_USER:-monuser}"
 NUT_PORT="${NUT_PORT:-3493}"
 UDEV_RULE="/etc/udev/rules.d/99-nut-ups.rules"
 UPS_CONF="/etc/nut/ups.conf"
 UPSD_CONF="/etc/nut/upsd.conf"
 UPSD_USERS="/etc/nut/upsd.users"
 UPS_CONF_MON="/etc/nut/upsmon.conf"
 NUT_CONF="/etc/nut/nut.conf"
 # --- Helpers ---
 gen_pw() { head -c 24 /dev/urandom | base64 | tr -d '/+=' | cut -c1-20; }
 # Reuse existing passwords if config already present (idempotent re-runs)
 extract_pw() {  # $1=user
  if [ -f "$UPSD_USERS" ]; then
    awk -v u="[$1]" '
      $0==u {inblk=1; next}
      /^\[/ {inblk=0}
      inblk && $1=="password" {gsub(/"/,"",$3); print $3; exit}
    ' "$UPSD_USERS" 2>/dev/null
  fi
 }
 echo "============================================"
 echo "  NUT UPS Setup on $(hostname)"
 echo "  APC:        $APC_NAME  ($APC_VID:$APC_PID serial $APC_SERIAL)"
 if [ "$TRIPP_ENABLED" = "1" ]; then
  echo "  Tripp Lite: $TRIPP_NAME ($TRIPP_VID:$TRIPP_PID serial $TRIPP_SERIAL)"
 else
  echo "  Tripp Lite: DISABLED (TRIPP_ENABLED=0)"
 fi
 echo "============================================"
 # --- 0. Resolve / generate passwords (idempotent) ---
 MON_PASSWORD="${MON_PASSWORD:-$(extract_pw "$MON_USER")}"
 HA_PASSWORD="${HA_PASSWORD:-$(extract_pw "$HA_USER")}"
 [ -n "$MON_PASSWORD" ] || MON_PASSWORD="$(gen_pw)"
 [ -n "$HA_PASSWORD" ] || HA_PASSWORD="$(gen_pw)"
 # --- 0b. Auto-detect listen IPs for upsd ---
 # Tailscale IP: tailnet clients (workstation, etc.)
 # LAN IP: HAOS VMs where Tailscale runs as an isolated add-on (HA container
 #         cannot route to Tailscale IPs, so the shared-LAN bridge is required)
 if [ -z "${NUT_LISTEN_IPS:-}" ]; then
  NUT_LISTEN_IPS="127.0.0.1"
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -n "$TS_IP" ]; then
    NUT_LISTEN_IPS="${NUT_LISTEN_IPS} ${TS_IP}"
  else
    echo "  WARNING: No Tailscale IP detected."
  fi
  if [ "${NUT_INCLUDE_LAN:-1}" = "1" ]; then
    LAN_IP=$(ip -4 addr show vmbr0 2>/dev/null | awk '/scope global/{print $2}' | cut -d/ -f1 | head -1)
    if [ -z "$LAN_IP" ]; then
      LAN_IP=$(hostname -I 2>/dev/null | awk '{print $1}')
    fi
    if [ -n "$LAN_IP" ]; then
      NUT_LISTEN_IPS="${NUT_LISTEN_IPS} ${LAN_IP}"
    fi
  fi
 fi
 echo "  upsd LISTEN IPs: ${NUT_LISTEN_IPS:-<none>}"
 # --- 1. Install NUT ---
 echo ""
 echo "--- [1/8] Installing NUT (nut-server, nut-client) ---"
 if dpkg -l nut-server 2>/dev/null | grep -q '^ii'; then
  echo "  NUT already installed: $(dpkg -l nut-server | awk '/^ii/{print $3}')"
 else
  apt-get update -qq && apt-get install -y -qq nut-server nut-client
 fi
 mkdir -p /etc/nut
 # --- 2. udev rules: grant nut group access to BOTH raw USB + hidraw devices ---
 # CRITICAL: usbhid-ups opens /dev/bus/usb/BBB/DDD (raw USB), not /dev/hidraw.
 # The driver drops to the nut user via setuid(), so the nut group needs write
 # access to the raw USB device files. Matching on subsystem=="usb" by VID:PID
 # is required because ATTRS{serial} does not reliably traverse for usb devices.
 echo ""
 echo "--- [2/8] Writing udev rules (raw USB + hidraw, group nut) ---"
 {
  echo "# Stable permissions for NUT USB HID UPS units"
  echo "# Generated by ups/setup.sh — grants the 'nut' group access to both"
  echo "# the raw USB device files (/dev/bus/usb) and hidraw devices."
  echo "# Match BOTH subsystems: the usbhid-ups driver opens the raw USB device"
  echo "# after dropping to the nut user via setuid()."
  echo ""
  echo "# APC Smart-UPS C 1500 ($APC_VID:$APC_PID)"
  echo "SUBSYSTEM==\"usb\",    ATTR{idVendor}==\"$APC_VID\", ATTR{idProduct}==\"$APC_PID\", GROUP=\"nut\", MODE=\"0664\""
  echo "SUBSYSTEM==\"hidraw\", ATTRS{serial}==\"$APC_SERIAL\",                   GROUP=\"nut\", MODE=\"0660\""
  echo ""
  echo "# Tripp Lite UPS ($TRIPP_VID:$TRIPP_PID)"
  echo "SUBSYSTEM==\"usb\",    ATTR{idVendor}==\"$TRIPP_VID\", ATTR{idProduct}==\"$TRIPP_PID\", GROUP=\"nut\", MODE=\"0664\""
  echo "SUBSYSTEM==\"hidraw\", ATTRS{serial}==\"$TRIPP_SERIAL\",                   GROUP=\"nut\", MODE=\"0660\""
 } > "$UDEV_RULE"
 echo "  Written: $UDEV_RULE"
 udevadm control --reload-rules 2>/dev/null || true
 udevadm trigger --subsystem-match=usb 2>/dev/null || true
 udevadm trigger --subsystem-match=hidraw 2>/dev/null || true
 sleep 1
 # --- 3. ups.conf ---
 echo ""
 echo "--- [3/8] Writing ups.conf ---"
 {
  echo "# NUT UPS devices — generated by ups/setup.sh on $(date)"
  echo ""
  echo "maxretry = 3"
  echo ""
  echo "[${APC_NAME}]"
  echo "    driver = usbhid-ups"
  echo "    port = auto"
  echo "    vendorid = ${APC_VID}"
  echo "    productid = ${APC_PID}"
  echo "    serial = ${APC_SERIAL}"
  echo "    desc = \"APC Smart-UPS C 1500\""
  if [ "$TRIPP_ENABLED" = "1" ]; then
    echo ""
    echo "[${TRIPP_NAME}]"
    echo "    driver = usbhid-ups"
    echo "    port = auto"
    echo "    vendorid = ${TRIPP_VID}"
    echo "    productid = ${TRIPP_PID}"
    echo "    serial = ${TRIPP_SERIAL}"
    echo "    subdriver = \"${TRIPP_SUBDRIVER}\""
    echo "    desc = \"Tripp Lite UPS\""
  fi
 } > "$UPS_CONF"
 echo "  Written: $UPS_CONF"
 # --- 4. upsd.conf: network server (localhost + Tailscale for HA) ---
 echo ""
 echo "--- [4/8] Writing upsd.conf ---"
 {
  echo "# NUT upsd — generated by ups/setup.sh on $(date)"
  for ip in $NUT_LISTEN_IPS; do
    echo "LISTEN ${ip} ${NUT_PORT}"
  done
  echo "MAXAGE 25"
 } > "$UPSD_CONF"
 echo "  Written: $UPSD_CONF (LISTEN: $(echo "$NUT_LISTEN_IPS" | tr '\n' ' '))"
 # --- 5. upsd.users: monuser (master) + homeassistant (read-only monitor) ---
 echo ""
 echo "--- [5/8] Writing upsd.users ---"
 cat > "$UPSD_USERS" <<USERS
 # NUT upsd users — generated by ups/setup.sh on $(date)
 # monuser       : local upsmon (master) — graceful hypervisor shutdown
 # ${HA_USER} : Home Assistant NUT integration (read-only polling)
 [${MON_USER}]
    password = "${MON_PASSWORD}"
    upsmon master
 [${HA_USER}]
    password = "${HA_PASSWORD}"
    upsmon slave
 USERS
 echo "  Written: $UPSD_USERS"
 # --- 6. upsmon.conf + nut.conf ---
 echo ""
 echo "--- [6/8] Writing upsmon.conf + nut.conf ---"
 {
  echo "# NUT upsmon (local monitor) — generated by ups/setup.sh on $(date)"
  echo "# Monitors the APC UPS as master. On battery-low, shuts the host down."
  echo ""
  echo "MONITOR ${APC_NAME}@localhost 1 ${MON_USER} \"${MON_PASSWORD}\" master"
  if [ "$TRIPP_ENABLED" = "1" ]; then
    echo "MONITOR ${TRIPP_NAME}@localhost 1 ${MON_USER} \"${MON_PASSWORD}\" master"
  fi
  echo ""
  echo "SHUTDOWNCMD \"/sbin/shutdown -h now\""
  echo "POWERDOWNFLAG /etc/killpower"
  echo "NOTIFYFLAG ONLINE  SYSLOG+WALL"
  echo "NOTIFYFLAG ONBATT  SYSLOG+WALL"
  echo "NOTIFYFLAG LOWBATT SYSLOG+WALL"
  echo "POLLFREQ 15"
  echo "POLLFREQALERT 5"
  echo "HOSTSYNC 15"
 } > "$UPS_CONF_MON"
 cat > "$NUT_CONF" <<NUTCONF
 # NUT mode — generated by ups/setup.sh on $(date)
 # netserver = this host runs drivers + upsd; serves UPS data to clients (HA).
 MODE=netserver
 NUTCONF
 echo "  Written: $UPS_CONF_MON + $NUT_CONF"
 # --- 6b. Fix ownership/permissions (Debian: nut group must read config) ---
 chown root:nut "$UPS_CONF" "$UPSD_CONF" "$UPSD_USERS" "$UPS_CONF_MON" 2>/dev/null || true
 chmod 640 "$UPS_CONF" "$UPSD_CONF" "$UPSD_USERS" "$UPS_CONF_MON" 2>/dev/null || true
 chmod 644 "$NUT_CONF" 2>/dev/null || true
 # --- 7. Start services (Debian uses templated nut-driver@<name> units) ---
 echo ""
 echo "--- [7/8] Starting NUT services ---"
 # Re-read ups.conf to generate per-UPS driver instances
 systemctl restart nut-driver-enumerator 2>/dev/null || true
 sleep 2
 # Start per-UPS driver instances
 systemctl restart "nut-driver@${APC_NAME}" 2>/dev/null || true
 if [ "$TRIPP_ENABLED" = "1" ]; then
  systemctl restart "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
 else
  systemctl stop "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
  systemctl mask "nut-driver@${TRIPP_NAME}" 2>/dev/null || true
 fi
 sleep 3
 systemctl restart nut-server 2>/dev/null || true
 sleep 1
 systemctl restart nut-monitor 2>/dev/null || true
 echo ""
 echo "  Service status:"
 for svc in "nut-driver@${APC_NAME}" "nut-driver@${TRIPP_NAME}" nut-server nut-monitor; do
  if systemctl list-unit-files "$svc" >/dev/null 2>&1; then
    printf "    %-42s " "$svc"
    systemctl is-active "$svc" 2>/dev/null || echo "(unknown)"
  fi
 done
 # --- 8. Validate ---
 echo ""
 echo "--- [8/8] Validation ---"
 echo ""
 echo "  upsc — ${APC_NAME}:"
 upsc "${APC_NAME}@localhost" 2>&1 | head -25 || echo "    (APC UPS not responding yet)"
 if [ "$TRIPP_ENABLED" = "1" ]; then
  echo ""
  echo "  upsc — ${TRIPP_NAME}:"
  upsc "${TRIPP_NAME}@localhost" 2>&1 | head -25 || echo "    (Tripp Lite UPS not responding yet)"
 fi
 echo ""
 echo "============================================"
 echo "  Setup complete."
 echo ""
 echo "  Home Assistant NUT integration:"
 echo "    Host:     $(echo "$NUT_LISTEN_IPS" | awk '{print $2}')   (or any LISTEN IP above)"
 echo "    Port:     ${NUT_PORT}"
 echo "    Username: ${HA_USER}"
 echo "    Password: ${HA_PASSWORD}"
 if [ "$TRIPP_ENABLED" = "1" ]; then
  echo "    UPS names: ${APC_NAME}, ${TRIPP_NAME}"
 else
  echo "    UPS names: ${APC_NAME}"
 fi
 echo ""
 echo "  Save the HA password now — it is stored in ${UPSD_USERS}."
 echo "============================================"
@@ -1,59 +0,0 @@
 #!/usr/bin/bash
 #
 # ups/status.sh — query NUT UPS state + service health (read-only)
 #
 # Usage (run ON the target host via remote.sh):
 #   PROX_HOST=pfv-tsys1 bash tests/remote.sh prox-file ups/status.sh
 #
 # shellcheck disable=SC2012 # ss/awk field extraction is intentional
 set -uo pipefail
 APC_NAME="${APC_NAME:-apc-smartups-c1500}"
 TRIPP_NAME="${TRIPP_NAME:-tripp-lite-ups}"
 echo "======================================================"
 echo "  NUT UPS Status on $(hostname)"
 echo "======================================================"
 echo ""
 echo "--- Services ---"
 for svc in "nut-driver@${APC_NAME}" "nut-driver@${TRIPP_NAME}" nut-server nut-monitor; do
  if systemctl list-unit-files "$svc" >/dev/null 2>&1; then
    printf "  %-42s " "$svc"
    systemctl is-active "$svc" 2>/dev/null || echo "(unknown)"
  fi
 done
 for ups in "$APC_NAME" "$TRIPP_NAME"; do
  echo ""
  echo "--- ${ups} ---"
  if upsc "${ups}@localhost" >/tmp/.nutstatus.$$ 2>&1; then
    awk -v u="$ups" '
      BEGIN{printf "  %s\n", u}
      /^battery\.charge:/      {printf "    battery.charge:      %s\n", $3}
      /^battery\.runtime:/     {printf "    battery.runtime:     %ss (%.0f min)\n", $3, $3/60}
      /^battery\.voltage:/     {printf "    battery.voltage:     %s\n", $3}
      /^ups\.status:/          {printf "    ups.status:          %s\n", $3}
      /^ups\.load:/            {printf "    ups.load:            %s%%\n", $3}
      /^ups\.power:/           {printf "    ups.power:           %s\n", $3}
      /^ups\.realpower:/       {printf "    ups.realpower:       %s W\n", $3}
      /^input\.voltage:/       {printf "    input.voltage:       %s\n", $3}
      /^output\.voltage:/      {printf "    output.voltage:      %s\n", $3}
      /^ups\.model:/           {printf "    ups.model:           %s\n", $3}
      /^ups\.serial:/          {printf "    ups.serial:          %s\n", $3}
      /^device\.mfr:/          {printf "    device.mfr:          %s\n", $3}
    ' /tmp/.nutstatus.$$
    echo "    (full dump: upsc ${ups}@localhost)"
  else
    echo "  NOT RESPONDING:"
    sed 's/^/    /' /tmp/.nutstatus.$$
  fi
  rm -f /tmp/.nutstatus.$$
 done
 echo ""
 echo "--- upsd LISTEN sockets ---"
 ss -ltnp 2>/dev/null | grep -E "3493|nut" | sed 's/^/  /' || echo "  (upsd not listening on 3493)"
 echo ""
 echo "======================================================"
@@ -1,297 +0,0 @@
 #!/usr/bin/env bash
 # =============================================================================
 # audit-netboot.sh
 # -----------------------------------------------------------------------------
 # READ-ONLY audit of the reference node "pfv-netboot".
 #
 # Purpose: gather enough information to replicate its Pi-hole, Technitium DNS,
 # and NTP services onto pfv-netinfra-01 and pfv-netinfra-02.
 #
 # Guarantees:
 #   * NO writes, NO installs, NO service restarts, NO network changes.
 #   * Every command below is read-only (status, cat, ls, ss, ps, sqlite3 SELECT).
 #
 # Intended to be run as the "localuser" account on pfv-netboot:
 #
 #     ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh > netboot-audit.txt
 #
 # or, if sudo is needed for a few reads, the script will try `sudo -n` for
 # specific files that are normally root-readable only. It will NEVER use sudo
 # to write or modify anything.
 # =============================================================================
 set -u
 AUDIT_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 # Helper: print a clearly delimited section header.
 section() {
    printf '\n========================================================================\n'
    printf 'SECTION: %s\n' "$1"
    printf '========================================================================\n'
 }
 # Helper: read a file with sudo if needed, silently skip if missing.
 read_file() {
    local path="$1"
    if [ -r "$path" ]; then
        printf '--- %s (uid-readable) ---\n' "$path"
        cat "$path" 2>/dev/null
    elif sudo -n true 2>/dev/null; then
        if sudo -n test -e "$path"; then
            printf '--- %s (via sudo -n) ---\n' "$path"
            sudo -n cat "$path" 2>/dev/null
        else
            printf '--- %s : NOT FOUND ---\n' "$path"
        fi
    else
        printf '--- %s : NOT READABLE (no passwordless sudo) ---\n' "$path"
    fi
    printf '\n'
 }
 list_dir() {
    local path="$1"
    printf '--- ls -la %s ---\n' "$path"
    ls -la "$path" 2>/dev/null || sudo -n ls -la "$path" 2>/dev/null || printf '(cannot list %s)\n' "$path"
    printf '\n'
 }
 printf '##### AUDIT START %s #####\n' "$AUDIT_DATE"
 printf 'Audit host: %s\n' "$(hostname -f 2>/dev/null || hostname)"
 printf 'Audit user: %s\n' "$(id -un 2>/dev/null)"
 printf 'Script:     audit-netboot.sh (READ-ONLY)\n'
 # -----------------------------------------------------------------------------
 # 1. System basics
 # -----------------------------------------------------------------------------
 section "1. SYSTEM BASICS"
 echo "-- uname --"; uname -a
 echo "-- /etc/os-release --"; cat /etc/os-release 2>/dev/null
 echo "-- uptime --"; uptime
 echo "-- timezone --"; timedatectl 2>/dev/null || cat /etc/timezone 2>/dev/null || date
 echo "-- arch --"; dpkg --print-architecture 2>/dev/null || uname -m
 echo "-- memory --"; free -h 2>/dev/null
 echo "-- disk --"; df -h / 2>/dev/null
 echo "-- cpu count --"; nproc 2>/dev/null
 # -----------------------------------------------------------------------------
 # 2. Network configuration
 # -----------------------------------------------------------------------------
 section "2. NETWORK"
 echo "-- hostname --"; hostname; hostname -f 2>/dev/null; hostname -I 2>/dev/null
 echo "-- ip addr --"; ip -br addr 2>/dev/null; echo; ip addr 2>/dev/null
 echo "-- default route --"; ip route 2>/dev/null
 echo "-- /etc/resolv.conf --"; cat /etc/resolv.conf 2>/dev/null
 echo "-- /etc/hosts --"; cat /etc/hosts 2>/dev/null
 echo "-- listening TCP/UDP sockets --"
 ss -tlnup 2>/dev/null || sudo -n ss -tlnup 2>/dev/null
 echo "-- /etc/network/interfaces --"; cat /etc/network/interfaces 2>/dev/null
 echo "-- netplan --"; ls -la /etc/netplan/ 2>/dev/null; for f in /etc/netplan/*.yaml; do [ -e "$f" ] && { echo "--- $f ---"; cat "$f"; }; done 2>/dev/null
 echo "-- systemd-networkd --"; ls -la /etc/systemd/network/ 2>/dev/null; networkctl status 2>/dev/null | head -40
 # -----------------------------------------------------------------------------
 # 3. DNS / DHCP / NTP related systemd units
 # -----------------------------------------------------------------------------
 section "3. RELEVANT SYSTEMD UNITS"
 echo "-- all units matching dns|pihole|dnsmasq|technitium|ftl|ntp|chrony|timesync --"
 systemctl list-units --type=service --all --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- unit files (enabled state) --"
 systemctl list-unit-files --no-pager 2>/dev/null \
    | grep -Ei 'pihole|dnsmasq|ftl|technitium|dns|ntp|chrony|timesync|resolv|resolved' || true
 echo "-- installed packages of interest --"
 dpkg -l 2>/dev/null | grep -Ei 'pihole|dnsmasq|ftl|technitium|ntp|chrony|timesync|unbound|resolved|resolvconf' || true
 # -----------------------------------------------------------------------------
 # 4. NTP service details
 # -----------------------------------------------------------------------------
 section "4. NTP"
 echo "-- chrony --"
 systemctl status chrony --no-pager 2>/dev/null | head -15 || echo "(no chrony unit)"
 read_file /etc/chrony/chrony.conf
 echo "-- chronyc sources/stats (if available) --"
 chronyc -n sources 2>/dev/null || sudo -n chronyc -n sources 2>/dev/null || true
 chronyc -n tracking 2>/dev/null || sudo -n chronyc -n tracking 2>/dev/null || true
 echo "-- ntpsec / ntp classic --"
 systemctl status ntp --no-pager 2>/dev/null | head -15 || echo "(no ntp unit)"
 systemctl status ntpsec --no-pager 2>/dev/null | head -15 || true
 read_file /etc/ntp.conf
 read_file /etc/ntpsec/ntp.conf
 echo "-- systemd-timesyncd --"
 systemctl status systemd-timesyncd --no-pager 2>/dev/null | head -15 || echo "(no timesyncd)"
 read_file /etc/systemd/timesyncd.conf
 echo "-- openntpd --"
 systemctl status openntpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/openntpd/ntpd.conf
 # -----------------------------------------------------------------------------
 # 5. Docker (Pi-hole and/or Technitium may be containerized)
 # -----------------------------------------------------------------------------
 section "5. DOCKER"
 if command -v docker >/dev/null 2>&1; then
    echo "-- docker version --"; docker version 2>/dev/null || sudo -n docker version 2>/dev/null || true
    echo "-- containers (running) --"; docker ps 2>/dev/null || sudo -n docker ps 2>/dev/null || true
    echo "-- containers (all) --"; docker ps -a 2>/dev/null || sudo -n docker ps -a 2>/dev/null || true
    echo "-- images --"; docker images 2>/dev/null || sudo -n docker images 2>/dev/null || true
    echo "-- volumes --"; docker volume ls 2>/dev/null || sudo -n docker volume ls 2>/dev/null || true
    echo "-- networks --"; docker network ls 2>/dev/null || sudo -n docker network ls 2>/dev/null || true
 else
    echo "(docker not installed / not on PATH)"
 fi
 echo "-- compose files in common locations --"
 for d in /etc/docker-compose /opt/docker-compose /root/docker-compose /home/localuser/docker-compose /srv/docker-compose /opt/pihole /opt/technitium /opt; do
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        echo "### $d ###"
        sudo -n ls -la "$d" 2>/dev/null || ls -la "$d" 2>/dev/null || true
        for f in docker-compose.yml docker-compose.yaml compose.yml compose.yaml; do
            if sudo -n test -f "$d/$f" 2>/dev/null; then
                read_file "$d/$f"
            elif [ -f "$d/$f" ]; then
                read_file "$d/$f"
            fi
        done
    fi
 done
 # -----------------------------------------------------------------------------
 # 6. Pi-hole
 # -----------------------------------------------------------------------------
 section "6. PI-HOLE"
 if command -v pihole >/dev/null 2>&1; then
    echo "-- pihole version --"; pihole -v 2>/dev/null || sudo -n pihole -v 2>/dev/null || true
    echo "-- pihole status --"; pihole status 2>/dev/null || sudo -n pihole status 2>/dev/null || true
 else
    echo "(pihole command not on PATH)"
 fi
 echo "-- /etc/pihole listing --"
 list_dir /etc/pihole
 # Key Pi-hole config files
 for f in \
    /etc/pihole/setupVars.conf \
    /etc/pihole/pihole-FTL.conf \
    /etc/pihole/adlists.list \
    /etc/pihole/whitelist.txt \
    /etc/pihole/blacklist.txt \
    /etc/pihole/regex.list \
    /etc/pihole/custom.list \
    /etc/pihole/local.list \
    /etc/pihole/hostnameMappings.txt \
    /etc/pihole/dhcp.leases \
    /etc/pihole/static_ip.conf \
    /etc/pihole/GitHubVersions \
    /etc/pihole/macvendor.db ; do
    read_file "$f"
 done
 echo "-- /etc/pihole/*.conf (all) --"
 for f in /etc/pihole/*.conf; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.d listing --"
 list_dir /etc/dnsmasq.d
 for f in /etc/dnsmasq.d/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- /etc/dnsmasq.conf (if present) --"
 read_file /etc/dnsmasq.conf
 echo "-- gravity.db schema + row counts (Pi-hole v5+) --"
 GRAVITY=""
 for g in /etc/pihole/gravity.db /etc/pihole/gravity.db.*; do
    if [ -e "$g" ]; then GRAVITY="$g"; break; fi
 done
 if [ -n "${GRAVITY:-}" ]; then
    echo "gravity.db = $GRAVITY"
    if command -v sqlite3 >/dev/null 2>&1; then
        sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" ".tables" 2>/dev/null || true
        for tbl in adlist adlist_by_group domainlist domainlist_by_group client client_by_group group info; do
            cnt=$(sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT COUNT(*) FROM $tbl;" 2>/dev/null || echo "n/a")
            printf 'gravity.%s count = %s\n' "$tbl" "$cnt"
        done
        echo "-- adlist entries (url, enabled, comment) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,address,enabled,comment FROM adlist;" 2>/dev/null || true
        echo "-- domainlist sample (first 50) --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,type,domain,enabled,comment FROM domainlist LIMIT 50;" 2>/dev/null || true
        echo "-- client list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,ip,comment FROM client;" 2>/dev/null || true
        echo "-- group list --"
        sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT id,name,enabled,comment FROM 'group';" 2>/dev/null || true
        echo "-- info table --"
        sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null \
            || sudo -n sqlite3 -readonly "$GRAVITY" "SELECT * FROM info;" 2>/dev/null || true
    else
        echo "(sqlite3 not installed; gravity.db present at $GRAVITY)"
    fi
 else
    echo "(no gravity.db found)"
 fi
 echo "-- lighttpd / pihole web admin --"
 systemctl status lighttpd --no-pager 2>/dev/null | head -15 || true
 read_file /etc/lighttpd/lighttpd.conf
 for f in /etc/lighttpd/conf-enabled/*; do [ -e "$f" ] && read_file "$f"; done 2>/dev/null
 echo "-- pihole-FTL service --"
 systemctl status pihole-FTL --no-pager 2>/dev/null | head -20 || true
 # -----------------------------------------------------------------------------
 # 7. Technitium DNS
 # -----------------------------------------------------------------------------
 section "7. TECHNITIUM DNS"
 echo "-- technitium systemd unit --"
 systemctl status technitium-dns --no-pager 2>/dev/null | head -20 || echo "(no technitium-dns unit)"
 echo "-- any unit containing technitium --"
 systemctl list-units --all --no-pager 2>/dev/null | grep -i technitium || true
 echo "-- unit file path --"
 sudo -n systemctl cat technitium-dns 2>/dev/null || systemctl cat technitium-dns 2>/dev/null || true
 # Technitium common install locations
 for d in /etc/technitium /etc/technitium/dns /opt/technitium /opt/technitium/dns /var/lib/technitium /usr/local/technitium; do
    echo "### checking $d ###"
    if sudo -n test -d "$d" 2>/dev/null || [ -d "$d" ]; then
        list_dir "$d"
        # recurse one level for config files
        for sub in "$d" "$d"/*; do
            [ -e "$sub" ] || continue
            if [ -f "$sub" ] && echo "$sub" | grep -Eq '\.(xml|json|conf|config|txt)$'; then
                read_file "$sub"
            fi
        done
    fi
 done
 echo "-- technitium config.xml / dnsServer.conf (search) --"
 sudo -n find /etc/technitium /opt/technitium /var/lib/technitium -maxdepth 4 \
    \( -name 'config.xml' -o -name '*.config' -o -name 'dnsServer.conf' -o -name 'blockList.txt' \) \
    -print 2>/dev/null || true
 # -----------------------------------------------------------------------------
 # 8. Firewall / SELinux / misc
 # -----------------------------------------------------------------------------
 section "8. FIREWALL & MISC"
 echo "-- nftables --"; sudo -n nft list ruleset 2>/dev/null | head -80 || true
 echo "-- iptables --"; sudo -n iptables -S 2>/dev/null | head -40 || true
 echo "-- ufw --"; sudo -n ufw status verbose 2>/dev/null || true
 echo "-- firewalld --"; systemctl status firewalld --no-pager 2>/dev/null | head -8 || true
 echo "-- selinux/apparmor --"; getenforce 2>/dev/null || echo "(SELinux not present)"; aa-status 2>/dev/null | head -5 || true
 # -----------------------------------------------------------------------------
 # 9. Cron / timers that maintain these services
 # -----------------------------------------------------------------------------
 section "9. CRON & TIMERS"
 echo "-- root crontab --"; sudo -n crontab -l 2>/dev/null || echo "(n/a)"
 echo "-- localuser crontab --"; crontab -l 2>/dev/null || echo "(none)"
 echo "-- /etc/cron.d --"; ls -la /etc/cron.d 2>/dev/null
 for f in /etc/cron.d/*pihole* /etc/cron.d/*technitium* /etc/cron.d/*gravity*; do
    [ -e "$f" ] && read_file "$f"
 done 2>/dev/null
 echo "-- pihole timer --"; systemctl list-timers --all --no-pager 2>/dev/null | grep -Ei 'pihole|gravity|technitium' || true
 printf '\n##### AUDIT END %s #####\n' "$AUDIT_DATE"
@@ -1,15 +0,0 @@
 #!/usr/bin/env bash
 # shellcheck disable=SC2012 # diagnostic baseline script; ls -la listings are intentional
 # baseline.sh — quick read-only baseline of a target node.
 set -u
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "WHO/SUDO"; id; echo "--- sudo -n ---"; sudo -n true 2>&1 && echo "sudo OK" || echo "sudo FAIL"
 hdr "OS"; cat /etc/os-release 2>/dev/null | grep PRETTY; uname -r
 hdr "NET"; hostname -I 2>/dev/null; ip -br addr 2>/dev/null; echo "-- tailscale --"; tailscale ip -4 2>/dev/null || echo "(tailscale CLI absent or no ip)"
 hdr "DISK/MEM"; df -h / 2>/dev/null | tail -2; free -h 2>/dev/null | head -2
 hdr "DOCKER"; docker --version 2>&1; sudo docker version --format '{{.Server.Version}}' 2>&1 | tail -1; id -nG | tr ' ' '\n' | grep -qx docker && echo "localuser IN docker group" || echo "localuser NOT in docker group"
 hdr "EXISTING RELEVANT SERVICES"; systemctl is-active docker 2>/dev/null; systemctl is-enabled docker 2>/dev/null
 hdr "PORTS 53/67/80/123/443/5380/53443"; ss -tlnup 2>/dev/null | grep -E ':53|:67|:80|:123|:443|:5380|:53443' || echo "(none of those ports listening)"
 hdr "EXISTING SERVICES DIRS"; ls -la /home/localuser/services 2>/dev/null || echo "(no ~/services)"; ls -la /root/pihole /root/NTP 2>/dev/null || sudo -n ls -la /root 2>/dev/null | head
 hdr "DNS RESOLV"; cat /etc/resolv.conf 2>/dev/null
 hdr "HOME"; ls -la /home/localuser 2>/dev/null | head
@@ -1,69 +0,0 @@
 #!/usr/bin/env bash
 # deep-audit-netboot.sh — READ-ONLY deep inspection. Uses `sudo` for docker/root files.
 set -u
 DG="sudo docker"
 hdr() { printf '\n=== %s ===\n' "$1"; }
 hdr "COMPOSE FILES: /root/pihole and /root/NTP"
 for f in /root/pihole/docker-compose.yml /root/NTP/docker-compose.yml; do
    printf '\n--- %s ---\n' "$f"
    sudo cat "$f" 2>&1
 done
 hdr "DIR LAYOUT of compose project dirs"
 sudo ls -la /root/pihole 2>&1
 sudo ls -la /root/NTP 2>&1
 sudo ls -la /root 2>&1
 hdr "SEARCH entire FS for any technitium / dns compose files"
 sudo find / -xdev -maxdepth 6 \( -iname 'docker-compose.y*ml' -o -iname 'compose.y*ml' \) -print 2>/dev/null \
    | grep -Ei 'dns|technitium|tsys' || true
 hdr "ALL CONTAINERS with compose labels"
 $DG ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\tWDIR={{.Label "com.docker.compose.project.working_dir"}}\tCFG={{.Label "com.docker.compose.project.config_files"}}\tSVC={{.Label "com.docker.compose.service"}}' 2>&1
 hdr "DOCKER INSPECT pihole + tsys-ntp (and any dns/technitium container)"
 for c in $($DG ps -aq 2>/dev/null); do
    nm=$($DG inspect --format '{{.Name}} | image={{.Config.Image}} | proj={{index .Config.Labels "com.docker.compose.project"}}' "$c" 2>/dev/null)
    printf '\n###### %s ######\n' "$nm"
    $DG inspect "$c" 2>&1
 done
 hdr "PI-HOLE internal config (sudo docker exec, READ-ONLY)"
 echo "-- ls /etc/pihole --";          $DG exec pihole ls -la /etc/pihole 2>&1 || true
 echo "-- ls /etc/dnsmasq.d --";       $DG exec pihole ls -la /etc/dnsmasq.d 2>&1 || true
 for f in setupVars.conf pihole-FTL.conf adlists.list custom.list local.list regex.list dhcp.leases static_ip.conf; do
    echo "--- /etc/pihole/$f ---"; $DG exec pihole cat "/etc/pihole/$f" 2>&1 || true
 done
 echo "-- /etc/dnsmasq.d/* --"
 # shellcheck disable=SC2016 # $f/$t expand inside the container's sh, not locally — single quotes are intentional
 $DG exec pihole sh -c 'for f in /etc/dnsmasq.d/*; do echo "--- $f ---"; cat "$f"; done' 2>&1 || true
 echo "-- pihole version --";          $DG exec pihole pihole -v 2>&1 || true
 echo "-- gravity row counts --"
 # shellcheck disable=SC2016 # $t expands inside the container's sh, not locally
 $DG exec pihole sh -c 'for t in adlist domainlist client "group" info; do printf "%s=" "$t"; sqlite3 /etc/pihole/gravity.db "SELECT COUNT(*) FROM $t;" 2>/dev/null; done' 2>&1 || true
 echo "-- adlist addresses --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT address,enabled,comment FROM adlist;" 2>&1 || true
 echo "-- domainlist (allow+deny, first 60) --"
 $DG exec pihole sqlite3 /etc/pihole/gravity.db "SELECT type,domain,enabled,comment FROM domainlist LIMIT 60;" 2>&1 || true
 hdr "CHRONY/NTP container config (tsys-ntp)"
 $DG exec tsys-ntp sh -c 'echo "--- chrony.conf ---"; cat /etc/chrony/chrony.conf 2>/dev/null || cat /etc/chrony.conf 2>/dev/null || echo none; echo "--- ls /etc ---"; ls -la /etc 2>/dev/null | head -30' 2>&1 || true
 $DG exec tsys-ntp chronyc -n sources 2>&1 || true
 $DG exec tsys-ntp chronyc -n tracking 2>&1 || true
 echo "-- bare metal ntpsec.conf --"; sudo cat /etc/ntpsec/ntp.conf 2>&1
 hdr "TECHNITIUM volumes"
 for v in dns_tsys-dns-config dns_tyss-dns-config; do
    mnt=$($DG volume inspect --format '{{.Mountpoint}}' "$v" 2>/dev/null)
    printf '\n--- volume %s -> %s ---\n' "$v" "$mnt"
    [ -n "$mnt" ] || continue
    sudo find "$mnt" -maxdepth 4 -type f 2>/dev/null | head -80
    echo "-- config dir listing --"
    sudo ls -laR "$mnt"/config 2>/dev/null | head -60 || sudo ls -laR "$mnt" 2>/dev/null | head -60 || true
    echo "-- config.xml --"
    sudo cat "$mnt"/config/config.xml 2>/dev/null | head -250 || true
 done
 hdr "DONE"
@@ -1,10 +0,0 @@
 # netinfra/dhcp-migration.md
 > **Documentation moved to Discourse — the canonical source of truth.**
 >
 > **DHCP migration to netinfra-01/02**
 >
 > **Read it here:** https://community.turnsys.com/t/306
 >
 > *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
 > Do not update content here — edit the Discourse wiki topic instead.*
@@ -1,333 +0,0 @@
 # dhcpd.conf — pfv-netinfra-01 (PRIMARY)
 # Migrated from pfv-netboot 2026-07-29
 # Managed via Webmin DHCP module
 #
 # FAILOVER: this node is PRIMARY; peer is pfv-netinfra-02 (192.168.3.253)
 # DESIGN [#420, founder ruling 2026-08-27]: DHCP failover = active/passive
 #   hot standby (split 255: primary serves all hashes; secondary answers only
 #   when primary unreachable for >3s via 'load balance max seconds').
 #   DNS + NTP options hand out BOTH servers (active/active). Tailscale hosts
 #   use 100.100.100.100 MagicDNS, redundant to both netinfra TS IPs.
 # Global defaults
 option domain-name "knel.net";
 option domain-name-servers 192.168.3.252, 192.168.3.253;
 option ntp-servers 192.168.3.252, 192.168.3.253;
 default-lease-time 600;
 max-lease-time 7200;
 ddns-update-style none;
 authoritative;
 # ----- failover peer (PRIMARY) -----
 failover peer "pfv-dhcp" {
  primary;
  address 192.168.3.252;
  port 647;
  peer address 192.168.3.253;
  peer port 647;
  max-response-delay 30;
  max-unacked-updates 10;
  mclt 600;
  split 255;
  load balance max seconds 3;
 }
 # ----- subnet (shared /22) -----
 subnet 192.168.0.0 netmask 255.255.252.0 {
  option routers 192.168.3.254;
  option domain-name-servers 192.168.3.252, 192.168.3.253;
  option ntp-servers 192.168.3.252, 192.168.3.253;
  option domain-name "knel.net";
  authoritative;
  allow unknown-clients;
  pool {
    failover peer "pfv-dhcp";
    range 192.168.0.1 192.168.3.200;
  }
  # ---- host reservations (fixed-address; not subject to failover pool) ----
  host pfv-r3-tor-mgmt-01 {
    hardware ethernet 00:14:22:69:1c:37;
    fixed-address 192.168.0.7;
  }
  host pfv-r3-tor-stor-01 {
    hardware ethernet 00:13:72:46:95:e4;
    fixed-address 192.168.0.9;
  }
  host pfv-printer {
    hardware ethernet 40:9f:38:b0:b5:2f;
    fixed-address 192.168.1.84;
  }
  host pfv-r2-tor-01 {
    hardware ethernet 00:0d:56:41:7a:4d;
    fixed-address 192.168.0.10;
  }
  host pfv-r5-core-01 {
    hardware ethernet a4:ba:db:6f:ce:28;
    fixed-address 192.168.0.12;
  }
  host upstairs-receiver {
    hardware ethernet 74:5e:1c:76:e2:60;
    fixed-address 192.168.0.21;
  }
  host ap-tablemount {
    hardware ethernet e0:63:da:36:73:39;
    fixed-address 192.168.3.54;
  }
  host ap-wallmount {
    hardware ethernet e0:63:da:33:bb:1d;
    fixed-address 192.168.1.182;
  }
  host pfv-consrv {
    hardware ethernet 00:60:2e:01:50:aa;
    fixed-address 192.168.3.56;
  }
  host pfv-garage-pdu-1 {
    hardware ethernet 00:c0:b7:7e:49:78;
    fixed-address 192.168.3.18;
  }
  host pfv-dvr {
    hardware ethernet 54:2b:57:37:a7:d9;
    fixed-address 192.168.3.84;
  }
  host appletv-livingroom {
    hardware ethernet d0:d2:b0:97:81:c2;
    fixed-address 192.168.1.81;
  }
  host pfv-stor1 {
    hardware ethernet 00:00:c0:34:0c:dc;
    fixed-address 192.168.1.166;
  }
  host stlp-3dscanner {
    hardware ethernet b8:27:eb:91:31:82;
    fixed-address 192.168.0.4;
  }
  host pfv-jetson-nano-1 {
    hardware ethernet 00:04:4b:e4:17:7b;
    fixed-address 192.168.3.186;
  }
  host pfv-tsys7-oob {
    hardware ethernet f8:bc:12:35:1e:c6;
    fixed-address 192.168.3.197;
  }
  host pfv-tsys6-oob {
    # 2026-08-27 #460: was a4:ba:db:0b:df:a0 (core-switch OUI mis-migrated from
    # pfv-netboot, never matched tsys6 iDRAC); corrected to live iDRAC6 MAC
    # 00:21:9b:a2:7c:5b read in-band via ipmitool lan print
    hardware ethernet 00:21:9b:a2:7c:5b;
    fixed-address 192.168.3.196;
  }
  host brother-label-printer {
    hardware ethernet 04:fe:a1:56:72:e2;
    fixed-address 192.168.3.52;
  }
  host pfv-bms {
    hardware ethernet 02:5A:39:38:3E:9F;
    fixed-address 192.168.3.12;
  }
  host stl-canon-scanner-artroom {
    hardware ethernet 74:38:b7:24:fa:4e;
    fixed-address 192.168.3.142;
  }
  host tsys-ucs-01 {
    hardware ethernet bc:24:11:86:ea:1a;
    fixed-address 192.168.2.51;
  }
  host tsys-ucs-02 {
    hardware ethernet bc:24:11:c8:da:34;
    fixed-address 192.168.2.54;
  }
  host dell-openmanage-enterprise {
    hardware ethernet bc:24:11:ac:f4:6b;
    fixed-address 192.168.2.113;
  }
  host pfv-rrinfra-rtr {
    hardware ethernet 00:1d:70:0b:4f:41;
    fixed-address 192.168.3.94;
  }
  host pfv-tsys1 {
    hardware ethernet 34:17:eb:b3:b1:2d;
    fixed-address 192.168.3.11;
  }
  host pfv-tsys3 {
    hardware ethernet a4:4c:c8:08:d1:b8;
    fixed-address 192.168.2.5;
  }
  host pfv-tsys4 {
    hardware ethernet 98:90:96:c4:96:9a;
    fixed-address 192.168.3.191;
  }
  host pfv-tsys5 {
    hardware ethernet 18:03:73:43:ce:de;
    fixed-address 192.168.0.20;
  }
  host pfv-tsys6 {
    hardware ethernet 00:21:9b:a2:7c:53;
    fixed-address 192.168.3.169;
  }
  host pfv-tsys7 {
    hardware ethernet f8:bc:12:34:e0:74;
    fixed-address 192.168.0.250;
  }
  host pfv-tsys9 {
    hardware ethernet a4:bb:6d:e3:56:86;
    fixed-address 192.168.3.58;
  }
 	# umbrel
 	host tsys-umbrel {
 		hardware ethernet 02:2E:FF:8E:A2:D2;
 		fixed-address 192.168.1.97;
 		}
 	# ultix-streaming
 	host ultix-streaming {
 		hardware ethernet bc:24:11:1a:8f:6f;
 		fixed-address 192.168.3.78;
 		}
 	# ultix-highside
 	host ultix-highside {
 		hardware ethernet a0:4a:5e:ca:46:f3;
 		fixed-address 192.168.3.32;
 		}
 	# pfv-k8s-cnode1
 	host pfv-k8s-cnode1 {
 		hardware ethernet bc:24:11:cb:97:10;
 		fixed-address 192.168.1.91;
 		}
 	# pfv-k8s-cnode2
 	host pfv-k8s-cnode2 {
 		hardware ethernet bc:24:11:40:25:f8;
 		fixed-address 192.168.3.113;
 		}
 	# pfv-k8s-cnode3
 	host pfv-k8s-cnode3 {
 		hardware ethernet bc:24:11:38:c0:58;
 		fixed-address 192.168.1.228;
 		}
 	# devbox-cloudron
 	host devbox-cloudron {
 		hardware ethernet bc:24:11:f7:b1:07;
 		fixed-address 192.168.1.6;
 		}
 	# hfnoc-uisp
 	host hfnoc-uisp {
 		hardware ethernet bc:24:11:a3:87:61;
 		fixed-address 192.168.3.193;
 		}
 	# kali-rd
 	host kali-rd {
 		hardware ethernet bc:24:11:9e:1c:e9;
 		fixed-address 192.168.2.37;
 		}
 	# kali-tsys
 	host kali-tsys {
 		hardware ethernet bc:24:11:16:22:d4;
 		fixed-address 192.168.1.114;
 		}
 }
 # ---- host declarations outside subnet (global scope, same as netboot) ----
 host pfv-r6-mgmt-01 {
  hardware ethernet 00:14:22:69:18:a7;
  fixed-address 192.168.0.8;
 }
 host pfv-r1-tor-top {
  hardware ethernet 00:23:ae:c1:ad:e8;
  fixed-address 192.168.0.11;
 }
 # --- VM DHCP reservations (generated 2026-08-11, ticket #420) ---
 # All pinned to current ARP-observed IPs. No forward DNS needed
 # (forward records point to Tailscale 100.x addresses).
  host tsys-ca {
    hardware ethernet bc:24:11:32:d0:36;
    fixed-address 192.168.1.181;
  }
  host pfv-netinfra-01 {
    hardware ethernet bc:24:11:65:b2:ac;
    fixed-address 192.168.3.252;
  }
  host pfv-netinfra-02 {
    hardware ethernet bc:24:11:e4:37:53;
    fixed-address 192.168.3.253;
  }
  host tsys-librenms {
    hardware ethernet bc:24:11:5c:96:1e;
    fixed-address 192.168.3.176;
  }
  host tsys-proxmox-datacenter {
    hardware ethernet bc:24:11:e6:03:2d;
    fixed-address 192.168.2.44;
  }
  host pfv-k8s-wnode-tsys3 {
    hardware ethernet bc:24:11:ee:7e:7b;
    fixed-address 192.168.1.98;
  }
  host pfv-proxmox-backup-server {
    hardware ethernet bc:24:11:6e:12:69;
    fixed-address 192.168.2.193;
  }
  host pfv-k8s-wnode-tsys5 {
    hardware ethernet bc:24:11:c7:a8:6c;
    fixed-address 192.168.1.5;
  }
  host preprod-hfnoc-uisp {
    hardware ethernet bc:24:11:74:d6:8a;
    fixed-address 192.168.3.192;
  }
  host tsys-awx {
    hardware ethernet bc:24:11:80:0d:16;
    fixed-address 192.168.3.115;
  }
  host pfv-rr-middleware-02 {
    hardware ethernet bc:24:11:96:0e:ee;
    fixed-address 192.168.1.117;
  }
  host tsys-proxmox-mailgw-01 {
    hardware ethernet bc:24:11:56:61:18;
    fixed-address 192.168.1.11;
  }
  host pfv-k8s-wnode-tsys7 {
    hardware ethernet bc:24:11:30:b8:07;
    fixed-address 192.168.1.109;
  }
  host pfv-rr-middleware-01 {
    hardware ethernet bc:24:11:1e:61:cf;
    fixed-address 192.168.1.110;
  }
  host tsys-voip {
    hardware ethernet bc:24:11:23:ce:04;
    fixed-address 192.168.1.70;
  }
  host tsys-proxmox-mailgw-02 {
    hardware ethernet bc:24:11:5f:e5:2c;
    fixed-address 192.168.1.10;
  }
  host pfv-k8s-wnode-tsys6 {
    hardware ethernet bc:24:11:fa:6e:b5;
    fixed-address 192.168.1.111;
  }
  host tsys-siem {
    hardware ethernet bc:24:11:ee:67:e2;
    fixed-address 192.168.1.223;
  }
 # --- added 2026-08-26, ticket #420 (missing reservations) ---
 host pfv-k8s-wnode-tsys9 { hardware ethernet BC:24:11:EE:2B:B6; fixed-address 192.168.1.104; }
 host stlpc-artroom { hardware ethernet ca:fe:0f:0e:b0:c3; fixed-address 192.168.3.87; }
 host stlpc-garage { hardware ethernet 96:86:dd:1b:81:5a; fixed-address 192.168.0.100; }
 host subopi3 { hardware ethernet b8:27:eb:11:47:b2; fixed-address 192.168.1.217; }
 host subopi-dev-3 { hardware ethernet b8:27:eb:1b:35:95; fixed-address 192.168.1.232; }
 host subopi-dev-4 { hardware ethernet dc:a6:32:d4:da:5e; fixed-address 192.168.1.231; }
 host pfvsvrpi { hardware ethernet b8:27:eb:c2:f8:9f; fixed-address 192.168.1.230; }
 # --- added 2026-09-01, tickets #614/#629 (minisplit smart dongle) ---
 host pfv-minisplit-dongle { hardware ethernet c4:39:60:59:0c:01; fixed-address 192.168.1.149; }
 host pfv-dirigera { hardware ethernet 68:ec:8a:0e:36:69; fixed-address 192.168.3.123; }
 host pfv-wiz-bulb-1 { hardware ethernet cc:40:85:a4:d4:36; fixed-address 192.168.1.150; }
 host pfv-wiz-bulb-2 { hardware ethernet cc:40:85:a4:f8:22; fixed-address 192.168.1.151; }
@@ -1,332 +0,0 @@
 # dhcpd.conf — pfv-netinfra-02 (SECONDARY)
 # Migrated from pfv-netboot 2026-07-29
 # Managed via Webmin DHCP module
 #
 # FAILOVER: this node is SECONDARY; peer is pfv-netinfra-01 (192.168.3.252)
 # DESIGN [#420, founder ruling 2026-08-27]: DHCP failover = active/passive
 #   hot standby (primary split 255 serves all clients while healthy; this
 #   node answers only when primary unreachable for >3s via
 #   'load balance max seconds 3').
 #   DNS + NTP options hand out BOTH servers (active/active). Tailscale hosts
 #   use 100.100.100.100 MagicDNS, redundant to both netinfra TS IPs.
 # Global defaults
 option domain-name "knel.net";
 option domain-name-servers 192.168.3.252, 192.168.3.253;
 option ntp-servers 192.168.3.252, 192.168.3.253;
 default-lease-time 600;
 max-lease-time 7200;
 ddns-update-style none;
 authoritative;
 # ----- failover peer (SECONDARY) -----
 failover peer "pfv-dhcp" {
  secondary;
  address 192.168.3.253;
  port 647;
  peer address 192.168.3.252;
  peer port 647;
  max-response-delay 30;
  max-unacked-updates 10;
  load balance max seconds 3;
 }
 # ----- subnet (shared /22) -----
 subnet 192.168.0.0 netmask 255.255.252.0 {
  option routers 192.168.3.254;
  option domain-name-servers 192.168.3.252, 192.168.3.253;
  option ntp-servers 192.168.3.252, 192.168.3.253;
  option domain-name "knel.net";
  authoritative;
  allow unknown-clients;
  pool {
    failover peer "pfv-dhcp";
    range 192.168.0.1 192.168.3.200;
  }
  # ---- host reservations (fixed-address; not subject to failover pool) ----
  host pfv-r3-tor-mgmt-01 {
    hardware ethernet 00:14:22:69:1c:37;
    fixed-address 192.168.0.7;
  }
  host pfv-r3-tor-stor-01 {
    hardware ethernet 00:13:72:46:95:e4;
    fixed-address 192.168.0.9;
  }
  host pfv-printer {
    hardware ethernet 40:9f:38:b0:b5:2f;
    fixed-address 192.168.1.84;
  }
  host pfv-r2-tor-01 {
    hardware ethernet 00:0d:56:41:7a:4d;
    fixed-address 192.168.0.10;
  }
  host pfv-r5-core-01 {
    hardware ethernet a4:ba:db:6f:ce:28;
    fixed-address 192.168.0.12;
  }
  host upstairs-receiver {
    hardware ethernet 74:5e:1c:76:e2:60;
    fixed-address 192.168.0.21;
  }
  host ap-tablemount {
    hardware ethernet e0:63:da:36:73:39;
    fixed-address 192.168.3.54;
  }
  host ap-wallmount {
    hardware ethernet e0:63:da:33:bb:1d;
    fixed-address 192.168.1.182;
  }
  host pfv-consrv {
    hardware ethernet 00:60:2e:01:50:aa;
    fixed-address 192.168.3.56;
  }
  host pfv-garage-pdu-1 {
    hardware ethernet 00:c0:b7:7e:49:78;
    fixed-address 192.168.3.18;
  }
  host pfv-dvr {
    hardware ethernet 54:2b:57:37:a7:d9;
    fixed-address 192.168.3.84;
  }
  host appletv-livingroom {
    hardware ethernet d0:d2:b0:97:81:c2;
    fixed-address 192.168.1.81;
  }
  host pfv-stor1 {
    hardware ethernet 00:00:c0:34:0c:dc;
    fixed-address 192.168.1.166;
  }
  host stlp-3dscanner {
    hardware ethernet b8:27:eb:91:31:82;
    fixed-address 192.168.0.4;
  }
  host pfv-jetson-nano-1 {
    hardware ethernet 00:04:4b:e4:17:7b;
    fixed-address 192.168.3.186;
  }
  host pfv-tsys7-oob {
    hardware ethernet f8:bc:12:35:1e:c6;
    fixed-address 192.168.3.197;
  }
  host pfv-tsys6-oob {
    # 2026-08-27 #460: was a4:ba:db:0b:df:a0 (core-switch OUI mis-migrated from
    # pfv-netboot, never matched tsys6 iDRAC); corrected to live iDRAC6 MAC
    # 00:21:9b:a2:7c:5b read in-band via ipmitool lan print
    hardware ethernet 00:21:9b:a2:7c:5b;
    fixed-address 192.168.3.196;
  }
  host brother-label-printer {
    hardware ethernet 04:fe:a1:56:72:e2;
    fixed-address 192.168.3.52;
  }
  host pfv-bms {
    hardware ethernet 02:5A:39:38:3E:9F;
    fixed-address 192.168.3.12;
  }
  host stl-canon-scanner-artroom {
    hardware ethernet 74:38:b7:24:fa:4e;
    fixed-address 192.168.3.142;
  }
  host tsys-ucs-01 {
    hardware ethernet bc:24:11:86:ea:1a;
    fixed-address 192.168.2.51;
  }
  host tsys-ucs-02 {
    hardware ethernet bc:24:11:c8:da:34;
    fixed-address 192.168.2.54;
  }
  host dell-openmanage-enterprise {
    hardware ethernet bc:24:11:ac:f4:6b;
    fixed-address 192.168.2.113;
  }
  host pfv-rrinfra-rtr {
    hardware ethernet 00:1d:70:0b:4f:41;
    fixed-address 192.168.3.94;
  }
  host pfv-tsys1 {
    hardware ethernet 34:17:eb:b3:b1:2d;
    fixed-address 192.168.3.11;
  }
  host pfv-tsys3 {
    hardware ethernet a4:4c:c8:08:d1:b8;
    fixed-address 192.168.2.5;
  }
  host pfv-tsys4 {
    hardware ethernet 98:90:96:c4:96:9a;
    fixed-address 192.168.3.191;
  }
  host pfv-tsys5 {
    hardware ethernet 18:03:73:43:ce:de;
    fixed-address 192.168.0.20;
  }
  host pfv-tsys6 {
    hardware ethernet 00:21:9b:a2:7c:53;
    fixed-address 192.168.3.169;
  }
  host pfv-tsys7 {
    hardware ethernet f8:bc:12:34:e0:74;
    fixed-address 192.168.0.250;
  }
  host pfv-tsys9 {
    hardware ethernet a4:bb:6d:e3:56:86;
    fixed-address 192.168.3.58;
  }
 	# umbrel
 	host tsys-umbrel {
 		hardware ethernet 02:2E:FF:8E:A2:D2;
 		fixed-address 192.168.1.97;
 		}
 	# ultix-streaming
 	host ultix-streaming {
 		hardware ethernet bc:24:11:1a:8f:6f;
 		fixed-address 192.168.3.78;
 		}
 	# ultix-highside
 	host ultix-highside {
 		hardware ethernet a0:4a:5e:ca:46:f3;
 		fixed-address 192.168.3.32;
 		}
 	# pfv-k8s-cnode1
 	host pfv-k8s-cnode1 {
 		hardware ethernet bc:24:11:cb:97:10;
 		fixed-address 192.168.1.91;
 		}
 	# pfv-k8s-cnode2
 	host pfv-k8s-cnode2 {
 		hardware ethernet bc:24:11:40:25:f8;
 		fixed-address 192.168.3.113;
 		}
 	# pfv-k8s-cnode3
 	host pfv-k8s-cnode3 {
 		hardware ethernet bc:24:11:38:c0:58;
 		fixed-address 192.168.1.228;
 		}
 	# devbox-cloudron
 	host devbox-cloudron {
 		hardware ethernet bc:24:11:f7:b1:07;
 		fixed-address 192.168.1.6;
 		}
 	# hfnoc-uisp
 	host hfnoc-uisp {
 		hardware ethernet bc:24:11:a3:87:61;
 		fixed-address 192.168.3.193;
 		}
 	# kali-rd
 	host kali-rd {
 		hardware ethernet bc:24:11:9e:1c:e9;
 		fixed-address 192.168.2.37;
 		}
 	# kali-tsys
 	host kali-tsys {
 		hardware ethernet bc:24:11:16:22:d4;
 		fixed-address 192.168.1.114;
 		}
 }
 # ---- host declarations outside subnet (global scope, same as netboot) ----
 host pfv-r6-mgmt-01 {
  hardware ethernet 00:14:22:69:18:a7;
  fixed-address 192.168.0.8;
 }
 host pfv-r1-tor-top {
  hardware ethernet 00:23:ae:c1:ad:e8;
  fixed-address 192.168.0.11;
 }
 # --- VM DHCP reservations (generated 2026-08-11, ticket #420) ---
 # All pinned to current ARP-observed IPs. No forward DNS needed
 # (forward records point to Tailscale 100.x addresses).
  host tsys-ca {
    hardware ethernet bc:24:11:32:d0:36;
    fixed-address 192.168.1.181;
  }
  host pfv-netinfra-01 {
    hardware ethernet bc:24:11:65:b2:ac;
    fixed-address 192.168.3.252;
  }
  host pfv-netinfra-02 {
    hardware ethernet bc:24:11:e4:37:53;
    fixed-address 192.168.3.253;
  }
  host tsys-librenms {
    hardware ethernet bc:24:11:5c:96:1e;
    fixed-address 192.168.3.176;
  }
  host tsys-proxmox-datacenter {
    hardware ethernet bc:24:11:e6:03:2d;
    fixed-address 192.168.2.44;
  }
  host pfv-k8s-wnode-tsys3 {
    hardware ethernet bc:24:11:ee:7e:7b;
    fixed-address 192.168.1.98;
  }
  host pfv-proxmox-backup-server {
    hardware ethernet bc:24:11:6e:12:69;
    fixed-address 192.168.2.193;
  }
  host pfv-k8s-wnode-tsys5 {
    hardware ethernet bc:24:11:c7:a8:6c;
    fixed-address 192.168.1.5;
  }
  host preprod-hfnoc-uisp {
    hardware ethernet bc:24:11:74:d6:8a;
    fixed-address 192.168.3.192;
  }
  host tsys-awx {
    hardware ethernet bc:24:11:80:0d:16;
    fixed-address 192.168.3.115;
  }
  host pfv-rr-middleware-02 {
    hardware ethernet bc:24:11:96:0e:ee;
    fixed-address 192.168.1.117;
  }
  host tsys-proxmox-mailgw-01 {
    hardware ethernet bc:24:11:56:61:18;
    fixed-address 192.168.1.11;
  }
  host pfv-k8s-wnode-tsys7 {
    hardware ethernet bc:24:11:30:b8:07;
    fixed-address 192.168.1.109;
  }
  host pfv-rr-middleware-01 {
    hardware ethernet bc:24:11:1e:61:cf;
    fixed-address 192.168.1.110;
  }
  host tsys-voip {
    hardware ethernet bc:24:11:23:ce:04;
    fixed-address 192.168.1.70;
  }
  host tsys-proxmox-mailgw-02 {
    hardware ethernet bc:24:11:5f:e5:2c;
    fixed-address 192.168.1.10;
  }
  host pfv-k8s-wnode-tsys6 {
    hardware ethernet bc:24:11:fa:6e:b5;
    fixed-address 192.168.1.111;
  }
  host tsys-siem {
    hardware ethernet bc:24:11:ee:67:e2;
    fixed-address 192.168.1.223;
  }
 # --- added 2026-08-26, ticket #420 (missing reservations) ---
 host pfv-k8s-wnode-tsys9 { hardware ethernet BC:24:11:EE:2B:B6; fixed-address 192.168.1.104; }
 host stlpc-artroom { hardware ethernet ca:fe:0f:0e:b0:c3; fixed-address 192.168.3.87; }
 host stlpc-garage { hardware ethernet 96:86:dd:1b:81:5a; fixed-address 192.168.0.100; }
 host subopi3 { hardware ethernet b8:27:eb:11:47:b2; fixed-address 192.168.1.217; }
 host subopi-dev-3 { hardware ethernet b8:27:eb:1b:35:95; fixed-address 192.168.1.232; }
 host subopi-dev-4 { hardware ethernet dc:a6:32:d4:da:5e; fixed-address 192.168.1.231; }
 host pfvsvrpi { hardware ethernet b8:27:eb:c2:f8:9f; fixed-address 192.168.1.230; }
 # --- added 2026-09-01, tickets #614/#629 (minisplit smart dongle) ---
 host pfv-minisplit-dongle { hardware ethernet c4:39:60:59:0c:01; fixed-address 192.168.1.149; }
 host pfv-dirigera { hardware ethernet 68:ec:8a:0e:36:69; fixed-address 192.168.3.123; }
 host pfv-wiz-bulb-1 { hardware ethernet cc:40:85:a4:d4:36; fixed-address 192.168.1.150; }
 host pfv-wiz-bulb-2 { hardware ethernet cc:40:85:a4:f8:22; fixed-address 192.168.1.151; }
@@ -1,31 +0,0 @@
 #!/bin/bash
 # install-dhcp.sh — installs isc-dhcp-server + Webmin on a netinfra node.
 # Does NOT start the DHCP service. Run on the target node itself.
 set -e
 echo "=== Installing isc-dhcp-server ==="
 apt-get update -qq
 apt-get install -y isc-dhcp-server
 echo "=== Installing Webmin ==="
 if ! dpkg -l | grep -q '^ii.*webmin'; then
  curl -fsSL https://raw.githubusercontent.com/webmin/webmin/master/webmin-setup-repo.sh -o /tmp/webmin-setup.sh
  sh /tmp/webmin-setup.sh -f
  rm -f /tmp/webmin-setup.sh
  apt-get install -y webmin
 else
  echo "Webmin already installed"
 fi
 echo "=== Writing /etc/default/isc-dhcp-server ==="
 cat > /etc/default/isc-dhcp-server <<'EOF'
 # Defaults for isc-dhcp-server (sourced by /etc/init.d/isc-dhcp-server)
 INTERFACESv4="ens18"
 INTERFACESv6=""
 EOF
 echo "=== Stopping DHCP service (should not serve yet) ==="
 systemctl stop isc-dhcp-server 2>/dev/null || true
 systemctl disable isc-dhcp-server 2>/dev/null || true
 echo "=== Done. DHCP installed but NOT started. ==="
@@ -1,10 +0,0 @@
 # netinfra/dns-cluster-setup/README.md
 > **Documentation moved to Discourse — the canonical source of truth.**
 >
 > **Technitium DNS cluster setup**
 >
 > **Read it here:** https://community.turnsys.com/t/306
 >
 > *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
 > Do not update content here — edit the Discourse wiki topic instead.*
@@ -1,89 +0,0 @@
 #!/usr/bin/bash
 #
 # remote-dns.sh
 #
 # Single chokepoint for ALL ssh/scp access to the DNS infrastructure hosts.
 # Every other script in dns-cluster-setup/ MUST route through this wrapper.
 # Never call ssh/scp directly.
 #
 # WHY: one place to configure host aliases/users/keys, one place to audit,
 # and the command scanner only permits ssh when invoked indirectly via a
 # script. Mirrors the pattern of tests/remote.sh.
 #
 # HOSTS (override IPs via env if needed):
 #   netinfra01     pfv-netinfra-01.knel.net   (Technitium primary target)
 #   netinfra02     pfv-netinfra-02.knel.net   (Technitium secondary target)
 #   netboot        pfv-netboot.knel.net       (reference / validation client)
 #   sandbox        sectestbed-sandbox.knel.net (validation client)
 #
 # NOTE: the former tsrouter alias (tailscale-router.knel.net) was removed
 # 2026-09-02 — that host is retired; subnet routing is now the netinfra pair.
 #
 # All hosts are accessed as $VM_USER (default: localuser) over SSH with key auth
 # and passwordless sudo.
 #
 # USAGE:
 #   remote-dns.sh <host-alias> <cmd...>          run command on host
 #   remote-dns.sh <host-alias>-root <cmd...>     run command on host as root (sudo)
 #   remote-dns.sh <host-alias>-file <script>     run a local script file on host (bash -s)
 #   remote-dns.sh <host-alias>-copy <local> <remote-dest>   copy a file to host
 #
 #   e.g.
 #   remote-dns.sh netinfra01-root 'systemctl status dnsServer'
 #   remote-dns.sh netboot-file ./probe.sh
 #
 set -uo pipefail
 VM_USER="${VM_USER:-localuser}"
 # Hostname -> FQDN map. Override individual IPs via env if a host moves.
 NETINFRA01_HOST="${NETINFRA01_HOST:-pfv-netinfra-01.knel.net}"
 NETINFRA02_HOST="${NETINFRA02_HOST:-pfv-netinfra-02.knel.net}"
 NETBOOT_HOST="${NETBOOT_HOST:-pfv-netboot.knel.net}"
 SANDBOX_HOST="${SANDBOX_HOST:-sectestbed-sandbox.knel.net}"
 SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15)
 die() { echo "remote-dns.sh: $*" >&2; exit 1; }
 host_fqdn() {
  case "$1" in
    netinfra01)  printf '%s' "$NETINFRA01_HOST" ;;
    netinfra02)  printf '%s' "$NETINFRA02_HOST" ;;
    netboot)     printf '%s' "$NETBOOT_HOST" ;;
    sandbox)     printf '%s' "$SANDBOX_HOST" ;;
    *) return 1 ;;
  esac
 }
 _run() { ssh "${SSH_OPTS[@]}" "${VM_USER}@$1" "$2"; }
 _run_root() { ssh "${SSH_OPTS[@]}" "${VM_USER}@$1" "sudo -n bash -c $(printf '%q' "$2")"; }
 _run_file() { ssh "${SSH_OPTS[@]}" "${VM_USER}@$1" "bash -s" < "$2"; }
 _copy() {
  local fqdn="$1" local="$2" dest="$3"
  if command -v rsync >/dev/null 2>&1 \
     && ssh "${SSH_OPTS[@]}" "${VM_USER}@${fqdn}" 'command -v rsync' >/dev/null 2>&1; then
    rsync -az -e "ssh ${SSH_OPTS[*]}" "$local" "${VM_USER}@${fqdn}:${dest}"
  else
    ssh "${SSH_OPTS[@]}" "${VM_USER}@${fqdn}" "cat > '$dest'" < "$local"
  fi
 }
 spec="${1:-}"; shift || true
 # Split host alias from mode: "netinfra01", "netinfra01-root", "netinfra01-file", "netinfra01-copy"
 mode="run"
 alias="$spec"
 case "$spec" in
  *-root) mode="root"; alias="${spec%-root}" ;;
  *-file) mode="file"; alias="${spec%-file}" ;;
  *-copy) mode="copy"; alias="${spec%-copy}" ;;
 esac
 fqdn="$(host_fqdn "$alias")" || die "unknown host alias '$alias' (try: netinfra01|netinfra02|netboot|sandbox)"
 case "$mode" in
  run)  _run "$fqdn" "$*" ;;
  root) [ "$#" -ge 1 ] || die "need command"; _run_root "$fqdn" "$*" ;;
  file) [ -f "${1:-}" ] || die "need local script file"; _run_file "$fqdn" "$1" ;;
  copy) [ -f "${1:-}" ] || die "need local file"; _copy "$fqdn" "$1" "${2:-}" ;;
  *) die "bad mode" ;;
 esac
@@ -1,477 +0,0 @@
 #!/usr/bin/bash
 #
 # setup.sh — Technitium DNS Cluster Setup
 #
 # BOOTSTRAP (historical): originally exported the production config from the
 # retired tailscale-router host to the pfv-netinfra-01/02 pair. Since
 # 2026-09-02 (router retired) the export source is pfv-netinfra-01 itself,
 # the live primary and SoR for knel.net zones.
 #
 # Configures 01 as primary and 02 as secondary with automatic zone
 # transfers (AXFR).
 #
 # PRODUCTION SAFETY: the export source is accessed READ-ONLY. No file on it
 # is modified. The only operation is a docker cp (read) to export the config.
 #
 # ARCHITECTURE AFTER SETUP:
 #
 #   pfv-netinfra-01 (192.168.3.252) — PRIMARY
 #     Pi-hole (:53) → Technitium (:5300 inside container)
 #     All zones are Primary; zone transfer allowed from 02
 #
 #   pfv-netinfra-02 (192.168.3.253) — SECONDARY
 #     Pi-hole (:53) → Technitium (:5300 inside container)
 #     All zones are Secondary; AXFR from 01 on changes
 #
 # CLUSTERING MECHANISM:
 #   Technitium primary/secondary via DNS zone transfers (AXFR/IXFR + NOTIFY).
 #   01 serves all zones as Primary. 02 fetches them as Secondary from
 #   01's address (192.168.3.252:5300). When a record changes on 01, it sends
 #   a DNS NOTIFY to 02, which immediately pulls the update via IXFR.
 #
 # CREDENTIALS:
 #   The production auth.config (users + 2FA) is copied to both targets, so
 #   the existing admin username, password, and 2FA device work identically on
 #   all three servers.
 #
 # USAGE:
 #   ./setup.sh export     # Step 1: read-only export from primary (netinfra-01)
 #   ./setup.sh deploy01   # Step 2: deploy config to netinfra-01 (primary)
 #   ./setup.sh deploy02   # Step 3: deploy config to netinfra-02 (secondary)
 #   ./setup.sh cluster    # Step 4: configure clustering (01 primary, 02 secondary)
 #   ./setup.sh verify     # Step 5: test everything
 #   ./setup.sh all        # Steps 1-5 in sequence
 #
 set -euo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REMOTE="$HERE/remote-dns.sh"
 # Host aliases (defined in remote-dns.sh)
 PROD="$PRIMARY"        # export source: primary (was tailscale-router, retired 2026-09-02)
 PRIMARY="netinfra01"   # pfv-netinfra-01
 SECONDARY="netinfra02" # pfv-netinfra-02
 # Network addresses for zone transfer
 PRIMARY_IP="${PRIMARY_IP:-192.168.3.252}"
 SECONDARY_IP="${SECONDARY_IP:-192.168.3.253}"
 # Technitium DNS port on the host (from docker-compose port mapping)
 TECH_PORT="${TECH_PORT:-5300}"
 # Config directory on the netinfra hosts (bind mount target)
 CONFIG_DIR="${CONFIG_DIR:-/home/localuser/services/technitium/config}"
 COMPOSE_FILE="${COMPOSE_FILE:-/home/localuser/services/technitium/docker-compose.yml}"
 # Temporary admin password used ONLY during clustering API calls.
 # After configuration, the production auth.config (with 2FA) is restored.
 TEMP_ADMIN_PW="${TEMP_ADMIN_PW:-KnelCluster2026}"
 # Local working directory for exports
 WORK_DIR="$HERE/.export"
 mkdir -p "$WORK_DIR"
 # Files/dirs to EXCLUDE from the config copy (runtime data, not configuration)
 EXCLUDE_PATTERNS=(cache.bin stats logs)
 log()  { printf '\033[0;36m[%s]\033[0m %s\n' "$(date +%H:%M:%S)" "$*"; }
 die()  { log "ERROR: $*"; exit 1; }
 # -----------------------------------------------------------------------------
 # Helpers
 # -----------------------------------------------------------------------------
 # Build an exclude-args string for tar
 exclude_args() {
  local args=""
  for p in "${EXCLUDE_PATTERNS[@]}"; do
    args+=" --exclude=$p"
  done
  printf '%s' "$args"
 }
 # Run a command on a host as root via the wrapper
 run_root() { bash "$REMOTE" "$1-root" "${@:2}"; }
 run()      { bash "$REMOTE" "$1" "${@:2}"; }
 # Get a Technitium API token on a host (temporary admin, no 2FA)
 # Uses root to avoid PATH issues with non-interactive SSH sessions.
 # Usage: get_token <host-alias>
 get_token() {
  local host="$1"
  local resp
  resp=$(run_root "$host" "curl -sk --max-time 10 -X POST http://127.0.0.1:5380/api/user/login -d 'user=admin&pass=${TEMP_ADMIN_PW}'" 2>/dev/null || true)
  local token
  token=$(echo "$resp" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('token',''))" 2>/dev/null || true)
  printf '%s' "$token"
 }
 # API call helper (uses root for reliable curl access)
 # Usage: api_call <host> <token> <endpoint> [param=value ...]
 api_call() {
  local host="$1" token="$2" endpoint="$3"; shift 3
  local url="http://127.0.0.1:5380/api/${endpoint}?token=${token}"
  local p
  for p in "$@"; do url+="&${p}"; done
  run_root "$host" "curl -sk --max-time 10 '$url'" 2>/dev/null || true
 }
 # -----------------------------------------------------------------------------
 # Step 1: Export production config (READ-ONLY on the primary; historically
 # the retired tailscale-router)
 # -----------------------------------------------------------------------------
 do_export() {
  log "=== STEP 1: Exporting production config from $PROD (READ-ONLY) ==="
  local export_tar="$WORK_DIR/technitium-production-config.tar.gz"
  log "Exporting config volume from $PROD (piped, no disk writes on prod)..."
  # Read the Docker volume directory directly from the host filesystem.
  # No docker exec needed (avoids /tmp space issues on the prod host).
  # Pipe tar → ssh → local file. Nothing is written on production's disk.
  local vol_path
  vol_path=$(bash "$REMOTE" "$PROD-root" \
    "docker volume inspect -f '{{.Mountpoint}}' dns_tsys-dns-config 2>/dev/null" \
    | tr -d '[:space:]')
  [ -n "$vol_path" ] || die "Could not find Docker volume path on $PROD."
  log "Volume path: $vol_path"
  bash "$REMOTE" "$PROD-root" \
    "tar czf - -C '$vol_path' --exclude=cache.bin --exclude=stats --exclude=logs ." \
    > "$export_tar" 2>/dev/null || die "Export pipe failed."
  [ -s "$export_tar" ] || die "Export tarball is empty."
  # Inspect
  local zone_count
  zone_count=$(tar tzf "$export_tar" | grep -c '\.zone$' || true)
  log "Export complete: $(du -h "$export_tar" | cut -f1), $zone_count zones."
  # Save the zone name list for clustering
  tar tzf "$export_tar" | grep '\.zone$' | sed 's|^\./||; s|^zones/||; s|\.zone$||' | sort > "$WORK_DIR/zones.txt"
  log "Zone list saved ($zone_count zones): $(head -5 "$WORK_DIR/zones.txt" | tr '\n' ' ')..."
 }
 # -----------------------------------------------------------------------------
 # Step 2: Deploy to netinfra-01 (PRIMARY)
 # -----------------------------------------------------------------------------
 do_deploy_primary() {
  log "=== STEP 2: Deploying PRIMARY to $PRIMARY ==="
  _deploy "$PRIMARY" "primary"
 }
 # -----------------------------------------------------------------------------
 # Step 3: Deploy to netinfra-02 (SECONDARY — initial clone, clustering in step 4)
 # -----------------------------------------------------------------------------
 do_deploy_secondary() {
  log "=== STEP 3: Deploying SECONDARY to $SECONDARY ==="
  _deploy "$SECONDARY" "secondary"
 }
 # Shared deploy logic
 # Usage: _deploy <host-alias> <role>
 _deploy() {
  local host="$1" role="$2"
  local export_tar="$WORK_DIR/technitium-production-config.tar.gz"
  [ -f "$export_tar" ] || die "No export found. Run '$0 export' first."
  log "Stopping Technitium on $host..."
  run_root "$host" "cd $CONFIG_DIR/.. && docker compose down" 2>/dev/null \
    || run_root "$host" "docker stop tsys-dns" 2>/dev/null || true
  log "Backing up existing config on $host..."
  run_root "$host" "
    if [ -d '$CONFIG_DIR' ]; then
      mv '$CONFIG_DIR' '${CONFIG_DIR}.backup-$(date +%Y%m%d-%H%M%S)'
    fi
    mkdir -p '$CONFIG_DIR'
  " || die "Backup failed."
  log "Uploading production config to $host..."
  bash "$REMOTE" "$host-root" "cat > /tmp/technitium-config.tar.gz" < "$export_tar" \
    || die "Upload failed."
  log "Extracting config on $host..."
  run_root "$host" "
    cd '$CONFIG_DIR'
    tar xzf /tmp/technitium-config.tar.gz
    rm -f /tmp/technitium-config.tar.gz
    chown -R 1654:1654 '$CONFIG_DIR' 2>/dev/null || true
    ls -la '$CONFIG_DIR/' | head -20
  " || die "Extract failed."
  # Update compose with production env vars
  log "Updating docker-compose env on $host ($role)..."
  run_root "$host" "
    cat > /tmp/compose-patch.py << 'PYEOF'
 import re, sys
 f = sys.argv[1]
 with open(f) as fh: c = fh.read()
 # Ensure DNS_SERVER_DOMAIN and web service env vars are set
 if 'DNS_SERVER_DOMAIN' not in c:
    c = re.sub(r'(image:.*\n)', r'\1    environment:\n      - DNS_SERVER_DOMAIN=knel.net\n', c, count=1)
 print(c)
 PYEOF
    python3 /tmp/compose-patch.py '$COMPOSE_FILE' > '${COMPOSE_FILE}.new' 2>/dev/null && mv '${COMPOSE_FILE}.new' '$COMPOSE_FILE' || true
    rm -f /tmp/compose-patch.py
  " || log "WARN: compose patch skipped (non-critical)."
  log "Starting Technitium on $host..."
  run_root "$host" "cd $CONFIG_DIR/.. && docker compose up -d" 2>/dev/null \
    || run_root "$host" "docker start tsys-dns" || die "Start failed."
  log "Waiting for Technitium to come up on $host..."
  local i
  for i in $(seq 1 20); do
    if run "$host" "curl -sk --max-time 3 http://127.0.0.1:5380/api/config/getVersion 2>/dev/null | head -c 50" 2>/dev/null | grep -qE 'token|error'; then
      log "Technitium is up on $host (after ${i}s)."
      return 0
    fi
    sleep 2
  done
  die "Technitium did not come up on $host within 40s."
 }
 # -----------------------------------------------------------------------------
 # Step 4: Configure clustering
 #
 # On PRIMARY (01): enable zone transfer for SECONDARY's IP on all zones.
 # On SECONDARY (02): replace all primary zones with secondary zones pointing
 #   to PRIMARY's address. Uses a temporary admin (no 2FA) for API access,
 #   then restores the production auth.config.
 # -----------------------------------------------------------------------------
 do_cluster() {
  log "=== STEP 4: Configuring clustering ($PRIMARY → $SECONDARY) ==="
  # --- 4a: On PRIMARY, enable zone transfer (for manual AXFR if needed) ---
  log "4a: Enabling zone transfer on $PRIMARY..."
  _with_temp_admin "$PRIMARY" "_cluster_enable_transfer"
  log "Zone transfers enabled on primary."
  # --- 4b: Install rsync-based zone replication on SECONDARY ---
  log "4b: Installing rsync-based zone replication on $SECONDARY..."
  _install_rsync_replication
  log "Replication installed."
 }
 # Install rsync-based zone sync on the secondary as a systemd timer.
 _install_rsync_replication() {
  local sync_script="$HERE/sync-zones.sh"
  [ -f "$sync_script" ] || die "sync-zones.sh not found."
  # Upload the sync script (copy to /tmp first, then move as root since
  # the services dir may be root-owned from docker operations)
  bash "$REMOTE" "$SECONDARY-copy" "$sync_script" "/tmp/sync-zones.sh" \
    || die "Could not copy sync-zones.sh to /tmp."
  run_root "$SECONDARY" "cp /tmp/sync-zones.sh /home/localuser/services/technitium/sync-zones.sh && chmod +x /home/localuser/services/technitium/sync-zones.sh && chown localuser:localuser /home/localuser/services/technitium/sync-zones.sh && rm /tmp/sync-zones.sh" \
    || die "Could not install sync-zones.sh."
  # Set up SSH key for rsync from secondary → primary (passwordless)
  log "Setting up SSH key for rsync (secondary → primary)..."
  run_root "$SECONDARY" "
    if [ ! -f /home/localuser/.ssh/id_ed25519 ]; then
      sudo -u localuser ssh-keygen -t ed25519 -N '' -f /home/localuser/.ssh/id_ed25519 -q
    fi
    cat /home/localuser/.ssh/id_ed25519.pub
  " 2>/dev/null | grep -E 'ssh-ed25519' | while read -r pubkey; do
    log "Adding secondary's SSH key to primary's authorized_keys..."
    run_root "$PRIMARY" "mkdir -p /home/localuser/.ssh && echo '$pubkey' >> /home/localuser/.ssh/authorized_keys && chmod 600 /home/localuser/.ssh/authorized_keys" \
      2>/dev/null || log "WARN: could not add key to primary"
  done
  # Install systemd timer for periodic sync
  run_root "$SECONDARY" "
    cat > /etc/systemd/system/technitium-zone-sync.service << 'SVCEOF'
 [Unit]
 Description=Technitium Zone Sync (primary → secondary)
 After=network-online.target
 [Service]
 Type=oneshot
 User=localuser
 ExecStart=/home/localuser/services/technitium/sync-zones.sh
 SVCEOF
    cat > /etc/systemd/system/technitium-zone-sync.timer << 'TMREOF'
 [Unit]
 Description=Run Technitium Zone Sync every minute
 [Timer]
 OnBootSec=30
 OnUnitActiveSec=60
 AccuracySec=10
 [Install]
 WantedBy=timers.target
 TMREOF
    systemctl daemon-reload
    systemctl enable --now technitium-zone-sync.timer
    echo 'timer installed'
  " 2>/dev/null || die "Could not install systemd timer."
  # Trigger an immediate sync
  log "Triggering initial sync..."
  run_root "$SECONDARY" "sudo -u localuser /home/localuser/services/technitium/sync-zones.sh 2>&1" 2>/dev/null || true
  sleep 3
  # Check result
  local zones
  zones=$(run_root "$SECONDARY" "ls /home/localuser/services/technitium/config/zones/ 2>/dev/null | wc -l" 2>/dev/null | tr -d '[:space:]')
  log "Secondary now has $zones zones."
 }
 # Enable zone transfer for the secondary IP on all primary zones.
 # Runs inside _with_temp_admin, so $1 = host.
 _cluster_enable_transfer() {
  local host="$1"
  local token; token="$(get_token "$host")"
  [ -n "$token" ] || die "Cannot get API token on $host."
  # Set global zone transfer allow list to include the secondary.
  # Technitium per-zone "allow zone transfer" — use the API to set it.
  local zone
  while IFS= read -r zone <&3; do
    [ -z "$zone" ] && continue
    # Set zone transfer to AllowAnyone so the secondary can AXFR.
    # Technitium API param: zoneTransfer (not allowZoneTransfer).
    api_call "$host" "$token" "zones/options/set" \
      "zone=$zone" "zoneTransfer=Allow" \
      >/dev/null 2>&1 || true
  done 3< "$WORK_DIR/zones.txt"
  log "Zone transfer set to AllowAnyone for ${SECONDARY_IP} on all zones."
 }
 # Delete all primary zones and recreate as secondary zones.
 # Runs inside _with_temp_admin, so $1 = host.
 _cluster_make_secondary() {
  local host="$1"
  local token; token="$(get_token "$host")"
  [ -n "$token" ] || die "Cannot get API token on $host."
  local zone total
  total=$(wc -l < "$WORK_DIR/zones.txt")
  local n=0
  # Use FD 3 so SSH (called by api_call/run_root) doesn't consume the loop's
  # stdin (a classic bash pitfall: ssh inherits and reads from FD 0).
  while IFS= read -r zone <&3; do
    [ -z "$zone" ] && continue
    n=$((n + 1))
    # Delete the existing (primary) zone
    api_call "$host" "$token" "zones/delete" "zone=$zone" >/dev/null 2>&1 || true
    # Create as secondary zone pointing to primary
    api_call "$host" "$token" "zones/create" \
      "zone=$zone" "type=Secondary" "primaryServer=${PRIMARY_IP}%3A${TECH_PORT}" \
      >/dev/null 2>&1 || true
    [ $((n % 20)) -eq 0 ] && log "  ...converted $n/$total zones"
  done 3< "$WORK_DIR/zones.txt"
  log "Converted $n zones to secondary (AXFR from ${PRIMARY_IP}:${TECH_PORT})."
  # Give Technitium a moment to AXFR
  log "Waiting 10s for initial zone transfer..."
  sleep 10
 }
 # Helper: temporarily replace auth.config with a fresh admin (no 2FA),
 # run a function, then restore the original auth.config.
 # Uses a docker-compose.override.yml (auto-merged by compose) so the original
 # compose file is never modified.
 # Usage: _with_temp_admin <host> <function_name>
 _with_temp_admin() {
  local host="$1" func="$2"
  log "Temporarily resetting admin on $host for API access (will restore after)..."
  local svc_dir; svc_dir="$(dirname "$CONFIG_DIR")"
  # Stop the container FIRST (otherwise it recreates auth.config from memory
  # before we can delete it), then back up + delete auth.config, then create
  # the override file, then restart.
  log "Stopping Technitium on $host..."
  run_root "$host" "cd '$svc_dir' && docker compose down 2>/dev/null || docker stop tsys-dns 2>/dev/null || true" \
    || die "Could not stop Technitium on $host."
  # Back up production auth.config, then remove it so Technitium creates a
  # fresh admin on next start.
  run_root "$host" "
    cp '$CONFIG_DIR/auth.config' '$CONFIG_DIR/auth.config.production'
    rm -f '$CONFIG_DIR/auth.config'
  " || die "Could not back up/remove auth.config on $host."
  # Create a compose override that injects the temp admin password.
  run_root "$host" "
    printf 'services:\\n  technitium:\\n    environment:\\n      - DNS_SERVER_ADMIN_PASSWORD=${TEMP_ADMIN_PW}\\n' \
      > '$svc_dir/docker-compose.override.yml'
  " || die "Could not create compose override on $host."
  # Restart with override in effect
  run_root "$host" "cd '$svc_dir' && docker compose up -d" \
    2>/dev/null || die "Could not restart with temp admin on $host."
  # Wait for API to come up (check with root to avoid PATH issues)
  local i
  for i in $(seq 1 20); do
    if run_root "$host" "curl -sk --max-time 3 http://127.0.0.1:5380/api/config/getVersion 2>/dev/null" 2>/dev/null | grep -q .; then
      log "Temp admin API is up on $host."
      # Give the auth subsystem a few seconds to finish creating the admin user.
      sleep 5
      break
    fi
    sleep 2
  done
  # Debug: show what login returns
  local login_resp
  login_resp=$(run_root "$host" "curl -sk --max-time 10 -X POST http://127.0.0.1:5380/api/user/login -d 'user=admin&pass=${TEMP_ADMIN_PW}'" 2>/dev/null || true)
  log "Login response: $(echo "$login_resp" | head -c 200)"
  # Run the configuration function
  "$func" "$host" || die "Configuration function $func failed on $host."
  # Restore: production auth.config + remove override + restart
  log "Restoring production auth.config (with 2FA) on $host..."
  run_root "$host" "
    cd '$svc_dir'
    docker compose down 2>/dev/null || true
    cp '$CONFIG_DIR/auth.config.production' '$CONFIG_DIR/auth.config'
    rm -f '$CONFIG_DIR/auth.config.production'
    chown 1654:1654 '$CONFIG_DIR/auth.config' 2>/dev/null || true
    rm -f docker-compose.override.yml
    docker compose up -d 2>/dev/null || true
  " || die "Could not restore auth.config on $host."
  sleep 3
  log "Production auth restored on $host."
 }
 # -----------------------------------------------------------------------------
 # Step 5: Verify
 # -----------------------------------------------------------------------------
 do_verify() {
  log "=== STEP 5: Verification ==="
  bash "$HERE/verify.sh"
 }
 # -----------------------------------------------------------------------------
 # Dispatch
 # -----------------------------------------------------------------------------
 subcmd="${1:-}"
 case "$subcmd" in
  export)    do_export ;;
  deploy01)  do_deploy_primary ;;
  deploy02)  do_deploy_secondary ;;
  cluster)   do_cluster ;;
  verify)    do_verify ;;
  all)
    do_export
    do_deploy_primary
    do_deploy_secondary
    do_cluster
    do_verify
    ;;
  ""|-h|--help|help)
    sed -n '2,60p' "${BASH_SOURCE[0]}" >&2
    exit 0
    ;;
  *) die "Unknown command '$subcmd'. Run '$0 help'." ;;
 esac
 log "=== DONE: $subcmd ==="
@@ -1,72 +0,0 @@
 #!/usr/bin/bash
 #
 # sync-zones.sh — rsync-based zone replication from primary to secondary
 #
 # Runs on the SECONDARY (netinfra-02). Syncs the zones/ directory from the
 # primary (netinfra-01) every 60 seconds. When rsync changes any zone file,
 # Technitium is reloaded (container restart) so it serves the fresh zones —
 # Technitium does NOT reliably auto-detect externally modified zone files,
 # which left the secondary serving stale records (seen 2026-09-01 [#344]).
 #
 # This is used instead of AXFR-based zone transfer because Technitium's zone
 # transfer mechanism uses port 53 (standard DNS), but on the netinfra hosts
 # port 53 is Pi-hole and Technitium is on port 5300. rsync-based replication
 # avoids the port conflict entirely. Native clustering tracked in [#469].
 #
 # Install as a systemd service/timer or run via cron:
 #   * * * * * /home/localuser/services/technitium/sync-zones.sh
 #
 set -uo pipefail
 PRIMARY_HOST="${PRIMARY_HOST:-pfv-netinfra-01.knel.net}"
 CONFIG_DIR="${CONFIG_DIR:-/home/localuser/services/technitium/config}"
 ZONE_DIR="$CONFIG_DIR/zones"
 LOCK_FILE="/tmp/technitium-zone-sync.lock"
 LOG_FILE="${LOG_FILE:-/home/localuser/services/technitium/sync.log}"
 # Reload command; array form so tests can substitute a fake.
 read -r -a RELOAD_CMD <<< "${RELOAD_CMD:-docker restart tsys-dns}"
 export RELOAD_CMD
 log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >> "$LOG_FILE"; }
 # Sync zones from the primary and reload Technitium when anything changed.
 # Returns non-zero when rsync fails (reload failures are logged, not fatal).
 sync_and_reload() {
  local changed rc zone_count changed_count
  # --temp-dir avoids partial writes; --delete removes zones deleted on
  # primary; --itemize-changes reports exactly what changed.
  changed=$(rsync -az --delete --itemize-changes --out-format='%n' --temp-dir=/tmp \
    "${PRIMARY_HOST}:$ZONE_DIR/" "$ZONE_DIR/" 2>> "$LOG_FILE")
  rc=$?
  if [ "$rc" -ne 0 ]; then
    return "$rc"
  fi
  zone_count=$(find "$ZONE_DIR" -maxdepth 1 -type f | wc -l)
  if [ -n "$changed" ]; then
    changed_count=$(printf '%s\n' "$changed" | wc -l)
    log "Sync complete: $zone_count zones, $changed_count changed — reloading Technitium"
    if "${RELOAD_CMD[@]}" >> "$LOG_FILE" 2>&1; then
      log "Technitium reload triggered"
    else
      log "WARN: reload command failed (rc=$?) — records may be stale until next change"
    fi
  else
    log "Sync complete: $zone_count zones (no changes)"
  fi
 }
 main() {
  # Prevent overlapping runs
  exec 9>"$LOCK_FILE" || exit 0
  flock -n 9 || { log "another sync is running; skipping"; exit 0; }
  mkdir -p "$ZONE_DIR"
  log "Syncing zones from $PRIMARY_HOST..."
  sync_and_reload || { log "ERROR: rsync failed (rc=$?)"; exit 1; }
 }
 # Allow sourcing (unit tests) without executing the sync
 if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
  main
 fi
@@ -1,206 +0,0 @@
 #!/usr/bin/bash
 #
 # verify.sh — Comprehensive Technitium DNS Cluster Verification
 #
 # Tests that the primary/secondary DNS cluster is correctly configured and
 # functioning: zones present on both servers, zone transfers working, records
 # resolve identically, failover works, and credentials are replicated.
 #
 set -uo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REMOTE="$HERE/remote-dns.sh"
 PRIMARY="netinfra01"
 SECONDARY="netinfra02"
 # The retired tailscale-router was the old PROD comparison target; the
 # primary itself is production since 2026-09-02.
 PROD="$PRIMARY"
 PRIMARY_IP="${PRIMARY_IP:-192.168.3.252}"
 SECONDARY_IP="${SECONDARY_IP:-192.168.3.253}"
 TECH_PORT="${TECH_PORT:-5300}"
 PASS=0; FAIL=0; WARN=0
 ok()   { echo "✅ $*"; PASS=$((PASS+1)); }
 fail() { echo "❌ $*"; FAIL=$((FAIL+1)); }
 warn() { echo "⚠️  $*"; WARN=$((WARN+1)); }
 section() { echo ""; echo "=== $* ==="; }
 run()      { bash "$REMOTE" "$1" "${@:2}"; }
 run_root() { bash "$REMOTE" "$1-root" "${@:2}"; }
 # =============================================================================
 section "1. Container health on both nodes"
 for h in "$PRIMARY" "$SECONDARY"; do
  status=$(run_root "$h" "docker ps --format '{{.Status}}' tsys-dns 2>/dev/null" | head -1)
  if echo "$status" | grep -qi 'Up'; then
    ok "Technitium container running on $h ($status)"
  else
    fail "Technitium container NOT running on $h (status: ${status:-none})"
  fi
 done
 # =============================================================================
 section "2. Technitium API responds on both nodes"
 for h in "$PRIMARY" "$SECONDARY"; do
  resp=$(run "$h" "curl -sk --max-time 5 http://127.0.0.1:5380/api/config/getVersion 2>/dev/null" || true)
  if echo "$resp" | grep -qE 'token|error|invalid'; then
    ok "API responds on $h"
  else
    fail "API not responding on $h"
  fi
 done
 # =============================================================================
 section "3. Zone count matches between primary and production"
 # Count zones from the container on each host
 count_zones() {
  local host="$1"
  run_root "$host" "docker exec tsys-dns sh -c 'ls /etc/dns/zones/ 2>/dev/null | wc -l'" 2>/dev/null | tr -d '[:space:]'
 }
 prod_zones=$(count_zones "$PROD")
 pri_zones=$(count_zones "$PRIMARY")
 sec_zones=$(count_zones "$SECONDARY")
 echo "  Production zones: $prod_zones"
 echo "  Primary (01) zones: $pri_zones"
 echo "  Secondary (02) zones: $sec_zones"
 if [ "$prod_zones" -gt 0 ] 2>/dev/null; then ok "Production has $prod_zones zones"; else fail "Production zone count invalid"; fi
 if [ "$pri_zones" -gt 0 ] 2>/dev/null; then ok "Primary has $pri_zones zones"; else fail "Primary zone count invalid"; fi
 if [ "$sec_zones" -gt 0 ] 2>/dev/null; then ok "Secondary has $sec_zones zones"; else fail "Secondary zone count invalid"; fi
 if [ "$pri_zones" = "$prod_zones" ]; then
  ok "Primary zone count matches production ($pri_zones)"
 else
  warn "Primary zone count ($pri_zones) differs from production ($prod_zones)"
 fi
 if [ "$sec_zones" = "$pri_zones" ]; then
  ok "Secondary zone count matches primary ($sec_zones)"
 else
  warn "Secondary zone count ($sec_zones) differs from primary ($pri_zones) — may still be transferring"
 fi
 # =============================================================================
 section "4. knel.net zone resolves identically on primary and secondary"
 # Query a known record on both servers directly via Technitium's port
 for name in pfv-netinfra-01 pfv-netinfra-02 tsys-cloudron tsys-siem tsys-wazuh; do
  fqdn="${name}.knel.net"
  # Query via dig against each Technitium instance (through Pi-hole on :53)
  pri_ans=$(run "$PRIMARY" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 $fqdn A 2>/dev/null | head -1" 2>/dev/null || true)
  sec_ans=$(run "$SECONDARY" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 $fqdn A 2>/dev/null | head -1" 2>/dev/null || true)
  if [ -n "$pri_ans" ] && [ "$pri_ans" = "$sec_ans" ]; then
    ok "$fqdn resolves identically: $pri_ans"
  elif [ -n "$pri_ans" ] && [ -z "$sec_ans" ]; then
    warn "$fqdn: primary=$pri_ans secondary=<no answer> (may still be syncing)"
  elif [ -z "$pri_ans" ] && [ -z "$sec_ans" ]; then
    warn "$fqdn: no answer on either server"
  else
    fail "$fqdn MISMATCH: primary=$pri_ans secondary=$sec_ans"
  fi
 done
 # =============================================================================
 section "5. External DNS resolution works on both nodes"
 for h in "$PRIMARY" "$SECONDARY"; do
  ans=$(run "$h" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 github.com A 2>/dev/null | head -1" 2>/dev/null || true)
  if [ -n "$ans" ]; then
    ok "$h resolves github.com → $ans"
  else
    fail "$h cannot resolve github.com"
  fi
 done
 # =============================================================================
 section "6. Zone transfer (AXFR) from primary to secondary"
 # Test AXFR of knel.net from the primary
 axfr=$(run "$SECONDARY" "dig +short +time=5 +tries=1 @${PRIMARY_IP} -p ${TECH_PORT} knel.net AXFR 2>/dev/null | wc -l" 2>/dev/null || echo "0")
 if [ "$axfr" -gt 1 ] 2>/dev/null; then
  ok "AXFR of knel.net from primary succeeds ($axfr records transferred)"
 else
  warn "AXFR test returned $axfr records — zone transfer may be restricted or in progress"
 fi
 # =============================================================================
 section "7. Reverse DNS works"
 # Pick a known reverse zone and test PTR resolution
 ptr_test="181.103.100.in-addr.arpa"
 ptr_ans=$(run "$PRIMARY" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 $ptr_test SOA 2>/dev/null | head -1" 2>/dev/null || true)
 if [ -n "$ptr_ans" ]; then
  ok "Reverse zone $ptr_test has SOA on primary"
 else
  warn "Reverse zone $ptr_test: no SOA on primary"
 fi
 ptr_ans2=$(run "$SECONDARY" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 $ptr_test SOA 2>/dev/null | head -1" 2>/dev/null || true)
 if [ -n "$ptr_ans2" ]; then
  ok "Reverse zone $ptr_test has SOA on secondary"
 else
  warn "Reverse zone $ptr_test: no SOA on secondary"
 fi
 # =============================================================================
 section "8. Production untouched (read-only verification)"
 # Verify production container is still running and unchanged
 prod_status=$(run_root "$PROD" "docker ps --format '{{.Status}}' tsys-dns 2>/dev/null" | head -1)
 if echo "$prod_status" | grep -qi 'Up'; then
  ok "Production container still running on $PROD ($prod_status)"
 else
  fail "Production container NOT running on $PROD!"
 fi
 prod_zones_after=$(count_zones "$PROD")
 if [ "$prod_zones_after" = "$prod_zones" ]; then
  ok "Production zone count unchanged ($prod_zones_after = $prod_zones before)"
 else
  fail "Production zone count CHANGED: $prod_zones → $prod_zones_after"
 fi
 # =============================================================================
 section "9. Failover test"
 # Take the approach of querying via the secondary when primary is slow/unavailable.
 # We test that the secondary answers independently.
 sec_soa=$(run "$SECONDARY" "dig +short +time=3 +tries=1 @127.0.0.1 -p 53 knel.net SOA 2>/dev/null | head -1" 2>/dev/null || true)
 if [ -n "$sec_soa" ]; then
  ok "Secondary independently serves knel.net SOA: $sec_soa"
 else
  fail "Secondary cannot serve knel.net SOA independently"
 fi
 # =============================================================================
 section "10. Credentials check — auth.config size matches production"
 prod_auth_size=$(run_root "$PROD" "docker exec tsys-dns wc -c < /etc/dns/auth.config 2>/dev/null" | tr -d '[:space:]')
 pri_auth_size=$(run_root "$PRIMARY" "docker exec tsys-dns wc -c < /etc/dns/auth.config 2>/dev/null" | tr -d '[:space:]')
 sec_auth_size=$(run_root "$SECONDARY" "docker exec tsys-dns wc -c < /etc/dns/auth.config 2>/dev/null" | tr -d '[:space:]')
 echo "  auth.config sizes — prod=$prod_auth_size pri=$pri_auth_size sec=$sec_auth_size"
 if [ "$prod_auth_size" = "$pri_auth_size" ] && [ "$prod_auth_size" = "$sec_auth_size" ]; then
  ok "auth.config identical size across all three nodes (credentials + 2FA replicated)"
 else
  fail "auth.config sizes differ — credentials may not be replicated correctly"
 fi
 # =============================================================================
 # Summary
 echo ""
 echo "=========================================="
 echo "  PASSED: $PASS"
 echo "  FAILED: $FAIL"
 echo "  WARNED: $WARN"
 echo "=========================================="
 [ "$FAIL" -eq 0 ] && exit 0 || exit 1
@@ -1,115 +0,0 @@
 #!/usr/bin/bash
 #
 # drift-check.sh — compare live netinfra DNS/DHCP config against git [#420][#469]
 #
 # Verifies that the running configuration on netinfra-01/02 matches the
 # files tracked in this repo. Founded after the 2026-09-01 DNS incident:
 # the live systems are production; git is the source of truth; drift is
 # a defect.
 #
 # Checks per node:
 #   - /etc/dhcp/dhcpd.conf        vs netinfra/dhcp/dhcpd-{primary,secondary}.conf
 #   - /etc/pihole/pihole.toml     vs netinfra/dns/pihole/netinfra-0{1,2}.pihole.toml
 #     (secrets redacted on both sides before compare; "Last updated" line ignored)
 #   - /etc/ntpsec/ntp.conf        vs netinfra/ntp/ntp.conf (nodes are identical)
 #   - Technitium zones (node 01, the replication primary): md5 manifest of the
 #     binary DZ store vs netinfra/dns/technitium/zones/ [#630]
 #
 # Usage:
 #   drift-check.sh [--node 01|02|all]   (default: all)
 # Exit: 0 = in sync, 1 = drift detected, 2 = fetch failure
 #
 set -uo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 DNS_SETUP="$HERE/dns-cluster-setup"
 DHCP_DIR="$HERE/dhcp"
 PIHOLE_DIR="$HERE/dns/pihole"
 NTP_FILE="$HERE/ntp/ntp.conf"
 ZONES_DIR="$HERE/dns/technitium/zones"
 REMOTE_ZONE_DIR="/home/localuser/services/technitium/config/zones"
 REDACT='s/^( *pwhash *=).*/\1 "REDACTED"/; s/^( *totp_secret *=).*/\1 "REDACTED"/; s/^( *password *=).*/\1 "REDACTED"/'
 # redact_config <stdin> <stdout> — strip secrets from pihole.toml content
 redact_config() { sed -E "$REDACT"; }
 # normalize_toml <stdin> <stdout> — redact + drop churn lines (timestamps)
 normalize_toml() { redact_config | grep -v "Last updated on"; }
 # gen_manifest <dir> — sorted 'md5  name' manifest of *.zone files (name-sorted)
 gen_manifest() { (cd "$1" && md5sum -- *.zone 2>/dev/null | sort -k2); }
 # fetch <node> <remote-cmd> — run via the remote-dns.sh chokepoint (env IPs honored)
 fetch() {
  local node="$1" cmd="$2"
  bash "$DNS_SETUP/remote-dns.sh" "netinfra${node}-root" "$cmd" 2>/dev/null
 }
 # check_file <label> <node> <remote-cat-cmd> <local-file> <normalize-fn>
 check_file() {
  local label="$1" node="$2" rcmd="$3" local_file="$4" norm="$5" tmp rc
  tmp="$(mktemp)"
  fetch "$node" "$rcmd" | "$norm" > "$tmp"
  if [ ! -s "$tmp" ]; then
    echo "DRIFT-ERROR: $label: live fetch empty (node $node unreachable?)"
    rm -f "$tmp"
    return 2
  fi
  "$norm" < "$local_file" | diff -q - "$tmp" >/dev/null 2>&1
  rc=$?
  if [ "$rc" -ne 0 ]; then
    echo "DRIFT: $label (node $node) differs from git: $local_file"
    "$norm" < "$local_file" | diff - "$tmp" | head -10
  else
    echo "OK: $label (node $node) in sync"
  fi
  rm -f "$tmp"
  return "$rc"
 }
 main() {
  local nodes="${1:-all}" node rc_total=0 rc
  [ "$nodes" = "all" ] && nodes="01 02"
  for node in $nodes; do
    local_dhcp="$DHCP_DIR/dhcpd-primary.conf"
    [ "$node" = "02" ] && local_dhcp="$DHCP_DIR/dhcpd-secondary.conf"
    local_pihole="$PIHOLE_DIR/netinfra-01.pihole.toml"
    [ "$node" = "02" ] && local_pihole="$PIHOLE_DIR/netinfra-02.pihole.toml"
    check_file "dhcpd.conf" "$node" "cat /etc/dhcp/dhcpd.conf" "$local_dhcp" cat || rc_total=1
    check_file "pihole.toml" "$node" \
      "docker exec pihole cat /etc/pihole/pihole.toml" "$local_pihole" normalize_toml || rc_total=1
    check_file "ntp.conf" "$node" "cat /etc/ntpsec/ntp.conf" "$NTP_FILE" cat || rc_total=1
  done
  # Technitium zones: binary DZ files, compared by md5 manifest against the
  # primary (netinfra-01) — the replication source for -02.
  if [[ "$nodes" == *01* ]]; then
    remote_manifest="$(fetch 01 "cd $REMOTE_ZONE_DIR && md5sum -- *.zone | sort -k2")"
    if [ -z "$remote_manifest" ]; then
      echo "DRIFT-ERROR: technitium zones: live fetch empty (node 01 unreachable?)"
      rc_total=1
    else
      if diff -q <(gen_manifest "$ZONES_DIR") <(printf '%s\n' "$remote_manifest") >/dev/null 2>&1; then
        echo "OK: technitium zones (node 01) in sync"
      else
        echo "DRIFT: technitium zones (node 01) manifest differs from git: $ZONES_DIR"
        diff <(gen_manifest "$ZONES_DIR") <(printf '%s\n' "$remote_manifest") | head -10
        rc_total=1
      fi
    fi
  fi
  if [ "$rc_total" -eq 0 ]; then
    echo "drift-check: ALL IN SYNC"
  else
    echo "drift-check: DRIFT DETECTED — reconcile git <-> live before any change"
  fi
  return "$rc_total"
 }
 if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
  node="${2:-all}"
  [ "${1:-}" = "--node" ] && node="${2:-}" || node="all"
  main "$node"
 fi
@@ -1,57 +0,0 @@
 #!/usr/bin/bash
 #
 # zone-snapshot.sh — refresh the git SoR snapshot of the Technitium zone
 # store [#630][#728]
 #
 # Pulls the binary DZ zone files from netinfra-01 (the replication primary)
 # into netinfra/dns/technitium/zones/ via the remote-dns.sh chokepoint, then
 # shows which snapshot files changed.
 #
 # RULE (founder, 2026-09-02): run this after EVERY Technitium record change
 # and commit the refreshed snapshots in the same session, so the git SoR
 # never goes stale. netinfra/dns/drift-check.sh verifies the result; the
 # pre-push rule audit fails on a stale questions file and drift-check is the
 # DNS counterpart: green before you push any DNS-adjacent change.
 #
 # Usage:
 #   zone-snapshot.sh            refresh snapshots from the primary
 #   zone-snapshot.sh --check    verify only (delegates to drift-check.sh)
 #
 set -euo pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"          # netinfra/dns/technitium
 REPO="$(cd "$HERE/../../.." && pwd)"
 REMOTE_DNS="$HERE/../../dns-cluster-setup/remote-dns.sh"
 ZONES_DIR="$HERE/zones"
 REMOTE_ZONE_DIR="/home/localuser/services/technitium/config/zones"
 if [ "${1:-}" = "--check" ]; then
  exec bash "$HERE/../drift-check.sh"
 fi
 tmp="$(mktemp -d)"
 trap 'rm -rf "$tmp"' EXIT
 echo "Fetching zone store from netinfra-01 (primary)..."
 bash "$REMOTE_DNS" netinfra01-root "cd '$REMOTE_ZONE_DIR' && tar cf - *.zone" > "$tmp/zones.tar"
 # Validate the archive before touching the tracked tree.
 tar -tf "$tmp/zones.tar" >/dev/null
 mkdir -p "$ZONES_DIR"
 find "$ZONES_DIR" -maxdepth 1 -name '*.zone' -delete
 tar -xf "$tmp/zones.tar" -C "$ZONES_DIR"
 count="$(find "$ZONES_DIR" -maxdepth 1 -name '*.zone' | wc -l)"
 echo "Snapshotted $count zone files into ${ZONES_DIR#"$REPO"/}"
 echo "Changed snapshot files (git):"
 changed="$(git -C "$REPO" status --porcelain -- "$ZONES_DIR")"
 if [ -n "$changed" ]; then
  printf '%s\n' "$changed" | sed "s|^$REPO/||" | head -20
  printf '%s\n' "$changed" | wc -l | xargs -I{} echo "{} files changed — commit these in the same session (DNS sync rule)"
 else
  echo "  (none — snapshots were already current)"
 fi
 echo "Verify with: netinfra/dns/drift-check.sh"
--- a/Show More
+++ b/Show More