chore: reconcile live working copy from reachableceo lift/shift [#478]

Phase A of the reachableceo drop merge. Working-tree diff (excluding
.git/.ansible/.crush/.fact_cache) showed zero content differences vs
this clone; drop HEAD matched df22156. Preserved the two loose analysis
notes from the drop root under docs/. Session state (.ansible,
.fact_cache) archived outside git per procedure.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

docs/kneliac-basis-analysis.md

@@ -0,0 +1,169 @@
+# KNELIAC Basis Analysis: KNELServerBuild → KNELIAC, and PFVCluster netinfra relocation (Redmine #472)
+
+Date: 2026-08-28. Sources inspected read-only: `KNEL/KNELServerBuild`, `KNEL/KNELIAC`, `PFVCluster/netinfra` (+ targeted greps elsewhere in PFVCluster).
+
+---
+
+## 1. KNELServerBuild inventory (legacy bash fleet build)
+
+**Self-description:** "KNEL FetchApply" — TSYS IaC via bash, intended to run under FetchApply (`P5vc/fetch-apply`), which never materialized.
+
+### Top-level layout
+
+| Path | Contents |
+|---|---|
+| `ProjectCode/SetupNewSystem.sh` | 431-line monolithic runner; `global-*` phase functions |
+| `ProjectCode/Modules/{Security,OAM,Auth,RandD}` | 9 bash modules (~830 lines total) |
+| `ProjectCode/ConfigFiles/` | 17 static config dirs (SSH, SNMP, NTP, DHCP, Syslog, AuditD, ZSH, BANNERS, ...) |
+| `ProjectCode/Agents/librenms` | PHP files disguised as `.sh` (`mysql.sh`, `#!/usr/bin/php`) |
+| `ProjectCode/{Dell,scripts,legacy}` | Dell CPU perf fix, `up2date.sh`, dead profile scripts |
+| `Project-Includes/` | `PreflightCheck.sh`, `pi-detect.sh`, `LocalHelp.sh` |
+| `Project-ConfigFiles/CONFIG_VARS` | `DL_ROOT` pointing at deprecated `dl.knownelement.com` CDN |
+| `Project-Tests/{unit,security,validation}` | Bash tests incl. `dns-ntp-redundancy.sh` (duplicated in PFVCluster) |
+| `dns-cluster-setup/` | Technitium cluster scripts at repo ROOT (outside ProjectCode; also duplicated in PFVCluster) |
+| `ProjectDocs/` | 13 docs, mostly AI review/audit artifacts + todo files |
+| `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework` | Runtime-sourced shell framework (SSH URL as directory name) |
+| `logs/` | Runtime logs inside repo tree |
+
+### Patterns / languages
+
+- Bash; runner sources vendored `KNELShellFramework` + `Project-Includes/*` at startup (`SetupNewSystem.sh:28-36`).
+- Self-locating via `BASH_SOURCE`; host-class detection: Dell dmidecode → physical, `virt-what` → guest, Pi detect, distro checks.
+- Modules = `secharden-*.sh` (security) / `oam-*.sh` (monitoring); configs deployed via `cat`/`cp` from `ConfigFiles/`.
+- ISC dhcpd / ntp / snmpd conf, PHP agents, embedded systemd units in heredocs.
+
+### Generic vs PFV-specific
+
+| Generic (fleet-wide) | PFV-specific |
+|---|---|
+| All Security modules, OAM/LibreNMS, packages, shells, banners, Dell fixes | `dns-cluster-setup/` (pfv-netinfra-01/02 Technitium, LAN IPs 192.168.3.x hardcoded) |
+| ConfigFiles for SSH/SNMP/NTP/DHCP-client/SMTP | `Project-Tests/validation/dns-ntp-redundancy.sh` (192.168.3.252/.253 hardcoded) |
+
+### Quality / rot assessment
+
+- **Documented debt:** `ProjectDocs/CODE-REVIEW-FINDINGS.md` (2025-07-14) lists unquoted expansions, `curl` without timeout/retry, repeated `apt-get` invocations.
+- **Fragile foundation:** runtime sourcing of a vendored framework whose path is an SSH URL; `CONFIG_VARS` still points at the deprecated CDN (contradicted by its own AGENTS.md).
+- **Drift:** `dns-cluster-setup` exists in two repos; the PFVCluster copy carries fixes (`find` vs `ls`, `if/else` vs `&&/||`) the KNELServerBuild copy lacks — the "wrong" copy is in the IaC repo.
+- **Rot signals:** `legacy/` dir, 3-line stub module (`secharden-auto-upgrade.sh`), committed `logs/`, 13 process-artifact docs, FetchApply framing with no FetchApply.
+- **Verdict:** architecture (phases, modules, preflight, tests) is sound and was deliberately ported; the bash implementation is end-of-life. Treat as *specification*, not code to salvage verbatim.
+
+---
+
+## 2. KNELIAC current state
+
+Already a working Ansible repo (not greenfield), executed by AWX (`tsys-awx.knel.net`) against Debian-family hosts.
+
+| Item | State |
+|---|---|
+| `roles/` | `preflight`, `oam`, `packages`, `system_config`, `security_ssh`, `security_wazuh`, `security_scap_stig`, `security_2fa`, `security_audit` |
+| `playbooks/` | `hello_world.yml` (AWX smoke), `setup_new_system.yml` (full build, port of the bash runner), `setup_awx_https.yml` (host nginx TLS-termination proxy in front of k3s ServiceLB; self-signed cert w/ documented SANs) |
+| `inventory/hosts.yml` | Functional groups; **already contains `ntp_servers`, `dhcp_servers`, `dns_servers` groups with `pfv-netinfra-01/02`** |
+| `inventory/group_vars/all.yml` | Mirrors legacy hard-coded values; `run_*` phase toggles; `dns_servers`/`ntp_servers` = 192.168.3.252/.253 |
+| `scripts/awx_create_job_templates.py` | AWX job-template bootstrap (190 lines) |
+| Config/tooling | `ansible.cfg`, `.ansible-lint`, `requirements.yml` (empty — no collections yet), `.gitignore` |
+
+**Conventions observed**
+
+- Naming: `<area>_<thing>` roles (`security_ssh`, `system_config`); playbooks `setup_<purpose>.yml`.
+- `preflight` derives facts from group membership (`roles/preflight/tasks/main.yml:109-115`): `is_ntp_server`, `is_dhcp_server`, `is_dns_server`, ...
+- `system_config` already skips `dhclient.conf` on DHCP servers (`when: not is_dhcp_server`) — the server-side hook point exists.
+- snmpd.conf selected per host class (pi / physical / virtual / default).
+- `README.md` carries a legacy→Ansible mapping table; `AGENTS.md` carries a strict secrets policy (vault/env only, history purge on leak) and auto commit/push policy.
+- Note: `group_vars/all.yml` holds the SNMP RO community in plaintext today — tension with the AGENTS.md policy to resolve during the #472 work (move to Vault or AWX credentials).
+
+---
+
+## 3. KNELServerBuild → KNELIAC: what ports, what dies
+
+**Reality check:** the fleet-build port is ~done (the 9 roles + mapping table in KNELIAC README cover all `Modules/` + runner phases). Remaining port work is small:
+
+| Legacy item | Disposition |
+|---|---|
+| Modules/Security, OAM, runner phases, Preflight, pi-detect | ✅ Already ported — verify parity only |
+| `Project-Tests/*` | Port as role self-tests / assert tasks; `dns-ntp-redundancy.sh` folds into #472 verification |
+| `ProjectDocs/*` guides (TSYS-2FA, DEPLOYMENT, SECURITY) | Cherry-pick still-true content into KNELIAC docs; drop AI-review artifacts |
+| `dns-cluster-setup/` (repo root) | **Retire** — superseded by PFVCluster copy (which has the fixes) and then by #472 roles |
+| `CONFIG_VARS` DL_ROOT, FetchApply framing, `legacy/`, Dell/, Agents PHP | Drop; capture intent in KNELIAC wiki/README if still relevant |
+
+### Recommended KNELIAC skeleton (additions for #472)
+
+```
+KNELIAC/
+├── playbooks/
+│   └── setup_netinfra.yml          # repave playbook: preflight → base → netinfra roles → verify
+├── roles/
+│   ├── dhcp_server/                # ISC dhcpd failover pair (from netinfra/dhcp/)
+│   │   ├── tasks/  handlers/  templates/dhcpd.conf.j2
+│   ├── dns_pihole/                 # compose + dnsnet + gravity (from netinfra/pihole/, setup-netinfra.sh)
+│   │   └── tasks/ templates/docker-compose.yml.j2  handlers/
+│   ├── dns_technitium/             # compose, config bootstrap, zone-sync units
+│   │   └── tasks/ templates/ files/sync-zones.sh  handlers/
+│   └── ntp_server/                 # ntpsec (from host state; see gaps)
+│       └── tasks/ templates/ntp.conf.j2
+├── inventory/
+│   ├── group_vars/netinfra_servers.yml   # failover params, dnsnet subnet, ports, probe names
+│   └── host_vars/pfv-netinfra-0{1,2}.yml # dhcp_role: primary|secondary, lan_ip, tailscale_ip
+└── scripts/                         # keep verify.sh-style bash here or convert to asserts
+```
+
+- Naming follows existing convention (`dhcp_server` ~ `security_ssh`); one playbook `setup_netinfra.yml` matches `setup_new_system.yml`.
+- `system_config` gains a netinfra snmpd.conf variant (extends/ACL) selected by `is_dns_server`/group — smallest change, matches its existing per-class selection pattern.
+- Decide on `community.docker` vs `ansible.builtin.command`+compose for containers; if the former, it becomes the first entry in `requirements.yml`.
+
+---
+
+## 4. Gap analysis: PFVCluster `netinfra/` → KNELIAC (#472)
+
+### Current source of truth (PFVCluster/netinfra, read-only)
+
+| Item | File(s) | Notes |
+|---|---|---|
+| Node repave orchestrator | `setup-netinfra.sh` (322 L) | SSH-driven; deploys compose services under `/home/localuser/services/{pihole,ntp,technitium}`; relays config from `pfv-netboot` (read-only ref) via workstation tarball; per-node Tailscale IPs at lines 42-45 |
+| DHCP failover pair | `dhcp/dhcpd-{primary,secondary}.conf` (332/331 L) | Peer `pfv-dhcp`, hot-standby `split 255` + `load balance max seconds 3`; subnet 192.168.0.0/22; DNS/NTP options hand out both nodes; header says "Managed via Webmin" |
+| DHCP install | `dhcp/install-dhcp.sh` | isc-dhcp-server + Webmin; `INTERFACESv4=ens18`; installs disabled |
+| Pi-hole | `pihole/docker-compose.yml`, `.env.example`, `gravity-validate.sh` | :53, web 10002/10003; `.env.example` correctly redacted |
+| Technitium cluster | `dns-cluster-setup/{setup,remote-dns,sync-zones,verify}.sh` | 01 primary / 02 secondary; **zone sync = rsync via `technitium-zone-sync.service` + `.timer`** (60 s), installed by `setup.sh:253-307`; rsync chosen because Technitium AXFR uses :53 which Pi-hole owns ( Technitium on :5300) |
+| snmpd | `snmp/snmpd-netinfra-0{1,2}.conf` | Scoped ACL (LibreNMS host + Tailscale only, founder ruling 2026-08-27); `extend` for ntpq + dhcpd lease counts |
+| NTP | (no repo file) | Host bare-metal **ntpsec** serves :123 incl. Tailscale IP; chrony container deliberately removed as redundant (`setup-netinfra.sh` §6b) |
+| Switch configs | `switches/*.cmds` | Out of scope for KNELIAC (device CLI dumps) — leave in PFVCluster |
+| Docs | `pfv-netinfra-setup.md`, `dhcp-migration.md`, `dns-cluster-setup/README.md` | Already pointer stubs to Discourse topic 306 |
+
+### Mapping into KNELIAC
+
+| #472 component | KNELIAC home | Conversion |
+|---|---|---|
+| dhcpd failover pair | `roles/dhcp_server` | Two confs → one `dhcpd.conf.j2` + `dhcp_role: primary\|secondary` host_var; params (peer name, mclt, split, subnet, options) to `group_vars/netinfra_servers.yml`; `install-dhcp.sh` → `apt` + `template` + handlers (`dhcpd -t` validate → restart) |
+| Pi-hole container | `roles/dns_pihole` | compose → template; password via Vault/AWX credential, never group_vars (matches `.env.example` stance + AGENTS.md) |
+| Technitium container + zone sync | `roles/dns_technitium` | compose → template (dnsnet fixed IP 10.53.0.53); `sync-zones.sh` → `files/` + systemd unit/timer templates; **idempotent: never wipe `config/` on re-run** (keep quarantine-fallback semantics of `setup-netinfra.sh` §8) |
+| ntpsec | `roles/ntp_server` | **Gap: no managed ntp.conf exists anywhere in the repo** — capture from live hosts first, then template; gate on `is_ntp_server` |
+| snmpd | extend `roles/system_config` | Add netinfra conf variant (scoped ACL + extends) selected by group; RO community moves to Vault (it is currently plaintext in KNELIAC `group_vars/all.yml:49` and in PFV snmpd confs) |
+| Verification | `playbooks/setup_netinfra.yml` verify block or `scripts/` | Port `verify.sh` + `tests/validation/dns-ntp-redundancy.sh` (dig/ntpq probes already codified there) |
+
+### Gaps / risks
+
+1. **ntpsec config unmanaged** — lives only on hosts (and stale in `netboot-audit.txt`). Must be captured before repave or NTP service definition is lost.
+2. **Webmin hand-edits** — dhcpd confs are declared "Managed via Webmin"; templates will fight Webmin. Decide: Webmin becomes read-only lease viewer, or is dropped from netinfra nodes.
+3. **Secrets** — Technitium `auth.config` (admin creds + 2FA) is copied host-to-host by `setup.sh`; it must NOT enter git. Keep out of repo; rotate to env/vault or accept manual one-time bootstrap.
+4. **Zone data gravity** — zones live inside Technitium config dir on hosts. Repave playbook must be zone-preserving (no `state: absent` on data dirs).
+5. **Duplicate sources** — `dns-cluster-setup` ×2 repos, `dns-ntp-redundancy.sh` ×2 repos, with drift favoring the PFV copies; port from **PFVCluster** versions (they have the fixes), then stub both old locations.
+6. **No collections yet** — container management approach undecided (`requirements.yml` empty).
+7. **Legacy bootstrap pattern** — `setup-netinfra.sh` relays config *from pfv-netboot*; the Ansible repave should bootstrap from first principles (templates + captured zones) so the retired netboot host is no longer a deploy-time dependency.
+
+---
+
+## 5. Migration sequencing (founder wants dns/ntp/dhcp code relocated PFVCluster → KNELIAC)
+
+Blast radius is small: no Makefile/AWX/test references to `netinfra/` exist outside the directory itself; docs are already Discourse stubs.
+
+| Phase | Action | Why this order |
+|---|---|---|
+| 0. Capture | Pull live `/etc/ntpsec/ntp.conf` from both nodes; snapshot Technitium `config/` + Pi-hole gravity as offline backups (not committed) | Closes gap #1; makes repave non-destructive |
+| 1. Scaffolding | Add `netinfra_servers` group_vars + host_vars (roles primary/secondary, IPs); `requirements.yml` decision | Pure additive; no behavior change |
+| 2. `ntp_server` role | Template captured ntp.conf; deploy to both nodes; verify with `dns-ntp-redundancy.sh` probes | Least coupled service; proves AWX pipeline to these hosts |
+| 3. `dhcp_server` role | Convert pair confs to template + failover vars; `dhcpd -t` gate; flip from Webmin-managed to Ansible-managed in one maintenance window | Configs already declarative → low conversion risk; highest service impact, so do while DNS/NTP verified stable |
+| 4. `dns_pihole` + `dns_technitium` roles | Compose templates, dnsnet, zone-sync units; **port PFVCluster versions** of the scripts; zone-preserving idempotency; run `verify.sh` port after | Containers are self-contained; quarantine fallback already designed |
+| 5. snmp + AWX wiring | netinfra snmpd variant; add job templates via `scripts/awx_create_job_templates.py`; snmp community → Vault | Cosmetic until monitoring depends on it |
+| 6. Decommission in PFVCluster | Replace `netinfra/{dhcp,dns-cluster-setup,snmp,pihole,setup-netinfra.sh}` with pointer READMEs (→ KNELIAC paths + Discourse 306), matching the existing stub pattern; keep `switches/` and audit artifacts (`netboot-*.txt`) in place (history) | Keeps PFVCluster git-browsers correct; preserves non-IaC content |
+| 7. KNELServerBuild cleanup | Stub/remove its `dns-cluster-setup/` copy and duplicated validation test | Ends three-way drift |
+
+**Keeping PFVCluster correct during transition:** each phase leaves the old script in place until the KNELIAC role has run green against both nodes (toggle with `run_*` per repo convention); pointer stubs land only in phase 6/7; Discourse topic 306 gets a "source of truth = KNELIAC" note at phase 6. Nothing else in PFVCluster imports netinfra, so no other references need updating.

docs/kneliac-basis-summary.md

@@ -0,0 +1,15 @@
+# KNELIAC Basis — Executive Summary
+
+**Question:** How does legacy `KNELServerBuild` (bash) become `KNELIAC` (Ansible), and how does the PFVCluster netinfra dns/dhcp/ntp code (Redmine #472) move into KNELIAC?
+
+**Headline:** KNELIAC is not greenfield — the fleet-build port from KNELServerBuild is already ~complete (9 roles, AWX-driven, inventory already lists `pfv-netinfra-01/02` in `ntp_servers`/`dhcp_servers`/`dns_servers` groups). The real work is #472: porting PFVCluster's `netinfra/` into four new roles.
+
+**Key findings**
+
+- **KNELServerBuild**: architecturally sound bash (phases, modules, preflight, tests) but end-of-life — vendored-framework runtime dependency, deprecated CDN var, documented code-review debt, and a drifted `dns-cluster-setup` copy at repo root. Treat as *specification only*; ~90% already ported. Nothing else needs porting except tests and a few docs.
+- **KNELIAC conventions to follow:** `<area>_<thing>` role names, `preflight` facts (`is_dhcp_server`...) from inventory groups, `run_*` toggles, strict no-secrets AGENTS.md, `setup_<purpose>.yml` playbooks.
+- **#472 mapping (PFVCluster `netinfra/` → new roles):** `dhcp/` → `roles/dhcp_server` (dhcpd.conf.j2, failover vars, primary/secondary host_vars); Pi-hole + Technitium compose + `technitium-zone-sync` systemd timer → `roles/dns_pihole` / `roles/dns_technitium` (rsync sync, not AXFR — Pi-hole owns :53); host ntpsec → `roles/ntp_server`; netinfra snmpd (scoped ACL + lease extends) → variant in `system_config`. All driven by one `playbooks/setup_netinfra.yml`.
+- **Gaps:** ntpsec config exists **only on hosts** (capture before repave); Webmin hand-edits will fight templates; Technitium `auth.config` contains creds and must never enter git; zone data must survive re-runs (idempotency); SNMP community currently plaintext in group_vars → move to Vault; port from the **PFVCluster** copies of duplicated scripts (they carry the fixes).
+- **Sequencing:** (0) capture ntpsec + zone/gravity snapshots → (1) group/host_vars → (2) ntp_server role → (3) dhcp_server role → (4) DNS containers + zone-sync → (5) snmp + AWX job templates → (6) replace PFVCluster `netinfra/` subdirs with pointer stubs (Discourse 306 pattern), keep `switches/` and audit artifacts → (7) stub the KNELServerBuild duplicate. Old scripts stay until each role runs green on both nodes; nothing else in PFVCluster references `netinfra/`, so blast radius is minimal.
+
+Full detail: `kneliac-basis-analysis.md`