KNELIAC/docs/kneliac-basis-analysis.md

> Lane index + ops status: https://community.turnsys.com/t/297 — env map: https://community.turnsys.com/t/331

# KNELIAC Basis Analysis: KNELServerBuild → KNELIAC, and PFVCluster netinfra relocation (Redmine #472)

Date: 2026-08-28. Sources inspected read-only: `KNEL/KNELServerBuild`, `KNEL/KNELIAC`, `PFVCluster/netinfra` (+ targeted greps elsewhere in PFVCluster).

---

## 1. KNELServerBuild inventory (legacy bash fleet build)

**Self-description:** "KNEL FetchApply" — TSYS IaC via bash, intended to run under FetchApply (`P5vc/fetch-apply`), which never materialized.

### Top-level layout

| Path | Contents |
|---|---|
| `ProjectCode/SetupNewSystem.sh` | 431-line monolithic runner; `global-*` phase functions |
| `ProjectCode/Modules/{Security,OAM,Auth,RandD}` | 9 bash modules (~830 lines total) |
| `ProjectCode/ConfigFiles/` | 17 static config dirs (SSH, SNMP, NTP, DHCP, Syslog, AuditD, ZSH, BANNERS, ...) |
| `ProjectCode/Agents/librenms` | PHP files disguised as `.sh` (`mysql.sh`, `#!/usr/bin/php`) |
| `ProjectCode/{Dell,scripts,legacy}` | Dell CPU perf fix, `up2date.sh`, dead profile scripts |
| `Project-Includes/` | `PreflightCheck.sh`, `pi-detect.sh`, `LocalHelp.sh` |
| `Project-ConfigFiles/CONFIG_VARS` | `DL_ROOT` pointing at deprecated `dl.knownelement.com` CDN |
| `Project-Tests/{unit,security,validation}` | Bash tests incl. `dns-ntp-redundancy.sh` (duplicated in PFVCluster) |
| `dns-cluster-setup/` | Technitium cluster scripts at repo ROOT (outside ProjectCode; also duplicated in PFVCluster) |
| `ProjectDocs/` | 13 docs, mostly AI review/audit artifacts + todo files |
| `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework` | Runtime-sourced shell framework (SSH URL as directory name) |
| `logs/` | Runtime logs inside repo tree |

### Patterns / languages

- Bash; runner sources vendored `KNELShellFramework` + `Project-Includes/*` at startup (`SetupNewSystem.sh:28-36`).
- Self-locating via `BASH_SOURCE`; host-class detection: Dell dmidecode → physical, `virt-what` → guest, Pi detect, distro checks.
- Modules = `secharden-*.sh` (security) / `oam-*.sh` (monitoring); configs deployed via `cat`/`cp` from `ConfigFiles/`.
- ISC dhcpd / ntp / snmpd conf, PHP agents, embedded systemd units in heredocs.

### Generic vs PFV-specific

| Generic (fleet-wide) | PFV-specific |
|---|---|
| All Security modules, OAM/LibreNMS, packages, shells, banners, Dell fixes | `dns-cluster-setup/` (pfv-netinfra-01/02 Technitium, LAN IPs 192.168.3.x hardcoded) |
| ConfigFiles for SSH/SNMP/NTP/DHCP-client/SMTP | `Project-Tests/validation/dns-ntp-redundancy.sh` (192.168.3.252/.253 hardcoded) |

### Quality / rot assessment

- **Documented debt:** `ProjectDocs/CODE-REVIEW-FINDINGS.md` (2025-07-14) lists unquoted expansions, `curl` without timeout/retry, repeated `apt-get` invocations.
- **Fragile foundation:** runtime sourcing of a vendored framework whose path is an SSH URL; `CONFIG_VARS` still points at the deprecated CDN (contradicted by its own AGENTS.md).
- **Drift:** `dns-cluster-setup` exists in two repos; the PFVCluster copy carries fixes (`find` vs `ls`, `if/else` vs `&&/||`) the KNELServerBuild copy lacks — the "wrong" copy is in the IaC repo.
- **Rot signals:** `legacy/` dir, 3-line stub module (`secharden-auto-upgrade.sh`), committed `logs/`, 13 process-artifact docs, FetchApply framing with no FetchApply.
- **Verdict:** architecture (phases, modules, preflight, tests) is sound and was deliberately ported; the bash implementation is end-of-life. Treat as *specification*, not code to salvage verbatim.

---

## 2. KNELIAC current state

Already a working Ansible repo (not greenfield), executed by AWX (`tsys-awx.knel.net`) against Debian-family hosts.

| Item | State |
|---|---|
| `roles/` | `preflight`, `oam`, `packages`, `system_config`, `security_ssh`, `security_wazuh`, `security_scap_stig`, `security_2fa`, `security_audit` |
| `playbooks/` | `hello_world.yml` (AWX smoke), `setup_new_system.yml` (full build, port of the bash runner), `setup_awx_https.yml` (host nginx TLS-termination proxy in front of k3s ServiceLB; self-signed cert w/ documented SANs) |
| `inventory/hosts.yml` | Functional groups; **already contains `ntp_servers`, `dhcp_servers`, `dns_servers` groups with `pfv-netinfra-01/02`** |
| `inventory/group_vars/all.yml` | Mirrors legacy hard-coded values; `run_*` phase toggles; `dns_servers`/`ntp_servers` = 192.168.3.252/.253 |
| `scripts/awx_create_job_templates.py` | AWX job-template bootstrap (190 lines) |
| Config/tooling | `ansible.cfg`, `.ansible-lint`, `requirements.yml` (empty — no collections yet), `.gitignore` |

**Conventions observed**

- Naming: `<area>_<thing>` roles (`security_ssh`, `system_config`); playbooks `setup_<purpose>.yml`.
- `preflight` derives facts from group membership (`roles/preflight/tasks/main.yml:109-115`): `is_ntp_server`, `is_dhcp_server`, `is_dns_server`, ...
- `system_config` already skips `dhclient.conf` on DHCP servers (`when: not is_dhcp_server`) — the server-side hook point exists.
- snmpd.conf selected per host class (pi / physical / virtual / default).
- `README.md` carries a legacy→Ansible mapping table; `AGENTS.md` carries a strict secrets policy (vault/env only, history purge on leak) and auto commit/push policy.
- Note: `group_vars/all.yml` holds the SNMP RO community in plaintext today — tension with the AGENTS.md policy to resolve during the #472 work (move to Vault or AWX credentials).

---

## 3. KNELServerBuild → KNELIAC: what ports, what dies

**Reality check:** the fleet-build port is ~done (the 9 roles + mapping table in KNELIAC README cover all `Modules/` + runner phases). Remaining port work is small:

| Legacy item | Disposition |
|---|---|
| Modules/Security, OAM, runner phases, Preflight, pi-detect | ✅ Already ported — verify parity only |
| `Project-Tests/*` | Port as role self-tests / assert tasks; `dns-ntp-redundancy.sh` folds into #472 verification |
| `ProjectDocs/*` guides (TSYS-2FA, DEPLOYMENT, SECURITY) | Cherry-pick still-true content into KNELIAC docs; drop AI-review artifacts |
| `dns-cluster-setup/` (repo root) | **Retire** — superseded by PFVCluster copy (which has the fixes) and then by #472 roles |
| `CONFIG_VARS` DL_ROOT, FetchApply framing, `legacy/`, Dell/, Agents PHP | Drop; capture intent in KNELIAC wiki/README if still relevant |

### Recommended KNELIAC skeleton (additions for #472)

```
KNELIAC/
├── playbooks/
│   └── setup_netinfra.yml          # repave playbook: preflight → base → netinfra roles → verify
├── roles/
│   ├── dhcp_server/                # ISC dhcpd failover pair (from netinfra/dhcp/)
│   │   ├── tasks/  handlers/  templates/dhcpd.conf.j2
│   ├── dns_pihole/                 # compose + dnsnet + gravity (from netinfra/pihole/, setup-netinfra.sh)
│   │   └── tasks/ templates/docker-compose.yml.j2  handlers/
│   ├── dns_technitium/             # compose, config bootstrap, zone-sync units
│   │   └── tasks/ templates/ files/sync-zones.sh  handlers/
│   └── ntp_server/                 # ntpsec (from host state; see gaps)
│       └── tasks/ templates/ntp.conf.j2
├── inventory/
│   ├── group_vars/netinfra_servers.yml   # failover params, dnsnet subnet, ports, probe names
│   └── host_vars/pfv-netinfra-0{1,2}.yml # dhcp_role: primary|secondary, lan_ip, tailscale_ip
└── scripts/                         # keep verify.sh-style bash here or convert to asserts
```

- Naming follows existing convention (`dhcp_server` ~ `security_ssh`); one playbook `setup_netinfra.yml` matches `setup_new_system.yml`.
- `system_config` gains a netinfra snmpd.conf variant (extends/ACL) selected by `is_dns_server`/group — smallest change, matches its existing per-class selection pattern.
- Decide on `community.docker` vs `ansible.builtin.command`+compose for containers; if the former, it becomes the first entry in `requirements.yml`.

---

## 4. Gap analysis: PFVCluster `netinfra/` → KNELIAC (#472)

### Current source of truth (PFVCluster/netinfra, read-only)

| Item | File(s) | Notes |
|---|---|---|
| Node repave orchestrator | `setup-netinfra.sh` (322 L) | SSH-driven; deploys compose services under `/home/localuser/services/{pihole,ntp,technitium}`; relays config from `pfv-netboot` (read-only ref) via workstation tarball; per-node Tailscale IPs at lines 42-45 |
| DHCP failover pair | `dhcp/dhcpd-{primary,secondary}.conf` (332/331 L) | Peer `pfv-dhcp`, hot-standby `split 255` + `load balance max seconds 3`; subnet 192.168.0.0/22; DNS/NTP options hand out both nodes; header says "Managed via Webmin" |
| DHCP install | `dhcp/install-dhcp.sh` | isc-dhcp-server + Webmin; `INTERFACESv4=ens18`; installs disabled |
| Pi-hole | `pihole/docker-compose.yml`, `.env.example`, `gravity-validate.sh` | :53, web 10002/10003; `.env.example` correctly redacted |
| Technitium cluster | `dns-cluster-setup/{setup,remote-dns,sync-zones,verify}.sh` | 01 primary / 02 secondary; **zone sync = rsync via `technitium-zone-sync.service` + `.timer`** (60 s), installed by `setup.sh:253-307`; rsync chosen because Technitium AXFR uses :53 which Pi-hole owns ( Technitium on :5300) |
| snmpd | `snmp/snmpd-netinfra-0{1,2}.conf` | Scoped ACL (LibreNMS host + Tailscale only, founder ruling 2026-08-27); `extend` for ntpq + dhcpd lease counts |
| NTP | (no repo file) | Host bare-metal **ntpsec** serves :123 incl. Tailscale IP; chrony container deliberately removed as redundant (`setup-netinfra.sh` §6b) |
| Switch configs | `switches/*.cmds` | Out of scope for KNELIAC (device CLI dumps) — leave in PFVCluster |
| Docs | `pfv-netinfra-setup.md`, `dhcp-migration.md`, `dns-cluster-setup/README.md` | Already pointer stubs to Discourse topic 306 |

### Mapping into KNELIAC

| #472 component | KNELIAC home | Conversion |
|---|---|---|
| dhcpd failover pair | `roles/dhcp_server` | Two confs → one `dhcpd.conf.j2` + `dhcp_role: primary\|secondary` host_var; params (peer name, mclt, split, subnet, options) to `group_vars/netinfra_servers.yml`; `install-dhcp.sh` → `apt` + `template` + handlers (`dhcpd -t` validate → restart) |
| Pi-hole container | `roles/dns_pihole` | compose → template; password via Vault/AWX credential, never group_vars (matches `.env.example` stance + AGENTS.md) |
| Technitium container + zone sync | `roles/dns_technitium` | compose → template (dnsnet fixed IP 10.53.0.53); `sync-zones.sh` → `files/` + systemd unit/timer templates; **idempotent: never wipe `config/` on re-run** (keep quarantine-fallback semantics of `setup-netinfra.sh` §8) |
| ntpsec | `roles/ntp_server` | **Gap: no managed ntp.conf exists anywhere in the repo** — capture from live hosts first, then template; gate on `is_ntp_server` |
| snmpd | extend `roles/system_config` | Add netinfra conf variant (scoped ACL + extends) selected by group; RO community moves to Vault (it is currently plaintext in KNELIAC `group_vars/all.yml:49` and in PFV snmpd confs) |
| Verification | `playbooks/setup_netinfra.yml` verify block or `scripts/` | Port `verify.sh` + `tests/validation/dns-ntp-redundancy.sh` (dig/ntpq probes already codified there) |

### Gaps / risks

1. **ntpsec config unmanaged** — lives only on hosts (and stale in `netboot-audit.txt`). Must be captured before repave or NTP service definition is lost.
2. **Webmin hand-edits** — dhcpd confs are declared "Managed via Webmin"; templates will fight Webmin. Decide: Webmin becomes read-only lease viewer, or is dropped from netinfra nodes.
3. **Secrets** — Technitium `auth.config` (admin creds + 2FA) is copied host-to-host by `setup.sh`; it must NOT enter git. Keep out of repo; rotate to env/vault or accept manual one-time bootstrap.
4. **Zone data gravity** — zones live inside Technitium config dir on hosts. Repave playbook must be zone-preserving (no `state: absent` on data dirs).
5. **Duplicate sources** — `dns-cluster-setup` ×2 repos, `dns-ntp-redundancy.sh` ×2 repos, with drift favoring the PFV copies; port from **PFVCluster** versions (they have the fixes), then stub both old locations.
6. **No collections yet** — container management approach undecided (`requirements.yml` empty).
7. **Legacy bootstrap pattern** — `setup-netinfra.sh` relays config *from pfv-netboot*; the Ansible repave should bootstrap from first principles (templates + captured zones) so the retired netboot host is no longer a deploy-time dependency.

---

## 5. Migration sequencing (founder wants dns/ntp/dhcp code relocated PFVCluster → KNELIAC)

Blast radius is small: no Makefile/AWX/test references to `netinfra/` exist outside the directory itself; docs are already Discourse stubs.

| Phase | Action | Why this order |
|---|---|---|
| 0. Capture | Pull live `/etc/ntpsec/ntp.conf` from both nodes; snapshot Technitium `config/` + Pi-hole gravity as offline backups (not committed) | Closes gap #1; makes repave non-destructive |
| 1. Scaffolding | Add `netinfra_servers` group_vars + host_vars (roles primary/secondary, IPs); `requirements.yml` decision | Pure additive; no behavior change |
| 2. `ntp_server` role | Template captured ntp.conf; deploy to both nodes; verify with `dns-ntp-redundancy.sh` probes | Least coupled service; proves AWX pipeline to these hosts |
| 3. `dhcp_server` role | Convert pair confs to template + failover vars; `dhcpd -t` gate; flip from Webmin-managed to Ansible-managed in one maintenance window | Configs already declarative → low conversion risk; highest service impact, so do while DNS/NTP verified stable |
| 4. `dns_pihole` + `dns_technitium` roles | Compose templates, dnsnet, zone-sync units; **port PFVCluster versions** of the scripts; zone-preserving idempotency; run `verify.sh` port after | Containers are self-contained; quarantine fallback already designed |
| 5. snmp + AWX wiring | netinfra snmpd variant; add job templates via `scripts/awx_create_job_templates.py`; snmp community → Vault | Cosmetic until monitoring depends on it |
| 6. Decommission in PFVCluster | Replace `netinfra/{dhcp,dns-cluster-setup,snmp,pihole,setup-netinfra.sh}` with pointer READMEs (→ KNELIAC paths + Discourse 306), matching the existing stub pattern; keep `switches/` and audit artifacts (`netboot-*.txt`) in place (history) | Keeps PFVCluster git-browsers correct; preserves non-IaC content |
| 7. KNELServerBuild cleanup | Stub/remove its `dns-cluster-setup/` copy and duplicated validation test | Ends three-way drift |

**Keeping PFVCluster correct during transition:** each phase leaves the old script in place until the KNELIAC role has run green against both nodes (toggle with `run_*` per repo convention); pointer stubs land only in phase 6/7; Discourse topic 306 gets a "source of truth = KNELIAC" note at phase 6. Nothing else in PFVCluster imports netinfra, so no other references need updating.