KNELIAC/docs/kneliac-basis-analysis.md

KNELIAC Basis Analysis: KNELServerBuild → KNELIAC, and PFVCluster netinfra relocation (Redmine #472)

Date: 2026-08-28. Sources inspected read-only: KNEL/KNELServerBuild, KNEL/KNELIAC, PFVCluster/netinfra (+ targeted greps elsewhere in PFVCluster).

---

1. KNELServerBuild inventory (legacy bash fleet build)

Self-description: "KNEL FetchApply" — TSYS IaC via bash, intended to run under FetchApply (P5vc/fetch-apply), which never materialized.

Top-level layout

Path	Contents
ProjectCode/SetupNewSystem.sh	431-line monolithic runner; global-* phase functions
ProjectCode/Modules/{Security,OAM,Auth,RandD}	9 bash modules (~830 lines total)
ProjectCode/ConfigFiles/	17 static config dirs (SSH, SNMP, NTP, DHCP, Syslog, AuditD, ZSH, BANNERS, ...)
ProjectCode/Agents/librenms	PHP files disguised as .sh (mysql.sh, #!/usr/bin/php)
ProjectCode/{Dell,scripts,legacy}	Dell CPU perf fix, up2date.sh, dead profile scripts
Project-Includes/	PreflightCheck.sh, pi-detect.sh, LocalHelp.sh
Project-ConfigFiles/CONFIG_VARS	DL_ROOT pointing at deprecated dl.knownelement.com CDN
Project-Tests/{unit,security,validation}	Bash tests incl. dns-ntp-redundancy.sh (duplicated in PFVCluster)
dns-cluster-setup/	Technitium cluster scripts at repo ROOT (outside ProjectCode; also duplicated in PFVCluster)
ProjectDocs/	13 docs, mostly AI review/audit artifacts + todo files
vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework	Runtime-sourced shell framework (SSH URL as directory name)
logs/	Runtime logs inside repo tree

Patterns / languages

• Bash; runner sources vendored KNELShellFramework + Project-Includes/* at startup (SetupNewSystem.sh:28-36).
• Self-locating via BASH_SOURCE; host-class detection: Dell dmidecode → physical, virt-what → guest, Pi detect, distro checks.
• Modules = secharden-*.sh (security) / oam-*.sh (monitoring); configs deployed via cat/cp from ConfigFiles/.
• ISC dhcpd / ntp / snmpd conf, PHP agents, embedded systemd units in heredocs.

Generic vs PFV-specific

Generic (fleet-wide)	PFV-specific
All Security modules, OAM/LibreNMS, packages, shells, banners, Dell fixes	dns-cluster-setup/ (pfv-netinfra-01/02 Technitium, LAN IPs 192.168.3.x hardcoded)
ConfigFiles for SSH/SNMP/NTP/DHCP-client/SMTP	Project-Tests/validation/dns-ntp-redundancy.sh (192.168.3.252/.253 hardcoded)

Quality / rot assessment

• Documented debt: ProjectDocs/CODE-REVIEW-FINDINGS.md (2025-07-14) lists unquoted expansions, curl without timeout/retry, repeated apt-get invocations.
• Fragile foundation: runtime sourcing of a vendored framework whose path is an SSH URL; CONFIG_VARS still points at the deprecated CDN (contradicted by its own AGENTS.md).
• Drift: dns-cluster-setup exists in two repos; the PFVCluster copy carries fixes (find vs ls, if/else vs &&/||) the KNELServerBuild copy lacks — the "wrong" copy is in the IaC repo.
• Rot signals: legacy/ dir, 3-line stub module (secharden-auto-upgrade.sh), committed logs/, 13 process-artifact docs, FetchApply framing with no FetchApply.
• Verdict: architecture (phases, modules, preflight, tests) is sound and was deliberately ported; the bash implementation is end-of-life. Treat as specification, not code to salvage verbatim.

---

2. KNELIAC current state

Already a working Ansible repo (not greenfield), executed by AWX (tsys-awx.knel.net) against Debian-family hosts.

Item	State
roles/	preflight, oam, packages, system_config, security_ssh, security_wazuh, security_scap_stig, security_2fa, security_audit
playbooks/	hello_world.yml (AWX smoke), setup_new_system.yml (full build, port of the bash runner), setup_awx_https.yml (host nginx TLS-termination proxy in front of k3s ServiceLB; self-signed cert w/ documented SANs)
inventory/hosts.yml	Functional groups; already contains ntp_servers, dhcp_servers, dns_servers groups with pfv-netinfra-01/02
inventory/group_vars/all.yml	Mirrors legacy hard-coded values; run_* phase toggles; dns_servers/ntp_servers = 192.168.3.252/.253
scripts/awx_create_job_templates.py	AWX job-template bootstrap (190 lines)
Config/tooling	ansible.cfg, .ansible-lint, requirements.yml (empty — no collections yet), .gitignore

Conventions observed

• Naming: <area>_<thing> roles (security_ssh, system_config); playbooks setup_<purpose>.yml.
• preflight derives facts from group membership (roles/preflight/tasks/main.yml:109-115): is_ntp_server, is_dhcp_server, is_dns_server, ...
• system_config already skips dhclient.conf on DHCP servers (when: not is_dhcp_server) — the server-side hook point exists.
• snmpd.conf selected per host class (pi / physical / virtual / default).
• README.md carries a legacy→Ansible mapping table; AGENTS.md carries a strict secrets policy (vault/env only, history purge on leak) and auto commit/push policy.
• Note: group_vars/all.yml holds the SNMP RO community in plaintext today — tension with the AGENTS.md policy to resolve during the #472 work (move to Vault or AWX credentials).

---

3. KNELServerBuild → KNELIAC: what ports, what dies

Reality check: the fleet-build port is ~done (the 9 roles + mapping table in KNELIAC README cover all Modules/ + runner phases). Remaining port work is small:

Legacy item	Disposition
Modules/Security, OAM, runner phases, Preflight, pi-detect	✅ Already ported — verify parity only
Project-Tests/*	Port as role self-tests / assert tasks; dns-ntp-redundancy.sh folds into #472 verification
ProjectDocs/* guides (TSYS-2FA, DEPLOYMENT, SECURITY)	Cherry-pick still-true content into KNELIAC docs; drop AI-review artifacts
dns-cluster-setup/ (repo root)	Retire — superseded by PFVCluster copy (which has the fixes) and then by #472 roles
CONFIG_VARS DL_ROOT, FetchApply framing, legacy/, Dell/, Agents PHP	Drop; capture intent in KNELIAC wiki/README if still relevant

Recommended KNELIAC skeleton (additions for #472)

KNELIAC/
├── playbooks/
│   └── setup_netinfra.yml          # repave playbook: preflight → base → netinfra roles → verify
├── roles/
│   ├── dhcp_server/                # ISC dhcpd failover pair (from netinfra/dhcp/)
│   │   ├── tasks/  handlers/  templates/dhcpd.conf.j2
│   ├── dns_pihole/                 # compose + dnsnet + gravity (from netinfra/pihole/, setup-netinfra.sh)
│   │   └── tasks/ templates/docker-compose.yml.j2  handlers/
│   ├── dns_technitium/             # compose, config bootstrap, zone-sync units
│   │   └── tasks/ templates/ files/sync-zones.sh  handlers/
│   └── ntp_server/                 # ntpsec (from host state; see gaps)
│       └── tasks/ templates/ntp.conf.j2
├── inventory/
│   ├── group_vars/netinfra_servers.yml   # failover params, dnsnet subnet, ports, probe names
│   └── host_vars/pfv-netinfra-0{1,2}.yml # dhcp_role: primary|secondary, lan_ip, tailscale_ip
└── scripts/                         # keep verify.sh-style bash here or convert to asserts

• Naming follows existing convention (dhcp_server ~ security_ssh); one playbook setup_netinfra.yml matches setup_new_system.yml.
• system_config gains a netinfra snmpd.conf variant (extends/ACL) selected by is_dns_server/group — smallest change, matches its existing per-class selection pattern.
• Decide on community.docker vs ansible.builtin.command+compose for containers; if the former, it becomes the first entry in requirements.yml.

---

4. Gap analysis: PFVCluster netinfra/ → KNELIAC (#472)

Current source of truth (PFVCluster/netinfra, read-only)

Item	File(s)	Notes
Node repave orchestrator	setup-netinfra.sh (322 L)	SSH-driven; deploys compose services under /home/localuser/services/{pihole,ntp,technitium}; relays config from pfv-netboot (read-only ref) via workstation tarball; per-node Tailscale IPs at lines 42-45
DHCP failover pair	dhcp/dhcpd-{primary,secondary}.conf (332/331 L)	Peer pfv-dhcp, hot-standby split 255 + load balance max seconds 3; subnet 192.168.0.0/22; DNS/NTP options hand out both nodes; header says "Managed via Webmin"
DHCP install	dhcp/install-dhcp.sh	isc-dhcp-server + Webmin; INTERFACESv4=ens18; installs disabled
Pi-hole	pihole/docker-compose.yml, .env.example, gravity-validate.sh	:53, web 10002/10003; .env.example correctly redacted
Technitium cluster	dns-cluster-setup/{setup,remote-dns,sync-zones,verify}.sh	01 primary / 02 secondary; zone sync = rsync via technitium-zone-sync.service + .timer (60 s), installed by setup.sh:253-307; rsync chosen because Technitium AXFR uses :53 which Pi-hole owns ( Technitium on :5300)
snmpd	snmp/snmpd-netinfra-0{1,2}.conf	Scoped ACL (LibreNMS host + Tailscale only, founder ruling 2026-08-27); extend for ntpq + dhcpd lease counts
NTP	(no repo file)	Host bare-metal ntpsec serves :123 incl. Tailscale IP; chrony container deliberately removed as redundant (setup-netinfra.sh §6b)
Switch configs	switches/*.cmds	Out of scope for KNELIAC (device CLI dumps) — leave in PFVCluster
Docs	pfv-netinfra-setup.md, dhcp-migration.md, dns-cluster-setup/README.md	Already pointer stubs to Discourse topic 306

Mapping into KNELIAC

#472 component	KNELIAC home	Conversion
dhcpd failover pair	roles/dhcp_server	Two confs → one dhcpd.conf.j2 + dhcp_role: primary|secondary host_var; params (peer name, mclt, split, subnet, options) to group_vars/netinfra_servers.yml; install-dhcp.sh → apt + template + handlers (dhcpd -t validate → restart)
Pi-hole container	roles/dns_pihole	compose → template; password via Vault/AWX credential, never group_vars (matches .env.example stance + AGENTS.md)
Technitium container + zone sync	roles/dns_technitium	compose → template (dnsnet fixed IP 10.53.0.53); sync-zones.sh → files/ + systemd unit/timer templates; idempotent: never wipe config/ on re-run (keep quarantine-fallback semantics of setup-netinfra.sh §8)
ntpsec	roles/ntp_server	Gap: no managed ntp.conf exists anywhere in the repo — capture from live hosts first, then template; gate on is_ntp_server
snmpd	extend roles/system_config	Add netinfra conf variant (scoped ACL + extends) selected by group; RO community moves to Vault (it is currently plaintext in KNELIAC group_vars/all.yml:49 and in PFV snmpd confs)
Verification	playbooks/setup_netinfra.yml verify block or scripts/	Port verify.sh + tests/validation/dns-ntp-redundancy.sh (dig/ntpq probes already codified there)

Gaps / risks

1. ntpsec config unmanaged — lives only on hosts (and stale in netboot-audit.txt). Must be captured before repave or NTP service definition is lost.
2. Webmin hand-edits — dhcpd confs are declared "Managed via Webmin"; templates will fight Webmin. Decide: Webmin becomes read-only lease viewer, or is dropped from netinfra nodes.
3. Secrets — Technitium auth.config (admin creds + 2FA) is copied host-to-host by setup.sh; it must NOT enter git. Keep out of repo; rotate to env/vault or accept manual one-time bootstrap.
4. Zone data gravity — zones live inside Technitium config dir on hosts. Repave playbook must be zone-preserving (no state: absent on data dirs).
5. Duplicate sources — dns-cluster-setup ×2 repos, dns-ntp-redundancy.sh ×2 repos, with drift favoring the PFV copies; port from PFVCluster versions (they have the fixes), then stub both old locations.
6. No collections yet — container management approach undecided (requirements.yml empty).
7. Legacy bootstrap pattern — setup-netinfra.sh relays config from pfv-netboot; the Ansible repave should bootstrap from first principles (templates + captured zones) so the retired netboot host is no longer a deploy-time dependency.

---

5. Migration sequencing (founder wants dns/ntp/dhcp code relocated PFVCluster → KNELIAC)

Blast radius is small: no Makefile/AWX/test references to netinfra/ exist outside the directory itself; docs are already Discourse stubs.

Phase	Action	Why this order
0. Capture	Pull live /etc/ntpsec/ntp.conf from both nodes; snapshot Technitium config/ + Pi-hole gravity as offline backups (not committed)	Closes gap #1; makes repave non-destructive
1. Scaffolding	Add netinfra_servers group_vars + host_vars (roles primary/secondary, IPs); requirements.yml decision	Pure additive; no behavior change
2. ntp_server role	Template captured ntp.conf; deploy to both nodes; verify with dns-ntp-redundancy.sh probes	Least coupled service; proves AWX pipeline to these hosts
3. dhcp_server role	Convert pair confs to template + failover vars; dhcpd -t gate; flip from Webmin-managed to Ansible-managed in one maintenance window	Configs already declarative → low conversion risk; highest service impact, so do while DNS/NTP verified stable
4. dns_pihole + dns_technitium roles	Compose templates, dnsnet, zone-sync units; port PFVCluster versions of the scripts; zone-preserving idempotency; run verify.sh port after	Containers are self-contained; quarantine fallback already designed
5. snmp + AWX wiring	netinfra snmpd variant; add job templates via scripts/awx_create_job_templates.py; snmp community → Vault	Cosmetic until monitoring depends on it
6. Decommission in PFVCluster	Replace netinfra/{dhcp,dns-cluster-setup,snmp,pihole,setup-netinfra.sh} with pointer READMEs (→ KNELIAC paths + Discourse 306), matching the existing stub pattern; keep switches/ and audit artifacts (netboot-*.txt) in place (history)	Keeps PFVCluster git-browsers correct; preserves non-IaC content
7. KNELServerBuild cleanup	Stub/remove its dns-cluster-setup/ copy and duplicated validation test	Ends three-way drift

Keeping PFVCluster correct during transition: each phase leaves the old script in place until the KNELIAC role has run green against both nodes (toggle with run_* per repo convention); pointer stubs land only in phase 6/7; Discourse topic 306 gets a "source of truth = KNELIAC" note at phase 6. Nothing else in PFVCluster imports netinfra, so no other references need updating.