Set up a fully scripted, documented Technitium DNS cluster that
replicates the production instance from tailscale-router to
pfv-netinfra-01 (primary) and pfv-netinfra-02 (secondary).
What it does:
- EXPORT: reads the production Technitium config (auth.config with
users + 2FA, dns.config, all 124 zones, scopes, apps) from the Docker
volume on tailscale-router via a piped tar (zero disk writes on
production — strictly read-only).
- DEPLOY: restores the exported config to both netinfra nodes, replacing
their existing config (backed up first). Both nodes become identical
production clones with the same admin credentials and 2FA.
- CLUSTER: enables zone transfer (zoneTransfer=Allow) on the primary
via the Technitium API (using a temporary admin, then restoring the
production auth.config). Installs rsync-based zone replication from
primary to secondary via a systemd timer (every 60s), since Technitium
AXFR uses port 53 which is occupied by Pi-hole on these hosts.
- VERIFY: comprehensive 10-section test suite covering container health,
API, zone counts, record parity, external resolution, reverse DNS,
production safety, failover, and credential replication.
Scripts:
- remote-dns.sh: SSH chokepoint for all DNS host access
- setup.sh: master orchestrator (export → deploy → cluster → verify)
- sync-zones.sh: rsync-based zone replication (installed as systemd timer)
- verify.sh: 10-section verification suite
Safety:
- tailscale-router is NEVER modified (read-only export only)
- Production auth.config is backed up before any temporary admin swap
- Each node's existing config is backed up before replacement
- The export tarball is gitignored (contains production credentials)
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The ntp.conf hardening used `interface ignore wildcard` +
`interface listen 127.0.0.1`, which binds ntpd to loopback only. Outbound
NTP queries to the upstream servers then carried a 127.0.0.1 source
address that the servers cannot reply to, so the daemon's peers stayed
stuck in .INIT. with reach 0 — even though the servers are reachable
(verified: ntpdate -q succeeds, ntpd does not).
Replace the interface-based restriction with restrict-based hardening:
`restrict default ignore` blocks unsolicited queries from any host (so
the box never serves time to others), while explicit allow rules for the
two upstream servers and localhost let the client sync normally.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an architecture analysis for the tension between Tailscale's
default resolv.conf management (100.100.100.100) and the managed
LAN-resolver resolv.conf (.252/.253). Documents a key finding from
live-network probing: knel.net device records only resolve via the
Tailscale MagicDNS path; querying the LAN DNS servers directly returns
NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23).
Lays out four options (Tailscale-owned, LAN-pinned, split DNS,
Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS
to Tailscale in the short term (since wazuh/postfix/syslog depend on
knel.net names that only resolve there) and fixing the Technitium/Pi-hole
knel.net zone before pinning the LAN resolvers. Confirms the NTP
(LAN-IP) change is safe regardless. Flags that the managed-resolv.conf
change will be overwritten by Tailscale and would break knel.net
resolution if it ever sticks.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Route every host built by this project through the new
pfv-netinfra-01 (192.168.3.252) / pfv-netinfra-02 (192.168.3.253)
pair for both name resolution and time, with automatic failover.
- NTP: replace the single pfv-netboot.knel.net upstream with both
netinfra servers (iburst) so time sync survives either one failing.
- DNS: add a managed static /etc/resolv.conf (new ConfigFiles/Resolv/).
The repo previously had no resolver configuration at all. Both servers
are listed so glibc falls through to the secondary on failure.
- DHCP: request domain-name-servers/domain-search/ntp-servers and
supersede them to the netinfra pair, so a DHCP renew can't silently
revert to whatever the DHCP server advertises.
- SetupNewSystem.sh: deploy resolv.conf (robustly replacing any
systemd-resolved/NetworkManager symlink) and add pfv-netinfra to the
NTP-server self-exclusion guard so those boxes don't client off
themselves.
LAN IPs are used throughout (not the knel.net hostnames) because those
hostnames resolve to Tailscale CGNAT addresses, not the LAN addresses,
and NTP must come up before DNS. Add a validation test asserting the
config is present and both servers actually answer DNS and NTP queries.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The framework defines LOGFILENAME as "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$"
and PrettyPrint appends every print_info/print_error line to it, so
executing any script that sources the framework leaves a timestamped log
file beside it (e.g. run-tests.sh.Monday-2026-07-27-10:44:31.123). These
are runtime artifacts, not source, and were showing up as untracked
noise. Ignore them across the whole repo.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_webmin_2fa used `sed -i ... || echo ... >>` to add
twofactor_provider and twofactor to /etc/webmin/miniserv.conf. sed
returns 0 even when it matches nothing, so when the directives were
absent (the normal case on a fresh Webmin install) the `|| echo` branch
never ran. The script printed "Webmin 2FA configuration completed" while
leaving 2FA entirely unconfigured — caught by 2fa-validation reporting
"Webmin TOTP provider not configured".
Guard each directive with grep so it is appended when absent and updated
when present.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three tests produced false failures when run on the deployed host:
- safe-download: the read-only-location assertion expects a write to
fail, but the suite runs as root and root bypasses filesystem
permissions, so the write succeeded. Skip that assertion as root.
- 2fa-validation: package presence used `dpkg -l | grep`, whose
fixed-width output wraps long names when COLUMNS is narrow (as in a
non-interactive shell), falsely reporting libpam-google-authenticator
and qrencode as missing even though they were installed. Use dpkg -s.
- https-enforcement: SSL validation passed --cert-status, which requires
OCSP stapling that many valid CDNs do not provide, flagging valid
certificates as invalid. Drop it; --ssl-reqd still enforces TLS and
certificate-chain verification.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three bugs prevented the validation suite from running cleanly:
- run-tests.sh used `((TESTS_PASSED++))` under `set -e`. Post-increment
evaluates to the old value, so the first passing test (0 -> 1) made
`(( ))` return 1 and errexit aborted the whole run after exactly one
test. Use plain arithmetic assignment instead.
- https-enforcement.sh's comment filter ran `grep -n` (which prefixes
"linenum:") and then tried to drop comment lines with
`^[[:space:]]*#`, which never matched the line-number prefix. Every
http:// URL in a comment (deprecated curl lines, the strict-mode
attribution comment) was flagged as a violation. Match the prefix.
- 2fa-validation.sh hardcoded `/home/$user/` for the setup-instructions
check, so for root it looked in /home/root (which does not exist)
instead of /root. Resolve the home directory with getent.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-2fa enforces AuthenticationMethods publickey,keyboard-
interactive, so once setup completes no non-interactive SSH client can
authenticate (a TOTP token is required). The harness's post-deploy steps
— log fetch, repo path resolution, and the validation suite — all relied
on SSH and therefore failed after the first successful deploy, masking
the fact that setup itself had completed (rc=0).
- remote.sh: add a vm-guest mode that runs commands as root inside the
VM via the Proxmox qemu-guest-agent (qm guest exec), bypassing SSH/2FA
entirely. Output is parsed on the Proxmox host with python3.
- vm-validation.sh: resolve repo path, fetch the setup log, and run the
validation suite via vm-guest when SSH is unavailable. Detect the
setup exit marker from the always-available live stream as a fallback
to the fetched log. Make restore_vm_access 2FA-aware so a post-deploy
SSH failure is understood (not a hard error) once 2FA is in effect.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_ssh_2fa only enabled the deprecated ChallengeResponseAuthentication
directive (removed as a usable knob in modern OpenSSH; it no longer controls
keyboard-interactive). The base tsys-sshd-config ships
KbdInteractiveAuthentication no, so on Debian 13 (OpenSSH 9.x/10.x)
keyboard-interactive stayed disabled. With AuthenticationMethods set to
"publickey,keyboard-interactive", sshd -t then failed:
Disabled method "keyboard-interactive" in AuthenticationMethods list ...
AuthenticationMethods cannot be satisfied by enabled authentication methods
which aborted provisioning under errexit.
Add the modern KbdInteractiveAuthentication yes directive alongside the
legacy one so 2FA works on both current and older OpenSSH.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
29 files under ProjectCode/ConfigFiles lacked a trailing newline. They
deploy via `cat file > target`, and several targets are subsequently
appended to (notably /etc/ssh/sshd_config, which configure_ssh_2fa
appends `AuthenticationMethods publickey,keyboard-interactive` to).
Without a trailing newline the append fused onto the last line,
producing `LoginGraceTime 60AuthenticationMethods ...`, which sshd -t
rejected as an invalid time value and aborted provisioning under errexit.
This is the same defect class that already broke the managed
authorized_keys files. Add the trailing newline to every config file
that was missing one so the cat-then-append pattern is always safe.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
setup_user_2fa wrote each user's 2FA-setup instructions to the quoted
path "~$user/2fa-setup-instructions.txt". Tilde expansion does not occur
inside double quotes, so the path was treated literally and the write
failed with "No such file or directory", aborting the whole 2FA module
(and thus provisioning) under errexit.
Resolve the home directory explicitly with `getent passwd` and use that
absolute path for both the instructions file and the chown. Skip the
user cleanly if no home directory exists.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-ssh intentionally replaces authorized_keys with the managed
production key set, which locks out the bootstrap/dev key the validation
harness uses to drive the VM. After the first deploy that reaches SSH
hardening, the harness could no longer connect to fetch logs or run the
test suite, breaking the iteration loop.
Add restore_vm_access(): after each deploy, if SSH is unreachable, it
re-injects the validation pubkey OUT OF BAND via the Proxmox guest agent
(qm guest exec runs as root inside the VM and does not depend on SSH).
The injected payload is prefixed with a newline to avoid key
concatenation when the managed file lacks a trailing newline.
Config: ACCESS_PUBKEY (default ~/.ssh/id_ed25519.pub), RESTORE_ACCESS=1.
Disable with RESTORE_ACCESS=0.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-wazuh.sh did a hard `systemctl start wazuh-agent`. The agent
attempts to reach its manager (tsys-nsm.knel.net) during startup; when
that host is unreachable (e.g. an isolated lab/sandbox VM, or the SIEM
being temporarily down during a fresh build), systemd's start exceeds
its timeout and the whole provisioning aborts under the framework's
errexit.
The agent is already installed and enabled, so it will keep retrying the
manager on its own. Make the start non-fatal with `|| true` so a host can
finish building even when the manager isn't reachable at deploy time.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Both managed authorized_keys files lacked a trailing newline. sshd reads
the final key fine on its own, but any subsequent append (tooling, a
follow-up key, or the validation harness) concatenated onto the last key
line, fusing two keys into one unparseable blob and silently breaking
public-key auth for both.
Add the trailing newline so the files concatenate safely.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The KexAlgorithms line in ssh-audit-hardening.conf began with a space.
In sshd_config a leading whitespace marks a line continuation, so the
entire directive was absorbed as arguments to the (non-existent)
previous directive. The effective kexalgorithms collapsed to only the
two trailing GSSAPI entries (gss-curve25519-sha256-, gss-group16-sha512-),
which no normal OpenSSH client can negotiate.
Result: after secharden-ssh deployed this file, every SSH connection to
the host died in [preauth] with no usable key exchange algorithm. sshd -t
still returned 0, so the breakage was completely silent.
Drop the leading space so the directive is parsed as intended. This
restores normal client compatibility while keeping the hardened
algorithm set.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The `apt-get --purge autoremove` after the package removal pass had no
`-y`, so once removing modemmanager/wpasupplicant orphaned eleven
dependent packages, autoremove printed its "[Y/n] Abort." confirmation
and exited 1, aborting the run under errexit.
Add `-y` so the now-orphaned dependencies are purged without prompting,
consistent with the surrounding non-interactive apt invocations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
global-installPackages installed latencytop and cockpit-tests on every
non-Kali host, but both have been dropped from Debian trixie (latencytop
is dead upstream; cockpit-tests has no candidate). Under the framework's
errexit the failing apt-get aborted the entire run with rc=100 right
after the core package install completed.
These are optional monitoring/test extras, not core requirements, so
make the install best-effort with `|| true` to match the script's
existing treatment of non-critical operations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an end-to-end validation loop for the sandbox VM and the single
SSH/SCP chokepoint it depends on:
- Project-Tests/remote.sh: the only place ssh/scp is invoked. Provides
prox / vm / vmroot / *-file / *-copy modes. Centralizes host/user/key
config and keeps remote access auditable and reusable.
- Project-Tests/vm-validation.sh: drives a Proxmox VM through
snapshot -> deploy -> validate with one-command rollback. Deployment is
GIT-BASED: the VM clones/pulls the public repo itself, exactly as a
fresh server would, so results are identical regardless of who runs it
(no reliance on a local working copy or rsync). Resolves the absolute
repo path on the VM before sudo to avoid the '~' -> root's home trap.
- logs/.gitignore: ignore generated validation/test logs (was a no-op
`!.gitignore` with no ignore rule; logs would have been committed).
Also fixes a `help`-branch typo (`${BASH_SOURCE[0]}`) and adds
PROX_USER (defaults to root) since the bare Proxmox hostname defaulted
to the wrong SSH user.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The framework sources ErrorHandling.sh which enables `set -o errexit`
globally, yet PrettyPrint's print_info/print_error invoked `tput bold`
and `tput sgr0` with no protection. In any TERM-less context (SSH
automation, CI, cron) `tput` fails with "unknown terminal" and, under
errexit, aborts the entire script on the very first status message.
Suppress tput stderr and add `|| true` so the color helpers degrade
gracefully to plain output instead of crashing every consumer script.
This is an internal framework consistency fix: strict mode + unguarded
external command were mutually incompatible.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Make AGENTS.md actionable for future agents by recording what was learned
while fixing the provisioning scripts:
- Add a Repository Layout section: the KNELShellFramework is vendored
under vendor/.../KNELShellFramework (not at repo root), scripts must
self-locate via BASH_SOURCE, configs are read locally (no CDN), and
some .sh agents are actually PHP
- Replace the vague "commit immediately" note with an explicit
Autonomous Git Workflow section authorizing agents to commit AND push
without being asked, grouped into coherent atomic commits
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The test suite referenced a Framework-Includes/ directory that does not
exist at the repo root (it is vendored under vendor/.../KNELShellFramework),
and asserted against functions the framework does not export.
- Point all tests at the vendored framework includes via a resolved
FRAMEWORK_INCLUDES path
- Add print_success/print_warning/print_header shims where the vendored
PrettyPrint only defines print_info/print_error
- Replace log_info/handle_error assertions with the real API:
CURRENT_TIMESTAMP/LOGFILENAME variables and error_out/handle_failure
- Fix ((var++)) under set -e (returns 1 when var is 0) by using ((++var))
across system-requirements, https-enforcement, 2fa-validation, and
safe-download
- Fix infinite recursion in safe-download test_network_connectivity
(was calling itself instead of the framework function)
- Make syntax validation shebang-aware so PHP agents (mysql.sh) are
skipped instead of flagged as bash syntax errors
Framework unit test now passes (exit 0).
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
All provisioning scripts and modules now resolve their own location via
BASH_SOURCE and derive PROJECT_ROOT_PATH from it, removing a hard
dependency on the current working directory.
- Derive PROJECT_ROOT_PATH/CONFIGFILES_PATH/MODULES_PATH/SCRIPTS_PATH
from BASH_SOURCE in SetupNewSystem.sh and every module
- Replace all curl ${DL_ROOT}/... downloads with cat of the matching
local files under ProjectCode/ConfigFiles (the dl.knownelement.com CDN
is no longer required for a git clone)
- Invoke modules by absolute path instead of cd ./Modules/X && bash ./x
- Fix secharden-audit-agents.sh: wrong path depth (../../ vs ../../..),
wrong Project-Includes glob, and ConfigFiles/AudidD -> AuditD typo,
all of which previously crashed the script
- Remove duplicate FrameworkVars source lines
Run from anywhere with: sudo bash ProjectCode/SetupNewSystem.sh
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add agent guidelines for AI assistants working on this repository:
- Document atomic commit requirements
- Specify conventional commit format with examples
- Require verbose, formatted commit messages
- Emphasize immediate commit/push behavior
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles
Charles N Wyble