Files
mrcharles b1088e8487 feat(dns-cluster): replicate Technitium production to netinfra pair
Set up a fully scripted, documented Technitium DNS cluster that
replicates the production instance from tailscale-router to
pfv-netinfra-01 (primary) and pfv-netinfra-02 (secondary).

What it does:
- EXPORT: reads the production Technitium config (auth.config with
  users + 2FA, dns.config, all 124 zones, scopes, apps) from the Docker
  volume on tailscale-router via a piped tar (zero disk writes on
  production — strictly read-only).
- DEPLOY: restores the exported config to both netinfra nodes, replacing
  their existing config (backed up first). Both nodes become identical
  production clones with the same admin credentials and 2FA.
- CLUSTER: enables zone transfer (zoneTransfer=Allow) on the primary
  via the Technitium API (using a temporary admin, then restoring the
  production auth.config). Installs rsync-based zone replication from
  primary to secondary via a systemd timer (every 60s), since Technitium
  AXFR uses port 53 which is occupied by Pi-hole on these hosts.
- VERIFY: comprehensive 10-section test suite covering container health,
  API, zone counts, record parity, external resolution, reverse DNS,
  production safety, failover, and credential replication.

Scripts:
- remote-dns.sh: SSH chokepoint for all DNS host access
- setup.sh: master orchestrator (export → deploy → cluster → verify)
- sync-zones.sh: rsync-based zone replication (installed as systemd timer)
- verify.sh: 10-section verification suite

Safety:
- tailscale-router is NEVER modified (read-only export only)
- Production auth.config is backed up before any temporary admin swap
- Each node's existing config is backed up before replacement
- The export tarball is gitignored (contains production credentials)

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 08:50:14 -05:00
..

Technitium DNS Cluster Setup

Replicates the production Technitium DNS Server from tailscale-router to the pfv-netinfra-01/02 pair and configures them as a primary/secondary cluster with automatic zone transfers.

Architecture

 tailscale-router (PRODUCTION — READ ONLY)
   └─ tsys-dns container (technitium/dns-server)
      └─ 124 zones (knel.net + reverse DNS)
      └─ Users + 2FA in auth.config
                │
          docker cp (export)
                │
                ▼
 ┌─ pfv-netinfra-01 (192.168.3.252) ──── PRIMARY ──────────┐
 │  tsys-dns container (Technitium on :5300)                │
 │  pihole container (Pi-hole on :53 → Technitium :5300)   │
 │  All zones are Primary                                   │
 │  Zone transfer allowed from 192.168.3.253               │
 └──────────────────────────────────────────────────────────┘
                │
          AXFR / IXFR + NOTIFY (DNS zone transfer, port 5300)
                │
                ▼
 ┌─ pfv-netinfra-02 (192.168.3.253) ─── SECONDARY ────────┐
 │  tsys-dns container (Technitium on :5300)                │
 │  pihole container (Pi-hole on :53 → Technitium :5300)   │
 │  All zones are Secondary (AXFR from 01)                  │
 └──────────────────────────────────────────────────────────┘

How clustering works

Technitium uses standard DNS zone transfers (AXFR/IXFR) for primary/secondary replication, not a proprietary protocol:

  1. Primary (01) holds all zones as authoritative primary zones.
  2. Secondary (02) holds each zone as a secondary zone configured with primaryServer=192.168.3.252:5300.
  3. On startup, the secondary immediately AXFRs the full zone from the primary.
  4. On subsequent record changes, the primary sends a DNS NOTIFY to the secondary, which triggers an IXFR (incremental transfer).
  5. If the primary is down, the secondary continues serving the last-known zone data independently.

Credentials and 2FA

The production auth.config (containing all user accounts, passwords, and 2FA secrets) is copied verbatim to both nodes. This means:

  • The same username, password, and 2FA device work on all three servers.
  • The web console is at http://<host>:5380/ on each node.
  • No credential changes are needed.

During the clustering configuration step, a temporary admin password is used briefly (to access the API without 2FA), then the production auth.config is restored. See "Security notes" below.

Prerequisites

  • SSH key access to all hosts as localuser with passwordless sudo.
  • The remote-dns.sh wrapper must be able to reach all hosts via Tailscale FQDN.
  • Docker + Docker Compose on netinfra-01/02 (already installed).
  • The production Technitium on tailscale-router must be running.

Usage

cd dns-cluster-setup/

# Step-by-step (recommended for first run):
./setup.sh export      # 1. Export config from tailscale-router (READ-ONLY)
./setup.sh deploy01    # 2. Deploy to netinfra-01 as primary
./setup.sh deploy02    # 3. Deploy to netinfra-02 as secondary clone
./setup.sh cluster     # 4. Configure clustering (01→02 zone transfers)
./setup.sh verify      # 5. Run all verification tests

# Or all at once:
./setup.sh all

Configuration overrides

All defaults can be overridden via environment variables:

Variable Default Description
PRIMARY_IP 192.168.3.252 netinfra-01 LAN IP
SECONDARY_IP 192.168.3.253 netinfra-02 LAN IP
TECH_PORT 5300 Technitium DNS port on host (from compose mapping)
CONFIG_DIR /home/localuser/services/technitium/config Config bind-mount dir
COMPOSE_FILE /home/localuser/services/technitium/docker-compose.yml Compose file
TEMP_ADMIN_PW KnelClusterSetup!2026 Temp admin password (used only during clustering, then discarded)

Scripts

Script Purpose
remote-dns.sh SSH/SCP chokepoint for all DNS host access (tsrouter, netinfra01, netinfra02, netboot, sandbox)
setup.sh Master orchestrator: export → deploy → cluster → verify
verify.sh Comprehensive 10-section verification suite
discover*.sh Read-only discovery probes (used during development, safe to keep)

What gets copied

From production /etc/dns/ (inside the container), excluding runtime data:

Copied (configuration) Excluded (runtime)
auth.config (users, passwords, 2FA) cache.bin (DNS cache)
dns.config (server settings) stats/ (query statistics)
webservice.config (web console) logs/ (log files)
allowed.config (zone transfer ACL)
blocked.config (blocked domains)
blocklist.config (blocklist settings)
blocklists/ (blocklist data)
zones/ (all 124 zone files)
scopes/ (DHCP scopes)
apps/ (Technitium apps)

Verification tests

The verify.sh script runs 10 categories of tests:

  1. Container health — both Technitium containers are Up
  2. API responds — web console API is reachable on both nodes
  3. Zone count — primary matches production; secondary matches primary
  4. Forward DNS — known knel.net records resolve identically on both nodes
  5. External DNS — both nodes can resolve external domains (github.com)
  6. Zone transfer (AXFR) — secondary can AXFR knel.net from primary
  7. Reverse DNS — PTR zones have SOA records on both nodes
  8. Production untouched — container still running, zone count unchanged
  9. Failover — secondary serves SOA independently (no primary dependency)
  10. Credentialsauth.config byte-size matches across all three nodes

Security notes

  • tailscale-router is never modified. The only operation is docker cp (read) to export the config. No writes, no restarts, no config changes.
  • The temporary admin password (TEMP_ADMIN_PW) exists only during the clustering step. After configuration, the production auth.config (with 2FA) is restored. The temp password is never persisted.
  • The export tarball (.export/technitium-production-config.tar.gz) contains production credentials. It is in .gitignore and should be deleted after setup: rm -rf dns-cluster-setup/.export/
  • Each node's existing config is backed up to config.backup-<timestamp> before replacement, so the change is reversible.

Recovery

If something goes wrong, each node has a backup:

# On netinfra-01 or netinfra-02:
cd /home/localuser/services/technitium/
docker compose down
mv config config.failed
mv config.backup-<timestamp> config
docker compose up -d

Validation on sandbox

After cluster setup, validate that client hosts use the pair correctly:

# From sectestbed-sandbox (or any client):
# Query primary directly:
dig @192.168.3.252 pfv-netinfra-01.knel.net

# Query secondary directly:
dig @192.168.3.253 pfv-netinfra-01.knel.net

# Both should return the same answer.

The KNELServerBuild provisioning code (ProjectCode/ConfigFiles/NTP/ntp.conf and ProjectCode/ConfigFiles/Resolv/resolv.conf) points clients at both servers for DNS and NTP redundancy. See ProjectDocs/tailscale.md for the full DNS architecture analysis.