KNELServerBuild/dns-cluster-setup/README.md

# Technitium DNS Cluster Setup

Replicates the production Technitium DNS Server from `tailscale-router` to the
`pfv-netinfra-01/02` pair and configures them as a primary/secondary cluster
with automatic zone transfers.

## Architecture

```
 tailscale-router (PRODUCTION — READ ONLY)
   └─ tsys-dns container (technitium/dns-server)
      └─ 124 zones (knel.net + reverse DNS)
      └─ Users + 2FA in auth.config
                │
          docker cp (export)
                │
                ▼
 ┌─ pfv-netinfra-01 (192.168.3.252) ──── PRIMARY ──────────┐
 │  tsys-dns container (Technitium on :5300)                │
 │  pihole container (Pi-hole on :53 → Technitium :5300)   │
 │  All zones are Primary                                   │
 │  Zone transfer allowed from 192.168.3.253               │
 └──────────────────────────────────────────────────────────┘
                │
          AXFR / IXFR + NOTIFY (DNS zone transfer, port 5300)
                │
                ▼
 ┌─ pfv-netinfra-02 (192.168.3.253) ─── SECONDARY ────────┐
 │  tsys-dns container (Technitium on :5300)                │
 │  pihole container (Pi-hole on :53 → Technitium :5300)   │
 │  All zones are Secondary (AXFR from 01)                  │
 └──────────────────────────────────────────────────────────┘
```

### How clustering works

Technitium uses standard DNS zone transfers (AXFR/IXFR) for primary/secondary
replication, not a proprietary protocol:

1. **Primary (01)** holds all zones as authoritative primary zones.
2. **Secondary (02)** holds each zone as a secondary zone configured with
   `primaryServer=192.168.3.252:5300`.
3. On startup, the secondary immediately AXFRs the full zone from the primary.
4. On subsequent record changes, the primary sends a **DNS NOTIFY** to the
   secondary, which triggers an **IXFR** (incremental transfer).
5. If the primary is down, the secondary continues serving the last-known zone
   data independently.

### Credentials and 2FA

The production `auth.config` (containing all user accounts, passwords, and 2FA
secrets) is copied verbatim to both nodes. This means:

- The **same username, password, and 2FA device** work on all three servers.
- The web console is at `http://<host>:5380/` on each node.
- No credential changes are needed.

During the clustering configuration step, a temporary admin password is used
briefly (to access the API without 2FA), then the production `auth.config` is
restored. See "Security notes" below.

## Prerequisites

- SSH key access to all hosts as `localuser` with passwordless sudo.
- The `remote-dns.sh` wrapper must be able to reach all hosts via Tailscale FQDN.
- Docker + Docker Compose on netinfra-01/02 (already installed).
- The production Technitium on tailscale-router must be running.

## Usage

```bash
cd dns-cluster-setup/

# Step-by-step (recommended for first run):
./setup.sh export      # 1. Export config from tailscale-router (READ-ONLY)
./setup.sh deploy01    # 2. Deploy to netinfra-01 as primary
./setup.sh deploy02    # 3. Deploy to netinfra-02 as secondary clone
./setup.sh cluster     # 4. Configure clustering (01→02 zone transfers)
./setup.sh verify      # 5. Run all verification tests

# Or all at once:
./setup.sh all
```

### Configuration overrides

All defaults can be overridden via environment variables:

| Variable | Default | Description |
|---|---|---|
| `PRIMARY_IP` | `192.168.3.252` | netinfra-01 LAN IP |
| `SECONDARY_IP` | `192.168.3.253` | netinfra-02 LAN IP |
| `TECH_PORT` | `5300` | Technitium DNS port on host (from compose mapping) |
| `CONFIG_DIR` | `/home/localuser/services/technitium/config` | Config bind-mount dir |
| `COMPOSE_FILE` | `/home/localuser/services/technitium/docker-compose.yml` | Compose file |
| `TEMP_ADMIN_PW` | `KnelClusterSetup!2026` | Temp admin password (used only during clustering, then discarded) |

## Scripts

| Script | Purpose |
|---|---|
| `remote-dns.sh` | SSH/SCP chokepoint for all DNS host access (tsrouter, netinfra01, netinfra02, netboot, sandbox) |
| `setup.sh` | Master orchestrator: export → deploy → cluster → verify |
| `verify.sh` | Comprehensive 10-section verification suite |
| `discover*.sh` | Read-only discovery probes (used during development, safe to keep) |

## What gets copied

From production `/etc/dns/` (inside the container), **excluding** runtime data:

| Copied (configuration) | Excluded (runtime) |
|---|---|
| `auth.config` (users, passwords, 2FA) | `cache.bin` (DNS cache) |
| `dns.config` (server settings) | `stats/` (query statistics) |
| `webservice.config` (web console) | `logs/` (log files) |
| `allowed.config` (zone transfer ACL) | |
| `blocked.config` (blocked domains) | |
| `blocklist.config` (blocklist settings) | |
| `blocklists/` (blocklist data) | |
| `zones/` (all 124 zone files) | |
| `scopes/` (DHCP scopes) | |
| `apps/` (Technitium apps) | |

## Verification tests

The `verify.sh` script runs 10 categories of tests:

1. **Container health** — both Technitium containers are Up
2. **API responds** — web console API is reachable on both nodes
3. **Zone count** — primary matches production; secondary matches primary
4. **Forward DNS** — known knel.net records resolve identically on both nodes
5. **External DNS** — both nodes can resolve external domains (github.com)
6. **Zone transfer (AXFR)** — secondary can AXFR knel.net from primary
7. **Reverse DNS** — PTR zones have SOA records on both nodes
8. **Production untouched** — container still running, zone count unchanged
9. **Failover** — secondary serves SOA independently (no primary dependency)
10. **Credentials** — `auth.config` byte-size matches across all three nodes

## Security notes

- **tailscale-router is never modified.** The only operation is `docker cp`
  (read) to export the config. No writes, no restarts, no config changes.
- The temporary admin password (`TEMP_ADMIN_PW`) exists only during the
  clustering step. After configuration, the production `auth.config` (with 2FA)
  is restored. The temp password is never persisted.
- The export tarball (`.export/technitium-production-config.tar.gz`) contains
  production credentials. It is in `.gitignore` and should be deleted after
  setup: `rm -rf dns-cluster-setup/.export/`
- Each node's existing config is backed up to `config.backup-<timestamp>` before
  replacement, so the change is reversible.

## Recovery

If something goes wrong, each node has a backup:

```bash
# On netinfra-01 or netinfra-02:
cd /home/localuser/services/technitium/
docker compose down
mv config config.failed
mv config.backup-<timestamp> config
docker compose up -d
```

## Validation on sandbox

After cluster setup, validate that client hosts use the pair correctly:

```bash
# From sectestbed-sandbox (or any client):
# Query primary directly:
dig @192.168.3.252 pfv-netinfra-01.knel.net

# Query secondary directly:
dig @192.168.3.253 pfv-netinfra-01.knel.net

# Both should return the same answer.
```

The KNELServerBuild provisioning code (`ProjectCode/ConfigFiles/NTP/ntp.conf`
and `ProjectCode/ConfigFiles/Resolv/resolv.conf`) points clients at both
servers for DNS and NTP redundancy. See `ProjectDocs/tailscale.md` for the
full DNS architecture analysis.