docs(network): analyze Tailscale vs managed DNS conflict

Add an architecture analysis for the tension between Tailscale's
default resolv.conf management (100.100.100.100) and the managed
LAN-resolver resolv.conf (.252/.253). Documents a key finding from
live-network probing: knel.net device records only resolve via the
Tailscale MagicDNS path; querying the LAN DNS servers directly returns
NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23).

Lays out four options (Tailscale-owned, LAN-pinned, split DNS,
Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS
to Tailscale in the short term (since wazuh/postfix/syslog depend on
knel.net names that only resolve there) and fixing the Technitium/Pi-hole
knel.net zone before pinning the LAN resolvers. Confirms the NTP
(LAN-IP) change is safe regardless. Flags that the managed-resolv.conf
change will be overwritten by Tailscale and would break knel.net
resolution if it ever sticks.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>

ProjectDocs/tailscale.md

@@ -0,0 +1,217 @@
+# Tailscale vs. Managed DNS — Architecture Analysis
+
+> **Status:** analysis for review. No code decisions are final. Read the
+> "Known issues" section before acting on the managed-resolv.conf change.
+
+## 1. Executive summary
+
+Every host in this build runs the Tailscale client, and Tailscale — by default —
+**manages `/etc/resolv.conf` itself**, pointing it at `100.100.100.100`
+(Tailscale's MagicDNS resolver). This directly conflicts with the managed
+`resolv.conf` (pointing at `192.168.3.252`/`192.168.3.253`) that
+`SetupNewSystem.sh` deploys: whichever runs last wins, and Tailscale's daemon
+re-wins on every `tailscale up` and on reboot.
+
+Worse, a probe of the live network shows that **knel.net device records only
+resolve through the Tailscale 100.100.100.100 path** — querying the LAN IPs of
+the DNS servers directly returns NXDOMAIN for current hostnames (the Technitium
+`knel.net` zone has the SOA but is stale/empty of actual records). So pointing
+`resolv.conf` at the LAN IPs would break resolution of the very names this
+project's modules depend on (`tsys-nsm.knel.net`, `tsys-cloudron.knel.net`,
+`tsys-librenms.knel.net`).
+
+This document lays out the options and a recommended path forward.
+
+## 2. How name resolution actually works today (as measured)
+
+Probed from `sectestbed-sandbox` (192.168.3.50):
+
+| Query path | External name (`github.com`) | knel.net device name (`pfv-netinfra-01.knel.net`) |
+|---|---|---|
+| Via current resolver = `100.100.100.100` (Tailscale) | resolves | **resolves** → `100.70.181.72` (Tailscale CGNAT) |
+| Direct `dig @192.168.3.252` (Technitium, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) |
+| Direct `dig @192.168.3.253` (Pi-hole, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) |
+
+Other measured facts:
+
+- `dig @192.168.3.252 knel.net SOA` → `NOERROR`, returns
+  `knel.net. 900 IN SOA dns.knel.net. hostadmin.knel.net. 2025062313 …`
+  (serial dated **2025-06-23** — the zone exists but is stale).
+- NTP on both `.252` and `.253` answers time queries (stratum 2/3).
+- The live `/etc/resolv.conf` on a deployed host reads:
+  ```
+  # resolv.conf(5) file generated by tailscale
+  # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
+  nameserver 100.100.100.100
+  nameserver fd7a:115c:a1e0::53
+  search knel.net
+  ```
+
+**Interpretation:** the `knel.net` device→Tailscale-IP mappings are synthesised
+by Tailscale's MagicDNS from the tailnet device registry (every device that
+joins the tailnet gets `hostname.knel.net` → its `100.x.x.x` address). The
+Technitium `knel.net` zone is a separate, manually-maintained zone that has
+fallen out of date. The two are not the same source of truth.
+
+## 3. The core tension
+
+| Goal | Who provides it today |
+|---|---|
+| Resolve `*.knel.net` device names (→ Tailscale IPs) | Tailscale MagicDNS via `100.100.100.100` |
+| Resolve external names with ad-blocking | Pi-hole (`.253`), reachable via Tailscale → Technitium → Pi-hole chain |
+| Redundant, low-latency, tunnel-independent DNS | LAN resolvers `.252`/`.253` — **but these lack knel.net records** |
+| Authoritative time | NTP on `.252`/`.253` (works on either path) |
+
+The conflict: you cannot simply point `resolv.conf` at the LAN resolvers,
+because they do not know about the current `knel.net` device records, and
+several modules in this project resolve `knel.net` hostnames at runtime
+(wazuh manager, postfix relay, syslog target). You also cannot ignore Tailscale,
+because it is the only thing that resolves those names today.
+
+## 4. Options
+
+### Option A — Let Tailscale own DNS (status quo, `accept-dns=true`)
+
+Leave the default. Tailscale writes `100.100.100.100` to `resolv.conf`; the
+control-plane forwarding (`100.100.100.100` → Technitium → Pi-hole) handles
+external names and ad-blocking; MagicDNS handles `knel.net` device names.
+
+| Pros | Cons |
+|---|---|
+| Zero per-host config; new machines "just work" on `tailscale up` | **All DNS depends on the Tailscale daemon being up.** If `tailscaled` dies, every name lookup fails — including the ones you need to SSH in and fix it. |
+| MagicDNS + knel.net names resolve automatically | Latency: every query goes host→tailscaled→100.100.100.100→(tunnel)→Technitium→Pi-hole→upstream |
+| Ad-blocking preserved (via the Pi-hole hop) | Overwrites the managed `resolv.conf` — the `.252`/`.253` redundancy is lost |
+| Centralised in the Tailscale admin console | Single resolver in `resolv.conf` (`100.100.100.100`); no glibc-level failover |
+| | Boot-order risk: early-boot processes have no DNS until `tailscaled` is up |
+
+### Option B — Pin resolv.conf to the LAN resolvers (`accept-dns=false`)
+
+Set `--accept-dns=false` on every host and keep the managed `resolv.conf`
+pointing at `.252`/`.253`.
+
+| Pros | Cons |
+|---|---|
+| DNS independent of Tailscale — survives `tailscaled` outages | **`*.knel.net` device names break (NXDOMAIN)** because the LAN resolvers' knel.net zone is stale. This breaks wazuh/postfix/syslog hostname resolution. |
+| Lowest latency, full glibc-level failover across two servers | MagicDNS names (`*.ts.net`) do not resolve |
+| Managed `resolv.conf` wins uncontested | Requires fixing the Technitium/Pi-hole `knel.net` zone to mirror the Tailscale device records before this is viable |
+| Boot-time DNS works immediately | Off-LAN hosts (laptops) can't reach `.252`/`.253` without the tunnel — back to needing Tailscale |
+
+> **Not recommended as-is.** Only viable **after** the `knel.net` zone on
+> `.252`/`.253` is repopulated with current device records (see §6).
+
+### Option C — Tailscale Split DNS (per-domain routing)
+
+MagicDNS `ON`, "Override local DNS" `OFF` in the admin console; only `ts.net`
+(and explicitly split domains) route to `100.100.100.100`, everything else stays
+on the system resolver.
+
+| Pros | Cons |
+|---|---|
+| Best of both worlds: MagicDNS names resolve AND general queries go direct | Requires `systemd-resolved` (or NetworkManager `dns=dnsmasq`) for per-domain routing. These hosts use a **plain `/etc/resolv.conf`** — on which Tailscale **cannot** do per-domain split; it replaces the whole file. |
+| Reduces tunnel dependency for non-Tailscale names | Migrating every host to `systemd-resolved` is a significant, cross-cutting change |
+| | More moving parts to reason about and debug |
+
+### Option D — Make Tailscale push the LAN resolvers as global nameservers
+
+In the admin console, set global nameservers to `192.168.3.252`/`192.168.3.253`,
+keep `accept-dns=true`.
+
+| Pros | Cons |
+|---|---|
+| Clients get the LAN resolvers via Tailscale config (consistent) | Tailscale still overwrites `resolv.conf` |
+| MagicDNS still works (100.100.100.100 added for `ts.net`/`knel.net`) | On-LAN hosts don't need Tailscale to find `.252`/`.253` — pure indirection |
+| Centralised management | Still depends on `tailscaled` for DNS |
+| | `knel.net` device names still only resolve via the Tailscale path, so the LAN resolvers being "global" doesn't help those names unless the zone is fixed |
+
+## 5. Recommendation
+
+**Short term (unblock now): Option A — let Tailscale own DNS.** Revert/disable
+the managed-`resolv.conf` deployment so provisioning stops fighting Tailscale.
+Today, `knel.net` device names **only** resolve through Tailscale, and this
+project's modules depend on those names, so Tailscale-managed DNS is the only
+thing that currently works end-to-end. Keep the NTP change (LAN IPs, no DNS
+dependency) — that part is safe and beneficial regardless.
+
+**Medium term (the real fix): populate the `knel.net` zone on the LAN
+resolvers**, then choose B or C. Concretely:
+
+1. Make Technitium (`.252`) authoritative for `knel.net` **with current records**
+   (mirror the Tailscale device→IP mappings, or enable a zone-transfer/sync from
+   the Tailscale device registry, or use Technitium's "Tailscale" DNS app if
+   available). Confirm `dig @192.168.3.252 pfv-netinfra-01.knel.net` returns an
+   answer, not NXDOMAIN.
+2. Make Pi-hole (`.253`) forward `knel.net` to Technitium (or also serve the
+   zone), so both resolvers in the pair can answer internal names — otherwise
+   glibc failover to `.253` would silently break knel.net lookups.
+3. *Then* pin `resolv.conf` to `.252`/`.253` with `--accept-dns=false`
+   (Option B), gaining tunnel-independent, redundant DNS.
+
+**Long term (optional, if per-domain routing is wanted): Option C** — adopt
+`systemd-resolved` and configure Tailscale Split DNS so `ts.net`/`knel.net` go
+to MagicDNS and everything else goes direct. Only worth the migration cost if
+you specifically need `*.ts.net` short-name resolution alongside direct LAN DNS.
+
+### Why not just force `.252`/`.253` today?
+
+Because it regresses name resolution for the hostnames this project already
+uses. Concretely, with `resolv.conf` pinned to the LAN resolvers the following
+would fail to resolve:
+
+- `ProjectCode/Modules/Security/secharden-wazuh.sh` → `WAZUH_MANAGER="tsys-nsm.knel.net"`
+- `ProjectCode/SetupNewSystem.sh` → `postconf -e "relayhost = tsys-cloudron.knel.net"`
+- `ProjectCode/ConfigFiles/Syslog/rsyslog.conf` → `*.* @tsys-librenms.knel.net:514`
+
+All three resolve cleanly via `100.100.100.100` today and return NXDOMAIN via
+`.252`/`.253`. Pinning the LAN resolvers before the zone is fixed would break
+wazuh, mail relay, and syslog.
+
+## 6. Known issues / action items
+
+1. **Technitium `knel.net` zone is stale.** SOA serial `2025062313`
+   (2025-06-23); current device names return NXDOMAIN from the LAN interface.
+   Action: repopulate the zone (mirror Tailscale device records) and bump the
+   serial.
+2. **Pi-hole (`.253`) has no `knel.net` device records either.** For the pair
+   to be truly redundant for internal names, `.253` must either serve the same
+   zone or conditional-forward `knel.net` to `.252`. Action: configure Pi-hole
+   to forward `knel.net` to Technitium.
+3. **The managed-`resolv.conf` change (commit f010fa9) conflicts with
+   Tailscale.** As written, `SetupNewSystem.sh` writes `resolv.conf` with
+   `.252`/`.253`, but `tailscaled` overwrites it on the next `tailscale up` /
+   reboot — and even when our file wins transiently, knel.net names break. See
+   §5 for the recommended handling.
+4. **NTP change is safe and good.** `ntp.conf` now uses LAN IPs
+   (`192.168.3.252`/`192.168.3.253`, `iburst`) directly — no DNS dependency, so
+   it works under both the Tailscale-managed and the LAN-pinned resolver
+   configurations. Keep this regardless of the DNS decision.
+5. **Split-horizon possibility (unconfirmed).** It is possible Technitium serves
+   a richer `knel.net` zone on its Tailscale interface (`100.x`) than on its LAN
+   interface (`192.168.3.252`). If so, the fix is to make the LAN view match the
+   Tailscale view. Worth confirming with `dig @<technitium-tailscale-ip> knel.net host`.
+
+## 7. Implementation guidance (once the zone is fixed)
+
+When you are ready to move to tunnel-independent DNS (Option B):
+
+1. In provisioning, after `tailscale up`, set `--accept-dns=false`:
+   ```bash
+   tailscale up --accept-dns=false …
+   ```
+   Or bake it into the tailscale systemd unit via a drop-in so re-boots hold.
+2. *Then* deploy the managed `resolv.conf` (`.252`/`.253`). Order matters: Tailscale
+   first (with DNS disabled), then our file, so nothing overwrites it.
+3. Add a watchdog (timer) that restores `resolv.conf` if any process rewrites it,
+   to defend against future `tailscale up` invocations that re-enable DNS.
+4. Validate with `Project-Tests/validation/dns-ntp-redundancy.sh` — and extend
+   its probe to assert `*.knel.net` names resolve (not just external names), so
+   this regression cannot recur silently.
+
+## 8. TL;DR
+
+- **DNS**: don't fight Tailscale yet. Today `knel.net` names only resolve via
+  Tailscale, and this project depends on them. Fix the Technitium/Pi-hole
+  `knel.net` zone first, *then* pin the LAN resolvers.
+- **NTP**: the LAN-IP change is correct and safe; keep it.
+- **The managed `resolv.conf` (`.252`/`.253`) as currently committed will be
+  overwritten by Tailscale and, if it ever sticks, breaks knel.net resolution —
+  see §5/§6 before relying on it.**