From 9a4961d94b450f40a36f1f54d3be9c7331a3f9ac Mon Sep 17 00:00:00 2001 From: reachableceo Date: Tue, 28 Jul 2026 05:31:47 -0500 Subject: [PATCH] docs(network): analyze Tailscale vs managed DNS conflict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add an architecture analysis for the tension between Tailscale's default resolv.conf management (100.100.100.100) and the managed LAN-resolver resolv.conf (.252/.253). Documents a key finding from live-network probing: knel.net device records only resolve via the Tailscale MagicDNS path; querying the LAN DNS servers directly returns NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23). Lays out four options (Tailscale-owned, LAN-pinned, split DNS, Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS to Tailscale in the short term (since wazuh/postfix/syslog depend on knel.net names that only resolve there) and fixing the Technitium/Pi-hole knel.net zone before pinning the LAN resolvers. Confirms the NTP (LAN-IP) change is safe regardless. Flags that the managed-resolv.conf change will be overwritten by Tailscale and would break knel.net resolution if it ever sticks. 🤖 Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush --- ProjectDocs/tailscale.md | 217 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100644 ProjectDocs/tailscale.md diff --git a/ProjectDocs/tailscale.md b/ProjectDocs/tailscale.md new file mode 100644 index 0000000..e0b78b1 --- /dev/null +++ b/ProjectDocs/tailscale.md @@ -0,0 +1,217 @@ +# Tailscale vs. Managed DNS — Architecture Analysis + +> **Status:** analysis for review. No code decisions are final. Read the +> "Known issues" section before acting on the managed-resolv.conf change. + +## 1. Executive summary + +Every host in this build runs the Tailscale client, and Tailscale — by default — +**manages `/etc/resolv.conf` itself**, pointing it at `100.100.100.100` +(Tailscale's MagicDNS resolver). This directly conflicts with the managed +`resolv.conf` (pointing at `192.168.3.252`/`192.168.3.253`) that +`SetupNewSystem.sh` deploys: whichever runs last wins, and Tailscale's daemon +re-wins on every `tailscale up` and on reboot. + +Worse, a probe of the live network shows that **knel.net device records only +resolve through the Tailscale 100.100.100.100 path** — querying the LAN IPs of +the DNS servers directly returns NXDOMAIN for current hostnames (the Technitium +`knel.net` zone has the SOA but is stale/empty of actual records). So pointing +`resolv.conf` at the LAN IPs would break resolution of the very names this +project's modules depend on (`tsys-nsm.knel.net`, `tsys-cloudron.knel.net`, +`tsys-librenms.knel.net`). + +This document lays out the options and a recommended path forward. + +## 2. How name resolution actually works today (as measured) + +Probed from `sectestbed-sandbox` (192.168.3.50): + +| Query path | External name (`github.com`) | knel.net device name (`pfv-netinfra-01.knel.net`) | +|---|---|---| +| Via current resolver = `100.100.100.100` (Tailscale) | resolves | **resolves** → `100.70.181.72` (Tailscale CGNAT) | +| Direct `dig @192.168.3.252` (Technitium, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) | +| Direct `dig @192.168.3.253` (Pi-hole, LAN) | resolves (recurses) | **NXDOMAIN** (SOA present, no record) | + +Other measured facts: + +- `dig @192.168.3.252 knel.net SOA` → `NOERROR`, returns + `knel.net. 900 IN SOA dns.knel.net. hostadmin.knel.net. 2025062313 …` + (serial dated **2025-06-23** — the zone exists but is stale). +- NTP on both `.252` and `.253` answers time queries (stratum 2/3). +- The live `/etc/resolv.conf` on a deployed host reads: + ``` + # resolv.conf(5) file generated by tailscale + # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN + nameserver 100.100.100.100 + nameserver fd7a:115c:a1e0::53 + search knel.net + ``` + +**Interpretation:** the `knel.net` device→Tailscale-IP mappings are synthesised +by Tailscale's MagicDNS from the tailnet device registry (every device that +joins the tailnet gets `hostname.knel.net` → its `100.x.x.x` address). The +Technitium `knel.net` zone is a separate, manually-maintained zone that has +fallen out of date. The two are not the same source of truth. + +## 3. The core tension + +| Goal | Who provides it today | +|---|---| +| Resolve `*.knel.net` device names (→ Tailscale IPs) | Tailscale MagicDNS via `100.100.100.100` | +| Resolve external names with ad-blocking | Pi-hole (`.253`), reachable via Tailscale → Technitium → Pi-hole chain | +| Redundant, low-latency, tunnel-independent DNS | LAN resolvers `.252`/`.253` — **but these lack knel.net records** | +| Authoritative time | NTP on `.252`/`.253` (works on either path) | + +The conflict: you cannot simply point `resolv.conf` at the LAN resolvers, +because they do not know about the current `knel.net` device records, and +several modules in this project resolve `knel.net` hostnames at runtime +(wazuh manager, postfix relay, syslog target). You also cannot ignore Tailscale, +because it is the only thing that resolves those names today. + +## 4. Options + +### Option A — Let Tailscale own DNS (status quo, `accept-dns=true`) + +Leave the default. Tailscale writes `100.100.100.100` to `resolv.conf`; the +control-plane forwarding (`100.100.100.100` → Technitium → Pi-hole) handles +external names and ad-blocking; MagicDNS handles `knel.net` device names. + +| Pros | Cons | +|---|---| +| Zero per-host config; new machines "just work" on `tailscale up` | **All DNS depends on the Tailscale daemon being up.** If `tailscaled` dies, every name lookup fails — including the ones you need to SSH in and fix it. | +| MagicDNS + knel.net names resolve automatically | Latency: every query goes host→tailscaled→100.100.100.100→(tunnel)→Technitium→Pi-hole→upstream | +| Ad-blocking preserved (via the Pi-hole hop) | Overwrites the managed `resolv.conf` — the `.252`/`.253` redundancy is lost | +| Centralised in the Tailscale admin console | Single resolver in `resolv.conf` (`100.100.100.100`); no glibc-level failover | +| | Boot-order risk: early-boot processes have no DNS until `tailscaled` is up | + +### Option B — Pin resolv.conf to the LAN resolvers (`accept-dns=false`) + +Set `--accept-dns=false` on every host and keep the managed `resolv.conf` +pointing at `.252`/`.253`. + +| Pros | Cons | +|---|---| +| DNS independent of Tailscale — survives `tailscaled` outages | **`*.knel.net` device names break (NXDOMAIN)** because the LAN resolvers' knel.net zone is stale. This breaks wazuh/postfix/syslog hostname resolution. | +| Lowest latency, full glibc-level failover across two servers | MagicDNS names (`*.ts.net`) do not resolve | +| Managed `resolv.conf` wins uncontested | Requires fixing the Technitium/Pi-hole `knel.net` zone to mirror the Tailscale device records before this is viable | +| Boot-time DNS works immediately | Off-LAN hosts (laptops) can't reach `.252`/`.253` without the tunnel — back to needing Tailscale | + +> **Not recommended as-is.** Only viable **after** the `knel.net` zone on +> `.252`/`.253` is repopulated with current device records (see §6). + +### Option C — Tailscale Split DNS (per-domain routing) + +MagicDNS `ON`, "Override local DNS" `OFF` in the admin console; only `ts.net` +(and explicitly split domains) route to `100.100.100.100`, everything else stays +on the system resolver. + +| Pros | Cons | +|---|---| +| Best of both worlds: MagicDNS names resolve AND general queries go direct | Requires `systemd-resolved` (or NetworkManager `dns=dnsmasq`) for per-domain routing. These hosts use a **plain `/etc/resolv.conf`** — on which Tailscale **cannot** do per-domain split; it replaces the whole file. | +| Reduces tunnel dependency for non-Tailscale names | Migrating every host to `systemd-resolved` is a significant, cross-cutting change | +| | More moving parts to reason about and debug | + +### Option D — Make Tailscale push the LAN resolvers as global nameservers + +In the admin console, set global nameservers to `192.168.3.252`/`192.168.3.253`, +keep `accept-dns=true`. + +| Pros | Cons | +|---|---| +| Clients get the LAN resolvers via Tailscale config (consistent) | Tailscale still overwrites `resolv.conf` | +| MagicDNS still works (100.100.100.100 added for `ts.net`/`knel.net`) | On-LAN hosts don't need Tailscale to find `.252`/`.253` — pure indirection | +| Centralised management | Still depends on `tailscaled` for DNS | +| | `knel.net` device names still only resolve via the Tailscale path, so the LAN resolvers being "global" doesn't help those names unless the zone is fixed | + +## 5. Recommendation + +**Short term (unblock now): Option A — let Tailscale own DNS.** Revert/disable +the managed-`resolv.conf` deployment so provisioning stops fighting Tailscale. +Today, `knel.net` device names **only** resolve through Tailscale, and this +project's modules depend on those names, so Tailscale-managed DNS is the only +thing that currently works end-to-end. Keep the NTP change (LAN IPs, no DNS +dependency) — that part is safe and beneficial regardless. + +**Medium term (the real fix): populate the `knel.net` zone on the LAN +resolvers**, then choose B or C. Concretely: + +1. Make Technitium (`.252`) authoritative for `knel.net` **with current records** + (mirror the Tailscale device→IP mappings, or enable a zone-transfer/sync from + the Tailscale device registry, or use Technitium's "Tailscale" DNS app if + available). Confirm `dig @192.168.3.252 pfv-netinfra-01.knel.net` returns an + answer, not NXDOMAIN. +2. Make Pi-hole (`.253`) forward `knel.net` to Technitium (or also serve the + zone), so both resolvers in the pair can answer internal names — otherwise + glibc failover to `.253` would silently break knel.net lookups. +3. *Then* pin `resolv.conf` to `.252`/`.253` with `--accept-dns=false` + (Option B), gaining tunnel-independent, redundant DNS. + +**Long term (optional, if per-domain routing is wanted): Option C** — adopt +`systemd-resolved` and configure Tailscale Split DNS so `ts.net`/`knel.net` go +to MagicDNS and everything else goes direct. Only worth the migration cost if +you specifically need `*.ts.net` short-name resolution alongside direct LAN DNS. + +### Why not just force `.252`/`.253` today? + +Because it regresses name resolution for the hostnames this project already +uses. Concretely, with `resolv.conf` pinned to the LAN resolvers the following +would fail to resolve: + +- `ProjectCode/Modules/Security/secharden-wazuh.sh` → `WAZUH_MANAGER="tsys-nsm.knel.net"` +- `ProjectCode/SetupNewSystem.sh` → `postconf -e "relayhost = tsys-cloudron.knel.net"` +- `ProjectCode/ConfigFiles/Syslog/rsyslog.conf` → `*.* @tsys-librenms.knel.net:514` + +All three resolve cleanly via `100.100.100.100` today and return NXDOMAIN via +`.252`/`.253`. Pinning the LAN resolvers before the zone is fixed would break +wazuh, mail relay, and syslog. + +## 6. Known issues / action items + +1. **Technitium `knel.net` zone is stale.** SOA serial `2025062313` + (2025-06-23); current device names return NXDOMAIN from the LAN interface. + Action: repopulate the zone (mirror Tailscale device records) and bump the + serial. +2. **Pi-hole (`.253`) has no `knel.net` device records either.** For the pair + to be truly redundant for internal names, `.253` must either serve the same + zone or conditional-forward `knel.net` to `.252`. Action: configure Pi-hole + to forward `knel.net` to Technitium. +3. **The managed-`resolv.conf` change (commit f010fa9) conflicts with + Tailscale.** As written, `SetupNewSystem.sh` writes `resolv.conf` with + `.252`/`.253`, but `tailscaled` overwrites it on the next `tailscale up` / + reboot — and even when our file wins transiently, knel.net names break. See + §5 for the recommended handling. +4. **NTP change is safe and good.** `ntp.conf` now uses LAN IPs + (`192.168.3.252`/`192.168.3.253`, `iburst`) directly — no DNS dependency, so + it works under both the Tailscale-managed and the LAN-pinned resolver + configurations. Keep this regardless of the DNS decision. +5. **Split-horizon possibility (unconfirmed).** It is possible Technitium serves + a richer `knel.net` zone on its Tailscale interface (`100.x`) than on its LAN + interface (`192.168.3.252`). If so, the fix is to make the LAN view match the + Tailscale view. Worth confirming with `dig @ knel.net host`. + +## 7. Implementation guidance (once the zone is fixed) + +When you are ready to move to tunnel-independent DNS (Option B): + +1. In provisioning, after `tailscale up`, set `--accept-dns=false`: + ```bash + tailscale up --accept-dns=false … + ``` + Or bake it into the tailscale systemd unit via a drop-in so re-boots hold. +2. *Then* deploy the managed `resolv.conf` (`.252`/`.253`). Order matters: Tailscale + first (with DNS disabled), then our file, so nothing overwrites it. +3. Add a watchdog (timer) that restores `resolv.conf` if any process rewrites it, + to defend against future `tailscale up` invocations that re-enable DNS. +4. Validate with `Project-Tests/validation/dns-ntp-redundancy.sh` — and extend + its probe to assert `*.knel.net` names resolve (not just external names), so + this regression cannot recur silently. + +## 8. TL;DR + +- **DNS**: don't fight Tailscale yet. Today `knel.net` names only resolve via + Tailscale, and this project depends on them. Fix the Technitium/Pi-hole + `knel.net` zone first, *then* pin the LAN resolvers. +- **NTP**: the LAN-IP change is correct and safe; keep it. +- **The managed `resolv.conf` (`.252`/`.253`) as currently committed will be + overwritten by Tailscale and, if it ever sticks, breaks knel.net resolution — + see §5/§6 before relying on it.**