KNELServerBuild/ProjectDocs/tailscale.md

Tailscale vs. Managed DNS — Architecture Analysis

Status: analysis for review. No code decisions are final. Read the "Known issues" section before acting on the managed-resolv.conf change.

1. Executive summary

Every host in this build runs the Tailscale client, and Tailscale — by default — manages /etc/resolv.conf itself, pointing it at 100.100.100.100 (Tailscale's MagicDNS resolver). This directly conflicts with the managed resolv.conf (pointing at 192.168.3.252/192.168.3.253) that SetupNewSystem.sh deploys: whichever runs last wins, and Tailscale's daemon re-wins on every tailscale up and on reboot.

Worse, a probe of the live network shows that knel.net device records only resolve through the Tailscale 100.100.100.100 path — querying the LAN IPs of the DNS servers directly returns NXDOMAIN for current hostnames (the Technitium knel.net zone has the SOA but is stale/empty of actual records). So pointing resolv.conf at the LAN IPs would break resolution of the very names this project's modules depend on (tsys-nsm.knel.net, tsys-cloudron.knel.net, tsys-librenms.knel.net).

This document lays out the options and a recommended path forward.

2. How name resolution actually works today (as measured)

Probed from sectestbed-sandbox (192.168.3.50):

Query path	External name (github.com)	knel.net device name (pfv-netinfra-01.knel.net)
Via current resolver = 100.100.100.100 (Tailscale)	resolves	resolves → 100.70.181.72 (Tailscale CGNAT)
Direct dig @192.168.3.252 (Technitium, LAN)	resolves (recurses)	NXDOMAIN (SOA present, no record)
Direct dig @192.168.3.253 (Pi-hole, LAN)	resolves (recurses)	NXDOMAIN (SOA present, no record)

Other measured facts:

• dig @192.168.3.252 knel.net SOA → NOERROR, returns knel.net. 900 IN SOA dns.knel.net. hostadmin.knel.net. 2025062313 … (serial dated 2025-06-23 — the zone exists but is stale).
• NTP on both .252 and .253 answers time queries (stratum 2/3).
• The live /etc/resolv.conf on a deployed host reads:

  # resolv.conf(5) file generated by tailscale
  # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
  nameserver 100.100.100.100
  nameserver fd7a:115c:a1e0::53
  search knel.net
  

Interpretation: the knel.net device→Tailscale-IP mappings are synthesised by Tailscale's MagicDNS from the tailnet device registry (every device that joins the tailnet gets hostname.knel.net → its 100.x.x.x address). The Technitium knel.net zone is a separate, manually-maintained zone that has fallen out of date. The two are not the same source of truth.

3. The core tension

Goal	Who provides it today
Resolve *.knel.net device names (→ Tailscale IPs)	Tailscale MagicDNS via 100.100.100.100
Resolve external names with ad-blocking	Pi-hole (.253), reachable via Tailscale → Technitium → Pi-hole chain
Redundant, low-latency, tunnel-independent DNS	LAN resolvers .252/.253 — but these lack knel.net records
Authoritative time	NTP on .252/.253 (works on either path)

The conflict: you cannot simply point resolv.conf at the LAN resolvers, because they do not know about the current knel.net device records, and several modules in this project resolve knel.net hostnames at runtime (wazuh manager, postfix relay, syslog target). You also cannot ignore Tailscale, because it is the only thing that resolves those names today.

4. Options

Option A — Let Tailscale own DNS (status quo, accept-dns=true)

Leave the default. Tailscale writes 100.100.100.100 to resolv.conf; the control-plane forwarding (100.100.100.100 → Technitium → Pi-hole) handles external names and ad-blocking; MagicDNS handles knel.net device names.

Pros	Cons
Zero per-host config; new machines "just work" on tailscale up	All DNS depends on the Tailscale daemon being up. If tailscaled dies, every name lookup fails — including the ones you need to SSH in and fix it.
MagicDNS + knel.net names resolve automatically	Latency: every query goes host→tailscaled→100.100.100.100→(tunnel)→Technitium→Pi-hole→upstream
Ad-blocking preserved (via the Pi-hole hop)	Overwrites the managed resolv.conf — the .252/.253 redundancy is lost
Centralised in the Tailscale admin console	Single resolver in resolv.conf (100.100.100.100); no glibc-level failover
	Boot-order risk: early-boot processes have no DNS until tailscaled is up

Option B — Pin resolv.conf to the LAN resolvers (accept-dns=false)

Set --accept-dns=false on every host and keep the managed resolv.conf pointing at .252/.253.

Pros	Cons
DNS independent of Tailscale — survives tailscaled outages	*.knel.net device names break (NXDOMAIN) because the LAN resolvers' knel.net zone is stale. This breaks wazuh/postfix/syslog hostname resolution.
Lowest latency, full glibc-level failover across two servers	MagicDNS names (*.ts.net) do not resolve
Managed resolv.conf wins uncontested	Requires fixing the Technitium/Pi-hole knel.net zone to mirror the Tailscale device records before this is viable
Boot-time DNS works immediately	Off-LAN hosts (laptops) can't reach .252/.253 without the tunnel — back to needing Tailscale

Not recommended as-is. Only viable after the knel.net zone on .252/.253 is repopulated with current device records (see §6).

Option C — Tailscale Split DNS (per-domain routing)

MagicDNS ON, "Override local DNS" OFF in the admin console; only ts.net (and explicitly split domains) route to 100.100.100.100, everything else stays on the system resolver.

Pros	Cons
Best of both worlds: MagicDNS names resolve AND general queries go direct	Requires systemd-resolved (or NetworkManager dns=dnsmasq) for per-domain routing. These hosts use a plain /etc/resolv.conf — on which Tailscale cannot do per-domain split; it replaces the whole file.
Reduces tunnel dependency for non-Tailscale names	Migrating every host to systemd-resolved is a significant, cross-cutting change
	More moving parts to reason about and debug

Option D — Make Tailscale push the LAN resolvers as global nameservers

In the admin console, set global nameservers to 192.168.3.252/192.168.3.253, keep accept-dns=true.

Pros	Cons
Clients get the LAN resolvers via Tailscale config (consistent)	Tailscale still overwrites resolv.conf
MagicDNS still works (100.100.100.100 added for ts.net/knel.net)	On-LAN hosts don't need Tailscale to find .252/.253 — pure indirection
Centralised management	Still depends on tailscaled for DNS
	knel.net device names still only resolve via the Tailscale path, so the LAN resolvers being "global" doesn't help those names unless the zone is fixed

5. Recommendation

Short term (unblock now): Option A — let Tailscale own DNS. Revert/disable the managed-resolv.conf deployment so provisioning stops fighting Tailscale. Today, knel.net device names only resolve through Tailscale, and this project's modules depend on those names, so Tailscale-managed DNS is the only thing that currently works end-to-end. Keep the NTP change (LAN IPs, no DNS dependency) — that part is safe and beneficial regardless.

Medium term (the real fix): populate the knel.net zone on the LAN resolvers, then choose B or C. Concretely:

1. Make Technitium (.252) authoritative for knel.net with current records (mirror the Tailscale device→IP mappings, or enable a zone-transfer/sync from the Tailscale device registry, or use Technitium's "Tailscale" DNS app if available). Confirm dig @192.168.3.252 pfv-netinfra-01.knel.net returns an answer, not NXDOMAIN.
2. Make Pi-hole (.253) forward knel.net to Technitium (or also serve the zone), so both resolvers in the pair can answer internal names — otherwise glibc failover to .253 would silently break knel.net lookups.
3. Then pin resolv.conf to .252/.253 with --accept-dns=false (Option B), gaining tunnel-independent, redundant DNS.

Long term (optional, if per-domain routing is wanted): Option C — adopt systemd-resolved and configure Tailscale Split DNS so ts.net/knel.net go to MagicDNS and everything else goes direct. Only worth the migration cost if you specifically need *.ts.net short-name resolution alongside direct LAN DNS.

Why not just force .252/.253 today?

Because it regresses name resolution for the hostnames this project already uses. Concretely, with resolv.conf pinned to the LAN resolvers the following would fail to resolve:

• ProjectCode/Modules/Security/secharden-wazuh.sh → WAZUH_MANAGER="tsys-nsm.knel.net"
• ProjectCode/SetupNewSystem.sh → postconf -e "relayhost = tsys-cloudron.knel.net"
• ProjectCode/ConfigFiles/Syslog/rsyslog.conf → *.* @tsys-librenms.knel.net:514

All three resolve cleanly via 100.100.100.100 today and return NXDOMAIN via .252/.253. Pinning the LAN resolvers before the zone is fixed would break wazuh, mail relay, and syslog.

6. Known issues / action items

1. Technitium knel.net zone is stale. SOA serial 2025062313 (2025-06-23); current device names return NXDOMAIN from the LAN interface. Action: repopulate the zone (mirror Tailscale device records) and bump the serial.
2. Pi-hole (.253) has no knel.net device records either. For the pair to be truly redundant for internal names, .253 must either serve the same zone or conditional-forward knel.net to .252. Action: configure Pi-hole to forward knel.net to Technitium.
3. The managed-resolv.conf change (commit f010fa9) conflicts with Tailscale. As written, SetupNewSystem.sh writes resolv.conf with .252/.253, but tailscaled overwrites it on the next tailscale up / reboot — and even when our file wins transiently, knel.net names break. See §5 for the recommended handling.
4. NTP change is safe and good. ntp.conf now uses LAN IPs (192.168.3.252/192.168.3.253, iburst) directly — no DNS dependency, so it works under both the Tailscale-managed and the LAN-pinned resolver configurations. Keep this regardless of the DNS decision.
5. Split-horizon possibility (unconfirmed). It is possible Technitium serves a richer knel.net zone on its Tailscale interface (100.x) than on its LAN interface (192.168.3.252). If so, the fix is to make the LAN view match the Tailscale view. Worth confirming with dig @<technitium-tailscale-ip> knel.net host.

7. Implementation guidance (once the zone is fixed)

When you are ready to move to tunnel-independent DNS (Option B):

1. In provisioning, after tailscale up, set --accept-dns=false:

   tailscale up --accept-dns=false …
   

   Or bake it into the tailscale systemd unit via a drop-in so re-boots hold.
2. Then deploy the managed resolv.conf (.252/.253). Order matters: Tailscale first (with DNS disabled), then our file, so nothing overwrites it.
3. Add a watchdog (timer) that restores resolv.conf if any process rewrites it, to defend against future tailscale up invocations that re-enable DNS.
4. Validate with Project-Tests/validation/dns-ntp-redundancy.sh — and extend its probe to assert *.knel.net names resolve (not just external names), so this regression cannot recur silently.

8. TL;DR

• DNS: don't fight Tailscale yet. Today knel.net names only resolve via Tailscale, and this project depends on them. Fix the Technitium/Pi-hole knel.net zone first, then pin the LAN resolvers.
• NTP: the LAN-IP change is correct and safe; keep it.
• The managed resolv.conf (.252/.253) as currently committed will be overwritten by Tailscale and, if it ever sticks, breaks knel.net resolution — see §5/§6 before relying on it.