KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
netinfra/dhcp-migration.md
+9 -124
View File
@@ -1,125 +1,10 @@
# DHCP Migration: pfv-netboot → pfv-netinfra-01/02
# netinfra/dhcp-migration.md
**Date:** 2026-07-29
**Status:** ✅ **COMPLETE** — both servers in `normal` failover state, 333
active leases load-balanced. Netboot DHCP stopped + disabled.
> **Safety nets:** pfv-netboot, pfv-netinfra-01/02, sectestbed-sandbox, and
> ultix-streaming all have static IPs (reachable via Tailscale even if DHCP
> fails). sectestbed-sandbox snapshot `pre-dhcp-migration` exists as rollback.
## Migration results (2026-07-29 17:20 CDT)
| Check | Result |
|-------|--------|
| netinfra-01 (primary) | `active`, failover `normal`, 244 active leases |
| netinfra-02 (secondary) | `active`, failover `normal`, 89 active leases |
| Failover partnership | "Both servers normal" — load-balanced at split=128 |
| DHCP test (sandbox) | Lease obtained from 192.168.3.252, IP 192.168.3.50/22 |
| DNS handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
| NTP handed out | `192.168.3.252, 192.168.3.253` (netinfra pair) |
| Domain | `knel.net` |
| Gateway | `192.168.3.254` |
| Webmin | Port 10000 (SSL) on both nodes; DHCP module configured |
| Netboot DHCP | Stopped + disabled (won't start on reboot) |
| Netmask fix | Both nodes corrected /24 → /22 |
## 1. Current state (discovered 2026-07-29)
| | pfv-netboot (SOURCE) | pfv-netinfra-01 (TARGET primary) | pfv-netinfra-02 (TARGET secondary) |
|---|---|---|---|
| IP | 192.168.3.250/22 | 192.168.3.252/24 | 192.168.3.253/24 |
| isc-dhcp-server | 4.4.3, **active** | NOT installed | NOT installed |
| Webmin | 2.652 (port 10000, SSL) | NOT installed | NOT installed |
| DHCP leases | **532 active** | — | — |
| DHCP config | `/etc/dhcp/dhcpd.conf` | — | — |
**Source DHCP config summary:**
- Subnet: `192.168.0.0/22` (covers .0.x/.1.x/.2.x/.3.x)
- Pool range: `192.168.0.1 — 192.168.3.200`
- Router/GW: `192.168.3.254`
- DNS: `192.168.3.250` (netboot itself — **changing to 252/253**)
- Domain: `knel.net`
- Lease times: 600s default, 7200s max
- DDNS: `ddns-update-style none` (globally disabled)
- ~37 host reservations (MAC → fixed IP)
- 3 global-scope host declarations (outside subnet)
## 2. What changes in the migrated config
| Setting | Old (netboot) | New (netinfra) | Why |
|---------|--------------|----------------|-----|
| DNS servers handed out | `192.168.3.250` | `192.168.3.252, 192.168.3.253` | Point clients at the netinfra Pi-hole pair (netboot will be decommissioned) |
| Failover | none (single server) | ISC DHCP failover-peer | Primary/secondary HA |
| DHCP interface | `eth0` | `ens18` | Different NIC name on netinfra VMs |
| Netmask on servers | /22 | **fix to /22** (currently /24) | Match the /22 network |
Everything else (subnet, pool range, reservations, lease times, domain) stays
identical. The host reservation list is copied verbatim.
## 3. Configs prepared
| File | Destination |
|------|-------------|
| [`dhcp/dhcpd-primary.conf`](dhcp/dhcpd-primary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-01 |
| [`dhcp/dhcpd-secondary.conf`](dhcp/dhcpd-secondary.conf) | `/etc/dhcp/dhcpd.conf` on netinfra-02 |
Both are identical except for the `failover peer` block (primary vs secondary).
## 4. Execution sequence
### Phase 1 — pre-flight (non-disruptive)
1. **Snapshot sectestbed-sandbox** (VMID 6000 on tsys5) — rollback point
2. **Fix netmask** on netinfra-01/02: `/24` → `/22` in `/etc/network/interfaces`
(then `ifdown ens18 && ifup ens18`, or `systemctl restart networking`)
3. Verify Tailscale still reachable after netmask change
### Phase 2 — install prerequisites (non-disruptive, nothing serving yet)
On **both** netinfra-01 and netinfra-02:
1. `apt-get update && apt-get install -y isc-dhcp-server`
2. Install Webmin via official setup script
3. Copy `dhcpd-{primary,secondary}.conf` → `/etc/dhcp/dhcpd.conf`
4. Write `/etc/default/isc-dhcp-server` with `INTERFACESv4="ens18"`
5. Syntax check: `dhcpd -t -cf /etc/dhcp/dhcpd.conf`
6. **Do NOT start the DHCP service yet**
### Phase 3 — cutover (the critical moment)
**Goal:** minimize the DHCP gap. Existing clients hold valid leases for 600
seconds — they won't notice a brief outage.
1. **Stop netboot DHCP:** `ssh netboot 'sudo systemctl stop isc-dhcp-server'`
2. **Immediately start netinfra-01:** `systemctl start isc-dhcp-server`
3. **Immediately start netinfra-02:** `systemctl start isc-dhcp-server`
4. **Verify failover partnership:** check `journalctl -u isc-dhcp-server` on both
— look for "failover peer pfv-dhcp: I move from recover to recover-done"
5. Wait ~10 seconds for lease sync
### Phase 4 — test
1. On sectestbed-sandbox: edit `/etc/network/interfaces` to DHCP
2. `dhclient -r` (release) then `dhclient` (request new lease)
3. Verify IP assigned, correct DNS, correct gateway
4. If fails: `qm rollback 6000 <snapshot>` and debug
5. If succeeds: change back to static (sandbox's normal config)
### Phase 5 — cleanup
1. **Disable netboot DHCP:** `ssh netboot 'sudo systemctl disable isc-dhcp-server'`
2. Verify Webmin DHCP module visible at `https://192.168.3.252:10000`
3. Document + commit
## 5. Failover mechanics (reference)
ISC DHCP failover uses a TCP connection on port **647** between the two servers.
- **Primary** owns the pool initially and can assign new leases immediately.
- **Secondary** enters `recover` state, syncs the lease database from primary,
then transitions to `recover-done` → `normal`.
- If primary dies, secondary continues serving after the MCLT timeout (600s).
- If secondary dies, primary continues serving alone.
- **split 128** = load balance 50/50 (primary handles even hashes, secondary odd).
- Host reservations (fixed-address) are NOT subject to failover — both servers
can independently assign reserved IPs.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **DHCP migration to netinfra-01/02**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
netinfra/dns-cluster-setup/README.md
+9 -184
View File
@@ -1,185 +1,10 @@
# Technitium DNS Cluster Setup
# netinfra/dns-cluster-setup/README.md
> **Redmine:** [#357](https://projects.knownelement.com/issues/357) (deployment, closed)
Replicates the production Technitium DNS Server from `tailscale-router` to the
`pfv-netinfra-01/02` pair and configures them as a primary/secondary cluster
with automatic zone transfers.
## Architecture
```
tailscale-router (PRODUCTION — READ ONLY)
└─ tsys-dns container (technitium/dns-server)
└─ 124 zones (knel.net + reverse DNS)
└─ Users + 2FA in auth.config
│
docker cp (export)
│
▼
┌─ pfv-netinfra-01 (192.168.3.252) ──── PRIMARY ──────────┐
│ tsys-dns container (Technitium on :5300) │
│ pihole container (Pi-hole on :53 → Technitium :5300) │
│ All zones are Primary │
│ Zone transfer allowed from 192.168.3.253 │
└──────────────────────────────────────────────────────────┘
│
AXFR / IXFR + NOTIFY (DNS zone transfer, port 5300)
│
▼
┌─ pfv-netinfra-02 (192.168.3.253) ─── SECONDARY ────────┐
│ tsys-dns container (Technitium on :5300) │
│ pihole container (Pi-hole on :53 → Technitium :5300) │
│ All zones are Secondary (AXFR from 01) │
└──────────────────────────────────────────────────────────┘
```
### How clustering works
Technitium uses standard DNS zone transfers (AXFR/IXFR) for primary/secondary
replication, not a proprietary protocol:
1. **Primary (01)** holds all zones as authoritative primary zones.
2. **Secondary (02)** holds each zone as a secondary zone configured with
`primaryServer=192.168.3.252:5300`.
3. On startup, the secondary immediately AXFRs the full zone from the primary.
4. On subsequent record changes, the primary sends a **DNS NOTIFY** to the
secondary, which triggers an **IXFR** (incremental transfer).
5. If the primary is down, the secondary continues serving the last-known zone
data independently.
### Credentials and 2FA
The production `auth.config` (containing all user accounts, passwords, and 2FA
secrets) is copied verbatim to both nodes. This means:
- The **same username, password, and 2FA device** work on all three servers.
- The web console is at `http://<host>:5380/` on each node.
- No credential changes are needed.
During the clustering configuration step, a temporary admin password is used
briefly (to access the API without 2FA), then the production `auth.config` is
restored. See "Security notes" below.
## Prerequisites
- SSH key access to all hosts as `localuser` with passwordless sudo.
- The `remote-dns.sh` wrapper must be able to reach all hosts via Tailscale FQDN.
- Docker + Docker Compose on netinfra-01/02 (already installed).
- The production Technitium on tailscale-router must be running.
## Usage
```bash
cd dns-cluster-setup/
# Step-by-step (recommended for first run):
./setup.sh export # 1. Export config from tailscale-router (READ-ONLY)
./setup.sh deploy01 # 2. Deploy to netinfra-01 as primary
./setup.sh deploy02 # 3. Deploy to netinfra-02 as secondary clone
./setup.sh cluster # 4. Configure clustering (01→02 zone transfers)
./setup.sh verify # 5. Run all verification tests
# Or all at once:
./setup.sh all
```
### Configuration overrides
All defaults can be overridden via environment variables:
| Variable | Default | Description |
|---|---|---|
| `PRIMARY_IP` | `192.168.3.252` | netinfra-01 LAN IP |
| `SECONDARY_IP` | `192.168.3.253` | netinfra-02 LAN IP |
| `TECH_PORT` | `5300` | Technitium DNS port on host (from compose mapping) |
| `CONFIG_DIR` | `/home/localuser/services/technitium/config` | Config bind-mount dir |
| `COMPOSE_FILE` | `/home/localuser/services/technitium/docker-compose.yml` | Compose file |
| `TEMP_ADMIN_PW` | `KnelClusterSetup!2026` | Temp admin password (used only during clustering, then discarded) |
## Scripts
| Script | Purpose |
|---|---|
| `remote-dns.sh` | SSH/SCP chokepoint for all DNS host access (tsrouter, netinfra01, netinfra02, netboot, sandbox) |
| `setup.sh` | Master orchestrator: export → deploy → cluster → verify |
| `verify.sh` | Comprehensive 10-section verification suite |
| `discover*.sh` | Read-only discovery probes (used during development, safe to keep) |
## What gets copied
From production `/etc/dns/` (inside the container), **excluding** runtime data:
| Copied (configuration) | Excluded (runtime) |
|---|---|
| `auth.config` (users, passwords, 2FA) | `cache.bin` (DNS cache) |
| `dns.config` (server settings) | `stats/` (query statistics) |
| `webservice.config` (web console) | `logs/` (log files) |
| `allowed.config` (zone transfer ACL) | |
| `blocked.config` (blocked domains) | |
| `blocklist.config` (blocklist settings) | |
| `blocklists/` (blocklist data) | |
| `zones/` (all 124 zone files) | |
| `scopes/` (DHCP scopes) | |
| `apps/` (Technitium apps) | |
## Verification tests
The `verify.sh` script runs 10 categories of tests:
1. **Container health** — both Technitium containers are Up
2. **API responds** — web console API is reachable on both nodes
3. **Zone count** — primary matches production; secondary matches primary
4. **Forward DNS** — known knel.net records resolve identically on both nodes
5. **External DNS** — both nodes can resolve external domains (github.com)
6. **Zone transfer (AXFR)** — secondary can AXFR knel.net from primary
7. **Reverse DNS** — PTR zones have SOA records on both nodes
8. **Production untouched** — container still running, zone count unchanged
9. **Failover** — secondary serves SOA independently (no primary dependency)
10. **Credentials** — `auth.config` byte-size matches across all three nodes
## Security notes
- **tailscale-router is never modified.** The only operation is `docker cp`
(read) to export the config. No writes, no restarts, no config changes.
- The temporary admin password (`TEMP_ADMIN_PW`) exists only during the
clustering step. After configuration, the production `auth.config` (with 2FA)
is restored. The temp password is never persisted.
- The export tarball (`.export/technitium-production-config.tar.gz`) contains
production credentials. It is in `.gitignore` and should be deleted after
setup: `rm -rf dns-cluster-setup/.export/`
- Each node's existing config is backed up to `config.backup-<timestamp>` before
replacement, so the change is reversible.
## Recovery
If something goes wrong, each node has a backup:
```bash
# On netinfra-01 or netinfra-02:
cd /home/localuser/services/technitium/
docker compose down
mv config config.failed
mv config.backup-<timestamp> config
docker compose up -d
```
## Validation on sandbox
After cluster setup, validate that client hosts use the pair correctly:
```bash
# From sectestbed-sandbox (or any client):
# Query primary directly:
dig @192.168.3.252 pfv-netinfra-01.knel.net
# Query secondary directly:
dig @192.168.3.253 pfv-netinfra-01.knel.net
# Both should return the same answer.
```
The KNELServerBuild provisioning code (`provisioning/ConfigFiles/NTP/ntp.conf`
and `provisioning/ConfigFiles/Resolv/resolv.conf`) points clients at both
servers for DNS and NTP redundancy. See `docs/server-build/tailscale.md` for the
full DNS architecture analysis.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Technitium DNS cluster setup**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
netinfra/pfv-netboot-setup.md
+9 -115
View File
@@ -1,116 +1,10 @@
# pfv-netboot — Reference Network Infrastructure (READ-ONLY reference)
# netinfra/pfv-netboot-setup.md
> **Status:** REFERENCE SOURCE ONLY. This node is production infrastructure.
> Do **not** modify it. This document describes it as audited so its services can
> be replicated to `pfv-netinfra-01` / `pfv-netinfra-02`. All data below was
> collected by **read-only** audit scripts (`audit-netboot.sh`,
> `deep-audit-netboot.sh`, `gather-configs.sh`) on 2026-07-27/28.
## 1. Host
| Item | Value |
|---|---|
| Hostname / FQDN | `pfv-netboot` / `pfv-netboot.knel.net` |
| OS | Debian GNU/Linux 12 (bookworm), kernel 6.1.0-44-amd64 |
| Hardware | 2 vCPU, ~1.9 GiB RAM, 491 GB disk (18 GB used) |
| Timezone | `America/Chicago` (US/Central) |
| LAN | `eth0` static `192.168.3.250/22`, gw `192.168.3.254` (`/etc/network/interfaces`) |
| Tailscale | `100.103.64.82` (`tailscale0`) |
| DNS resolver | Tailscale MagicDNS — `/etc/resolv.conf` → `100.100.100.100` |
| Docker | Docker Engine 29.6.2 (containerd v2.2.6, runc 1.3.6) |
| Access | `localuser` has passwordless sudo; **not** in `docker` group (uses `sudo docker`) |
`eth1` is up but unconfigured; many docker bridges exist (`pihole_default`,
`ntp_default`, `dns_default`, and several stale ones).
## 2. Services overview
| Service | Form | Running? |
|---|---|---|
| **Pi-hole** (DNS sinkhole, recursive resolver) | Docker container `pihole` | ✅ healthy |
| **NTP** — overlay on Tailscale IP | Docker container `tsys-ntp` (`dockurr/chrony`) | ✅ healthy |
| **NTP** — system clock + LAN serving | bare-metal `ntpsec` (`ntpd`) | ✅ active, enabled |
| **Technitium DNS** (authoritative for `knel.net`) | Docker container | ❌ **not running**; config preserved in orphaned volume |
## 3. Pi-hole (container)
- **Compose:** `/root/pihole/docker-compose.yml` (compose project `pihole`)
- **Image:** `pihole/pihole:latest` — Core **v6.1.2**, Web v6.2.1, FTL v6.2.2
- **Container:** `pihole`, `restart: always`, `cap_add: [SYS_NICE]`, network `pihole_default`
- **Ports (host):**
| Host | Container | Purpose |
|---|---|---|
| `53/tcp`, `53/udp` | 53 | DNS |
| `10002/tcp` | 80 | Web admin (HTTP) |
| `10003/tcp` | 443 | Web admin (HTTPS, self-signed) |
- **Environment:** `TZ=America/Chicago`, `FTLCONF_webserver_api_password=REDACTED_PASSWORD`, `FTLCONF_dns_listeningMode=all`
- **Data:** bind mount `/root/pihole/etc-pihole:/etc/pihole` (dir owned by `localuser`; files by container `pihole` uid)
- **Config (Pi-hole v6 TOML):** `pihole.toml`. Key settings:
- Upstream DNS: `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`
- `listeningMode = "ALL"`, `interface = "eth0"`, `dns.port = 53`, `dns.domain = "lan"`
- `queryLogging = true`, DNSSEC off
- **Adlists:** one entry — `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` (in `gravity.db`/`adlists.list`)
- **Gravity DB:** `/etc/pihole/gravity.db` (~5.5 MB) holds adlists/domainlists/clients/groups
- **Web admin:** `http://pfv-netboot:10002/admin/` (password `REDACTED_PASSWORD`)
- Note: query history `pihole-FTL.db` (~2.5 GB) is transient and **excluded** from replication.
## 4. NTP (two layers)
### 4a. chrony container (`tsys-ntp`) — overlay on the Tailscale IP
- **Compose:** `/root/NTP/docker-compose.yml` (project `ntp`)
- **Image:** `dockurr/chrony`
- **Env:** `NTP_SERVERS=pool.ntp.org`
- **Ports:** `100.103.64.82:123:123/udp` — bound specifically to the **Tailscale IP**
- `restart: always`
- chrony.conf (generated): `server pool.ntp.org iburst`, `allow all`, `rtcsync`
- On netboot this coexists with bare-metal ntpsec because ntpsec here does **not** pre-bind the specific Tailscale-IP socket, letting Docker claim it.
### 4b. bare-metal `ntpsec`
- Unit `ntpsec.service` — active, enabled; `/usr/sbin/ntpd -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec`
- **Config** (`/etc/ntpsec/ntp.conf`):
```
driftfile /var/lib/ntp/ntp.drift
leapfile /usr/share/zoneinfo/leap-seconds.list
server pfvsvrpi.knel.net
restrict 127.0.0.1
restrict ::1
```
- Listens on all local addresses (incl. Tailscale) for UDP/123; serves LAN clients.
## 5. Technitium DNS (currently stopped)
- **Not running** — no container and **no compose file** exists for it.
- A previous deployment left an **orphaned Docker volume** `dns_tsys-dns-config`
(mountpoint `/var/lib/docker/volumes/dns_tsys-dns-config/_data`) whose contents
are intact (last activity 2025-06-23). A second typo'd volume
`dns_tyss-dns-config` is empty.
- Config files are **binary** (Technitium's own serialization), but copy verbatim:
`dns.config`, `auth.config`, `log.config`, `scopes/Default.scope`,
`self-signed-cert.pfx`, `cache.bin`, `zones/`, `stats/`, `logs/`.
- **Zones present** (12 reverse + 1 forward):
- `knel.net.zone` — forward zone; SOA `dns.knel.net. hostadmin.knel.net.` (serial `2025062313`). A-records for the internal fleet, including: `tsys1`, `rr-middleware`, `pfv-netboot`, `pfv-k8s-cnode1`…`cnode5`, `pfv-k8s-wnode3`, `tsys-k8scloud-netcup-1`, `tsys-kali-vptechops`, `tsys-kali-dev`; NS `dns.knel.net`.
- Reverse zones for Tailscale CGNAT ranges (`100.x.in-addr.arpa`): `199.86`, `145.105`, `181.103`, `184.108`, `194.67`, `2.108`, `211.114`, `46.96`, `64.103`, `75.110`, `97.82`, `119.127`.
- **Auth:** `auth.config` defines user `admin` (Administrators group) with a stored password hash; the plaintext password is whatever was set on the original Technitium instance.
- The compose project name historically was `dns` (network `dns_default` still exists).
## 6. Firewall / misc
- nftables/iptables: mostly Docker + Tailscale chains (`ts-input`, `ts-forward`,
`DOCKER`, `DOCKER-FORWARD`); default `INPUT ACCEPT`, `FORWARD DROP`,
`OUTPUT ACCEPT`. No UFW / firewalld.
- Also runs (out of scope for this replication): Samba (137/138/139, 445), NFS
(2049), rpcbind (111), Postfix (25), Cockpit (9090), Beszel agent, webmin/
usermin (10000/10002/20000), Tailscale (41641).
- SELinux absent; AppArmor default docker profile.
## 7. How it was audited (no changes made)
```bash
ssh localuser@pfv-netboot 'bash -s' < audit-netboot.sh # broad read-only sweep
ssh localuser@pfv-netboot 'bash -s' < deep-audit-netboot.sh # docker inspect + compose
ssh localuser@pfv-netboot 'bash -s' < gather-configs.sh # pihole.toml + technitium
```
Artifacts: `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt`.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **pfv-netboot reference node setup**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
netinfra/pfv-netinfra-setup.md
+9 -219
View File
@@ -1,220 +1,10 @@
# pfv-netinfra-01 / pfv-netinfra-02 — Network Services Setup
# netinfra/pfv-netinfra-setup.md
These two nodes replicate the network-infrastructure services of **pfv-netboot**
(Pi-hole, Technitium DNS, NTP). They were deployed by `setup-netinfra.sh`, which
reads config from pfv-netboot (read-only) and relays it to each target.
## 1. Nodes
| | pfv-netinfra-01 | pfv-netinfra-02 |
|---|---|---|
| OS | Debian 13 (trixie), kernel 6.12.96+deb13 | Debian 13 (trixie) |
| LAN | `ens18` `192.168.3.252/24` | `ens18` `192.168.3.253/24` |
| Tailscale | `100.70.181.72` | `100.93.194.82` |
| RAM / Disk | 1.9 GiB / 30 GB (27 GB free) | 3.7 GiB / 30 GB (27 GB free) |
| Resolver | Tailscale MagicDNS (`100.100.100.100`) | same |
| Docker | 29.6.2 (pre-installed, enabled) | 29.6.2 |
| Access | `ssh localuser@pfv-netinfra-0X`, passwordless sudo; `localuser` **not** in docker group → use `sudo docker` | same |
## 2. Service layout
All services live under `/home/localuser/services/<svc>/` (owned by `localuser`
so the compose files are directly editable; data dirs keep container uids):
```
/home/localuser/services/
├── pihole/
│ ├── docker-compose.yml
│ └── etc-pihole/ # copied from netboot /root/pihole/etc-pihole
│ ├── pihole.toml # Pi-hole v6 config (upstreams, etc.)
│ ├── gravity.db # adlists / domainlists / clients / groups
│ ├── adlists.list
│ ├── dnsmasq.conf
│ ├── tls.{crt,pem,crt_ca}
│ └── versions
├── ntp/
│ └── docker-compose.yml # chrony container (see §5 — not used; host ntpsec serves)
└── technitium/
├── docker-compose.yml
└── config/ # copied from netboot orphaned volume dns_tsys-dns-config/_data
├── dns.config
├── auth.config
├── scopes/Default.scope
├── self-signed-cert.pfx
└── zones/ # knel.net.zone + 12 Tailscale reverse zones
```
## 3. Pi-hole (container `pihole`)
Image `pihole/pihole:latest`; `restart: always`; `cap_add: [SYS_NICE]`.
| Host port | Container | Purpose |
|---|---|---|
| `53/tcp`, `53/udp` | 53 | DNS (the LAN/Tailscale recursive resolver) |
| `10002/tcp` | 80 | Web admin (HTTP) |
| `10003/tcp` | 443 | Web admin (HTTPS) |
`docker-compose.yml`:
```yaml
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
hostname: pihole
ports:
- "53:53/tcp"
- "53:53/udp"
- "10002:80/tcp"
- "10003:443/tcp"
environment:
TZ: 'America/Chicago'
FTLCONF_webserver_api_password: 'REDACTED_PASSWORD'
FTLCONF_dns_listeningMode: 'all'
volumes:
- './etc-pihole:/etc/pihole'
cap_add:
- SYS_NICE
restart: always
```
- Upstream DNS (from copied `pihole.toml`): `192.168.3.16`, `8.8.8.8`, `2001:4860:4860::8888`.
- Adlist: `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`.
- `pihole.toml` `interface` was adapted from netboot's `eth0` to the target's `ens18`.
- Web admin: `http://<node>:10002/admin/` — password **`REDACTED_PASSWORD`** (same as netboot).
- Web UI URL per node: `http://100.70.181.72:10002/admin/` (-01), `http://100.93.194.82:10002/admin/` (-02).
## 4. Technitium DNS (container `tsys-dns`)
Image `technitium/dns-server`; `restart: always`. Authoritative DNS for
`knel.net` (and Tailscale reverse zones), config copied verbatim from netboot's
orphaned `dns_tsys-dns-config` volume.
| Host port | Container | Purpose |
|---|---|---|
| `5300/tcp`, `5300/udp` | 53 | DNS (remapped — see note) |
| `5380/tcp` | 5380 | Web console (HTTP) |
| `53443/tcp` | 53443 | Web console (HTTPS) |
`docker-compose.yml`:
```yaml
services:
technitium:
image: technitium/dns-server
container_name: tsys-dns
ports:
- "5300:53/tcp"
- "5300:53/udp"
- "5380:5380/tcp"
- "53443:53443/tcp"
volumes:
- './config:/etc/dns'
restart: always
```
- Zones loaded (verified): `knel.net` SOA → `dns.knel.net. hostadmin.knel.net. 2025062313 900 300 604800 900`, plus 12 Tailscale reverse zones.
- Web console: `http://<node>:5380/` → user **`admin`** + the original Technitium
password (carried over via `auth.config`). If the password is unknown, reset it
from the console or by removing `config/auth.config` and recreating the container.
- **Port note:** Technitium's native DNS port (53) is remapped to host **5300**
because Pi-hole already owns host :53 (they cannot both bind 0.0.0.0:53). To
query the authoritative server: `dig -p 5300 @<node> knel.net SOA`. To make
Pi-hole resolve `knel.net` via Technitium, add a conditional/local upstream in
Pi-hole pointing to the container (e.g. `127.0.0.1#5300` is not host-reachable
from Pi-hole's netns — use the docker bridge IP of `tsys-dns`, or add
`knel.net` A-records directly in Pi-hole's Local DNS).
## 5. NTP (host `ntpsec`, not a container)
Both targets **already run a bare-metal `ntpsec` daemon** (active, enabled) that
serves NTP on every local address — including the Tailscale IP — and keeps the
system clock synced. This is the **same daemon family as netboot's own bare-metal
ntpsec**.
- **Why no chrony container?** netboot's chrony container (`tsys-ntp`) binds the
Tailscale IP `100.103.64.82:123`; on netboot that works only because its ntpsec
does **not** pre-bind the specific Tailscale-IP socket. On these targets ntpsec
**does** bind the Tailscale IP, so the container cannot claim it (`address
already in use`) and would be a non-functional duplicate (verified: the
container started but never synced — Stratum 0). It is therefore intentionally
**omitted**; host ntpsec provides NTP. `setup-netinfra.sh` detects an active
host NTP unit and removes any stale `tsys-ntp` container.
- ntpsec config (`/etc/ntpsec/ntp.conf`): Debian NTP pool (`0-3.debian.pool.ntp.org`),
`restrict default kod nomodify noquery limited` (serves time, blocks mgmt queries).
- Verified sync: -01 stratum 2 (~2 ms offset), -02 stratum 3 (~0.2 ms offset),
leap normal.
The `ntp/docker-compose.yml` is still written on each node for parity/reference
(and in case the host NTP is ever disabled — then `sudo docker compose -f
/home/localuser/services/ntp/docker-compose.yml up -d` brings up chrony).
## 6. Verification results (2026-07-28)
| Check | pfv-netinfra-01 | pfv-netinfra-02 |
|---|---|---|
| `pihole` health | healthy | healthy |
| `dig @127.0.0.1:53 pi.hole` | `172.18.0.2` | `172.18.0.2` |
| Pi-hole web `:10002` | HTTP 302 (→login) | HTTP 302 |
| `dig @127.0.0.1:5300 knel.net SOA` | SOA answered | SOA answered |
| Technitium web `:5380` | HTTP 200 | HTTP 200 |
| NTP daemon | ntpsec, stratum 2, synced | ntpsec, stratum 3, synced |
## 7. Operating the services
```bash
# status
sudo docker ps
# Pi-hole
sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml ps
sudo docker compose -f /home/localuser/services/pihole/docker-compose.yml logs -f
sudo docker exec pihole pihole -v # version
sudo docker exec pihole pihole -g # rebuild gravity
sudo docker exec pihole pihole -a -p # set/change web password
# Technitium
sudo docker compose -f /home/localuser/services/technitium/docker-compose.yml logs -f
sudo docker exec tsys-dns sh # explore /etc/dns
# NTP (host)
systemctl status ntpsec
ntpq -pn
```
## 8. Differences from pfv-netboot (intentional)
1. **Layout** under `/home/localuser/services/` instead of `/root` (so `localuser`
can manage compose files); Pi-hole data dir still owned by `localuser`, as on netboot.
2. **Pi-hole `interface`** set to `ens18` (targets' NIC) instead of netboot's `eth0`.
3. **NTP:** host `ntpsec` (Debian pool) used instead of netboot's chrony container
(the container cannot bind the Tailscale IP here; see §5).
4. **Technitium DNS** host port remapped `53 → 5300` to avoid clashing with Pi-hole
on `:53`. The `knel.net` zone and all reverse zones are identical to netboot's.
5. Pi-hole query logs (`pihole-FTL.db*`) and regenerable caches/backups are not
copied (transient); gravity DB and all configuration are.
## 9. Re-running / reproducing
`setup-netinfra.sh` is **idempotent** — it skips re-copying config if already
present and uses `docker compose up -d` (no-ops when unchanged). It reads
pfv-netboot read-only and never mutates it.
```bash
./setup-netinfra.sh # deploy to both nodes
./setup-netinfra.sh pfv-netinfra-01 # deploy one node
./setup-netinfra.sh pfv-netinfra-01 verify # verify only
```
Prerequisites: SSH key access to all three hosts as `localuser` with passwordless
sudo; the targets reach `192.168.3.16`/`8.8.8.8` for Pi-hole upstream and the
internet for image pulls.
## 10. Files in this directory
| File | Purpose |
|---|---|
| `setup-netinfra.sh` | orchestrator: deploys + verifies the clone on -01/-02 |
| `audit-netboot.sh` | broad read-only audit of pfv-netboot |
| `deep-audit-netboot.sh` | docker inspect / compose / volume deep audit (read-only) |
| `gather-configs.sh` | targeted config pull (pihole.toml, technitium) (read-only) |
| `baseline.sh` | read-only baseline of a target node |
| `netboot-audit.txt`, `netboot-deep-audit.txt`, `netboot-configs.txt` | audit output |
| [`pfv-netboot-setup.md`](pfv-netboot-setup.md) | reference-node documentation |
| [`pfv-netinfra-setup.md`](pfv-netinfra-setup.md) | this document |
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **pfv-netinfra-01/02 initial setup**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*
netinfra/pihole/README.md
+9 -60
View File
@@ -1,61 +1,10 @@
# netinfra/pihole/ — Pi-hole recursive DNS (pfv-netinfra-01/02)
# netinfra/pihole/README.md
> **Redmine:** [#376](https://projects.knownelement.com/issues/376) (up/down alerts + commit hardening) · [#357](https://projects.knownelement.com/issues/357) (cluster build, closed)
Pi-hole v6 runs as the recursive resolver on port 53 of both DNS nodes.
Technitium (`tsys-dns`) runs as the authoritative server on port 5300; the two
share the `dnsnet` Docker network so Pi-hole can conditional-forward `knel.net`
zones to Technitium.
## Hardening (defense-in-depth against gravity.db corruption)
The operator hit a production outage when Pi-hole's `/dev/shm` was too small,
corrupting `gravity.db`. The live config on both nodes now includes:
- **`shm_size: 1024M`** — fixes the root cause (default 64M was too small).
- **`gravity-validate.sh`** — entrypoint that checks the SQLite header of
`gravity.db` before start; auto-moves a corrupt/empty DB aside so Pi-hole
can regenerate it cleanly.
- **Healthcheck** — `dig +norecurse @127.0.0.1 pi.hole` + gravity.db non-empty;
fails the container if DNS or the DB is broken.
- **`autoheal`** sidecar — restarts any container labeled `autoheal=true` that
goes unhealthy.
## Deploy
The compose reads the web UI password from a gitignored `.env`:
```bash
cd netinfra/pihole/
cp .env.example .env # then edit .env and set PIHOLE_WEB_PASSWORD
docker compose up -d
```
Files are deployed to `/home/localuser/services/pihole/` on each node. Volumes
(`./etc-pihole`, `./etc-dnsmasq.d`) hold the persistent state.
## IPv6 disabled + Rate-limiting disabled
### IPv6
Both netinfra nodes run **IPv4-only**. IPv6 is disabled at the kernel level
(`/etc/sysctl.d/99-disable-ipv6.conf`) because netinfra-01 has no IPv6 internet
route, and Pi-hole's default IPv6 upstream (Google `2001:4860:4860::8888`) was
causing continuous "Network unreachable" errors + intermittent DNS failures
detected by Uptime Kuma. The upstream is now `8.8.8.8` (IPv4 only).
### Rate-limiting
Pi-hole's default rate-limit (1000 queries / 60 seconds per client) was
**the root cause of Uptime Kuma DNS flapping**. Uptime Kuma runs on the
Cloudron VPS (`tsys-cloudron`, `100.107.35.78`) and sends high-volume DNS
queries to monitor dozens of hosts. When it exceeded 1000 queries/60s,
Pi-hole responded with REFUSED, which Uptime Kuma detected as DNS being
down. Rate-limiting is now disabled (`count=0, interval=0`) since this is a
private tailnet with no risk of DNS amplification attacks.
## Verify
```bash
dig @127.0.0.1 +short google.com # recursive
dig @127.0.0.1 +short git.knownelement.com # knel.net via Technitium forward
docker inspect pihole --format '{{.State.Health.Status}}'
```
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Pi-hole recursive DNS hardening**
>
> **Read it here:** https://community.turnsys.com/t/306
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*