diff --git a/.gitignore b/.gitignore index 2705d3f..a2b3250 100644 --- a/.gitignore +++ b/.gitignore @@ -37,3 +37,6 @@ returned-logs/ # Kubernetes secrets (kubeconfig contains embedded client certs) k8s/kubeconfig.yaml k8s/*.token + +# Pi-hole web UI password (real value only in on-box .env, never committed) +netinfra/pihole/.env diff --git a/STATUS.md b/STATUS.md index 24d9d60..5d5d9df 100644 --- a/STATUS.md +++ b/STATUS.md @@ -7,7 +7,7 @@ > checklists or status items here. All work is tracked as Redmine tickets. > When work is done, update the ticket (done ratio, notes, close). -> **Last updated:** 2026-08-06 — migrated all tracking to Redmine. +> **Last updated:** 2026-08-06 — Pi-hole hardening config committed (`netinfra/pihole/`, [#376]). ## Current State: MIXED (infra stable; k8s needs rebuild) @@ -35,6 +35,7 @@ are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum). | Ticket | Component | Notes | |--------|-----------|-------| +| [#376](https://projects.knownelement.com/issues/376) | DNS cluster up/down alerts + Pi-hole hardening committed | Config now in `netinfra/pihole/`; flapping source under investigation | | [#367](https://projects.knownelement.com/issues/367) | k8s cluster rebuild (cnode wiped, no quorum) | Blocks #368 | | [#368](https://projects.knownelement.com/issues/368) | k8s worker nodes join cluster | Blocked by #367 | | [#369](https://projects.knownelement.com/issues/369) | Switch config fixes (core-sw01 LAG, LACP, member list) | Onsite Friday | diff --git a/docs/docmap.md b/docs/docmap.md index cf9f21b..d68ba1a 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -3,7 +3,7 @@ > **Index of all documentation in this repo.** Agents must update this file > whenever a doc is added, removed, or substantively changed. -> **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra) +> **Last updated:** 2026-08-06 (Pi-hole hardening config committed to netinfra/pihole/ — [#376]) ## Kubernetes Architecture ([`k8s/`](k8s/)) @@ -61,6 +61,7 @@ Server provisioning, security hardening, DNS/NTP configuration. | [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | | [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 | +| [`../netinfra/pihole/README.md`](../netinfra/pihole/README.md) | Pi-hole recursive DNS on netinfra-01/02: shm_size fix, gravity-validate.sh, healthcheck, autoheal (defense-in-depth vs gravity.db corruption) | 2026-08-06 | | [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 | ## Archive ([`archive/`](archive/)) diff --git a/netinfra/pihole/.env.example b/netinfra/pihole/.env.example new file mode 100644 index 0000000..0f77783 --- /dev/null +++ b/netinfra/pihole/.env.example @@ -0,0 +1,3 @@ +# Pi-hole web UI password. NEVER commit the real .env — only this template. +# Copy to .env and set the value before `docker compose up -d`. +PIHOLE_WEB_PASSWORD=changeme diff --git a/netinfra/pihole/README.md b/netinfra/pihole/README.md new file mode 100644 index 0000000..78b7b44 --- /dev/null +++ b/netinfra/pihole/README.md @@ -0,0 +1,43 @@ +# netinfra/pihole/ — Pi-hole recursive DNS (pfv-netinfra-01/02) + +> **Redmine:** [#376](https://projects.knownelement.com/issues/376) (up/down alerts + commit hardening) · [#357](https://projects.knownelement.com/issues/357) (cluster build, closed) + +Pi-hole v6 runs as the recursive resolver on port 53 of both DNS nodes. +Technitium (`tsys-dns`) runs as the authoritative server on port 5300; the two +share the `dnsnet` Docker network so Pi-hole can conditional-forward `knel.net` +zones to Technitium. + +## Hardening (defense-in-depth against gravity.db corruption) + +The operator hit a production outage when Pi-hole's `/dev/shm` was too small, +corrupting `gravity.db`. The live config on both nodes now includes: + +- **`shm_size: 1024M`** — fixes the root cause (default 64M was too small). +- **`gravity-validate.sh`** — entrypoint that checks the SQLite header of + `gravity.db` before start; auto-moves a corrupt/empty DB aside so Pi-hole + can regenerate it cleanly. +- **Healthcheck** — `dig +norecurse @127.0.0.1 pi.hole` + gravity.db non-empty; + fails the container if DNS or the DB is broken. +- **`autoheal`** sidecar — restarts any container labeled `autoheal=true` that + goes unhealthy. + +## Deploy + +The compose reads the web UI password from a gitignored `.env`: + +```bash +cd netinfra/pihole/ +cp .env.example .env # then edit .env and set PIHOLE_WEB_PASSWORD +docker compose up -d +``` + +Files are deployed to `/home/localuser/services/pihole/` on each node. Volumes +(`./etc-pihole`, `./etc-dnsmasq.d`) hold the persistent state. + +## Verify + +```bash +dig @127.0.0.1 +short google.com # recursive +dig @127.0.0.1 +short git.knownelement.com # knel.net via Technitium forward +docker inspect pihole --format '{{.State.Health.Status}}' +``` diff --git a/netinfra/pihole/docker-compose.yml b/netinfra/pihole/docker-compose.yml new file mode 100644 index 0000000..060860c --- /dev/null +++ b/netinfra/pihole/docker-compose.yml @@ -0,0 +1,49 @@ +services: + pihole: + container_name: pihole + # Root cause of the 2026-08 gravity.db corruption: default /dev/shm (64M) + # was too small for FTL's shared-memory metrics. 1024M has been stable. + shm_size: '1024M' + image: pihole/pihole:latest + hostname: pihole + entrypoint: ["/usr/local/bin/gravity-validate.sh"] + ports: + - "53:53/tcp" + - "53:53/udp" + - "10002:80/tcp" + - "10003:443/tcp" + environment: + TZ: 'America/Chicago' + FTLCONF_webserver_api_password: '${PIHOLE_WEB_PASSWORD}' + FTLCONF_dns_listeningMode: 'all' + volumes: + - './etc-pihole:/etc/pihole' + - './etc-dnsmasq.d:/etc/dnsmasq.d' + - './gravity-validate.sh:/usr/local/bin/gravity-validate.sh:ro' + cap_add: + - SYS_NICE + restart: always + healthcheck: + test: ["CMD-SHELL", "dig +short +norecurse @127.0.0.1 pi.hole >/dev/null 2>&1 && test -s /etc/pihole/gravity.db || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + labels: + autoheal: "true" + networks: + - default + - dnsnet + autoheal: + container_name: autoheal + image: willfarrell/autoheal:latest + environment: + AUTOHEAL_CONTAINER_LABEL: autoheal + AUTOHEAL_INTERVAL: 30 + AUTOHEAL_START_PERIOD: 60 + volumes: + - '/var/run/docker.sock:/var/run/docker.sock:ro' + restart: always +networks: + dnsnet: + external: true diff --git a/netinfra/pihole/gravity-validate.sh b/netinfra/pihole/gravity-validate.sh new file mode 100755 index 0000000..b71ab1e --- /dev/null +++ b/netinfra/pihole/gravity-validate.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# gravity-validate.sh — pre-start integrity check for Pi-hole's gravity.db +# +# Runs as the container entrypoint. If gravity.db is empty or has an invalid +# SQLite header (the symptom of the /dev/shm corruption outage), move it aside +# so Pi-hole regenerates a clean DB on start instead of crashing. +set -e +GRAVITY_DB="/etc/pihole/gravity.db" +TIMESTAMP=$(date +%Y%m%d%H%M%S) + +if [ -f "$GRAVITY_DB" ]; then + if [ ! -s "$GRAVITY_DB" ]; then + echo "[gravity-validate] gravity.db is empty, moving aside" + mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}" + else + HEADER=$(head -c 15 "$GRAVITY_DB" 2>/dev/null || true) + if [ "$HEADER" != "SQLite format 3" ]; then + echo "[gravity-validate] gravity.db invalid header, moving aside" + mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}" + fi + fi +fi +# Keep only the 3 most recent corrupt backups (names carry a timestamp, +# so lexical reverse-sort = newest-first). +find /etc/pihole -maxdepth 1 -name 'gravity.db.corrupt.*' -print 2>/dev/null \ + | sort -r | tail -n +4 | xargs -r rm -f +echo "[gravity-validate] OK, starting Pi-hole" +exec /usr/bin/start.sh "$@"