fix(netinfra): commit Pi-hole hardening config previously missing from repo

Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only modified markdown — the working docker-compose.yml, gravity-validate.sh, healthcheck, and autoheal config were never written to the repo, leaving the DNS hardening unreproducible from version control. This commits the live, verified-working config from the boxes into netinfra/pihole/: - docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal) - gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB) - .env.example (web UI password templated; real .env gitignored) Defends against the gravity.db / /dev/shm corruption production outage. The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret enters git. [#376]
2026-08-06 12:37:46 -05:00
parent 6c58671d15
commit 30ddb37b34
7 changed files with 130 additions and 2 deletions
@@ -0,0 +1,3 @@
+# Pi-hole web UI password. NEVER commit the real .env — only this template.
+# Copy to .env and set the value before `docker compose up -d`.
+PIHOLE_WEB_PASSWORD=changeme
@@ -0,0 +1,43 @@
+# netinfra/pihole/ — Pi-hole recursive DNS (pfv-netinfra-01/02)
+
+> **Redmine:** [#376](https://projects.knownelement.com/issues/376) (up/down alerts + commit hardening) · [#357](https://projects.knownelement.com/issues/357) (cluster build, closed)
+
+Pi-hole v6 runs as the recursive resolver on port 53 of both DNS nodes.
+Technitium (`tsys-dns`) runs as the authoritative server on port 5300; the two
+share the `dnsnet` Docker network so Pi-hole can conditional-forward `knel.net`
+zones to Technitium.
+
+## Hardening (defense-in-depth against gravity.db corruption)
+
+The operator hit a production outage when Pi-hole's `/dev/shm` was too small,
+corrupting `gravity.db`. The live config on both nodes now includes:
+
+- **`shm_size: 1024M`** — fixes the root cause (default 64M was too small).
+- **`gravity-validate.sh`** — entrypoint that checks the SQLite header of
+  `gravity.db` before start; auto-moves a corrupt/empty DB aside so Pi-hole
+  can regenerate it cleanly.
+- **Healthcheck** — `dig +norecurse @127.0.0.1 pi.hole` + gravity.db non-empty;
+  fails the container if DNS or the DB is broken.
+- **`autoheal`** sidecar — restarts any container labeled `autoheal=true` that
+  goes unhealthy.
+
+## Deploy
+
+The compose reads the web UI password from a gitignored `.env`:
+
+```bash
+cd netinfra/pihole/
+cp .env.example .env          # then edit .env and set PIHOLE_WEB_PASSWORD
+docker compose up -d
+```
+
+Files are deployed to `/home/localuser/services/pihole/` on each node. Volumes
+(`./etc-pihole`, `./etc-dnsmasq.d`) hold the persistent state.
+
+## Verify
+
+```bash
+dig @127.0.0.1 +short google.com        # recursive
+dig @127.0.0.1 +short git.knownelement.com   # knel.net via Technitium forward
+docker inspect pihole --format '{{.State.Health.Status}}'
+```
@@ -0,0 +1,49 @@
+services:
+  pihole:
+    container_name: pihole
+    # Root cause of the 2026-08 gravity.db corruption: default /dev/shm (64M)
+    # was too small for FTL's shared-memory metrics. 1024M has been stable.
+    shm_size: '1024M'
+    image: pihole/pihole:latest
+    hostname: pihole
+    entrypoint: ["/usr/local/bin/gravity-validate.sh"]
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: '${PIHOLE_WEB_PASSWORD}'
+      FTLCONF_dns_listeningMode: 'all'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+      - './etc-dnsmasq.d:/etc/dnsmasq.d'
+      - './gravity-validate.sh:/usr/local/bin/gravity-validate.sh:ro'
+    cap_add:
+      - SYS_NICE
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "dig +short +norecurse @127.0.0.1 pi.hole >/dev/null 2>&1 && test -s /etc/pihole/gravity.db || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+    labels:
+      autoheal: "true"
+    networks:
+      - default
+      - dnsnet
+  autoheal:
+    container_name: autoheal
+    image: willfarrell/autoheal:latest
+    environment:
+      AUTOHEAL_CONTAINER_LABEL: autoheal
+      AUTOHEAL_INTERVAL: 30
+      AUTOHEAL_START_PERIOD: 60
+    volumes:
+      - '/var/run/docker.sock:/var/run/docker.sock:ro'
+    restart: always
+networks:
+  dnsnet:
+    external: true
@@ -0,0 +1,28 @@
+#!/bin/bash
+# gravity-validate.sh — pre-start integrity check for Pi-hole's gravity.db
+#
+# Runs as the container entrypoint. If gravity.db is empty or has an invalid
+# SQLite header (the symptom of the /dev/shm corruption outage), move it aside
+# so Pi-hole regenerates a clean DB on start instead of crashing.
+set -e
+GRAVITY_DB="/etc/pihole/gravity.db"
+TIMESTAMP=$(date +%Y%m%d%H%M%S)
+
+if [ -f "$GRAVITY_DB" ]; then
+    if [ ! -s "$GRAVITY_DB" ]; then
+        echo "[gravity-validate] gravity.db is empty, moving aside"
+        mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+    else
+        HEADER=$(head -c 15 "$GRAVITY_DB" 2>/dev/null || true)
+        if [ "$HEADER" != "SQLite format 3" ]; then
+            echo "[gravity-validate] gravity.db invalid header, moving aside"
+            mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+        fi
+    fi
+fi
+# Keep only the 3 most recent corrupt backups (names carry a timestamp,
+# so lexical reverse-sort = newest-first).
+find /etc/pihole -maxdepth 1 -name 'gravity.db.corrupt.*' -print 2>/dev/null \
+  | sort -r | tail -n +4 | xargs -r rm -f
+echo "[gravity-validate] OK, starting Pi-hole"
+exec /usr/bin/start.sh "$@"