fix(netinfra): commit Pi-hole hardening config previously missing from repo

Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only
modified markdown — the working docker-compose.yml, gravity-validate.sh,
healthcheck, and autoheal config were never written to the repo, leaving
the DNS hardening unreproducible from version control.

This commits the live, verified-working config from the boxes into
netinfra/pihole/:
- docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal)
- gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB)
- .env.example (web UI password templated; real .env gitignored)

Defends against the gravity.db / /dev/shm corruption production outage.
The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret
enters git.

[#376]

.gitignore

@@ -37,3 +37,6 @@ returned-logs/
 # Kubernetes secrets (kubeconfig contains embedded client certs)
 k8s/kubeconfig.yaml
 k8s/*.token
+
+# Pi-hole web UI password (real value only in on-box .env, never committed)
+netinfra/pihole/.env

STATUS.md

@@ -7,7 +7,7 @@
 > checklists or status items here. All work is tracked as Redmine tickets.
 > When work is done, update the ticket (done ratio, notes, close).
 
-> **Last updated:** 2026-08-06 — migrated all tracking to Redmine.
+> **Last updated:** 2026-08-06 — Pi-hole hardening config committed (`netinfra/pihole/`, [#376]).
 
 ## Current State: MIXED (infra stable; k8s needs rebuild)
 
@@ -35,6 +35,7 @@ are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum).
 
 | Ticket | Component | Notes |
 |--------|-----------|-------|
+| [#376](https://projects.knownelement.com/issues/376) | DNS cluster up/down alerts + Pi-hole hardening committed | Config now in `netinfra/pihole/`; flapping source under investigation |
 | [#367](https://projects.knownelement.com/issues/367) | k8s cluster rebuild (cnode wiped, no quorum) | Blocks #368 |
 | [#368](https://projects.knownelement.com/issues/368) | k8s worker nodes join cluster | Blocked by #367 |
 | [#369](https://projects.knownelement.com/issues/369) | Switch config fixes (core-sw01 LAG, LACP, member list) | Onsite Friday |

docs/docmap.md

@@ -3,7 +3,7 @@
 > **Index of all documentation in this repo.** Agents must update this file
 > whenever a doc is added, removed, or substantively changed.
 
-> **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra)
+> **Last updated:** 2026-08-06 (Pi-hole hardening config committed to netinfra/pihole/ — [#376])
 
 ## Kubernetes Architecture ([`k8s/`](k8s/))
 
@@ -61,6 +61,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
 | [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
 | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
 | [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
+| [`../netinfra/pihole/README.md`](../netinfra/pihole/README.md) | Pi-hole recursive DNS on netinfra-01/02: shm_size fix, gravity-validate.sh, healthcheck, autoheal (defense-in-depth vs gravity.db corruption) | 2026-08-06 |
 | [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
 
 ## Archive ([`archive/`](archive/))

netinfra/pihole/.env.example

@@ -0,0 +1,3 @@
+# Pi-hole web UI password. NEVER commit the real .env — only this template.
+# Copy to .env and set the value before `docker compose up -d`.
+PIHOLE_WEB_PASSWORD=changeme

netinfra/pihole/README.md

@@ -0,0 +1,43 @@
+# netinfra/pihole/ — Pi-hole recursive DNS (pfv-netinfra-01/02)
+
+> **Redmine:** [#376](https://projects.knownelement.com/issues/376) (up/down alerts + commit hardening) · [#357](https://projects.knownelement.com/issues/357) (cluster build, closed)
+
+Pi-hole v6 runs as the recursive resolver on port 53 of both DNS nodes.
+Technitium (`tsys-dns`) runs as the authoritative server on port 5300; the two
+share the `dnsnet` Docker network so Pi-hole can conditional-forward `knel.net`
+zones to Technitium.
+
+## Hardening (defense-in-depth against gravity.db corruption)
+
+The operator hit a production outage when Pi-hole's `/dev/shm` was too small,
+corrupting `gravity.db`. The live config on both nodes now includes:
+
+- **`shm_size: 1024M`** — fixes the root cause (default 64M was too small).
+- **`gravity-validate.sh`** — entrypoint that checks the SQLite header of
+  `gravity.db` before start; auto-moves a corrupt/empty DB aside so Pi-hole
+  can regenerate it cleanly.
+- **Healthcheck** — `dig +norecurse @127.0.0.1 pi.hole` + gravity.db non-empty;
+  fails the container if DNS or the DB is broken.
+- **`autoheal`** sidecar — restarts any container labeled `autoheal=true` that
+  goes unhealthy.
+
+## Deploy
+
+The compose reads the web UI password from a gitignored `.env`:
+
+```bash
+cd netinfra/pihole/
+cp .env.example .env          # then edit .env and set PIHOLE_WEB_PASSWORD
+docker compose up -d
+```
+
+Files are deployed to `/home/localuser/services/pihole/` on each node. Volumes
+(`./etc-pihole`, `./etc-dnsmasq.d`) hold the persistent state.
+
+## Verify
+
+```bash
+dig @127.0.0.1 +short google.com        # recursive
+dig @127.0.0.1 +short git.knownelement.com   # knel.net via Technitium forward
+docker inspect pihole --format '{{.State.Health.Status}}'
+```

netinfra/pihole/docker-compose.yml

@@ -0,0 +1,49 @@
+services:
+  pihole:
+    container_name: pihole
+    # Root cause of the 2026-08 gravity.db corruption: default /dev/shm (64M)
+    # was too small for FTL's shared-memory metrics. 1024M has been stable.
+    shm_size: '1024M'
+    image: pihole/pihole:latest
+    hostname: pihole
+    entrypoint: ["/usr/local/bin/gravity-validate.sh"]
+    ports:
+      - "53:53/tcp"
+      - "53:53/udp"
+      - "10002:80/tcp"
+      - "10003:443/tcp"
+    environment:
+      TZ: 'America/Chicago'
+      FTLCONF_webserver_api_password: '${PIHOLE_WEB_PASSWORD}'
+      FTLCONF_dns_listeningMode: 'all'
+    volumes:
+      - './etc-pihole:/etc/pihole'
+      - './etc-dnsmasq.d:/etc/dnsmasq.d'
+      - './gravity-validate.sh:/usr/local/bin/gravity-validate.sh:ro'
+    cap_add:
+      - SYS_NICE
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "dig +short +norecurse @127.0.0.1 pi.hole >/dev/null 2>&1 && test -s /etc/pihole/gravity.db || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+    labels:
+      autoheal: "true"
+    networks:
+      - default
+      - dnsnet
+  autoheal:
+    container_name: autoheal
+    image: willfarrell/autoheal:latest
+    environment:
+      AUTOHEAL_CONTAINER_LABEL: autoheal
+      AUTOHEAL_INTERVAL: 30
+      AUTOHEAL_START_PERIOD: 60
+    volumes:
+      - '/var/run/docker.sock:/var/run/docker.sock:ro'
+    restart: always
+networks:
+  dnsnet:
+    external: true

netinfra/pihole/gravity-validate.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# gravity-validate.sh — pre-start integrity check for Pi-hole's gravity.db
+#
+# Runs as the container entrypoint. If gravity.db is empty or has an invalid
+# SQLite header (the symptom of the /dev/shm corruption outage), move it aside
+# so Pi-hole regenerates a clean DB on start instead of crashing.
+set -e
+GRAVITY_DB="/etc/pihole/gravity.db"
+TIMESTAMP=$(date +%Y%m%d%H%M%S)
+
+if [ -f "$GRAVITY_DB" ]; then
+    if [ ! -s "$GRAVITY_DB" ]; then
+        echo "[gravity-validate] gravity.db is empty, moving aside"
+        mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+    else
+        HEADER=$(head -c 15 "$GRAVITY_DB" 2>/dev/null || true)
+        if [ "$HEADER" != "SQLite format 3" ]; then
+            echo "[gravity-validate] gravity.db invalid header, moving aside"
+            mv "$GRAVITY_DB" "${GRAVITY_DB}.corrupt.${TIMESTAMP}"
+        fi
+    fi
+fi
+# Keep only the 3 most recent corrupt backups (names carry a timestamp,
+# so lexical reverse-sort = newest-first).
+find /etc/pihole -maxdepth 1 -name 'gravity.db.corrupt.*' -print 2>/dev/null \
+  | sort -r | tail -n +4 | xargs -r rm -f
+echo "[gravity-validate] OK, starting Pi-hole"
+exec /usr/bin/start.sh "$@"