feat(nuisance): edge nuisance-block sweep + founder-directed blocks [#796]

Cloudron-native enforcement (ipset cloudron_blocklist via panel setting). https://projects.knownelement.com/issues/796
2026-09-04 12:10:46 -05:00
parent 3b218e7659
commit 9959f86295
7 changed files with 329 additions and 0 deletions
@@ -0,0 +1,53 @@
+# perf/nuisance — edge nuisance-block (my.knownelement.com)
+
+Runbook for the founder-directed hostile-source block (#796, #787 Option B).
+Ops knowledge lives here (runs beside code); the durable audit trail is
+Redmine #796 + Discourse t/298.
+
+## How it works
+
+- Enforcement is **Cloudron-native**: the blocklist setting
+  (`firewall_blocklist`) drives ipset `cloudron_blocklist`, referenced by the
+  INPUT-side `CLOUDRON` chain and FORWARD-side `DOCKER-USER`. Cloudron
+  re-applies it at boot — no custom firewall persistence to maintain.
+- Detection is **dual**:
+  - `nuisance-sweep.sh` (workstation, daily 09:00 automation): scans Haraka
+    spamhaus-DENY sources, adds new ones (>=3 denials/24h, cap 64/run) via
+    the blocklist API, prints every change for the daily report.
+  - `nuisance-detect.sh` + `nuisance-detect.timer` (host, 07:15 daily, no
+    secrets): writes ranked candidates to
+    `/home/yellowtent/platformdata/nuisance/candidates.txt` — cross-check
+    that detection runs even if the LLM automation misses a day.
+
+## View current blocks
+
+- Panel: **my.knownelement.com → Network → IP block list**
+- API: `GET https://my.knownelement.com/api/v1/network/blocklist`
+- Kernel: `ipset list cloudron_blocklist` (members = what is actually dropped)
+
+## Unblock something
+
+1. Add the IP/CIDR to `allowlist.txt` (repo) — the sweep will never re-add it.
+2. Remove it from the live setting: GET the blocklist, delete the line, POST
+   the remainder (POST /api/v1/network/blocklist, body
+   `{"blocklist": "<full text>"}`). Takes effect immediately.
+3. Update `blocklist.txt` (repo) and note why in Redmine #796.
+4. Verify: `ipset test cloudron_blocklist <ip>` must FAIL after removal.
+
+## Files
+
+| file | role |
+|------|------|
+| `nuisance-sweep.sh` | daily auto-block sweep (workstation, automation-driven) |
+| `nuisance-detect.sh` | host-side candidate scan (no secrets) |
+| `nuisance-detect.{service,timer}` | systemd arming for the detector |
+| `blocklist.txt` | git SoR mirror of the live blocklist |
+| `allowlist.txt` | never-block overrides (always wins) |
+
+## Known limits (v1)
+
+- IPv4 only. IPv6 sources are logged by Haraka but not blocked.
+- The sweep only ever blocks IPs our own mail server already DENIED on
+  Spamhaus evidence — a listed-but-legit sender is unaffected in practice
+  (it could not deliver anyway). If a shared IP needs unblocking, follow the
+  procedure above.
@@ -0,0 +1,6 @@
+# nuisance allowlist — entries here are NEVER auto-blocked by the sweep [#796]
+# One IP or CIDR per line; '#' comments allowed. Put any IP that must never
+# lose connectivity (offices, monitoring probes, key partners) here.
+# The sweep reads this file every run; changes take effect next run (or at
+# the 09:00 automation pass).
+# (empty — add entries as needed)
@@ -0,0 +1,67 @@
+# Cloudron blocklist SoR mirror — my.knownelement.com
+# LIVE SOURCE OF TRUTH: Cloudron setting firewall_blocklist
+#   GET/POST https://my.knownelement.com/api/v1/network/blocklist (admin token)
+#   panel: Network > IP block list; enforced via ipset cloudron_blocklist on
+#   INPUT(CLOUDRON) + FORWARD(DOCKER-USER); Cloudron re-applies at boot.
+# This file mirrors the live setting (last sync 2026-09-04 12:15 CDT, #796).
+# Full-line comments allowed in both.
+
+# founder-directed 2026-09-04 (#796, #787 root cause): hostile SMTP ranges
+92.118.39.0/24
+80.94.92.0/24
+
+# sweep-added 2026-09-04 (#796 auto: spamhaus-DENY sources, hits>=MIN_HITS)
+213.176.24.83
+193.32.162.9
+193.32.162.71
+158.94.208.191
+195.224.203.50
+161.35.78.82
+3.129.187.38
+18.116.101.220
+34.14.19.55
+45.148.10.25
+45.148.10.26
+45.148.10.29
+45.148.10.30
+45.148.10.31
+45.148.10.34
+45.148.10.35
+45.148.10.36
+45.148.10.37
+45.148.10.39
+45.91.64.7
+142.93.104.135
+203.188.168.251
+35.236.139.55
+58.211.39.86
+102.134.107.34
+104.154.249.213
+111.26.106.117
+111.26.177.216
+111.26.62.42
+111.26.63.85
+111.42.175.101
+111.46.77.2
+116.98.104.138
+131.100.49.158
+155.4.31.215
+164.164.198.32
+168.235.210.157
+177.27.76.210
+179.184.218.49
+180.165.11.113
+186.238.89.142
+187.8.64.94
+197.219.208.58
+218.219.193.151
+218.25.233.22
+221.203.187.222
+34.182.9.35
+34.86.40.181
+35.198.252.246
+36.135.107.57
+45.164.115.86
+46.44.0.185
+61.37.150.6
+
@@ -0,0 +1,7 @@
+[Unit]
+Description=Nuisance candidate detection (Haraka spamhaus-DENY scan) [#796]
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/bash /home/yellowtent/platformdata/nuisance/nuisance-detect.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# perf/nuisance/nuisance-detect.sh — HOST-side nuisance candidate detection [#796]
+#
+# Runs on the Cloudron host (systemd timer, 07:15 CDT daily + at boot).
+# No secrets, no API: scans the mail container's Haraka log for sources
+# DENIED via zen.spamhaus.org in the last 24h and writes a ranked
+# candidates file the daily automation cross-checks. Detection is
+# independent of the LLM automation; enforcement stays in
+# nuisance-sweep.sh (Cloudron blocklist API).
+#
+# Output: /home/yellowtent/platformdata/nuisance/candidates.txt
+#   lines: <hits24h> <ip> <BLOCKED|seen>   (BLOCKED = already in ipset)
+
+set -u
+
+OUT_DIR=/home/yellowtent/platformdata/nuisance
+OUT="$OUT_DIR/candidates.txt"
+SET_NAME=cloudron_blocklist
+
+mkdir -p "$OUT_DIR"
+TMP=$(mktemp) || exit 1
+trap 'rm -f "$TMP"' EXIT
+
+docker logs mail --since 24h 2>&1 \
+    | grep -a "plugin=dns-list" \
+    | grep -a "retval=DENY" \
+    | grep -aoE "host \[[0-9.]+\]" \
+    | sed "s/^host \[//; s/\]$//" | sort | uniq -c | sort -rn > "$TMP"
+
+{
+    printf '# nuisance candidates — Haraka spamhaus-DENY sources, 24h, generated %s\n' \
+        "$(date '+%F %T %Z')"
+    printf '# format: <hits24h> <ip> <BLOCKED|seen>  (BLOCKED = covered by %s)\n' "$SET_NAME"
+    while read -r hits ip; do
+        [ -n "$ip" ] || continue
+        if ipset test "$SET_NAME" "$ip" > /dev/null 2>&1; then
+            state=BLOCKED
+        else
+            state=seen
+        fi
+        printf '%s %s %s\n' "$hits" "$ip" "$state"
+    done < "$TMP"
+} > "$OUT"
@@ -0,0 +1,11 @@
+[Unit]
+Description=Daily nuisance candidate detection (07:15 local) [#796]
+
+[Timer]
+OnCalendar=*-*-* 07:15:00
+RandomizedDelaySec=300
+Persistent=true
+Unit=nuisance-detect.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+# perf/nuisance/nuisance-sweep.sh — daily hostile-source auto-block sweep [#796]
+#
+# Founder-directed 2026-09-04 (#787 Option B): scan the mail container's
+# Haraka log for sources DENIED at connect via zen.spamhaus.org; any source
+# with >=MIN_HITS denials in 24h is added to the Cloudron-native blocklist
+# (panel: Network > IP block list; setting firewall_blocklist; enforced via
+# ipset cloudron_blocklist on INPUT(CLOUDRON) + FORWARD(DOCKER-USER) —
+# re-applied by Cloudron at boot).
+#
+# Guardrails:
+#   - allowlist.txt entries are NEVER added (and reported if conflicts arise)
+#   - RFC1918 / loopback / link-local / Tailscale 100.64/10 / reserved nets
+#     are never blockable
+#   - already-covered sources are skipped (CIDR containment check)
+#   - hard cap MAX_NEW additions per run; every change printed for the report
+#
+# Only Spamhaus-corroborated sources are ever blocked: an IP must have been
+# DENIED by our own mail server on Spamhaus evidence — legitimate senders
+# are untouched in practice.
+#
+# Run (workstation; needs tests/remote.sh chokepoint + ~/.creds/cloudron.env):
+#   cd ~/projects/PFVCluster && timeout 300 env VM_IP=my.knownelement.com VM_USER=root \
+#     bash ~/projects/cloudron/perf/nuisance/nuisance-sweep.sh [--dry]
+
+set -u
+
+PFV_DIR=${PFV_CLUSTER_DIR:-/home/reachableceo/projects/PFVCluster}
+NUIS_DIR=${NUISANCE_DIR:-/home/reachableceo/projects/cloudron/perf/nuisance}
+ALLOWLIST="$NUIS_DIR/allowlist.txt"
+HOST_NAME=my.knownelement.com
+MIN_HITS=3
+MAX_NEW=64
+DRY=0
+[ "${1:-}" = "--dry" ] && DRY=1
+
+TMP=$(mktemp -d) || exit 1
+trap 'rm -rf "$TMP"' EXIT
+
+fail() { echo "SWEEP ERROR: $1"; exit 1; }
+
+# --- 1. scan Haraka spamhaus-DENY sources through the SSH chokepoint ---
+cd "$PFV_DIR" || fail "PFVCluster checkout not found at $PFV_DIR"
+timeout 120 env VM_IP="$HOST_NAME" VM_USER=root bash tests/remote.sh vm \
+    'docker logs mail --since 24h 2>&1 | grep -a "plugin=dns-list" | grep -a "retval=DENY" | grep -aoE "host \[[0-9.]+\]" | sed "s/^host \[//; s/\]$//" | sort | uniq -c' \
+    > "$TMP/scan" || fail "mail log scan failed"
+deny_lines=$(wc -l < "$TMP/scan")
+
+# --- 2. current blocklist from the Cloudron API ---
+[ -f "$HOME/.creds/cloudron.env" ] || fail "$HOME/.creds/cloudron.env missing"
+set -a
+# shellcheck disable=SC1091  # creds file path built above
+. "$HOME/.creds/cloudron.env"
+set +a
+[ -n "${CLOUDRON_URL:-}" ] && [ -n "${CLOUDRON_API_TOKEN:-}" ] || fail "creds file lacks CLOUDRON_URL/CLOUDRON_API_TOKEN"
+curl -sf --max-time 15 "$CLOUDRON_URL/api/v1/network/blocklist" \
+    -H "Authorization: Bearer $CLOUDRON_API_TOKEN" \
+    | jq -r .blocklist > "$TMP/current" || fail "blocklist GET failed"
+current_entries=$(grep -cvE '^[[:space:]]*(#|$)' "$TMP/current")
+
+# --- 3. candidates: hits>=MIN_HITS, IPv4, not allowlisted, not never-net, not already covered ---
+awk -v min="$MIN_HITS" -v allowf="$ALLOWLIST" -v curf="$TMP/current" '
+    function ip2int(ip, a) {
+        if (split(ip, a, ".") != 4) return -1
+        return a[1]*16777216 + a[2]*65536 + a[3]*256 + a[4]
+    }
+    function innet(ip, cidr,  n, m, base, mask, size, i, v) {
+        v = ip2int(ip); if (v < 0) return 0
+        n = split(cidr, m, "/")
+        base = ip2int(m[1]); if (base < 0) return 0
+        mask = (n == 2) ? m[2] + 0 : 32
+        size = 4294967296
+        for (i = 0; i < mask; i++) size /= 2
+        return v >= base && v < base + size
+    }
+    BEGIN {
+        nn = split("0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16 172.16.0.0/12 192.168.0.0/16 224.0.0.0/4 240.0.0.0/4", NN, " ")
+    }
+    FILENAME == allowf { if ($1 !~ /^#/) allow[$1] = 1; next }
+    FILENAME == curf   { if ($0 !~ /^[[:space:]]*(#|$)/) cur[++nc] = $1; next }
+    {
+        hits = $1 + 0; ip = $2
+        if (ip2int(ip) < 0) next
+        if (hits < min) next
+        if (ip in allow) { print "SKIP-ALLOW", ip, hits > "/dev/stderr"; next }
+        bad = 0
+        for (i = 1; i <= nn; i++) if (innet(ip, NN[i])) { bad = 1; break }
+        if (bad) { print "SKIP-RESERVED", ip, hits > "/dev/stderr"; next }
+        for (i = 1; i <= nc; i++) if (innet(ip, cur[i])) { bad = 1; break }
+        if (bad) next
+        print ip, hits
+    }
+' "$ALLOWLIST" "$TMP/current" "$TMP/scan" 2>"$TMP/skips" \
+    | sort -k2,2rn > "$TMP/new"
+new_count=$(wc -l < "$TMP/new")
+
+echo "== nuisance sweep summary =="
+echo "haraka spamhaus-DENY sources 24h: $deny_lines distinct"
+echo "current blocklist entries: $current_entries"
+echo "new block candidates (hits>=$MIN_HITS, uncovered): $new_count"
+
+# --- 4. apply (cap MAX_NEW) ---
+if [ "$new_count" -eq 0 ]; then
+    echo "SWEEP RESULT: no new blocks needed"
+    exit 0
+fi
+if [ "$new_count" -gt "$MAX_NEW" ]; then
+    echo "SWEEP ABORT: $new_count candidates exceed per-run cap $MAX_NEW — pathological day, human review required" >&2
+    exit 2
+fi
+
+stamp=$(date +%F)
+{
+    echo "# sweep-added $stamp (#796 auto: spamhaus-DENY sources, hits>=MIN_HITS)"
+    awk '{ print $1 }' "$TMP/new"
+} > "$TMP/addition"
+
+if [ "$DRY" -eq 1 ]; then
+    echo "DRY RUN — would append these entries:"
+    cat "$TMP/addition"
+    exit 0
+fi
+
+{ cat "$TMP/current"; echo; cat "$TMP/addition"; } > "$TMP/merged"
+payload=$(jq -Rs '{ blocklist: . }' "$TMP/merged")
+resp=$(printf '%s' "$payload" | curl -sf --max-time 30 -X POST \
+    "$CLOUDRON_URL/api/v1/network/blocklist" \
+    -H "Authorization: Bearer $CLOUDRON_API_TOKEN" \
+    -H "Content-Type: application/json" -d @- -w '\n%{http_code}') || fail "blocklist POST failed"
+echo "$resp" | tail -1 | grep -q 200 || fail "blocklist POST not 200: $resp"
+
+# --- 5. verify ---
+curl -sf --max-time 15 "$CLOUDRON_URL/api/v1/network/blocklist" \
+    -H "Authorization: Bearer $CLOUDRON_API_TOKEN" \
+    | jq -r .blocklist > "$TMP/after" || fail "verify GET failed"
+verified=$(grep -cvE '^[[:space:]]*(#|$)' "$TMP/after")
+expected=$((current_entries + new_count))
+[ "$verified" -eq "$expected" ] || fail "verify mismatch: expected $expected entries, got $verified"
+
+echo "SWEEP RESULT: BLOCKED $new_count new source(s):"
+cat "$TMP/new"
+echo "blocklist now $verified entries"