feat(mail): fleet mail-env audit + map; PMG VIP proposal [#696][#694]

Audit findings note (55/64 systems, read-only):
https://projects.knownelement.com/issues/696#note-4088

Map + deploy plan: https://community.turnsys.com/t/322
Key: relayhost empty fleet-wide (direct-to-MX today); prod PMG pair
already installed (VM 604/711, PMG 9.1, unclustered); VIP slot
proposed 192.168.3.249; open PQs in questions-v8.md.

netinfra/mail/README.md

@@ -0,0 +1,21 @@
+# netinfra/mail — PMG / fleet mail tooling
+
+Pointers + usage. Knowledge doc (map, findings, deploy plan):
+[Discourse t/322 — Mail environment map + PMG deploy plan](https://community.turnsys.com/t/322).
+Ticket: [Redmine #696](https://projects.knownelement.com/issues/696).
+
+## Files
+
+- `audit-mail-env.sh` — read-only mail probe, runs ON a target
+  (postfix config, alt relays, identity, :25 listeners, relay log
+  evidence). Delivered via the ssh chokepoints only
+  (`remote.sh prox-file|vm-file`, `remote-dns.sh <host>-file`).
+- `run-mail-audit.sh` — drives the probe across
+  `audit-targets.txt`; output per host under `.crush/audit/mail/`.
+- `audit-targets.txt` — fleet target list (`mode|name|user`).
+
+## Re-run the audit
+
+```bash
+bash netinfra/mail/run-mail-audit.sh
+```

netinfra/mail/audit-mail-env.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# audit-mail-env.sh — read-only mail environment probe.
+#
+# Runs ON the target host, delivered via the ssh chokepoints:
+#   tests/remote.sh prox-file|vm-file   OR   remote-dns.sh <host>-file
+# Collects: installed MTAs, postfix relay config, alternative relay configs
+# (ssmtp/msmtp/nullmailer/dma), identity (mailname/aliases/forward),
+# smtp listeners, and recent relay evidence from mail logs.
+# Makes NO changes. Redacts auth material. Output is "KEY value" text.
+set -u
+
+as_root() {
+  if [ "$(id -u)" -eq 0 ]; then
+    "$@"
+  else
+    if sudo -n "$@" 2>/dev/null; then
+      :
+    else
+      "$@"
+    fi
+  fi
+}
+
+echo "== host $(hostname -f 2>/dev/null || hostname) | $(date '+%F %T %Z')"
+
+echo "== mtas"
+for b in postconf postfix sendmail exim4 msmtp ssmtp nullmailer-send dma; do
+  p="$(command -v "$b" 2>/dev/null)"
+  if [ -n "${p}" ]; then
+    echo "bin ${b}: ${p}"
+  fi
+done
+
+echo "== postfix"
+if command -v postconf >/dev/null 2>&1; then
+  for k in myhostname myorigin relayhost mydestination inet_interfaces; do
+    v="$(as_root postconf -h "$k" 2>/dev/null)"
+    echo "postconf ${k} = ${v:-<empty>}"
+  done
+else
+  echo "postconf: not installed"
+  grep -Hs '^relayhost' /etc/postfix/main.cf 2>/dev/null
+fi
+
+echo "== alt-relay"
+if [ -r /etc/ssmtp/ssmtp.conf ]; then
+  sed -n -E 's/^(AuthPass|AuthUser)=.*/\1=<redacted>/p; s/^(mailhub|hostname|rewriteDomain|UseSTARTTLS|UseTLS)=/\1=/p' /etc/ssmtp/ssmtp.conf | sed 's/^/ssmtp /'
+fi
+for f in /etc/msmtp.conf /root/.msmtprc "${HOME}/.msmtprc"; do
+  if [ -r "$f" ]; then
+    grep -Es '^(account|host|port|from|auth |tls|syslog)' "$f" | sed "s|^|msmtp ${f} |"
+  fi
+done
+for f in /etc/nullmailer/remotes /var/spool/nullmailer/remotes; do
+  if [ -r "$f" ]; then
+    sed -E 's/ --[^ ]*//g' "$f" | sed 's/^/nullmailer-remote /'
+  fi
+done
+if [ -r /etc/dma/dma.conf ]; then
+  grep -Es '^(SMARTHOST|PORT|MAILNAME)' /etc/dma/dma.conf | sed 's/^/dma /'
+fi
+
+echo "== identity"
+if [ -r /etc/mailname ]; then
+  echo "mailname: $(cat /etc/mailname)"
+fi
+grep -hsE '^root:' /etc/aliases /etc/mail/aliases 2>/dev/null | sed 's/^/alias /'
+if [ -r /root/.forward ]; then
+  echo "root-forward: $(head -1 /root/.forward)"
+fi
+
+echo "== listeners-25-465-587"
+as_root ss -ltn 2>/dev/null | awk 'NR==1 || $4 ~ /:(25|465|587|2525)$/'
+
+echo "== recent-relay-evidence"
+as_root sh -c 'grep -h "relay=" /var/log/mail.log /var/log/mail.log.1 /var/log/maillog 2>/dev/null | tail -n 6'
+
+echo "== done $(hostname -f 2>/dev/null || hostname)"

netinfra/mail/audit-targets.txt

@@ -0,0 +1,75 @@
+# audit-targets.txt — fleet targets for run-mail-audit.sh (#696 mail audit)
+# format: mode|name|user   (user only applies to vm mode; default root)
+#   prox = PVE host            -> tests/remote.sh prox-file
+#   vm   = Linux VM / physical -> tests/remote.sh vm-file
+#   dns  = DNS infra alias     -> remote-dns.sh <alias>-file
+# Deliberately NOT targeted (no ssh path / dark / not Linux):
+#   pfv-bms (HAOS core_ssh only; HA SMTP notify lives in KNEL/pfv-bms repo),
+#   tsys-umbrel, kali-tsys (sshd gated), subopi3 (#733 key-denied),
+#   siter-bizoffice (Windows), stlp-3dscanner (offline 184d+)
+prox|pfv-tsys1|
+prox|pfv-tsys3|
+prox|pfv-tsys4|
+prox|pfv-tsys5|
+prox|pfv-tsys6|
+prox|pfv-tsys7|
+prox|pfv-tsys9|
+dns|netinfra01|
+dns|netinfra02|
+vm|tsys-ca|
+vm|tsys-librenms|
+vm|tsys-proxmox-datacenter|
+vm|tsys-proxmox-mailgw-01|
+vm|tsys-proxmox-mailgw-02|
+vm|tsys-ucs-01|
+vm|tsys-ucs-02|
+vm|tsys-awx|
+vm|pfv-rr-middleware-02|
+vm|rr-middleware-01|
+vm|hfnoc-uisp|
+vm|hfnoc-uisp-preprod|
+vm|tsys-siem|
+vm|tsys-voip|
+vm|devbox-cloudron|
+vm|ultix-streaming|
+vm|kali-rd|
+vm|pfv-k8s-cnode1|localuser
+vm|pfv-k8s-wnode-tsys3|localuser
+vm|pfv-k8s-cnode2|localuser
+vm|pfv-k8s-cnode3|localuser
+vm|pfv-k8s-wnode-tsys6|localuser
+vm|pfv-k8s-wnode-tsys7|localuser
+vm|pfv-k8s-wnode-tsys9|localuser
+vm|pfv-proxmox-backup-server|
+vm|pfvsvrpi|localuser
+vm|pfv-jetson-nano-1|
+vm|stlpc-artroom|localuser
+vm|stlpc-garage|localuser
+vm|subopi-dev-3|localuser
+vm|subopi-dev-4|localuser
+vm|netbird|
+vm|sectestbed-sandbox|
+vm|sectestbed-siem|
+vm|sectestbed-proxmox-pve|
+vm|sectestbed-proxmox-datacenter|
+vm|sectestbed-proxmox-pbs|
+vm|sectestbed-awx|
+vm|sectestbed-k8s-cnode|
+vm|sectestbed-k8s-wnode|
+vm|sectestbed-librenms|
+vm|sectestbed-netinfra|
+vm|sectestbed-cloudron|
+vm|sectestbed-proxmox-mailgw|
+vm|sectestbed-ca|
+vm|sectestbed-voip|
+vm|sectestbed-hfnoc-uisp|
+vm|sectestbed-rancherplatform|
+vm|preprod-awx|
+vm|preprod-siem|
+vm|preprod-rancherplatform|
+vm|preprod-proxmox-mailgw|
+vm|preprod-ca|
+vm|preprod-proxmox-datacenter|
+vm|preprod-librenms|
+vm|preprod-voip|
+vm|preprod-cloudron|

netinfra/mail/run-mail-audit.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# run-mail-audit.sh — drive audit-mail-env.sh across the fleet, read-only.
+#
+# All remote access goes through the sanctioned chokepoints:
+#   prox targets  -> tests/remote.sh prox-file      (PVE hosts, root)
+#   vm targets    -> tests/remote.sh vm-file        (VMs + physical Linux)
+#   dns targets   -> remote-dns.sh <alias>-file     (netinfra pair, netboot)
+# Output: one text block per target under .crush/audit/mail/<name>.txt
+#
+# Usage: bash netinfra/mail/run-mail-audit.sh [targets-file]
+set -u
+
+REPO="$(cd "$(dirname "$0")/../.." && pwd)"
+TARGETS="${1:-${REPO}/netinfra/mail/audit-targets.txt}"
+PAYLOAD="${REPO}/netinfra/mail/audit-mail-env.sh"
+OUTDIR="${REPO}/.crush/audit/mail"
+REMOTE_SH="${REPO}/tests/remote.sh"
+DNS_SH="${REPO}/netinfra/dns-cluster-setup/remote-dns.sh"
+TMO="${MAIL_AUDIT_TIMEOUT:-45}"
+
+mkdir -p "${OUTDIR}"
+ok=0
+fail=0
+failed=""
+
+while IFS='|' read -r mode name user; do
+  case "${mode}" in '' | '#'*) continue ;; esac
+  outfile="${OUTDIR}/${name}.txt"
+  rc=0
+  case "${mode}" in
+    prox)
+      timeout "${TMO}" env PROX_HOST="${name}" bash "${REMOTE_SH}" prox-file "${PAYLOAD}" >"${outfile}" 2>&1
+      rc=$?
+      ;;
+    vm)
+      for tgt in "${name}" "${name}.knel.net"; do
+        timeout "${TMO}" env VM_IP="${tgt}" VM_USER="${user:-root}" bash "${REMOTE_SH}" vm-file "${PAYLOAD}" >"${outfile}" 2>&1
+        rc=$?
+        if [ "${rc}" -eq 0 ]; then
+          break
+        fi
+        case "${tgt}" in
+          *.*) break ;;
+        esac
+      done
+      ;;
+    dns)
+      timeout "${TMO}" bash "${DNS_SH}" "${name}-file" "${PAYLOAD}" >"${outfile}" 2>&1
+      rc=$?
+      ;;
+    *)
+      echo "unknown mode '${mode}' for ${name}" >&2
+      continue
+      ;;
+  esac
+  if [ "${rc}" -eq 0 ] && grep -q '^== done' "${outfile}"; then
+    ok=$((ok + 1))
+    echo "[ok] ${name}"
+  else
+    fail=$((fail + 1))
+    failed="${failed} ${name}(rc=${rc})"
+    echo "[FAIL] ${name} rc=${rc}"
+  fi
+done <"${TARGETS}"
+
+echo "=== audit run: ${ok} ok, ${fail} failed${failed}"

questions-v8.md

@@ -0,0 +1,69 @@
+# Questions — v8 (current round)
+
+v7's PMG round is partially resolved — the 2026-09-03 mail audit (55
+systems, read-only) answered several questions with ground truth and
+Charles answered others in chat. Answers recorded below; new questions
+PQ8–PQ10 need your go. Full map + deploy plan:
+[Discourse t/322](https://community.turnsys.com/t/322).
+
+## PMG round — resolved
+
+**PQ1 (backend).** ✅ Cloudron VPS confirmed as backend:
+`mail.knownelement.com` (152.53.37.179). Audit adds: public MX already
+points there; fleet delivers direct today; knel.net zone has no MX.
+
+**PQ3 (outbound smarthost).** ✅ Charles ruled: all hosts send through
+a shared **VIP** in front of the PMG pair. (Mechanics in PQ8 below.)
+
+**PQ4 (placement/sizing).** ✅ Resolved by reality: the pair already
+exists — VM 604 on tsys6 (192.168.1.11) and VM 711 on tsys7
+(192.168.1.10), both 2vCPU/4G/32G, running, onboot, PMG 9.1.0
+installed, unclustered, default config. Nothing to build; configure
+only.
+
+**"Old relay in postfix?"** ✅ None. `relayhost` is empty on every
+audited system (16 postfix installs); no ssmtp/msmtp/nullmailer
+anywhere. Fleet sends direct-to-MX today.
+
+## PMG round — still open
+
+**PQ2 (inbound path).** Where does public :25 DNAT land — rrinfra
+Cisco? Any existing mail port-forwards? (Founder-side; only needed
+before the MX cut-over, not for outbound value.)
+
+**PQ5 (native PMG cluster?).** Recommended: yes — config + quarantine
+replication between .1.11/.1.10, managed strictly serial. OK?
+
+**PQ6 (TLS from fleet CA #697?).** Recommended: yes for the two admin
+UIs; SMTP stays opportunistic. OK?
+
+**PQ7 (notify address).** Evidence says the pair already forwards to
+`coo@turnsys.com`. Confirm that address for quarantine digests + PMG
+alerts?
+
+**PQ8 (VIP specifics).** Proposal: **VIP 192.168.3.249** (pool ends
+.3.200; .250-.254 is your infra block), keepalived VRRP VRID 10 on the
+pair, dhcpd host-declaration pin (VRRP MAC 00:00:5e:00:01:0a), DNS A
+record `smtp.knel.net` → 192.168.3.249. Confirm the number + name, or
+name your preferred slot.
+
+**PQ9 (outbound cutover staging).** Recommended: serial staged — PVE
+hosts + PBS first (verify a week), then the VM postfix set, one
+blue/green hop at a time. OK, or all at once?
+
+**PQ10 (side findings — sub-tickets?).** (a) tsys-ucs-02 root alias
+blackholes to `systemmail@unassigned-hostname...`; (b) Wazuh alerting
+never configured (stock example smtp). Fix as ride-alongs in #696 or
+separate sub-tickets? (Will create them either way — just tell me
+where they land.)
+
+## Carried from v6 — still open
+
+**Q4. Second human approver in gitea?** "Planned for very near
+future"; until then KNEL/access-roster merges are impossible by
+design. Does not block code/deploy work.
+
+## Carried from v7 — superseded questions
+
+PQ1/PQ3/PQ4 answers above replace the v7 wording; v7 file remains in
+git history.