From d3b9099d5ffeb3142c759c70095549a4c5f41422 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Thu, 3 Sep 2026 12:49:59 -0500 Subject: [PATCH] feat(mail): fleet mail-env audit + map; PMG VIP proposal [#696][#694] Audit findings note (55/64 systems, read-only): https://projects.knownelement.com/issues/696#note-4088 Map + deploy plan: https://community.turnsys.com/t/322 Key: relayhost empty fleet-wide (direct-to-MX today); prod PMG pair already installed (VM 604/711, PMG 9.1, unclustered); VIP slot proposed 192.168.3.249; open PQs in questions-v8.md. --- netinfra/mail/README.md | 21 +++++++++ netinfra/mail/audit-mail-env.sh | 78 +++++++++++++++++++++++++++++++++ netinfra/mail/audit-targets.txt | 75 +++++++++++++++++++++++++++++++ netinfra/mail/run-mail-audit.sh | 66 ++++++++++++++++++++++++++++ questions-v8.md | 69 +++++++++++++++++++++++++++++ 5 files changed, 309 insertions(+) create mode 100644 netinfra/mail/README.md create mode 100644 netinfra/mail/audit-mail-env.sh create mode 100644 netinfra/mail/audit-targets.txt create mode 100644 netinfra/mail/run-mail-audit.sh create mode 100644 questions-v8.md diff --git a/netinfra/mail/README.md b/netinfra/mail/README.md new file mode 100644 index 0000000..226ce14 --- /dev/null +++ b/netinfra/mail/README.md @@ -0,0 +1,21 @@ +# netinfra/mail — PMG / fleet mail tooling + +Pointers + usage. Knowledge doc (map, findings, deploy plan): +[Discourse t/322 — Mail environment map + PMG deploy plan](https://community.turnsys.com/t/322). +Ticket: [Redmine #696](https://projects.knownelement.com/issues/696). + +## Files + +- `audit-mail-env.sh` — read-only mail probe, runs ON a target + (postfix config, alt relays, identity, :25 listeners, relay log + evidence). Delivered via the ssh chokepoints only + (`remote.sh prox-file|vm-file`, `remote-dns.sh -file`). +- `run-mail-audit.sh` — drives the probe across + `audit-targets.txt`; output per host under `.crush/audit/mail/`. +- `audit-targets.txt` — fleet target list (`mode|name|user`). + +## Re-run the audit + +```bash +bash netinfra/mail/run-mail-audit.sh +``` diff --git a/netinfra/mail/audit-mail-env.sh b/netinfra/mail/audit-mail-env.sh new file mode 100644 index 0000000..df09ad2 --- /dev/null +++ b/netinfra/mail/audit-mail-env.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# audit-mail-env.sh — read-only mail environment probe. +# +# Runs ON the target host, delivered via the ssh chokepoints: +# tests/remote.sh prox-file|vm-file OR remote-dns.sh -file +# Collects: installed MTAs, postfix relay config, alternative relay configs +# (ssmtp/msmtp/nullmailer/dma), identity (mailname/aliases/forward), +# smtp listeners, and recent relay evidence from mail logs. +# Makes NO changes. Redacts auth material. Output is "KEY value" text. +set -u + +as_root() { + if [ "$(id -u)" -eq 0 ]; then + "$@" + else + if sudo -n "$@" 2>/dev/null; then + : + else + "$@" + fi + fi +} + +echo "== host $(hostname -f 2>/dev/null || hostname) | $(date '+%F %T %Z')" + +echo "== mtas" +for b in postconf postfix sendmail exim4 msmtp ssmtp nullmailer-send dma; do + p="$(command -v "$b" 2>/dev/null)" + if [ -n "${p}" ]; then + echo "bin ${b}: ${p}" + fi +done + +echo "== postfix" +if command -v postconf >/dev/null 2>&1; then + for k in myhostname myorigin relayhost mydestination inet_interfaces; do + v="$(as_root postconf -h "$k" 2>/dev/null)" + echo "postconf ${k} = ${v:-}" + done +else + echo "postconf: not installed" + grep -Hs '^relayhost' /etc/postfix/main.cf 2>/dev/null +fi + +echo "== alt-relay" +if [ -r /etc/ssmtp/ssmtp.conf ]; then + sed -n -E 's/^(AuthPass|AuthUser)=.*/\1=/p; s/^(mailhub|hostname|rewriteDomain|UseSTARTTLS|UseTLS)=/\1=/p' /etc/ssmtp/ssmtp.conf | sed 's/^/ssmtp /' +fi +for f in /etc/msmtp.conf /root/.msmtprc "${HOME}/.msmtprc"; do + if [ -r "$f" ]; then + grep -Es '^(account|host|port|from|auth |tls|syslog)' "$f" | sed "s|^|msmtp ${f} |" + fi +done +for f in /etc/nullmailer/remotes /var/spool/nullmailer/remotes; do + if [ -r "$f" ]; then + sed -E 's/ --[^ ]*//g' "$f" | sed 's/^/nullmailer-remote /' + fi +done +if [ -r /etc/dma/dma.conf ]; then + grep -Es '^(SMARTHOST|PORT|MAILNAME)' /etc/dma/dma.conf | sed 's/^/dma /' +fi + +echo "== identity" +if [ -r /etc/mailname ]; then + echo "mailname: $(cat /etc/mailname)" +fi +grep -hsE '^root:' /etc/aliases /etc/mail/aliases 2>/dev/null | sed 's/^/alias /' +if [ -r /root/.forward ]; then + echo "root-forward: $(head -1 /root/.forward)" +fi + +echo "== listeners-25-465-587" +as_root ss -ltn 2>/dev/null | awk 'NR==1 || $4 ~ /:(25|465|587|2525)$/' + +echo "== recent-relay-evidence" +as_root sh -c 'grep -h "relay=" /var/log/mail.log /var/log/mail.log.1 /var/log/maillog 2>/dev/null | tail -n 6' + +echo "== done $(hostname -f 2>/dev/null || hostname)" diff --git a/netinfra/mail/audit-targets.txt b/netinfra/mail/audit-targets.txt new file mode 100644 index 0000000..c4e42e4 --- /dev/null +++ b/netinfra/mail/audit-targets.txt @@ -0,0 +1,75 @@ +# audit-targets.txt — fleet targets for run-mail-audit.sh (#696 mail audit) +# format: mode|name|user (user only applies to vm mode; default root) +# prox = PVE host -> tests/remote.sh prox-file +# vm = Linux VM / physical -> tests/remote.sh vm-file +# dns = DNS infra alias -> remote-dns.sh -file +# Deliberately NOT targeted (no ssh path / dark / not Linux): +# pfv-bms (HAOS core_ssh only; HA SMTP notify lives in KNEL/pfv-bms repo), +# tsys-umbrel, kali-tsys (sshd gated), subopi3 (#733 key-denied), +# siter-bizoffice (Windows), stlp-3dscanner (offline 184d+) +prox|pfv-tsys1| +prox|pfv-tsys3| +prox|pfv-tsys4| +prox|pfv-tsys5| +prox|pfv-tsys6| +prox|pfv-tsys7| +prox|pfv-tsys9| +dns|netinfra01| +dns|netinfra02| +vm|tsys-ca| +vm|tsys-librenms| +vm|tsys-proxmox-datacenter| +vm|tsys-proxmox-mailgw-01| +vm|tsys-proxmox-mailgw-02| +vm|tsys-ucs-01| +vm|tsys-ucs-02| +vm|tsys-awx| +vm|pfv-rr-middleware-02| +vm|rr-middleware-01| +vm|hfnoc-uisp| +vm|hfnoc-uisp-preprod| +vm|tsys-siem| +vm|tsys-voip| +vm|devbox-cloudron| +vm|ultix-streaming| +vm|kali-rd| +vm|pfv-k8s-cnode1|localuser +vm|pfv-k8s-wnode-tsys3|localuser +vm|pfv-k8s-cnode2|localuser +vm|pfv-k8s-cnode3|localuser +vm|pfv-k8s-wnode-tsys6|localuser +vm|pfv-k8s-wnode-tsys7|localuser +vm|pfv-k8s-wnode-tsys9|localuser +vm|pfv-proxmox-backup-server| +vm|pfvsvrpi|localuser +vm|pfv-jetson-nano-1| +vm|stlpc-artroom|localuser +vm|stlpc-garage|localuser +vm|subopi-dev-3|localuser +vm|subopi-dev-4|localuser +vm|netbird| +vm|sectestbed-sandbox| +vm|sectestbed-siem| +vm|sectestbed-proxmox-pve| +vm|sectestbed-proxmox-datacenter| +vm|sectestbed-proxmox-pbs| +vm|sectestbed-awx| +vm|sectestbed-k8s-cnode| +vm|sectestbed-k8s-wnode| +vm|sectestbed-librenms| +vm|sectestbed-netinfra| +vm|sectestbed-cloudron| +vm|sectestbed-proxmox-mailgw| +vm|sectestbed-ca| +vm|sectestbed-voip| +vm|sectestbed-hfnoc-uisp| +vm|sectestbed-rancherplatform| +vm|preprod-awx| +vm|preprod-siem| +vm|preprod-rancherplatform| +vm|preprod-proxmox-mailgw| +vm|preprod-ca| +vm|preprod-proxmox-datacenter| +vm|preprod-librenms| +vm|preprod-voip| +vm|preprod-cloudron| diff --git a/netinfra/mail/run-mail-audit.sh b/netinfra/mail/run-mail-audit.sh new file mode 100644 index 0000000..2cda91d --- /dev/null +++ b/netinfra/mail/run-mail-audit.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# run-mail-audit.sh — drive audit-mail-env.sh across the fleet, read-only. +# +# All remote access goes through the sanctioned chokepoints: +# prox targets -> tests/remote.sh prox-file (PVE hosts, root) +# vm targets -> tests/remote.sh vm-file (VMs + physical Linux) +# dns targets -> remote-dns.sh -file (netinfra pair, netboot) +# Output: one text block per target under .crush/audit/mail/.txt +# +# Usage: bash netinfra/mail/run-mail-audit.sh [targets-file] +set -u + +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +TARGETS="${1:-${REPO}/netinfra/mail/audit-targets.txt}" +PAYLOAD="${REPO}/netinfra/mail/audit-mail-env.sh" +OUTDIR="${REPO}/.crush/audit/mail" +REMOTE_SH="${REPO}/tests/remote.sh" +DNS_SH="${REPO}/netinfra/dns-cluster-setup/remote-dns.sh" +TMO="${MAIL_AUDIT_TIMEOUT:-45}" + +mkdir -p "${OUTDIR}" +ok=0 +fail=0 +failed="" + +while IFS='|' read -r mode name user; do + case "${mode}" in '' | '#'*) continue ;; esac + outfile="${OUTDIR}/${name}.txt" + rc=0 + case "${mode}" in + prox) + timeout "${TMO}" env PROX_HOST="${name}" bash "${REMOTE_SH}" prox-file "${PAYLOAD}" >"${outfile}" 2>&1 + rc=$? + ;; + vm) + for tgt in "${name}" "${name}.knel.net"; do + timeout "${TMO}" env VM_IP="${tgt}" VM_USER="${user:-root}" bash "${REMOTE_SH}" vm-file "${PAYLOAD}" >"${outfile}" 2>&1 + rc=$? + if [ "${rc}" -eq 0 ]; then + break + fi + case "${tgt}" in + *.*) break ;; + esac + done + ;; + dns) + timeout "${TMO}" bash "${DNS_SH}" "${name}-file" "${PAYLOAD}" >"${outfile}" 2>&1 + rc=$? + ;; + *) + echo "unknown mode '${mode}' for ${name}" >&2 + continue + ;; + esac + if [ "${rc}" -eq 0 ] && grep -q '^== done' "${outfile}"; then + ok=$((ok + 1)) + echo "[ok] ${name}" + else + fail=$((fail + 1)) + failed="${failed} ${name}(rc=${rc})" + echo "[FAIL] ${name} rc=${rc}" + fi +done <"${TARGETS}" + +echo "=== audit run: ${ok} ok, ${fail} failed${failed}" diff --git a/questions-v8.md b/questions-v8.md new file mode 100644 index 0000000..18f9898 --- /dev/null +++ b/questions-v8.md @@ -0,0 +1,69 @@ +# Questions — v8 (current round) + +v7's PMG round is partially resolved — the 2026-09-03 mail audit (55 +systems, read-only) answered several questions with ground truth and +Charles answered others in chat. Answers recorded below; new questions +PQ8–PQ10 need your go. Full map + deploy plan: +[Discourse t/322](https://community.turnsys.com/t/322). + +## PMG round — resolved + +**PQ1 (backend).** ✅ Cloudron VPS confirmed as backend: +`mail.knownelement.com` (152.53.37.179). Audit adds: public MX already +points there; fleet delivers direct today; knel.net zone has no MX. + +**PQ3 (outbound smarthost).** ✅ Charles ruled: all hosts send through +a shared **VIP** in front of the PMG pair. (Mechanics in PQ8 below.) + +**PQ4 (placement/sizing).** ✅ Resolved by reality: the pair already +exists — VM 604 on tsys6 (192.168.1.11) and VM 711 on tsys7 +(192.168.1.10), both 2vCPU/4G/32G, running, onboot, PMG 9.1.0 +installed, unclustered, default config. Nothing to build; configure +only. + +**"Old relay in postfix?"** ✅ None. `relayhost` is empty on every +audited system (16 postfix installs); no ssmtp/msmtp/nullmailer +anywhere. Fleet sends direct-to-MX today. + +## PMG round — still open + +**PQ2 (inbound path).** Where does public :25 DNAT land — rrinfra +Cisco? Any existing mail port-forwards? (Founder-side; only needed +before the MX cut-over, not for outbound value.) + +**PQ5 (native PMG cluster?).** Recommended: yes — config + quarantine +replication between .1.11/.1.10, managed strictly serial. OK? + +**PQ6 (TLS from fleet CA #697?).** Recommended: yes for the two admin +UIs; SMTP stays opportunistic. OK? + +**PQ7 (notify address).** Evidence says the pair already forwards to +`coo@turnsys.com`. Confirm that address for quarantine digests + PMG +alerts? + +**PQ8 (VIP specifics).** Proposal: **VIP 192.168.3.249** (pool ends +.3.200; .250-.254 is your infra block), keepalived VRRP VRID 10 on the +pair, dhcpd host-declaration pin (VRRP MAC 00:00:5e:00:01:0a), DNS A +record `smtp.knel.net` → 192.168.3.249. Confirm the number + name, or +name your preferred slot. + +**PQ9 (outbound cutover staging).** Recommended: serial staged — PVE +hosts + PBS first (verify a week), then the VM postfix set, one +blue/green hop at a time. OK, or all at once? + +**PQ10 (side findings — sub-tickets?).** (a) tsys-ucs-02 root alias +blackholes to `systemmail@unassigned-hostname...`; (b) Wazuh alerting +never configured (stock example smtp). Fix as ride-alongs in #696 or +separate sub-tickets? (Will create them either way — just tell me +where they land.) + +## Carried from v6 — still open + +**Q4. Second human approver in gitea?** "Planned for very near +future"; until then KNEL/access-roster merges are impossible by +design. Does not block code/deploy work. + +## Carried from v7 — superseded questions + +PQ1/PQ3/PQ4 answers above replace the v7 wording; v7 file remains in +git history.