From 290245349f67beee0a157a47380876f2ada76242 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Thu, 30 Jul 2026 23:27:54 -0500 Subject: [PATCH] docs(drift): fleet drift report with executive briefing format MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dedicated drift report covering package/service/tuning/security inconsistencies across all 7 Proxmox hosts. Executive briefing at the top (what needs decisions, quick severity summary), detailed matrices in appendixes. Key drift findings: - lldpd inactive on tsys1 (blind spot in topology) - tsys9 missing 2 SSH keys vs fleet standard - iperf3 missing on tsys9, net-tools missing on tsys1/6/7 - sysstat missing on tsys5, nvme-cli missing on tsys4/5 - tsys4 tuning drift: 16MB TCP buffers, low backlog, wrong tuned profile - rsyslog + snmpd + beszel inactive fleet-wide (Saturday OAM Day items) - noatime only on tsys5 root fs (all others use relatime) Added perf/scripts/probe-drift.sh as a portable reusable drift probe. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- AGENTS.md | 1 + docs/docmap.md | 1 + perf/scripts/probe-drift.sh | 79 ++++++++++++++ proxmox/docs/DRIFT-2026-07-30.md | 175 +++++++++++++++++++++++++++++++ 4 files changed, 256 insertions(+) create mode 100644 perf/scripts/probe-drift.sh create mode 100644 proxmox/docs/DRIFT-2026-07-30.md diff --git a/AGENTS.md b/AGENTS.md index bfa57bb..ea80db3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -103,6 +103,7 @@ vendor/ Vendored KNELShellFramework | [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit | | [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) | | [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) | +| [`perf/scripts/probe-drift.sh`](perf/scripts/probe-drift.sh) | Read-only fleet drift probe (packages, services, tuning, SSH, security) | ## Key Docs diff --git a/docs/docmap.md b/docs/docmap.md index c7b6037..c4775de 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -30,6 +30,7 @@ Fleet operations, hardware, performance tuning, storage architecture. | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | | [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 | | [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 | +| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | diff --git a/perf/scripts/probe-drift.sh b/perf/scripts/probe-drift.sh new file mode 100644 index 0000000..b78be7e --- /dev/null +++ b/perf/scripts/probe-drift.sh @@ -0,0 +1,79 @@ +#!/bin/bash +# Read-only fleet drift probe — gathers package/tooling/config state for +# consistency comparison across hosts. Writes only stdout. +set -u +echo "===== DRIFT PROBE: $(hostname -s) $(date -u +%FT%TZ) =====" +echo +echo "##### OS / kernel / PVE #####" +pveversion 2>&1 +cat /etc/debian_version 2>&1 +echo +echo "##### Installed key packages (versions) #####" +dpkg-query -W -f='${Package}\t${Version}\n' 2>/dev/null | grep -iE 'lldpd|lldpad|smartmontools|nfs-common|nfs-kernel-server|iperf3|tcpdump|htop|rsyslog|qemu-guest-agent|snmpd|snmp|net-tools|ethtool|sysstat|ioping|fio|nvme-cli|conman|ser2net|nut-server|nut-client|tuned-adm|tuned' 2>/dev/null | sort +echo +echo "##### lldpcli present? #####" +command -v lldpcli >/dev/null 2>&1 && lldpcli -v 2>&1 | head -1 || echo "lldpcli: NOT INSTALLED" +echo +echo "##### lldpd service #####" +systemctl is-active lldpd 2>&1 || true +systemctl is-enabled lldpd 2>&1 || true +echo +echo "##### smartmontools service #####" +systemctl is-active smartd 2>&1 || echo "smartd: inactive/disabled" +systemctl is-enabled smartd 2>&1 || true +echo +echo "##### snmpd service #####" +systemctl is-active snmpd 2>&1 || echo "snmpd: not running" +systemctl is-enabled snmpd 2>&1 || true +echo +echo "##### rsyslog #####" +systemctl is-active rsyslog 2>&1 || echo "rsyslog: not running" +echo +echo "##### tuned profile #####" +tuned-adm active 2>&1 | head -2 || echo "tuned: not available" +echo +echo "##### sshd config (key settings) #####" +grep -E '^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|KbdInteractiveAuthentication|Port |PermitEmptyPasswords)' /etc/ssh/sshd_config 2>/dev/null || echo "(defaults)" +grep -E '^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|KbdInteractiveAuthentication|Port |PermitEmptyPasswords)' /etc/ssh/sshd_config.d/*.conf 2>/dev/null || true +echo +echo "##### authorized_keys (root) — count + key types #####" +if [ -f /root/.ssh/authorized_keys ]; then + echo " total keys: $(wc -l < /root/.ssh/authorized_keys)" + awk '{print " "$1}' /root/.ssh/authorized_keys | sort | uniq -c +else + echo " NO authorized_keys for root" +fi +echo +echo "##### ssh host key fingerprints #####" +for f in /etc/ssh/ssh_host_*_key.pub; do + [ -f "$f" ] && ssh-keygen -lf "$f" 2>/dev/null +done +echo +echo "##### sysctl tuning (key values) #####" +echo " net.core.rmem_max=$(cat /proc/sys/net/core/rmem_max 2>/dev/null)" +echo " net.core.wmem_max=$(cat /proc/sys/net/core/wmem_max 2>/dev/null)" +echo " net.ipv4.tcp_congestion_control=$(cat /proc/sys/net/ipv4/tcp_congestion_control 2>/dev/null)" +echo " net.ipv4.tcp_rmem=$(cat /proc/sys/net/ipv4/tcp_rmem 2>/dev/null)" +echo " net.ipv4.tcp_wmem=$(cat /proc/sys/net/ipv4/tcp_wmem 2>/dev/null)" +echo " net.ipv4.tcp_max_syn_backlog=$(cat /proc/sys/net/ipv4/tcp_max_syn_backlog 2>/dev/null)" +echo " net.core.netdev_max_backlog=$(cat /proc/sys/net/core/netdev_max_backlog 2>/dev/null)" +echo " vm.swappiness=$(cat /proc/sys/vm/swappiness 2>/dev/null)" +echo +echo "##### CPU governor #####" +cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "no cpufreq driver" +echo +echo "##### beszel agent? #####" +systemctl is-active beszel-hub 2>/dev/null || systemctl is-active beszel-agent 2>/dev/null || echo "beszel: not installed" +echo +echo "##### /root/bin or custom scripts present? #####" +# shellcheck disable=SC2012 # diagnostic listing, not for processing +find /root/bin/ -maxdepth 1 -type f 2>/dev/null | head -10 || echo "(no /root/bin)" +find /root/ -maxdepth 1 -name '*.sh' -type f 2>/dev/null | head -10 || echo "(no /root/*.sh)" +echo +echo "##### mount options (noatime check on root) #####" +mount | grep ' / ' | head -1 +echo +echo "##### NFS server threads (if NFS server) #####" +grep -E '^[[:space:]]*RPCNFSDCOUNT' /etc/default/nfs-kernel-server 2>/dev/null || grep -E 'RPCNFSDCOUNT' /etc/default/nfs-kernel-server 2>/dev/null || echo "(default, not a NFS server or default threads)" +echo +echo "===== END $(hostname -s) =====" diff --git a/proxmox/docs/DRIFT-2026-07-30.md b/proxmox/docs/DRIFT-2026-07-30.md new file mode 100644 index 0000000..3f3fd6a --- /dev/null +++ b/proxmox/docs/DRIFT-2026-07-30.md @@ -0,0 +1,175 @@ +# Fleet Drift Report + +> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts) +> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel) +> **Report generated:** 2026-07-30 + +--- + +## Executive Briefing + +### What needs your decision now + +| # | Issue | Impact | Recommendation | +|---|-------|--------|---------------| +| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` | +| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing | +| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` | +| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` | +| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` | +| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` | +| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 | +| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) | +| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog | +| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS | +| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide | + +### Consistent across the fleet (good) + +These items are the SAME on all 7 hosts — no drift: + +- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task) +- **Debian version:** all on trixie/13 +- **tcp_congestion_control:** bbr everywhere +- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected) +- **ethtool:** same version everywhere +- **nfs-common:** same version everywhere +- **tcpdump:** same version everywhere +- **smartmontools:** 7.5 everywhere, smartd active everywhere +- **tuned:** installed everywhere +- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM) +- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module + +### Quick severity summary + +- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5 +- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`) +- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment +- **Investigate:** tsys9 SSH key count (2 keys missing) + +--- + +## Appendix A: Package Presence Matrix + +> MISSING = not installed. Version shown = installed version (truncated). + +| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | +|---------|-------|-------|-------|-------|-------|-------|-------| +| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | +| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | +| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | +| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — | +| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** | +| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | +| rsyslog | installed | installed | installed | installed | installed | installed | installed | +| qemu-guest-agent | — | — | — | — | — | — | — | +| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | +| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** | +| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | +| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 | +| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 | +| conman | — | — | 0.3.1 | — | — | — | — | +| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — | +| nut-server | 2.8.1 | — | — | — | — | — | — | +| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | + +> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are +> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config. +> conman/ser2net on tsys4 is expected (console server). These are not drift. + +## Appendix B: Service State Matrix + +| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | +|---------|-------|-------|-------|-------|-------|-------|-------| +| lldpd | **inactive** | active | active | active | active | active | active | +| smartd | active | active | active | active | active | active | active | +| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | +| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | +| beszel | — | — | — | — | — | — | — | + +> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM +> Day items, not drift — they haven't been deployed yet. + +## Appendix C: Sysctl Tuning Matrix + +| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | +|---------|-------|-------|-------|-------|-------|-------|-------| +| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB | +| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB | +| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr | +| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 | +| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 | +| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 | +| governor | perf | perf | perf | N/A | perf | perf | perf | + +> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB), +> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000). +> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection +> window size, and the low backlog values can cause packet drops under load. +> swappiness=1 on tsys4/5 is intentional (storage hosts). + +## Appendix D: Tuned Profile Matrix + +| Host | Profile | Expected | Match? | +|------|---------|----------|--------| +| tsys1 | virtual-host | virtual-host | ✓ | +| tsys3 | virtual-host | virtual-host | ✓ | +| **tsys4** | **throughput-performance** | **network-throughput** | **✗** | +| tsys5 | network-throughput | network-throughput | ✓ | +| tsys6 | virtual-host | virtual-host | ✓ | +| tsys7 | virtual-host | virtual-host | ✓ | +| tsys9 | virtual-host | virtual-host | ✓ | + +## Appendix E: SSH + Security State + +### Authorized keys (root) + +| Host | Key count | Notes | +|------|-----------|-------| +| tsys1 | 5 | Standard | +| tsys3 | 5 | Standard | +| tsys4 | 5 | Standard | +| tsys5 | 5 | Standard | +| tsys6 | 5 | Standard | +| tsys7 | 5 | Standard | +| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** | + +### SSHD config (consistent across fleet) + +| Setting | Value | All hosts | +|---------|-------|-----------| +| PermitRootLogin | yes | ✓ (all) | +| PasswordAuthentication | (default — not set) | ✓ (all) | +| KbdInteractiveAuthentication | no | ✓ (all) | + +> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's +> KbdInteractiveAuthentication. The PAM approach is consistent. + +## Appendix F: Filesystem Mount Options + +| Host | root fs mount option | Notes | +|------|---------------------|-------| +| tsys1 | relatime | Default | +| tsys3 | relatime | Default | +| tsys4 | relatime | Default | +| **tsys5** | **noatime** | **Only host with noatime on root** | +| tsys6 | relatime | Default | +| tsys7 | relatime | Default | +| tsys9 | relatime | Default | + +> Minor: noatime reduces metadata writes (slight improvement on HDD). +> Storage hosts (tsys4/5) would benefit most from noatime. + +## Appendix G: Raw Data Location + +All raw drift probe output is stored in `/tmp/-drift.txt` on this +workstation (not committed — ephemeral). Re-gather any time with: + +```bash +for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do + PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt +done +``` + +The probe script should be saved to the repo as `perf/scripts/probe-drift.sh` +for reuse. It is read-only and portable.