KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(drift): fleet drift report with executive briefing format

Browse Source 
Dedicated drift report covering package/service/tuning/security
inconsistencies across all 7 Proxmox hosts. Executive briefing at the
top (what needs decisions, quick severity summary), detailed matrices
in appendixes.

Key drift findings:
- lldpd inactive on tsys1 (blind spot in topology)
- tsys9 missing 2 SSH keys vs fleet standard
- iperf3 missing on tsys9, net-tools missing on tsys1/6/7
- sysstat missing on tsys5, nvme-cli missing on tsys4/5
- tsys4 tuning drift: 16MB TCP buffers, low backlog, wrong tuned profile
- rsyslog + snmpd + beszel inactive fleet-wide (Saturday OAM Day items)
- noatime only on tsys5 root fs (all others use relatime)

Added perf/scripts/probe-drift.sh as a portable reusable drift probe.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-30 23:27:54 -05:00
parent 0a8865de38
commit 290245349f
4 changed files with 256 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1
View File
@@ -103,6 +103,7 @@ vendor/ Vendored KNELShellFramework
| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit |
| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) |
| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) |
| [`perf/scripts/probe-drift.sh`](perf/scripts/probe-drift.sh) | Read-only fleet drift probe (packages, services, tuning, SSH, security) |
## Key Docs
docs/docmap.md
+1
View File
@@ -30,6 +30,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 |
| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 |
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
perf/scripts/probe-drift.sh
+79
View File
@@ -0,0 +1,79 @@
#!/bin/bash
# Read-only fleet drift probe — gathers package/tooling/config state for
# consistency comparison across hosts. Writes only stdout.
set -u
echo "===== DRIFT PROBE: $(hostname -s) $(date -u +%FT%TZ) ====="
echo
echo "##### OS / kernel / PVE #####"
pveversion 2>&1
cat /etc/debian_version 2>&1
echo
echo "##### Installed key packages (versions) #####"
dpkg-query -W -f='${Package}\t${Version}\n' 2>/dev/null | grep -iE 'lldpd|lldpad|smartmontools|nfs-common|nfs-kernel-server|iperf3|tcpdump|htop|rsyslog|qemu-guest-agent|snmpd|snmp|net-tools|ethtool|sysstat|ioping|fio|nvme-cli|conman|ser2net|nut-server|nut-client|tuned-adm|tuned' 2>/dev/null | sort
echo
echo "##### lldpcli present? #####"
command -v lldpcli >/dev/null 2>&1 && lldpcli -v 2>&1 | head -1 || echo "lldpcli: NOT INSTALLED"
echo
echo "##### lldpd service #####"
systemctl is-active lldpd 2>&1 || true
systemctl is-enabled lldpd 2>&1 || true
echo
echo "##### smartmontools service #####"
systemctl is-active smartd 2>&1 || echo "smartd: inactive/disabled"
systemctl is-enabled smartd 2>&1 || true
echo
echo "##### snmpd service #####"
systemctl is-active snmpd 2>&1 || echo "snmpd: not running"
systemctl is-enabled snmpd 2>&1 || true
echo
echo "##### rsyslog #####"
systemctl is-active rsyslog 2>&1 || echo "rsyslog: not running"
echo
echo "##### tuned profile #####"
tuned-adm active 2>&1 | head -2 || echo "tuned: not available"
echo
echo "##### sshd config (key settings) #####"
grep -E '^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|KbdInteractiveAuthentication|Port |PermitEmptyPasswords)' /etc/ssh/sshd_config 2>/dev/null || echo "(defaults)"
grep -E '^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication|KbdInteractiveAuthentication|Port |PermitEmptyPasswords)' /etc/ssh/sshd_config.d/*.conf 2>/dev/null || true
echo
echo "##### authorized_keys (root) — count + key types #####"
if [ -f /root/.ssh/authorized_keys ]; then
echo " total keys: $(wc -l < /root/.ssh/authorized_keys)"
awk '{print " "$1}' /root/.ssh/authorized_keys | sort | uniq -c
else
echo " NO authorized_keys for root"
fi
echo
echo "##### ssh host key fingerprints #####"
for f in /etc/ssh/ssh_host_*_key.pub; do
[ -f "$f" ] && ssh-keygen -lf "$f" 2>/dev/null
done
echo
echo "##### sysctl tuning (key values) #####"
echo " net.core.rmem_max=$(cat /proc/sys/net/core/rmem_max 2>/dev/null)"
echo " net.core.wmem_max=$(cat /proc/sys/net/core/wmem_max 2>/dev/null)"
echo " net.ipv4.tcp_congestion_control=$(cat /proc/sys/net/ipv4/tcp_congestion_control 2>/dev/null)"
echo " net.ipv4.tcp_rmem=$(cat /proc/sys/net/ipv4/tcp_rmem 2>/dev/null)"
echo " net.ipv4.tcp_wmem=$(cat /proc/sys/net/ipv4/tcp_wmem 2>/dev/null)"
echo " net.ipv4.tcp_max_syn_backlog=$(cat /proc/sys/net/ipv4/tcp_max_syn_backlog 2>/dev/null)"
echo " net.core.netdev_max_backlog=$(cat /proc/sys/net/core/netdev_max_backlog 2>/dev/null)"
echo " vm.swappiness=$(cat /proc/sys/vm/swappiness 2>/dev/null)"
echo
echo "##### CPU governor #####"
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null || echo "no cpufreq driver"
echo
echo "##### beszel agent? #####"
systemctl is-active beszel-hub 2>/dev/null || systemctl is-active beszel-agent 2>/dev/null || echo "beszel: not installed"
echo
echo "##### /root/bin or custom scripts present? #####"
# shellcheck disable=SC2012 # diagnostic listing, not for processing
find /root/bin/ -maxdepth 1 -type f 2>/dev/null | head -10 || echo "(no /root/bin)"
find /root/ -maxdepth 1 -name '*.sh' -type f 2>/dev/null | head -10 || echo "(no /root/*.sh)"
echo
echo "##### mount options (noatime check on root) #####"
mount | grep ' / ' | head -1
echo
echo "##### NFS server threads (if NFS server) #####"
grep -E '^[[:space:]]*RPCNFSDCOUNT' /etc/default/nfs-kernel-server 2>/dev/null || grep -E 'RPCNFSDCOUNT' /etc/default/nfs-kernel-server 2>/dev/null || echo "(default, not a NFS server or default threads)"
echo
echo "===== END $(hostname -s) ====="
proxmox/docs/DRIFT-2026-07-30.md
+175
View File
@@ -0,0 +1,175 @@
# Fleet Drift Report
> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts)
> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel)
> **Report generated:** 2026-07-30
---
## Executive Briefing
### What needs your decision now
| # | Issue | Impact | Recommendation |
|---|-------|--------|---------------|
| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` |
| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing |
| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` |
| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` |
| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` |
| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` |
| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 |
| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) |
| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog |
| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS |
| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide |
### Consistent across the fleet (good)
These items are the SAME on all 7 hosts — no drift:
- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task)
- **Debian version:** all on trixie/13
- **tcp_congestion_control:** bbr everywhere
- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected)
- **ethtool:** same version everywhere
- **nfs-common:** same version everywhere
- **tcpdump:** same version everywhere
- **smartmontools:** 7.5 everywhere, smartd active everywhere
- **tuned:** installed everywhere
- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM)
- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module
### Quick severity summary
- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5
- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`)
- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment
- **Investigate:** tsys9 SSH key count (2 keys missing)
---
## Appendix A: Package Presence Matrix
> MISSING = not installed. Version shown = installed version (truncated).
| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 |
| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 |
| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 |
| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — |
| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** |
| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 |
| rsyslog | installed | installed | installed | installed | installed | installed | installed |
| qemu-guest-agent | — | — | — | — | — | — | — |
| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** |
| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** |
| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 |
| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 |
| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 |
| conman | — | — | 0.3.1 | — | — | — | — |
| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — |
| nut-server | 2.8.1 | — | — | — | — | — | — |
| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 |
> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are
> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config.
> conman/ser2net on tsys4 is expected (console server). These are not drift.
## Appendix B: Service State Matrix
| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **inactive** | active | active | active | active | active | active |
| smartd | active | active | active | active | active | active | active |
| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| beszel | — | — | — | — | — | — | — |
> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM
> Day items, not drift — they haven't been deployed yet.
## Appendix C: Sysctl Tuning Matrix
| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr |
| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 |
| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 |
| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 |
| governor | perf | perf | perf | N/A | perf | perf | perf |
> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB),
> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000).
> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection
> window size, and the low backlog values can cause packet drops under load.
> swappiness=1 on tsys4/5 is intentional (storage hosts).
## Appendix D: Tuned Profile Matrix
| Host | Profile | Expected | Match? |
|------|---------|----------|--------|
| tsys1 | virtual-host | virtual-host | ✓ |
| tsys3 | virtual-host | virtual-host | ✓ |
| **tsys4** | **throughput-performance** | **network-throughput** | **✗** |
| tsys5 | network-throughput | network-throughput | ✓ |
| tsys6 | virtual-host | virtual-host | ✓ |
| tsys7 | virtual-host | virtual-host | ✓ |
| tsys9 | virtual-host | virtual-host | ✓ |
## Appendix E: SSH + Security State
### Authorized keys (root)
| Host | Key count | Notes |
|------|-----------|-------|
| tsys1 | 5 | Standard |
| tsys3 | 5 | Standard |
| tsys4 | 5 | Standard |
| tsys5 | 5 | Standard |
| tsys6 | 5 | Standard |
| tsys7 | 5 | Standard |
| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** |
### SSHD config (consistent across fleet)
| Setting | Value | All hosts |
|---------|-------|-----------|
| PermitRootLogin | yes | ✓ (all) |
| PasswordAuthentication | (default — not set) | ✓ (all) |
| KbdInteractiveAuthentication | no | ✓ (all) |
> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's
> KbdInteractiveAuthentication. The PAM approach is consistent.
## Appendix F: Filesystem Mount Options
| Host | root fs mount option | Notes |
|------|---------------------|-------|
| tsys1 | relatime | Default |
| tsys3 | relatime | Default |
| tsys4 | relatime | Default |
| **tsys5** | **noatime** | **Only host with noatime on root** |
| tsys6 | relatime | Default |
| tsys7 | relatime | Default |
| tsys9 | relatime | Default |
> Minor: noatime reduces metadata writes (slight improvement on HDD).
> Storage hosts (tsys4/5) would benefit most from noatime.
## Appendix G: Raw Data Location
All raw drift probe output is stored in `/tmp/<hostname>-drift.txt` on this
workstation (not committed — ephemeral). Re-gather any time with:
```bash
for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do
PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt
done
```
The probe script should be saved to the repo as `perf/scripts/probe-drift.sh`
for reuse. It is read-only and portable.