KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6c58671d15ccf24d23b3265d322c20efd24bc445
PFVCluster/proxmox/docs/DRIFT-2026-07-30.md
T
mrcharles 290245349f docs(drift): fleet drift report with executive briefing format 
Dedicated drift report covering package/service/tuning/security
inconsistencies across all 7 Proxmox hosts. Executive briefing at the
top (what needs decisions, quick severity summary), detailed matrices
in appendixes.

Key drift findings:
- lldpd inactive on tsys1 (blind spot in topology)
- tsys9 missing 2 SSH keys vs fleet standard
- iperf3 missing on tsys9, net-tools missing on tsys1/6/7
- sysstat missing on tsys5, nvme-cli missing on tsys4/5
- tsys4 tuning drift: 16MB TCP buffers, low backlog, wrong tuned profile
- rsyslog + snmpd + beszel inactive fleet-wide (Saturday OAM Day items)
- noatime only on tsys5 root fs (all others use relatime)

Added perf/scripts/probe-drift.sh as a portable reusable drift probe.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:27:54 -05:00

176 lines
8.8 KiB
Markdown
Raw Blame History

# Fleet Drift Report
> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts)
> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel)
> **Report generated:** 2026-07-30
---
## Executive Briefing
### What needs your decision now
| # | Issue | Impact | Recommendation |
|---|-------|--------|---------------|
| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` |
| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing |
| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` |
| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` |
| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` |
| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` |
| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 |
| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) |
| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog |
| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS |
| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide |
### Consistent across the fleet (good)
These items are the SAME on all 7 hosts — no drift:
- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task)
- **Debian version:** all on trixie/13
- **tcp_congestion_control:** bbr everywhere
- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected)
- **ethtool:** same version everywhere
- **nfs-common:** same version everywhere
- **tcpdump:** same version everywhere
- **smartmontools:** 7.5 everywhere, smartd active everywhere
- **tuned:** installed everywhere
- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM)
- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module
### Quick severity summary
- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5
- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`)
- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment
- **Investigate:** tsys9 SSH key count (2 keys missing)
---
## Appendix A: Package Presence Matrix
> MISSING = not installed. Version shown = installed version (truncated).
| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 |
| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 |
| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 |
| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — |
| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** |
| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 |
| rsyslog | installed | installed | installed | installed | installed | installed | installed |
| qemu-guest-agent | — | — | — | — | — | — | — |
| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** |
| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** |
| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 |
| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 |
| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 |
| conman | — | — | 0.3.1 | — | — | — | — |
| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — |
| nut-server | 2.8.1 | — | — | — | — | — | — |
| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 |
> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are
> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config.
> conman/ser2net on tsys4 is expected (console server). These are not drift.
## Appendix B: Service State Matrix
| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **inactive** | active | active | active | active | active | active |
| smartd | active | active | active | active | active | active | active |
| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| beszel | — | — | — | — | — | — | — |
> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM
> Day items, not drift — they haven't been deployed yet.
## Appendix C: Sysctl Tuning Matrix
| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr |
| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 |
| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 |
| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 |
| governor | perf | perf | perf | N/A | perf | perf | perf |
> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB),
> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000).
> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection
> window size, and the low backlog values can cause packet drops under load.
> swappiness=1 on tsys4/5 is intentional (storage hosts).
## Appendix D: Tuned Profile Matrix
| Host | Profile | Expected | Match? |
|------|---------|----------|--------|
| tsys1 | virtual-host | virtual-host | ✓ |
| tsys3 | virtual-host | virtual-host | ✓ |
| **tsys4** | **throughput-performance** | **network-throughput** | **✗** |
| tsys5 | network-throughput | network-throughput | ✓ |
| tsys6 | virtual-host | virtual-host | ✓ |
| tsys7 | virtual-host | virtual-host | ✓ |
| tsys9 | virtual-host | virtual-host | ✓ |
## Appendix E: SSH + Security State
### Authorized keys (root)
| Host | Key count | Notes |
|------|-----------|-------|
| tsys1 | 5 | Standard |
| tsys3 | 5 | Standard |
| tsys4 | 5 | Standard |
| tsys5 | 5 | Standard |
| tsys6 | 5 | Standard |
| tsys7 | 5 | Standard |
| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** |
### SSHD config (consistent across fleet)
| Setting | Value | All hosts |
|---------|-------|-----------|
| PermitRootLogin | yes | ✓ (all) |
| PasswordAuthentication | (default — not set) | ✓ (all) |
| KbdInteractiveAuthentication | no | ✓ (all) |
> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's
> KbdInteractiveAuthentication. The PAM approach is consistent.
## Appendix F: Filesystem Mount Options
| Host | root fs mount option | Notes |
|------|---------------------|-------|
| tsys1 | relatime | Default |
| tsys3 | relatime | Default |
| tsys4 | relatime | Default |
| **tsys5** | **noatime** | **Only host with noatime on root** |
| tsys6 | relatime | Default |
| tsys7 | relatime | Default |
| tsys9 | relatime | Default |
> Minor: noatime reduces metadata writes (slight improvement on HDD).
> Storage hosts (tsys4/5) would benefit most from noatime.
## Appendix G: Raw Data Location
All raw drift probe output is stored in `/tmp/<hostname>-drift.txt` on this
workstation (not committed — ephemeral). Re-gather any time with:
```bash
for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do
PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt
done
```
The probe script should be saved to the repo as `perf/scripts/probe-drift.sh`
for reuse. It is read-only and portable.
Reference in New Issue View Git Blame Copy Permalink