PFVCluster/proxmox/docs/DRIFT-2026-07-30.md

Fleet Drift Report

Data gathered: 2026-07-30 ~04:15 UTC (live, via tests/remote.sh from all 7 hosts) Probe script: /tmp/pfv-probe-drift.sh (read-only, all 7 hosts in parallel) Report generated: 2026-07-30

---

Executive Briefing

What needs your decision now

#	Issue	Impact	Recommendation
1	lldpd INACTIVE on tsys1	No LLDP neighbor data from tsys1 — blind spot in network topology map	Enable: systemctl enable --now lldpd
2	tsys9 missing 2 SSH keys	Only 3 authorized_keys vs 5 on every other host — possible access gap	Verify which keys should be present; add missing
3	iperf3 missing on tsys9	Can't run throughput tests from the best compute host	Install: apt install iperf3
4	net-tools missing on tsys1/6/7	ifconfig/netstat unavailable (minor — ip command works)	Install for consistency: apt install net-tools
5	sysstat missing on tsys5	No iostat/mpstat/sar on a storage host	Install: apt install sysstat
6	nvme-cli missing on tsys4/5	Can't query NVMe health/SMART on the storage servers that have NVMe	Install: apt install nvme-cli
7	tsys4 tuning drift	rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput)	Run apply-tunings.sh --apply on tsys4
8	noatime only on tsys5 root	All other hosts use relatime on root fs — minor perf difference	Standardize (noatime preferred for storage hosts)
9	rsyslog inactive fleet-wide	No syslog forwarding/receiving anywhere	Saturday OAM Day: set up centralized syslog
10	snmpd inactive fleet-wide	No SNMP polling on hosts (switches have it)	Saturday OAM Day: wire snmpd for LibreNMS
11	beszel not deployed	No metrics agent on any host yet	Saturday OAM Day: deploy Beszel agent fleet-wide

Consistent across the fleet (good)

These items are the SAME on all 7 hosts — no drift:

• PVE version: 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task)
• Debian version: all on trixie/13
• tcp_congestion_control: bbr everywhere
• CPUs scaling_governor: performance everywhere (except tsys5: no cpufreq driver — Westmere, expected)
• ethtool: same version everywhere
• nfs-common: same version everywhere
• tcpdump: same version everywhere
• smartmontools: 7.5 everywhere, smartd active everywhere
• tuned: installed everywhere
• SSHD config: PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM)
• PasswordAuthentication: not explicitly set (default) — relies on 2FA PAM module

Quick severity summary

• Fix now (5 min each, no risk): lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5
• Fix with tuning pass: tsys4 rmem/wmem/backlog/tuned-profile (via existing apply-tunings.sh --apply)
• Saturday OAM Day: rsyslog, snmpd, beszel fleet-wide deployment
• Investigate: tsys9 SSH key count (2 keys missing)

---

Appendix A: Package Presence Matrix

MISSING = not installed. Version shown = installed version (truncated).

Package	tsys1	tsys3	tsys4	tsys5	tsys6	tsys7	tsys9
lldpd	MISSING	1.0.18	1.0.18	1.0.18	1.0.18	1.0.18	1.0.18
smartmontools	7.5	7.5	7.5	7.5	7.5	7.5	7.5
nfs-common	2.8.3	2.8.3	2.8.3	2.8.3	2.8.3	2.8.3	2.8.3
nfs-kernel-server	—	—	2.8.3	2.8.3	—	—	—
iperf3	3.18	3.18	3.18	3.18	3.18	3.18	MISSING
tcpdump	4.99.5	4.99.5	4.99.5	4.99.5	4.99.5	4.99.5	4.99.5
rsyslog	installed	installed	installed	installed	installed	installed	installed
qemu-guest-agent	—	—	—	—	—	—	—
snmpd	MISSING	MISSING	MISSING	MISSING	MISSING	MISSING	MISSING
net-tools	MISSING	2.10	2.10	2.10	MISSING	MISSING	MISSING
ethtool	6.14.2	6.14.2	6.14.2	6.14.2	6.14.2	6.14.2	6.14.2
sysstat	12.7.5	12.7.5	12.7.5	MISSING	12.7.5	12.7.5	12.7.5
nvme-cli	2.13	2.13	MISSING	MISSING	2.13	2.13	2.13
conman	—	—	0.3.1	—	—	—	—
ser2net	4.6.4	—	4.6.4	—	—	—	—
nut-server	2.8.1	—	—	—	—	—	—
tuned	2.25.1	2.25.1	2.25.1	2.25.1	2.25.1	2.25.1	2.25.1

Note: conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config. conman/ser2net on tsys4 is expected (console server). These are not drift.

Appendix B: Service State Matrix

Service	tsys1	tsys3	tsys4	tsys5	tsys6	tsys7	tsys9
lldpd	inactive	active	active	active	active	active	active
smartd	active	active	active	active	active	active	active
snmpd	inactive	inactive	inactive	inactive	inactive	inactive	inactive
rsyslog	inactive	inactive	inactive	inactive	inactive	inactive	inactive
beszel	—	—	—	—	—	—	—

snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM Day items, not drift — they haven't been deployed yet.

Appendix C: Sysctl Tuning Matrix

Setting	tsys1	tsys3	tsys4	tsys5	tsys6	tsys7	tsys9
rmem_max	128MB	128MB	16MB	128MB	128MB	128MB	128MB
wmem_max	128MB	128MB	16MB	128MB	128MB	128MB	128MB
tcp_congestion	bbr	bbr	bbr	bbr	bbr	bbr	bbr
swappiness	10	10	1	1	10	10	10
tcp_max_syn_backlog	2048	2048	1024	4096	4096	4096	2048
netdev_max_backlog	250000	250000	5000	250000	250000	250000	250000
governor	perf	perf	perf	N/A	perf	perf	perf

tsys4 is the outlier on 4 settings: rmem/wmem (16MB vs 128MB), tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000). These directly affect NFS throughput — the 16MB TCP buffers cap per-connection window size, and the low backlog values can cause packet drops under load. swappiness=1 on tsys4/5 is intentional (storage hosts).

Appendix D: Tuned Profile Matrix

Host	Profile	Expected	Match?
tsys1	virtual-host	virtual-host	✓
tsys3	virtual-host	virtual-host	✓
tsys4	throughput-performance	network-throughput	✗
tsys5	network-throughput	network-throughput	✓
tsys6	virtual-host	virtual-host	✓
tsys7	virtual-host	virtual-host	✓
tsys9	virtual-host	virtual-host	✓

Appendix E: SSH + Security State

Authorized keys (root)

Host	Key count	Notes
tsys1	5	Standard
tsys3	5	Standard
tsys4	5	Standard
tsys5	5	Standard
tsys6	5	Standard
tsys7	5	Standard
tsys9	3	2 fewer keys than rest of fleet — investigate

SSHD config (consistent across fleet)

Setting	Value	All hosts
PermitRootLogin	yes	✓ (all)
PasswordAuthentication	(default — not set)	✓ (all)
KbdInteractiveAuthentication	no	✓ (all)

2FA is enforced via PAM module (secharden-2fa), not via sshd's KbdInteractiveAuthentication. The PAM approach is consistent.

Appendix F: Filesystem Mount Options

Host	root fs mount option	Notes
tsys1	relatime	Default
tsys3	relatime	Default
tsys4	relatime	Default
tsys5	noatime	Only host with noatime on root
tsys6	relatime	Default
tsys7	relatime	Default
tsys9	relatime	Default

Minor: noatime reduces metadata writes (slight improvement on HDD). Storage hosts (tsys4/5) would benefit most from noatime.

Appendix G: Raw Data Location

All raw drift probe output is stored in /tmp/<hostname>-drift.txt on this workstation (not committed — ephemeral). Re-gather any time with:

for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do
  PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt
done

The probe script should be saved to the repo as perf/scripts/probe-drift.sh for reuse. It is read-only and portable.