KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/DRIFT-2026-07-30.md
+9 -174
View File
@@ -1,175 +1,10 @@
# Fleet Drift Report
# proxmox/docs/DRIFT-2026-07-30.md
> **Data gathered:** 2026-07-30 ~04:15 UTC (live, via `tests/remote.sh` from all 7 hosts)
> **Probe script:** `/tmp/pfv-probe-drift.sh` (read-only, all 7 hosts in parallel)
> **Report generated:** 2026-07-30
---
## Executive Briefing
### What needs your decision now
| # | Issue | Impact | Recommendation |
|---|-------|--------|---------------|
| **1** | **lldpd INACTIVE on tsys1** | No LLDP neighbor data from tsys1 — blind spot in network topology map | Enable: `systemctl enable --now lldpd` |
| **2** | **tsys9 missing 2 SSH keys** | Only 3 authorized_keys vs 5 on every other host — possible access gap | Verify which keys should be present; add missing |
| **3** | **iperf3 missing on tsys9** | Can't run throughput tests from the best compute host | Install: `apt install iperf3` |
| **4** | **net-tools missing on tsys1/6/7** | `ifconfig`/`netstat` unavailable (minor — `ip` command works) | Install for consistency: `apt install net-tools` |
| **5** | **sysstat missing on tsys5** | No `iostat`/`mpstat`/`sar` on a storage host | Install: `apt install sysstat` |
| **6** | **nvme-cli missing on tsys4/5** | Can't query NVMe health/SMART on the storage servers that have NVMe | Install: `apt install nvme-cli` |
| **7** | **tsys4 tuning drift** | rmem/wmem=16MB (fleet: 128MB), tcp_max_syn_backlog=1024 (fleet: 2048+), netdev_max_backlog=5000 (fleet: 250000), tuned=throughput-performance (should be network-throughput) | Run `apply-tunings.sh --apply` on tsys4 |
| **8** | **noatime only on tsys5 root** | All other hosts use relatime on root fs — minor perf difference | Standardize (noatime preferred for storage hosts) |
| **9** | **rsyslog inactive fleet-wide** | No syslog forwarding/receiving anywhere | Saturday OAM Day: set up centralized syslog |
| **10** | **snmpd inactive fleet-wide** | No SNMP polling on hosts (switches have it) | Saturday OAM Day: wire snmpd for LibreNMS |
| **11** | **beszel not deployed** | No metrics agent on any host yet | Saturday OAM Day: deploy Beszel agent fleet-wide |
### Consistent across the fleet (good)
These items are the SAME on all 7 hosts — no drift:
- **PVE version:** 9.2.5 / kernel 7.0.14-6-pve (except tsys4: 9.1.5 / 6.17.9 — known, separate upgrade task)
- **Debian version:** all on trixie/13
- **tcp_congestion_control:** bbr everywhere
- **CPUs scaling_governor:** performance everywhere (except tsys5: no cpufreq driver — Westmere, expected)
- **ethtool:** same version everywhere
- **nfs-common:** same version everywhere
- **tcpdump:** same version everywhere
- **smartmontools:** 7.5 everywhere, smartd active everywhere
- **tuned:** installed everywhere
- **SSHD config:** PermitRootLogin=yes, KbdInteractiveAuthentication=no everywhere (pubkey+2FA via PAM)
- **PasswordAuthentication:** not explicitly set (default) — relies on 2FA PAM module
### Quick severity summary
- **Fix now (5 min each, no risk):** lldpd on tsys1, iperf3 on tsys9, sysstat on tsys5, nvme-cli on tsys4/5
- **Fix with tuning pass:** tsys4 rmem/wmem/backlog/tuned-profile (via existing `apply-tunings.sh --apply`)
- **Saturday OAM Day:** rsyslog, snmpd, beszel fleet-wide deployment
- **Investigate:** tsys9 SSH key count (2 keys missing)
---
## Appendix A: Package Presence Matrix
> MISSING = not installed. Version shown = installed version (truncated).
| Package | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **MISSING** | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 | 1.0.18 |
| smartmontools | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 | 7.5 |
| nfs-common | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 | 2.8.3 |
| nfs-kernel-server | — | — | 2.8.3 | 2.8.3 | — | — | — |
| iperf3 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | 3.18 | **MISSING** |
| tcpdump | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 | 4.99.5 |
| rsyslog | installed | installed | installed | installed | installed | installed | installed |
| qemu-guest-agent | — | — | — | — | — | — | — |
| snmpd | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** | **MISSING** |
| net-tools | **MISSING** | 2.10 | 2.10 | 2.10 | **MISSING** | **MISSING** | **MISSING** |
| ethtool | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 | 6.14.2 |
| sysstat | 12.7.5 | 12.7.5 | 12.7.5 | **MISSING** | 12.7.5 | 12.7.5 | 12.7.5 |
| nvme-cli | 2.13 | 2.13 | **MISSING** | **MISSING** | 2.13 | 2.13 | 2.13 |
| conman | — | — | 0.3.1 | — | — | — | — |
| ser2net | 4.6.4 | — | 4.6.4 | — | — | — | — |
| nut-server | 2.8.1 | — | — | — | — | — | — |
| tuned | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 | 2.25.1 |
> **Note:** conman on tsys1, ser2net on tsys1, and nut-server on tsys1 are
> expected — tsys1 hosts the UPS (NUT) and has ser2net from a previous config.
> conman/ser2net on tsys4 is expected (console server). These are not drift.
## Appendix B: Service State Matrix
| Service | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| lldpd | **inactive** | active | active | active | active | active | active |
| smartd | active | active | active | active | active | active | active |
| snmpd | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| rsyslog | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** | **inactive** |
| beszel | — | — | — | — | — | — | — |
> snmpd, rsyslog, and beszel are inactive on ALL hosts. These are Saturday OAM
> Day items, not drift — they haven't been deployed yet.
## Appendix C: Sysctl Tuning Matrix
| Setting | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|---------|-------|-------|-------|-------|-------|-------|-------|
| rmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| wmem_max | 128MB | 128MB | **16MB** | 128MB | 128MB | 128MB | 128MB |
| tcp_congestion | bbr | bbr | bbr | bbr | bbr | bbr | bbr |
| swappiness | 10 | 10 | 1 | 1 | 10 | 10 | 10 |
| tcp_max_syn_backlog | 2048 | 2048 | **1024** | 4096 | 4096 | 4096 | 2048 |
| netdev_max_backlog | 250000 | 250000 | **5000** | 250000 | 250000 | 250000 | 250000 |
| governor | perf | perf | perf | N/A | perf | perf | perf |
> **tsys4 is the outlier** on 4 settings: rmem/wmem (16MB vs 128MB),
> tcp_max_syn_backlog (1024 vs 2048+), netdev_max_backlog (5000 vs 250000).
> These directly affect NFS throughput — the 16MB TCP buffers cap per-connection
> window size, and the low backlog values can cause packet drops under load.
> swappiness=1 on tsys4/5 is intentional (storage hosts).
## Appendix D: Tuned Profile Matrix
| Host | Profile | Expected | Match? |
|------|---------|----------|--------|
| tsys1 | virtual-host | virtual-host | ✓ |
| tsys3 | virtual-host | virtual-host | ✓ |
| **tsys4** | **throughput-performance** | **network-throughput** | **✗** |
| tsys5 | network-throughput | network-throughput | ✓ |
| tsys6 | virtual-host | virtual-host | ✓ |
| tsys7 | virtual-host | virtual-host | ✓ |
| tsys9 | virtual-host | virtual-host | ✓ |
## Appendix E: SSH + Security State
### Authorized keys (root)
| Host | Key count | Notes |
|------|-----------|-------|
| tsys1 | 5 | Standard |
| tsys3 | 5 | Standard |
| tsys4 | 5 | Standard |
| tsys5 | 5 | Standard |
| tsys6 | 5 | Standard |
| tsys7 | 5 | Standard |
| **tsys9** | **3** | **2 fewer keys than rest of fleet — investigate** |
### SSHD config (consistent across fleet)
| Setting | Value | All hosts |
|---------|-------|-----------|
| PermitRootLogin | yes | ✓ (all) |
| PasswordAuthentication | (default — not set) | ✓ (all) |
| KbdInteractiveAuthentication | no | ✓ (all) |
> 2FA is enforced via PAM module (`secharden-2fa`), not via sshd's
> KbdInteractiveAuthentication. The PAM approach is consistent.
## Appendix F: Filesystem Mount Options
| Host | root fs mount option | Notes |
|------|---------------------|-------|
| tsys1 | relatime | Default |
| tsys3 | relatime | Default |
| tsys4 | relatime | Default |
| **tsys5** | **noatime** | **Only host with noatime on root** |
| tsys6 | relatime | Default |
| tsys7 | relatime | Default |
| tsys9 | relatime | Default |
> Minor: noatime reduces metadata writes (slight improvement on HDD).
> Storage hosts (tsys4/5) would benefit most from noatime.
## Appendix G: Raw Data Location
All raw drift probe output is stored in `/tmp/<hostname>-drift.txt` on this
workstation (not committed — ephemeral). Re-gather any time with:
```bash
for h in pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7 pfv-tsys9; do
PROX_HOST=$h bash tests/remote.sh prox-file /tmp/pfv-probe-drift.sh > /tmp/$h-drift.txt
done
```
The probe script should be saved to the repo as `perf/scripts/probe-drift.sh`
for reuse. It is read-only and portable.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Fleet drift report 2026-07-30**
>
> **Read it here:** https://community.turnsys.com/t/298
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*