mrcharles
422999bf3c
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.
Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
distribution, StorageClass design, and ETL/HPC workload planning
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
58 lines
2.8 KiB
Plaintext
58 lines
2.8 KiB
Plaintext
=== LACP retransmit cause investigation (sender) ===
|
|
Host: pfv-tsys7 Role: sender
|
|
Date: 2026-07-27T13:57:55Z
|
|
Bond: bond0
|
|
|
|
--- bond0 hash + driver ---
|
|
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
|
|
Transmit Hash Policy: layer3+4 (1)
|
|
Number of ports: 2
|
|
Partner Mac Address: 00:13:72:46:95:e4
|
|
nic1 driver=tg3 speed=1000 current RX ring=2047
|
|
nic2 driver=tg3 speed=1000 current RX ring=2047
|
|
|
|
|
|
--- iperf3 summary ---
|
|
[SUM] 0.00-12.00 sec 2.37 GBytes 1.69 Gbits/sec 69187 sender
|
|
[SUM] 0.00-12.00 sec 2.36 GBytes 1.69 Gbits/sec receiver
|
|
|
|
--- TCP SNMP deltas (/proc/net/snmp) ---
|
|
ActiveOpens 80334 -> 80371 delta=37
|
|
EstabResets 14 -> 21 delta=7
|
|
InSegs 3887134 -> 4130320 delta=243186
|
|
OutSegs 10112473 -> 12172271 delta=2059798
|
|
RetransSegs 259532 -> 340946 delta=81414
|
|
|
|
--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---
|
|
|
|
--- NIC counter deltas (only non-zero) ---
|
|
[nic1] sysfs multicast 16340 -> 16342 delta=2
|
|
[nic2] sysfs multicast 38862 -> 38872 delta=10
|
|
|
|
--- raw iperf3 tail (last 12 lines) ---
|
|
[ 13] 0.00-12.00 sec 87.9 MBytes 61.4 Mbits/sec 7050 sender
|
|
[ 13] 0.00-12.00 sec 87.2 MBytes 61.0 Mbits/sec receiver
|
|
[ 15] 0.00-12.00 sec 538 MBytes 376 Mbits/sec 0 sender
|
|
[ 15] 0.00-12.00 sec 538 MBytes 376 Mbits/sec receiver
|
|
[ 17] 0.00-12.00 sec 378 MBytes 264 Mbits/sec 19608 sender
|
|
[ 17] 0.00-12.00 sec 378 MBytes 264 Mbits/sec receiver
|
|
[ 19] 0.00-12.00 sec 327 MBytes 229 Mbits/sec 12049 sender
|
|
[ 19] 0.00-12.00 sec 327 MBytes 228 Mbits/sec receiver
|
|
[SUM] 0.00-12.00 sec 2.37 GBytes 1.69 Gbits/sec 69187 sender
|
|
[SUM] 0.00-12.00 sec 2.36 GBytes 1.69 Gbits/sec receiver
|
|
|
|
iperf Done.
|
|
|
|
===================================================================
|
|
HOW TO READ THIS
|
|
===================================================================
|
|
- If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:
|
|
-> ring buffers too small. Fix: ethtool -G $IF rx 4096 (or max).
|
|
- If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:
|
|
-> softirq starvation. Fix: increase net.core.netdev_budget,
|
|
check IRQ affinity, consider RPS.
|
|
- If NEITHER of the above climbed but TCP RetransSegs did:
|
|
-> the retransmits are LACP reordering (out-of-order segments
|
|
triggering fast retransmit), NOT packet loss. NOT fixable.
|
|
===================================================================
|