mrcharles
422999bf3c
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.
Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
distribution, StorageClass design, and ETL/HPC workload planning
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
64 lines
3.0 KiB
Plaintext
64 lines
3.0 KiB
Plaintext
=== LACP retransmit cause investigation (receiver) ===
|
|
Host: pfv-tsys6 Role: receiver
|
|
Date: 2026-07-27T13:57:50Z
|
|
Bond: bond0
|
|
|
|
--- bond0 hash + driver ---
|
|
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
|
|
Transmit Hash Policy: layer3+4 (1)
|
|
Number of ports: 2
|
|
Partner Mac Address: 00:13:72:46:95:e4
|
|
nic1 driver=bnx2 speed=1000 current RX ring=2040
|
|
nic2 driver=bnx2 speed=1000 current RX ring=2040
|
|
|
|
--- BEFORE snapshots ---
|
|
[receiver] starting iperf3 -s -1 on 10.100.100.6 ...
|
|
--- AFTER snapshots ---
|
|
|
|
--- iperf3 summary ---
|
|
[SUM] 0.00-12.00 sec 2.36 GBytes 1.69 Gbits/sec receiver
|
|
|
|
--- TCP SNMP deltas (/proc/net/snmp) ---
|
|
ActiveOpens 82825 -> 82853 delta=28
|
|
PassiveOpens 12139 -> 12148 delta=9
|
|
EstabResets 3 -> 4 delta=1
|
|
InSegs 4295069 -> 4586141 delta=291072
|
|
OutSegs 9518841 -> 9761558 delta=242717
|
|
RetransSegs 68841 -> 68886 delta=45
|
|
OutRsts 1605 -> 1661 delta=56
|
|
|
|
--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---
|
|
|
|
--- NIC counter deltas (only non-zero) ---
|
|
[nic1] sysfs multicast 38589 -> 38603 delta=14
|
|
[nic2] sysfs multicast 16188 -> 16194 delta=6
|
|
|
|
--- raw iperf3 tail (last 12 lines) ---
|
|
[SUM] 12.00-12.00 sec 640 KBytes 1.69 Gbits/sec
|
|
- - - - - - - - - - - - - - - - - - - - - - - - -
|
|
[ ID] Interval Transfer Bitrate
|
|
[ 5] 0.00-12.00 sec 538 MBytes 376 Mbits/sec receiver
|
|
[ 8] 0.00-12.00 sec 81.5 MBytes 57.0 Mbits/sec receiver
|
|
[ 10] 0.00-12.00 sec 69.4 MBytes 48.5 Mbits/sec receiver
|
|
[ 12] 0.00-12.00 sec 402 MBytes 281 Mbits/sec receiver
|
|
[ 14] 0.00-12.00 sec 87.2 MBytes 61.0 Mbits/sec receiver
|
|
[ 16] 0.00-12.00 sec 538 MBytes 376 Mbits/sec receiver
|
|
[ 18] 0.00-12.00 sec 378 MBytes 264 Mbits/sec receiver
|
|
[ 20] 0.00-12.00 sec 327 MBytes 228 Mbits/sec receiver
|
|
[SUM] 0.00-12.00 sec 2.36 GBytes 1.69 Gbits/sec receiver
|
|
|
|
===================================================================
|
|
HOW TO READ THIS
|
|
===================================================================
|
|
- If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:
|
|
-> ring buffers too small. Fix: ethtool -G $IF rx 4096 (or max).
|
|
- If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:
|
|
-> softirq starvation. Fix: increase net.core.netdev_budget,
|
|
check IRQ affinity, consider RPS.
|
|
- If NEITHER of the above climbed but TCP RetransSegs did:
|
|
-> the retransmits are LACP reordering (out-of-order segments
|
|
triggering fast retransmit), NOT packet loss. NOT fixable.
|
|
===================================================================
|
|
|
|
Log: /root/lacp-retrans-receiver.log
|