chore: initialize repo with full project state

Performance optimization engagement for a 7-host Proxmox R&D cluster. Captures the accumulated work across host tuning, network analysis, fleet assessment, and kubernetes architecture planning. Contents: - Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS nconnect, tuned profiles -- complete on 5 of 7 hosts - Validation + benchmarking scripts: iperf matrix, bond/NFS fixes - Collected host data (returned-logs/): check.sh output from all 7 hosts + iperf results, including newly-validated pfv-tsys9 - AGENTS.md: operating context for AI agents - PROJECT.md: board-ready fleet assessment with VM placement and storage redundancy analysis (40 VMs across 7 hosts) - K8S.md: kubernetes architecture deep-dive covering cnode/wnode distribution, StorageClass design, and ETL/HPC workload planning 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-07-27 11:31:29 -05:00
commit 422999bf3c
106 changed files with 114159 additions and 0 deletions
@@ -0,0 +1,57 @@
+=== LACP retransmit cause investigation (sender) ===
+Host: pfv-tsys7   Role: sender
+Date: 2026-07-27T13:57:55Z
+Bond: bond0
+
+--- bond0 hash + driver ---
+Bonding Mode: IEEE 802.3ad Dynamic link aggregation
+Transmit Hash Policy: layer3+4 (1)
+	Number of ports: 2
+	Partner Mac Address: 00:13:72:46:95:e4
+  nic1     driver=tg3                  speed=1000   current RX ring=2047
+  nic2     driver=tg3                  speed=1000   current RX ring=2047
+
+
+--- iperf3 summary ---
+[SUM]   0.00-12.00  sec  2.37 GBytes  1.69 Gbits/sec  69187             sender
+[SUM]   0.00-12.00  sec  2.36 GBytes  1.69 Gbits/sec                  receiver
+
+--- TCP SNMP deltas (/proc/net/snmp) ---
+  ActiveOpens                   80334 ->        80371  delta=37
+  EstabResets                      14 ->           21  delta=7
+  InSegs                      3887134 ->      4130320  delta=243186
+  OutSegs                    10112473 ->     12172271  delta=2059798
+  RetransSegs                  259532 ->       340946  delta=81414
+
+--- softnet_stat deltas (look for non-zero dropped_delta/squeezed_delta) ---
+
+--- NIC counter deltas (only non-zero) ---
+  [nic1]   sysfs multicast                       16340 ->        16342  delta=2
+  [nic2]   sysfs multicast                       38862 ->        38872  delta=10
+
+--- raw iperf3 tail (last 12 lines) ---
+[ 13]   0.00-12.00  sec  87.9 MBytes  61.4 Mbits/sec  7050            sender
+[ 13]   0.00-12.00  sec  87.2 MBytes  61.0 Mbits/sec                  receiver
+[ 15]   0.00-12.00  sec   538 MBytes   376 Mbits/sec    0            sender
+[ 15]   0.00-12.00  sec   538 MBytes   376 Mbits/sec                  receiver
+[ 17]   0.00-12.00  sec   378 MBytes   264 Mbits/sec  19608            sender
+[ 17]   0.00-12.00  sec   378 MBytes   264 Mbits/sec                  receiver
+[ 19]   0.00-12.00  sec   327 MBytes   229 Mbits/sec  12049            sender
+[ 19]   0.00-12.00  sec   327 MBytes   228 Mbits/sec                  receiver
+[SUM]   0.00-12.00  sec  2.37 GBytes  1.69 Gbits/sec  69187             sender
+[SUM]   0.00-12.00  sec  2.36 GBytes  1.69 Gbits/sec                  receiver
+
+iperf Done.
+
+===================================================================
+ HOW TO READ THIS
+===================================================================
+ - If NIC rx_dropped / rx_missed_errors / rx_fifo_errors climbed:
+     -> ring buffers too small. Fix: ethtool -G $IF rx 4096 (or max).
+ - If softnet_stat 'dropped' or 'squeezed' climbed on any CPU:
+     -> softirq starvation. Fix: increase net.core.netdev_budget,
+        check IRQ affinity, consider RPS.
+ - If NEITHER of the above climbed but TCP RetransSegs did:
+     -> the retransmits are LACP reordering (out-of-order segments
+        triggering fast retransmit), NOT packet loss. NOT fixable.
+===================================================================