mrcharles
422999bf3c
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.
Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
distribution, StorageClass design, and ETL/HPC workload planning
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
89 lines
3.8 KiB
Plaintext
89 lines
3.8 KiB
Plaintext
!###############################################################################
|
|
!# pfv-r3-tor-stor — TARGETED LACP hash verification
|
|
!#
|
|
!# Purpose: confirm whether `port-channel load-balance layer-2-3-4` actually
|
|
!# took effect, and gather the LAG state needed to decide whether the LAG
|
|
!# must be bounced.
|
|
!#
|
|
!# Context: tsys6<->tsys7 is capped at exactly single-link speed (~943 Mbps)
|
|
!# with 90K-136K retransmits on multi-stream TCP, but a clean 928 Mbps / 0
|
|
!# retransmits on single-stream. That = offered load above 1 Gbps is being
|
|
!# dropped at a single egress port. We need to know if the switch hash is
|
|
!# really layer-2-3-4, and whether tsys6 + tsys7 ports are correctly
|
|
!# bundled into LACP groups.
|
|
!#
|
|
!# Paste from enable mode. Capture to:
|
|
!# returned-logs/pfv-r3-tor-stor-lacp-verify.log
|
|
!###############################################################################
|
|
|
|
enable
|
|
terminal datadump
|
|
terminal length 0
|
|
|
|
! --- 1. THE KEY COMMAND: what hash is actually active? -----------------------
|
|
! Expected: layer-2-3-4. If it still shows source-mac / dst-mac /
|
|
! src-dst-mac (layer 2 only), the change did NOT take effect.
|
|
show port-channel load-balance
|
|
show port-channel protocol
|
|
|
|
! --- 2. Which LAGs exist and which physical ports are members ----------------
|
|
! Look for the port-channels carrying tsys6 and tsys7. Confirm each has
|
|
! TWO member ports both in "bundled" / "Forwarding" state.
|
|
show port-channel summary
|
|
show port-channel detail
|
|
|
|
! --- 3. LACP actor/partner state per member port ----------------------------
|
|
! Every member port should show: Actor State Flag = AcLg (Active, Aggr),
|
|
! Partner Operational, and a non-zero Partner System ID.
|
|
show lacp
|
|
show lacp neighbor
|
|
|
|
! --- 4. Per-port counters during/after an iperf run -------------------------
|
|
! THIS is the switch-side equivalent of the host RX-counter test.
|
|
! Run an 8-stream iperf tsys7->tsys6 first, then immediately capture
|
|
! these so the byte counters reflect the test. The TWO tsys6-facing
|
|
! ports should show SIMILAR "RX bytes" (switch ingress from tsys6's
|
|
! bond is not the test direction; look at TX-out bytes toward tsys6,
|
|
! i.e. the ports' output counters). Replace Te1/0/x with the real port
|
|
! IDs shown by "show port-channel summary" above:
|
|
show interfaces counters
|
|
show interfaces counters errors
|
|
|
|
! --- 5. Running config line that set the hash (proof it was committed) ------
|
|
show running-config | include port-channel
|
|
|
|
! --- 6. Switch identity (to look up correct hash syntax for this model) -----
|
|
show version
|
|
show system
|
|
|
|
exit
|
|
|
|
!###############################################################################
|
|
!# DECISION TREE after reading the output:
|
|
!#
|
|
!# A) show port-channel load-balance reports layer-2-3-4 AND per-port
|
|
!# counters show traffic split across BOTH tsys6-facing ports:
|
|
!# -> Hash is fine. Re-test iperf; if still ~943 Mbps, the cap is
|
|
!# host-side (receiver RX bridge / IRQ / CPU).
|
|
!#
|
|
!# B) load-balance reports layer-2-3-4 BUT per-port counters show all traffic
|
|
!# on ONE tsys6-facing port:
|
|
!# -> Hash is configured but the LAG needs a bounce to pick it up.
|
|
!# Bounce the relevant port-channel:
|
|
!# config
|
|
!# interface port-channel N
|
|
!# shutdown
|
|
!# no shutdown
|
|
!# end
|
|
!# WARNING: drops all traffic on that LAG for ~2-5 s. Coordinate.
|
|
!#
|
|
!# C) load-balance still shows a layer-2-only policy:
|
|
!# -> The earlier change didn't commit to this context. Re-apply:
|
|
!# config
|
|
!# port-channel load-balance layer-2-3-4
|
|
!# end
|
|
!# show port-channel load-balance
|
|
!# copy running-config startup-config
|
|
!# Then handle as case (B) if counters still aren't split.
|
|
!###############################################################################
|