docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts

Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-28 20:07:09 -05:00
parent 0600321968
commit 28e0b0c7a6
13 changed files with 281 additions and 24 deletions
+1 -1
View File
@@ -3,7 +3,7 @@
set -uo pipefail
HOST="$1"
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts"
echo "=== Uploading fix script to $HOST ==="
scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
+1 -1
View File
@@ -18,7 +18,7 @@
###############################################################################
set -uo pipefail
SCRIPT_DIR="/home/reachableceo/projects/perfopt"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CHECK_SH="$SCRIPT_DIR/scripts/check.sh"
LOG_DIR="$SCRIPT_DIR/returned-logs"
mkdir -p "$LOG_DIR"
+1 -1
View File
@@ -4,7 +4,7 @@
# Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section.
set -uo pipefail
SCRIPT="/home/reachableceo/projects/perfopt/scripts/apply-tunings.sh"
SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts/apply-tunings.sh"
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
MODE=""
EXTRA_FLAGS=""
+1 -1
View File
@@ -22,7 +22,7 @@ if [ -z "$HOST" ]; then
fi
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
SCRIPT_DIR="/home/reachableceo/projects/perfopt/scripts"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts"
echo "==================================================================="
echo " finish-host.sh — $HOST (mode: $MODE)"
+1 -1
View File
@@ -11,7 +11,7 @@
set -uo pipefail
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
mkdir -p "$LOG_DIR"
HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
+1 -1
View File
@@ -17,7 +17,7 @@
set -uo pipefail
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
mkdir -p "$LOG_DIR"
ALL_HOSTS=(pfv-tsys1 pfv-tsys3 pfv-tsys4 pfv-tsys5 pfv-tsys6 pfv-tsys7)
+1 -1
View File
@@ -2,7 +2,7 @@
# iperf-tsys6-tsys7.sh - validate 2Gbps LACP between the two tuned hosts.
set -uo pipefail
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/returned-logs/iperf"
mkdir -p "$LOG_DIR"
TSYS6="10.100.100.6"
+2 -2
View File
@@ -12,9 +12,9 @@ SSH=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 \
SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
RECV="pfv-tsys6"; SEND="pfv-tsys7"
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
LOCAL_LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/returned-logs/iperf"
SCRIPT="lacp-retrans-cause.sh"
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
LOCAL_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/scripts/${SCRIPT}"
REMOTE_SCRIPT="/root/${SCRIPT}"
mkdir -p "$LOCAL_LOG_DIR"
+2 -2
View File
@@ -19,9 +19,9 @@ SCP=(-o BatchMode=yes -o ConnectTimeout=8 -o StrictHostKeyChecking=accept-new)
RECV="pfv-tsys6" # receiver (we care most about its RX split)
SEND="pfv-tsys7" # sender (control: its TX split)
LOCAL_LOG_DIR="/home/reachableceo/projects/perfopt/returned-logs/iperf"
LOCAL_LOG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/returned-logs/iperf"
SCRIPT="lacp-rx-distribution.sh"
LOCAL_SCRIPT="/home/reachableceo/projects/perfopt/scripts/${SCRIPT}"
LOCAL_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/scripts/${SCRIPT}"
REMOTE_SCRIPT="/root/${SCRIPT}"
mkdir -p "$LOCAL_LOG_DIR"
+1 -1
View File
@@ -51,7 +51,7 @@ esac
BAUD=9600
HOST=root@pfv-tsys4
HERE=/home/reachableceo/projects/perfopt
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
LOCAL_DRIVER=$HERE/scripts/sw-capture.py
LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
LOCAL_LOG=$HERE/returned-logs/$SWITCH.log