mrcharles
21adb89d4e
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM, CPU, and storage for all 7 reachable hosts. Written to docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in PROJECT.md sections 4-8. Key audit findings: - CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4 storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These migrations were the #1 recommendation from the previous audit and have not been done. - CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS. tsys4 failure = etcd quorum lost. - 59% of running VMs still on tsys4 storage (improved from 68%). - cnode VMIDs have changed since PROJECT.md was written (cnode1 is now VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.) Gardening fixes: - Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already exist for this purpose) - Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/ scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md self-locating scripts convention) - Updated STATUS.md Known Issues with the two critical findings - Updated STATUS.md Pending with prioritized pre-k8s action items - Registered AUDIT-2026-07-28.md in docmap.md 💘 Generated with Crush Assisted-by: Crush:glm-5.2
52 lines
2.0 KiB
Bash
Executable File
52 lines
2.0 KiB
Bash
Executable File
#!/bin/bash
|
|
# deploy-and-fix.sh - uploads fix script, runs it, starts VMs, verifies.
|
|
set -uo pipefail
|
|
HOST="$1"
|
|
SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new)
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts"
|
|
|
|
echo "=== Uploading fix script to $HOST ==="
|
|
scp "${SSH_OPTS[@]}" "$SCRIPT_DIR/fix-bond-nfs.sh" "root@$HOST:/root/fix-bond-nfs.sh" >/dev/null 2>&1
|
|
|
|
echo "=== Running fix ==="
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'chmod +x /root/fix-bond-nfs.sh && bash /root/fix-bond-nfs.sh' 2>&1
|
|
|
|
echo ""
|
|
echo "=== Starting VMs ==="
|
|
for vmid in $(ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list 2>/dev/null | awk "NR>1{print \$1}"'); do
|
|
status=$(ssh "${SSH_OPTS[@]}" "root@$HOST" "qm status $vmid 2>/dev/null | awk '{print \$2}'")
|
|
if [ "$status" != "running" ]; then
|
|
echo " Starting VM $vmid..."
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" "qm start $vmid" 2>&1 | sed 's/^/ /'
|
|
else
|
|
echo " VM $vmid already running"
|
|
fi
|
|
done
|
|
|
|
echo ""
|
|
echo "Waiting 20s for VMs to boot..."
|
|
sleep 20
|
|
|
|
echo ""
|
|
echo "=== FULL VERIFICATION ==="
|
|
echo ""
|
|
echo "--- VMs ---"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'qm list'
|
|
echo ""
|
|
echo "--- NFS mounts ---"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'nfsstat -m 2>/dev/null | head -24'
|
|
echo ""
|
|
echo "--- NFS TCP connections ---"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null'
|
|
echo " Count:"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'ss -tn state established "( dport = :2049 )" 2>/dev/null | tail -n +2 | wc -l'
|
|
echo ""
|
|
echo "--- bond0 hash policy ---"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'cat /proc/net/bonding/bond0 | head -6'
|
|
echo ""
|
|
echo "--- Summary ---"
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "tcp_cc: $(sysctl -n net.ipv4.tcp_congestion_control)"'
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "swappiness: $(sysctl -n vm.swappiness)"'
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'echo "governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null)"'
|
|
ssh "${SSH_OPTS[@]}" "root@$HOST" 'tuned-adm active 2>/dev/null'
|