Compare commits
4
Commits
48cdd81779
...
24b08aa592
| Author | SHA1 | Date | |
|---|---|---|---|
 mrcharles
|
24b08aa592 | ||
|
mrcharles
|
26de312c42 | ||
|
mrcharles
|
d9b32e4eef | ||
|
mrcharles
|
e7312ad460 |
@@ -99,6 +99,10 @@ vendor/ Vendored KNELShellFramework
|
||||
| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
|
||||
| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
|
||||
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
|
||||
| [`perf/scripts/probe-storage.sh`](perf/scripts/probe-storage.sh) | Read-only disk/mount/export/SMART audit |
|
||||
| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit |
|
||||
| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) |
|
||||
| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) |
|
||||
|
||||
## Key Docs
|
||||
|
||||
|
||||
@@ -3,12 +3,14 @@
|
||||
> **Human read-only. Agents maintain this file automatically after each work
|
||||
> session.** Do not edit by hand — the next agent run will overwrite it.
|
||||
|
||||
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — HA NUT integration live
|
||||
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute)
|
||||
>
|
||||
> **Capacity targets are now codified** in
|
||||
> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md)
|
||||
> — network topology, switch LAG configs, corrected storage topology, VM config
|
||||
> audit, fleet consistency, cross-rack NFS root-cause analysis.
|
||||
> Capacity targets are codified in
|
||||
> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
|
||||
> steady-state, ~50% idle CPU. The current placement analysis lives in
|
||||
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md).
|
||||
> steady-state, ~50% idle CPU.
|
||||
|
||||
## Current State: STABLE
|
||||
|
||||
@@ -36,15 +38,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
|
||||
### Proxmox Cluster Ops
|
||||
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
|
||||
- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
|
||||
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73%
|
||||
- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
|
||||
- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg
|
||||
- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
|
||||
- [x] **tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator** (verified: 8/8 NFS stores active on both)
|
||||
- [x] **Full ground-truth audit 2026-07-30** (network + storage + compute + switches)
|
||||
- [x] **Cross-rack NFS root cause identified:** core-sw01 LAG hash=layer-2-3 (no L4),
|
||||
g16/g17 cable mismatch (3 of 4 links active), static mode (no LACP)
|
||||
- [x] **tsys4/tsys5 storage changes documented:** new NVMe (CT500P1 500GB), relocated
|
||||
SK hynix SSD, D3 changed to USB HDD, /mnt/albert + /mnt/backup added
|
||||
- [x] **Console symlink bug fixed:** udev rule singular→plural mismatch broke all 7
|
||||
console ports after reboot
|
||||
- [x] **Portable audit tooling built:** probe-storage.sh, probe-network.sh,
|
||||
conman-console.py, snmp-switch-audit.py (AGPLv3-friendly, config-driven)
|
||||
- [ ] **Memory goal (≤80% actual): PASS fleet-wide** — highest is tsys9 at 69%
|
||||
- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
|
||||
- [ ] tsys4: blocked on PCIe NIC + RAM install
|
||||
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
|
||||
- [ ] tsys4: PVE 9.1.5 (behind fleet 9.2.5), needs upgrade
|
||||
- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
|
||||
- [ ] tsys2: pending rebuild from Win10 to Proxmox
|
||||
|
||||
### Kubernetes Control Plane (k3s HA — LIVE)
|
||||
@@ -131,50 +137,55 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
|
||||
| Issue | Impact | Status |
|
||||
|-------|--------|--------|
|
||||
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
|
||||
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 73%/37%/62% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §11 |
|
||||
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
|
||||
| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ |
|
||||
| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
|
||||
| **D2 IOPS hotspot** | 17 of 26 NFS-backed VMs on one spindle; S1/S4 underused | Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11) |
|
||||
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
|
||||
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
|
||||
| **core-sw01 LAG hash = layer-2-3** | NFS cross-rack capped at 1 Gbps per host pair | Fix: change to layer-2-3-4 (switch change, pending approval) |
|
||||
| **core-sw01 ch1 g16/g17 mismatch** | Only 3 of 4 cross-rack links active | Fix: change member list g(13-15,17)→g(13-16) |
|
||||
| **Cross-rack trunk uses `mode on`** | No LACP failure detection | Fix: change to `mode active` (both switches) |
|
||||
| **tsys5 bond0 degraded** (1 of 2 slaves) | Single link, no redundancy | Physical: 2nd ethernet cable needed |
|
||||
| **tsys5 NVMe + SSD not configured** | Best local storage unused | Config: mount + add to Proxmox storage.cfg |
|
||||
| **D2 carries 13 running VMs** | IOPS hotspot persists | Migrate to S2/S3/new disks |
|
||||
| **tsys4 PVE 9.1.5** | Behind fleet (9.2.5) | Upgrade needed |
|
||||
| **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Offload to new SK hynix SSD |
|
||||
| **Port errors on core-sw01** | g9:2902, g38:590, g26:73 inErr | Investigate CRC/cable |
|
||||
| **tsys5 is single-CPU** | CPU2 socket unpopulated | Consider seating spare LGA771 Xeon |
|
||||
| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; grow via workloads |
|
||||
|
||||
## Pending (next session priorities)
|
||||
|
||||
> **Full phased action checklist with current/target inventory is in**
|
||||
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §11.
|
||||
> Summary below. SSD/NVMe tiering is deferred to post-Friday.
|
||||
> **Full audit with findings + recommendations is in**
|
||||
> [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md).
|
||||
|
||||
**Phase 0 — Storage config fixes — DONE ✓** (tsys6 S1 typo, tsys9 S1 missing)
|
||||
**Awaiting operator review of AUDIT-2026-07-30 before executing any fixes.**
|
||||
|
||||
**Phase 1 — Cleanup:**
|
||||
- Delete cnode4 (601) on tsys6 (stopped, stale)
|
||||
- cnode5 (706) already deleted ✓
|
||||
**Switch changes (need approval):**
|
||||
- Fix core-sw01 ch1 member: g(13-15,17) → g(13-16) [restores 4th link]
|
||||
- Fix core-sw01 LAG hash: layer-2-3 → layer-2-3-4 [unlocks multi-link NFS]
|
||||
- Convert trunk to LACP: mode on → mode active [failure detection]
|
||||
- Remove stale tsys8 config from tor3-stor (ch4/g13/g14)
|
||||
|
||||
**Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):**
|
||||
- wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4
|
||||
- rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2
|
||||
|
||||
**Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):**
|
||||
- tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117)
|
||||
- tsys9→tsys6: tsys-siem(901)
|
||||
|
||||
**Phase 4 — Grow wnode-tsys9:** 4c/4GB → 6c/8GB (after siem leaves tsys9)
|
||||
|
||||
**Phase 5 — Join wnodes to k3s** (after migrations free resources)
|
||||
**Host-side changes (no switch access needed):**
|
||||
- Mount + configure tsys5 NVMe (CT500P1 500GB) as LVM-thin
|
||||
- Mount + configure tsys5 SK hynix SSD (512GB, relocated from tsys4)
|
||||
- Clean stale S3/S4 exports + storage.cfg on tsys5
|
||||
- Upgrade tsys4 to PVE 9.2.5 / kernel 7.0.14
|
||||
- Apply apply-tunings.sh to tsys4/5 (nconnect, bond hash, rmem/wmem)
|
||||
- Audit VM cache modes (writethrough → writeback for NFS)
|
||||
|
||||
**Other:**
|
||||
- **Friday maintenance:** tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet)
|
||||
- **PDU:** Rename outlets in powerman.conf (Friday onsite)
|
||||
- **k8s deferred:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out
|
||||
- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. HA NUT integration live (battery charge, status sensors). No ups.load over USB HID (firmware limitation, confirmed via apcupsd test). Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port).
|
||||
- **Saturday OAM Day (2026-08-01):** 100% instrumentation coverage — Beszel
|
||||
(metrics) on all hosts, Uptime Kuma (up/down) → Pushover alerts, LibreNMS
|
||||
(SNMP) for switches, Home Assistant (power/temp) for physical infra,
|
||||
centralized syslog (tool TBD)
|
||||
- Investigate core-sw01 port errors (g9/g38/g26)
|
||||
- k8s: join wnodes, label by workload class
|
||||
- k8s deferred: ETL tooling, HPC scheduler, vcluster policy
|
||||
|
||||
## Infrastructure Summary
|
||||
|
||||
| Component | Details |
|
||||
|-----------|---------|
|
||||
| Proxmox hosts | 7 standalone PVE, managed via PDM |
|
||||
| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) |
|
||||
| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + new NVMe unconfigured) |
|
||||
| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3), + 1 unidentified (192.168.0.8) |
|
||||
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
|
||||
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
|
||||
| DNS production | tailscale-router (read-only source of truth) |
|
||||
|
||||
@@ -121,7 +121,7 @@ for entry in "${ENTRIES[@]}"; do
|
||||
{
|
||||
echo ""
|
||||
echo "# $name (TCP $tcp_port): $comment"
|
||||
echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"console/$name\""
|
||||
echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"consoles/$name\""
|
||||
} >> "$UDEV_RULES"
|
||||
done
|
||||
|
||||
|
||||
+3
-2
@@ -3,7 +3,7 @@
|
||||
> **Index of all documentation in this repo.** Agents must update this file
|
||||
> whenever a doc is added, removed, or substantively changed.
|
||||
|
||||
> **Last updated:** 2026-07-30 (UPS/NUT)
|
||||
> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute)
|
||||
|
||||
## Kubernetes Architecture ([`k8s/`](k8s/))
|
||||
|
||||
@@ -24,7 +24,8 @@ Fleet operations, hardware, performance tuning, storage architecture.
|
||||
|----------|-------------|---------------|
|
||||
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current & only audit.** Consolidated RAM/CPU + HDD disk placement with corrected storage topology, current/target inventory, action checklist | 2026-07-29 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 |
|
||||
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
# read-only, and pulls the resulting log back to returned-logs/.
|
||||
#
|
||||
# EXPLICITLY SKIPS:
|
||||
# - pfv-tsys2 (off the air per user)
|
||||
# - pfv-tsys9 (off the air per user; also not in original inventory)
|
||||
# - pfv-tsys2 (off the air per user; Win10 pending rebuild)
|
||||
# - pfv-tsys8 (retired / permanently offline per user)
|
||||
#
|
||||
# Safety features:
|
||||
# - BatchMode=yes : never hang on a password prompt
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
conman-console.py — Drive a serial console via a conman server, read-only.
|
||||
|
||||
Connects to a conmand server (RFC 127-style multiplexer) over the network,
|
||||
opens a named console, sends the commands from a .cmds file, captures all
|
||||
output, and disconnects. Requires no expect/tcl — pure stdlib PTY.
|
||||
|
||||
This replaces the old sw-capture-remote.sh workflow that killed the serial
|
||||
device holder (conflicting with conman/ser2net). Instead, it talks to conman
|
||||
over TCP, which multiplexes safely with other sessions.
|
||||
|
||||
All endpoints are configurable via environment variables so this works on
|
||||
any network with a conman server:
|
||||
|
||||
CONMAN_SERVER conman server host:port (default: via CONSOLE_HOST)
|
||||
CONSOLE console name to open (required)
|
||||
CMDS_FILE file of commands to send (required)
|
||||
TIMEOUT overall timeout in seconds (default: 45)
|
||||
CMD_DELAY seconds between commands (default: 3)
|
||||
WAKE_DELAY seconds after connect (default: 2)
|
||||
|
||||
Usage:
|
||||
CONMAN_SERVER=console-host:7890 \\
|
||||
python3 conman-console.py --console pfv-core-sw01 --cmds switches/pfv-core-sw01.cmds
|
||||
|
||||
Lines starting with '!' or '#' in the cmds file are comments (skipped).
|
||||
Blank lines are skipped. The conman escape sequence (&.) is sent automatically
|
||||
to disconnect. A password prompt aborts immediately (we never send creds).
|
||||
|
||||
Exit codes:
|
||||
0 clean run
|
||||
1 usage / setup error
|
||||
2 could not connect to conman server
|
||||
3 timeout (partial output still printed)
|
||||
4 password prompt encountered (aborted)
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import pty
|
||||
import re
|
||||
import select
|
||||
import sys
|
||||
import time
|
||||
|
||||
PWD_RE = re.compile(rb"[Pp]assword:\s*$")
|
||||
MORE_RE = re.compile(rb"--\s*[Mm]ore\s*--|[Mm]ore:\s*<space>")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Drive a conman console session read-only via PTY")
|
||||
ap.add_argument("--console", required=True,
|
||||
help="console name (e.g. pfv-core-sw01)")
|
||||
ap.add_argument("--cmds", required=True,
|
||||
help="command file (one command per line; !/# = comment)")
|
||||
ap.add_argument("--server",
|
||||
default=os.environ.get("CONMAN_SERVER", ""),
|
||||
help="conman server host:port (env: CONMAN_SERVER)")
|
||||
ap.add_argument("--timeout", type=int,
|
||||
default=int(os.environ.get("TIMEOUT", "45")),
|
||||
help="overall timeout seconds (env: TIMEOUT)")
|
||||
ap.add_argument("--cmd-delay", type=float,
|
||||
default=float(os.environ.get("CMD_DELAY", "3")),
|
||||
help="seconds between commands (env: CMD_DELAY)")
|
||||
ap.add_argument("--wake-delay", type=float,
|
||||
default=float(os.environ.get("WAKE_DELAY", "2")),
|
||||
help="seconds after connect before first command (env: WAKE_DELAY)")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not args.server:
|
||||
sys.stderr.write("ERROR: --server or CONMAN_SERVER env required\n")
|
||||
return 1
|
||||
|
||||
with open(args.cmds) as f:
|
||||
cmds = [l.strip() for l in f
|
||||
if l.strip() and not l.strip().startswith(("!", "#"))]
|
||||
|
||||
pid, fd = pty.fork()
|
||||
if pid == 0:
|
||||
try:
|
||||
os.execvp("conman",
|
||||
["conman", "-d", args.server, "-f", args.console])
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"ERROR: cannot exec conman: {e}\n")
|
||||
os._exit(2)
|
||||
os._exit(2)
|
||||
|
||||
output = b""
|
||||
cmd_queue = list(cmds)
|
||||
sent_disconnect = False
|
||||
start = time.time()
|
||||
last_action = 0.0
|
||||
phase = "connect"
|
||||
|
||||
while time.time() - start < args.timeout:
|
||||
ready, _, _ = select.select([fd], [], [], 0.5)
|
||||
if ready:
|
||||
try:
|
||||
data = os.read(fd, 8192)
|
||||
except OSError:
|
||||
break
|
||||
if not data:
|
||||
break
|
||||
output += data
|
||||
|
||||
if PWD_RE.search(output.split(b"\n")[-1] if output else b""):
|
||||
sys.stderr.write("[ABORT] password prompt detected — "
|
||||
"never sending credentials\n")
|
||||
os.write(fd, b"&.\n")
|
||||
break
|
||||
|
||||
# Handle pagination: send space to continue
|
||||
if MORE_RE.search(output[-200:] if output else b""):
|
||||
os.write(fd, b" ")
|
||||
time.sleep(0.5)
|
||||
|
||||
elapsed = time.time() - start
|
||||
gap = elapsed - last_action
|
||||
|
||||
if phase == "connect" and gap >= args.wake_delay:
|
||||
os.write(fd, b"\n")
|
||||
phase = "send"
|
||||
last_action = elapsed
|
||||
elif phase == "send" and gap >= args.cmd_delay:
|
||||
if cmd_queue:
|
||||
cmd = cmd_queue.pop(0)
|
||||
os.write(fd, (cmd + "\n").encode())
|
||||
last_action = elapsed
|
||||
else:
|
||||
phase = "drain"
|
||||
last_action = elapsed
|
||||
elif phase == "drain" and gap >= args.cmd_delay:
|
||||
os.write(fd, b"&.\n")
|
||||
sent_disconnect = True
|
||||
phase = "done"
|
||||
last_action = elapsed
|
||||
elif phase == "done" and gap >= 2:
|
||||
break
|
||||
|
||||
if not sent_disconnect:
|
||||
try:
|
||||
os.write(fd, b"&.\n")
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
try:
|
||||
os.close(fd)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.waitpid(pid, 0)
|
||||
except ChildProcessError:
|
||||
pass
|
||||
|
||||
sys.stdout.buffer.write(output)
|
||||
sys.stdout.flush()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# probe-network.sh
|
||||
#
|
||||
# READ-ONLY network + NFS ground-truth probe. Writes only stdout.
|
||||
# Run on any Proxmox host to inventory its NICs, bonds, LLDP neighbors,
|
||||
# NFS client mounts (including nconnect), ethtool link state, and error
|
||||
# counters. No hardcoded values — fully portable.
|
||||
#
|
||||
# Usage (via tests/remote.sh):
|
||||
# PROX_HOST=pfv-tsys6 bash tests/remote.sh prox-file perf/scripts/probe-network.sh
|
||||
###############################################################################
|
||||
set -u
|
||||
echo "===== HOST: $(hostname -s) $(date -u +%FT%TZ) ====="
|
||||
echo
|
||||
echo "##### ip -br link #####"
|
||||
ip -br link 2>&1
|
||||
echo
|
||||
echo "##### ip -br addr #####"
|
||||
ip -br addr 2>&1
|
||||
echo
|
||||
echo "##### /etc/network/interfaces #####"
|
||||
cat /etc/network/interfaces 2>&1
|
||||
echo
|
||||
echo "##### bond0 state (if present) #####"
|
||||
if [ -r /proc/net/bonding/bond0 ]; then
|
||||
cat /proc/net/bonding/bond0 2>&1
|
||||
else
|
||||
echo "(no bond0)"
|
||||
fi
|
||||
echo
|
||||
echo "##### ethtool per physical NIC #####"
|
||||
for nic in /sys/class/net/*; do
|
||||
nic=$(basename "$nic")
|
||||
case "$nic" in lo|bond*|br*|venet*|veth*|docker*|tap*|vnet*|fw*) continue;; esac
|
||||
echo "--- ethtool $nic ---"
|
||||
ethtool "$nic" 2>&1 | grep -iE 'Speed|Duplex|Port|Link|Supported link modes|Advertising|Auto-neg|Settings' || echo "(ethtool failed for $nic)"
|
||||
done
|
||||
echo
|
||||
echo "##### lldpcli (if installed) #####"
|
||||
if command -v lldpcli >/dev/null 2>&1; then
|
||||
echo "--- lldpcli show neighbors ---"
|
||||
lldpcli show neighbors 2>&1
|
||||
echo
|
||||
echo "--- lldpcli show interfaces ---"
|
||||
lldpcli show interfaces 2>&1
|
||||
echo
|
||||
echo "--- lldpcli show chassis ---"
|
||||
lldpcli show chassis 2>&1
|
||||
else
|
||||
echo "(lldpcli not installed)"
|
||||
fi
|
||||
echo
|
||||
echo "##### lldpd / lldpad service #####"
|
||||
systemctl is-active lldpd 2>&1 || true
|
||||
systemctl is-enabled lldpd 2>&1 || true
|
||||
echo
|
||||
echo "##### NFS mounts (mount | grep nfs) #####"
|
||||
mount | grep -i nfs 2>&1 || echo "(no nfs mounts)"
|
||||
echo
|
||||
echo "##### mount nconnect detail (nfsstat -m) #####"
|
||||
nfsstat -m 2>&1
|
||||
echo
|
||||
echo "##### storage.cfg NFS stanzas (options) #####"
|
||||
grep -A3 '^nfs:' /etc/pve/storage.cfg 2>&1
|
||||
echo
|
||||
echo "##### ip route #####"
|
||||
ip route 2>&1
|
||||
echo
|
||||
echo "##### ethtool -S bond slaves (key counters) #####"
|
||||
if [ -r /proc/net/bonding/bond0 ]; then
|
||||
# shellcheck disable=SC2013 # intentional: extract NIC names from bonding info
|
||||
for nic in $(grep -oE 'eth[0-9]+|en[psx][a-z0-9]+' /proc/net/bonding/bond0 2>/dev/null | sort -u); do
|
||||
echo "--- ethtool -S $nic (errors) ---"
|
||||
ethtool -S "$nic" 2>/dev/null | grep -iE 'error|drop|discard|crc|pause|miss' || echo "(no error counters)"
|
||||
done
|
||||
fi
|
||||
echo
|
||||
echo "##### ip neigh (ARP table, reachable/stale) #####"
|
||||
ip neigh show 2>&1 | grep -vE ' FAILED|INCOMPLETE' | sort -t. -k4 -n
|
||||
echo
|
||||
echo "===== END $(hostname -s) ====="
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# probe-storage.sh
|
||||
#
|
||||
# READ-ONLY storage + disk ground-truth probe. Writes only stdout.
|
||||
# Run on any Proxmox host (or any Linux NFS server) to inventory its physical
|
||||
# disks, mounts, exports, SMART health, and Proxmox storage config.
|
||||
#
|
||||
# Portable: no hardcoded values. Uses only standard CLI tools + smartmontools.
|
||||
#
|
||||
# Usage (via tests/remote.sh):
|
||||
# PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file perf/scripts/probe-storage.sh
|
||||
#
|
||||
# Or directly on a host:
|
||||
# bash probe-storage.sh > storage-audit.txt
|
||||
###############################################################################
|
||||
set -u
|
||||
echo "===== HOST: $(hostname -s) $(date -u +%FT%TZ) ====="
|
||||
echo
|
||||
echo "##### lsblk (tree, with model/serial/size/type) #####"
|
||||
lsblk -o NAME,MAJ:MIN,SIZE,TYPE,MOUNTPOINT,MODEL,SERIAL,STATE,ROTA,TRAN,REV 2>&1
|
||||
echo
|
||||
echo "##### block devices by-id #####"
|
||||
for dev in /dev/disk/by-id/*; do
|
||||
[ -L "$dev" ] || continue
|
||||
case "$(basename "$dev")" in *part[0-9]*) continue;; esac
|
||||
ls -l "$dev"
|
||||
done 2>&1
|
||||
echo
|
||||
echo "##### nvme list (if any) #####"
|
||||
command -v nvme >/dev/null 2>&1 && nvme list 2>&1 || echo "(no nvme-cli or no nvme devices)"
|
||||
echo
|
||||
echo "##### blkid #####"
|
||||
blkid 2>&1
|
||||
echo
|
||||
echo "##### mounted filesystems #####"
|
||||
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS 2>&1
|
||||
echo
|
||||
echo "##### /etc/fstab #####"
|
||||
cat /etc/fstab 2>&1
|
||||
echo
|
||||
echo "##### /etc/exports (+ exports.d) #####"
|
||||
cat /etc/exports 2>&1
|
||||
for f in /etc/exports.d/*.exports; do [ -f "$f" ] && echo "--- $f ---" && cat "$f"; done 2>&1
|
||||
echo
|
||||
echo "##### df -h (all mounts) #####"
|
||||
df -h 2>&1
|
||||
echo
|
||||
echo "##### smartctl -a per block device #####"
|
||||
command -v smartctl >/dev/null 2>&1 || echo "(smartctl not installed)"
|
||||
for d in /dev/sd? /dev/nvme?n1; do
|
||||
[ -b "$d" ] || continue
|
||||
echo "----- smartctl -a $d -----"
|
||||
smartctl -a "$d" 2>&1 | grep -iE 'Device Model|Model Number|Serial|Firmware|User Capacity|Rotation Rate|Form Factor|SATA Version|NVMe|SMART overall|Reallocated|Pending|Uncorrect|Power On|Temperature|Media and Data Integrity' || true
|
||||
done
|
||||
echo
|
||||
echo "##### /etc/pve/storage.cfg #####"
|
||||
cat /etc/pve/storage.cfg 2>&1
|
||||
echo
|
||||
echo "##### pvesm status #####"
|
||||
pvesm status 2>&1
|
||||
echo
|
||||
echo "##### pvesm list per store #####"
|
||||
for s in $(pvesm status 2>/dev/null | awk 'NR>1 && $3>0 {print $1}'); do
|
||||
echo "--- pvesm list $s ---"
|
||||
pvesm list "$s" 2>&1 | head -40
|
||||
done
|
||||
echo
|
||||
echo "##### zpool status (if any) #####"
|
||||
command -v zpool >/dev/null 2>&1 && zpool status 2>&1 || echo "(no zfs)"
|
||||
echo
|
||||
echo "##### lvm: pvs/vgs/lvs #####"
|
||||
command -v pvs >/dev/null 2>&1 && { pvs 2>&1; echo; vgs 2>&1; echo; lvs 2>&1; } || echo "(no lvm tools)"
|
||||
echo
|
||||
echo "===== END $(hostname -s) ====="
|
||||
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
snmp-switch-audit.py — READ-ONLY switch inventory via SNMP.
|
||||
|
||||
Gathers interface status/speed/errors, LLDP neighbor topology, port-channel
|
||||
(LAG) membership, and VLAN membership from any SNMPv2c-capable switch.
|
||||
Designed for Dell/Radlan (Neyland) and standard IF/LLDP/Q-BRIDGE MIB switches,
|
||||
but works on any SNMP-manageable device.
|
||||
|
||||
All parameters configurable via env vars or CLI flags so this works on any
|
||||
network:
|
||||
|
||||
SNMP_COMMUNITY SNMPv2c community string (env, default: public)
|
||||
SWITCH_IPS space-separated switch IPs (env, or pass as args)
|
||||
OUTPUT_DIR where to write per-switch (env, default: returned-logs/snmp)
|
||||
|
||||
Usage:
|
||||
SNMP_COMMUNITY=kn3lmgmt SWITCH_IPS="192.168.0.9 192.168.0.12" \\
|
||||
python3 snmp-switch-audit.py
|
||||
|
||||
# or pass IPs as positional args:
|
||||
SNMP_COMMUNITY=kn3lmgmt python3 snmp-switch-audit.py 192.168.0.9 192.168.0.12
|
||||
|
||||
Requires: pysnmp (pip install pysnmp) or net-snmp utils (snmpwalk) on PATH.
|
||||
Outputs: per-switch JSON + human-readable text in OUTPUT_DIR.
|
||||
|
||||
Read-only: sends only SNMP GET/GETNEXT/GETBULK. Never SETs anything.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
COMMUNITY = os.environ.get("SNMP_COMMUNITY", "public")
|
||||
OUTPUT_DIR = os.environ.get("OUTPUT_DIR",
|
||||
os.path.join(os.path.dirname(__file__), "..",
|
||||
"..", "returned-logs", "snmp"))
|
||||
|
||||
# OID constants
|
||||
OID_SYSDESCR = "1.3.6.1.2.1.1.1.0"
|
||||
OID_SYSNAME = "1.3.6.1.2.1.1.5.0"
|
||||
OID_IF_NAME = "1.3.6.1.2.1.31.1.1.1.1"
|
||||
OID_IF_SPEED = "1.3.6.1.2.1.2.2.1.5"
|
||||
OID_IF_OPER = "1.3.6.1.2.1.2.2.1.8"
|
||||
OID_IF_INERR = "1.3.6.1.2.1.2.2.1.14"
|
||||
OID_IF_OUTERR = "1.3.6.1.2.1.2.2.1.20"
|
||||
OID_IF_INOCT = "1.3.6.1.2.1.31.1.1.1.6"
|
||||
OID_IF_OUTOCT = "1.3.6.1.2.1.31.1.1.1.10"
|
||||
OID_LACP_LAG = "1.2.840.10006.300.43.1.1.1.1"
|
||||
OID_LLDP_REM_PORT = "1.0.8802.1.1.2.1.4.1.1.7"
|
||||
OID_LLDP_REM_SYSNAME = "1.0.8802.1.1.2.1.4.1.1.9"
|
||||
OID_LLDP_REM_CHASSIS = "1.0.8802.1.1.2.1.4.1.1.6"
|
||||
OID_LLDP_REM_LOCALPORT = "1.0.8802.1.1.2.1.4.1.1.3"
|
||||
OID_QBRIDGE_VLAN = "1.3.6.1.2.1.17.7.1.4.3.1.1"
|
||||
|
||||
|
||||
def snmpget(ip, oid):
|
||||
"""Single SNMP GET, returns string value or None."""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["snmpget", "-Oqv", "-v2c", "-c", COMMUNITY, ip, oid],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
if r.returncode == 0 and r.stdout.strip():
|
||||
return r.stdout.strip().strip('"')
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def snmpwalk(ip, oid):
|
||||
"""SNMP BULKWALK, returns dict of ifIndex -> value."""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["snmpbulkwalk", "-Oqv", "-v2c", "-c", COMMUNITY, ip, oid],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
if r.returncode != 0:
|
||||
r = subprocess.run(
|
||||
["snmpwalk", "-Oqv", "-v2c", "-c", COMMUNITY, ip, oid],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
return {}
|
||||
result = {}
|
||||
for i, line in enumerate(r.stdout.strip().split("\n"), 1):
|
||||
line = line.strip().strip('"')
|
||||
if line:
|
||||
result[i] = line
|
||||
return result
|
||||
|
||||
|
||||
def walk_indexed(ip, oid):
|
||||
"""SNMP walk preserving OID index. Returns dict: index_str -> value."""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["snmpbulkwalk", "-v2c", "-c", COMMUNITY, ip, oid],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
if r.returncode != 0:
|
||||
r = subprocess.run(
|
||||
["snmpwalk", "-v2c", "-c", COMMUNITY, ip, oid],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
return {}
|
||||
result = {}
|
||||
for line in r.stdout.strip().split("\n"):
|
||||
m = re.search(r'(\d+)\s*=\s*(.+)', line)
|
||||
if not m:
|
||||
m = re.search(r'\.(\d+)\s*=\s*(.+)', line)
|
||||
if m:
|
||||
idx = m.group(1).split(".")[-1]
|
||||
val = m.group(2).strip()
|
||||
val = re.sub(r'^(INTEGER: |STRING: |Hex-STRING: |Gauge32: |Counter32: |Counter64: )', '', val)
|
||||
result[idx] = val
|
||||
return result
|
||||
|
||||
|
||||
def audit_switch(ip):
|
||||
"""Gather all data for one switch."""
|
||||
data = {"ip": ip}
|
||||
data["sysDescr"] = snmpget(ip, OID_SYSDESCR)
|
||||
data["sysName"] = snmpget(ip, OID_SYSNAME)
|
||||
if not data["sysDescr"]:
|
||||
return data
|
||||
|
||||
names = snmpwalk(ip, OID_IF_NAME)
|
||||
speeds = snmpwalk(ip, OID_IF_SPEED)
|
||||
oper = snmpwalk(ip, OID_IF_OPER)
|
||||
inerr = snmpwalk(ip, OID_IF_INERR)
|
||||
outerr = snmpwalk(ip, OID_IF_OUTERR)
|
||||
|
||||
interfaces = []
|
||||
for idx in sorted(names.keys()):
|
||||
if idx not in names:
|
||||
continue
|
||||
speed_raw = speeds.get(idx, "0")
|
||||
try:
|
||||
speed_mbps = int(re.sub(r'\D', '', str(speed_raw))) // 1000000
|
||||
except (ValueError, TypeError):
|
||||
speed_mbps = 0
|
||||
is_up = str(oper.get(idx, "0")).strip() == "1"
|
||||
interfaces.append({
|
||||
"ifIndex": idx,
|
||||
"name": names[idx],
|
||||
"speedMbps": speed_mbps,
|
||||
"up": is_up,
|
||||
"inErrors": inerr.get(idx, "0"),
|
||||
"outErrors": outerr.get(idx, "0"),
|
||||
})
|
||||
data["interfaces"] = interfaces
|
||||
|
||||
# LLDP neighbors
|
||||
rem_ports = walk_indexed(ip, OID_LLDP_REM_PORT)
|
||||
rem_sysnames = walk_indexed(ip, OID_LLDP_REM_SYSNAME)
|
||||
rem_chassis = walk_indexed(ip, OID_LLDP_REM_CHASSIS)
|
||||
rem_local = walk_indexed(ip, OID_LLDP_REM_LOCALPORT)
|
||||
lldp = []
|
||||
for idx in rem_ports:
|
||||
lldp.append({
|
||||
"localPort": rem_local.get(idx, "?"),
|
||||
"remotePort": rem_ports[idx],
|
||||
"remoteSysName": rem_sysnames.get(idx, ""),
|
||||
"remoteChassis": rem_chassis.get(idx, ""),
|
||||
})
|
||||
data["lldpNeighbors"] = lldp
|
||||
|
||||
# LACP LAG table
|
||||
lag_data = walk_indexed(ip, OID_LACP_LAG)
|
||||
data["lagTable"] = lag_data
|
||||
|
||||
# VLAN membership
|
||||
vlan_data = walk_indexed(ip, OID_QBRIDGE_VLAN)
|
||||
data["vlans"] = vlan_data
|
||||
|
||||
return data
|
||||
|
||||
|
||||
def print_switch(data):
|
||||
"""Human-readable summary."""
|
||||
print(f"\n{'='*60}")
|
||||
print(f" {data.get('sysName', data['ip'])} ({data['ip']})")
|
||||
print(f" {data.get('sysDescr', '?')}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
print(f"\n Active ports (UP only):")
|
||||
print(f" {'Port':<12} {'Speed':>10} {'InErrors':>10} {'OutErrors':>10}")
|
||||
print(f" {'-'*12} {'-'*10} {'-'*10} {'-'*10}")
|
||||
for iface in data.get("interfaces", []):
|
||||
if iface["up"]:
|
||||
print(f" {iface['name']:<12} {iface['speedMbps']:>8}Mb "
|
||||
f"{iface['inErrors']:>10} {iface['outErrors']:>10}")
|
||||
|
||||
err_ports = [i for i in data.get("interfaces", [])
|
||||
if i["up"] and (int(i["inErrors"] or 0) > 0
|
||||
or int(i["outErrors"] or 0) > 0)]
|
||||
if err_ports:
|
||||
print(f"\n *** PORTS WITH ERRORS ***")
|
||||
for p in err_ports:
|
||||
print(f" {p['name']}: inErr={p['inErrors']} outErr={p['outErrors']}")
|
||||
|
||||
if data.get("lldpNeighbors"):
|
||||
print(f"\n LLDP neighbors:")
|
||||
for n in data["lldpNeighbors"]:
|
||||
sysname = n.get("remoteSysName", "") or "(unknown)"
|
||||
print(f" local={n['localPort']:<6} remote={n['remotePort']:<20} {sysname}")
|
||||
|
||||
if data.get("lagTable"):
|
||||
print(f"\n LACP/LAG table entries: {len(data['lagTable'])}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="READ-ONLY SNMP switch audit (portable, config-driven)")
|
||||
ap.add_argument("switches", nargs="*",
|
||||
help="switch IPs (env: SWITCH_IPS)")
|
||||
ap.add_argument("--community", default=COMMUNITY,
|
||||
help=f"SNMPv2c community (env: SNMP_COMMUNITY, default: {COMMUNITY})")
|
||||
ap.add_argument("--output", default=OUTPUT_DIR,
|
||||
help=f"output dir (env: OUTPUT_DIR)")
|
||||
args = ap.parse_args()
|
||||
|
||||
community = args.community
|
||||
|
||||
ips = args.switches
|
||||
if not ips:
|
||||
env_ips = os.environ.get("SWITCH_IPS", "")
|
||||
ips = env_ips.split()
|
||||
|
||||
if not ips:
|
||||
ap.error("no switch IPs provided (pass as args or set SWITCH_IPS)")
|
||||
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
all_data = []
|
||||
|
||||
for ip in ips:
|
||||
globals()["COMMUNITY"] = community
|
||||
data = audit_switch(ip.strip())
|
||||
all_data.append(data)
|
||||
print_switch(data)
|
||||
outpath = os.path.join(args.output, f"switch-{ip}.json")
|
||||
with open(outpath, "w") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
print(f"\n -> {outpath}")
|
||||
|
||||
combined = os.path.join(args.output, "switches-all.json")
|
||||
with open(combined, "w") as f:
|
||||
json.dump(all_data, f, indent=2)
|
||||
print(f"\n Combined: {combined}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,128 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# sw-capture-remote.sh - orchestrate a serial capture from this workstation.
|
||||
#
|
||||
# Flow:
|
||||
# 1. De-conflict: abort if any local ssh to pfv-tsys4 is in flight
|
||||
# (other agent could be there).
|
||||
# 2. Free the serial port: kill whatever holds /dev/ttyUSBx
|
||||
# (typically a screen session). Targeted, not blanket.
|
||||
# 3. scp driver + .cmds to pfv-tsys4.
|
||||
# 4. Run driver over ssh, capture stderr to console.
|
||||
# 5. scp the resulting log back to returned-logs/.
|
||||
#
|
||||
# Usage:
|
||||
# sw-capture-remote.sh <switch-name> [device]
|
||||
#
|
||||
# <switch-name> e.g. pfv-core-sw01 (must have switches/<name>.cmds)
|
||||
# [device] /dev/ttyUSBx on pfv-tsys4. Defaults per switch map below.
|
||||
#
|
||||
# Currently scoped to pfv-core-sw01 only (per user direction). The other
|
||||
# two switches are deferred; their defaults are placeholders.
|
||||
set -u
|
||||
|
||||
SWITCH=${1:-}
|
||||
DEVICE=${2:-}
|
||||
|
||||
if [ -z "$SWITCH" ]; then
|
||||
echo "Usage: $0 <switch-name> [device]" >&2
|
||||
echo " e.g. $0 pfv-core-sw01 /dev/ttyUSB2" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Switch -> default device map (ttyUSB2 = core-sw01 confirmed by user).
|
||||
case "$SWITCH" in
|
||||
pfv-core-sw01)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB2 ;;
|
||||
pfv-r3-tor-mgmt)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB0 # TENTATIVE - unconfirmed
|
||||
if [ "${2:-}" = "" ]; then
|
||||
echo "NOTE: pfv-r3-tor-mgmt device is tentative (/dev/ttyUSB0)." >&2
|
||||
echo " Pass the device explicitly if different." >&2
|
||||
fi ;;
|
||||
pfv-r3-tor-stor)
|
||||
[ -z "$DEVICE" ] && DEVICE=/dev/ttyUSB1 # TENTATIVE - unconfirmed
|
||||
if [ "${2:-}" = "" ]; then
|
||||
echo "NOTE: pfv-r3-tor-stor device is tentative (/dev/ttyUSB1)." >&2
|
||||
echo " Pass the device explicitly if different." >&2
|
||||
fi ;;
|
||||
*)
|
||||
echo "unknown switch: $SWITCH" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
BAUD=9600
|
||||
HOST=root@pfv-tsys4
|
||||
HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
|
||||
LOCAL_DRIVER=$HERE/scripts/sw-capture.py
|
||||
LOCAL_CMDS=$HERE/switches/$SWITCH.cmds
|
||||
LOCAL_LOG=$HERE/returned-logs/$SWITCH.log
|
||||
REMOTE_DRIVER=/root/sw-capture.py
|
||||
REMOTE_CMDS=/root/$SWITCH.cmds
|
||||
REMOTE_LOG=/root/$SWITCH.log
|
||||
|
||||
[ -f "$LOCAL_DRIVER" ] || { echo "missing $LOCAL_DRIVER" >&2; exit 2; }
|
||||
[ -f "$LOCAL_CMDS" ] || { echo "missing $LOCAL_CMDS" >&2; exit 2; }
|
||||
|
||||
ts() { date +%H:%M:%S; }
|
||||
|
||||
echo "[$(ts)] switch=$SWITCH device=$DEVICE baud=$BAUD host=$HOST"
|
||||
|
||||
# 1. De-conflict: any local ssh to pfv-tsys4 in flight?
|
||||
echo "[$(ts)] checking for in-flight ssh to pfv-tsys4..."
|
||||
# shellcheck disable=SC2009 # intentional: need full ps columns filtered by process args
|
||||
if ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys4|scp.*pfv-tsys4' | grep -v grep >/tmp/.swcap.ps 2>&1; then
|
||||
cat /tmp/.swcap.ps
|
||||
echo "[$(ts)] ABORT: another ssh/scp to pfv-tsys4 is running (other agent?)." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "[$(ts)] clear."
|
||||
rm -f /tmp/.swcap.ps
|
||||
|
||||
# 2. Free the serial port: kill whatever holds $DEVICE.
|
||||
echo "[$(ts)] freeing $DEVICE on $HOST (targeted; other screen sessions untouched)..."
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=8 "$HOST" \
|
||||
"fuser -v $DEVICE 2>&1 | tee /dev/stderr; \
|
||||
fuser -k -TERM $DEVICE 2>/dev/null; sleep 1; \
|
||||
if fuser $DEVICE 2>/dev/null; then \
|
||||
echo 'still held after SIGTERM, escalating to SIGKILL'; \
|
||||
fuser -k -KILL $DEVICE 2>/dev/null; sleep 1; \
|
||||
fi; \
|
||||
fuser $DEVICE 2>/dev/null && echo 'STILL HELD' || echo 'FREE'"
|
||||
|
||||
# Re-check; abort if still held.
|
||||
HELD=$(ssh -o BatchMode=yes "$HOST" "fuser $DEVICE 2>/dev/null && echo HELD || echo FREE")
|
||||
if [ "$HELD" = "HELD" ]; then
|
||||
echo "[$(ts)] ABORT: $DEVICE still held on $HOST." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 3. Copy driver + cmds.
|
||||
echo "[$(ts)] copying driver + cmds to $HOST..."
|
||||
scp -q "$LOCAL_DRIVER" "$HOST:$REMOTE_DRIVER"
|
||||
scp -q "$LOCAL_CMDS" "$HOST:$REMOTE_CMDS"
|
||||
|
||||
# 4. Run the capture on pfv-tsys4. Stream stderr (progress) to console.
|
||||
echo "[$(ts)] running capture..."
|
||||
ssh -o BatchMode=yes -o ServerAliveInterval=10 "$HOST" \
|
||||
"python3 $REMOTE_DRIVER \
|
||||
--device $DEVICE --baud $BAUD \
|
||||
--cmds $REMOTE_CMDS --log $REMOTE_LOG"
|
||||
RC=$?
|
||||
echo "[$(ts)] capture exit code: $RC"
|
||||
|
||||
# 5. Pull log back.
|
||||
echo "[$(ts)] pulling log back to $LOCAL_LOG..."
|
||||
mkdir -p "$(dirname "$LOCAL_LOG")"
|
||||
scp -q "$HOST:$REMOTE_LOG" "$LOCAL_LOG"
|
||||
if [ -f "$LOCAL_LOG" ]; then
|
||||
SZ=$(wc -c < "$LOCAL_LOG")
|
||||
echo "[$(ts)] OK: $LOCAL_LOG ($SZ bytes)"
|
||||
echo "----- head -----"
|
||||
head -30 "$LOCAL_LOG"
|
||||
echo "----- tail -----"
|
||||
tail -10 "$LOCAL_LOG"
|
||||
else
|
||||
echo "[$(ts)] ERROR: log not pulled back." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit $RC
|
||||
@@ -1,255 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
sw-capture.py - drive a Dell switch over a serial console and log all output.
|
||||
|
||||
Read-only. Sends ONLY the commands listed in the supplied .cmds file
|
||||
(comment lines starting with '!' and blank lines are skipped). Handles
|
||||
`--More--` pagination by sending a space. Aborts cleanly on any password
|
||||
prompt (we never supply credentials).
|
||||
|
||||
Pure stdlib (termios + select). No pyserial/expect required.
|
||||
|
||||
Exit codes:
|
||||
0 clean run, every command saw a prompt again
|
||||
2 could not synchronize with a prompt during wake
|
||||
3 one or more commands timed out (log still written)
|
||||
4 password prompt encountered (aborted)
|
||||
|
||||
Usage:
|
||||
sw-capture.py --device /dev/ttyUSB2 --baud 9600 \\
|
||||
--cmds pfv-core-sw01.cmds --log /root/pfv-core-sw01.log
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import select
|
||||
import sys
|
||||
import termios
|
||||
import time
|
||||
|
||||
PROMPT_RE = re.compile(rb'[>#]\s*$') # ends in # or > + spaces
|
||||
MORE_RE = re.compile(rb'--\s*More\s*--') # pagination prompt
|
||||
PWD_RE = re.compile(rb'[Pp]assword:\s*$') # enable / login password
|
||||
|
||||
BAUDS = {
|
||||
'9600': termios.B9600,
|
||||
'19200': termios.B19200,
|
||||
'38400': termios.B38400,
|
||||
'57600': termios.B57600,
|
||||
'115200': termios.B115200,
|
||||
}
|
||||
|
||||
|
||||
def log(msg, level='INFO'):
|
||||
sys.stderr.write(f'[{level}] {msg}\n')
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def open_port(device, baud):
|
||||
"""Open the serial device raw at the requested baud, 8N1, no flow ctrl."""
|
||||
fd = os.open(device, os.O_RDWR | os.O_NOCTTY)
|
||||
try:
|
||||
attrs = termios.tcgetattr(fd)
|
||||
except termios.error:
|
||||
log(f'{device} is not a termios-capable device', 'WARN')
|
||||
return fd
|
||||
|
||||
# raw input
|
||||
attrs[0] &= ~(termios.IGNBRK | termios.BRKINT | termios.PARMRK |
|
||||
termios.ISTRIP | termios.INLCR | termios.IGNCR |
|
||||
termios.ICRNL | termios.IXON)
|
||||
# raw output
|
||||
attrs[1] &= ~termios.OPOST
|
||||
# 8N1, enable receiver, ignore modem control lines
|
||||
attrs[2] &= ~(termios.CSIZE | termios.PARENB | termios.CSTOPB)
|
||||
attrs[2] |= termios.CS8 | termios.CREAD | termios.CLOCAL
|
||||
# raw local
|
||||
attrs[3] &= ~(termios.ECHO | termios.ECHONL | termios.ICANON |
|
||||
termios.ISIG | termios.IEXTEN)
|
||||
# non-blocking-ish reads (select is the primary gate)
|
||||
attrs[6][termios.VMIN] = 0
|
||||
attrs[6][termios.VTIME] = 1
|
||||
|
||||
b = BAUDS.get(str(baud))
|
||||
if b is None:
|
||||
raise SystemExit(f'unsupported baud: {baud}')
|
||||
# Set ispeed/ospeed directly on the attribute list. (Equivalent to
|
||||
# termios.cfsetispeed/cfsetospeed, which are missing on some Python
|
||||
# builds — e.g. the one on pfv-tsys4.)
|
||||
attrs[4] = b # ispeed
|
||||
attrs[5] = b # ospeed
|
||||
termios.tcsetattr(fd, termios.TCSANOW, attrs)
|
||||
return fd
|
||||
|
||||
|
||||
def read_chunk(fd, timeout):
|
||||
"""Read whatever arrives within `timeout`. Extends briefly on activity."""
|
||||
buf = b''
|
||||
deadline = time.time() + timeout
|
||||
while True:
|
||||
remaining = deadline - time.time()
|
||||
if remaining <= 0:
|
||||
return buf
|
||||
r, _, _ = select.select([fd], [], [], min(0.5, remaining))
|
||||
if not r:
|
||||
if buf:
|
||||
return buf
|
||||
continue
|
||||
try:
|
||||
chunk = os.read(fd, 4096)
|
||||
except OSError:
|
||||
return buf
|
||||
if not chunk:
|
||||
return buf
|
||||
buf += chunk
|
||||
# keep collecting as long as bytes are flowing
|
||||
deadline = time.time() + 0.3
|
||||
|
||||
|
||||
def drain(fd, timeout=1.0):
|
||||
total = 0
|
||||
while True:
|
||||
b = read_chunk(fd, timeout=timeout)
|
||||
if not b:
|
||||
return total
|
||||
total += len(b)
|
||||
|
||||
|
||||
def send(fd, s):
|
||||
if isinstance(s, str):
|
||||
s = s.encode()
|
||||
os.write(fd, s)
|
||||
|
||||
|
||||
def wait_for(fd, regex, timeout, on_more=None, on_pwd=None):
|
||||
"""Read until `regex` matches the tail of the buffer, or timeout."""
|
||||
buf = b''
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
remaining = deadline - time.time()
|
||||
chunk = read_chunk(fd, timeout=min(1.0, remaining))
|
||||
if chunk:
|
||||
buf += chunk
|
||||
tail64 = buf[-64:]
|
||||
tail32 = buf[-32:]
|
||||
tail128 = buf[-128:]
|
||||
if on_more and MORE_RE.search(tail64):
|
||||
on_more(fd)
|
||||
continue
|
||||
if on_pwd and PWD_RE.search(tail32):
|
||||
on_pwd(buf)
|
||||
return buf, 'pwd'
|
||||
if regex.search(tail128):
|
||||
return buf, 'ok'
|
||||
return buf, 'timeout'
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--device', required=True)
|
||||
ap.add_argument('--baud', type=int, default=9600)
|
||||
ap.add_argument('--cmds', required=True)
|
||||
ap.add_argument('--log', required=True)
|
||||
ap.add_argument('--per-cmd-timeout', type=float, default=45.0)
|
||||
ap.add_argument('--wake-timeout', type=float, default=15.0)
|
||||
ap.add_argument('--session-max', type=float, default=600.0)
|
||||
args = ap.parse_args()
|
||||
|
||||
cmds = []
|
||||
with open(args.cmds) as f:
|
||||
for raw in f:
|
||||
s = raw.rstrip('\n').strip()
|
||||
if not s or s.startswith('!'):
|
||||
continue
|
||||
cmds.append(s)
|
||||
log(f'parsed {len(cmds)} commands from {args.cmds}')
|
||||
|
||||
logf = open(args.log, 'wb', buffering=0)
|
||||
|
||||
def w(b):
|
||||
if isinstance(b, str):
|
||||
b = b.encode()
|
||||
logf.write(b)
|
||||
|
||||
w(f'==== sw-capture {time.strftime("%Y-%m-%d %H:%M:%S")} ====\n')
|
||||
w(f'device={args.device} baud={args.baud} cmds={args.cmds} '
|
||||
f'n={len(cmds)} per_cmd_timeout={args.per_cmd_timeout}\n\n')
|
||||
|
||||
fd = open_port(args.device, args.baud)
|
||||
log(f'opened {args.device} @ {args.baud} 8N1 raw')
|
||||
|
||||
session_start = time.time()
|
||||
abort = False
|
||||
|
||||
def on_more(fd_):
|
||||
log('--More-- -> space')
|
||||
send(fd_, b' ')
|
||||
|
||||
def on_pwd(buf):
|
||||
nonlocal abort
|
||||
abort = True
|
||||
log('password prompt detected (enable or login) - aborting; '
|
||||
'no credentials supplied', 'ERROR')
|
||||
w(buf)
|
||||
w(b'\n[PASSWORD PROMPT - ABORTED]\n')
|
||||
|
||||
# WAKE: nudge with Ctrl-C + Enter, look for any prompt
|
||||
drain(fd, 0.5)
|
||||
synced = False
|
||||
wake_deadline = time.time() + args.wake_timeout
|
||||
attempt = 0
|
||||
while time.time() < wake_deadline:
|
||||
attempt += 1
|
||||
send(fd, b'\x03')
|
||||
time.sleep(0.2)
|
||||
send(fd, b'\r')
|
||||
buf, status = wait_for(fd, PROMPT_RE, timeout=3.0,
|
||||
on_more=on_more, on_pwd=on_pwd)
|
||||
w(buf)
|
||||
if status == 'pwd':
|
||||
logf.close(); os.close(fd); sys.exit(4)
|
||||
if status == 'ok':
|
||||
synced = True
|
||||
log(f'prompt synced after {attempt} attempt(s)')
|
||||
break
|
||||
if not synced:
|
||||
w(b'\n[NO PROMPT - ABORT]\n')
|
||||
log('no prompt detected during wake window', 'ERROR')
|
||||
logf.close(); os.close(fd); sys.exit(2)
|
||||
|
||||
# RUN commands verbatim from the .cmds list
|
||||
failures = 0
|
||||
for idx, cmd in enumerate(cmds, 1):
|
||||
if time.time() - session_start > args.session_max:
|
||||
log('session_max exceeded - stopping early', 'ERROR')
|
||||
w(b'\n[SESSION_MAX - STOP]\n')
|
||||
break
|
||||
if abort:
|
||||
break
|
||||
log(f'[{idx}/{len(cmds)}] {cmd}')
|
||||
send(fd, cmd + '\r')
|
||||
buf, status = wait_for(fd, PROMPT_RE,
|
||||
timeout=args.per_cmd_timeout,
|
||||
on_more=on_more, on_pwd=on_pwd)
|
||||
w(buf)
|
||||
if status == 'pwd':
|
||||
failures += 1
|
||||
break
|
||||
if status == 'timeout':
|
||||
log(f'timeout after: {cmd}', 'WARN')
|
||||
failures += 1
|
||||
# try to resync: Ctrl-C + drain
|
||||
send(fd, b'\x03')
|
||||
time.sleep(0.3)
|
||||
drain(fd, 0.5)
|
||||
|
||||
w(f'\n==== end {time.strftime("%Y-%m-%d %H:%M:%S")} '
|
||||
f'failures={failures} ====\n')
|
||||
logf.close()
|
||||
os.close(fd)
|
||||
log(f'done -> {args.log} failures={failures}')
|
||||
sys.exit(0 if failures == 0 else 3)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,56 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Probe conman state + expect availability on pfv-tsys4.
|
||||
# Read-only. Decides whether we drive via conman+expect or expect-only.
|
||||
set -u
|
||||
|
||||
# De-conflict: any ssh to pfv-tsys4 right now?
|
||||
echo "===== LOCAL ssh activity ====="
|
||||
# shellcheck disable=SC2009 # intentional: need full ps columns filtered by process args
|
||||
ps -eo pid,etime,args | grep -E 'ssh.*pfv-tsys' | grep -v grep || echo "(none to pfv-tsys4)"
|
||||
|
||||
echo
|
||||
echo "===== pfv-tsys4: conman + expect state ====="
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
|
||||
echo "--- conmand service ---"
|
||||
systemctl is-active conmand 2>&1 || true
|
||||
systemctl is-enabled conmand 2>&1 || true
|
||||
systemctl status conmand --no-pager 2>&1 | head -15 || true
|
||||
|
||||
echo
|
||||
echo "--- conman binary ---"
|
||||
command -v conman && conman --version 2>&1 | head -2 || echo "conman: MISSING"
|
||||
command -v conmand && echo "conmand present" || echo "conmand: MISSING"
|
||||
|
||||
echo
|
||||
echo "--- /etc/conman.conf: ttyUSB2 entries ---"
|
||||
grep -nE "ttyUSB2|core-sw|CONSOLE|LOG|SERIAL|BAUD" /etc/conman.conf 2>/dev/null | head -40 || echo "(no matches / no file)"
|
||||
|
||||
echo
|
||||
echo "--- conman log dir ---"
|
||||
ls -la /var/log/conman/ 2>&1 | head -20 || echo "(no /var/log/conman)"
|
||||
ls -la /var/consoles/ 2>&1 | head -20 || echo "(no /var/consoles)"
|
||||
|
||||
echo
|
||||
echo "--- expect availability ---"
|
||||
command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
|
||||
echo "apt-cache policy expect:"
|
||||
apt-cache policy expect 2>/dev/null | head -10 || echo "(apt-cache failed)"
|
||||
|
||||
echo
|
||||
echo "--- other useful drivers ---"
|
||||
for t in tclsh socat cu tip; do
|
||||
command -v "$t" 2>/dev/null && echo " $t: present" || true
|
||||
done
|
||||
|
||||
echo
|
||||
echo "--- apt network reachability (quick) ---"
|
||||
timeout 5 bash -c 'echo > /dev/tcp/deb.debian.org/80' 2>&1 && echo "apt network: OK" || echo "apt network: UNREACHABLE"
|
||||
|
||||
echo
|
||||
echo "--- disk space for log ---"
|
||||
df -h /root 2>&1 | tail -2
|
||||
|
||||
echo
|
||||
echo "--- screen sessions (still 3?) ---"
|
||||
screen -ls 2>&1 || true
|
||||
REMOTE
|
||||
@@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Probe pfv-tsys4 for what's available to drive the serial console.
|
||||
# Also snapshots local ssh/scp activity so we can de-conflict with the
|
||||
# other agent running in this directory.
|
||||
set -u
|
||||
|
||||
echo "===== LOCAL ssh/scp activity (other-agent de-confliction) ====="
|
||||
# shellcheck disable=SC2009 # intentional: need full ps columns (etime,args) filtered by process args
|
||||
ps -eo pid,ppid,etime,user,args | grep -E 'ssh|scp' | grep -v grep || echo "(none)"
|
||||
|
||||
echo
|
||||
echo "===== Ping pfv-tsys4 ====="
|
||||
ping -c1 -W2 pfv-tsys4 >/dev/null 2>&1 && echo "ping OK" || echo "ping FAIL"
|
||||
|
||||
echo
|
||||
echo "===== Probe pfv-tsys4 over ssh ====="
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=5 root@pfv-tsys4 'bash -s' <<'REMOTE'
|
||||
echo "--- host ---"
|
||||
hostname; uname -a
|
||||
echo "--- tools ---"
|
||||
for t in python3 python expect screen minicom picocom stty fuser lsof; do
|
||||
p=$(command -v "$t" 2>/dev/null) && echo "$t -> $p" || echo "$t -> MISSING"
|
||||
done
|
||||
echo "--- pyserial ---"
|
||||
python3 -c "import serial; print('pyserial', serial.__version__)" 2>&1
|
||||
echo "--- device node ---"
|
||||
ls -l /dev/ttyUSB2 2>&1
|
||||
stat -c '%n owner=%U:%G mode=%a' /dev/ttyUSB2 2>&1 || true
|
||||
echo "--- who holds /dev/ttyUSB2 ---"
|
||||
fuser -v /dev/ttyUSB2 2>&1 || echo "(fuser: none or n/a)"
|
||||
lsof /dev/ttyUSB2 2>&1 | head -20 || true
|
||||
echo "--- screen sessions on this host ---"
|
||||
screen -ls 2>&1 || echo "(no screen / not installed)"
|
||||
echo "--- current tty settings (only readable if not held exclusively) ---"
|
||||
stty -F /dev/ttyUSB2 2>&1 || echo "(held exclusively - expected if screen is up)"
|
||||
echo "--- baud hints in config/history ---"
|
||||
grep -riE "ttyUSB2|115200|9600|baud" /etc/ ~/.screenrc ~/.bash_history 2>/dev/null | head -20 || true
|
||||
echo "--- recent console-related processes ---"
|
||||
ps -eo pid,etime,user,args | grep -E 'screen|minicom|picocom|ttyUSB' | grep -v grep || echo "(none)"
|
||||
REMOTE
|
||||
@@ -8,7 +8,7 @@
|
||||
dumps + ~05:40 CDT capacity probe). Storage configs read directly from
|
||||
`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host.
|
||||
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
|
||||
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline)
|
||||
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (retired / permanently offline)
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,478 @@
|
||||
# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)
|
||||
|
||||
> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md)
|
||||
> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are
|
||||
> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
|
||||
|
||||
**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)
|
||||
+ switch SNMP/conman captures (~03:50–04:00 UTC).
|
||||
**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`,
|
||||
`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all
|
||||
read-only. No changes made to any system.
|
||||
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
|
||||
**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild)
|
||||
|
||||
---
|
||||
|
||||
## 1. Physical network topology
|
||||
|
||||
### 1.1 Diagram
|
||||
|
||||
```
|
||||
RACK 5 (core/data) RACK 3 (compute/RackRental)
|
||||
┌─────────────────────────┐ ┌──────────────────────────┐
|
||||
│ │ │ │
|
||||
│ pfv-core-sw01 │ ch1 LAG │ pfv-tor3-stor │
|
||||
│ PowerConnect 5448 │◄───3×1G──────────►│ Neyland 24T (PC5324) │
|
||||
│ (mgmt+storage VLAN) │ (g13-15) │ (storage TOR) │
|
||||
│ │ g16 UP, not │ │
|
||||
│ VLAN1 (mgmt) │ in LAG; g17 │ g8-9 → tsys6 bond0 │
|
||||
│ VLAN1000 (storage) │ DOWN │ g10-11 → tsys7 bond0 │
|
||||
│ │ │ g20-23 → ch1 (3 active) │
|
||||
│ g8 → tsys4 nic0 │ │ │
|
||||
│ g19 → tsys5 nic0 │ │ pfv-tor3-mgmt │
|
||||
│ g31 → tsys4 enx (USB) │ │ Neyland 24T (PC5324) │
|
||||
│ g33-34 → tsys5 ch3 │ │ (mgmt TOR) │
|
||||
│ ch3 → tsys5 bond (1 ln)│ │ g7 → tsys6 nic0 │
|
||||
│ │ │ g3 → tsys7 nic0 │
|
||||
│ tsys4 (storage+PBS) │ │ │
|
||||
│ tsys5 (storage+sectest)│ │ tsys6 (RackRental) │
|
||||
│ tsys1 (infra) │ │ tsys7 (RackRental) │
|
||||
│ tsys3 (compute) │ │ │
|
||||
│ tsys9 (infra+compute) │ │ │
|
||||
└─────────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK
|
||||
|
||||
| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? |
|
||||
|----------|-------------------|--------------------|----------|
|
||||
| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below |
|
||||
| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG |
|
||||
| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate |
|
||||
| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 |
|
||||
| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection |
|
||||
| **VLAN** | access vlan 1000 | access vlan 1000 | OK |
|
||||
|
||||
**Three root causes of cross-rack underperformance:**
|
||||
|
||||
1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on
|
||||
`g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th
|
||||
physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps,
|
||||
not the intended 4).
|
||||
|
||||
2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4`
|
||||
(includes TCP/UDP source+dest ports — correct for NFS with nconnect).
|
||||
core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With
|
||||
layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one
|
||||
physical link**. Even though the clients use `nconnect=4` (4 TCP connections
|
||||
per mount), they share the same src/dst IP → same hash → **max 1 Gbps per
|
||||
server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.
|
||||
|
||||
3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the
|
||||
cross-rack trunk. `mode on` provides no link-failure detection, no partner
|
||||
validation, no dynamic negotiation. A dead cable stays "active" until
|
||||
manually discovered.
|
||||
|
||||
### 1.3 Per-host network state (ground truth)
|
||||
|
||||
| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP |
|
||||
|------|------|----------|-------------|-------|------|---------------|-------|------------|
|
||||
| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 |
|
||||
| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 |
|
||||
| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 |
|
||||
| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 |
|
||||
| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 |
|
||||
| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 |
|
||||
| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 |
|
||||
|
||||
**Key observations:**
|
||||
- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously.
|
||||
- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy.
|
||||
- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is
|
||||
the LACP problem the operator has been struggling with. The switch side (core-sw01
|
||||
ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
|
||||
- **tsys1 also uses a USB dongle** for storage.
|
||||
- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5
|
||||
use layer2+3 (suboptimal for NFS multi-connection).
|
||||
|
||||
### 1.4 Switch inventory (SNMP + conman ground truth)
|
||||
|
||||
| Switch | IP | Model | Firmware | Role | Console |
|
||||
|--------|----|-------|----------|------|---------|
|
||||
| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 |
|
||||
| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 |
|
||||
| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 |
|
||||
| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? |
|
||||
|
||||
**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c.
|
||||
**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit
|
||||
(udev symlink naming bug — singular vs plural — prevented consoles from working
|
||||
after reboot).
|
||||
|
||||
### 1.5 Port error counters (core-sw01)
|
||||
|
||||
| Port | In Errors | Notes |
|
||||
|------|-----------|-------|
|
||||
| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. |
|
||||
| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. |
|
||||
| **g26** | **73** | Connected to a management-network device. |
|
||||
| All others | 0 | Healthy. |
|
||||
|
||||
tor3-stor shows **zero errors** on all active ports.
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage topology (CORRECTED — current ground truth)
|
||||
|
||||
### 2.1 Storage tier definitions
|
||||
|
||||
| Tier | Definition | Performance | Use case |
|
||||
|------|-----------|-------------|----------|
|
||||
| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads |
|
||||
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives |
|
||||
| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only |
|
||||
|
||||
### 2.2 tsys4 disk inventory (CHANGED since last audit)
|
||||
|
||||
| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes |
|
||||
|--------|-------|------|-----------|-------|------------|------|-------|
|
||||
| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. |
|
||||
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. |
|
||||
| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** |
|
||||
| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
|
||||
| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. |
|
||||
| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). |
|
||||
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. |
|
||||
|
||||
**Changes from AUDIT-2026-07-29:**
|
||||
- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB).
|
||||
D3 is now Tier 3 (was previously listed as unreliable USB SSD).
|
||||
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox.
|
||||
Potential new Tier 2 storage.
|
||||
- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup.
|
||||
- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone.
|
||||
|
||||
### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)
|
||||
|
||||
| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes |
|
||||
|--------|-------|------|-----------|-------|-----------|------|-------|
|
||||
| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
|
||||
| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
|
||||
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
|
||||
| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). |
|
||||
| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. |
|
||||
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. |
|
||||
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. |
|
||||
|
||||
**Changes from AUDIT-2026-07-29:**
|
||||
- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest-
|
||||
performance local storage in the fleet (true NVMe).
|
||||
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal
|
||||
SATA. Not yet mounted/configured.
|
||||
- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented
|
||||
out. /etc/exports still exports S3 (stale).
|
||||
- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented.
|
||||
storage.cfg still has a broken `dir: S4` entry (points to root filesystem).
|
||||
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate
|
||||
916GB" in old audit).
|
||||
- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention.
|
||||
|
||||
### 2.4 NFS export map (current, verified)
|
||||
|
||||
| Server | Export | NFS ID | Disk | Tier | Used |
|
||||
|--------|--------|--------|------|------|------|
|
||||
| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% |
|
||||
| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) |
|
||||
| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% |
|
||||
| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) |
|
||||
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) |
|
||||
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) |
|
||||
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — |
|
||||
|
||||
### 2.5 NFS client mount status (nconnect)
|
||||
|
||||
| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash |
|
||||
|------|-------------------|-------------------|-------------|------|
|
||||
| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 |
|
||||
| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 |
|
||||
| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
|
||||
| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
|
||||
|
||||
**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4).
|
||||
**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client).
|
||||
|
||||
### 2.6 Per-spindle IOPS load (running VMs only)
|
||||
|
||||
| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes |
|
||||
|---------|--------|-----------------|----------------|-------|------|-------|
|
||||
| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 |
|
||||
| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 |
|
||||
| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) |
|
||||
| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 |
|
||||
| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. |
|
||||
| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
|
||||
| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
|
||||
| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) |
|
||||
|
||||
**D2 still carries 13 running VMs** — the IOPS hotspot persists.
|
||||
|
||||
---
|
||||
|
||||
## 3. Host capacity (live ground truth, 2026-07-30)
|
||||
|
||||
| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy |
|
||||
|------|-----|---------|-----|-------------|-------------|------|----------|
|
||||
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% |
|
||||
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% |
|
||||
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% |
|
||||
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% |
|
||||
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% |
|
||||
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% |
|
||||
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% |
|
||||
|
||||
**Changes since AUDIT-2026-07-29:**
|
||||
- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped.
|
||||
- **tsys9 RAM 62%→69%** — approaching 80% goal.
|
||||
- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit).
|
||||
- **All other hosts healthy** — well under 80%.
|
||||
|
||||
---
|
||||
|
||||
## 4. Fleet consistency audit
|
||||
|
||||
### 4.1 PVE/kernel versions
|
||||
|
||||
| Host | PVE | Kernel | Status |
|
||||
|------|-----|--------|--------|
|
||||
| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** |
|
||||
| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
|
||||
|
||||
### 4.2 Performance tuning state
|
||||
|
||||
| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect |
|
||||
|------|----------|------------|--------|---------------|---------------|-----------|----------|
|
||||
| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) |
|
||||
| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A |
|
||||
| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** |
|
||||
| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** |
|
||||
| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) |
|
||||
| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** |
|
||||
| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A |
|
||||
|
||||
**Inconsistencies:**
|
||||
- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong
|
||||
(`throughput-performance` should be `network-throughput`). Missing nconnect.
|
||||
- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere —
|
||||
expected, not a bug).
|
||||
- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7).
|
||||
|
||||
---
|
||||
|
||||
## 5. VM inventory + config audit (running VMs only)
|
||||
|
||||
### 5.1 Per-VM disk + CPU + network settings
|
||||
|
||||
> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good
|
||||
> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS.
|
||||
> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled.
|
||||
|
||||
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes |
|
||||
|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------|
|
||||
| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 |
|
||||
| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt |
|
||||
| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | |
|
||||
| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
|
||||
| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
|
||||
| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | |
|
||||
| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | |
|
||||
| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | |
|
||||
| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | |
|
||||
| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | |
|
||||
| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server |
|
||||
| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode |
|
||||
| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) |
|
||||
| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB |
|
||||
| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM |
|
||||
| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | |
|
||||
| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | |
|
||||
| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk |
|
||||
| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | |
|
||||
| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | |
|
||||
| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | |
|
||||
| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk |
|
||||
| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk |
|
||||
| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | |
|
||||
| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | |
|
||||
| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G |
|
||||
| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | |
|
||||
|
||||
### 5.2 VM optimization findings
|
||||
|
||||
| Finding | VMs affected | Impact | Fix |
|
||||
|---------|-------------|--------|-----|
|
||||
| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
|
||||
| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
|
||||
| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
|
||||
| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
|
||||
| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |
|
||||
|
||||
---
|
||||
|
||||
## 6. Console infrastructure status (fixed during this audit)
|
||||
|
||||
**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with
|
||||
`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
|
||||
`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
|
||||
console ports failed until `setup.sh`'s manual fallback re-created symlinks.
|
||||
|
||||
**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
|
||||
all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
|
||||
consoles now working (verified via conman-console.py).
|
||||
|
||||
**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
|
||||
conmand systemd service from starting.
|
||||
|
||||
---
|
||||
|
||||
## 7. Findings summary (prioritized)
|
||||
|
||||
### Critical (cross-rack performance)
|
||||
|
||||
| # | Finding | Impact | Type |
|
||||
|---|---------|--------|------|
|
||||
| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
|
||||
| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
|
||||
| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
|
||||
| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |
|
||||
|
||||
### High (storage configuration)
|
||||
|
||||
| # | Finding | Impact | Type |
|
||||
|---|---------|--------|------|
|
||||
| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
|
||||
| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
|
||||
| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
|
||||
| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity |
|
||||
| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config |
|
||||
|
||||
### Medium (fleet consistency)
|
||||
|
||||
| # | Finding | Impact | Type |
|
||||
|---|---------|--------|------|
|
||||
| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade |
|
||||
| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning |
|
||||
| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning |
|
||||
| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning |
|
||||
| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config |
|
||||
| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup |
|
||||
|
||||
### Low (monitoring / investigation)
|
||||
|
||||
| # | Finding | Impact | Type |
|
||||
|---|---------|--------|------|
|
||||
| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate |
|
||||
| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring |
|
||||
| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config |
|
||||
| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup |
|
||||
|
||||
---
|
||||
|
||||
## 9. Observability stack (operator direction)
|
||||
|
||||
The operator has defined a clear monitoring architecture with three tools,
|
||||
each serving a distinct role. Do NOT duplicate functionality across tools.
|
||||
|
||||
### 9.1 Current architecture
|
||||
|
||||
| Tool | Where it runs | Scope | Role |
|
||||
|------|---------------|-------|------|
|
||||
| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts |
|
||||
| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover |
|
||||
| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent |
|
||||
| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) |
|
||||
|
||||
### 9.2 Division of responsibility (no duplication)
|
||||
|
||||
| Metric class | Tool | Notes |
|
||||
|-------------|------|-------|
|
||||
| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) |
|
||||
| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications |
|
||||
| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) |
|
||||
| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations |
|
||||
|
||||
### 9.3 Saturday "OAM Day" plan (2026-08-01)
|
||||
|
||||
The operator has designated Saturday as a dedicated operations/admin/management
|
||||
day with the goal of achieving **100% instrumentation coverage**:
|
||||
|
||||
- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover)
|
||||
- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog
|
||||
relay, or Loki/Grafana stack)
|
||||
- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale
|
||||
devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12)
|
||||
- [ ] Beszel agent coverage on all 7 Proxmox hosts
|
||||
- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU
|
||||
via powerman, any temp sensors)
|
||||
- [ ] Verify alerting paths end-to-end (test each alert fires correctly)
|
||||
|
||||
> Much of the groundwork is already done: NUT is live on tsys1, powerman is
|
||||
> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live
|
||||
> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring
|
||||
> continuous monitoring + alerting on top of these.
|
||||
|
||||
---
|
||||
|
||||
## 8. Recommendations (for operator review — NO changes applied)
|
||||
|
||||
### 8.1 Switch changes (require maintenance window + operator approval)
|
||||
|
||||
1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes
|
||||
the 4th live cable (g16) part of the LAG.
|
||||
2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor).
|
||||
This unlocks multi-link distribution for nconnect NFS traffic.
|
||||
3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches
|
||||
for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
|
||||
4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor.
|
||||
5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr)
|
||||
on core-sw01.
|
||||
|
||||
> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the
|
||||
> highest-impact changes with the lowest risk. LACP conversion (#3) is optional
|
||||
> if the static trunk is working reliably otherwise. All changes can be done
|
||||
> read-only-verified via the conman + SNMP tooling in this repo.
|
||||
|
||||
### 8.2 Storage configuration (host-side, no switch changes needed)
|
||||
|
||||
1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`)
|
||||
per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact)
|
||||
and sectestbed scratch.
|
||||
2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or
|
||||
similar. Offloads the 93%-full Samsung 860 PRO.
|
||||
3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken
|
||||
`dir: S4` entry in storage.cfg.
|
||||
4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or
|
||||
keep local-only for PBS scratch.
|
||||
|
||||
### 8.3 Fleet consistency + tuning (host-side)
|
||||
|
||||
1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
|
||||
2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5:
|
||||
- Fix rmem/wmem to 128MB (tsys4)
|
||||
- Fix tuned profile (tsys4 → network-throughput)
|
||||
- Add nconnect=4 to tsys5 NFS client mounts
|
||||
- Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
|
||||
3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS-
|
||||
backed qcow2 disks (requires VM stop/start).
|
||||
|
||||
### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md))
|
||||
|
||||
The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty
|
||||
Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading
|
||||
that didn't exist in the 07-29 audit. Detailed placement plan deferred to the
|
||||
operator review of this document.
|
||||
@@ -1,93 +1,7 @@
|
||||
!###############################################################################
|
||||
!# pfv-core-sw01
|
||||
!#
|
||||
!# Dell PowerConnect 5448 (48-port 1GbE copper, 4× combo SFP / 10G via stacking)
|
||||
!# OS: DNOS 4.x / PowerConnect CLI
|
||||
!#
|
||||
!# This file is a flat list of READ-ONLY show commands.
|
||||
!# Paste them in order from enable mode. Capture the whole session to
|
||||
!# returned-logs/pfv-core-sw01.log
|
||||
!#
|
||||
!# To capture from a terminal:
|
||||
!# 1. SSH/telnet in
|
||||
!# 2. enable
|
||||
!# 3. terminal datadump (disables --More-- pause)
|
||||
!# 4. paste everything below
|
||||
!###############################################################################
|
||||
|
||||
enable
|
||||
! pfv-core-sw01 — Dell PowerConnect 5448 (core switch, rack 5)
|
||||
! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down)
|
||||
terminal datadump
|
||||
|
||||
! --- Identify & versions ---------------------------------------------------
|
||||
show version
|
||||
show system
|
||||
show inventory
|
||||
show bootvar
|
||||
enable
|
||||
show running-config
|
||||
show stack
|
||||
show cpu utilization
|
||||
show memory
|
||||
show logging
|
||||
|
||||
! --- Interfaces ------------------------------------------------------------
|
||||
show interfaces status
|
||||
show interfaces configuration
|
||||
show interfaces description
|
||||
show interfaces counters
|
||||
show interfaces counters errors
|
||||
show interfaces counters detail
|
||||
show interfaces counters cable 1-52
|
||||
|
||||
! --- Port-channels / LAG (4-port LACP to pfv-r3-tor-stor) -----------------
|
||||
show interfaces port-channel
|
||||
show lacp 1
|
||||
show lacp 2
|
||||
show lacp 3
|
||||
show lacp 4
|
||||
show lacp 5
|
||||
show lacp 6
|
||||
|
||||
! --- VLANs -----------------------------------------------------------------
|
||||
show vlan
|
||||
show vlan detail
|
||||
show interfaces vlan 1
|
||||
show interfaces vlan 1000
|
||||
show vlan info
|
||||
|
||||
! --- Spanning tree ---------------------------------------------------------
|
||||
show spanning-tree
|
||||
show spanning-tree active
|
||||
show spanning-tree summary
|
||||
show spanning-tree detail
|
||||
|
||||
! --- IGMP / multicast (matters for storage/iSCSI-NFS sometimes) -----------
|
||||
show ip igmp snooping
|
||||
show ip igmp snooping vlan 1
|
||||
show ip igmp snooping vlan 1000
|
||||
|
||||
! --- QoS / CoS -------------------------------------------------------------
|
||||
show qos
|
||||
show qos interface
|
||||
show class-of-service
|
||||
|
||||
! --- Storm control / loop protection --------------------------------------
|
||||
show storm-control
|
||||
show ports jumbo-frame
|
||||
|
||||
! --- ARP / MAC / FDB -------------------------------------------------------
|
||||
show mac address-table
|
||||
show arp
|
||||
show ip route
|
||||
|
||||
! --- Environmental ---------------------------------------------------------
|
||||
show environment
|
||||
show fan
|
||||
show power
|
||||
show hardware
|
||||
|
||||
! --- Flow control ----------------------------------------------------------
|
||||
show flowcontrol
|
||||
show interfaces flowcontrol
|
||||
|
||||
! --- Save the capture, then exit ------------------------------------------
|
||||
exit
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
!###############################################################################
|
||||
!# pfv-r3-tor-mgmt
|
||||
!#
|
||||
!# Rack 3 management TOR (VLAN1 only for this switch).
|
||||
!# Dell switch - OS variant unknown. Commands below cover the common
|
||||
!# Dell CLI flavors (DNOS 4/6/9, OS6/OS9, N-Series). Some may not be
|
||||
!# recognized on every OS; that is fine - skip errors and keep going.
|
||||
!#
|
||||
!# Paste from enable mode after `terminal datadump` (or `terminal length 0`
|
||||
!# on DNOS 6/9). Capture session to returned-logs/pfv-r3-tor-mgmt.log
|
||||
!###############################################################################
|
||||
|
||||
enable
|
||||
terminal datadump
|
||||
terminal length 0
|
||||
|
||||
! --- Identify ---------------------------------------------------------------
|
||||
show version
|
||||
show system
|
||||
show inventory
|
||||
show running-config
|
||||
show bootvar
|
||||
show cpu
|
||||
show cpu utilization
|
||||
show memory
|
||||
show processes
|
||||
show logging
|
||||
|
||||
! --- Interfaces -------------------------------------------------------------
|
||||
show interface status
|
||||
show interface description
|
||||
show interface configuration
|
||||
show interface counters
|
||||
show interface counters errors
|
||||
show interface counters detail
|
||||
show interfaces status
|
||||
show interfaces description
|
||||
show interfaces counters
|
||||
show interfaces counters errors
|
||||
|
||||
! --- Port channels / LAG ----------------------------------------------------
|
||||
show port-channel summary
|
||||
show port-channel detail
|
||||
show interfaces port-channel
|
||||
show lacp
|
||||
show lacp neighbor
|
||||
show lag
|
||||
|
||||
! --- VLANs ------------------------------------------------------------------
|
||||
show vlan
|
||||
show vlan brief
|
||||
show vlan detail
|
||||
|
||||
! --- Spanning tree ----------------------------------------------------------
|
||||
show spanning-tree
|
||||
show spanning-tree summary
|
||||
show spanning-tree active
|
||||
|
||||
! --- Multicast / snooping ---------------------------------------------------
|
||||
show ip igmp snooping
|
||||
show ip igmp snooping vlan 1
|
||||
|
||||
! --- QoS --------------------------------------------------------------------
|
||||
show qos
|
||||
show qos interface
|
||||
|
||||
! --- Jumbo / storm control --------------------------------------------------
|
||||
show ports jumbo-frame
|
||||
show storm-control
|
||||
|
||||
! --- FDB / ARP / routes -----------------------------------------------------
|
||||
show mac address-table
|
||||
show mac-address-table
|
||||
show arp
|
||||
show ip route
|
||||
|
||||
! --- Environment / hardware -------------------------------------------------
|
||||
show environment
|
||||
show fan
|
||||
show power supplies
|
||||
show power
|
||||
show hardware
|
||||
show transceiver
|
||||
show inventory
|
||||
|
||||
! --- Flow control -----------------------------------------------------------
|
||||
show flowcontrol
|
||||
show interfaces flowcontrol
|
||||
|
||||
exit
|
||||
@@ -1,88 +0,0 @@
|
||||
!###############################################################################
|
||||
!# pfv-r3-tor-stor — TARGETED LACP hash verification
|
||||
!#
|
||||
!# Purpose: confirm whether `port-channel load-balance layer-2-3-4` actually
|
||||
!# took effect, and gather the LAG state needed to decide whether the LAG
|
||||
!# must be bounced.
|
||||
!#
|
||||
!# Context: tsys6<->tsys7 is capped at exactly single-link speed (~943 Mbps)
|
||||
!# with 90K-136K retransmits on multi-stream TCP, but a clean 928 Mbps / 0
|
||||
!# retransmits on single-stream. That = offered load above 1 Gbps is being
|
||||
!# dropped at a single egress port. We need to know if the switch hash is
|
||||
!# really layer-2-3-4, and whether tsys6 + tsys7 ports are correctly
|
||||
!# bundled into LACP groups.
|
||||
!#
|
||||
!# Paste from enable mode. Capture to:
|
||||
!# returned-logs/pfv-r3-tor-stor-lacp-verify.log
|
||||
!###############################################################################
|
||||
|
||||
enable
|
||||
terminal datadump
|
||||
terminal length 0
|
||||
|
||||
! --- 1. THE KEY COMMAND: what hash is actually active? -----------------------
|
||||
! Expected: layer-2-3-4. If it still shows source-mac / dst-mac /
|
||||
! src-dst-mac (layer 2 only), the change did NOT take effect.
|
||||
show port-channel load-balance
|
||||
show port-channel protocol
|
||||
|
||||
! --- 2. Which LAGs exist and which physical ports are members ----------------
|
||||
! Look for the port-channels carrying tsys6 and tsys7. Confirm each has
|
||||
! TWO member ports both in "bundled" / "Forwarding" state.
|
||||
show port-channel summary
|
||||
show port-channel detail
|
||||
|
||||
! --- 3. LACP actor/partner state per member port ----------------------------
|
||||
! Every member port should show: Actor State Flag = AcLg (Active, Aggr),
|
||||
! Partner Operational, and a non-zero Partner System ID.
|
||||
show lacp
|
||||
show lacp neighbor
|
||||
|
||||
! --- 4. Per-port counters during/after an iperf run -------------------------
|
||||
! THIS is the switch-side equivalent of the host RX-counter test.
|
||||
! Run an 8-stream iperf tsys7->tsys6 first, then immediately capture
|
||||
! these so the byte counters reflect the test. The TWO tsys6-facing
|
||||
! ports should show SIMILAR "RX bytes" (switch ingress from tsys6's
|
||||
! bond is not the test direction; look at TX-out bytes toward tsys6,
|
||||
! i.e. the ports' output counters). Replace Te1/0/x with the real port
|
||||
! IDs shown by "show port-channel summary" above:
|
||||
show interfaces counters
|
||||
show interfaces counters errors
|
||||
|
||||
! --- 5. Running config line that set the hash (proof it was committed) ------
|
||||
show running-config | include port-channel
|
||||
|
||||
! --- 6. Switch identity (to look up correct hash syntax for this model) -----
|
||||
show version
|
||||
show system
|
||||
|
||||
exit
|
||||
|
||||
!###############################################################################
|
||||
!# DECISION TREE after reading the output:
|
||||
!#
|
||||
!# A) show port-channel load-balance reports layer-2-3-4 AND per-port
|
||||
!# counters show traffic split across BOTH tsys6-facing ports:
|
||||
!# -> Hash is fine. Re-test iperf; if still ~943 Mbps, the cap is
|
||||
!# host-side (receiver RX bridge / IRQ / CPU).
|
||||
!#
|
||||
!# B) load-balance reports layer-2-3-4 BUT per-port counters show all traffic
|
||||
!# on ONE tsys6-facing port:
|
||||
!# -> Hash is configured but the LAG needs a bounce to pick it up.
|
||||
!# Bounce the relevant port-channel:
|
||||
!# config
|
||||
!# interface port-channel N
|
||||
!# shutdown
|
||||
!# no shutdown
|
||||
!# end
|
||||
!# WARNING: drops all traffic on that LAG for ~2-5 s. Coordinate.
|
||||
!#
|
||||
!# C) load-balance still shows a layer-2-only policy:
|
||||
!# -> The earlier change didn't commit to this context. Re-apply:
|
||||
!# config
|
||||
!# port-channel load-balance layer-2-3-4
|
||||
!# end
|
||||
!# show port-channel load-balance
|
||||
!# copy running-config startup-config
|
||||
!# Then handle as case (B) if counters still aren't split.
|
||||
!###############################################################################
|
||||
@@ -1,108 +0,0 @@
|
||||
!###############################################################################
|
||||
!# pfv-r3-tor-stor
|
||||
!#
|
||||
!# Rack 3 storage TOR (VLAN1000 storage + VLAN1).
|
||||
!# Carries the 4-port LACP to pfv-core-sw01 (Rack 5).
|
||||
!# Most important switch for cross-rack storage throughput.
|
||||
!#
|
||||
!# Paste from enable mode after `terminal datadump` (DNOS 4/6) or
|
||||
!# `terminal length 0` (DNOS 9 / OS9). Capture to returned-logs/pfv-r3-tor-stor.log
|
||||
!###############################################################################
|
||||
|
||||
enable
|
||||
terminal datadump
|
||||
terminal length 0
|
||||
|
||||
! --- Identify ---------------------------------------------------------------
|
||||
show version
|
||||
show system
|
||||
show inventory
|
||||
show running-config
|
||||
show bootvar
|
||||
show cpu utilization
|
||||
show cpu
|
||||
show memory
|
||||
show processes
|
||||
show logging
|
||||
|
||||
! --- Interfaces (most important section) ------------------------------------
|
||||
show interface status
|
||||
show interface description
|
||||
show interface counters
|
||||
show interface counters errors
|
||||
show interface counters detail
|
||||
show interfaces status
|
||||
show interfaces description
|
||||
show interfaces counters
|
||||
show interfaces counters errors
|
||||
show interfaces configuration
|
||||
|
||||
! --- LACP / port-channel (the 4-port LAG to Rack 5) ------------------------
|
||||
show port-channel summary
|
||||
show port-channel detail
|
||||
show port-channel 1
|
||||
show port-channel 2
|
||||
show port-channel 3
|
||||
show port-channel 4
|
||||
show interfaces port-channel
|
||||
show lacp
|
||||
show lacp neighbor
|
||||
show lacp 1
|
||||
show lag
|
||||
|
||||
! --- VLANs (VLAN1000 is the storage VLAN) ----------------------------------
|
||||
show vlan
|
||||
show vlan brief
|
||||
show vlan id 1000
|
||||
show vlan detail
|
||||
show interfaces vlan 1000
|
||||
show interfaces switchport
|
||||
|
||||
! --- Spanning tree ----------------------------------------------------------
|
||||
show spanning-tree
|
||||
show spanning-tree summary
|
||||
show spanning-tree active
|
||||
show spanning-tree detail
|
||||
show spanning-tree mst
|
||||
|
||||
! --- Multicast / IGMP (storage traffic) -------------------------------------
|
||||
show ip igmp snooping
|
||||
show ip igmp snooping vlan 1000
|
||||
show ip igmp snooping vlan 1
|
||||
|
||||
! --- QoS / CoS / DCB / PFC (matters for storage) ---------------------------
|
||||
show qos
|
||||
show qos interface
|
||||
show class-of-service
|
||||
show dcb
|
||||
show dcb priority
|
||||
show priority-flow-control
|
||||
show pfc
|
||||
|
||||
! --- Jumbo / storm / MTU ----------------------------------------------------
|
||||
show ports jumbo-frame
|
||||
show storm-control
|
||||
show interfaces mtu
|
||||
|
||||
! --- FDB / ARP / routing ----------------------------------------------------
|
||||
show mac address-table
|
||||
show mac-address-table
|
||||
show arp
|
||||
show ip route
|
||||
|
||||
! --- Environment / hardware / optics ---------------------------------------
|
||||
show environment
|
||||
show fan
|
||||
show power supplies
|
||||
show power
|
||||
show hardware
|
||||
show inventory
|
||||
show transceiver
|
||||
show interface transceiver
|
||||
show media
|
||||
|
||||
! --- Flow control (important for storage) ----------------------------------
|
||||
show flowcontrol
|
||||
show interfaces flowcontrol
|
||||
|
||||
exit
|
||||
@@ -0,0 +1,12 @@
|
||||
! pfv-tor3-stor — Neyland 24T (Radlan-based, rack 3 storage TOR)
|
||||
! Radlan CLI uses different keywords than DNOS
|
||||
enable
|
||||
show system
|
||||
show inventory
|
||||
show interfaces configuration
|
||||
show interfaces description
|
||||
show port-channel
|
||||
show lag
|
||||
show vlan database
|
||||
show vlan
|
||||
show running-config
|
||||
Reference in New Issue
Block a user