feat(console): manage 7 switch consoles via ser2net+conman on pfv-tsys4
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.
Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.
Architecture (layered, no port sharing):
USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)
Port assignments (all on Tailscale IP 100.70.77.93):
2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor
2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw
2007 = pfv-r2-sw
Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
entries from mapping.txt. Idempotent (markers in conman.conf for clean
regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
+ manual fallback for already-discovered devices), create conmand
systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture
Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
/etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
/var/log/conman/. Fixed by adding server logdir directive.
Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/bash
|
||||
#
|
||||
# console/validate-conman.sh — verify conman can actually reach devices via
|
||||
# ser2net TCP ports and is capturing log output to files.
|
||||
#
|
||||
# This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device
|
||||
#
|
||||
set -uo pipefail
|
||||
|
||||
TS_IP=$(tailscale ip -4)
|
||||
LOGDIR="/var/log/conman"
|
||||
|
||||
echo "============================================"
|
||||
echo " Conman Data Path + Log Validation"
|
||||
echo " Host: $(hostname) TS IP: $TS_IP"
|
||||
echo "============================================"
|
||||
|
||||
echo ""
|
||||
echo "--- 1. conman.conf log settings ---"
|
||||
grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo " (no explicit logdir — defaults to /var/log/conman)"
|
||||
echo " Log dir: $LOGDIR"
|
||||
ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo " ($LOGDIR does not exist yet)"
|
||||
|
||||
echo ""
|
||||
echo "--- 2. CONSOLE entries: each has a log= directive? ---"
|
||||
# Extract the auto-generated block and check each CONSOLE line has log=
|
||||
sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do
|
||||
name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p')
|
||||
if echo "$line" | grep -q 'log='; then
|
||||
logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p')
|
||||
echo " [OK] $name → log=$logfile"
|
||||
else
|
||||
echo " [FAIL] $name has NO log= directive"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- 3. Trigger log capture: connect to each console briefly ---"
|
||||
# conman -e changes the escape char. We use -j (join, read-only) with a timeout.
|
||||
# Actually, conman doesn't have a built-in "connect for N seconds" — but conmand
|
||||
# connects to each device ON STARTUP and keeps the connection open for logging.
|
||||
# The log files should already be created. Let's check timestamps.
|
||||
|
||||
echo " conmand connects to all consoles on startup. Checking if logs exist..."
|
||||
echo ""
|
||||
echo "--- 4. Log file inventory ---"
|
||||
for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
|
||||
logfile="$LOGDIR/${name}.log"
|
||||
if [ -f "$logfile" ]; then
|
||||
SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
|
||||
MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1)
|
||||
echo " [OK] $logfile ($SIZE bytes, modified $MTIME)"
|
||||
else
|
||||
echo " [MISSING] $logfile — conmand may not be writing yet"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- 5. conmand connection status (journal) ---"
|
||||
# conmand logs connection attempts/errors to syslog
|
||||
journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo " (no relevant journal entries)"
|
||||
|
||||
echo ""
|
||||
echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---"
|
||||
# Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded)
|
||||
echo " Probing TCP $TS_IP:2007 for data..."
|
||||
RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
|
||||
if [ -n "$RESPONSE" ]; then
|
||||
echo " [OK] Data flowing through ser2net TCP 2007:"
|
||||
echo "$RESPONSE" | sed 's/^/ /'
|
||||
else
|
||||
echo " (no immediate response — device may need more interaction)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- 7. Check if conmand has open connections to ser2net ports ---"
|
||||
CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
|
||||
if [ -n "$CONMAND_PID" ]; then
|
||||
echo " conmand PID: $CONMAND_PID"
|
||||
echo " Open connections to ser2net (expect 7 to 100.x:200X):"
|
||||
ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/ /'
|
||||
COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
|
||||
echo " Total conmand→ser2net connections: $COUNT (expect 7)"
|
||||
else
|
||||
echo " [FAIL] conmand not running"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
Reference in New Issue
Block a user