prep for next ai session
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
# Console Management (ser2net + conman)
|
||||
|
||||
Network-accessible serial console management for all production network
|
||||
switches and routers, running on **pfv-tsys4** (storage server).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
|
||||
```
|
||||
|
||||
ser2net owns the physical serial devices and exposes them on TCP ports
|
||||
using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
|
||||
only** (`100.70.77.93:200X`). conman connects to those TCP ports via
|
||||
telnet for session logging, output capture, and multi-user console
|
||||
sharing.
|
||||
|
||||
**Why telnet(rfc2217)?** The serial devices send `
|
||||
␍` (LF+CR) line
|
||||
endings instead of standard `
|
||||
`. Raw TCP transport caused conman's
|
||||
telnet NVT to strip bare CR characters, producing stair-stepped output.
|
||||
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
|
||||
translation is handled correctly by the telnet layer.
|
||||
|
||||
**conman and ser2net do NOT share ports** — only one process can open a
|
||||
serial device at a time. ser2net owns the physical device; conman connects
|
||||
over TCP.
|
||||
|
||||
## The USB Enumeration Problem (SOLVED)
|
||||
|
||||
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
|
||||
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
|
||||
enumeration order, which shifts on every boot. This made the old
|
||||
`/root/conmap` + manual `screen` workflow break after every reboot.
|
||||
|
||||
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
|
||||
topology), which is stable across reboots regardless of enumeration order.
|
||||
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
|
||||
|
||||
The udev rules are generated from `mapping.txt`, which maps each adapter's
|
||||
ID_PATH to a console name and TCP port. To re-map after physically moving
|
||||
an adapter, update `mapping.txt` and re-run `setup.sh`.
|
||||
|
||||
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
|
||||
devices (common on first run), `setup.sh` creates them manually by matching
|
||||
ID_PATH. On subsequent boots, udev creates them automatically.
|
||||
|
||||
## Port Assignments
|
||||
|
||||
| TCP Port | Console Name | ID_PATH | Description |
|
||||
|----------|-------------|---------|-------------|
|
||||
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
|
||||
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
|
||||
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
|
||||
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
|
||||
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
|
||||
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
|
||||
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
|
||||
|
||||
All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
|
||||
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
|
||||
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
|
||||
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
|
||||
|
||||
## Usage
|
||||
|
||||
### Connect to a console
|
||||
|
||||
**Primary method — conman client (with logging + multiplexing):**
|
||||
|
||||
```bash
|
||||
# From any Tailscale-connected workstation:
|
||||
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
|
||||
conman -d pfv-tsys4:7890 -q # list all consoles
|
||||
```
|
||||
|
||||
Escape sequence: `&.` to disconnect, `&?` for help.
|
||||
|
||||
**Direct telnet (emergency only — conflicts with conman):**
|
||||
|
||||
```bash
|
||||
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
||||
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
||||
ssh pfv-tsys4 'systemctl stop conmand'
|
||||
telnet pfv-tsys4 2001 # pfv-core-sw01
|
||||
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
||||
```
|
||||
|
||||
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
||||
and kick your telnet session immediately ("Connection closed by foreign host").
|
||||
The correct workflow is conman client → conmand → ser2net → device.
|
||||
|
||||
### Re-deploy after changing mapping.txt
|
||||
|
||||
```bash
|
||||
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
|
||||
```
|
||||
|
||||
### Find the ID_PATH for a new adapter
|
||||
|
||||
```bash
|
||||
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
|
||||
```
|
||||
|
||||
Then match the new adapter's ID_PATH to its physical location and add a line
|
||||
to `mapping.txt`.
|
||||
|
||||
## Files on pfv-tsys4
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
|
||||
| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
|
||||
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
|
||||
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
|
||||
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
|
||||
| `/root/console/setup.sh` | Setup script (re-runnable) |
|
||||
| `/root/console/generate-config.sh` | Config generator |
|
||||
|
||||
## Old workflow (replaced)
|
||||
|
||||
The old `/root/conmap` file and manual `screen` sessions are no longer
|
||||
needed. The new setup is fully automated and survives reboots.
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/bash
|
||||
# shellcheck disable=SC2010,SC2012 # diagnostic script; ls|grep/ls -la on sysfs & log dirs is intentional for human-readable output
|
||||
#
|
||||
# console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4
|
||||
#
|
||||
# Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
|
||||
#
|
||||
# This script is strictly read-only. No writes to the system.
|
||||
#
|
||||
set -uo pipefail
|
||||
|
||||
echo "============================================"
|
||||
echo " Console Setup Discovery"
|
||||
echo " Host: $(hostname)"
|
||||
echo " Date: $(date)"
|
||||
echo " READ-ONLY"
|
||||
echo "============================================"
|
||||
|
||||
echo ""
|
||||
echo "=== 1. USB devices ==="
|
||||
lsusb 2>/dev/null || echo "(lsusb not available)"
|
||||
|
||||
echo ""
|
||||
echo "=== 2. All ttyUSB* devices (with major/minor) ==="
|
||||
ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
|
||||
|
||||
echo ""
|
||||
echo "=== 3. USB-serial driver bindings ==="
|
||||
echo "-- pl2303 --"
|
||||
ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
|
||||
echo "-- cp210x --"
|
||||
ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
|
||||
echo "-- ftdi_sio --"
|
||||
ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
|
||||
echo "-- ch341 --"
|
||||
ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
|
||||
|
||||
echo ""
|
||||
echo "=== 4. USB serial adapter details (vendor/model/serial per port) ==="
|
||||
for tty in /dev/ttyUSB*; do
|
||||
[ -e "$tty" ] || continue
|
||||
echo "--- $tty ---"
|
||||
udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/ /'
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "=== 5. Existing /root/conmap ==="
|
||||
if [ -f /root/conmap ]; then
|
||||
cat /root/conmap
|
||||
else
|
||||
echo "(no /root/conmap)"
|
||||
fi
|
||||
ls -la /root/conmap* 2>/dev/null
|
||||
|
||||
echo ""
|
||||
echo "=== 6. Screen sessions (running) ==="
|
||||
screen -ls 2>&1 || echo "(screen not running or not installed)"
|
||||
|
||||
echo ""
|
||||
echo "=== 7. Existing screen wrappers/scripts in /root ==="
|
||||
ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)"
|
||||
|
||||
echo ""
|
||||
echo "=== 8. ser2net ==="
|
||||
which ser2net 2>/dev/null || echo "(ser2net not installed)"
|
||||
dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)"
|
||||
cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)"
|
||||
systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)"
|
||||
|
||||
echo ""
|
||||
echo "=== 9. conman ==="
|
||||
which conman 2>/dev/null || echo "(conman not installed)"
|
||||
which conmand 2>/dev/null || echo "(conmand not installed)"
|
||||
dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)"
|
||||
echo "--- /etc/conman.conf (console lines only) ---"
|
||||
grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)"
|
||||
echo "--- conmand service ---"
|
||||
systemctl is-active conmand 2>/dev/null || echo "(conmand not running)"
|
||||
systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)"
|
||||
|
||||
echo ""
|
||||
echo "=== 10. Existing console logs ==="
|
||||
ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)"
|
||||
ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)"
|
||||
|
||||
echo ""
|
||||
echo "=== 11. udev rules for ttyUSB ==="
|
||||
grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)"
|
||||
grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true
|
||||
|
||||
echo ""
|
||||
echo "=== 12. expect availability ==="
|
||||
command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
|
||||
command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed"
|
||||
|
||||
echo ""
|
||||
echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ==="
|
||||
ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)"
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " Discovery complete (read-only)."
|
||||
echo "============================================"
|
||||
@@ -0,0 +1,254 @@
|
||||
#!/usr/bin/bash
|
||||
#
|
||||
# console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf
|
||||
#
|
||||
# Reads console/mapping.txt (the source of truth) and generates all three
|
||||
# config files. This is the fix for the USB enumeration shift problem:
|
||||
#
|
||||
# 1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
|
||||
# to a named symlink like /dev/consoles/pfv-core-sw01
|
||||
# 2. ser2net opens those stable symlinks and exposes them on TCP ports
|
||||
# (2001, 2002, ...) bound to the Tailscale IP
|
||||
# 3. conman connects to those TCP ports for logging + multiplexing
|
||||
#
|
||||
# Run this script ON the target host. It writes to:
|
||||
# /etc/udev/rules.d/99-console-ports.rules
|
||||
# /etc/ser2net.yaml
|
||||
# /etc/conman/console-consoles.conf (included by /etc/conman.conf)
|
||||
#
|
||||
# Usage:
|
||||
# PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh
|
||||
#
|
||||
# Environment overrides:
|
||||
# MAPPING_FILE — path to mapping.txt (default: auto-detect next to this script)
|
||||
# TS_IP — Tailscale IP to bind ser2net on (default: auto-detect)
|
||||
# CONMAN_LOGDIR — conman log directory (default: /var/log/conman)
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}"
|
||||
CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}"
|
||||
|
||||
UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
|
||||
SER2NET_CONF="/etc/ser2net.yaml"
|
||||
CONMAN_CONF="/etc/conman.conf"
|
||||
|
||||
echo "============================================"
|
||||
echo " Console Config Generator"
|
||||
echo " Host: $(hostname) $(date)"
|
||||
echo "============================================"
|
||||
|
||||
# --- Locate mapping file ---
|
||||
# When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong.
|
||||
# Search common locations.
|
||||
if [ ! -f "$MAPPING_FILE" ]; then
|
||||
for candidate in \
|
||||
"/root/console/mapping.txt" \
|
||||
"/tmp/mapping.txt" \
|
||||
"$(dirname "$0")/mapping.txt"; do
|
||||
if [ -f "$candidate" ]; then
|
||||
MAPPING_FILE="$candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [ ! -f "$MAPPING_FILE" ]; then
|
||||
echo "FATAL: mapping file not found. Tried: $MAPPING_FILE"
|
||||
echo "Copy mapping.txt to the target host first."
|
||||
exit 1
|
||||
fi
|
||||
echo " Mapping file: $MAPPING_FILE"
|
||||
|
||||
# --- Auto-detect Tailscale IP ---
|
||||
if [ -z "${TS_IP:-}" ]; then
|
||||
TS_IP=$(tailscale ip -4 2>/dev/null || true)
|
||||
if [ -z "$TS_IP" ]; then
|
||||
echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
echo " Tailscale IP: $TS_IP"
|
||||
echo " ser2net will bind to: $TS_IP"
|
||||
|
||||
# --- Parse mapping file (skip comments and blank lines) ---
|
||||
echo ""
|
||||
echo "--- Parsing mapping file ---"
|
||||
ENTRIES=()
|
||||
while IFS= read -r line; do
|
||||
# Skip comments and blank lines
|
||||
line="${line%%#*}"
|
||||
line="$(echo "$line" | xargs)" # trim whitespace
|
||||
[ -z "$line" ] && continue
|
||||
ENTRIES+=("$line")
|
||||
echo " $line"
|
||||
done < "$MAPPING_FILE"
|
||||
|
||||
if [ "${#ENTRIES[@]}" -eq 0 ]; then
|
||||
echo "FATAL: no entries found in mapping file."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo " ${#ENTRIES[@]} console ports configured."
|
||||
|
||||
# ============================================================
|
||||
# 1. Generate udev rules
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo "--- [1/3] Generating udev rules: $UDEV_RULES ---"
|
||||
|
||||
cat > "$UDEV_RULES" <<'UDEV_HEADER'
|
||||
# Stable symlinks for USB-DB9 console adapters
|
||||
# Generated by console/generate-config.sh
|
||||
# DO NOT EDIT — edit mapping.txt and re-run generate-config.sh
|
||||
#
|
||||
# These rules pin each adapter to a named symlink based on its physical
|
||||
# USB port path (ID_PATH), which is stable across reboots regardless of
|
||||
# enumeration order. This is the fix for the "USB adapters shift on reboot"
|
||||
# problem.
|
||||
#
|
||||
# To find the ID_PATH for a device:
|
||||
# udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH
|
||||
UDEV_HEADER
|
||||
|
||||
for entry in "${ENTRIES[@]}"; do
|
||||
IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
|
||||
# Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4"
|
||||
# The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0"
|
||||
# We match on the substring to be portable across PCI bus changes.
|
||||
{
|
||||
echo ""
|
||||
echo "# $name (TCP $tcp_port): $comment"
|
||||
echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"consoles/$name\""
|
||||
} >> "$UDEV_RULES"
|
||||
done
|
||||
|
||||
echo " Written: $UDEV_RULES"
|
||||
echo " Symlinks: /dev/consoles/<name> for each device"
|
||||
|
||||
# ============================================================
|
||||
# 2. Generate ser2net.yaml
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---"
|
||||
|
||||
# Backup existing config if not already backed up
|
||||
if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then
|
||||
cp "$SER2NET_CONF" "${SER2NET_CONF}.orig"
|
||||
echo " Backed up original to ${SER2NET_CONF}.orig"
|
||||
fi
|
||||
|
||||
{
|
||||
echo "%YAML 1.1"
|
||||
echo "---"
|
||||
echo "# ser2net configuration for pfv-tsys4 console ports"
|
||||
echo "# Generated by console/generate-config.sh on $(date)"
|
||||
echo "#"
|
||||
echo "# All ports use telnet(rfc2217) accepter so conman and telnet clients"
|
||||
echo "# negotiate proper telnet binary mode — this prevents CR stripping"
|
||||
printf '%s\n' "# and stair-stepping on devices that send \\n\\r (LF+CR) line endings."
|
||||
echo "# Ports bound to Tailscale IP ($TS_IP) for secure remote access."
|
||||
echo "#"
|
||||
echo "# Direct telnet: telnet $TS_IP 2001"
|
||||
echo "# Via conman: conman -f <name>"
|
||||
echo ""
|
||||
printf '%s\n' "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
|
||||
echo ""
|
||||
|
||||
for entry in "${ENTRIES[@]}"; do
|
||||
IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
|
||||
# ser2net connection block — telnet(rfc2217) accepter so conman and
|
||||
# telnet clients negotiate proper telnet binary mode. This prevents
|
||||
# CR stripping that occurs with raw TCP + conman's telnet NVT.
|
||||
echo "connection: &con${tcp_port}"
|
||||
echo " accepter: telnet(rfc2217),tcp,${TS_IP},${tcp_port}"
|
||||
echo " enable: on"
|
||||
echo " options:"
|
||||
echo " banner: *banner"
|
||||
echo " kickolduser: true"
|
||||
echo " telnet-brk-on-sync: true"
|
||||
echo " connector: serialdev,"
|
||||
echo " /dev/consoles/${name},"
|
||||
echo " ${baud},local"
|
||||
echo ""
|
||||
done
|
||||
} > "$SER2NET_CONF"
|
||||
|
||||
echo " Written: $SER2NET_CONF"
|
||||
echo " ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)"
|
||||
|
||||
# ============================================================
|
||||
# 3. Write conman console entries directly into conman.conf
|
||||
# ============================================================
|
||||
# conman 0.3.x does NOT support the 'include' directive, so we write
|
||||
# CONSOLE entries directly into /etc/conman.conf between idempotent markers.
|
||||
echo ""
|
||||
echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---"
|
||||
|
||||
# Ensure logdir exists
|
||||
mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true
|
||||
|
||||
# Ensure LOGDIR is set in conman.conf (server-level directive for log file paths)
|
||||
if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
|
||||
# Insert near the top, after the first SERVER directives
|
||||
sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF"
|
||||
echo " Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
|
||||
fi
|
||||
|
||||
# Ensure loopback=off so conmand is reachable over Tailscale (not localhost-only)
|
||||
if ! grep -qiE '^\s*server\s+loopback\s*=' "$CONMAN_CONF" 2>/dev/null; then
|
||||
sed -i "/^server logdir/a server loopback=off" "$CONMAN_CONF"
|
||||
echo " Added server loopback=off to $CONMAN_CONF (enables remote access)"
|
||||
fi
|
||||
|
||||
# Remove any previous auto-generated block (between markers)
|
||||
# Then append the new block
|
||||
MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
|
||||
MARKER_END="# END PFV CONSOLE DEFINITIONS"
|
||||
|
||||
# Strip old block if present
|
||||
if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then
|
||||
sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF"
|
||||
echo " Removed previous console definitions."
|
||||
fi
|
||||
|
||||
# Append new block
|
||||
{
|
||||
echo ""
|
||||
echo "$MARKER_BEGIN"
|
||||
echo "# Generated by console/generate-config.sh on $(date)"
|
||||
echo "# Each console connects to a ser2net TCP port via telnet protocol."
|
||||
echo "# ser2net uses telnet(rfc2217) accepter so binary mode is negotiated"
|
||||
echo "# and CR/LF translation is handled correctly by the telnet NVT layer."
|
||||
echo "# Access: conman -f <name>"
|
||||
echo ""
|
||||
for entry in "${ENTRIES[@]}"; do
|
||||
IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
|
||||
echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
|
||||
done
|
||||
echo "$MARKER_END"
|
||||
} >> "$CONMAN_CONF"
|
||||
|
||||
CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0)
|
||||
echo " Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF"
|
||||
|
||||
# ============================================================
|
||||
# Summary
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " Configuration generated successfully."
|
||||
echo ""
|
||||
echo " Files written:"
|
||||
echo " $UDEV_RULES ($(wc -l < "$UDEV_RULES") lines)"
|
||||
echo " $SER2NET_CONF ($(wc -l < "$SER2NET_CONF") lines)"
|
||||
echo " $CONMAN_CONF (CONSOLE entries appended between markers)"
|
||||
echo ""
|
||||
echo " Next steps:"
|
||||
echo " 1. Reload udev: udevadm control --reload-rules && udevadm trigger"
|
||||
echo " 2. Restart ser2net: systemctl restart ser2net"
|
||||
echo " 3. Start conman: systemctl enable --now conmand"
|
||||
echo " 4. Or run: bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
|
||||
echo "============================================"
|
||||
@@ -0,0 +1,29 @@
|
||||
# console/mapping.txt — Source of Truth for console port assignments
|
||||
#
|
||||
# Format: <tcp_port>|<name>|<id_path_substring>|<baud>|<comment>
|
||||
#
|
||||
# Delimiter is | (pipe) because ID_PATH values contain colons.
|
||||
#
|
||||
# - tcp_port: TCP port ser2net listens on (also the conman console name suffix)
|
||||
# - name: Device name (used for /dev/console/<name> symlink, conman console name)
|
||||
# - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH`
|
||||
# These are STABLE across reboots as long as adapters aren't moved
|
||||
# to different physical USB ports.
|
||||
# - baud: Serial baud rate (9600n81 = 9600 8N1, no flow control)
|
||||
# - comment: Free-form description
|
||||
#
|
||||
# To RE-MAP after physically moving an adapter:
|
||||
# 1. Run: bash console/discover.sh (find the new ID_PATH for the device)
|
||||
# 2. Update the id_path_substring in this file
|
||||
# 3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
|
||||
#
|
||||
2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
|
||||
2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
|
||||
2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
|
||||
2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
|
||||
2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
|
||||
2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
|
||||
2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
|
||||
# Unassigned (no device detected):
|
||||
# 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
|
||||
# 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
|
||||
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/bash
|
||||
#
|
||||
# console/query-remote.sh — install conman client and connect to a console
|
||||
# on pfv-tsys4 over Tailscale.
|
||||
#
|
||||
# Usage:
|
||||
# bash console/query-remote.sh # list consoles
|
||||
# bash console/query-remote.sh pfv-core-sw01 # connect to a console
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
REMOTE_HOST="${REMOTE_HOST:-pfv-tsys4}"
|
||||
REMOTE_PORT="${REMOTE_PORT:-7890}"
|
||||
|
||||
echo "============================================"
|
||||
echo " Conman Remote Console Access"
|
||||
echo " Server: ${REMOTE_HOST}:${REMOTE_PORT} (Tailscale)"
|
||||
echo "============================================"
|
||||
|
||||
# --- 1. Install conman client if missing ---
|
||||
if ! command -v conman >/dev/null 2>&1; then
|
||||
echo ""
|
||||
echo "--- Installing conman client ---"
|
||||
if sudo -n true 2>/dev/null; then
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq conman
|
||||
else
|
||||
echo " Passwordless sudo not available. Please run:"
|
||||
echo " sudo apt-get update && sudo apt-get install -y conman"
|
||||
echo " Then re-run this script."
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo " conman client already installed."
|
||||
fi
|
||||
|
||||
# --- 2. Verify connectivity ---
|
||||
echo ""
|
||||
echo "--- Connectivity check ---"
|
||||
if timeout 3 bash -c "echo > /dev/tcp/${REMOTE_HOST}/${REMOTE_PORT}" 2>/dev/null; then
|
||||
echo " [OK] ${REMOTE_HOST}:${REMOTE_PORT} reachable"
|
||||
else
|
||||
echo " [FAIL] Cannot reach ${REMOTE_HOST}:${REMOTE_PORT}"
|
||||
echo " Is Tailscale up? Is conmand running on ${REMOTE_HOST}?"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. List or connect ---
|
||||
CONSOLE="${1:-}"
|
||||
if [ -z "$CONSOLE" ]; then
|
||||
echo ""
|
||||
echo "--- Available consoles ---"
|
||||
conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -q
|
||||
echo ""
|
||||
echo "To connect: bash $0 <console-name>"
|
||||
echo " e.g: bash $0 pfv-core-sw01"
|
||||
else
|
||||
echo ""
|
||||
echo "--- Connecting to: $CONSOLE ---"
|
||||
echo " Escape sequence: &. (to disconnect)"
|
||||
echo ""
|
||||
conman -d "${REMOTE_HOST}:${REMOTE_PORT}" -f "$CONSOLE"
|
||||
fi
|
||||
@@ -0,0 +1,217 @@
|
||||
#!/usr/bin/bash
|
||||
# shellcheck disable=SC2010 # diagnostic; ls|grep on /dev listing is intentional
|
||||
#
|
||||
# console/setup.sh — deploy console management on pfv-tsys4
|
||||
#
|
||||
# Orchestrates the full setup:
|
||||
# 1. Ensures ser2net + conman are installed
|
||||
# 2. Copies mapping.txt to the target host (if running remotely)
|
||||
# 3. Runs generate-config.sh to produce udev rules + ser2net.yaml + conman.conf
|
||||
# 4. Reloads udev, creates /dev/consoles/ symlinks
|
||||
# 5. Restarts ser2net (TCP ports on Tailscale IP)
|
||||
# 6. Enables + starts conmand (logging + multiplexing)
|
||||
# 7. Verifies
|
||||
#
|
||||
# This script is IDEMPOTENT — safe to run multiple times.
|
||||
#
|
||||
# Usage:
|
||||
# PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/setup.sh
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
echo "============================================"
|
||||
echo " Console Management Setup"
|
||||
echo " Host: $(hostname) $(date)"
|
||||
echo "============================================"
|
||||
|
||||
# --- 1. Install dependencies ---
|
||||
echo ""
|
||||
echo "--- [1/7] Checking dependencies ---"
|
||||
NEED_INSTALL=()
|
||||
dpkg -l ser2net 2>/dev/null | grep -q '^ii' && echo " ser2net: installed" || NEED_INSTALL+=(ser2net)
|
||||
dpkg -l conman 2>/dev/null | grep -q '^ii' && echo " conman: installed" || NEED_INSTALL+=(conman)
|
||||
|
||||
if [ "${#NEED_INSTALL[@]}" -gt 0 ]; then
|
||||
echo " Installing: ${NEED_INSTALL[*]}"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq "${NEED_INSTALL[@]}"
|
||||
else
|
||||
echo " All dependencies present."
|
||||
fi
|
||||
|
||||
# --- 2. Ensure mapping file is available ---
|
||||
echo ""
|
||||
echo "--- [2/7] Locating mapping file ---"
|
||||
|
||||
MAPPING_FILE=""
|
||||
for candidate in \
|
||||
"$SCRIPT_DIR/mapping.txt" \
|
||||
"$(dirname "$0")/mapping.txt" \
|
||||
"/root/console/mapping.txt" \
|
||||
"/tmp/mapping.txt"; do
|
||||
if [ -f "$candidate" ]; then
|
||||
MAPPING_FILE="$candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$MAPPING_FILE" ]; then
|
||||
echo "FATAL: mapping.txt not found. Copy it to the target host."
|
||||
exit 1
|
||||
fi
|
||||
echo " Using: $MAPPING_FILE"
|
||||
|
||||
# --- 3. Generate configs ---
|
||||
echo ""
|
||||
echo "--- [3/7] Generating configs ---"
|
||||
export MAPPING_FILE
|
||||
bash "$(dirname "$0")/generate-config.sh" 2>&1 || bash "$SCRIPT_DIR/generate-config.sh" 2>&1 || {
|
||||
echo "FATAL: generate-config.sh failed."
|
||||
exit 1
|
||||
}
|
||||
|
||||
# --- 4. Reload udev + create symlinks ---
|
||||
echo ""
|
||||
echo "--- [4/7] Reloading udev rules ---"
|
||||
udevadm control --reload-rules
|
||||
# Try trigger first (works on some systems)
|
||||
for tty in /sys/class/tty/ttyUSB*; do
|
||||
[ -e "$tty" ] && udevadm trigger --action=add "$tty" 2>/dev/null || true
|
||||
done
|
||||
# Also try writing to uevent (forces udev reprocessing)
|
||||
for tty in /sys/class/tty/ttyUSB*; do
|
||||
[ -e "$tty/uevent" ] && echo "add" > "$tty/uevent" 2>/dev/null || true
|
||||
done
|
||||
sleep 2
|
||||
|
||||
# FALLBACK: if udev symlinks don't exist (common when devices are already
|
||||
# discovered — udev trigger doesn't always re-create symlinks for existing
|
||||
# devices), create them manually by matching ID_PATH. The udev rules will
|
||||
# handle future boots/hotplugs automatically.
|
||||
if [ ! -d /dev/consoles ] || [ -z "$(ls /dev/consoles/ 2>/dev/null)" ]; then
|
||||
echo " udev trigger didn't create symlinks. Creating manually..."
|
||||
mkdir -p /dev/consoles
|
||||
while IFS= read -r line; do
|
||||
line="${line%%#*}"
|
||||
line="$(echo "$line" | xargs)"
|
||||
[ -z "$line" ] && continue
|
||||
IFS='|' read -r _ name id_path _ _ <<< "$line"
|
||||
# Find the ttyUSB whose ID_PATH contains the mapping's id_path substring
|
||||
for tty in /dev/ttyUSB*; do
|
||||
[ -e "$tty" ] || continue
|
||||
DEV_IDPATH=$(udevadm info -q property -n "$tty" 2>/dev/null | grep ^ID_PATH= | cut -d= -f2)
|
||||
if echo "$DEV_IDPATH" | grep -q "$id_path"; then
|
||||
ln -sf "$tty" "/dev/consoles/$name"
|
||||
echo " ln -s $tty -> /dev/consoles/$name"
|
||||
break
|
||||
fi
|
||||
done
|
||||
done < "$MAPPING_FILE"
|
||||
fi
|
||||
|
||||
echo " Stable symlinks:"
|
||||
ls -la /dev/consoles/ 2>/dev/null | grep -v '^total\|^d' | sed 's/^/ /' || echo " (none created)"
|
||||
|
||||
# Verify each symlink resolves
|
||||
echo ""
|
||||
echo " Symlink verification:"
|
||||
while IFS= read -r line; do
|
||||
line="${line%%#*}"
|
||||
line="$(echo "$line" | xargs)"
|
||||
[ -z "$line" ] && continue
|
||||
IFS='|' read -r _ name id_path _ _ <<< "$line"
|
||||
if [ -e "/dev/consoles/$name" ]; then
|
||||
TARGET=$(readlink -f "/dev/consoles/$name")
|
||||
echo " [OK] /dev/consoles/$name -> $TARGET"
|
||||
else
|
||||
echo " [MISSING] /dev/consoles/$name (adapter unplugged or ID_PATH changed)"
|
||||
fi
|
||||
done < "$MAPPING_FILE"
|
||||
|
||||
# --- 5. Restart ser2net ---
|
||||
echo ""
|
||||
echo "--- [5/7] Restarting ser2net ---"
|
||||
systemctl enable ser2net
|
||||
systemctl restart ser2net
|
||||
sleep 2
|
||||
|
||||
if systemctl is-active --quiet ser2net; then
|
||||
echo " ser2net is running (telnet rfc2217 accepters)."
|
||||
TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1")
|
||||
echo " Listening ports:"
|
||||
ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/ /'
|
||||
else
|
||||
echo " WARNING: ser2net failed to start. Checking journal..."
|
||||
journalctl -u ser2net --no-pager -n 20
|
||||
fi
|
||||
|
||||
# --- 6. Enable + start conmand ---
|
||||
echo ""
|
||||
echo "--- [6/7] Starting conmand ---"
|
||||
|
||||
# conman package on Debian may not ship a systemd unit. Create one if missing.
|
||||
if ! systemctl cat conmand >/dev/null 2>&1; then
|
||||
echo " No systemd unit for conmand — creating one..."
|
||||
cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT'
|
||||
[Unit]
|
||||
Description=ConMan (Console Manager)
|
||||
After=network.target ser2net.service
|
||||
Requires=ser2net.service
|
||||
|
||||
[Service]
|
||||
Type=forking
|
||||
ExecStart=/usr/sbin/conmand -c /etc/conman.conf
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
CONMAND_UNIT
|
||||
systemctl daemon-reload
|
||||
echo " Created /etc/systemd/system/conmand.service"
|
||||
fi
|
||||
|
||||
# Kill any manually-started conmand first
|
||||
pkill -x conmand 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
systemctl enable conmand 2>/dev/null || true
|
||||
systemctl restart conmand 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
if systemctl is-active --quiet conmand; then
|
||||
echo " conmand is running."
|
||||
echo " Consoles:"
|
||||
conman -q 2>&1 | sed 's/^/ /' || true
|
||||
else
|
||||
echo " WARNING: conmand failed to start. Checking journal..."
|
||||
journalctl -u conmand --no-pager -n 20 2>/dev/null || true
|
||||
# Try manual start as fallback
|
||||
echo " Attempting manual start..."
|
||||
/usr/sbin/conmand -c /etc/conman.conf 2>&1 || true
|
||||
fi
|
||||
|
||||
# --- 7. Summary ---
|
||||
echo ""
|
||||
echo "--- [7/7] Setup complete ---"
|
||||
echo ""
|
||||
echo " ser2net + conman architecture (telnet rfc2217):"
|
||||
echo " ser2net owns serial devices, exposes telnet(rfc2217) TCP ports"
|
||||
echo " conman connects via telnet for logging + multiplexing"
|
||||
echo ""
|
||||
echo " Connect from any Tailscale workstation:"
|
||||
echo " conman -d pfv-tsys4:7890 -f pfv-core-sw01"
|
||||
echo " conman -d pfv-tsys4:7890 -q # list consoles"
|
||||
echo ""
|
||||
echo " Direct telnet (emergency, conflicts with conman):"
|
||||
echo " ssh pfv-tsys4 'systemctl stop conmand'"
|
||||
echo " telnet pfv-tsys4 2001"
|
||||
echo " ssh pfv-tsys4 'systemctl start conmand'"
|
||||
echo ""
|
||||
echo " To regenerate after changing mapping.txt:"
|
||||
echo " bash generate-config.sh"
|
||||
echo " udevadm trigger"
|
||||
echo " systemctl restart ser2net conmand"
|
||||
echo "============================================"
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/bash
|
||||
# shellcheck disable=SC2012,SC2001 # diagnostic script; ls -la listings and sed line-prefixing are intentional
|
||||
#
|
||||
# console/validate-conman.sh — verify conman can actually reach devices via
|
||||
# ser2net TCP ports and is capturing log output to files.
|
||||
#
|
||||
# This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device
|
||||
#
|
||||
set -uo pipefail
|
||||
|
||||
TS_IP=$(tailscale ip -4)
|
||||
LOGDIR="/var/log/conman"
|
||||
|
||||
echo "============================================"
|
||||
echo " Conman Data Path + Log Validation"
|
||||
echo " Host: $(hostname) TS IP: $TS_IP"
|
||||
echo "============================================"
|
||||
|
||||
echo ""
|
||||
echo "--- 1. conman.conf log settings ---"
|
||||
grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo " (no explicit logdir — defaults to /var/log/conman)"
|
||||
echo " Log dir: $LOGDIR"
|
||||
ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo " ($LOGDIR does not exist yet)"
|
||||
|
||||
echo ""
|
||||
echo "--- 2. CONSOLE entries: each has a log= directive? ---"
|
||||
# Extract the auto-generated block and check each CONSOLE line has log=
|
||||
sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do
|
||||
name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p')
|
||||
if echo "$line" | grep -q 'log='; then
|
||||
logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p')
|
||||
echo " [OK] $name → log=$logfile"
|
||||
else
|
||||
echo " [FAIL] $name has NO log= directive"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- 3. Trigger log capture: connect to each console briefly ---"
|
||||
# conman -e changes the escape char. We use -j (join, read-only) with a timeout.
|
||||
# Actually, conman doesn't have a built-in "connect for N seconds" — but conmand
|
||||
# connects to each device ON STARTUP and keeps the connection open for logging.
|
||||
# The log files should already be created. Let's check timestamps.
|
||||
|
||||
echo " conmand connects to all consoles on startup. Checking if logs exist..."
|
||||
echo ""
|
||||
echo "--- 4. Log file inventory ---"
|
||||
for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
|
||||
logfile="$LOGDIR/${name}.log"
|
||||
if [ -f "$logfile" ]; then
|
||||
SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
|
||||
MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1)
|
||||
echo " [OK] $logfile ($SIZE bytes, modified $MTIME)"
|
||||
else
|
||||
echo " [MISSING] $logfile — conmand may not be writing yet"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- 5. conmand connection status (journal) ---"
|
||||
# conmand logs connection attempts/errors to syslog
|
||||
journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo " (no relevant journal entries)"
|
||||
|
||||
echo ""
|
||||
echo "--- 6. Verify ser2net is proxying data (telnet rfc2217) ---"
|
||||
echo " Probing TCP $TS_IP:2007 for data..."
|
||||
RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
|
||||
if [ -n "$RESPONSE" ]; then
|
||||
echo " [OK] Data flowing through ser2net TCP 2007:"
|
||||
echo "$RESPONSE" | sed 's/^/ /'
|
||||
else
|
||||
echo " (no immediate response — device may need more interaction)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- 7. Check if conmand has open connections to ser2net ports ---"
|
||||
CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
|
||||
if [ -n "$CONMAND_PID" ]; then
|
||||
echo " conmand PID: $CONMAND_PID"
|
||||
echo " Open connections to ser2net (expect 7 to 100.x:200X):"
|
||||
ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/ /'
|
||||
COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
|
||||
echo " Total conmand→ser2net connections: $COUNT (expect 7)"
|
||||
else
|
||||
echo " [FAIL] conmand not running"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
Reference in New Issue
Block a user