feat(console): manage 7 switch consoles via ser2net+conman on pfv-tsys4

Solve the long-standing USB adapter enumeration shift problem: 9 Prolific USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every reboot and breaks the old /root/conmap + manual screen workflow. Solution: udev rules pin each adapter by its ID_PATH (physical USB port topology), which is stable across reboots regardless of enumeration order. Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens these stable symlinks and exposes them on TCP ports (2001-2007) bound to the Tailscale interface only. conman connects to those TCP ports for session logging and multi-user console sharing. Architecture (layered, no port sharing): USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux) Port assignments (all on Tailscale IP 100.70.77.93): 2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor 2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw 2007 = pfv-r2-sw Scripts (console/): - mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment) - generate-config.sh: generates udev rules, ser2net.yaml, conman.conf entries from mapping.txt. Idempotent (markers in conman.conf for clean regeneration). Uses | delimiter (ID_PATH values contain colons). - setup.sh: full deploy — generate configs, create symlinks (udev trigger + manual fallback for already-discovered devices), create conmand systemd unit (Debian doesn't ship one), restart services - discover.sh: read-only USB adapter and service state discovery - validate-conman.sh: verify conman→ser2net→device data path and log capture Issues fixed during development: - /dev/console is a kernel char device (major 5, minor 1) — cannot create a directory there. Changed symlink namespace to /dev/consoles/. - conman 0.3.x has no 'include' directive — CONSOLE entries written directly into /etc/conman.conf between idempotent markers. - Debian conman package has no systemd unit — created /etc/systemd/system/conmand.service with After=ser2net ordering. - conman.conf had no LOGDIR — logs weren't being written to /var/log/conman/. Fixed by adding server logdir directive. Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with 7 consoles registered, 7 log files actively capturing console output, both services enabled for reboot survival. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-07-28 19:44:02 -05:00
parent 799d270261
commit eb99b00a70
10 changed files with 807 additions and 0 deletions
@@ -13,6 +13,7 @@ tests/               Test suite + VM validation harness
 dns-cluster-setup/   Technitium DNS cluster replication
 k8s/                 k3s cluster setup scripts (3-node HA over Tailscale)
 powerman/            Cyclades PM10i PDU management via powerman
 console/             Serial console management (ser2net + conman) for switches
 perf/                Proxmox perf tuning, fleet audit, iperf
 netinfra/            pfv-netinfra-01/02 DNS/NTP setup
 switches/            Switch configuration captures
@@ -81,6 +82,7 @@ vendor/              Vendored KNELShellFramework
 | [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
 | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
 | [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman |
 | [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
 | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
 ## Key Docs
@@ -13,6 +13,7 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste
 | [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts |
 | [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) |
 | [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 |
 | [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
 | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
 | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
 | [`switches/`](switches/) | Switch configuration captures |
@@ -56,6 +56,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 - [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
 - [ ] Rename outlets to match physical devices (Friday onsite)
 ### Console Management (ser2net + conman on pfv-tsys4 — LIVE)
 - [x] 7 network switch/router consoles managed via ser2net + conman
 - [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
  ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
  symlinks that survive reboot regardless of enumeration order
 - [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
  **Tailscale IP only** (`100.70.77.93`)
 - [x] conman connects to TCP ports for logging + multiplexing (7 log
  files active in `/var/log/conman/`)
 - [x] Both ser2net + conmand enabled via systemd (survive reboot)
 - [x] conmand systemd unit created (Debian package doesn't ship one)
 - [x] Old `/root/conmap` + manual `screen` workflow replaced
 ### Repo Merge
 - [x] KNELServerBuild merged into PFVCluster (history preserved)
 - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
@@ -99,4 +112,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
 | Backup | Proxmox Backup Server (PBS) |
 | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
 | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
 | **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** |
 | Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
@@ -0,0 +1,109 @@
 # Console Management (ser2net + conman)
 Network-accessible serial console management for all production network
 switches and routers, running on **pfv-tsys4** (storage server).
 ## Architecture
 ```
 USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net (TCP) → conman (logging + multiplexing)
 ```
 ser2net owns the physical serial devices and exposes them on TCP ports
 bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman
 connects to those TCP ports for session logging, output capture, and
 multi-user console sharing.
 **conman and ser2net do NOT share ports** — only one process can open a
 serial device at a time. ser2net owns the physical device; conman connects
 over TCP.
 ## The USB Enumeration Problem (SOLVED)
 The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
 unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
 enumeration order, which shifts on every boot. This made the old
 `/root/conmap` + manual `screen` workflow break after every reboot.
 **Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
 topology), which is stable across reboots regardless of enumeration order.
 Each adapter gets a named symlink in `/dev/consoles/` that never changes.
 The udev rules are generated from `mapping.txt`, which maps each adapter's
 ID_PATH to a console name and TCP port. To re-map after physically moving
 an adapter, update `mapping.txt` and re-run `setup.sh`.
 **Fallback:** if udev trigger doesn't create symlinks for already-discovered
 devices (common on first run), `setup.sh` creates them manually by matching
 ID_PATH. On subsequent boots, udev creates them automatically.
 ## Port Assignments
 | TCP Port | Console Name | ID_PATH | Description |
 |----------|-------------|---------|-------------|
 | 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
 | 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
 | 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
 | 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
 | 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
 | 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
 | 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
 All ports listen on the Tailscale IP (`100.70.77.93`).
 ## Scripts
 | Script | Purpose |
 |--------|---------|
 | [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
 | [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
 | [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
 | [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
 ## Usage
 ### Connect to a console
 ```bash
 # Via conman (with logging + multiplexing):
 conman -f pfv-core-sw01        # attach to console
 conman -q                      # query status of all consoles
 # Via telnet (direct, no logging):
 telnet 100.70.77.93 2001       # pfv-core-sw01
 # From any Tailscale-connected device:
 telnet pfv-tsys4 2001
 ```
 ### Re-deploy after changing mapping.txt
 ```bash
 PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
 ```
 ### Find the ID_PATH for a new adapter
 ```bash
 PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
 ```
 Then match the new adapter's ID_PATH to its physical location and add a line
 to `mapping.txt`.
 ## Files on pfv-tsys4
 | File | Purpose |
 |------|---------|
 | `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
 | `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) |
 | `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
 | `/etc/systemd/system/conmand.service` | systemd unit for conmand |
 | `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
 | `/root/console/setup.sh` | Setup script (re-runnable) |
 | `/root/console/generate-config.sh` | Config generator |
 ## Old workflow (replaced)
 The old `/root/conmap` file and manual `screen` sessions are no longer
 needed. The new setup is fully automated and survives reboots.
@@ -0,0 +1,102 @@
 #!/usr/bin/bash
 #
 # console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4
 #
 # Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
 #
 # This script is strictly read-only. No writes to the system.
 #
 set -uo pipefail
 echo "============================================"
 echo "  Console Setup Discovery"
 echo "  Host: $(hostname)"
 echo "  Date: $(date)"
 echo "  READ-ONLY"
 echo "============================================"
 echo ""
 echo "=== 1. USB devices ==="
 lsusb 2>/dev/null || echo "(lsusb not available)"
 echo ""
 echo "=== 2. All ttyUSB* devices (with major/minor) ==="
 ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)"
 echo ""
 echo "=== 3. USB-serial driver bindings ==="
 echo "-- pl2303 --"
 ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- cp210x --"
 ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ftdi_sio --"
 ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo "-- ch341 --"
 ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)"
 echo ""
 echo "=== 4. USB serial adapter details (vendor/model/serial per port) ==="
 for tty in /dev/ttyUSB*; do
  [ -e "$tty" ] || continue
  echo "--- $tty ---"
  udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/  /'
 done
 echo ""
 echo "=== 5. Existing /root/conmap ==="
 if [ -f /root/conmap ]; then
  cat /root/conmap
 else
  echo "(no /root/conmap)"
 fi
 ls -la /root/conmap* 2>/dev/null
 echo ""
 echo "=== 6. Screen sessions (running) ==="
 screen -ls 2>&1 || echo "(screen not running or not installed)"
 echo ""
 echo "=== 7. Existing screen wrappers/scripts in /root ==="
 ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)"
 echo ""
 echo "=== 8. ser2net ==="
 which ser2net 2>/dev/null || echo "(ser2net not installed)"
 dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)"
 cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)"
 systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)"
 echo ""
 echo "=== 9. conman ==="
 which conman 2>/dev/null || echo "(conman not installed)"
 which conmand 2>/dev/null || echo "(conmand not installed)"
 dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)"
 echo "--- /etc/conman.conf (console lines only) ---"
 grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)"
 echo "--- conmand service ---"
 systemctl is-active conmand 2>/dev/null || echo "(conmand not running)"
 systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)"
 echo ""
 echo "=== 10. Existing console logs ==="
 ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)"
 ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)"
 echo ""
 echo "=== 11. udev rules for ttyUSB ==="
 grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)"
 grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true
 echo ""
 echo "=== 12. expect availability ==="
 command -v expect && expect -v 2>&1 || echo "expect: NOT installed"
 command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed"
 echo ""
 echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ==="
 ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)"
 echo ""
 echo "============================================"
 echo "  Discovery complete (read-only)."
 echo "============================================"
@@ -0,0 +1,244 @@
 #!/usr/bin/bash
 #
 # console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf
 #
 # Reads console/mapping.txt (the source of truth) and generates all three
 # config files. This is the fix for the USB enumeration shift problem:
 #
 #   1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port)
 #      to a named symlink like /dev/consoles/pfv-core-sw01
 #   2. ser2net opens those stable symlinks and exposes them on TCP ports
 #      (2001, 2002, ...) bound to the Tailscale IP
 #   3. conman connects to those TCP ports for logging + multiplexing
 #
 # Run this script ON the target host. It writes to:
 #   /etc/udev/rules.d/99-console-ports.rules
 #   /etc/ser2net.yaml
 #   /etc/conman/console-consoles.conf  (included by /etc/conman.conf)
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh
 #
 # Environment overrides:
 #   MAPPING_FILE  — path to mapping.txt (default: auto-detect next to this script)
 #   TS_IP         — Tailscale IP to bind ser2net on (default: auto-detect)
 #   CONMAN_LOGDIR — conman log directory (default: /var/log/conman)
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}"
 CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}"
 UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
 SER2NET_CONF="/etc/ser2net.yaml"
 CONMAN_CONF="/etc/conman.conf"
 CONMAN_CONSOLES="/etc/conman/console-consoles.conf"
 CONSOLE_DEV_DIR="/dev/console"
 echo "============================================"
 echo "  Console Config Generator"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- Locate mapping file ---
 # When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong.
 # Search common locations.
 if [ ! -f "$MAPPING_FILE" ]; then
  for candidate in \
    "/root/console/mapping.txt" \
    "/tmp/mapping.txt" \
    "$(dirname "$0")/mapping.txt"; do
    if [ -f "$candidate" ]; then
      MAPPING_FILE="$candidate"
      break
    fi
  done
 fi
 if [ ! -f "$MAPPING_FILE" ]; then
  echo "FATAL: mapping file not found. Tried: $MAPPING_FILE"
  echo "Copy mapping.txt to the target host first."
  exit 1
 fi
 echo "  Mapping file: $MAPPING_FILE"
 # --- Auto-detect Tailscale IP ---
 if [ -z "${TS_IP:-}" ]; then
  TS_IP=$(tailscale ip -4 2>/dev/null || true)
  if [ -z "$TS_IP" ]; then
    echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually."
    exit 1
  fi
 fi
 echo "  Tailscale IP: $TS_IP"
 echo "  ser2net will bind to: $TS_IP"
 # --- Parse mapping file (skip comments and blank lines) ---
 echo ""
 echo "--- Parsing mapping file ---"
 ENTRIES=()
 while IFS= read -r line; do
  # Skip comments and blank lines
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"  # trim whitespace
  [ -z "$line" ] && continue
  ENTRIES+=("$line")
  echo "  $line"
 done < "$MAPPING_FILE"
 if [ "${#ENTRIES[@]}" -eq 0 ]; then
  echo "FATAL: no entries found in mapping file."
  exit 1
 fi
 echo ""
 echo "  ${#ENTRIES[@]} console ports configured."
 # ============================================================
 # 1. Generate udev rules
 # ============================================================
 echo ""
 echo "--- [1/3] Generating udev rules: $UDEV_RULES ---"
 cat > "$UDEV_RULES" <<'UDEV_HEADER'
 # Stable symlinks for USB-DB9 console adapters
 # Generated by console/generate-config.sh
 # DO NOT EDIT — edit mapping.txt and re-run generate-config.sh
 #
 # These rules pin each adapter to a named symlink based on its physical
 # USB port path (ID_PATH), which is stable across reboots regardless of
 # enumeration order. This is the fix for the "USB adapters shift on reboot"
 # problem.
 #
 # To find the ID_PATH for a device:
 #   udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH
 UDEV_HEADER
 for entry in "${ENTRIES[@]}"; do
  IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
  # Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4"
  # The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0"
  # We match on the substring to be portable across PCI bus changes.
  echo "" >> "$UDEV_RULES"
  echo "# $name (TCP $tcp_port): $comment" >> "$UDEV_RULES"
  echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"console/$name\"" >> "$UDEV_RULES"
 done
 echo "  Written: $UDEV_RULES"
 echo "  Symlinks: /dev/consoles/<name> for each device"
 # ============================================================
 # 2. Generate ser2net.yaml
 # ============================================================
 echo ""
 echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---"
 # Backup existing config if not already backed up
 if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then
  cp "$SER2NET_CONF" "${SER2NET_CONF}.orig"
  echo "  Backed up original to ${SER2NET_CONF}.orig"
 fi
 {
  echo "%YAML 1.1"
  echo "---"
  echo "# ser2net configuration for pfv-tsys4 console ports"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "#"
  echo "# All ports bound to Tailscale IP ($TS_IP) for secure remote access."
  echo "# Physical devices are accessed via stable udev symlinks in /dev/consoles/."
  echo "#"
  echo "# To connect directly:   telnet $TS_IP 2001"
  echo "# To connect via conman: conman -f <name>"
  echo ""
  echo "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    # ser2net connection block
    echo "connection: &con${tcp_port}"
    echo "    accepter: tcp,${TS_IP},${tcp_port}"
    echo "    enable: on"
    echo "    options:"
    echo "      banner: *banner"
    echo "      kickolduser: true"
    echo "      telnet-brk-on-sync: true"
    echo "    connector: serialdev,"
    echo "              /dev/consoles/${name},"
    echo "              ${baud},local"
    echo ""
  done
 } > "$SER2NET_CONF"
 echo "  Written: $SER2NET_CONF"
 echo "  ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)"
 # ============================================================
 # 3. Write conman console entries directly into conman.conf
 # ============================================================
 # conman 0.3.x does NOT support the 'include' directive, so we write
 # CONSOLE entries directly into /etc/conman.conf between idempotent markers.
 echo ""
 echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---"
 # Ensure logdir exists
 mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true
 # Ensure LOGDIR is set in conman.conf (server-level directive for log file paths)
 if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then
  # Insert near the top, after the first SERVER directives
  sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF"
  echo "  Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF"
 fi
 # Remove any previous auto-generated block (between markers)
 # Then append the new block
 MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)"
 MARKER_END="# END PFV CONSOLE DEFINITIONS"
 # Strip old block if present
 if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then
  sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF"
  echo "  Removed previous console definitions."
 fi
 # Append new block
 {
  echo ""
  echo "$MARKER_BEGIN"
  echo "# Generated by console/generate-config.sh on $(date)"
  echo "# Each console connects to a ser2net TCP port via telnet protocol."
  echo "# ser2net owns the physical serial device; conman provides logging"
  echo "# and multiplexing on top."
  echo "# Access: conman -f <name>"
  echo ""
  for entry in "${ENTRIES[@]}"; do
    IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
    echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
  done
  echo "$MARKER_END"
 } >> "$CONMAN_CONF"
 CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0)
 echo "  Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF"
 # ============================================================
 # Summary
 # ============================================================
 echo ""
 echo "============================================"
 echo "  Configuration generated successfully."
 echo ""
 echo "  Files written:"
 echo "    $UDEV_RULES       ($(wc -l < "$UDEV_RULES") lines)"
 echo "    $SER2NET_CONF     ($(wc -l < "$SER2NET_CONF") lines)"
 echo "    $CONMAN_CONF      (CONSOLE entries appended between markers)"
 echo ""
 echo "  Next steps:"
 echo "    1. Reload udev:   udevadm control --reload-rules && udevadm trigger"
 echo "    2. Restart ser2net: systemctl restart ser2net"
 echo "    3. Start conman:  systemctl enable --now conmand"
 echo "    4. Or run:       bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
 echo "============================================"
@@ -0,0 +1,29 @@
 # console/mapping.txt — Source of Truth for console port assignments
 #
 # Format: <tcp_port>|<name>|<id_path_substring>|<baud>|<comment>
 #
 # Delimiter is | (pipe) because ID_PATH values contain colons.
 #
 # - tcp_port:     TCP port ser2net listens on (also the conman console name suffix)
 # - name:         Device name (used for /dev/console/<name> symlink, conman console name)
 # - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH`
 #                      These are STABLE across reboots as long as adapters aren't moved
 #                      to different physical USB ports.
 # - baud:         Serial baud rate (9600n81 = 9600 8N1, no flow control)
 # - comment:      Free-form description
 #
 # To RE-MAP after physically moving an adapter:
 #   1. Run: bash console/discover.sh   (find the new ID_PATH for the device)
 #   2. Update the id_path_substring in this file
 #   3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand
 #
 2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch)
 2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch
 2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch
 2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra)
 2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch
 2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch
 2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch
 # Unassigned (no device detected):
 # 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare
 # 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare
@@ -0,0 +1,216 @@
 #!/usr/bin/bash
 #
 # console/setup.sh — deploy console management on pfv-tsys4
 #
 # Orchestrates the full setup:
 #   1. Ensures ser2net + conman are installed
 #   2. Copies mapping.txt to the target host (if running remotely)
 #   3. Runs generate-config.sh to produce udev rules + ser2net.yaml + conman.conf
 #   4. Reloads udev, creates /dev/consoles/ symlinks
 #   5. Restarts ser2net (TCP ports on Tailscale IP)
 #   6. Enables + starts conmand (logging + multiplexing)
 #   7. Verifies
 #
 # This script is IDEMPOTENT — safe to run multiple times.
 #
 # Usage:
 #   PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/setup.sh
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules"
 SER2NET_CONF="/etc/ser2net.yaml"
 CONMAN_CONSOLES="/etc/conman/console-consoles.conf"
 CONSOLE_DEV_DIR="/dev/consoles"
 echo "============================================"
 echo "  Console Management Setup"
 echo "  Host: $(hostname)   $(date)"
 echo "============================================"
 # --- 1. Install dependencies ---
 echo ""
 echo "--- [1/7] Checking dependencies ---"
 NEED_INSTALL=()
 dpkg -l ser2net 2>/dev/null | grep -q '^ii' && echo "  ser2net: installed" || NEED_INSTALL+=(ser2net)
 dpkg -l conman  2>/dev/null | grep -q '^ii' && echo "  conman: installed"  || NEED_INSTALL+=(conman)
 if [ "${#NEED_INSTALL[@]}" -gt 0 ]; then
  echo "  Installing: ${NEED_INSTALL[*]}"
  apt-get update -qq
  apt-get install -y -qq "${NEED_INSTALL[@]}"
 else
  echo "  All dependencies present."
 fi
 # --- 2. Ensure mapping file is available ---
 echo ""
 echo "--- [2/7] Locating mapping file ---"
 MAPPING_FILE=""
 for candidate in \
  "$SCRIPT_DIR/mapping.txt" \
  "$(dirname "$0")/mapping.txt" \
  "/root/console/mapping.txt" \
  "/tmp/mapping.txt"; do
  if [ -f "$candidate" ]; then
    MAPPING_FILE="$candidate"
    break
  fi
 done
 if [ -z "$MAPPING_FILE" ]; then
  echo "FATAL: mapping.txt not found. Copy it to the target host."
  exit 1
 fi
 echo "  Using: $MAPPING_FILE"
 # --- 3. Generate configs ---
 echo ""
 echo "--- [3/7] Generating configs ---"
 export MAPPING_FILE
 bash "$(dirname "$0")/generate-config.sh" 2>&1 || bash "$SCRIPT_DIR/generate-config.sh" 2>&1 || {
  echo "FATAL: generate-config.sh failed."
  exit 1
 }
 # --- 4. Reload udev + create symlinks ---
 echo ""
 echo "--- [4/7] Reloading udev rules ---"
 udevadm control --reload-rules
 # Try trigger first (works on some systems)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty" ] && udevadm trigger --action=add "$tty" 2>/dev/null || true
 done
 # Also try writing to uevent (forces udev reprocessing)
 for tty in /sys/class/tty/ttyUSB*; do
  [ -e "$tty/uevent" ] && echo "add" > "$tty/uevent" 2>/dev/null || true
 done
 sleep 2
 # FALLBACK: if udev symlinks don't exist (common when devices are already
 # discovered — udev trigger doesn't always re-create symlinks for existing
 # devices), create them manually by matching ID_PATH. The udev rules will
 # handle future boots/hotplugs automatically.
 if [ ! -d /dev/consoles ] || [ -z "$(ls /dev/consoles/ 2>/dev/null)" ]; then
  echo "  udev trigger didn't create symlinks. Creating manually..."
  mkdir -p /dev/consoles
  while IFS= read -r line; do
    line="${line%%#*}"
    line="$(echo "$line" | xargs)"
    [ -z "$line" ] && continue
    IFS='|' read -r tcp_port name id_path baud comment <<< "$line"
    # Find the ttyUSB whose ID_PATH contains the mapping's id_path substring
    for tty in /dev/ttyUSB*; do
      [ -e "$tty" ] || continue
      DEV_IDPATH=$(udevadm info -q property -n "$tty" 2>/dev/null | grep ^ID_PATH= | cut -d= -f2)
      if echo "$DEV_IDPATH" | grep -q "$id_path"; then
        ln -sf "$tty" "/dev/consoles/$name"
        echo "    ln -s $tty -> /dev/consoles/$name"
        break
      fi
    done
  done < "$MAPPING_FILE"
 fi
 echo "  Stable symlinks:"
 ls -la /dev/consoles/ 2>/dev/null | grep -v '^total\|^d' | sed 's/^/    /' || echo "    (none created)"
 # Verify each symlink resolves
 echo ""
 echo "  Symlink verification:"
 while IFS= read -r line; do
  line="${line%%#*}"
  line="$(echo "$line" | xargs)"
  [ -z "$line" ] && continue
  IFS='|' read -r tcp_port name id_path baud comment <<< "$line"
  if [ -e "/dev/consoles/$name" ]; then
    TARGET=$(readlink -f "/dev/consoles/$name")
    echo "    [OK] /dev/consoles/$name -> $TARGET"
  else
    echo "    [MISSING] /dev/consoles/$name (adapter unplugged or ID_PATH changed)"
  fi
 done < "$MAPPING_FILE"
 # --- 5. Restart ser2net ---
 echo ""
 echo "--- [5/7] Restarting ser2net ---"
 systemctl enable ser2net
 systemctl restart ser2net
 sleep 2
 if systemctl is-active --quiet ser2net; then
  echo "  ser2net is running."
  TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1")
  echo "  Listening ports:"
  ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/    /'
 else
  echo "  WARNING: ser2net failed to start. Checking journal..."
  journalctl -u ser2net --no-pager -n 20
 fi
 # --- 6. Enable + start conmand ---
 echo ""
 echo "--- [6/7] Starting conmand ---"
 # conman package on Debian may not ship a systemd unit. Create one if missing.
 if ! systemctl cat conmand >/dev/null 2>&1; then
  echo "  No systemd unit for conmand — creating one..."
  cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT'
 [Unit]
 Description=ConMan (Console Manager)
 After=network.target ser2net.service
 Requires=ser2net.service
 [Service]
 Type=forking
 ExecStart=/usr/sbin/conmand -c /etc/conman.conf
 Restart=on-failure
 RestartSec=5
 [Install]
 WantedBy=multi-user.target
 CONMAND_UNIT
  systemctl daemon-reload
  echo "  Created /etc/systemd/system/conmand.service"
 fi
 # Kill any manually-started conmand first
 pkill -x conmand 2>/dev/null || true
 sleep 1
 systemctl enable conmand 2>/dev/null || true
 systemctl restart conmand 2>/dev/null || true
 sleep 2
 if systemctl is-active --quiet conmand; then
  echo "  conmand is running."
  echo "  Consoles:"
  conman -q 2>&1 | sed 's/^/    /' || true
 else
  echo "  WARNING: conmand failed to start. Checking journal..."
  journalctl -u conmand --no-pager -n 20 2>/dev/null || true
  # Try manual start as fallback
  echo "  Attempting manual start..."
  /usr/sbin/conmand -c /etc/conman.conf 2>&1 || true
 fi
 # --- 7. Summary ---
 echo ""
 echo "--- [7/7] Setup complete ---"
 echo ""
 echo "  ser2net TCP ports (connect directly):"
 echo "    telnet <tailscale-ip> 2001    # pfv-core-sw01"
 echo "    telnet <tailscale-ip> 2002    # pfv-tor3-mgmt"
 echo "    ..."
 echo ""
 echo "  conman consoles (with logging):"
 echo "    conman -f pfv-core-sw01"
 echo "    conman -q                   # query status"
 echo ""
 echo "  To regenerate after changing mapping.txt:"
 echo "    bash generate-config.sh"
 echo "    udevadm trigger"
 echo "    systemctl restart ser2net conmand"
 echo "============================================"
@@ -0,0 +1,89 @@
 #!/usr/bin/bash
 #
 # console/validate-conman.sh — verify conman can actually reach devices via
 # ser2net TCP ports and is capturing log output to files.
 #
 # This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device
 #
 set -uo pipefail
 TS_IP=$(tailscale ip -4)
 LOGDIR="/var/log/conman"
 echo "============================================"
 echo "  Conman Data Path + Log Validation"
 echo "  Host: $(hostname)   TS IP: $TS_IP"
 echo "============================================"
 echo ""
 echo "--- 1. conman.conf log settings ---"
 grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo "  (no explicit logdir — defaults to /var/log/conman)"
 echo "  Log dir: $LOGDIR"
 ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo "  ($LOGDIR does not exist yet)"
 echo ""
 echo "--- 2. CONSOLE entries: each has a log= directive? ---"
 # Extract the auto-generated block and check each CONSOLE line has log=
 sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do
  name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p')
  if echo "$line" | grep -q 'log='; then
    logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p')
    echo "  [OK] $name → log=$logfile"
  else
    echo "  [FAIL] $name has NO log= directive"
  fi
 done
 echo ""
 echo "--- 3. Trigger log capture: connect to each console briefly ---"
 # conman -e changes the escape char. We use -j (join, read-only) with a timeout.
 # Actually, conman doesn't have a built-in "connect for N seconds" — but conmand
 # connects to each device ON STARTUP and keeps the connection open for logging.
 # The log files should already be created. Let's check timestamps.
 echo "  conmand connects to all consoles on startup. Checking if logs exist..."
 echo ""
 echo "--- 4. Log file inventory ---"
 for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do
  logfile="$LOGDIR/${name}.log"
  if [ -f "$logfile" ]; then
    SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
    MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1)
    echo "  [OK] $logfile ($SIZE bytes, modified $MTIME)"
  else
    echo "  [MISSING] $logfile — conmand may not be writing yet"
  fi
 done
 echo ""
 echo "--- 5. conmand connection status (journal) ---"
 # conmand logs connection attempts/errors to syslog
 journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo "  (no relevant journal entries)"
 echo ""
 echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---"
 # Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded)
 echo "  Probing TCP $TS_IP:2007 for data..."
 RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
 if [ -n "$RESPONSE" ]; then
  echo "  [OK] Data flowing through ser2net TCP 2007:"
  echo "$RESPONSE" | sed 's/^/    /'
 else
  echo "  (no immediate response — device may need more interaction)"
 fi
 echo ""
 echo "--- 7. Check if conmand has open connections to ser2net ports ---"
 CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
 if [ -n "$CONMAND_PID" ]; then
  echo "  conmand PID: $CONMAND_PID"
  echo "  Open connections to ser2net (expect 7 to 100.x:200X):"
  ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/    /'
  COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
  echo "  Total conmand→ser2net connections: $COUNT (expect 7)"
 else
  echo "  [FAIL] conmand not running"
 fi
 echo ""
 echo "============================================"
@@ -43,6 +43,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
 | Document | Description | Last Reviewed |
 |----------|-------------|---------------|
 | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
 | [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 |
 | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
 | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
 | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |