From eb99b00a70b37878a725f22b9283dd213ede8c86 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Tue, 28 Jul 2026 19:44:02 -0500 Subject: [PATCH] feat(console): manage 7 switch consoles via ser2net+conman on pfv-tsys4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Solve the long-standing USB adapter enumeration shift problem: 9 Prolific USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every reboot and breaks the old /root/conmap + manual screen workflow. Solution: udev rules pin each adapter by its ID_PATH (physical USB port topology), which is stable across reboots regardless of enumeration order. Each adapter gets a named symlink in /dev/consoles/. ser2net opens these stable symlinks and exposes them on TCP ports (2001-2007) bound to the Tailscale interface only. conman connects to those TCP ports for session logging and multi-user console sharing. Architecture (layered, no port sharing): USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux) Port assignments (all on Tailscale IP 100.70.77.93): 2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor 2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw 2007 = pfv-r2-sw Scripts (console/): - mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment) - generate-config.sh: generates udev rules, ser2net.yaml, conman.conf entries from mapping.txt. Idempotent (markers in conman.conf for clean regeneration). Uses | delimiter (ID_PATH values contain colons). - setup.sh: full deploy — generate configs, create symlinks (udev trigger + manual fallback for already-discovered devices), create conmand systemd unit (Debian doesn't ship one), restart services - discover.sh: read-only USB adapter and service state discovery - validate-conman.sh: verify conman→ser2net→device data path and log capture Issues fixed during development: - /dev/console is a kernel char device (major 5, minor 1) — cannot create a directory there. Changed symlink namespace to /dev/consoles/. - conman 0.3.x has no 'include' directive — CONSOLE entries written directly into /etc/conman.conf between idempotent markers. - Debian conman package has no systemd unit — created /etc/systemd/system/conmand.service with After=ser2net ordering. - conman.conf had no LOGDIR — logs weren't being written to /var/log/conman/. Fixed by adding server logdir directive. Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with 7 consoles registered, 7 log files actively capturing console output, both services enabled for reboot survival. 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- AGENTS.md | 2 + README.md | 1 + STATUS.md | 14 +++ console/README.md | 109 +++++++++++++++++ console/discover.sh | 102 ++++++++++++++++ console/generate-config.sh | 244 +++++++++++++++++++++++++++++++++++++ console/mapping.txt | 29 +++++ console/setup.sh | 216 ++++++++++++++++++++++++++++++++ console/validate-conman.sh | 89 ++++++++++++++ docs/docmap.md | 1 + 10 files changed, 807 insertions(+) create mode 100644 console/README.md create mode 100644 console/discover.sh create mode 100644 console/generate-config.sh create mode 100644 console/mapping.txt create mode 100644 console/setup.sh create mode 100644 console/validate-conman.sh diff --git a/AGENTS.md b/AGENTS.md index 04ea4bb..a9cd395 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,6 +13,7 @@ tests/ Test suite + VM validation harness dns-cluster-setup/ Technitium DNS cluster replication k8s/ k3s cluster setup scripts (3-node HA over Tailscale) powerman/ Cyclades PM10i PDU management via powerman +console/ Serial console management (ser2net + conman) for switches perf/ Proxmox perf tuning, fleet audit, iperf netinfra/ pfv-netinfra-01/02 DNS/NTP setup switches/ Switch configuration captures @@ -81,6 +82,7 @@ vendor/ Vendored KNELShellFramework | [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication | | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane | | [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman | +| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman | | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings | ## Key Docs diff --git a/README.md b/README.md index 33fcc16..5e1d1ca 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste | [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts | | [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) | | [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | +| [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 | | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics | | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts | | [`switches/`](switches/) | Switch configuration captures | diff --git a/STATUS.md b/STATUS.md index 0665a57..0b0e6d3 100644 --- a/STATUS.md +++ b/STATUS.md @@ -56,6 +56,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [x] Validated: outlet 10 cycled off → on (8/8 test checks passed) - [ ] Rename outlets to match physical devices (Friday onsite) +### Console Management (ser2net + conman on pfv-tsys4 — LIVE) +- [x] 7 network switch/router consoles managed via ser2net + conman +- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by + ID_PATH (physical USB port topology) to stable `/dev/consoles/` + symlinks that survive reboot regardless of enumeration order +- [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to + **Tailscale IP only** (`100.70.77.93`) +- [x] conman connects to TCP ports for logging + multiplexing (7 log + files active in `/var/log/conman/`) +- [x] Both ser2net + conmand enabled via systemd (survive reboot) +- [x] conmand systemd unit created (Debian package doesn't ship one) +- [x] Old `/root/conmap` + manual `screen` workflow replaced + ### Repo Merge - [x] KNELServerBuild merged into PFVCluster (history preserved) - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/) @@ -99,4 +112,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Backup | Proxmox Backup Server (PBS) | | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** | | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** | +| **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** | | Production | Cloudron VPS, Reston VA (this cluster is R&D only) | diff --git a/console/README.md b/console/README.md new file mode 100644 index 0000000..d73c88f --- /dev/null +++ b/console/README.md @@ -0,0 +1,109 @@ +# Console Management (ser2net + conman) + +Network-accessible serial console management for all production network +switches and routers, running on **pfv-tsys4** (storage server). + +## Architecture + +``` +USB-DB9 adapters → udev symlinks (/dev/consoles/) → ser2net (TCP) → conman (logging + multiplexing) +``` + +ser2net owns the physical serial devices and exposes them on TCP ports +bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman +connects to those TCP ports for session logging, output capture, and +multi-user console sharing. + +**conman and ser2net do NOT share ports** — only one process can open a +serial device at a time. ser2net owns the physical device; conman connects +over TCP. + +## The USB Enumeration Problem (SOLVED) + +The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no +unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on +enumeration order, which shifts on every boot. This made the old +`/root/conmap` + manual `screen` workflow break after every reboot. + +**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port +topology), which is stable across reboots regardless of enumeration order. +Each adapter gets a named symlink in `/dev/consoles/` that never changes. + +The udev rules are generated from `mapping.txt`, which maps each adapter's +ID_PATH to a console name and TCP port. To re-map after physically moving +an adapter, update `mapping.txt` and re-run `setup.sh`. + +**Fallback:** if udev trigger doesn't create symlinks for already-discovered +devices (common on first run), `setup.sh` creates them manually by matching +ID_PATH. On subsequent boots, udev creates them automatically. + +## Port Assignments + +| TCP Port | Console Name | ID_PATH | Description | +|----------|-------------|---------|-------------| +| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) | +| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch | +| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch | +| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) | +| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch | +| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch | +| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch | + +All ports listen on the Tailscale IP (`100.70.77.93`). + +## Scripts + +| Script | Purpose | +|--------|---------| +| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud | +| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt | +| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services | +| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services | + +## Usage + +### Connect to a console + +```bash +# Via conman (with logging + multiplexing): +conman -f pfv-core-sw01 # attach to console +conman -q # query status of all consoles + +# Via telnet (direct, no logging): +telnet 100.70.77.93 2001 # pfv-core-sw01 + +# From any Tailscale-connected device: +telnet pfv-tsys4 2001 +``` + +### Re-deploy after changing mapping.txt + +```bash +PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh' +``` + +### Find the ID_PATH for a new adapter + +```bash +PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh +``` + +Then match the new adapter's ID_PATH to its physical location and add a line +to `mapping.txt`. + +## Files on pfv-tsys4 + +| File | Purpose | +|------|---------| +| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH | +| `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) | +| `/etc/conman.conf` | conman config (CONSOLE entries between markers) | +| `/etc/systemd/system/conmand.service` | systemd unit for conmand | +| `/root/console/mapping.txt` | Copy of the source-of-truth mapping | +| `/root/console/setup.sh` | Setup script (re-runnable) | +| `/root/console/generate-config.sh` | Config generator | + +## Old workflow (replaced) + +The old `/root/conmap` file and manual `screen` sessions are no longer +needed. The new setup is fully automated and survives reboots. diff --git a/console/discover.sh b/console/discover.sh new file mode 100644 index 0000000..468eecd --- /dev/null +++ b/console/discover.sh @@ -0,0 +1,102 @@ +#!/usr/bin/bash +# +# console/discover.sh — READ-ONLY discovery of console setup on pfv-tsys4 +# +# Usage: PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh +# +# This script is strictly read-only. No writes to the system. +# +set -uo pipefail + +echo "============================================" +echo " Console Setup Discovery" +echo " Host: $(hostname)" +echo " Date: $(date)" +echo " READ-ONLY" +echo "============================================" + +echo "" +echo "=== 1. USB devices ===" +lsusb 2>/dev/null || echo "(lsusb not available)" + +echo "" +echo "=== 2. All ttyUSB* devices (with major/minor) ===" +ls -la /dev/ttyUSB* 2>/dev/null || echo "(no /dev/ttyUSB* devices)" + +echo "" +echo "=== 3. USB-serial driver bindings ===" +echo "-- pl2303 --" +ls -la /sys/bus/usb-serial/drivers/pl2303/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)" +echo "-- cp210x --" +ls -la /sys/bus/usb-serial/drivers/cp210x/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)" +echo "-- ftdi_sio --" +ls -la /sys/bus/usb-serial/drivers/ftdi_sio/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)" +echo "-- ch341 --" +ls -la /sys/bus/usb-serial/drivers/ch341/ 2>/dev/null | grep -v '^total\|^d\|module\|new_id\|uevent' || echo "(none)" + +echo "" +echo "=== 4. USB serial adapter details (vendor/model/serial per port) ===" +for tty in /dev/ttyUSB*; do + [ -e "$tty" ] || continue + echo "--- $tty ---" + udevadm info -q all -n "$tty" 2>/dev/null | grep -E 'ID_VENDOR_ID|ID_MODEL_ID|ID_SERIAL|ID_USB_DRIVER|ID_PATH=' | sed 's/^/ /' +done + +echo "" +echo "=== 5. Existing /root/conmap ===" +if [ -f /root/conmap ]; then + cat /root/conmap +else + echo "(no /root/conmap)" +fi +ls -la /root/conmap* 2>/dev/null + +echo "" +echo "=== 6. Screen sessions (running) ===" +screen -ls 2>&1 || echo "(screen not running or not installed)" + +echo "" +echo "=== 7. Existing screen wrappers/scripts in /root ===" +ls -la /root/ 2>/dev/null | grep -iE 'screen|con|console|tty|usb' || echo "(no obvious console scripts in /root)" + +echo "" +echo "=== 8. ser2net ===" +which ser2net 2>/dev/null || echo "(ser2net not installed)" +dpkg -l ser2net 2>/dev/null | tail -2 || echo "(ser2net not in dpkg)" +cat /etc/ser2net/ser2net.yaml 2>/dev/null || cat /etc/ser2net.conf 2>/dev/null || cat /etc/ser2net/ser2net.conf 2>/dev/null || echo "(no ser2net config)" +systemctl is-active ser2net 2>/dev/null || echo "(ser2net service not found)" + +echo "" +echo "=== 9. conman ===" +which conman 2>/dev/null || echo "(conman not installed)" +which conmand 2>/dev/null || echo "(conmand not installed)" +dpkg -l conman 2>/dev/null | tail -2 || echo "(conman not in dpkg)" +echo "--- /etc/conman.conf (console lines only) ---" +grep -nE 'CONSOLE|SERVER|LOG|SERIAL|DEV|BAUD|^[^#].*name=' /etc/conman.conf 2>/dev/null | head -60 || echo "(no conman.conf or no console entries)" +echo "--- conmand service ---" +systemctl is-active conmand 2>/dev/null || echo "(conmand not running)" +systemctl is-enabled conmand 2>/dev/null || echo "(conmand not enabled)" + +echo "" +echo "=== 10. Existing console logs ===" +ls -la /var/log/conman/ 2>/dev/null | head -20 || echo "(no /var/log/conman)" +ls -la /var/consoles/ 2>/dev/null | head -20 || echo "(no /var/consoles)" + +echo "" +echo "=== 11. udev rules for ttyUSB ===" +grep -r ttyUSB /etc/udev/rules.d/ 2>/dev/null || echo "(no udev rules for ttyUSB)" +grep -r 'console' /etc/udev/rules.d/ 2>/dev/null | head -10 || true + +echo "" +echo "=== 12. expect availability ===" +command -v expect && expect -v 2>&1 || echo "expect: NOT installed" +command -v socat && socat -V 2>&1 | head -1 || echo "socat: NOT installed" + +echo "" +echo "=== 13. Ports in use (2001-2099, 7000-7999, 7820-7899) ===" +ss -tlnp 2>/dev/null | grep -E ':200[0-9]|:700[0-9]|:782[0-9]|:789[0-9]' || echo "(no relevant ports listening)" + +echo "" +echo "============================================" +echo " Discovery complete (read-only)." +echo "============================================" diff --git a/console/generate-config.sh b/console/generate-config.sh new file mode 100644 index 0000000..dee2971 --- /dev/null +++ b/console/generate-config.sh @@ -0,0 +1,244 @@ +#!/usr/bin/bash +# +# console/generate-config.sh — generate udev rules + ser2net.yaml + conman.conf +# +# Reads console/mapping.txt (the source of truth) and generates all three +# config files. This is the fix for the USB enumeration shift problem: +# +# 1. udev rules pin each adapter by its STABLE ID_PATH (physical USB port) +# to a named symlink like /dev/consoles/pfv-core-sw01 +# 2. ser2net opens those stable symlinks and exposes them on TCP ports +# (2001, 2002, ...) bound to the Tailscale IP +# 3. conman connects to those TCP ports for logging + multiplexing +# +# Run this script ON the target host. It writes to: +# /etc/udev/rules.d/99-console-ports.rules +# /etc/ser2net.yaml +# /etc/conman/console-consoles.conf (included by /etc/conman.conf) +# +# Usage: +# PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/generate-config.sh +# +# Environment overrides: +# MAPPING_FILE — path to mapping.txt (default: auto-detect next to this script) +# TS_IP — Tailscale IP to bind ser2net on (default: auto-detect) +# CONMAN_LOGDIR — conman log directory (default: /var/log/conman) +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +MAPPING_FILE="${MAPPING_FILE:-$SCRIPT_DIR/mapping.txt}" +CONMAN_LOGDIR="${CONMAN_LOGDIR:-/var/log/conman}" + +UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules" +SER2NET_CONF="/etc/ser2net.yaml" +CONMAN_CONF="/etc/conman.conf" +CONMAN_CONSOLES="/etc/conman/console-consoles.conf" +CONSOLE_DEV_DIR="/dev/console" + +echo "============================================" +echo " Console Config Generator" +echo " Host: $(hostname) $(date)" +echo "============================================" + +# --- Locate mapping file --- +# When run via remote.sh prox-file, $0 is bash and $SCRIPT_DIR may be wrong. +# Search common locations. +if [ ! -f "$MAPPING_FILE" ]; then + for candidate in \ + "/root/console/mapping.txt" \ + "/tmp/mapping.txt" \ + "$(dirname "$0")/mapping.txt"; do + if [ -f "$candidate" ]; then + MAPPING_FILE="$candidate" + break + fi + done +fi + +if [ ! -f "$MAPPING_FILE" ]; then + echo "FATAL: mapping file not found. Tried: $MAPPING_FILE" + echo "Copy mapping.txt to the target host first." + exit 1 +fi +echo " Mapping file: $MAPPING_FILE" + +# --- Auto-detect Tailscale IP --- +if [ -z "${TS_IP:-}" ]; then + TS_IP=$(tailscale ip -4 2>/dev/null || true) + if [ -z "$TS_IP" ]; then + echo "FATAL: could not auto-detect Tailscale IP. Set TS_IP manually." + exit 1 + fi +fi +echo " Tailscale IP: $TS_IP" +echo " ser2net will bind to: $TS_IP" + +# --- Parse mapping file (skip comments and blank lines) --- +echo "" +echo "--- Parsing mapping file ---" +ENTRIES=() +while IFS= read -r line; do + # Skip comments and blank lines + line="${line%%#*}" + line="$(echo "$line" | xargs)" # trim whitespace + [ -z "$line" ] && continue + ENTRIES+=("$line") + echo " $line" +done < "$MAPPING_FILE" + +if [ "${#ENTRIES[@]}" -eq 0 ]; then + echo "FATAL: no entries found in mapping file." + exit 1 +fi + +echo "" +echo " ${#ENTRIES[@]} console ports configured." + +# ============================================================ +# 1. Generate udev rules +# ============================================================ +echo "" +echo "--- [1/3] Generating udev rules: $UDEV_RULES ---" + +cat > "$UDEV_RULES" <<'UDEV_HEADER' +# Stable symlinks for USB-DB9 console adapters +# Generated by console/generate-config.sh +# DO NOT EDIT — edit mapping.txt and re-run generate-config.sh +# +# These rules pin each adapter to a named symlink based on its physical +# USB port path (ID_PATH), which is stable across reboots regardless of +# enumeration order. This is the fix for the "USB adapters shift on reboot" +# problem. +# +# To find the ID_PATH for a device: +# udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH +UDEV_HEADER + +for entry in "${ENTRIES[@]}"; do + IFS='|' read -r tcp_port name id_path baud comment <<< "$entry" + # Build the full ID_PATH match. The mapping stores a substring like "usb-0:1.5.4.4" + # The actual ID_PATH is like "pci-0000:00:1a.0-usb-0:1.5.4.4:1.0" + # We match on the substring to be portable across PCI bus changes. + echo "" >> "$UDEV_RULES" + echo "# $name (TCP $tcp_port): $comment" >> "$UDEV_RULES" + echo "SUBSYSTEM==\"tty\", ENV{ID_PATH}==\"*$id_path*\", SYMLINK+=\"console/$name\"" >> "$UDEV_RULES" +done + +echo " Written: $UDEV_RULES" +echo " Symlinks: /dev/consoles/ for each device" + +# ============================================================ +# 2. Generate ser2net.yaml +# ============================================================ +echo "" +echo "--- [2/3] Generating ser2net config: $SER2NET_CONF ---" + +# Backup existing config if not already backed up +if [ -f "$SER2NET_CONF" ] && [ ! -f "${SER2NET_CONF}.orig" ]; then + cp "$SER2NET_CONF" "${SER2NET_CONF}.orig" + echo " Backed up original to ${SER2NET_CONF}.orig" +fi + +{ + echo "%YAML 1.1" + echo "---" + echo "# ser2net configuration for pfv-tsys4 console ports" + echo "# Generated by console/generate-config.sh on $(date)" + echo "#" + echo "# All ports bound to Tailscale IP ($TS_IP) for secure remote access." + echo "# Physical devices are accessed via stable udev symlinks in /dev/consoles/." + echo "#" + echo "# To connect directly: telnet $TS_IP 2001" + echo "# To connect via conman: conman -f " + echo "" + echo "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n" + echo "" + + for entry in "${ENTRIES[@]}"; do + IFS='|' read -r tcp_port name id_path baud comment <<< "$entry" + # ser2net connection block + echo "connection: &con${tcp_port}" + echo " accepter: tcp,${TS_IP},${tcp_port}" + echo " enable: on" + echo " options:" + echo " banner: *banner" + echo " kickolduser: true" + echo " telnet-brk-on-sync: true" + echo " connector: serialdev," + echo " /dev/consoles/${name}," + echo " ${baud},local" + echo "" + done +} > "$SER2NET_CONF" + +echo " Written: $SER2NET_CONF" +echo " ${#ENTRIES[@]} TCP ports configured ($TS_IP:2001-20XX)" + +# ============================================================ +# 3. Write conman console entries directly into conman.conf +# ============================================================ +# conman 0.3.x does NOT support the 'include' directive, so we write +# CONSOLE entries directly into /etc/conman.conf between idempotent markers. +echo "" +echo "--- [3/3] Writing conman consoles into $CONMAN_CONF ---" + +# Ensure logdir exists +mkdir -p "$CONMAN_LOGDIR" 2>/dev/null || true + +# Ensure LOGDIR is set in conman.conf (server-level directive for log file paths) +if ! grep -qiE '^\s*server\s+logdir\s*=' "$CONMAN_CONF" 2>/dev/null; then + # Insert near the top, after the first SERVER directives + sed -i "1i\\server logdir = \"$CONMAN_LOGDIR\"" "$CONMAN_CONF" + echo " Added server logdir = \"$CONMAN_LOGDIR\" to $CONMAN_CONF" +fi + +# Remove any previous auto-generated block (between markers) +# Then append the new block +MARKER_BEGIN="# BEGIN PFV CONSOLE DEFINITIONS (auto-generated — do not edit between markers)" +MARKER_END="# END PFV CONSOLE DEFINITIONS" + +# Strip old block if present +if grep -q "$MARKER_BEGIN" "$CONMAN_CONF" 2>/dev/null; then + sed -i "/$MARKER_BEGIN/,/$MARKER_END/d" "$CONMAN_CONF" + echo " Removed previous console definitions." +fi + +# Append new block +{ + echo "" + echo "$MARKER_BEGIN" + echo "# Generated by console/generate-config.sh on $(date)" + echo "# Each console connects to a ser2net TCP port via telnet protocol." + echo "# ser2net owns the physical serial device; conman provides logging" + echo "# and multiplexing on top." + echo "# Access: conman -f " + echo "" + for entry in "${ENTRIES[@]}"; do + IFS='|' read -r tcp_port name id_path baud comment <<< "$entry" + echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\"" + done + echo "$MARKER_END" +} >> "$CONMAN_CONF" + +CONSOLE_COUNT=$(grep -c "^CONSOLE " "$CONMAN_CONF" 2>/dev/null || echo 0) +echo " Written $CONSOLE_COUNT CONSOLE entries to $CONMAN_CONF" + +# ============================================================ +# Summary +# ============================================================ +echo "" +echo "============================================" +echo " Configuration generated successfully." +echo "" +echo " Files written:" +echo " $UDEV_RULES ($(wc -l < "$UDEV_RULES") lines)" +echo " $SER2NET_CONF ($(wc -l < "$SER2NET_CONF") lines)" +echo " $CONMAN_CONF (CONSOLE entries appended between markers)" +echo "" +echo " Next steps:" +echo " 1. Reload udev: udevadm control --reload-rules && udevadm trigger" +echo " 2. Restart ser2net: systemctl restart ser2net" +echo " 3. Start conman: systemctl enable --now conmand" +echo " 4. Or run: bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh" +echo "============================================" diff --git a/console/mapping.txt b/console/mapping.txt new file mode 100644 index 0000000..4a3fba9 --- /dev/null +++ b/console/mapping.txt @@ -0,0 +1,29 @@ +# console/mapping.txt — Source of Truth for console port assignments +# +# Format: |||| +# +# Delimiter is | (pipe) because ID_PATH values contain colons. +# +# - tcp_port: TCP port ser2net listens on (also the conman console name suffix) +# - name: Device name (used for /dev/console/ symlink, conman console name) +# - id_path_substring: Stable USB physical path from `udevadm info -q all -n /dev/ttyUSBN | grep ID_PATH` +# These are STABLE across reboots as long as adapters aren't moved +# to different physical USB ports. +# - baud: Serial baud rate (9600n81 = 9600 8N1, no flow control) +# - comment: Free-form description +# +# To RE-MAP after physically moving an adapter: +# 1. Run: bash console/discover.sh (find the new ID_PATH for the device) +# 2. Update the id_path_substring in this file +# 3. Run: bash console/generate-config.sh && udevadm trigger && systemctl restart ser2net conmand +# +2001|pfv-core-sw01|usb-0:1.5.4.4|9600n81|Dell PowerConnect 5448 (core switch) +2002|pfv-tor3-mgmt|usb-0:1.6.3.1|9600n81|Rack 3 management TOR switch +2003|pfv-tor3-stor|usb-0:1.6.3.3.2|9600n81|Rack 3 storage TOR switch +2004|pfv-rrinfra-rtr|usb-0:1.6.3.3.1|9600n81|Cisco router (rrinfra) +2005|pfv-r2-tor-top|usb-0:1.6.3.3.3|9600n81|Rack 2 top-of-rack switch +2006|subodev-torsw|usb-0:1.5.4.1|9600n81|Suborbital device TOR switch +2007|pfv-r2-sw|usb-0:1.6.3.2|9600n81|Rack 2 old Dell switch +# Unassigned (no device detected): +# 2008|spare-1|usb-0:1.6.3.4|9600n81|Empty / spare +# 2009|spare-2|usb-0:1.6.3.3.4|9600n81|Empty / spare diff --git a/console/setup.sh b/console/setup.sh new file mode 100644 index 0000000..462b398 --- /dev/null +++ b/console/setup.sh @@ -0,0 +1,216 @@ +#!/usr/bin/bash +# +# console/setup.sh — deploy console management on pfv-tsys4 +# +# Orchestrates the full setup: +# 1. Ensures ser2net + conman are installed +# 2. Copies mapping.txt to the target host (if running remotely) +# 3. Runs generate-config.sh to produce udev rules + ser2net.yaml + conman.conf +# 4. Reloads udev, creates /dev/consoles/ symlinks +# 5. Restarts ser2net (TCP ports on Tailscale IP) +# 6. Enables + starts conmand (logging + multiplexing) +# 7. Verifies +# +# This script is IDEMPOTENT — safe to run multiple times. +# +# Usage: +# PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/setup.sh +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +UDEV_RULES="/etc/udev/rules.d/99-console-ports.rules" +SER2NET_CONF="/etc/ser2net.yaml" +CONMAN_CONSOLES="/etc/conman/console-consoles.conf" +CONSOLE_DEV_DIR="/dev/consoles" + +echo "============================================" +echo " Console Management Setup" +echo " Host: $(hostname) $(date)" +echo "============================================" + +# --- 1. Install dependencies --- +echo "" +echo "--- [1/7] Checking dependencies ---" +NEED_INSTALL=() +dpkg -l ser2net 2>/dev/null | grep -q '^ii' && echo " ser2net: installed" || NEED_INSTALL+=(ser2net) +dpkg -l conman 2>/dev/null | grep -q '^ii' && echo " conman: installed" || NEED_INSTALL+=(conman) + +if [ "${#NEED_INSTALL[@]}" -gt 0 ]; then + echo " Installing: ${NEED_INSTALL[*]}" + apt-get update -qq + apt-get install -y -qq "${NEED_INSTALL[@]}" +else + echo " All dependencies present." +fi + +# --- 2. Ensure mapping file is available --- +echo "" +echo "--- [2/7] Locating mapping file ---" + +MAPPING_FILE="" +for candidate in \ + "$SCRIPT_DIR/mapping.txt" \ + "$(dirname "$0")/mapping.txt" \ + "/root/console/mapping.txt" \ + "/tmp/mapping.txt"; do + if [ -f "$candidate" ]; then + MAPPING_FILE="$candidate" + break + fi +done + +if [ -z "$MAPPING_FILE" ]; then + echo "FATAL: mapping.txt not found. Copy it to the target host." + exit 1 +fi +echo " Using: $MAPPING_FILE" + +# --- 3. Generate configs --- +echo "" +echo "--- [3/7] Generating configs ---" +export MAPPING_FILE +bash "$(dirname "$0")/generate-config.sh" 2>&1 || bash "$SCRIPT_DIR/generate-config.sh" 2>&1 || { + echo "FATAL: generate-config.sh failed." + exit 1 +} + +# --- 4. Reload udev + create symlinks --- +echo "" +echo "--- [4/7] Reloading udev rules ---" +udevadm control --reload-rules +# Try trigger first (works on some systems) +for tty in /sys/class/tty/ttyUSB*; do + [ -e "$tty" ] && udevadm trigger --action=add "$tty" 2>/dev/null || true +done +# Also try writing to uevent (forces udev reprocessing) +for tty in /sys/class/tty/ttyUSB*; do + [ -e "$tty/uevent" ] && echo "add" > "$tty/uevent" 2>/dev/null || true +done +sleep 2 + +# FALLBACK: if udev symlinks don't exist (common when devices are already +# discovered — udev trigger doesn't always re-create symlinks for existing +# devices), create them manually by matching ID_PATH. The udev rules will +# handle future boots/hotplugs automatically. +if [ ! -d /dev/consoles ] || [ -z "$(ls /dev/consoles/ 2>/dev/null)" ]; then + echo " udev trigger didn't create symlinks. Creating manually..." + mkdir -p /dev/consoles + while IFS= read -r line; do + line="${line%%#*}" + line="$(echo "$line" | xargs)" + [ -z "$line" ] && continue + IFS='|' read -r tcp_port name id_path baud comment <<< "$line" + # Find the ttyUSB whose ID_PATH contains the mapping's id_path substring + for tty in /dev/ttyUSB*; do + [ -e "$tty" ] || continue + DEV_IDPATH=$(udevadm info -q property -n "$tty" 2>/dev/null | grep ^ID_PATH= | cut -d= -f2) + if echo "$DEV_IDPATH" | grep -q "$id_path"; then + ln -sf "$tty" "/dev/consoles/$name" + echo " ln -s $tty -> /dev/consoles/$name" + break + fi + done + done < "$MAPPING_FILE" +fi + +echo " Stable symlinks:" +ls -la /dev/consoles/ 2>/dev/null | grep -v '^total\|^d' | sed 's/^/ /' || echo " (none created)" + +# Verify each symlink resolves +echo "" +echo " Symlink verification:" +while IFS= read -r line; do + line="${line%%#*}" + line="$(echo "$line" | xargs)" + [ -z "$line" ] && continue + IFS='|' read -r tcp_port name id_path baud comment <<< "$line" + if [ -e "/dev/consoles/$name" ]; then + TARGET=$(readlink -f "/dev/consoles/$name") + echo " [OK] /dev/consoles/$name -> $TARGET" + else + echo " [MISSING] /dev/consoles/$name (adapter unplugged or ID_PATH changed)" + fi +done < "$MAPPING_FILE" + +# --- 5. Restart ser2net --- +echo "" +echo "--- [5/7] Restarting ser2net ---" +systemctl enable ser2net +systemctl restart ser2net +sleep 2 + +if systemctl is-active --quiet ser2net; then + echo " ser2net is running." + TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1") + echo " Listening ports:" + ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/ /' +else + echo " WARNING: ser2net failed to start. Checking journal..." + journalctl -u ser2net --no-pager -n 20 +fi + +# --- 6. Enable + start conmand --- +echo "" +echo "--- [6/7] Starting conmand ---" + +# conman package on Debian may not ship a systemd unit. Create one if missing. +if ! systemctl cat conmand >/dev/null 2>&1; then + echo " No systemd unit for conmand — creating one..." + cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT' +[Unit] +Description=ConMan (Console Manager) +After=network.target ser2net.service +Requires=ser2net.service + +[Service] +Type=forking +ExecStart=/usr/sbin/conmand -c /etc/conman.conf +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target +CONMAND_UNIT + systemctl daemon-reload + echo " Created /etc/systemd/system/conmand.service" +fi + +# Kill any manually-started conmand first +pkill -x conmand 2>/dev/null || true +sleep 1 + +systemctl enable conmand 2>/dev/null || true +systemctl restart conmand 2>/dev/null || true +sleep 2 + +if systemctl is-active --quiet conmand; then + echo " conmand is running." + echo " Consoles:" + conman -q 2>&1 | sed 's/^/ /' || true +else + echo " WARNING: conmand failed to start. Checking journal..." + journalctl -u conmand --no-pager -n 20 2>/dev/null || true + # Try manual start as fallback + echo " Attempting manual start..." + /usr/sbin/conmand -c /etc/conman.conf 2>&1 || true +fi + +# --- 7. Summary --- +echo "" +echo "--- [7/7] Setup complete ---" +echo "" +echo " ser2net TCP ports (connect directly):" +echo " telnet 2001 # pfv-core-sw01" +echo " telnet 2002 # pfv-tor3-mgmt" +echo " ..." +echo "" +echo " conman consoles (with logging):" +echo " conman -f pfv-core-sw01" +echo " conman -q # query status" +echo "" +echo " To regenerate after changing mapping.txt:" +echo " bash generate-config.sh" +echo " udevadm trigger" +echo " systemctl restart ser2net conmand" +echo "============================================" diff --git a/console/validate-conman.sh b/console/validate-conman.sh new file mode 100644 index 0000000..bb46d1c --- /dev/null +++ b/console/validate-conman.sh @@ -0,0 +1,89 @@ +#!/usr/bin/bash +# +# console/validate-conman.sh — verify conman can actually reach devices via +# ser2net TCP ports and is capturing log output to files. +# +# This tests the real data path: conman → TCP 200X → ser2net → /dev/consoles/X → device +# +set -uo pipefail + +TS_IP=$(tailscale ip -4) +LOGDIR="/var/log/conman" + +echo "============================================" +echo " Conman Data Path + Log Validation" +echo " Host: $(hostname) TS IP: $TS_IP" +echo "============================================" + +echo "" +echo "--- 1. conman.conf log settings ---" +grep -E "logdir|LOGDIR|^GLOBAL LOG" /etc/conman.conf 2>/dev/null | grep -v "^#" || echo " (no explicit logdir — defaults to /var/log/conman)" +echo " Log dir: $LOGDIR" +ls -la "$LOGDIR"/ 2>/dev/null | head -15 || echo " ($LOGDIR does not exist yet)" + +echo "" +echo "--- 2. CONSOLE entries: each has a log= directive? ---" +# Extract the auto-generated block and check each CONSOLE line has log= +sed -n '/BEGIN PFV CONSOLE/,/END PFV CONSOLE/p' /etc/conman.conf | grep "^CONSOLE" | while read -r line; do + name=$(echo "$line" | sed -n 's/.*name="\([^"]*\)".*/\1/p') + if echo "$line" | grep -q 'log='; then + logfile=$(echo "$line" | sed -n 's/.*log="\([^"]*\)".*/\1/p') + echo " [OK] $name → log=$logfile" + else + echo " [FAIL] $name has NO log= directive" + fi +done + +echo "" +echo "--- 3. Trigger log capture: connect to each console briefly ---" +# conman -e changes the escape char. We use -j (join, read-only) with a timeout. +# Actually, conman doesn't have a built-in "connect for N seconds" — but conmand +# connects to each device ON STARTUP and keeps the connection open for logging. +# The log files should already be created. Let's check timestamps. + +echo " conmand connects to all consoles on startup. Checking if logs exist..." +echo "" +echo "--- 4. Log file inventory ---" +for name in pfv-core-sw01 pfv-tor3-mgmt pfv-tor3-stor pfv-rrinfra-rtr pfv-r2-tor-top subodev-torsw pfv-r2-sw; do + logfile="$LOGDIR/${name}.log" + if [ -f "$logfile" ]; then + SIZE=$(stat -c%s "$logfile" 2>/dev/null || echo 0) + MTIME=$(stat -c%y "$logfile" 2>/dev/null | cut -d. -f1) + echo " [OK] $logfile ($SIZE bytes, modified $MTIME)" + else + echo " [MISSING] $logfile — conmand may not be writing yet" + fi +done + +echo "" +echo "--- 5. conmand connection status (journal) ---" +# conmand logs connection attempts/errors to syslog +journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo " (no relevant journal entries)" + +echo "" +echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---" +# Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded) +echo " Probing TCP $TS_IP:2007 for data..." +RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5) +if [ -n "$RESPONSE" ]; then + echo " [OK] Data flowing through ser2net TCP 2007:" + echo "$RESPONSE" | sed 's/^/ /' +else + echo " (no immediate response — device may need more interaction)" +fi + +echo "" +echo "--- 7. Check if conmand has open connections to ser2net ports ---" +CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "") +if [ -n "$CONMAND_PID" ]; then + echo " conmand PID: $CONMAND_PID" + echo " Open connections to ser2net (expect 7 to 100.x:200X):" + ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/ /' + COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200") + echo " Total conmand→ser2net connections: $COUNT (expect 7)" +else + echo " [FAIL] conmand not running" +fi + +echo "" +echo "============================================" diff --git a/docs/docmap.md b/docs/docmap.md index cc95f0d..d0a0716 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -43,6 +43,7 @@ Server provisioning, security hardening, DNS/NTP configuration. | Document | Description | Last Reviewed | |----------|-------------|---------------| | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | +| [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |