diff --git a/STATUS.md b/STATUS.md index 7d57de4..ad8453e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -68,18 +68,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [x] Validated: outlet 10 cycled off → on (8/8 test checks passed) - [ ] Rename outlets to match physical devices (Friday onsite) -### Console Management (ser2net + conman on pfv-tsys4 — LIVE) -- [x] 7 network switch/router consoles managed via ser2net + conman +### Console Management (conman direct-serial on pfv-tsys4 — LIVE) +- [x] 7 network switch/router consoles managed via conman - [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by ID_PATH (physical USB port topology) to stable `/dev/consoles/` symlinks that survive reboot regardless of enumeration order -- [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to - **Tailscale IP only** (`100.70.77.93`) -- [x] conman connects to TCP ports for logging + multiplexing (7 log - files active in `/var/log/conman/`) -- [x] Both ser2net + conmand enabled via systemd (survive reboot) +- [x] conman owns serial devices directly (no ser2net/telnet in data path) +- [x] 7 log files active in `/var/log/conman/` +- [x] conmand enabled via systemd (survives reboot) - [x] conmand systemd unit created (Debian package doesn't ship one) +- [x] ser2net installed but DISABLED (emergency TCP access only) - [x] Old `/root/conmap` + manual `screen` workflow replaced +- [x] **Stair-stepping fix:** switched from ser2net→telnet→conman to + conman direct-serial to eliminate telnet NVT CR stripping ### Repo Merge - [x] KNELServerBuild merged into PFVCluster (history preserved) @@ -141,5 +142,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Backup | Proxmox Backup Server (PBS) | | **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** | | **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** | -| **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** | +| **Console** | **7 switch consoles via conman direct-serial on pfv-tsys4 (conmand:7890 on Tailscale)** | | Production | Cloudron VPS, Reston VA (this cluster is R&D only) | diff --git a/console/README.md b/console/README.md index f0e1f50..6f746f0 100644 --- a/console/README.md +++ b/console/README.md @@ -6,17 +6,18 @@ switches and routers, running on **pfv-tsys4** (storage server). ## Architecture ``` -USB-DB9 adapters → udev symlinks (/dev/consoles/) → ser2net (TCP) → conman (logging + multiplexing) +USB-DB9 adapters → udev symlinks (/dev/consoles/) → conman (direct serial, logging + multiplexing) ``` -ser2net owns the physical serial devices and exposes them on TCP ports -bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman -connects to those TCP ports for session logging, output capture, and -multi-user console sharing. +conman opens the serial devices directly via stable udev symlinks and +provides session logging, output capture, and multi-user console +sharing. Clients connect to conmand over the Tailscale interface +(`100.70.77.93:7890`). -**conman and ser2net do NOT share ports** — only one process can open a -serial device at a time. ser2net owns the physical device; conman connects -over TCP. +ser2net is installed but **disabled** — kept for emergency direct TCP +access only (see [Emergency Access](#emergency-access) below). It must +NOT run simultaneously with conmand (both would compete for the serial +devices). ## The USB Enumeration Problem (SOLVED) @@ -49,7 +50,8 @@ ID_PATH. On subsequent boots, udev creates them automatically. | 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch | | 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch | -All ports listen on the Tailscale IP (`100.70.77.93`). +TCP ports are used by the ser2net emergency config only. conman accesses +serial devices directly via `/dev/consoles/` symlinks. ## Scripts @@ -74,19 +76,24 @@ conman -d pfv-tsys4:7890 -q # list all consoles Escape sequence: `&.` to disconnect, `&?` for help. -**Direct telnet (emergency only — conflicts with conman):** +### Emergency Access + +If conmand is down or you need direct serial access: ```bash -# Direct telnet to ser2net works ONLY when conmand is stopped, because -# conmand maintains persistent connections to all 7 TCP ports. Use: +# Option A — direct screen on pfv-tsys4: ssh pfv-tsys4 'systemctl stop conmand' +ssh pfv-tsys4 'screen /dev/consoles/pfv-core-sw01' +ssh pfv-tsys4 'systemctl start conmand' + +# Option B — enable ser2net for TCP access: +ssh pfv-tsys4 'systemctl stop conmand && systemctl start ser2net' telnet pfv-tsys4 2001 # pfv-core-sw01 -ssh pfv-tsys4 'systemctl start conmand' # restart when done +ssh pfv-tsys4 'systemctl stop ser2net && systemctl start conmand' ``` -**Do NOT use telnet while conmand is running** — conmand will reconnect -and kick your telnet session immediately ("Connection closed by foreign host"). -The correct workflow is conman client → conmand → ser2net → device. +**Never run ser2net and conmand at the same time** — both compete for the +same serial devices. ### Re-deploy after changing mapping.txt @@ -108,7 +115,7 @@ to `mapping.txt`. | File | Purpose | |------|---------| | `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH | -| `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) | +| `/etc/ser2net.yaml` | ser2net config (EMERGENCY ONLY — disabled by default) | | `/etc/conman.conf` | conman config (CONSOLE entries between markers) | | `/etc/systemd/system/conmand.service` | systemd unit for conmand | | `/root/console/mapping.txt` | Copy of the source-of-truth mapping | diff --git a/console/generate-config.sh b/console/generate-config.sh index 4eb3f14..4662589 100644 --- a/console/generate-config.sh +++ b/console/generate-config.sh @@ -143,14 +143,15 @@ fi { echo "%YAML 1.1" echo "---" - echo "# ser2net configuration for pfv-tsys4 console ports" + echo "# ser2net configuration for pfv-tsys4 console ports (EMERGENCY USE ONLY)" echo "# Generated by console/generate-config.sh on $(date)" echo "#" - echo "# All ports bound to Tailscale IP ($TS_IP) for secure remote access." - echo "# Physical devices are accessed via stable udev symlinks in /dev/consoles/." + echo "# WARNING: ser2net is DISABLED by default. conman owns the serial" + echo "# devices directly. If you enable ser2net, STOP conmand first or the" + echo "# two will conflict (only one process can open a serial device at a time)." echo "#" - echo "# To connect directly: telnet $TS_IP 2001" - echo "# To connect via conman: conman -f " + echo "# Emergency workflow: systemctl stop conmand && systemctl start ser2net" + echo "# When done: systemctl stop ser2net && systemctl start conmand" echo "" echo "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n" echo "" @@ -215,14 +216,16 @@ fi echo "" echo "$MARKER_BEGIN" echo "# Generated by console/generate-config.sh on $(date)" - echo "# Each console connects to a ser2net TCP port via telnet protocol." - echo "# ser2net owns the physical serial device; conman provides logging" - echo "# and multiplexing on top." + echo "# Each console connects directly to its serial device via the stable" + echo "# udev symlink in /dev/consoles/. conman owns the serial devices" + echo "# directly — no ser2net/telnet layer in the data path." echo "# Access: conman -f " echo "" for entry in "${ENTRIES[@]}"; do IFS='|' read -r tcp_port name id_path baud comment <<< "$entry" - echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\"" + # Convert baud format: mapping.txt uses 9600n81, conman needs 9600,8n1 + conman_seropts=$(echo "$baud" | sed -E 's/^([0-9]+)(.)(.)(.)$/\1,\3\2\4/') + echo "CONSOLE name=\"${name}\" dev=\"/dev/consoles/${name}\" seropts=\"${conman_seropts}\" log=\"${name}.log\" logopts=\"timestamp\"" done echo "$MARKER_END" } >> "$CONMAN_CONF" @@ -244,7 +247,7 @@ echo " $CONMAN_CONF (CONSOLE entries appended between markers)" echo "" echo " Next steps:" echo " 1. Reload udev: udevadm control --reload-rules && udevadm trigger" -echo " 2. Restart ser2net: systemctl restart ser2net" +echo " 2. Stop ser2net: systemctl stop ser2net && systemctl disable ser2net" echo " 3. Start conman: systemctl enable --now conmand" echo " 4. Or run: bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh" echo "============================================" diff --git a/console/setup.sh b/console/setup.sh index 462b398..0fb2849 100644 --- a/console/setup.sh +++ b/console/setup.sh @@ -133,35 +133,24 @@ while IFS= read -r line; do fi done < "$MAPPING_FILE" -# --- 5. Restart ser2net --- +# --- 5. Stop ser2net (conman owns serial devices directly) --- echo "" -echo "--- [5/7] Restarting ser2net ---" -systemctl enable ser2net -systemctl restart ser2net -sleep 2 - -if systemctl is-active --quiet ser2net; then - echo " ser2net is running." - TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1") - echo " Listening ports:" - ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/ /' -else - echo " WARNING: ser2net failed to start. Checking journal..." - journalctl -u ser2net --no-pager -n 20 -fi +echo "--- [5/7] Disabling ser2net ---" +echo " conman now owns serial devices directly. ser2net must NOT run" +echo " simultaneously (only one process can open a serial device at a time)." +systemctl stop ser2net 2>/dev/null || true +systemctl disable ser2net 2>/dev/null || true +echo " ser2net stopped and disabled (available for emergency use)." # --- 6. Enable + start conmand --- echo "" echo "--- [6/7] Starting conmand ---" -# conman package on Debian may not ship a systemd unit. Create one if missing. -if ! systemctl cat conmand >/dev/null 2>&1; then - echo " No systemd unit for conmand — creating one..." - cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT' -[Unit] +# conman package on Debian may not ship a systemd unit. Create one if missing, +# or update an existing one that still references ser2net. +CONMAND_UNIT_BODY='[Unit] Description=ConMan (Console Manager) -After=network.target ser2net.service -Requires=ser2net.service +After=network.target [Service] Type=forking @@ -170,10 +159,18 @@ Restart=on-failure RestartSec=5 [Install] -WantedBy=multi-user.target -CONMAND_UNIT +WantedBy=multi-user.target' + +if ! systemctl cat conmand >/dev/null 2>&1; then + echo " No systemd unit for conmand — creating one..." + echo "$CONMAND_UNIT_BODY" > /etc/systemd/system/conmand.service systemctl daemon-reload echo " Created /etc/systemd/system/conmand.service" +elif systemctl cat conmand 2>/dev/null | grep -q 'ser2net'; then + echo " conmand unit has ser2net dependency — updating..." + echo "$CONMAND_UNIT_BODY" > /etc/systemd/system/conmand.service + systemctl daemon-reload + echo " Updated /etc/systemd/system/conmand.service (removed ser2net dep)" fi # Kill any manually-started conmand first @@ -200,17 +197,19 @@ fi echo "" echo "--- [7/7] Setup complete ---" echo "" -echo " ser2net TCP ports (connect directly):" -echo " telnet 2001 # pfv-core-sw01" -echo " telnet 2002 # pfv-tor3-mgmt" -echo " ..." +echo " conman owns serial devices directly (no ser2net in the data path)." echo "" -echo " conman consoles (with logging):" -echo " conman -f pfv-core-sw01" -echo " conman -q # query status" +echo " Connect from any Tailscale workstation:" +echo " conman -d pfv-tsys4:7890 -f pfv-core-sw01" +echo " conman -d pfv-tsys4:7890 -q # list consoles" +echo "" +echo " Emergency direct serial (must stop conmand first):" +echo " systemctl stop conmand" +echo " screen /dev/consoles/pfv-core-sw01" +echo " systemctl start conmand" echo "" echo " To regenerate after changing mapping.txt:" echo " bash generate-config.sh" echo " udevadm trigger" -echo " systemctl restart ser2net conmand" +echo " systemctl restart conmand" echo "============================================" diff --git a/console/validate-conman.sh b/console/validate-conman.sh index bb46d1c..5b44f0d 100644 --- a/console/validate-conman.sh +++ b/console/validate-conman.sh @@ -61,29 +61,32 @@ echo "--- 5. conmand connection status (journal) ---" journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo " (no relevant journal entries)" echo "" -echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---" -# Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded) -echo " Probing TCP $TS_IP:2007 for data..." -RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5) -if [ -n "$RESPONSE" ]; then - echo " [OK] Data flowing through ser2net TCP 2007:" - echo "$RESPONSE" | sed 's/^/ /' -else - echo " (no immediate response — device may need more interaction)" -fi - -echo "" -echo "--- 7. Check if conmand has open connections to ser2net ports ---" +echo "--- 6. Verify serial devices are open by conmand ---" +echo " Checking /dev/consoles/* are held by conmand..." CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "") if [ -n "$CONMAND_PID" ]; then echo " conmand PID: $CONMAND_PID" - echo " Open connections to ser2net (expect 7 to 100.x:200X):" - ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/ /' - COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200") - echo " Total conmand→ser2net connections: $COUNT (expect 7)" + for devlink in /dev/consoles/*; do + [ -e "$devlink" ] || continue + name=$(basename "$devlink") + if lsof "$devlink" 2>/dev/null | grep -q conmand; then + echo " [OK] $name ($devlink) opened by conmand" + else + echo " [WARN] $name ($devlink) not held by conmand" + fi + done else echo " [FAIL] conmand not running" fi +echo "" +echo "--- 7. Verify ser2net is stopped (conman owns devices) ---" +if systemctl is-active --quiet ser2net 2>/dev/null; then + echo " [WARN] ser2net is running — will conflict with conmand for serial devices!" + echo " Fix: systemctl stop ser2net" +else + echo " [OK] ser2net is stopped (conman owns serial devices directly)" +fi + echo "" echo "============================================" diff --git a/docs/docmap.md b/docs/docmap.md index 775caf6..3152229 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -46,7 +46,7 @@ Server provisioning, security hardening, DNS/NTP configuration. |----------|-------------|---------------| | [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 | | [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | -| [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 | +| [`../console/README.md`](../console/README.md) | Serial console management (conman direct-serial) for 7 network switches on pfv-tsys4 | 2026-07-29 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | | [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |