mrcharles
51db52f862
Replace all occurrences of the fleet admin password with REDACTED_PASSWORD.
Git history will be rewritten next to purge it from all prior commits.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Console Management (ser2net + conman)
Network-accessible serial console management for all production network switches and routers, running on pfv-tsys4 (storage server).
Architecture
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
ser2net owns the physical serial devices and exposes them on TCP ports
using the telnet(rfc2217) protocol bound to the Tailscale interface
only (100.70.77.93:200X). conman connects to those TCP ports via
telnet for session logging, output capture, and multi-user console
sharing.
Why telnet(rfc2217)? The serial devices send `
(LF+CR) line endings instead of standard
`. Raw TCP transport caused conman's
telnet NVT to strip bare CR characters, producing stair-stepped output.
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
translation is handled correctly by the telnet layer.
conman and ser2net do NOT share ports — only one process can open a serial device at a time. ser2net owns the physical device; conman connects over TCP.
The USB Enumeration Problem (SOLVED)
The 9 Prolific USB-to-DB9 adapters (067b:2303) on pfv-tsys4 have no
unique USB serial numbers and get assigned /dev/ttyUSB0-8 based on
enumeration order, which shifts on every boot. This made the old
/root/conmap + manual screen workflow break after every reboot.
Fix: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/ that never changes.
The udev rules are generated from mapping.txt, which maps each adapter's
ID_PATH to a console name and TCP port. To re-map after physically moving
an adapter, update mapping.txt and re-run setup.sh.
Fallback: if udev trigger doesn't create symlinks for already-discovered
devices (common on first run), setup.sh creates them manually by matching
ID_PATH. On subsequent boots, udev creates them automatically.
Port Assignments
| TCP Port | Console Name | ID_PATH | Description |
|---|---|---|---|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
All ports listen on the Tailscale IP (100.70.77.93) using telnet(rfc2217).
Scripts
| Script | Purpose |
|---|---|
mapping.txt |
Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
generate-config.sh |
Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
setup.sh |
Full deploy: generate configs, create symlinks, restart services |
discover.sh |
Read-only discovery of USB adapters, existing config, services |
Usage
Connect to a console
Primary method — conman client (with logging + multiplexing):
# From any Tailscale-connected workstation:
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
conman -d pfv-tsys4:7890 -q # list all consoles
Escape sequence: &. to disconnect, &? for help.
Direct telnet (emergency only — conflicts with conman):
# Direct telnet to ser2net works ONLY when conmand is stopped, because
# conmand maintains persistent connections to all 7 TCP ports. Use:
ssh pfv-tsys4 'systemctl stop conmand'
telnet pfv-tsys4 2001 # pfv-core-sw01
ssh pfv-tsys4 'systemctl start conmand' # restart when done
Do NOT use telnet while conmand is running — conmand will reconnect and kick your telnet session immediately ("Connection closed by foreign host"). The correct workflow is conman client → conmand → ser2net → device.
Re-deploy after changing mapping.txt
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
Find the ID_PATH for a new adapter
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
Then match the new adapter's ID_PATH to its physical location and add a line
to mapping.txt.
Files on pfv-tsys4
| File | Purpose |
|---|---|
/etc/udev/rules.d/99-console-ports.rules |
Stable symlinks by ID_PATH |
/etc/ser2net.yaml |
ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
/etc/conman.conf |
conman config (CONSOLE entries between markers) |
/etc/systemd/system/conmand.service |
systemd unit for conmand |
/root/console/mapping.txt |
Copy of the source-of-truth mapping |
/root/console/setup.sh |
Setup script (re-runnable) |
/root/console/generate-config.sh |
Config generator |
Old workflow (replaced)
The old /root/conmap file and manual screen sessions are no longer
needed. The new setup is fully automated and survives reboots.