mrcharles
fa0c77fb58
The ser2net layer between conman and the serial devices was causing
stair-stepping in terminal output. conman's dev="host:port" uses telnet
protocol, but ser2net's accepter was raw TCP (no telnet mode). The telnet
NVT state machine in conman was stripping bare CR characters from device
output — particularly from Dell switches that use old-style \n\r (LF+CR)
line endings instead of standard \r\n. In telnet, a bare \r not followed
by \n or NUL is non-compliant and gets dropped, leaving bare \n that
causes stair-stepping in the raw-mode terminal.
Fix: conman now opens serial devices directly via the stable udev
symlinks (/dev/consoles/<name>) with seropts, eliminating the telnet
layer entirely. ser2net is stopped and disabled but remains installed
for emergency TCP access (documented workflow: stop conmand, start
ser2net, use telnet, then reverse).
Architecture change:
Before: device → serial → ser2net (raw TCP) → conman (telnet NVT) → terminal
After: device → serial → conman (direct) → terminal
Verified: 7/7 serial devices held by conmand, 7/7 log files capturing,
all \r bytes preserved in terminal output (confirmed via PTY capture).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
4.8 KiB
Console Management (ser2net + conman)
Network-accessible serial console management for all production network switches and routers, running on pfv-tsys4 (storage server).
Architecture
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → conman (direct serial, logging + multiplexing)
conman opens the serial devices directly via stable udev symlinks and
provides session logging, output capture, and multi-user console
sharing. Clients connect to conmand over the Tailscale interface
(100.70.77.93:7890).
ser2net is installed but disabled — kept for emergency direct TCP access only (see Emergency Access below). It must NOT run simultaneously with conmand (both would compete for the serial devices).
The USB Enumeration Problem (SOLVED)
The 9 Prolific USB-to-DB9 adapters (067b:2303) on pfv-tsys4 have no
unique USB serial numbers and get assigned /dev/ttyUSB0-8 based on
enumeration order, which shifts on every boot. This made the old
/root/conmap + manual screen workflow break after every reboot.
Fix: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/ that never changes.
The udev rules are generated from mapping.txt, which maps each adapter's
ID_PATH to a console name and TCP port. To re-map after physically moving
an adapter, update mapping.txt and re-run setup.sh.
Fallback: if udev trigger doesn't create symlinks for already-discovered
devices (common on first run), setup.sh creates them manually by matching
ID_PATH. On subsequent boots, udev creates them automatically.
Port Assignments
| TCP Port | Console Name | ID_PATH | Description |
|---|---|---|---|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
TCP ports are used by the ser2net emergency config only. conman accesses
serial devices directly via /dev/consoles/<name> symlinks.
Scripts
| Script | Purpose |
|---|---|
mapping.txt |
Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
generate-config.sh |
Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
setup.sh |
Full deploy: generate configs, create symlinks, restart services |
discover.sh |
Read-only discovery of USB adapters, existing config, services |
Usage
Connect to a console
Primary method — conman client (with logging + multiplexing):
# From any Tailscale-connected workstation:
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
conman -d pfv-tsys4:7890 -q # list all consoles
Escape sequence: &. to disconnect, &? for help.
Emergency Access
If conmand is down or you need direct serial access:
# Option A — direct screen on pfv-tsys4:
ssh pfv-tsys4 'systemctl stop conmand'
ssh pfv-tsys4 'screen /dev/consoles/pfv-core-sw01'
ssh pfv-tsys4 'systemctl start conmand'
# Option B — enable ser2net for TCP access:
ssh pfv-tsys4 'systemctl stop conmand && systemctl start ser2net'
telnet pfv-tsys4 2001 # pfv-core-sw01
ssh pfv-tsys4 'systemctl stop ser2net && systemctl start conmand'
Never run ser2net and conmand at the same time — both compete for the same serial devices.
Re-deploy after changing mapping.txt
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
Find the ID_PATH for a new adapter
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
Then match the new adapter's ID_PATH to its physical location and add a line
to mapping.txt.
Files on pfv-tsys4
| File | Purpose |
|---|---|
/etc/udev/rules.d/99-console-ports.rules |
Stable symlinks by ID_PATH |
/etc/ser2net.yaml |
ser2net config (EMERGENCY ONLY — disabled by default) |
/etc/conman.conf |
conman config (CONSOLE entries between markers) |
/etc/systemd/system/conmand.service |
systemd unit for conmand |
/root/console/mapping.txt |
Copy of the source-of-truth mapping |
/root/console/setup.sh |
Setup script (re-runnable) |
/root/console/generate-config.sh |
Config generator |
Old workflow (replaced)
The old /root/conmap file and manual screen sessions are no longer
needed. The new setup is fully automated and survives reboots.