PFVCluster/dcinfra/console

Console Management (ser2net + conman)

Redmine: #360 (deployment, closed) · #373 (pfv-r2-sw fix, open)

Network-accessible serial console management for all production network switches and routers, running on pfv-tsys4 (storage server).

Architecture

USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)

ser2net owns the physical serial devices and exposes them on TCP ports using the telnet(rfc2217) protocol bound to the Tailscale interface only (100.70.77.93:200X). conman connects to those TCP ports via telnet for session logging, output capture, and multi-user console sharing.

Why telnet(rfc2217)? The serial devices send `

(LF+CR) line endings instead of standard `. Raw TCP transport caused conman's telnet NVT to strip bare CR characters, producing stair-stepped output. With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF translation is handled correctly by the telnet layer.

conman and ser2net do NOT share ports — only one process can open a serial device at a time. ser2net owns the physical device; conman connects over TCP.

The USB Enumeration Problem (SOLVED)

The 9 Prolific USB-to-DB9 adapters (067b:2303) on pfv-tsys4 have no unique USB serial numbers and get assigned /dev/ttyUSB0-8 based on enumeration order, which shifts on every boot. This made the old /root/conmap + manual screen workflow break after every reboot.

Fix: udev rules pin each adapter by its ID_PATH (physical USB port topology), which is stable across reboots regardless of enumeration order. Each adapter gets a named symlink in /dev/consoles/ that never changes.

The udev rules are generated from mapping.txt, which maps each adapter's ID_PATH to a console name and TCP port. To re-map after physically moving an adapter, update mapping.txt and re-run setup.sh.

Fallback: if udev trigger doesn't create symlinks for already-discovered devices (common on first run), setup.sh creates them manually by matching ID_PATH. On subsequent boots, udev creates them automatically.

Port Assignments

TCP Port	Console Name	ID_PATH	Description
2001	pfv-core-sw01	usb-0:1.5.4.4	Dell PowerConnect 5448 (core switch)
2002	pfv-tor3-mgmt	usb-0:1.6.3.1	Rack 3 management TOR switch
2003	pfv-tor3-stor	usb-0:1.6.3.3.2	Rack 3 storage TOR switch
2004	pfv-rrinfra-rtr	usb-0:1.6.3.3.1	Cisco router (rrinfra)
2005	pfv-r2-tor-top	usb-0:1.6.3.3.3	Rack 2 top-of-rack switch
2006	subodev-torsw	usb-0:1.5.4.1	Suborbital device TOR switch
2007	pfv-r2-sw	usb-0:1.6.3.2	Rack 2 old Dell switch

All ports listen on the Tailscale IP (100.70.77.93) using telnet(rfc2217).

Scripts

Script	Purpose
mapping.txt	Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud
generate-config.sh	Generates udev rules, ser2net.yaml, conman.conf from mapping.txt
setup.sh	Full deploy: generate configs, create symlinks, restart services
discover.sh	Read-only discovery of USB adapters, existing config, services

Usage

Connect to a console

Primary method — conman client (with logging + multiplexing):

# From any Tailscale-connected workstation:
conman -d pfv-tsys4:7890 -f pfv-core-sw01     # connect to console
conman -d pfv-tsys4:7890 -q                    # list all consoles

Escape sequence: &. to disconnect, &? for help.

Direct telnet (emergency only — conflicts with conman):

# Direct telnet to ser2net works ONLY when conmand is stopped, because
# conmand maintains persistent connections to all 7 TCP ports. Use:
ssh pfv-tsys4 'systemctl stop conmand'
telnet pfv-tsys4 2001                          # pfv-core-sw01
ssh pfv-tsys4 'systemctl start conmand'        # restart when done

Do NOT use telnet while conmand is running — conmand will reconnect and kick your telnet session immediately ("Connection closed by foreign host"). The correct workflow is conman client → conmand → ser2net → device.

Re-deploy after changing mapping.txt

PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'

Find the ID_PATH for a new adapter

PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh

Then match the new adapter's ID_PATH to its physical location and add a line to mapping.txt.

Files on pfv-tsys4

File	Purpose
/etc/udev/rules.d/99-console-ports.rules	Stable symlinks by ID_PATH
/etc/ser2net.yaml	ser2net config (telnet rfc2217 TCP ports → serial symlinks)
/etc/conman.conf	conman config (CONSOLE entries between markers)
/etc/systemd/system/conmand.service	systemd unit for conmand
/root/console/mapping.txt	Copy of the source-of-truth mapping
/root/console/setup.sh	Setup script (re-runnable)
/root/console/generate-config.sh	Config generator

Old workflow (replaced)

The old /root/conmap file and manual screen sessions are no longer needed. The new setup is fully automated and survives reboots.