mrcharles
6c58671d15
Each component README now links to its corresponding Redmine tickets (closed for completed work, open for pending items) for bidirectional traceability between code and system of record.
133 lines
5.4 KiB
Markdown
133 lines
5.4 KiB
Markdown
# Console Management (ser2net + conman)
|
|
|
|
> **Redmine:** [#360](https://projects.knownelement.com/issues/360) (deployment, closed) · [#373](https://projects.knownelement.com/issues/373) (pfv-r2-sw fix, open)
|
|
|
|
Network-accessible serial console management for all production network
|
|
switches and routers, running on **pfv-tsys4** (storage server).
|
|
|
|
## Architecture
|
|
|
|
```
|
|
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
|
|
```
|
|
|
|
ser2net owns the physical serial devices and exposes them on TCP ports
|
|
using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
|
|
only** (`100.70.77.93:200X`). conman connects to those TCP ports via
|
|
telnet for session logging, output capture, and multi-user console
|
|
sharing.
|
|
|
|
**Why telnet(rfc2217)?** The serial devices send `
|
|
|
|
` (LF+CR) line
|
|
endings instead of standard `
|
|
`. Raw TCP transport caused conman's
|
|
telnet NVT to strip bare CR characters, producing stair-stepped output.
|
|
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
|
|
translation is handled correctly by the telnet layer.
|
|
|
|
**conman and ser2net do NOT share ports** — only one process can open a
|
|
serial device at a time. ser2net owns the physical device; conman connects
|
|
over TCP.
|
|
|
|
## The USB Enumeration Problem (SOLVED)
|
|
|
|
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
|
|
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
|
|
enumeration order, which shifts on every boot. This made the old
|
|
`/root/conmap` + manual `screen` workflow break after every reboot.
|
|
|
|
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
|
|
topology), which is stable across reboots regardless of enumeration order.
|
|
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
|
|
|
|
The udev rules are generated from `mapping.txt`, which maps each adapter's
|
|
ID_PATH to a console name and TCP port. To re-map after physically moving
|
|
an adapter, update `mapping.txt` and re-run `setup.sh`.
|
|
|
|
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
|
|
devices (common on first run), `setup.sh` creates them manually by matching
|
|
ID_PATH. On subsequent boots, udev creates them automatically.
|
|
|
|
## Port Assignments
|
|
|
|
| TCP Port | Console Name | ID_PATH | Description |
|
|
|----------|-------------|---------|-------------|
|
|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
|
|
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
|
|
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
|
|
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
|
|
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
|
|
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
|
|
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
|
|
|
|
All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
|
|
|
|
## Scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
|
|
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
|
|
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
|
|
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
|
|
|
|
## Usage
|
|
|
|
### Connect to a console
|
|
|
|
**Primary method — conman client (with logging + multiplexing):**
|
|
|
|
```bash
|
|
# From any Tailscale-connected workstation:
|
|
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
|
|
conman -d pfv-tsys4:7890 -q # list all consoles
|
|
```
|
|
|
|
Escape sequence: `&.` to disconnect, `&?` for help.
|
|
|
|
**Direct telnet (emergency only — conflicts with conman):**
|
|
|
|
```bash
|
|
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
|
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
|
ssh pfv-tsys4 'systemctl stop conmand'
|
|
telnet pfv-tsys4 2001 # pfv-core-sw01
|
|
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
|
```
|
|
|
|
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
|
and kick your telnet session immediately ("Connection closed by foreign host").
|
|
The correct workflow is conman client → conmand → ser2net → device.
|
|
|
|
### Re-deploy after changing mapping.txt
|
|
|
|
```bash
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
|
|
```
|
|
|
|
### Find the ID_PATH for a new adapter
|
|
|
|
```bash
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
|
|
```
|
|
|
|
Then match the new adapter's ID_PATH to its physical location and add a line
|
|
to `mapping.txt`.
|
|
|
|
## Files on pfv-tsys4
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
|
|
| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
|
|
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
|
|
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
|
|
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
|
|
| `/root/console/setup.sh` | Setup script (re-runnable) |
|
|
| `/root/console/generate-config.sh` | Config generator |
|
|
|
|
## Old workflow (replaced)
|
|
|
|
The old `/root/conmap` file and manual `screen` sessions are no longer
|
|
needed. The new setup is fully automated and survives reboots.
|