KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(console): manage 7 switch consoles via ser2net+conman on pfv-tsys4

Browse Source 
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.

Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.

Architecture (layered, no port sharing):
  USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)

Port assignments (all on Tailscale IP 100.70.77.93):
  2001 = pfv-core-sw01     2002 = pfv-tor3-mgmt    2003 = pfv-tor3-stor
  2004 = pfv-rrinfra-rtr   2005 = pfv-r2-tor-top   2006 = subodev-torsw
  2007 = pfv-r2-sw

Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
  entries from mapping.txt. Idempotent (markers in conman.conf for clean
  regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
  + manual fallback for already-discovered devices), create conmand
  systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture

Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
  a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
  directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
  /etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
  /var/log/conman/. Fixed by adding server logdir directive.

Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-28 19:44:02 -05:00
parent e8969b5d12
commit 815d07bbca
10 changed files with 807 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
console/README.md
+109
View File
@@ -0,0 +1,109 @@
# Console Management (ser2net + conman)
Network-accessible serial console management for all production network
switches and routers, running on **pfv-tsys4** (storage server).
## Architecture
```
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net (TCP) → conman (logging + multiplexing)
```
ser2net owns the physical serial devices and exposes them on TCP ports
bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman
connects to those TCP ports for session logging, output capture, and
multi-user console sharing.
**conman and ser2net do NOT share ports** — only one process can open a
serial device at a time. ser2net owns the physical device; conman connects
over TCP.
## The USB Enumeration Problem (SOLVED)
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
enumeration order, which shifts on every boot. This made the old
`/root/conmap` + manual `screen` workflow break after every reboot.
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
The udev rules are generated from `mapping.txt`, which maps each adapter's
ID_PATH to a console name and TCP port. To re-map after physically moving
an adapter, update `mapping.txt` and re-run `setup.sh`.
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
devices (common on first run), `setup.sh` creates them manually by matching
ID_PATH. On subsequent boots, udev creates them automatically.
## Port Assignments
| TCP Port | Console Name | ID_PATH | Description |
|----------|-------------|---------|-------------|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
All ports listen on the Tailscale IP (`100.70.77.93`).
## Scripts
| Script | Purpose |
|--------|---------|
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
## Usage
### Connect to a console
```bash
# Via conman (with logging + multiplexing):
conman -f pfv-core-sw01 # attach to console
conman -q # query status of all consoles
# Via telnet (direct, no logging):
telnet 100.70.77.93 2001 # pfv-core-sw01
# From any Tailscale-connected device:
telnet pfv-tsys4 2001
```
### Re-deploy after changing mapping.txt
```bash
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
```
### Find the ID_PATH for a new adapter
```bash
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
```
Then match the new adapter's ID_PATH to its physical location and add a line
to `mapping.txt`.
## Files on pfv-tsys4
| File | Purpose |
|------|---------|
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
| `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) |
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
| `/root/console/setup.sh` | Setup script (re-runnable) |
| `/root/console/generate-config.sh` | Config generator |
## Old workflow (replaced)
The old `/root/conmap` file and manual `screen` sessions are no longer
needed. The new setup is fully automated and survives reboots.