|
|
@@ -1,130 +1,132 @@
|
|
|
|
# Console Management (ser2net + conman)
|
|
|
|
# Console Management (ser2net + conman)
|
|
|
|
|
|
|
|
|
|
|
|
Network-accessible serial console management for all production network
|
|
|
|
> **Redmine:** [#360](https://projects.knownelement.com/issues/360) (deployment, closed) · [#373](https://projects.knownelement.com/issues/373) (pfv-r2-sw fix, open)
|
|
|
|
switches and routers, running on **pfv-tsys4** (storage server).
|
|
|
|
|
|
|
|
|
|
|
|
Network-accessible serial console management for all production network
|
|
|
|
## Architecture
|
|
|
|
switches and routers, running on **pfv-tsys4** (storage server).
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
## Architecture
|
|
|
|
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
|
|
|
|
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
|
|
|
|
ser2net owns the physical serial devices and exposes them on TCP ports
|
|
|
|
```
|
|
|
|
using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
|
|
|
|
|
|
|
|
only** (`100.70.77.93:200X`). conman connects to those TCP ports via
|
|
|
|
ser2net owns the physical serial devices and exposes them on TCP ports
|
|
|
|
telnet for session logging, output capture, and multi-user console
|
|
|
|
using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
|
|
|
|
sharing.
|
|
|
|
only** (`100.70.77.93:200X`). conman connects to those TCP ports via
|
|
|
|
|
|
|
|
telnet for session logging, output capture, and multi-user console
|
|
|
|
**Why telnet(rfc2217)?** The serial devices send `
|
|
|
|
sharing.
|
|
|
|
␍` (LF+CR) line
|
|
|
|
|
|
|
|
|
|
|
|
**Why telnet(rfc2217)?** The serial devices send `
|
|
|
|
|
|
|
|
␍` (LF+CR) line
|
|
|
|
endings instead of standard `
|
|
|
|
endings instead of standard `
|
|
|
|
`. Raw TCP transport caused conman's
|
|
|
|
`. Raw TCP transport caused conman's
|
|
|
|
telnet NVT to strip bare CR characters, producing stair-stepped output.
|
|
|
|
telnet NVT to strip bare CR characters, producing stair-stepped output.
|
|
|
|
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
|
|
|
|
With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
|
|
|
|
translation is handled correctly by the telnet layer.
|
|
|
|
translation is handled correctly by the telnet layer.
|
|
|
|
|
|
|
|
|
|
|
|
**conman and ser2net do NOT share ports** — only one process can open a
|
|
|
|
**conman and ser2net do NOT share ports** — only one process can open a
|
|
|
|
serial device at a time. ser2net owns the physical device; conman connects
|
|
|
|
serial device at a time. ser2net owns the physical device; conman connects
|
|
|
|
over TCP.
|
|
|
|
over TCP.
|
|
|
|
|
|
|
|
|
|
|
|
## The USB Enumeration Problem (SOLVED)
|
|
|
|
## The USB Enumeration Problem (SOLVED)
|
|
|
|
|
|
|
|
|
|
|
|
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
|
|
|
|
The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
|
|
|
|
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
|
|
|
|
unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
|
|
|
|
enumeration order, which shifts on every boot. This made the old
|
|
|
|
enumeration order, which shifts on every boot. This made the old
|
|
|
|
`/root/conmap` + manual `screen` workflow break after every reboot.
|
|
|
|
`/root/conmap` + manual `screen` workflow break after every reboot.
|
|
|
|
|
|
|
|
|
|
|
|
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
|
|
|
|
**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
|
|
|
|
topology), which is stable across reboots regardless of enumeration order.
|
|
|
|
topology), which is stable across reboots regardless of enumeration order.
|
|
|
|
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
|
|
|
|
Each adapter gets a named symlink in `/dev/consoles/` that never changes.
|
|
|
|
|
|
|
|
|
|
|
|
The udev rules are generated from `mapping.txt`, which maps each adapter's
|
|
|
|
The udev rules are generated from `mapping.txt`, which maps each adapter's
|
|
|
|
ID_PATH to a console name and TCP port. To re-map after physically moving
|
|
|
|
ID_PATH to a console name and TCP port. To re-map after physically moving
|
|
|
|
an adapter, update `mapping.txt` and re-run `setup.sh`.
|
|
|
|
an adapter, update `mapping.txt` and re-run `setup.sh`.
|
|
|
|
|
|
|
|
|
|
|
|
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
|
|
|
|
**Fallback:** if udev trigger doesn't create symlinks for already-discovered
|
|
|
|
devices (common on first run), `setup.sh` creates them manually by matching
|
|
|
|
devices (common on first run), `setup.sh` creates them manually by matching
|
|
|
|
ID_PATH. On subsequent boots, udev creates them automatically.
|
|
|
|
ID_PATH. On subsequent boots, udev creates them automatically.
|
|
|
|
|
|
|
|
|
|
|
|
## Port Assignments
|
|
|
|
## Port Assignments
|
|
|
|
|
|
|
|
|
|
|
|
| TCP Port | Console Name | ID_PATH | Description |
|
|
|
|
| TCP Port | Console Name | ID_PATH | Description |
|
|
|
|
|----------|-------------|---------|-------------|
|
|
|
|
|----------|-------------|---------|-------------|
|
|
|
|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
|
|
|
|
| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
|
|
|
|
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
|
|
|
|
| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
|
|
|
|
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
|
|
|
|
| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
|
|
|
|
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
|
|
|
|
| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
|
|
|
|
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
|
|
|
|
| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
|
|
|
|
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
|
|
|
|
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
|
|
|
|
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
|
|
|
|
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
|
|
|
|
|
|
|
|
|
|
|
|
All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
|
|
|
|
All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
|
|
|
|
|
|
|
|
|
|
|
|
## Scripts
|
|
|
|
## Scripts
|
|
|
|
|
|
|
|
|
|
|
|
| Script | Purpose |
|
|
|
|
| Script | Purpose |
|
|
|
|
|--------|---------|
|
|
|
|
|--------|---------|
|
|
|
|
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
|
|
|
|
| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
|
|
|
|
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
|
|
|
|
| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
|
|
|
|
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
|
|
|
|
| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
|
|
|
|
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
|
|
|
|
| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
|
|
|
|
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
## Usage
|
|
|
|
|
|
|
|
|
|
|
|
### Connect to a console
|
|
|
|
### Connect to a console
|
|
|
|
|
|
|
|
|
|
|
|
**Primary method — conman client (with logging + multiplexing):**
|
|
|
|
**Primary method — conman client (with logging + multiplexing):**
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
```bash
|
|
|
|
# From any Tailscale-connected workstation:
|
|
|
|
# From any Tailscale-connected workstation:
|
|
|
|
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
|
|
|
|
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # connect to console
|
|
|
|
conman -d pfv-tsys4:7890 -q # list all consoles
|
|
|
|
conman -d pfv-tsys4:7890 -q # list all consoles
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Escape sequence: `&.` to disconnect, `&?` for help.
|
|
|
|
Escape sequence: `&.` to disconnect, `&?` for help.
|
|
|
|
|
|
|
|
|
|
|
|
**Direct telnet (emergency only — conflicts with conman):**
|
|
|
|
**Direct telnet (emergency only — conflicts with conman):**
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
```bash
|
|
|
|
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
|
|
|
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
|
|
|
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
|
|
|
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
|
|
|
ssh pfv-tsys4 'systemctl stop conmand'
|
|
|
|
ssh pfv-tsys4 'systemctl stop conmand'
|
|
|
|
telnet pfv-tsys4 2001 # pfv-core-sw01
|
|
|
|
telnet pfv-tsys4 2001 # pfv-core-sw01
|
|
|
|
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
|
|
|
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
|
|
|
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
|
|
|
and kick your telnet session immediately ("Connection closed by foreign host").
|
|
|
|
and kick your telnet session immediately ("Connection closed by foreign host").
|
|
|
|
The correct workflow is conman client → conmand → ser2net → device.
|
|
|
|
The correct workflow is conman client → conmand → ser2net → device.
|
|
|
|
|
|
|
|
|
|
|
|
### Re-deploy after changing mapping.txt
|
|
|
|
### Re-deploy after changing mapping.txt
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
```bash
|
|
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
|
|
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### Find the ID_PATH for a new adapter
|
|
|
|
### Find the ID_PATH for a new adapter
|
|
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
```bash
|
|
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
|
|
|
|
PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Then match the new adapter's ID_PATH to its physical location and add a line
|
|
|
|
Then match the new adapter's ID_PATH to its physical location and add a line
|
|
|
|
to `mapping.txt`.
|
|
|
|
to `mapping.txt`.
|
|
|
|
|
|
|
|
|
|
|
|
## Files on pfv-tsys4
|
|
|
|
## Files on pfv-tsys4
|
|
|
|
|
|
|
|
|
|
|
|
| File | Purpose |
|
|
|
|
| File | Purpose |
|
|
|
|
|------|---------|
|
|
|
|
|------|---------|
|
|
|
|
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
|
|
|
|
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
|
|
|
|
| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
|
|
|
|
| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
|
|
|
|
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
|
|
|
|
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
|
|
|
|
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
|
|
|
|
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
|
|
|
|
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
|
|
|
|
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
|
|
|
|
| `/root/console/setup.sh` | Setup script (re-runnable) |
|
|
|
|
| `/root/console/setup.sh` | Setup script (re-runnable) |
|
|
|
|
| `/root/console/generate-config.sh` | Config generator |
|
|
|
|
| `/root/console/generate-config.sh` | Config generator |
|
|
|
|
|
|
|
|
|
|
|
|
## Old workflow (replaced)
|
|
|
|
## Old workflow (replaced)
|
|
|
|
|
|
|
|
|
|
|
|
The old `/root/conmap` file and manual `screen` sessions are no longer
|
|
|
|
The old `/root/conmap` file and manual `screen` sessions are no longer
|
|
|
|
needed. The new setup is fully automated and survives reboots.
|
|
|
|
needed. The new setup is fully automated and survives reboots.
|
|
|
|
