prep for next ai session

2026-08-01 15:45:23 -05:00
parent 46c35106fb
commit 6244c1cc25
139 changed files with 16712 additions and 0 deletions
@@ -0,0 +1,130 @@
+# Console Management (ser2net + conman)
+
+Network-accessible serial console management for all production network
+switches and routers, running on **pfv-tsys4** (storage server).
+
+## Architecture
+
+```
+USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net telnet(rfc2217) TCP → conman (logging + multiplexing)
+```
+
+ser2net owns the physical serial devices and exposes them on TCP ports
+using the **telnet(rfc2217) protocol** bound to the **Tailscale interface
+only** (`100.70.77.93:200X`). conman connects to those TCP ports via
+telnet for session logging, output capture, and multi-user console
+sharing.
+
+**Why telnet(rfc2217)?** The serial devices send `
+␍` (LF+CR) line
+endings instead of standard `
+`. Raw TCP transport caused conman's
+telnet NVT to strip bare CR characters, producing stair-stepped output.
+With telnet(rfc2217) on both sides, binary mode is negotiated and CR/LF
+translation is handled correctly by the telnet layer.
+
+**conman and ser2net do NOT share ports** — only one process can open a
+serial device at a time. ser2net owns the physical device; conman connects
+over TCP.
+
+## The USB Enumeration Problem (SOLVED)
+
+The 9 Prolific USB-to-DB9 adapters (`067b:2303`) on pfv-tsys4 have **no
+unique USB serial numbers** and get assigned `/dev/ttyUSB0-8` based on
+enumeration order, which shifts on every boot. This made the old
+`/root/conmap` + manual `screen` workflow break after every reboot.
+
+**Fix:** udev rules pin each adapter by its **ID_PATH** (physical USB port
+topology), which is stable across reboots regardless of enumeration order.
+Each adapter gets a named symlink in `/dev/consoles/` that never changes.
+
+The udev rules are generated from `mapping.txt`, which maps each adapter's
+ID_PATH to a console name and TCP port. To re-map after physically moving
+an adapter, update `mapping.txt` and re-run `setup.sh`.
+
+**Fallback:** if udev trigger doesn't create symlinks for already-discovered
+devices (common on first run), `setup.sh` creates them manually by matching
+ID_PATH. On subsequent boots, udev creates them automatically.
+
+## Port Assignments
+
+| TCP Port | Console Name | ID_PATH | Description |
+|----------|-------------|---------|-------------|
+| 2001 | pfv-core-sw01 | usb-0:1.5.4.4 | Dell PowerConnect 5448 (core switch) |
+| 2002 | pfv-tor3-mgmt | usb-0:1.6.3.1 | Rack 3 management TOR switch |
+| 2003 | pfv-tor3-stor | usb-0:1.6.3.3.2 | Rack 3 storage TOR switch |
+| 2004 | pfv-rrinfra-rtr | usb-0:1.6.3.3.1 | Cisco router (rrinfra) |
+| 2005 | pfv-r2-tor-top | usb-0:1.6.3.3.3 | Rack 2 top-of-rack switch |
+| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
+| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
+
+All ports listen on the Tailscale IP (`100.70.77.93`) using telnet(rfc2217).
+
+## Scripts
+
+| Script | Purpose |
+|--------|---------|
+| [`mapping.txt`](mapping.txt) | Source of truth: TCP port ↔ ID_PATH ↔ name ↔ baud |
+| [`generate-config.sh`](generate-config.sh) | Generates udev rules, ser2net.yaml, conman.conf from mapping.txt |
+| [`setup.sh`](setup.sh) | Full deploy: generate configs, create symlinks, restart services |
+| [`discover.sh`](discover.sh) | Read-only discovery of USB adapters, existing config, services |
+
+## Usage
+
+### Connect to a console
+
+**Primary method — conman client (with logging + multiplexing):**
+
+```bash
+# From any Tailscale-connected workstation:
+conman -d pfv-tsys4:7890 -f pfv-core-sw01     # connect to console
+conman -d pfv-tsys4:7890 -q                    # list all consoles
+```
+
+Escape sequence: `&.` to disconnect, `&?` for help.
+
+**Direct telnet (emergency only — conflicts with conman):**
+
+```bash
+# Direct telnet to ser2net works ONLY when conmand is stopped, because
+# conmand maintains persistent connections to all 7 TCP ports. Use:
+ssh pfv-tsys4 'systemctl stop conmand'
+telnet pfv-tsys4 2001                          # pfv-core-sw01
+ssh pfv-tsys4 'systemctl start conmand'        # restart when done
+```
+
+**Do NOT use telnet while conmand is running** — conmand will reconnect
+and kick your telnet session immediately ("Connection closed by foreign host").
+The correct workflow is conman client → conmand → ser2net → device.
+
+### Re-deploy after changing mapping.txt
+
+```bash
+PROX_HOST=pfv-tsys4 bash tests/remote.sh prox 'bash /root/console/setup.sh'
+```
+
+### Find the ID_PATH for a new adapter
+
+```bash
+PROX_HOST=pfv-tsys4 bash tests/remote.sh prox-file console/discover.sh
+```
+
+Then match the new adapter's ID_PATH to its physical location and add a line
+to `mapping.txt`.
+
+## Files on pfv-tsys4
+
+| File | Purpose |
+|------|---------|
+| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
+| `/etc/ser2net.yaml` | ser2net config (telnet rfc2217 TCP ports → serial symlinks) |
+| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
+| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
+| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
+| `/root/console/setup.sh` | Setup script (re-runnable) |
+| `/root/console/generate-config.sh` | Config generator |
+
+## Old workflow (replaced)
+
+The old `/root/conmap` file and manual `screen` sessions are no longer
+needed. The new setup is fully automated and survives reboots.