fix(console): switch conman from ser2net-telnet to direct serial access
The ser2net layer between conman and the serial devices was causing
stair-stepping in terminal output. conman's dev="host:port" uses telnet
protocol, but ser2net's accepter was raw TCP (no telnet mode). The telnet
NVT state machine in conman was stripping bare CR characters from device
output — particularly from Dell switches that use old-style \n\r (LF+CR)
line endings instead of standard \r\n. In telnet, a bare \r not followed
by \n or NUL is non-compliant and gets dropped, leaving bare \n that
causes stair-stepping in the raw-mode terminal.
Fix: conman now opens serial devices directly via the stable udev
symlinks (/dev/consoles/<name>) with seropts, eliminating the telnet
layer entirely. ser2net is stopped and disabled but remains installed
for emergency TCP access (documented workflow: stop conmand, start
ser2net, use telnet, then reverse).
Architecture change:
Before: device → serial → ser2net (raw TCP) → conman (telnet NVT) → terminal
After: device → serial → conman (direct) → terminal
Verified: 7/7 serial devices held by conmand, 7/7 log files capturing,
all \r bytes preserved in terminal output (confirmed via PTY capture).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -68,18 +68,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
|
||||
- [ ] Rename outlets to match physical devices (Friday onsite)
|
||||
|
||||
### Console Management (ser2net + conman on pfv-tsys4 — LIVE)
|
||||
- [x] 7 network switch/router consoles managed via ser2net + conman
|
||||
### Console Management (conman direct-serial on pfv-tsys4 — LIVE)
|
||||
- [x] 7 network switch/router consoles managed via conman
|
||||
- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
|
||||
ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
|
||||
symlinks that survive reboot regardless of enumeration order
|
||||
- [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
|
||||
**Tailscale IP only** (`100.70.77.93`)
|
||||
- [x] conman connects to TCP ports for logging + multiplexing (7 log
|
||||
files active in `/var/log/conman/`)
|
||||
- [x] Both ser2net + conmand enabled via systemd (survive reboot)
|
||||
- [x] conman owns serial devices directly (no ser2net/telnet in data path)
|
||||
- [x] 7 log files active in `/var/log/conman/`
|
||||
- [x] conmand enabled via systemd (survives reboot)
|
||||
- [x] conmand systemd unit created (Debian package doesn't ship one)
|
||||
- [x] ser2net installed but DISABLED (emergency TCP access only)
|
||||
- [x] Old `/root/conmap` + manual `screen` workflow replaced
|
||||
- [x] **Stair-stepping fix:** switched from ser2net→telnet→conman to
|
||||
conman direct-serial to eliminate telnet NVT CR stripping
|
||||
|
||||
### Repo Merge
|
||||
- [x] KNELServerBuild merged into PFVCluster (history preserved)
|
||||
@@ -141,5 +142,5 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
| Backup | Proxmox Backup Server (PBS) |
|
||||
| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
|
||||
| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
|
||||
| **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** |
|
||||
| **Console** | **7 switch consoles via conman direct-serial on pfv-tsys4 (conmand:7890 on Tailscale)** |
|
||||
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
|
||||
|
||||
+24
-17
@@ -6,17 +6,18 @@ switches and routers, running on **pfv-tsys4** (storage server).
|
||||
## Architecture
|
||||
|
||||
```
|
||||
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → ser2net (TCP) → conman (logging + multiplexing)
|
||||
USB-DB9 adapters → udev symlinks (/dev/consoles/<name>) → conman (direct serial, logging + multiplexing)
|
||||
```
|
||||
|
||||
ser2net owns the physical serial devices and exposes them on TCP ports
|
||||
bound to the **Tailscale interface only** (`100.70.77.93:200X`). conman
|
||||
connects to those TCP ports for session logging, output capture, and
|
||||
multi-user console sharing.
|
||||
conman opens the serial devices directly via stable udev symlinks and
|
||||
provides session logging, output capture, and multi-user console
|
||||
sharing. Clients connect to conmand over the Tailscale interface
|
||||
(`100.70.77.93:7890`).
|
||||
|
||||
**conman and ser2net do NOT share ports** — only one process can open a
|
||||
serial device at a time. ser2net owns the physical device; conman connects
|
||||
over TCP.
|
||||
ser2net is installed but **disabled** — kept for emergency direct TCP
|
||||
access only (see [Emergency Access](#emergency-access) below). It must
|
||||
NOT run simultaneously with conmand (both would compete for the serial
|
||||
devices).
|
||||
|
||||
## The USB Enumeration Problem (SOLVED)
|
||||
|
||||
@@ -49,7 +50,8 @@ ID_PATH. On subsequent boots, udev creates them automatically.
|
||||
| 2006 | subodev-torsw | usb-0:1.5.4.1 | Suborbital device TOR switch |
|
||||
| 2007 | pfv-r2-sw | usb-0:1.6.3.2 | Rack 2 old Dell switch |
|
||||
|
||||
All ports listen on the Tailscale IP (`100.70.77.93`).
|
||||
TCP ports are used by the ser2net emergency config only. conman accesses
|
||||
serial devices directly via `/dev/consoles/<name>` symlinks.
|
||||
|
||||
## Scripts
|
||||
|
||||
@@ -74,19 +76,24 @@ conman -d pfv-tsys4:7890 -q # list all consoles
|
||||
|
||||
Escape sequence: `&.` to disconnect, `&?` for help.
|
||||
|
||||
**Direct telnet (emergency only — conflicts with conman):**
|
||||
### Emergency Access
|
||||
|
||||
If conmand is down or you need direct serial access:
|
||||
|
||||
```bash
|
||||
# Direct telnet to ser2net works ONLY when conmand is stopped, because
|
||||
# conmand maintains persistent connections to all 7 TCP ports. Use:
|
||||
# Option A — direct screen on pfv-tsys4:
|
||||
ssh pfv-tsys4 'systemctl stop conmand'
|
||||
ssh pfv-tsys4 'screen /dev/consoles/pfv-core-sw01'
|
||||
ssh pfv-tsys4 'systemctl start conmand'
|
||||
|
||||
# Option B — enable ser2net for TCP access:
|
||||
ssh pfv-tsys4 'systemctl stop conmand && systemctl start ser2net'
|
||||
telnet pfv-tsys4 2001 # pfv-core-sw01
|
||||
ssh pfv-tsys4 'systemctl start conmand' # restart when done
|
||||
ssh pfv-tsys4 'systemctl stop ser2net && systemctl start conmand'
|
||||
```
|
||||
|
||||
**Do NOT use telnet while conmand is running** — conmand will reconnect
|
||||
and kick your telnet session immediately ("Connection closed by foreign host").
|
||||
The correct workflow is conman client → conmand → ser2net → device.
|
||||
**Never run ser2net and conmand at the same time** — both compete for the
|
||||
same serial devices.
|
||||
|
||||
### Re-deploy after changing mapping.txt
|
||||
|
||||
@@ -108,7 +115,7 @@ to `mapping.txt`.
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/etc/udev/rules.d/99-console-ports.rules` | Stable symlinks by ID_PATH |
|
||||
| `/etc/ser2net.yaml` | ser2net config (TCP ports → serial symlinks) |
|
||||
| `/etc/ser2net.yaml` | ser2net config (EMERGENCY ONLY — disabled by default) |
|
||||
| `/etc/conman.conf` | conman config (CONSOLE entries between markers) |
|
||||
| `/etc/systemd/system/conmand.service` | systemd unit for conmand |
|
||||
| `/root/console/mapping.txt` | Copy of the source-of-truth mapping |
|
||||
|
||||
+13
-10
@@ -143,14 +143,15 @@ fi
|
||||
{
|
||||
echo "%YAML 1.1"
|
||||
echo "---"
|
||||
echo "# ser2net configuration for pfv-tsys4 console ports"
|
||||
echo "# ser2net configuration for pfv-tsys4 console ports (EMERGENCY USE ONLY)"
|
||||
echo "# Generated by console/generate-config.sh on $(date)"
|
||||
echo "#"
|
||||
echo "# All ports bound to Tailscale IP ($TS_IP) for secure remote access."
|
||||
echo "# Physical devices are accessed via stable udev symlinks in /dev/consoles/."
|
||||
echo "# WARNING: ser2net is DISABLED by default. conman owns the serial"
|
||||
echo "# devices directly. If you enable ser2net, STOP conmand first or the"
|
||||
echo "# two will conflict (only one process can open a serial device at a time)."
|
||||
echo "#"
|
||||
echo "# To connect directly: telnet $TS_IP 2001"
|
||||
echo "# To connect via conman: conman -f <name>"
|
||||
echo "# Emergency workflow: systemctl stop conmand && systemctl start ser2net"
|
||||
echo "# When done: systemctl stop ser2net && systemctl start conmand"
|
||||
echo ""
|
||||
echo "define: &banner \\r\\nPFV console port \\p device \\d [\\B]\\r\\n\\r\\n"
|
||||
echo ""
|
||||
@@ -215,14 +216,16 @@ fi
|
||||
echo ""
|
||||
echo "$MARKER_BEGIN"
|
||||
echo "# Generated by console/generate-config.sh on $(date)"
|
||||
echo "# Each console connects to a ser2net TCP port via telnet protocol."
|
||||
echo "# ser2net owns the physical serial device; conman provides logging"
|
||||
echo "# and multiplexing on top."
|
||||
echo "# Each console connects directly to its serial device via the stable"
|
||||
echo "# udev symlink in /dev/consoles/. conman owns the serial devices"
|
||||
echo "# directly — no ser2net/telnet layer in the data path."
|
||||
echo "# Access: conman -f <name>"
|
||||
echo ""
|
||||
for entry in "${ENTRIES[@]}"; do
|
||||
IFS='|' read -r tcp_port name id_path baud comment <<< "$entry"
|
||||
echo "CONSOLE name=\"${name}\" dev=\"${TS_IP}:${tcp_port}\" log=\"${name}.log\" logopts=\"timestamp\""
|
||||
# Convert baud format: mapping.txt uses 9600n81, conman needs 9600,8n1
|
||||
conman_seropts=$(echo "$baud" | sed -E 's/^([0-9]+)(.)(.)(.)$/\1,\3\2\4/')
|
||||
echo "CONSOLE name=\"${name}\" dev=\"/dev/consoles/${name}\" seropts=\"${conman_seropts}\" log=\"${name}.log\" logopts=\"timestamp\""
|
||||
done
|
||||
echo "$MARKER_END"
|
||||
} >> "$CONMAN_CONF"
|
||||
@@ -244,7 +247,7 @@ echo " $CONMAN_CONF (CONSOLE entries appended between markers)"
|
||||
echo ""
|
||||
echo " Next steps:"
|
||||
echo " 1. Reload udev: udevadm control --reload-rules && udevadm trigger"
|
||||
echo " 2. Restart ser2net: systemctl restart ser2net"
|
||||
echo " 2. Stop ser2net: systemctl stop ser2net && systemctl disable ser2net"
|
||||
echo " 3. Start conman: systemctl enable --now conmand"
|
||||
echo " 4. Or run: bash $(basename "$0" .sh | sed 's/generate-config/setup/') .sh"
|
||||
echo "============================================"
|
||||
|
||||
+31
-32
@@ -133,35 +133,24 @@ while IFS= read -r line; do
|
||||
fi
|
||||
done < "$MAPPING_FILE"
|
||||
|
||||
# --- 5. Restart ser2net ---
|
||||
# --- 5. Stop ser2net (conman owns serial devices directly) ---
|
||||
echo ""
|
||||
echo "--- [5/7] Restarting ser2net ---"
|
||||
systemctl enable ser2net
|
||||
systemctl restart ser2net
|
||||
sleep 2
|
||||
|
||||
if systemctl is-active --quiet ser2net; then
|
||||
echo " ser2net is running."
|
||||
TS_IP=$(tailscale ip -4 2>/dev/null || echo "127.0.0.1")
|
||||
echo " Listening ports:"
|
||||
ss -tlnp | grep ser2net | grep -oE "${TS_IP}:[0-9]+" | sort -t: -k2 -n | sed 's/^/ /'
|
||||
else
|
||||
echo " WARNING: ser2net failed to start. Checking journal..."
|
||||
journalctl -u ser2net --no-pager -n 20
|
||||
fi
|
||||
echo "--- [5/7] Disabling ser2net ---"
|
||||
echo " conman now owns serial devices directly. ser2net must NOT run"
|
||||
echo " simultaneously (only one process can open a serial device at a time)."
|
||||
systemctl stop ser2net 2>/dev/null || true
|
||||
systemctl disable ser2net 2>/dev/null || true
|
||||
echo " ser2net stopped and disabled (available for emergency use)."
|
||||
|
||||
# --- 6. Enable + start conmand ---
|
||||
echo ""
|
||||
echo "--- [6/7] Starting conmand ---"
|
||||
|
||||
# conman package on Debian may not ship a systemd unit. Create one if missing.
|
||||
if ! systemctl cat conmand >/dev/null 2>&1; then
|
||||
echo " No systemd unit for conmand — creating one..."
|
||||
cat > /etc/systemd/system/conmand.service <<'CONMAND_UNIT'
|
||||
[Unit]
|
||||
# conman package on Debian may not ship a systemd unit. Create one if missing,
|
||||
# or update an existing one that still references ser2net.
|
||||
CONMAND_UNIT_BODY='[Unit]
|
||||
Description=ConMan (Console Manager)
|
||||
After=network.target ser2net.service
|
||||
Requires=ser2net.service
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=forking
|
||||
@@ -170,10 +159,18 @@ Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
CONMAND_UNIT
|
||||
WantedBy=multi-user.target'
|
||||
|
||||
if ! systemctl cat conmand >/dev/null 2>&1; then
|
||||
echo " No systemd unit for conmand — creating one..."
|
||||
echo "$CONMAND_UNIT_BODY" > /etc/systemd/system/conmand.service
|
||||
systemctl daemon-reload
|
||||
echo " Created /etc/systemd/system/conmand.service"
|
||||
elif systemctl cat conmand 2>/dev/null | grep -q 'ser2net'; then
|
||||
echo " conmand unit has ser2net dependency — updating..."
|
||||
echo "$CONMAND_UNIT_BODY" > /etc/systemd/system/conmand.service
|
||||
systemctl daemon-reload
|
||||
echo " Updated /etc/systemd/system/conmand.service (removed ser2net dep)"
|
||||
fi
|
||||
|
||||
# Kill any manually-started conmand first
|
||||
@@ -200,17 +197,19 @@ fi
|
||||
echo ""
|
||||
echo "--- [7/7] Setup complete ---"
|
||||
echo ""
|
||||
echo " ser2net TCP ports (connect directly):"
|
||||
echo " telnet <tailscale-ip> 2001 # pfv-core-sw01"
|
||||
echo " telnet <tailscale-ip> 2002 # pfv-tor3-mgmt"
|
||||
echo " ..."
|
||||
echo " conman owns serial devices directly (no ser2net in the data path)."
|
||||
echo ""
|
||||
echo " conman consoles (with logging):"
|
||||
echo " conman -f pfv-core-sw01"
|
||||
echo " conman -q # query status"
|
||||
echo " Connect from any Tailscale workstation:"
|
||||
echo " conman -d pfv-tsys4:7890 -f pfv-core-sw01"
|
||||
echo " conman -d pfv-tsys4:7890 -q # list consoles"
|
||||
echo ""
|
||||
echo " Emergency direct serial (must stop conmand first):"
|
||||
echo " systemctl stop conmand"
|
||||
echo " screen /dev/consoles/pfv-core-sw01"
|
||||
echo " systemctl start conmand"
|
||||
echo ""
|
||||
echo " To regenerate after changing mapping.txt:"
|
||||
echo " bash generate-config.sh"
|
||||
echo " udevadm trigger"
|
||||
echo " systemctl restart ser2net conmand"
|
||||
echo " systemctl restart conmand"
|
||||
echo "============================================"
|
||||
|
||||
+20
-17
@@ -61,29 +61,32 @@ echo "--- 5. conmand connection status (journal) ---"
|
||||
journalctl -u conmand --no-pager -n 50 2>/dev/null | grep -iE "connect|error|fail|console|refused|timeout" | tail -15 || echo " (no relevant journal entries)"
|
||||
|
||||
echo ""
|
||||
echo "--- 6. Verify ser2net is actually proxying data (check for byte flow) ---"
|
||||
# Pick a known-active port (2007 = pfv-r2-sw, the old Dell with menu UI that responded)
|
||||
echo " Probing TCP $TS_IP:2007 for data..."
|
||||
RESPONSE=$(timeout 3 bash -c "printf '\r\r' | nc -w 2 $TS_IP 2007 2>/dev/null" | tr -cd '[:print:][:space:]' | head -5)
|
||||
if [ -n "$RESPONSE" ]; then
|
||||
echo " [OK] Data flowing through ser2net TCP 2007:"
|
||||
echo "$RESPONSE" | sed 's/^/ /'
|
||||
else
|
||||
echo " (no immediate response — device may need more interaction)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- 7. Check if conmand has open connections to ser2net ports ---"
|
||||
echo "--- 6. Verify serial devices are open by conmand ---"
|
||||
echo " Checking /dev/consoles/* are held by conmand..."
|
||||
CONMAND_PID=$(pgrep -x conmand 2>/dev/null || echo "")
|
||||
if [ -n "$CONMAND_PID" ]; then
|
||||
echo " conmand PID: $CONMAND_PID"
|
||||
echo " Open connections to ser2net (expect 7 to 100.x:200X):"
|
||||
ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -oE "100\.[0-9.]+:200[0-9]" | sort | sed 's/^/ /'
|
||||
COUNT=$(ss -tnp 2>/dev/null | grep "pid=$CONMAND_PID" | grep -c ":200")
|
||||
echo " Total conmand→ser2net connections: $COUNT (expect 7)"
|
||||
for devlink in /dev/consoles/*; do
|
||||
[ -e "$devlink" ] || continue
|
||||
name=$(basename "$devlink")
|
||||
if lsof "$devlink" 2>/dev/null | grep -q conmand; then
|
||||
echo " [OK] $name ($devlink) opened by conmand"
|
||||
else
|
||||
echo " [WARN] $name ($devlink) not held by conmand"
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo " [FAIL] conmand not running"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- 7. Verify ser2net is stopped (conman owns devices) ---"
|
||||
if systemctl is-active --quiet ser2net 2>/dev/null; then
|
||||
echo " [WARN] ser2net is running — will conflict with conmand for serial devices!"
|
||||
echo " Fix: systemctl stop ser2net"
|
||||
else
|
||||
echo " [OK] ser2net is stopped (conman owns serial devices directly)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
|
||||
|----------|-------------|---------------|
|
||||
| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
|
||||
| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
|
||||
| [`../console/README.md`](../console/README.md) | Serial console management (ser2net + conman) for 7 network switches on pfv-tsys4 | 2026-07-28 |
|
||||
| [`../console/README.md`](../console/README.md) | Serial console management (conman direct-serial) for 7 network switches on pfv-tsys4 | 2026-07-29 |
|
||||
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
|
||||
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
|
||||
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
|
||||
|
||||
Reference in New Issue
Block a user