Files
PFVCluster/AGENTS.md
T
mrcharles 815d07bbca feat(console): manage 7 switch consoles via ser2net+conman on pfv-tsys4
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.

Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.

Architecture (layered, no port sharing):
  USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)

Port assignments (all on Tailscale IP 100.70.77.93):
  2001 = pfv-core-sw01     2002 = pfv-tor3-mgmt    2003 = pfv-tor3-stor
  2004 = pfv-rrinfra-rtr   2005 = pfv-r2-tor-top   2006 = subodev-torsw
  2007 = pfv-r2-sw

Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
  entries from mapping.txt. Idempotent (markers in conman.conf for clean
  regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
  + manual fallback for already-discovered devices), create conmand
  systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture

Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
  a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
  directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
  /etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
  /var/log/conman/. Fixed by adding server logdir directive.

Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-28 19:44:02 -05:00

4.6 KiB

Agent Guidelines

Top-level files: README.md (project overview), STATUS.md (living status, agent-maintained), docs/docmap.md (documentation index). Everything else lives in subdirectories.

Repository Layout

provisioning/        Server provisioning (SetupNewSystem.sh, security, 2FA)
tests/               Test suite + VM validation harness
dns-cluster-setup/   Technitium DNS cluster replication
k8s/                 k3s cluster setup scripts (3-node HA over Tailscale)
powerman/            Cyclades PM10i PDU management via powerman
console/             Serial console management (ser2net + conman) for switches
perf/                Proxmox perf tuning, fleet audit, iperf
netinfra/            pfv-netinfra-01/02 DNS/NTP setup
switches/            Switch configuration captures
docs/                All documentation (see docs/docmap.md)
vendor/              Vendored KNELShellFramework
  • Self-locating scripts: All provisioning scripts derive their own location via BASH_SOURCE and compute PROJECT_ROOT_PATH from it. Run from anywhere.
  • Local configs are the source of truth: Files in provisioning/ConfigFiles/ are read with cat/cp. Do NOT re-introduce curl ${DL_ROOT}/... downloads.
  • Non-bash files: Some files under provisioning/Agents/ have .sh extension but are PHP (shebang #!/usr/bin/php). Skip in syntax checks.
  • SSH in Crush: Direct ssh/scp is blocked. Use tests/remote.sh or dns-cluster-setup/remote-dns.sh.

Git Policy

  1. ALWAYS commit + push. Never wait. Ever. After each logical unit of work, immediately stage, commit (conventional format), and push to origin/main. Do NOT pause for review. Do NOT ask permission. Do NOT stop to "let the user read it first." The user reviews rendered markdown on Gitea AFTER it is pushed. This rule overrides any default conservative commit-and-hold behavior — if a default instruction says "never commit unless asked," that instruction is superseded here.
  2. Atomic commits. Each commit coherent on its own.
  3. Conventional format: feat(scope): desc, fix(scope): desc, docs: desc, refactor(scope): desc, test(scope): desc.

Automatic Gardening Protocol

Docs and code must be kept in sync. After any work session, an agent MUST:

  1. Update STATUS.md — reflect completed work, new issues, changed infrastructure state. This file is human read-only; agents own it.
  2. Update docs/docmap.md — if a doc was added, removed, or substantively changed, update the table and "Last Reviewed" date.
  3. Grep for stale pathsgrep -rn 'old/path' --include='*.md' after any rename or restructure. Fix all references in the same commit.
  4. Verify new docs are linked — every new .md file must appear in docs/docmap.md and be linked from at least one other doc.
  5. If a new top-level directory was created, update ALL directory listings:
    • README.md → "Directory Structure" table
    • AGENTS.md → "Repository Layout" code block
    • AGENTS.md → "Key Scripts" table (if the directory has an entrypoint script) Missing any one of these is a protocol violation.
  6. Self-audit before commit. Before committing, run:
    grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md
    
    Every new top-level directory must appear in all four files.

Key Scripts

Script Purpose
provisioning/SetupNewSystem.sh Full server provisioning
tests/vm-validation.sh Deploy + validate on sandbox VM
tests/run-tests.sh Test suite
dns-cluster-setup/setup.sh DNS cluster replication
k8s/install-cp.sh Bootstrap k3s HA control plane
powerman/setup.sh Configure Cyclades PDU via powerman
console/setup.sh Configure serial console access via ser2net + conman
perf/deploy-tuning.sh Deploy perf tunings

Key Docs

See docs/docmap.md for the full documentation index.

Project Context

Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget. Production lives on a Cloudron VPS in Reston VA. See STATUS.md for current state and docs/proxmox/PROJECT.md for the fleet report.