mrcharles
eb99b00a70
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.
Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.
Architecture (layered, no port sharing):
USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)
Port assignments (all on Tailscale IP 100.70.77.93):
2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor
2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw
2007 = pfv-r2-sw
Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
entries from mapping.txt. Idempotent (markers in conman.conf for clean
regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
+ manual fallback for already-discovered devices), create conmand
systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture
Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
/etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
/var/log/conman/. Fixed by adding server logdir directive.
Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
117 lines
5.9 KiB
Markdown
117 lines
5.9 KiB
Markdown
# Project Status
|
|
|
|
> **Human read-only. Agents maintain this file automatically after each work
|
|
> session.** Do not edit by hand — the next agent run will overwrite it.
|
|
|
|
> **Last updated:** 2026-07-28 by Crush (GLM-5.2)
|
|
|
|
## Current State: STABLE
|
|
|
|
The merged PFVCluster repo is fully operational across provisioning, DNS
|
|
infrastructure, Proxmox cluster ops, and k8s control plane.
|
|
|
|
## Completed Work
|
|
|
|
### Server Provisioning (validated on sectestbed-sandbox)
|
|
- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
|
|
- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
|
|
- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
|
|
- [x] DNS resolv.conf: managed static file pointing at netinfra pair
|
|
- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
|
|
- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
|
|
|
|
### DNS Cluster (pfv-netinfra-01/02)
|
|
- [x] Production Technitium config replicated from tailscale-router (read-only)
|
|
- [x] 124 zones on both nodes (knel.net + reverse DNS)
|
|
- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
|
|
- [x] Zone replication via rsync systemd timer (every 60s)
|
|
- [x] Credentials + 2FA replicated identically to production
|
|
- [x] Both LAN IPs resolve knel.net device names + recurse externally
|
|
|
|
### Proxmox Cluster Ops
|
|
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
|
|
- [x] Fleet audit complete (PROJECT.md has ground truth)
|
|
- [ ] tsys4: blocked on PCIe NIC + RAM install
|
|
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
|
|
- [ ] tsys2: pending rebuild from Win10 to Proxmox
|
|
|
|
### Kubernetes Control Plane (k3s HA — LIVE)
|
|
- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
|
|
- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
|
|
- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
|
|
- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
|
|
Tailscale IPs, CoreDNS, API server, workload isolation
|
|
- [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
|
|
- [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
|
|
- [ ] Workers (wnodes) not yet joined to this cluster
|
|
- [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
|
|
preserved in `docs/k8s/` for future ITAR/classified cluster.
|
|
|
|
### PDU Management (powerman on pfv-tsys1 — LIVE)
|
|
- [x] Cyclades AlterPath PM10i (10 outlets) managed via powerman over serial
|
|
- [x] USB-DB9 adapter (Prolific pl2303) with stable udev symlink
|
|
`/dev/cyclades-pm10`
|
|
- [x] powermand listening on `0.0.0.0:10101` (network-accessible)
|
|
- [x] All 10 outlets defined as `outlet-1` through `outlet-10`
|
|
- [x] Validated: outlet 10 cycled off → on (8/8 test checks passed)
|
|
- [ ] Rename outlets to match physical devices (Friday onsite)
|
|
|
|
### Console Management (ser2net + conman on pfv-tsys4 — LIVE)
|
|
- [x] 7 network switch/router consoles managed via ser2net + conman
|
|
- [x] **USB enumeration problem SOLVED:** udev rules pin each adapter by
|
|
ID_PATH (physical USB port topology) to stable `/dev/consoles/<name>`
|
|
symlinks that survive reboot regardless of enumeration order
|
|
- [x] ser2net exposes all 7 consoles on TCP ports (2001-2007) bound to
|
|
**Tailscale IP only** (`100.70.77.93`)
|
|
- [x] conman connects to TCP ports for logging + multiplexing (7 log
|
|
files active in `/var/log/conman/`)
|
|
- [x] Both ser2net + conmand enabled via systemd (survive reboot)
|
|
- [x] conmand systemd unit created (Debian package doesn't ship one)
|
|
- [x] Old `/root/conmap` + manual `screen` workflow replaced
|
|
|
|
### Repo Merge
|
|
- [x] KNELServerBuild merged into PFVCluster (history preserved)
|
|
- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
|
|
- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
|
|
|
|
## Known Issues
|
|
|
|
| Issue | Impact | Status |
|
|
|-------|--------|--------|
|
|
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
|
|
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
|
|
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
|
|
|
|
## Pending (next session priorities)
|
|
|
|
1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical
|
|
devices (Friday onsite)
|
|
2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
|
2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
|
|
- Cnode count: 3 (recommended) vs 5
|
|
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
|
|
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
|
|
3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
|
4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
|
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
|
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
|
7. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
|
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
|
|
per-tenant vcluster policy templates, solar-aware scale-out
|
|
|
|
## Infrastructure Summary
|
|
|
|
| Component | Details |
|
|
|-----------|---------|
|
|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
|
|
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
|
|
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
|
|
| DNS production | tailscale-router (read-only source of truth) |
|
|
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
|
|
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
|
|
| Backup | Proxmox Backup Server (PBS) |
|
|
| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
|
|
| **PDU** | **Cyclades PM10i via powerman on pfv-tsys1 (port 10101)** |
|
|
| **Console** | **7 switch consoles via ser2net+conman on pfv-tsys4 (TCP 2001-2007 on Tailscale)** |
|
|
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
|