Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).
Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:
- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
qga, disk controller, k8s wnode sizing), stopped VM reference
Updated docmap.md with all four new entries.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deploy Network UPS Tools to monitor the server-room UPS units and feed data to
Home Assistant for power-usage tracking.
APC Smart-UPS C 1500 is fully operational: battery charge, runtime, load,
voltage, and status all reporting via usbhid-ups + APC HID 0.100 subdriver.
upsd listens on Tailscale (100.121.189.98:3493) for HA polling; local upsmon
provides graceful hypervisor shutdown on battery-low.
Key deployment fix: the udev rule must cover SUBSYSTEM=="usb" (raw
/dev/bus/usb) in addition to hidraw, because usbhid-ups opens the raw USB
device file after dropping to the nut user via setuid(). Matching only hidraw
causes EACCES on driver start.
Tripp Lite UPS (09ae:3016) is blocked — driver matches TrippLite HID 0.85
subdriver but fails reading the 878-byte HID Report Descriptor (EAGAIN via USB
hub). Needs physical reseat to direct motherboard USB port. Driver masked to
prevent restart-loop spam.
Files: ups/discover.sh, ups/setup.sh (idempotent), ups/status.sh, ups/README.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The original ser2net config used raw TCP accepters, but conman's
dev="host:port" speaks telnet protocol. This protocol mismatch caused
conman's telnet NVT state machine to strip bare CR characters from
device output — particularly from Dell switches that send \n\r (LF+CR)
line endings — producing stair-stepped and garbled terminal display.
Fix: change ser2net accepter from `tcp,IP,PORT` to
`telnet(rfc2217),tcp,IP,PORT`. With both sides speaking telnet, binary
mode is negotiated and CR/LF translation is handled correctly by the
telnet NVT layer. RFC2217 also enables baud rate negotiation for
future use.
Validated at the byte level: PTY capture confirms all prompts render
cleanly at column 0 with proper CRLF handling across 5+ Enter presses.
All 7 consoles connected, 7 log files actively capturing.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The ser2net layer between conman and the serial devices was causing
stair-stepping in terminal output. conman's dev="host:port" uses telnet
protocol, but ser2net's accepter was raw TCP (no telnet mode). The telnet
NVT state machine in conman was stripping bare CR characters from device
output — particularly from Dell switches that use old-style \n\r (LF+CR)
line endings instead of standard \r\n. In telnet, a bare \r not followed
by \n or NUL is non-compliant and gets dropped, leaving bare \n that
causes stair-stepping in the raw-mode terminal.
Fix: conman now opens serial devices directly via the stable udev
symlinks (/dev/consoles/<name>) with seropts, eliminating the telnet
layer entirely. ser2net is stopped and disabled but remains installed
for emergency TCP access (documented workflow: stop conmand, start
ser2net, use telnet, then reverse).
Architecture change:
Before: device → serial → ser2net (raw TCP) → conman (telnet NVT) → terminal
After: device → serial → conman (direct) → terminal
Verified: 7/7 serial devices held by conmand, 7/7 log files capturing,
all \r bytes preserved in terminal output (confirmed via PTY capture).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
ISC DHCP server with failover-peer (primary/secondary) deployed on
pfv-netinfra-01 (192.168.3.252) and pfv-netinfra-02 (192.168.3.253). Both
servers in "normal" state, load-balancing 333 active leases.
Migration details:
- Copied all 37 host reservations + subnet/pool config from pfv-netboot
- DHCP lease database copied from netboot for zero-disruption cutover
- DNS servers changed from 192.168.3.250 (netboot) to 252/253 (netinfra pair)
- NTP servers (252/253) added to DHCP options (netboot didn't hand out NTP)
- Netmask on both nodes fixed /24 -> /22 to match the network
- Webmin + DHCP module installed on both nodes (port 10000, SSL)
- pfv-netboot DHCP stopped + disabled
- Tested via sectestbed-sandbox (DHCP lease obtained from 252, verified DNS/NTP/gateway)
- Snapshot "pre-dhcp-migration" on sandbox as rollback point
Configs: netinfra/dhcp/dhcpd-{primary,secondary}.conf
Plan + results: netinfra/dhcp-migration.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Delete AUDIT-2026-07-28.md (old audit). Rewrite AUDIT-2026-07-29.md as the
sole authoritative audit after reading live /etc/pve/storage.cfg, /etc/exports,
and /etc/fstab on every host.
Key corrections:
- local-nonprod on tsys5 IS the S1 physical disk (dir storage, not separate).
S1 carries 16 sectestbed VMs, not 0 — it is NOT a migration target.
- storage.cfg is per-host, not cluster-wide — each node defines its own entries.
- BUG: tsys6 S1 export path typo (S21 instead of S1) — S1 broken on tsys6.
- GAP: tsys9 has no S1 defined in storage.cfg.
- Revised disk migration plan avoids S1, targets S2/S3/S4/D5 for IOPS spreading.
- D2 goes 17->5 VMs; S2/S3 absorb the load (7 each); S1 untouched (16 idle VMs).
- Refreshed ground truth: cnode5 deleted, cnode4 stopped (user actions this session).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add §13-§15 to AUDIT-2026-07-29: current per-VM host+disk inventory, target
end-state inventory (RAM/CPU relief + HDD IOPS rebalancing), and a phased action
checklist. Headline disk finding: D2 carries 17 of 26 NFS VMs while S1 sits
empty — the plan rebalances to 4-5 VMs per spindle (D2 17->5, S1 0->5) using HDD
exports only. Host moves relieve tsys1 (115%->77% alloc) and tsys9. All disks
stay on HDD (SSD/NVMe tiering deferred to post-Friday).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.
Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
migrations were the #1 recommendation from the previous audit and
have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)
Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.
Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.
Architecture (layered, no port sharing):
USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)
Port assignments (all on Tailscale IP 100.70.77.93):
2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor
2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw
2007 = pfv-r2-sw
Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
entries from mapping.txt. Idempotent (markers in conman.conf for clean
regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
+ manual fallback for already-discovered devices), create conmand
systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture
Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
/etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
/var/log/conman/. Fixed by adding server logdir directive.
Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Set up centralized PDU management for a Cyclades AlterPath PM10i (10
controllable AC outlets) connected to pfv-tsys1 via a Prolific USB-to-DB9
serial adapter. powermand is now listening on 0.0.0.0:10101, making the
PDU manageable over the network from any host on the tailnet.
Scripts (powerman/):
- discover.sh: gather USB adapter, powerman state, device definitions
- setup.sh: idempotent setup — udev rule (stable symlink by serial number),
powerman.conf with 10 outlet nodes, fix powermand dialout group, restart
service. Overridable via env vars for other hosts/PDU types
- test-pdu.sh: validate control by cycling outlet 10 off then on (8/8 pass)
- status.sh: quick PDU status check
Issues fixed during setup:
- Config pointed at /dev/ttyUSB0 but adapter is at /dev/ttyUSB1 (fixed
with udev symlink /dev/cyclades-pm10 pinned to adapter serial)
- powermand (user:powerman) lacked dialout group membership to open the
serial device (fixed with usermod + udev GROUP="dialout")
Validation: outlet 10 turned off (confirmed), turned on (confirmed), then
cycled. All 10 outlets currently ON and manageable.
TODO tracked for Friday: rename outlets from generic (outlet-1..10) to
match physical devices, and change PDU admin password from factory default.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.
Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check
All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.
Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Author the docs/k8s/ directory capturing the pfv-k8s control-plane design:
- README.md: TL;DR of all decisions (distro, runtime, cnode count, admin
access, identity, tenancy, registry, storage)
- DISTRO-DECISION.md: Talos vs k3s analysis. Recommend Talos because the
ITAR/classified requirement makes its immutable, API-only, measured-boot
posture structurally easier to certify than SCAP-hardened Debian. k3s was
only ever a plan (no cluster deployed), so cutover cost is ~zero.
- ARCHITECTURE.md: target arch with mermaid diagrams covering control
plane, LAN-only network with Tailscale subnet-router admin, Cilium CNI,
OIDC to Keycloak, per-tenant vcluster isolation (incl. ITAR tenant),
Harbor pull-through cache on D3 SSD, bootstrap sequence, and DR.
Gardening: register docs/k8s/ in docmap.md, update STATUS.md with the new
k8s section and the three pending user decisions (cnode count, host spread,
Tailscale pattern).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)
Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.
Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.
Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles