Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).
Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deploy Network UPS Tools to monitor the server-room UPS units and feed data to
Home Assistant for power-usage tracking.
APC Smart-UPS C 1500 is fully operational: battery charge, runtime, load,
voltage, and status all reporting via usbhid-ups + APC HID 0.100 subdriver.
upsd listens on Tailscale (100.121.189.98:3493) for HA polling; local upsmon
provides graceful hypervisor shutdown on battery-low.
Key deployment fix: the udev rule must cover SUBSYSTEM=="usb" (raw
/dev/bus/usb) in addition to hidraw, because usbhid-ups opens the raw USB
device file after dropping to the nut user via setuid(). Matching only hidraw
causes EACCES on driver start.
Tripp Lite UPS (09ae:3016) is blocked — driver matches TrippLite HID 0.85
subdriver but fails reading the 878-byte HID Report Descriptor (EAGAIN via USB
hub). Needs physical reseat to direct motherboard USB port. Driver masked to
prevent restart-loop spam.
Files: ups/discover.sh, ups/setup.sh (idempotent), ups/status.sh, ups/README.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Solve the long-standing USB adapter enumeration shift problem: 9 Prolific
USB-to-DB9 adapters on pfv-tsys4 have no unique serial numbers and get
assigned /dev/ttyUSB0-8 based on enumeration order, which changes on every
reboot and breaks the old /root/conmap + manual screen workflow.
Solution: udev rules pin each adapter by its ID_PATH (physical USB port
topology), which is stable across reboots regardless of enumeration order.
Each adapter gets a named symlink in /dev/consoles/<name>. ser2net opens
these stable symlinks and exposes them on TCP ports (2001-2007) bound to
the Tailscale interface only. conman connects to those TCP ports for
session logging and multi-user console sharing.
Architecture (layered, no port sharing):
USB adapter → udev symlink → ser2net (TCP) → conman (logging + mux)
Port assignments (all on Tailscale IP 100.70.77.93):
2001 = pfv-core-sw01 2002 = pfv-tor3-mgmt 2003 = pfv-tor3-stor
2004 = pfv-rrinfra-rtr 2005 = pfv-r2-tor-top 2006 = subodev-torsw
2007 = pfv-r2-sw
Scripts (console/):
- mapping.txt: source of truth (TCP port | name | ID_PATH | baud | comment)
- generate-config.sh: generates udev rules, ser2net.yaml, conman.conf
entries from mapping.txt. Idempotent (markers in conman.conf for clean
regeneration). Uses | delimiter (ID_PATH values contain colons).
- setup.sh: full deploy — generate configs, create symlinks (udev trigger
+ manual fallback for already-discovered devices), create conmand
systemd unit (Debian doesn't ship one), restart services
- discover.sh: read-only USB adapter and service state discovery
- validate-conman.sh: verify conman→ser2net→device data path and log capture
Issues fixed during development:
- /dev/console is a kernel char device (major 5, minor 1) — cannot create
a directory there. Changed symlink namespace to /dev/consoles/.
- conman 0.3.x has no 'include' directive — CONSOLE entries written
directly into /etc/conman.conf between idempotent markers.
- Debian conman package has no systemd unit — created
/etc/systemd/system/conmand.service with After=ser2net ordering.
- conman.conf had no LOGDIR — logs weren't being written to
/var/log/conman/. Fixed by adding server logdir directive.
Validation: 7 symlinks resolving, 7 TCP ports on Tailscale, conmand with
7 consoles registered, 7 log files actively capturing console output,
both services enabled for reboot survival.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Two variances from the gardening protocol were found during self-audit:
the k8s/ and powerman/ directories were added but the root README.md
directory table and the AGENTS.md Key Scripts table were not updated.
Root cause: the gardening protocol enumerated STATUS.md and docmap.md but
did not explicitly call out the root README.md directory table or the
AGENTS.md Key Scripts table, so they were easy to miss when adding a new
top-level directory.
Fixes:
- README.md: add k8s/ and powerman/ to the Directory Structure table
- AGENTS.md: add install-cp.sh and powerman/setup.sh to Key Scripts
Permanent fix (encode in protocol so it cannot recur):
- Add rule 5 to the Automatic Gardening Protocol: when a new top-level
directory is created, ALL directory listings must be updated
(README.md table, AGENTS.md layout block, AGENTS.md Key Scripts)
- Add rule 6: a grep-based self-audit command to run before commit,
verifying the new directory appears in all four canonical files
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Set up centralized PDU management for a Cyclades AlterPath PM10i (10
controllable AC outlets) connected to pfv-tsys1 via a Prolific USB-to-DB9
serial adapter. powermand is now listening on 0.0.0.0:10101, making the
PDU manageable over the network from any host on the tailnet.
Scripts (powerman/):
- discover.sh: gather USB adapter, powerman state, device definitions
- setup.sh: idempotent setup — udev rule (stable symlink by serial number),
powerman.conf with 10 outlet nodes, fix powermand dialout group, restart
service. Overridable via env vars for other hosts/PDU types
- test-pdu.sh: validate control by cycling outlet 10 off then on (8/8 pass)
- status.sh: quick PDU status check
Issues fixed during setup:
- Config pointed at /dev/ttyUSB0 but adapter is at /dev/ttyUSB1 (fixed
with udev symlink /dev/cyclades-pm10 pinned to adapter serial)
- powermand (user:powerman) lacked dialout group membership to open the
serial device (fixed with usermod + udev GROUP="dialout")
Validation: outlet 10 turned off (confirmed), turned on (confirmed), then
cycled. All 10 outlets currently ON and manageable.
TODO tracked for Friday: rename outlets from generic (outlet-1..10) to
match physical devices, and change PDU admin password from factory default.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.
Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check
All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.
Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Strengthen Git Policy point 1 so agents never hold work for review. The
user reviews rendered markdown on Gitea after push, so pausing to "let
them read first" defeats the workflow. Explicitly overrides any default
conservative commit-and-hold behavior.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)
Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.
Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.
Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.
The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).
Directory reorganization and doc gardening will follow in subsequent
commits.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.
Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds the PowerShell and cmd.exe commands the user needs to run on the
Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe
details before rebuilding it as Proxmox. Also documents the Friday
maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open
questions for the next session.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode
file. Add version control instructions to AGENTS.md for future sessions.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Make AGENTS.md actionable for future agents by recording what was learned
while fixing the provisioning scripts:
- Add a Repository Layout section: the KNELShellFramework is vendored
under vendor/.../KNELShellFramework (not at repo root), scripts must
self-locate via BASH_SOURCE, configs are read locally (no CDN), and
some .sh agents are actually PHP
- Replace the vague "commit immediately" note with an explicit
Autonomous Git Workflow section authorizing agents to commit AND push
without being asked, grouped into coherent atomic commits
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add agent guidelines for AI assistants working on this repository:
- Document atomic commit requirements
- Specify conventional commit format with examples
- Require verbose, formatted commit messages
- Emphasize immediate commit/push behavior
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles
Charles N Wyble