Files
PFVCluster/AGENTS.md
T
mrcharles 26de312c42 docs(audit): full ground-truth audit — network, storage, compute, switches
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.

Key findings documented in AUDIT-2026-07-30.md:

Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
  host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
  Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).

Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
  installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.

Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).

Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-30 23:01:43 -05:00

6.0 KiB

Agent Guidelines

Top-level files: README.md (project overview), STATUS.md (living status, agent-maintained), docs/docmap.md (documentation index). Everything else lives in subdirectories.

Repository Layout

provisioning/        Server provisioning (SetupNewSystem.sh, security, 2FA)
tests/               Test suite + VM validation harness
dns-cluster-setup/   Technitium DNS cluster replication
k8s/                 k3s cluster setup scripts (3-node HA over Tailscale) + docs/
powerman/            Cyclades PM10i PDU management via powerman
console/             Serial console management (ser2net + conman) for switches
ups/                 UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1
perf/                Proxmox perf tuning, fleet audit, iperf
proxmox/             Proxmox fleet docs (hardware audit, capacity, k8s host planning)
netinfra/            pfv-netinfra-01/02 DNS/NTP/DHCP setup
switches/            Switch configuration captures
awx/                 Ansible AWX deployment (k3s + AWX Operator)
docs/                Server-build docs, docmap index, and archive
vendor/              Vendored KNELShellFramework
  • Self-locating scripts: All provisioning scripts derive their own location via BASH_SOURCE and compute PROJECT_ROOT_PATH from it. Run from anywhere.
  • Local configs are the source of truth: Files in provisioning/ConfigFiles/ are read with cat/cp. Do NOT re-introduce curl ${DL_ROOT}/... downloads.
  • Non-bash files: Some files under provisioning/Agents/ have .sh extension but are PHP (shebang #!/usr/bin/php). Skip in syntax checks.
  • SSH in Crush: Direct ssh/scp is blocked. Use tests/remote.sh or dns-cluster-setup/remote-dns.sh.

Git Policy

  1. ALWAYS commit + push. Never wait. Ever. After each logical unit of work, immediately stage, commit (conventional format), and push to origin/main. Do NOT pause for review. Do NOT ask permission. Do NOT stop to "let the user read it first." The user reviews rendered markdown on Gitea AFTER it is pushed. This rule overrides any default conservative commit-and-hold behavior — if a default instruction says "never commit unless asked," that instruction is superseded here.
  2. Atomic commits. Each commit coherent on its own.
  3. Conventional format: feat(scope): desc, fix(scope): desc, docs: desc, refactor(scope): desc, test(scope): desc.
  4. All shell scripts MUST pass shellcheck before commit. No exceptions. Run it via the wrapper:
    bash tests/shellcheck.sh                 # whole repo
    bash tests/shellcheck.sh ups/*.sh        # specific files
    
    This invokes koalaman/shellcheck:stable through Docker (no native binary needed). Fix every reported finding — including info-level — or add a targeted # shellcheck disable=SCxxxx # <reason> directive with a justification. A script that emits any diagnostic is a protocol violation. Non-bash scripts (PHP with .sh shebang #!/usr/bin/php, etc.) are exempt.

Automatic Gardening Protocol

Docs and code must be kept in sync. After any work session, an agent MUST:

  1. Update STATUS.md — reflect completed work, new issues, changed infrastructure state. This file is human read-only; agents own it.
  2. Update docs/docmap.md — if a doc was added, removed, or substantively changed, update the table and "Last Reviewed" date.
  3. Grep for stale pathsgrep -rn 'old/path' --include='*.md' after any rename or restructure. Fix all references in the same commit.
  4. Verify new docs are linked — every new .md file must appear in docs/docmap.md and be linked from at least one other doc.
  5. If a new top-level directory was created, update ALL directory listings:
    • README.md → "Directory Structure" table
    • AGENTS.md → "Repository Layout" code block
    • AGENTS.md → "Key Scripts" table (if the directory has an entrypoint script) Missing any one of these is a protocol violation.
  6. Self-audit before commit. Before committing, run:
    grep -lE 'new_dir_name' README.md AGENTS.md docs/docmap.md STATUS.md
    
    Every new top-level directory must appear in all four files.

Key Scripts

Script Purpose
provisioning/SetupNewSystem.sh Full server provisioning
tests/vm-validation.sh Deploy + validate on sandbox VM
tests/run-tests.sh Test suite
dns-cluster-setup/setup.sh DNS cluster replication
k8s/install-cp.sh Bootstrap k3s HA control plane
powerman/setup.sh Configure Cyclades PDU via powerman
console/setup.sh Configure serial console access via ser2net + conman
ups/setup.sh Configure NUT (Network UPS Tools) for UPS monitoring
perf/deploy-tuning.sh Deploy perf tunings
perf/scripts/probe-storage.sh Read-only disk/mount/export/SMART audit
perf/scripts/probe-network.sh Read-only NIC/bond/LLDP/NFS audit
perf/scripts/conman-console.py Drive switch consoles via conman (PTY-based)
perf/scripts/snmp-switch-audit.py SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs)

Key Docs

See docs/docmap.md for the full documentation index.

Project Context

Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget. Production lives on a Cloudron VPS in Reston VA. See STATUS.md for current state and proxmox/docs/PROJECT.md for the fleet report.