mrcharles
26de312c42
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
6.9 KiB
6.9 KiB
Documentation Map
Index of all documentation in this repo. Agents must update this file whenever a doc is added, removed, or substantively changed.
Last updated: 2026-07-30 (full ground-truth audit: network+storage+compute)
Kubernetes Architecture (k8s/)
Distro decision, target architecture, control-plane design, bootstrap and DR procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
| Document | Description | Last Reviewed |
|---|---|---|
k8s/docs/README.md |
Index + TL;DR of all k8s decisions | 2026-07-28 |
k8s/docs/DISTRO-DECISION.md |
Talos vs k3s analysis. Decision: k3s for regular R&D cluster; Talos preserved for future ITAR/classified | 2026-07-28 |
k8s/docs/ARCHITECTURE.md |
Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
Proxmox Cluster (proxmox/)
Fleet operations, hardware, performance tuning, storage architecture.
| Document | Description | Last Reviewed |
|---|---|---|
proxmox/docs/PROJECT.md |
Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
proxmox/docs/CAPACITY-GOALS.md |
Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
proxmox/docs/AUDIT-2026-07-29.md |
Prior audit (consolidated RAM/CPU + HDD disk placement). Superseded by AUDIT-2026-07-30 | 2026-07-29 |
proxmox/docs/AUDIT-2026-07-30.md |
Current & authoritative. Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 |
proxmox/docs/TODO.md |
Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
proxmox/docs/K8S.md |
Kubernetes storage/host analysis (predecessor to k8s/) |
2026-07-27 |
Server Build (server-build/)
Server provisioning, security hardening, DNS/NTP configuration.
| Document | Description | Last Reviewed |
|---|---|---|
server-build/SECURITY.md |
Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
server-build/tailscale.md |
Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
server-build/DEPLOYMENT.md |
Server deployment procedures, package lists, config flow | 2026-07-25 |
server-build/TSYS-2FA-GUIDE.md |
End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
server-build/DEVELOPMENT-GUIDELINES.md |
Coding standards, commit conventions, script patterns | 2026-07-25 |
Operational Guides (outside docs/)
| Document | Description | Last Reviewed |
|---|---|---|
../netinfra/dhcp-migration.md |
DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
../awx/README.md |
Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
../powerman/README.md |
Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
../console/README.md |
Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
../ups/README.md |
UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
../k8s/README.md |
k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
../dns-cluster-setup/README.md |
Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
../tests/README.md |
Test suite documentation: unit, security, validation tests | 2026-07-28 |
../netinfra/pfv-netinfra-setup.md |
pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
../netinfra/pfv-netboot-setup.md |
pfv-netboot reference node setup | 2026-07-27 |
Archive (archive/)
Historical AI reviews, completed task lists, and pre-refactor examples. Read-only context — do not update; link to active docs instead.
| Document | Description |
|---|---|
archive/CODE-REVIEW-FINDINGS.md |
Early code review findings (most issues now fixed) |
archive/REFACTORING-EXAMPLES.md |
Pre-refactor code patterns (historical "before" examples) |
archive/Claude-Review.md |
Claude's initial code review |
archive/AIReview-QWEN.md |
Qwen AI review |
archive/AiOverview-Gemini.md |
Gemini project overview |
archive/AiOverview-OpenCode.md |
OpenCode project overview |
archive/AiSecurityAudit-Gemini.md |
Gemini security audit |
archive/charles-todo.md |
Charles's early task list (completed) |
archive/claude-todo.md |
Claude's early task list (completed) |
Top-Level Files
| File | Description |
|---|---|
../AGENTS.md |
Agent operating instructions (repo layout, git policy, gardening protocol) |
../STATUS.md |
Living project status (agent-maintained, human read-only) |
../README.md |
Project overview and quick start |
../LICENSE |
License |
Agent Gardening Protocol
When making changes to this repo, agents MUST:
- Update
../STATUS.mdif the work changes infrastructure state, completes/starts a task, or discovers a new issue. - Update this file (
docmap.md) if a doc is added, removed, or has a substantive content change. Update the "Last Reviewed" date. - Verify cross-references — any new
.mdfile must be linked from at least one existing doc or this map. - Check for stale paths — after any directory rename or file move,
grep -rn 'old/path' --include='*.md'and fix all references. - Keep code and docs in sync — if you change a script's interface, behavior, or location, update every doc that references it in the same commit.