mrcharles
26de312c42
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
102 lines
6.9 KiB
Markdown
102 lines
6.9 KiB
Markdown
# Documentation Map
|
|
|
|
> **Index of all documentation in this repo.** Agents must update this file
|
|
> whenever a doc is added, removed, or substantively changed.
|
|
|
|
> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute)
|
|
|
|
## Kubernetes Architecture ([`k8s/`](k8s/))
|
|
|
|
Distro decision, target architecture, control-plane design, bootstrap and DR
|
|
procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
|
|
|
|
| Document | Description | Last Reviewed |
|
|
|----------|-------------|---------------|
|
|
| [`k8s/docs/README.md`](k8s/docs/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
|
|
| [`k8s/docs/DISTRO-DECISION.md`](k8s/docs/DISTRO-DECISION.md) | Talos vs k3s analysis. Decision: k3s for regular R&D cluster; Talos preserved for future ITAR/classified | 2026-07-28 |
|
|
| [`k8s/docs/ARCHITECTURE.md`](k8s/docs/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
|
|
|
|
## Proxmox Cluster ([`proxmox/`](proxmox/))
|
|
|
|
Fleet operations, hardware, performance tuning, storage architecture.
|
|
|
|
| Document | Description | Last Reviewed |
|
|
|----------|-------------|---------------|
|
|
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
|
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
|
|
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 |
|
|
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 |
|
|
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
|
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
|
|
|
## Server Build ([`server-build/`](server-build/))
|
|
|
|
Server provisioning, security hardening, DNS/NTP configuration.
|
|
|
|
| Document | Description | Last Reviewed |
|
|
|----------|-------------|---------------|
|
|
| [`server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
|
|
| [`server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
|
|
| [`server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
|
|
| [`server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
|
|
| [`server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
|
|
|
|
## Operational Guides (outside docs/)
|
|
|
|
| Document | Description | Last Reviewed |
|
|
|----------|-------------|---------------|
|
|
| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
|
|
| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
|
|
| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
|
|
| [`../console/README.md`](../console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
|
|
| [`../ups/README.md`](../ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
|
|
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
|
|
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
|
|
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
|
|
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
|
|
| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
|
|
|
|
## Archive ([`archive/`](archive/))
|
|
|
|
Historical AI reviews, completed task lists, and pre-refactor examples. Read-only
|
|
context — do not update; link to active docs instead.
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [`archive/CODE-REVIEW-FINDINGS.md`](archive/CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) |
|
|
| [`archive/REFACTORING-EXAMPLES.md`](archive/REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) |
|
|
| [`archive/Claude-Review.md`](archive/Claude-Review.md) | Claude's initial code review |
|
|
| [`archive/AIReview-QWEN.md`](archive/AIReview-QWEN.md) | Qwen AI review |
|
|
| [`archive/AiOverview-Gemini.md`](archive/AiOverview-Gemini.md) | Gemini project overview |
|
|
| [`archive/AiOverview-OpenCode.md`](archive/AiOverview-OpenCode.md) | OpenCode project overview |
|
|
| [`archive/AiSecurityAudit-Gemini.md`](archive/AiSecurityAudit-Gemini.md) | Gemini security audit |
|
|
| [`archive/charles-todo.md`](archive/charles-todo.md) | Charles's early task list (completed) |
|
|
| [`archive/claude-todo.md`](archive/claude-todo.md) | Claude's early task list (completed) |
|
|
|
|
## Top-Level Files
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, gardening protocol) |
|
|
| [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) |
|
|
| [`../README.md`](../README.md) | Project overview and quick start |
|
|
| [`../LICENSE`](../LICENSE) | License |
|
|
|
|
---
|
|
|
|
## Agent Gardening Protocol
|
|
|
|
When making changes to this repo, agents MUST:
|
|
|
|
1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure
|
|
state, completes/starts a task, or discovers a new issue.
|
|
2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a
|
|
substantive content change. Update the "Last Reviewed" date.
|
|
3. **Verify cross-references** — any new `.md` file must be linked from at
|
|
least one existing doc or this map.
|
|
4. **Check for stale paths** — after any directory rename or file move,
|
|
`grep -rn 'old/path' --include='*.md'` and fix all references.
|
|
5. **Keep code and docs in sync** — if you change a script's interface,
|
|
behavior, or location, update every doc that references it in the same
|
|
commit.
|