From 4f2bfe18a5cf12f96d15e4e3b42fc00fff5aa06b Mon Sep 17 00:00:00 2001 From: reachableceo Date: Tue, 28 Jul 2026 11:32:03 -0500 Subject: [PATCH] docs: add STATUS.md, docmap.md, encode gardening protocol MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure top-level to exactly three .md files: - AGENTS.md (agent operating instructions) - README.md (project overview, links to status + docmap) - STATUS.md (living project status, agent-maintained, human read-only) Add docs/docmap.md as the single documentation index/map. All docs are categorized (active, operational guides, historical) with last-reviewed dates. Includes the agent gardening protocol requiring agents to update STATUS.md and docmap.md after every work session. Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery, encodes the automatic gardening protocol (keep docs/code in sync, grep for stale paths after renames, update STATUS.md after infrastructure changes). All references are Gitea-renderable relative links. Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md, doc table replaced with pointer to docmap.md. ๐Ÿค– Generated with [Crush](https://github.com/charmassociates/crush) Assisted-by: GLM-5 via Crush --- AGENTS.md | 124 ++++++++++++++++++++++--------------------------- README.md | 41 ++++------------ STATUS.md | 69 +++++++++++++++++++++++++++ docs/docmap.md | 74 +++++++++++++++++++++++++++++ 4 files changed, 208 insertions(+), 100 deletions(-) create mode 100644 STATUS.md create mode 100644 docs/docmap.md diff --git a/AGENTS.md b/AGENTS.md index afc4ef5..dffa0cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,87 +1,75 @@ # Agent Guidelines -This repo combines two formerly-separate projects: -- **Server provisioning** (formerly KNELServerBuild): [`provisioning/`](provisioning/), - [`tests/`](tests/), [`vendor/`](vendor/), [`dns-cluster-setup/`](dns-cluster-setup/) -- **Proxmox cluster ops** (formerly PFVCluster/perfopt): [`perf/`](perf/), - [`netinfra/`](netinfra/), [`switches/`](switches/) +**Top-level files:** [`README.md`](README.md) (project overview), +[`STATUS.md`](STATUS.md) (living status, agent-maintained), +[`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else +lives in subdirectories. ## Repository Layout -- **Vendored framework**: `KNELShellFramework` lives at - `vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes - are under `Framework-Includes/` there. Never assume `./Framework-Includes` - exists relative to the repo root. -- **Self-locating scripts**: All provisioning scripts derive their own location - via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never - depend on the current working directory. Run from anywhere. -- **Local config files are the source of truth**: Configs in +``` +provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA) +tests/ Test suite + VM validation harness +dns-cluster-setup/ Technitium DNS cluster replication +perf/ Proxmox perf tuning, fleet audit, iperf +netinfra/ pfv-netinfra-01/02 DNS/NTP setup +switches/ Switch configuration captures +docs/ All documentation (see docs/docmap.md) +vendor/ Vendored KNELShellFramework +``` + +- **Self-locating scripts**: All provisioning scripts derive their own + location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run + from anywhere. +- **Local configs are the source of truth**: Files in [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with - `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads โ€” that CDN - is deprecated. -- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh` - extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax - checkers must skip these. -- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox - Datacenter Manager). SSH keys deployed to `localuser` with passwordless sudo - on all hosts. -- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment. - Use the wrapper scripts: [`tests/remote.sh`](tests/remote.sh), - [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh), - or the `perf/deploy-check.sh` / `perf/deploy-tuning.sh` patterns. + `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads. +- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh` + extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks. +- **SSH in Crush**: Direct ssh/scp is blocked. Use + [`tests/remote.sh`](tests/remote.sh) or + [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh). -## Git Commit Requirements +## Git Policy -1. **Commit atomically**: each logical change its own commit. -2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`, - `docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`. -3. **Verbose messages**: title (50 chars max), blank line, body explaining - WHAT and WHY, footer with attribution. +1. **Autonomous commit + push.** Do not wait to be asked. After each logical + unit: stage, commit (conventional format), push to `origin/main`. +2. **Atomic commits.** Each commit coherent on its own. +3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`, + `docs: desc`, `refactor(scope): desc`, `test(scope): desc`. -## Autonomous Git Workflow +## Automatic Gardening Protocol -**Agents are authorized to commit AND push autonomously โ€” do not wait to be -asked.** After each logical unit of work: +**Docs and code must be kept in sync.** After any work session, an agent MUST: -1. Stage only the files belonging to that logical change. -2. Commit with a conventional, well-formed message. -3. Push to `origin` (`git push`). The branch tracks `origin/main`. -4. Repeat per logical unit. +1. **Update [`STATUS.md`](STATUS.md)** โ€” reflect completed work, new issues, + changed infrastructure state. This file is human read-only; agents own it. +2. **Update [`docs/docmap.md`](docs/docmap.md)** โ€” if a doc was added, + removed, or substantively changed, update the table and "Last Reviewed" + date. +3. **Grep for stale paths** โ€” `grep -rn 'old/path' --include='*.md'` after + any rename or restructure. Fix all references in the same commit. +4. **Verify new docs are linked** โ€” every new `.md` file must appear in + [`docs/docmap.md`](docs/docmap.md) and be linked from at least one other + doc. -Group changes so each commit is coherent on its own (a reader should -understand the commit without seeing the others). Never batch unrelated -changes into one commit. - -## Key scripts +## Key Scripts | Script | Purpose | |--------|---------| -| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning (packages, hardening, 2FA) | -| [`tests/vm-validation.sh`](tests/vm-validation.sh) | End-to-end deploy + validate on sandbox VM | -| [`tests/run-tests.sh`](tests/run-tests.sh) | Project test suite (unit/security/validation) | -| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | Technitium DNS cluster replication | -| [`perf/deploy-check.sh`](perf/deploy-check.sh) | Deploy read-only data collector to hosts | -| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings to hosts | -| [`perf/validate-fixes.sh`](perf/validate-fixes.sh) | Validate applied tuning changes | -| [`perf/iperf-full-matrix.sh`](perf/iperf-full-matrix.sh) | Full iperf throughput suite | +| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning | +| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM | +| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite | +| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication | +| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings | -## Key documentation +## Key Docs -| Doc | Contents | -|-----|----------| -| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) | -| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details | -| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) | -| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures | -| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) | -| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive | -| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide | -| [`tests/README.md`](tests/README.md) | Test suite documentation | +โ†’ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.** -## Project context +## Project Context -This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring -budget. Redundancy is not a concern for the R&D cluster. Backups DO matter -(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See -[`docs/PROJECT.md`](docs/PROJECT.md) for the comprehensive fleet report and -[`docs/TODO.md`](docs/TODO.md) for pending hardware work. +Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget. +Production lives on a Cloudron VPS in Reston VA. See +[`STATUS.md`](STATUS.md) for current state and +[`docs/PROJECT.md`](docs/PROJECT.md) for the fleet report. diff --git a/README.md b/README.md index 07de73e..77975a7 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ # PFVCluster Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster. -Combines server provisioning, Proxmox cluster operations, and DNS infrastructure. + +**[โ†’ Current Status](STATUS.md)** ยท **[โ†’ Documentation Index](docs/docmap.md)** ยท **[โ†’ Agent Guidelines](AGENTS.md)** ## Directory Structure @@ -13,7 +14,7 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics | | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts | | [`switches/`](switches/) | Switch configuration captures | -| [`docs/`](docs/) | All documentation | +| [`docs/`](docs/) | All documentation ([see docmap](docs/docmap.md)) | | [`vendor/`](vendor/) | Vendored KNELShellFramework | ## Quick Start @@ -22,50 +23,26 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure ```bash sudo bash provisioning/SetupNewSystem.sh ``` -Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh), -configures NTP/DNS/SNMP/syslog/postfix. -### Validate provisioning on the sandbox VM +### Validate on the sandbox VM ```bash VM_ID=6000 ./tests/vm-validation.sh all ``` -Snapshots, deploys, runs the test suite, auto-rolls back on failure. -### Run the test suite +### Deploy DNS cluster ```bash -./tests/run-tests.sh all +cd dns-cluster-setup/ && ./setup.sh all ``` -### Deploy DNS cluster setup +### Deploy perf tunings ```bash -cd dns-cluster-setup/ -./setup.sh all +cd perf/ && ./deploy-tuning.sh ``` -### Deploy perf tunings to hosts -```bash -cd perf/ -./deploy-check.sh # read-only data collection -./deploy-tuning.sh # apply sysctl/tuned/NFS tunings -``` - -## Key Documentation - -| Doc | Contents | -|-----|----------| -| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) | -| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details | -| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) | -| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures | -| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) | -| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive | -| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide | -| [`tests/README.md`](tests/README.md) | Test suite documentation | - ## Architecture - **Proxmox hosts**: 7 standalone PVE installs managed via PDM - **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02 -- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs) +- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) - **Production**: Cloudron VPS in Reston VA (this cluster is R&D only) - **Backups**: Proxmox Backup Server (PBS) diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..f2bcfdd --- /dev/null +++ b/STATUS.md @@ -0,0 +1,69 @@ +# Project Status + +> **Human read-only. Agents maintain this file automatically after each work +> session.** Do not edit by hand โ€” the next agent run will overwrite it. + +> **Last updated:** 2026-07-28 by Crush (GLM-5) + +## Current State: STABLE + +The merged PFVCluster repo is fully operational across provisioning, DNS +infrastructure, and Proxmox cluster ops. + +## Completed Work + +### Server Provisioning (validated on sectestbed-sandbox) +- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie) +- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin) +- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3 +- [x] DNS resolv.conf: managed static file pointing at netinfra pair +- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req) +- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access + +### DNS Cluster (pfv-netinfra-01/02) +- [x] Production Technitium config replicated from tailscale-router (read-only) +- [x] 124 zones on both nodes (knel.net + reverse DNS) +- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY +- [x] Zone replication via rsync systemd timer (every 60s) +- [x] Credentials + 2FA replicated identically to production +- [x] Both LAN IPs resolve knel.net device names + recurse externally + +### Proxmox Cluster Ops +- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9) +- [x] Fleet audit complete (PROJECT.md has ground truth) +- [ ] tsys4: blocked on PCIe NIC + RAM install +- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install +- [ ] tsys2: pending rebuild from Win10 to Proxmox + +### Repo Merge +- [x] KNELServerBuild merged into PFVCluster (history preserved) +- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/) +- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated + +## Known Issues + +| Issue | Impact | Status | +|-------|--------|--------| +| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place | +| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue | +| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install | + +## Pending (next session priorities) + +1. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy +2. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes +3. Perf: complete tsys4/5 tuning after hardware install (Friday) +4. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host) + +## Infrastructure Summary + +| Component | Details | +|-----------|---------| +| Proxmox hosts | 7 standalone PVE, managed via PDM | +| DNS primary | pfv-netinfra-01 (192.168.3.252) โ€” Technitium + Pi-hole | +| DNS secondary | pfv-netinfra-02 (192.168.3.253) โ€” Technitium + Pi-hole | +| DNS production | tailscale-router (read-only source of truth) | +| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) | +| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) | +| Backup | Proxmox Backup Server (PBS) | +| Production | Cloudron VPS, Reston VA (this cluster is R&D only) | diff --git a/docs/docmap.md b/docs/docmap.md new file mode 100644 index 0000000..d0bd2f4 --- /dev/null +++ b/docs/docmap.md @@ -0,0 +1,74 @@ +# Documentation Map + +> **Index of all documentation in this repo.** Agents must update this file +> whenever a doc is added, removed, or substantively changed. + +> **Last updated:** 2026-07-28 + +## Active Documentation + +These docs reflect current reality and should be kept up to date. + +| Document | Description | Last Reviewed | +|----------|-------------|---------------| +| [`PROJECT.md`](PROJECT.md) | Comprehensive fleet report: 7 Proxmox hosts, VM inventory, storage architecture, recommendations | 2026-07-27 | +| [`SECURITY.md`](SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 | +| [`tailscale.md`](tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED โ€” netinfra pair serves knel.net) | 2026-07-28 | +| [`DEPLOYMENT.md`](DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 | +| [`TODO.md`](TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | +| [`K8S.md`](K8S.md) | Kubernetes architecture deep-dive (for future k8s session) | 2026-07-27 | +| [`TSYS-2FA-GUIDE.md`](TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 | +| [`DEVELOPMENT-GUIDELINES.md`](DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 | + +## Operational Guides (outside docs/) + +| Document | Description | Last Reviewed | +|----------|-------------|---------------| +| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | +| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | +| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 | +| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 | + +## Historical / Reference (read-only context) + +These docs are preserved for historical context. They may reference pre-merge +paths or outdated architecture. Do not update them; link to active docs instead. + +| Document | Description | +|----------|-------------| +| [`CODE-REVIEW-FINDINGS.md`](CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) | +| [`REFACTORING-EXAMPLES.md`](REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) | +| [`Claude-Review.md`](Claude-Review.md) | Claude's initial code review | +| [`AIReview-QWEN.md`](AIReview-QWEN.md) | Qwen AI review | +| [`AiOverview-Gemini.md`](AiOverview-Gemini.md) | Gemini project overview | +| [`AiOverview-OpenCode.md`](AiOverview-OpenCode.md) | OpenCode project overview | +| [`AiSecurityAudit-Gemini.md`](AiSecurityAudit-Gemini.md) | Gemini security audit | +| [`charles-todo.md`](charles-todo.md) | Charles's early task list (completed) | +| [`claude-todo.md`](claude-todo.md) | Claude's early task list (completed) | + +## Top-Level Files + +| File | Description | +|------|-------------| +| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, key scripts) | +| [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) | +| [`../README.md`](../README.md) | Project overview and quick start | +| [`../LICENSE`](../LICENSE) | License | + +--- + +## Agent Gardening Protocol + +When making changes to this repo, agents MUST: + +1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure + state, completes/starts a task, or discovers a new issue. +2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a + substantive content change. Update the "Last Reviewed" date. +3. **Verify cross-references** โ€” any new `.md` file must be linked from at + least one existing doc or this map. +4. **Check for stale paths** โ€” after any directory rename or file move, + grep all `.md` files for old paths and fix them. +5. **Keep code and docs in sync** โ€” if you change a script's interface, + behavior, or location, update every doc that references it in the same + commit.