KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add STATUS.md, docmap.md, encode gardening protocol

Browse Source 
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)

Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.

Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.

Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
mrcharles
2026-07-28 11:32:03 -05:00
parent 37e59ca310
commit 3d5b6c859e
4 changed files with 208 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+56 -68
View File
@@ -1,87 +1,75 @@
# Agent Guidelines
This repo combines two formerly-separate projects:
- **Server provisioning** (formerly KNELServerBuild): [`provisioning/`](provisioning/),
[`tests/`](tests/), [`vendor/`](vendor/), [`dns-cluster-setup/`](dns-cluster-setup/)
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): [`perf/`](perf/),
[`netinfra/`](netinfra/), [`switches/`](switches/)
**Top-level files:** [`README.md`](README.md) (project overview),
[`STATUS.md`](STATUS.md) (living status, agent-maintained),
[`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else
lives in subdirectories.
## Repository Layout
- **Vendored framework**: `KNELShellFramework` lives at
`vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes
are under `Framework-Includes/` there. Never assume `./Framework-Includes`
exists relative to the repo root.
- **Self-locating scripts**: All provisioning scripts derive their own location
via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never
depend on the current working directory. Run from anywhere.
- **Local config files are the source of truth**: Configs in
```
provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA)
tests/ Test suite + VM validation harness
dns-cluster-setup/ Technitium DNS cluster replication
perf/ Proxmox perf tuning, fleet audit, iperf
netinfra/ pfv-netinfra-01/02 DNS/NTP setup
switches/ Switch configuration captures
docs/ All documentation (see docs/docmap.md)
vendor/ Vendored KNELShellFramework
```
- **Self-locating scripts**: All provisioning scripts derive their own
location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run
from anywhere.
- **Local configs are the source of truth**: Files in
[`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
`cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads — that CDN
is deprecated.
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh`
extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax
checkers must skip these.
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox
Datacenter Manager). SSH keys deployed to `localuser` with passwordless sudo
on all hosts.
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
Use the wrapper scripts: [`tests/remote.sh`](tests/remote.sh),
[`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh),
or the `perf/deploy-check.sh` / `perf/deploy-tuning.sh` patterns.
`cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **SSH in Crush**: Direct ssh/scp is blocked. Use
[`tests/remote.sh`](tests/remote.sh) or
[`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh).
## Git Commit Requirements
## Git Policy
1. **Commit atomically**: each logical change its own commit.
2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`,
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`.
3. **Verbose messages**: title (50 chars max), blank line, body explaining
WHAT and WHY, footer with attribution.
1. **Autonomous commit + push.** Do not wait to be asked. After each logical
unit: stage, commit (conventional format), push to `origin/main`.
2. **Atomic commits.** Each commit coherent on its own.
3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
## Autonomous Git Workflow
## Automatic Gardening Protocol
**Agents are authorized to commit AND push autonomously — do not wait to be
asked.** After each logical unit of work:
**Docs and code must be kept in sync.** After any work session, an agent MUST:
1. Stage only the files belonging to that logical change.
2. Commit with a conventional, well-formed message.
3. Push to `origin` (`git push`). The branch tracks `origin/main`.
4. Repeat per logical unit.
1. **Update [`STATUS.md`](STATUS.md)** — reflect completed work, new issues,
changed infrastructure state. This file is human read-only; agents own it.
2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added,
removed, or substantively changed, update the table and "Last Reviewed"
date.
3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after
any rename or restructure. Fix all references in the same commit.
4. **Verify new docs are linked** — every new `.md` file must appear in
[`docs/docmap.md`](docs/docmap.md) and be linked from at least one other
doc.
Group changes so each commit is coherent on its own (a reader should
understand the commit without seeing the others). Never batch unrelated
changes into one commit.
## Key scripts
## Key Scripts
| Script | Purpose |
|--------|---------|
| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning (packages, hardening, 2FA) |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | End-to-end deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Project test suite (unit/security/validation) |
| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | Technitium DNS cluster replication |
| [`perf/deploy-check.sh`](perf/deploy-check.sh) | Deploy read-only data collector to hosts |
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings to hosts |
| [`perf/validate-fixes.sh`](perf/validate-fixes.sh) | Validate applied tuning changes |
| [`perf/iperf-full-matrix.sh`](perf/iperf-full-matrix.sh) | Full iperf throughput suite |
| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
## Key documentation
## Key Docs
| Doc | Contents |
|-----|----------|
| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
→ **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.**
## Project context
## Project Context
This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring
budget. Redundancy is not a concern for the R&D cluster. Backups DO matter
(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See
[`docs/PROJECT.md`](docs/PROJECT.md) for the comprehensive fleet report and
[`docs/TODO.md`](docs/TODO.md) for pending hardware work.
Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
Production lives on a Cloudron VPS in Reston VA. See
[`STATUS.md`](STATUS.md) for current state and
[`docs/PROJECT.md`](docs/PROJECT.md) for the fleet report.
README.md
+9 -32
View File
@@ -1,7 +1,8 @@
# PFVCluster
Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
Combines server provisioning, Proxmox cluster operations, and DNS infrastructure.
**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)**
## Directory Structure
@@ -13,7 +14,7 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure
| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
| [`switches/`](switches/) | Switch configuration captures |
| [`docs/`](docs/) | All documentation |
| [`docs/`](docs/) | All documentation ([see docmap](docs/docmap.md)) |
| [`vendor/`](vendor/) | Vendored KNELShellFramework |
## Quick Start
@@ -22,50 +23,26 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure
```bash
sudo bash provisioning/SetupNewSystem.sh
```
Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh),
configures NTP/DNS/SNMP/syslog/postfix.
### Validate provisioning on the sandbox VM
### Validate on the sandbox VM
```bash
VM_ID=6000 ./tests/vm-validation.sh all
```
Snapshots, deploys, runs the test suite, auto-rolls back on failure.
### Run the test suite
### Deploy DNS cluster
```bash
./tests/run-tests.sh all
cd dns-cluster-setup/ && ./setup.sh all
```
### Deploy DNS cluster setup
### Deploy perf tunings
```bash
cd dns-cluster-setup/
./setup.sh all
cd perf/ && ./deploy-tuning.sh
```
### Deploy perf tunings to hosts
```bash
cd perf/
./deploy-check.sh # read-only data collection
./deploy-tuning.sh # apply sysctl/tuned/NFS tunings
```
## Key Documentation
| Doc | Contents |
|-----|----------|
| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
## Architecture
- **Proxmox hosts**: 7 standalone PVE installs managed via PDM
- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs)
- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
- **Backups**: Proxmox Backup Server (PBS)
STATUS.md
+69
View File
@@ -0,0 +1,69 @@
# Project Status
> **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-28 by Crush (GLM-5)
## Current State: STABLE
The merged PFVCluster repo is fully operational across provisioning, DNS
infrastructure, and Proxmox cluster ops.
## Completed Work
### Server Provisioning (validated on sectestbed-sandbox)
- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
- [x] DNS resolv.conf: managed static file pointing at netinfra pair
- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
### DNS Cluster (pfv-netinfra-01/02)
- [x] Production Technitium config replicated from tailscale-router (read-only)
- [x] 124 zones on both nodes (knel.net + reverse DNS)
- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
- [x] Zone replication via rsync systemd timer (every 60s)
- [x] Credentials + 2FA replicated identically to production
- [x] Both LAN IPs resolve knel.net device names + recurse externally
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit complete (PROJECT.md has ground truth)
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Repo Merge
- [x] KNELServerBuild merged into PFVCluster (history preserved)
- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
## Known Issues
| Issue | Impact | Status |
|-------|--------|--------|
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
## Pending (next session priorities)
1. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
2. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
3. Perf: complete tsys4/5 tuning after hardware install (Friday)
4. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
## Infrastructure Summary
| Component | Details |
|-----------|---------|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| DNS production | tailscale-router (read-only source of truth) |
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
| Backup | Proxmox Backup Server (PBS) |
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
docs/docmap.md
+74
View File
@@ -0,0 +1,74 @@
# Documentation Map
> **Index of all documentation in this repo.** Agents must update this file
> whenever a doc is added, removed, or substantively changed.
> **Last updated:** 2026-07-28
## Active Documentation
These docs reflect current reality and should be kept up to date.
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`PROJECT.md`](PROJECT.md) | Comprehensive fleet report: 7 Proxmox hosts, VM inventory, storage architecture, recommendations | 2026-07-27 |
| [`SECURITY.md`](SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
| [`tailscale.md`](tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
| [`DEPLOYMENT.md`](DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
| [`TODO.md`](TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`K8S.md`](K8S.md) | Kubernetes architecture deep-dive (for future k8s session) | 2026-07-27 |
| [`TSYS-2FA-GUIDE.md`](TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
| [`DEVELOPMENT-GUIDELINES.md`](DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
## Operational Guides (outside docs/)
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
## Historical / Reference (read-only context)
These docs are preserved for historical context. They may reference pre-merge
paths or outdated architecture. Do not update them; link to active docs instead.
| Document | Description |
|----------|-------------|
| [`CODE-REVIEW-FINDINGS.md`](CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) |
| [`REFACTORING-EXAMPLES.md`](REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) |
| [`Claude-Review.md`](Claude-Review.md) | Claude's initial code review |
| [`AIReview-QWEN.md`](AIReview-QWEN.md) | Qwen AI review |
| [`AiOverview-Gemini.md`](AiOverview-Gemini.md) | Gemini project overview |
| [`AiOverview-OpenCode.md`](AiOverview-OpenCode.md) | OpenCode project overview |
| [`AiSecurityAudit-Gemini.md`](AiSecurityAudit-Gemini.md) | Gemini security audit |
| [`charles-todo.md`](charles-todo.md) | Charles's early task list (completed) |
| [`claude-todo.md`](claude-todo.md) | Claude's early task list (completed) |
## Top-Level Files
| File | Description |
|------|-------------|
| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, key scripts) |
| [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) |
| [`../README.md`](../README.md) | Project overview and quick start |
| [`../LICENSE`](../LICENSE) | License |
---
## Agent Gardening Protocol
When making changes to this repo, agents MUST:
1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure
state, completes/starts a task, or discovers a new issue.
2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a
substantive content change. Update the "Last Reviewed" date.
3. **Verify cross-references** — any new `.md` file must be linked from at
least one existing doc or this map.
4. **Check for stale paths** — after any directory rename or file move,
grep all `.md` files for old paths and fix them.
5. **Keep code and docs in sync** — if you change a script's interface,
behavior, or location, update every doc that references it in the same
commit.