KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add STATUS.md, docmap.md, encode gardening protocol

Browse Source 
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)

Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.

Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.

Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
mrcharles
2026-07-28 11:32:03 -05:00
parent a7fbad60ab
commit 4f2bfe18a5
4 changed files with 208 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+56 -68
View File
@@ -1,87 +1,75 @@
# Agent Guidelines # Agent Guidelines
This repo combines two formerly-separate projects: **Top-level files:** [`README.md`](README.md) (project overview),
- **Server provisioning** (formerly KNELServerBuild): [`provisioning/`](provisioning/), [`STATUS.md`](STATUS.md) (living status, agent-maintained),
[`tests/`](tests/), [`vendor/`](vendor/), [`dns-cluster-setup/`](dns-cluster-setup/) [`docs/docmap.md`](docs/docmap.md) (documentation index). Everything else
- **Proxmox cluster ops** (formerly PFVCluster/perfopt): [`perf/`](perf/), lives in subdirectories.
[`netinfra/`](netinfra/), [`switches/`](switches/)
## Repository Layout ## Repository Layout
- **Vendored framework**: `KNELShellFramework` lives at ```
`vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/`. Its includes provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA)
are under `Framework-Includes/` there. Never assume `./Framework-Includes` tests/ Test suite + VM validation harness
exists relative to the repo root. dns-cluster-setup/ Technitium DNS cluster replication
- **Self-locating scripts**: All provisioning scripts derive their own location perf/ Proxmox perf tuning, fleet audit, iperf
via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. They must never netinfra/ pfv-netinfra-01/02 DNS/NTP setup
depend on the current working directory. Run from anywhere. switches/ Switch configuration captures
- **Local config files are the source of truth**: Configs in docs/ All documentation (see docs/docmap.md)
vendor/ Vendored KNELShellFramework
```
- **Self-locating scripts**: All provisioning scripts derive their own
location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run
from anywhere.
- **Local configs are the source of truth**: Files in
[`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
`cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads — that CDN `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
is deprecated. - **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
- **Non-bash agents**: Some files under `provisioning/Agents/` carry a `.sh` extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
extension but are PHP (e.g. `mysql.sh`, shebang `#!/usr/bin/php`). Syntax - **SSH in Crush**: Direct ssh/scp is blocked. Use
checkers must skip these. [`tests/remote.sh`](tests/remote.sh) or
- **Proxmox hosts** are standalone installs managed via **PDM** (Proxmox [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh).
Datacenter Manager). SSH keys deployed to `localuser` with passwordless sudo
on all hosts.
- **SSH in Crush**: Direct ssh/scp is blocked in the Crush bash environment.
Use the wrapper scripts: [`tests/remote.sh`](tests/remote.sh),
[`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh),
or the `perf/deploy-check.sh` / `perf/deploy-tuning.sh` patterns.
## Git Commit Requirements ## Git Policy
1. **Commit atomically**: each logical change its own commit. 1. **Autonomous commit + push.** Do not wait to be asked. After each logical
2. **Conventional commit format**: `feat(scope): desc`, `fix(scope): desc`, unit: stage, commit (conventional format), push to `origin/main`.
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`, `chore: desc`. 2. **Atomic commits.** Each commit coherent on its own.
3. **Verbose messages**: title (50 chars max), blank line, body explaining 3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
WHAT and WHY, footer with attribution. `docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
## Autonomous Git Workflow ## Automatic Gardening Protocol
**Agents are authorized to commit AND push autonomously — do not wait to be **Docs and code must be kept in sync.** After any work session, an agent MUST:
asked.** After each logical unit of work:
1. Stage only the files belonging to that logical change. 1. **Update [`STATUS.md`](STATUS.md)** — reflect completed work, new issues,
2. Commit with a conventional, well-formed message. changed infrastructure state. This file is human read-only; agents own it.
3. Push to `origin` (`git push`). The branch tracks `origin/main`. 2. **Update [`docs/docmap.md`](docs/docmap.md)** — if a doc was added,
4. Repeat per logical unit. removed, or substantively changed, update the table and "Last Reviewed"
date.
3. **Grep for stale paths** — `grep -rn 'old/path' --include='*.md'` after
any rename or restructure. Fix all references in the same commit.
4. **Verify new docs are linked** — every new `.md` file must appear in
[`docs/docmap.md`](docs/docmap.md) and be linked from at least one other
doc.
Group changes so each commit is coherent on its own (a reader should ## Key Scripts
understand the commit without seeing the others). Never batch unrelated
changes into one commit.
## Key scripts
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning (packages, hardening, 2FA) | | [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | End-to-end deploy + validate on sandbox VM | | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Project test suite (unit/security/validation) | | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | Technitium DNS cluster replication | | [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
| [`perf/deploy-check.sh`](perf/deploy-check.sh) | Deploy read-only data collector to hosts | | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings to hosts |
| [`perf/validate-fixes.sh`](perf/validate-fixes.sh) | Validate applied tuning changes |
| [`perf/iperf-full-matrix.sh`](perf/iperf-full-matrix.sh) | Full iperf throughput suite |
## Key documentation ## Key Docs
| Doc | Contents | → **See [`docs/docmap.md`](docs/docmap.md) for the full documentation index.**
|-----|----------|
| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
## Project context ## Project Context
This is a solo-founder R&D Proxmox cluster in a private residence. Shoestring Solo-founder R&D Proxmox cluster in a private residence. Shoestring budget.
budget. Redundancy is not a concern for the R&D cluster. Backups DO matter Production lives on a Cloudron VPS in Reston VA. See
(PBS in use). Production lives on a VPS in Reston VA (Cloudron). See [`STATUS.md`](STATUS.md) for current state and
[`docs/PROJECT.md`](docs/PROJECT.md) for the comprehensive fleet report and [`docs/PROJECT.md`](docs/PROJECT.md) for the fleet report.
[`docs/TODO.md`](docs/TODO.md) for pending hardware work.
README.md
+9 -32
View File
@@ -1,7 +1,8 @@
# PFVCluster # PFVCluster
Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster. Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluster.
Combines server provisioning, Proxmox cluster operations, and DNS infrastructure.
**[→ Current Status](STATUS.md)** · **[→ Documentation Index](docs/docmap.md)** · **[→ Agent Guidelines](AGENTS.md)**
## Directory Structure ## Directory Structure
@@ -13,7 +14,7 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure
| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics | | [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts | | [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP setup + audit scripts |
| [`switches/`](switches/) | Switch configuration captures | | [`switches/`](switches/) | Switch configuration captures |
| [`docs/`](docs/) | All documentation | | [`docs/`](docs/) | All documentation ([see docmap](docs/docmap.md)) |
| [`vendor/`](vendor/) | Vendored KNELShellFramework | | [`vendor/`](vendor/) | Vendored KNELShellFramework |
## Quick Start ## Quick Start
@@ -22,50 +23,26 @@ Combines server provisioning, Proxmox cluster operations, and DNS infrastructure
```bash ```bash
sudo bash provisioning/SetupNewSystem.sh sudo bash provisioning/SetupNewSystem.sh
``` ```
Installs packages, applies security hardening (SSH, SCAP-STIG, 2FA, Wazuh),
configures NTP/DNS/SNMP/syslog/postfix.
### Validate provisioning on the sandbox VM ### Validate on the sandbox VM
```bash ```bash
VM_ID=6000 ./tests/vm-validation.sh all VM_ID=6000 ./tests/vm-validation.sh all
``` ```
Snapshots, deploys, runs the test suite, auto-rolls back on failure.
### Run the test suite ### Deploy DNS cluster
```bash ```bash
./tests/run-tests.sh all cd dns-cluster-setup/ && ./setup.sh all
``` ```
### Deploy DNS cluster setup ### Deploy perf tunings
```bash ```bash
cd dns-cluster-setup/ cd perf/ && ./deploy-tuning.sh
./setup.sh all
``` ```
### Deploy perf tunings to hosts
```bash
cd perf/
./deploy-check.sh # read-only data collection
./deploy-tuning.sh # apply sysctl/tuned/NFS tunings
```
## Key Documentation
| Doc | Contents |
|-----|----------|
| [`docs/PROJECT.md`](docs/PROJECT.md) | Comprehensive fleet report (7 hosts, VM inventory, storage) |
| [`docs/SECURITY.md`](docs/SECURITY.md) | Security architecture and hardening details |
| [`docs/tailscale.md`](docs/tailscale.md) | Tailscale vs managed DNS analysis (resolved) |
| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | Deployment procedures |
| [`docs/TODO.md`](docs/TODO.md) | Pending hardware work (tsys2/4/5) |
| [`docs/K8S.md`](docs/K8S.md) | Kubernetes architecture deep-dive |
| [`dns-cluster-setup/README.md`](dns-cluster-setup/README.md) | DNS cluster setup guide |
| [`tests/README.md`](tests/README.md) | Test suite documentation |
## Architecture ## Architecture
- **Proxmox hosts**: 7 standalone PVE installs managed via PDM - **Proxmox hosts**: 7 standalone PVE installs managed via PDM
- **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02 - **DNS**: Technitium (authoritative) + Pi-hole (recursive) on pfv-netinfra-01/02
- **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs) - **NTP**: pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3)
- **Production**: Cloudron VPS in Reston VA (this cluster is R&D only) - **Production**: Cloudron VPS in Reston VA (this cluster is R&D only)
- **Backups**: Proxmox Backup Server (PBS) - **Backups**: Proxmox Backup Server (PBS)
STATUS.md
+69
View File
@@ -0,0 +1,69 @@
# Project Status
> **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-28 by Crush (GLM-5)
## Current State: STABLE
The merged PFVCluster repo is fully operational across provisioning, DNS
infrastructure, and Proxmox cluster ops.
## Completed Work
### Server Provisioning (validated on sectestbed-sandbox)
- [x] SetupNewSystem.sh deploys end-to-end to rc=0 (Debian 13 trixie)
- [x] Security hardening: SSH, SCAP-STIG, Wazuh, 2FA (SSH+Cockpit+Webmin)
- [x] NTP: redundant pfv-netinfra-01/02 (192.168.3.252/253), synced stratum 3
- [x] DNS resolv.conf: managed static file pointing at netinfra pair
- [x] Test suite: 5 tests (framework, safe-download, 2fa, https, system-req)
- [x] VM validation harness: git-based deploy + auto-rollback + guest-agent access
### DNS Cluster (pfv-netinfra-01/02)
- [x] Production Technitium config replicated from tailscale-router (read-only)
- [x] 124 zones on both nodes (knel.net + reverse DNS)
- [x] pfv-netinfra-01 = PRIMARY, pfv-netinfra-02 = SECONDARY
- [x] Zone replication via rsync systemd timer (every 60s)
- [x] Credentials + 2FA replicated identically to production
- [x] Both LAN IPs resolve knel.net device names + recurse externally
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit complete (PROJECT.md has ground truth)
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Repo Merge
- [x] KNELServerBuild merged into PFVCluster (history preserved)
- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
- [x] All docs gardened: links fixed, stale refs removed, tailscale.md updated
## Known Issues
| Issue | Impact | Status |
|-------|--------|--------|
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
## Pending (next session priorities)
1. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
2. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
3. Perf: complete tsys4/5 tuning after hardware install (Friday)
4. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
## Infrastructure Summary
| Component | Details |
|-----------|---------|
| Proxmox hosts | 7 standalone PVE, managed via PDM |
| DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole |
| DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole |
| DNS production | tailscale-router (read-only source of truth) |
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
| Backup | Proxmox Backup Server (PBS) |
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
docs/docmap.md
+74
View File
@@ -0,0 +1,74 @@
# Documentation Map
> **Index of all documentation in this repo.** Agents must update this file
> whenever a doc is added, removed, or substantively changed.
> **Last updated:** 2026-07-28
## Active Documentation
These docs reflect current reality and should be kept up to date.
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`PROJECT.md`](PROJECT.md) | Comprehensive fleet report: 7 Proxmox hosts, VM inventory, storage architecture, recommendations | 2026-07-27 |
| [`SECURITY.md`](SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
| [`tailscale.md`](tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
| [`DEPLOYMENT.md`](DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
| [`TODO.md`](TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`K8S.md`](K8S.md) | Kubernetes architecture deep-dive (for future k8s session) | 2026-07-27 |
| [`TSYS-2FA-GUIDE.md`](TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
| [`DEVELOPMENT-GUIDELINES.md`](DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
## Operational Guides (outside docs/)
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
## Historical / Reference (read-only context)
These docs are preserved for historical context. They may reference pre-merge
paths or outdated architecture. Do not update them; link to active docs instead.
| Document | Description |
|----------|-------------|
| [`CODE-REVIEW-FINDINGS.md`](CODE-REVIEW-FINDINGS.md) | Early code review findings (most issues now fixed) |
| [`REFACTORING-EXAMPLES.md`](REFACTORING-EXAMPLES.md) | Pre-refactor code patterns (historical "before" examples) |
| [`Claude-Review.md`](Claude-Review.md) | Claude's initial code review |
| [`AIReview-QWEN.md`](AIReview-QWEN.md) | Qwen AI review |
| [`AiOverview-Gemini.md`](AiOverview-Gemini.md) | Gemini project overview |
| [`AiOverview-OpenCode.md`](AiOverview-OpenCode.md) | OpenCode project overview |
| [`AiSecurityAudit-Gemini.md`](AiSecurityAudit-Gemini.md) | Gemini security audit |
| [`charles-todo.md`](charles-todo.md) | Charles's early task list (completed) |
| [`claude-todo.md`](claude-todo.md) | Claude's early task list (completed) |
## Top-Level Files
| File | Description |
|------|-------------|
| [`../AGENTS.md`](../AGENTS.md) | Agent operating instructions (repo layout, git policy, key scripts) |
| [`../STATUS.md`](../STATUS.md) | Living project status (agent-maintained, human read-only) |
| [`../README.md`](../README.md) | Project overview and quick start |
| [`../LICENSE`](../LICENSE) | License |
---
## Agent Gardening Protocol
When making changes to this repo, agents MUST:
1. **Update [`../STATUS.md`](../STATUS.md)** if the work changes infrastructure
state, completes/starts a task, or discovers a new issue.
2. **Update this file (`docmap.md`)** if a doc is added, removed, or has a
substantive content change. Update the "Last Reviewed" date.
3. **Verify cross-references** — any new `.md` file must be linked from at
least one existing doc or this map.
4. **Check for stale paths** — after any directory rename or file move,
grep all `.md` files for old paths and fix them.
5. **Keep code and docs in sync** — if you change a script's interface,
behavior, or location, update every doc that references it in the same
commit.