Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.
The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).
Directory reorganization and doc gardening will follow in subsequent
commits.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.
Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds the PowerShell and cmd.exe commands the user needs to run on the
Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe
details before rebuilding it as Proxmox. Also documents the Friday
maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open
questions for the next session.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode
file. Add version control instructions to AGENTS.md for future sessions.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Make AGENTS.md actionable for future agents by recording what was learned
while fixing the provisioning scripts:
- Add a Repository Layout section: the KNELShellFramework is vendored
under vendor/.../KNELShellFramework (not at repo root), scripts must
self-locate via BASH_SOURCE, configs are read locally (no CDN), and
some .sh agents are actually PHP
- Replace the vague "commit immediately" note with an explicit
Autonomous Git Workflow section authorizing agents to commit AND push
without being asked, grouped into coherent atomic commits
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add agent guidelines for AI assistants working on this repository:
- Document atomic commit requirements
- Specify conventional commit format with examples
- Require verbose, formatted commit messages
- Emphasize immediate commit/push behavior
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles
Charles N Wyble