Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.
Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check
All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.
Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Author the docs/k8s/ directory capturing the pfv-k8s control-plane design:
- README.md: TL;DR of all decisions (distro, runtime, cnode count, admin
access, identity, tenancy, registry, storage)
- DISTRO-DECISION.md: Talos vs k3s analysis. Recommend Talos because the
ITAR/classified requirement makes its immutable, API-only, measured-boot
posture structurally easier to certify than SCAP-hardened Debian. k3s was
only ever a plan (no cluster deployed), so cutover cost is ~zero.
- ARCHITECTURE.md: target arch with mermaid diagrams covering control
plane, LAN-only network with Tailscale subnet-router admin, Cilium CNI,
OIDC to Keycloak, per-tenant vcluster isolation (incl. ITAR tenant),
Harbor pull-through cache on D3 SSD, bootstrap sequence, and DR.
Gardening: register docs/k8s/ in docmap.md, update STATUS.md with the new
k8s section and the three pending user decisions (cnode count, host spread,
Tailscale pattern).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Strengthen Git Policy point 1 so agents never hold work for review. The
user reviews rendered markdown on Gitea after push, so pausing to "let
them read first" defeats the workflow. Explicitly overrides any default
conservative commit-and-hold behavior.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)
Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.
Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.
Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Replace the KNELServerBuild README with a unified PFVCluster README
covering both provisioning and cluster ops. Update AGENTS.md to document
the merged repo layout, key scripts, and project context. Consolidate
all documentation under docs/.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.
The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).
Directory reorganization and doc gardening will follow in subsequent
commits.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Set up a fully scripted, documented Technitium DNS cluster that
replicates the production instance from tailscale-router to
pfv-netinfra-01 (primary) and pfv-netinfra-02 (secondary).
What it does:
- EXPORT: reads the production Technitium config (auth.config with
users + 2FA, dns.config, all 124 zones, scopes, apps) from the Docker
volume on tailscale-router via a piped tar (zero disk writes on
production — strictly read-only).
- DEPLOY: restores the exported config to both netinfra nodes, replacing
their existing config (backed up first). Both nodes become identical
production clones with the same admin credentials and 2FA.
- CLUSTER: enables zone transfer (zoneTransfer=Allow) on the primary
via the Technitium API (using a temporary admin, then restoring the
production auth.config). Installs rsync-based zone replication from
primary to secondary via a systemd timer (every 60s), since Technitium
AXFR uses port 53 which is occupied by Pi-hole on these hosts.
- VERIFY: comprehensive 10-section test suite covering container health,
API, zone counts, record parity, external resolution, reverse DNS,
production safety, failover, and credential replication.
Scripts:
- remote-dns.sh: SSH chokepoint for all DNS host access
- setup.sh: master orchestrator (export → deploy → cluster → verify)
- sync-zones.sh: rsync-based zone replication (installed as systemd timer)
- verify.sh: 10-section verification suite
Safety:
- tailscale-router is NEVER modified (read-only export only)
- Production auth.config is backed up before any temporary admin swap
- Each node's existing config is backed up before replacement
- The export tarball is gitignored (contains production credentials)
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The ntp.conf hardening used `interface ignore wildcard` +
`interface listen 127.0.0.1`, which binds ntpd to loopback only. Outbound
NTP queries to the upstream servers then carried a 127.0.0.1 source
address that the servers cannot reply to, so the daemon's peers stayed
stuck in .INIT. with reach 0 — even though the servers are reachable
(verified: ntpdate -q succeeds, ntpd does not).
Replace the interface-based restriction with restrict-based hardening:
`restrict default ignore` blocks unsolicited queries from any host (so
the box never serves time to others), while explicit allow rules for the
two upstream servers and localhost let the client sync normally.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an architecture analysis for the tension between Tailscale's
default resolv.conf management (100.100.100.100) and the managed
LAN-resolver resolv.conf (.252/.253). Documents a key finding from
live-network probing: knel.net device records only resolve via the
Tailscale MagicDNS path; querying the LAN DNS servers directly returns
NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23).
Lays out four options (Tailscale-owned, LAN-pinned, split DNS,
Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS
to Tailscale in the short term (since wazuh/postfix/syslog depend on
knel.net names that only resolve there) and fixing the Technitium/Pi-hole
knel.net zone before pinning the LAN resolvers. Confirms the NTP
(LAN-IP) change is safe regardless. Flags that the managed-resolv.conf
change will be overwritten by Tailscale and would break knel.net
resolution if it ever sticks.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Route every host built by this project through the new
pfv-netinfra-01 (192.168.3.252) / pfv-netinfra-02 (192.168.3.253)
pair for both name resolution and time, with automatic failover.
- NTP: replace the single pfv-netboot.knel.net upstream with both
netinfra servers (iburst) so time sync survives either one failing.
- DNS: add a managed static /etc/resolv.conf (new ConfigFiles/Resolv/).
The repo previously had no resolver configuration at all. Both servers
are listed so glibc falls through to the secondary on failure.
- DHCP: request domain-name-servers/domain-search/ntp-servers and
supersede them to the netinfra pair, so a DHCP renew can't silently
revert to whatever the DHCP server advertises.
- SetupNewSystem.sh: deploy resolv.conf (robustly replacing any
systemd-resolved/NetworkManager symlink) and add pfv-netinfra to the
NTP-server self-exclusion guard so those boxes don't client off
themselves.
LAN IPs are used throughout (not the knel.net hostnames) because those
hostnames resolve to Tailscale CGNAT addresses, not the LAN addresses,
and NTP must come up before DNS. Add a validation test asserting the
config is present and both servers actually answer DNS and NTP queries.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
redundancy only, not for speed
D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.
Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.
Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds the PowerShell and cmd.exe commands the user needs to run on the
Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe
details before rebuilding it as Proxmox. Also documents the Friday
maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open
questions for the next session.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode
file. Add version control instructions to AGENTS.md for future sessions.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The framework defines LOGFILENAME as "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$"
and PrettyPrint appends every print_info/print_error line to it, so
executing any script that sources the framework leaves a timestamped log
file beside it (e.g. run-tests.sh.Monday-2026-07-27-10:44:31.123). These
are runtime artifacts, not source, and were showing up as untracked
noise. Ignore them across the whole repo.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_webmin_2fa used `sed -i ... || echo ... >>` to add
twofactor_provider and twofactor to /etc/webmin/miniserv.conf. sed
returns 0 even when it matches nothing, so when the directives were
absent (the normal case on a fresh Webmin install) the `|| echo` branch
never ran. The script printed "Webmin 2FA configuration completed" while
leaving 2FA entirely unconfigured — caught by 2fa-validation reporting
"Webmin TOTP provider not configured".
Guard each directive with grep so it is appended when absent and updated
when present.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three tests produced false failures when run on the deployed host:
- safe-download: the read-only-location assertion expects a write to
fail, but the suite runs as root and root bypasses filesystem
permissions, so the write succeeded. Skip that assertion as root.
- 2fa-validation: package presence used `dpkg -l | grep`, whose
fixed-width output wraps long names when COLUMNS is narrow (as in a
non-interactive shell), falsely reporting libpam-google-authenticator
and qrencode as missing even though they were installed. Use dpkg -s.
- https-enforcement: SSL validation passed --cert-status, which requires
OCSP stapling that many valid CDNs do not provide, flagging valid
certificates as invalid. Drop it; --ssl-reqd still enforces TLS and
certificate-chain verification.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three bugs prevented the validation suite from running cleanly:
- run-tests.sh used `((TESTS_PASSED++))` under `set -e`. Post-increment
evaluates to the old value, so the first passing test (0 -> 1) made
`(( ))` return 1 and errexit aborted the whole run after exactly one
test. Use plain arithmetic assignment instead.
- https-enforcement.sh's comment filter ran `grep -n` (which prefixes
"linenum:") and then tried to drop comment lines with
`^[[:space:]]*#`, which never matched the line-number prefix. Every
http:// URL in a comment (deprecated curl lines, the strict-mode
attribution comment) was flagged as a violation. Match the prefix.
- 2fa-validation.sh hardcoded `/home/$user/` for the setup-instructions
check, so for root it looked in /home/root (which does not exist)
instead of /root. Resolve the home directory with getent.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-2fa enforces AuthenticationMethods publickey,keyboard-
interactive, so once setup completes no non-interactive SSH client can
authenticate (a TOTP token is required). The harness's post-deploy steps
— log fetch, repo path resolution, and the validation suite — all relied
on SSH and therefore failed after the first successful deploy, masking
the fact that setup itself had completed (rc=0).
- remote.sh: add a vm-guest mode that runs commands as root inside the
VM via the Proxmox qemu-guest-agent (qm guest exec), bypassing SSH/2FA
entirely. Output is parsed on the Proxmox host with python3.
- vm-validation.sh: resolve repo path, fetch the setup log, and run the
validation suite via vm-guest when SSH is unavailable. Detect the
setup exit marker from the always-available live stream as a fallback
to the fetched log. Make restore_vm_access 2FA-aware so a post-deploy
SSH failure is understood (not a hard error) once 2FA is in effect.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_ssh_2fa only enabled the deprecated ChallengeResponseAuthentication
directive (removed as a usable knob in modern OpenSSH; it no longer controls
keyboard-interactive). The base tsys-sshd-config ships
KbdInteractiveAuthentication no, so on Debian 13 (OpenSSH 9.x/10.x)
keyboard-interactive stayed disabled. With AuthenticationMethods set to
"publickey,keyboard-interactive", sshd -t then failed:
Disabled method "keyboard-interactive" in AuthenticationMethods list ...
AuthenticationMethods cannot be satisfied by enabled authentication methods
which aborted provisioning under errexit.
Add the modern KbdInteractiveAuthentication yes directive alongside the
legacy one so 2FA works on both current and older OpenSSH.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
29 files under ProjectCode/ConfigFiles lacked a trailing newline. They
deploy via `cat file > target`, and several targets are subsequently
appended to (notably /etc/ssh/sshd_config, which configure_ssh_2fa
appends `AuthenticationMethods publickey,keyboard-interactive` to).
Without a trailing newline the append fused onto the last line,
producing `LoginGraceTime 60AuthenticationMethods ...`, which sshd -t
rejected as an invalid time value and aborted provisioning under errexit.
This is the same defect class that already broke the managed
authorized_keys files. Add the trailing newline to every config file
that was missing one so the cat-then-append pattern is always safe.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
setup_user_2fa wrote each user's 2FA-setup instructions to the quoted
path "~$user/2fa-setup-instructions.txt". Tilde expansion does not occur
inside double quotes, so the path was treated literally and the write
failed with "No such file or directory", aborting the whole 2FA module
(and thus provisioning) under errexit.
Resolve the home directory explicitly with `getent passwd` and use that
absolute path for both the instructions file and the chown. Skip the
user cleanly if no home directory exists.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-ssh intentionally replaces authorized_keys with the managed
production key set, which locks out the bootstrap/dev key the validation
harness uses to drive the VM. After the first deploy that reaches SSH
hardening, the harness could no longer connect to fetch logs or run the
test suite, breaking the iteration loop.
Add restore_vm_access(): after each deploy, if SSH is unreachable, it
re-injects the validation pubkey OUT OF BAND via the Proxmox guest agent
(qm guest exec runs as root inside the VM and does not depend on SSH).
The injected payload is prefixed with a newline to avoid key
concatenation when the managed file lacks a trailing newline.
Config: ACCESS_PUBKEY (default ~/.ssh/id_ed25519.pub), RESTORE_ACCESS=1.
Disable with RESTORE_ACCESS=0.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-wazuh.sh did a hard `systemctl start wazuh-agent`. The agent
attempts to reach its manager (tsys-nsm.knel.net) during startup; when
that host is unreachable (e.g. an isolated lab/sandbox VM, or the SIEM
being temporarily down during a fresh build), systemd's start exceeds
its timeout and the whole provisioning aborts under the framework's
errexit.
The agent is already installed and enabled, so it will keep retrying the
manager on its own. Make the start non-fatal with `|| true` so a host can
finish building even when the manager isn't reachable at deploy time.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Both managed authorized_keys files lacked a trailing newline. sshd reads
the final key fine on its own, but any subsequent append (tooling, a
follow-up key, or the validation harness) concatenated onto the last key
line, fusing two keys into one unparseable blob and silently breaking
public-key auth for both.
Add the trailing newline so the files concatenate safely.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The KexAlgorithms line in ssh-audit-hardening.conf began with a space.
In sshd_config a leading whitespace marks a line continuation, so the
entire directive was absorbed as arguments to the (non-existent)
previous directive. The effective kexalgorithms collapsed to only the
two trailing GSSAPI entries (gss-curve25519-sha256-, gss-group16-sha512-),
which no normal OpenSSH client can negotiate.
Result: after secharden-ssh deployed this file, every SSH connection to
the host died in [preauth] with no usable key exchange algorithm. sshd -t
still returned 0, so the breakage was completely silent.
Drop the leading space so the directive is parsed as intended. This
restores normal client compatibility while keeping the hardened
algorithm set.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The `apt-get --purge autoremove` after the package removal pass had no
`-y`, so once removing modemmanager/wpasupplicant orphaned eleven
dependent packages, autoremove printed its "[Y/n] Abort." confirmation
and exited 1, aborting the run under errexit.
Add `-y` so the now-orphaned dependencies are purged without prompting,
consistent with the surrounding non-interactive apt invocations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
global-installPackages installed latencytop and cockpit-tests on every
non-Kali host, but both have been dropped from Debian trixie (latencytop
is dead upstream; cockpit-tests has no candidate). Under the framework's
errexit the failing apt-get aborted the entire run with rc=100 right
after the core package install completed.
These are optional monitoring/test extras, not core requirements, so
make the install best-effort with `|| true` to match the script's
existing treatment of non-critical operations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an end-to-end validation loop for the sandbox VM and the single
SSH/SCP chokepoint it depends on:
- Project-Tests/remote.sh: the only place ssh/scp is invoked. Provides
prox / vm / vmroot / *-file / *-copy modes. Centralizes host/user/key
config and keeps remote access auditable and reusable.
- Project-Tests/vm-validation.sh: drives a Proxmox VM through
snapshot -> deploy -> validate with one-command rollback. Deployment is
GIT-BASED: the VM clones/pulls the public repo itself, exactly as a
fresh server would, so results are identical regardless of who runs it
(no reliance on a local working copy or rsync). Resolves the absolute
repo path on the VM before sudo to avoid the '~' -> root's home trap.
- logs/.gitignore: ignore generated validation/test logs (was a no-op
`!.gitignore` with no ignore rule; logs would have been committed).
Also fixes a `help`-branch typo (`${BASH_SOURCE[0]}`) and adds
PROX_USER (defaults to root) since the bare Proxmox hostname defaulted
to the wrong SSH user.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The framework sources ErrorHandling.sh which enables `set -o errexit`
globally, yet PrettyPrint's print_info/print_error invoked `tput bold`
and `tput sgr0` with no protection. In any TERM-less context (SSH
automation, CI, cron) `tput` fails with "unknown terminal" and, under
errexit, aborts the entire script on the very first status message.
Suppress tput stderr and add `|| true` so the color helpers degrade
gracefully to plain output instead of crashing every consumer script.
This is an internal framework consistency fix: strict mode + unguarded
external command were mutually incompatible.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Make AGENTS.md actionable for future agents by recording what was learned
while fixing the provisioning scripts:
- Add a Repository Layout section: the KNELShellFramework is vendored
under vendor/.../KNELShellFramework (not at repo root), scripts must
self-locate via BASH_SOURCE, configs are read locally (no CDN), and
some .sh agents are actually PHP
- Replace the vague "commit immediately" note with an explicit
Autonomous Git Workflow section authorizing agents to commit AND push
without being asked, grouped into coherent atomic commits
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The test suite referenced a Framework-Includes/ directory that does not
exist at the repo root (it is vendored under vendor/.../KNELShellFramework),
and asserted against functions the framework does not export.
- Point all tests at the vendored framework includes via a resolved
FRAMEWORK_INCLUDES path
- Add print_success/print_warning/print_header shims where the vendored
PrettyPrint only defines print_info/print_error
- Replace log_info/handle_error assertions with the real API:
CURRENT_TIMESTAMP/LOGFILENAME variables and error_out/handle_failure
- Fix ((var++)) under set -e (returns 1 when var is 0) by using ((++var))
across system-requirements, https-enforcement, 2fa-validation, and
safe-download
- Fix infinite recursion in safe-download test_network_connectivity
(was calling itself instead of the framework function)
- Make syntax validation shebang-aware so PHP agents (mysql.sh) are
skipped instead of flagged as bash syntax errors
Framework unit test now passes (exit 0).
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
All provisioning scripts and modules now resolve their own location via
BASH_SOURCE and derive PROJECT_ROOT_PATH from it, removing a hard
dependency on the current working directory.
- Derive PROJECT_ROOT_PATH/CONFIGFILES_PATH/MODULES_PATH/SCRIPTS_PATH
from BASH_SOURCE in SetupNewSystem.sh and every module
- Replace all curl ${DL_ROOT}/... downloads with cat of the matching
local files under ProjectCode/ConfigFiles (the dl.knownelement.com CDN
is no longer required for a git clone)
- Invoke modules by absolute path instead of cd ./Modules/X && bash ./x
- Fix secharden-audit-agents.sh: wrong path depth (../../ vs ../../..),
wrong Project-Includes glob, and ConfigFiles/AudidD -> AuditD typo,
all of which previously crashed the script
- Remove duplicate FrameworkVars source lines
Run from anywhere with: sudo bash ProjectCode/SetupNewSystem.sh
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add agent guidelines for AI assistants working on this repository:
- Document atomic commit requirements
- Specify conventional commit format with examples
- Require verbose, formatted commit messages
- Emphasize immediate commit/push behavior
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles
Charles N Wyble