Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
redundancy only, not for speed
D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.
Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.
Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.
Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.
Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.
Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds the PowerShell and cmd.exe commands the user needs to run on the
Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe
details before rebuilding it as Proxmox. Also documents the Friday
maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open
questions for the next session.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode
file. Add version control instructions to AGENTS.md for future sessions.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The framework defines LOGFILENAME as "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$"
and PrettyPrint appends every print_info/print_error line to it, so
executing any script that sources the framework leaves a timestamped log
file beside it (e.g. run-tests.sh.Monday-2026-07-27-10:44:31.123). These
are runtime artifacts, not source, and were showing up as untracked
noise. Ignore them across the whole repo.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_webmin_2fa used `sed -i ... || echo ... >>` to add
twofactor_provider and twofactor to /etc/webmin/miniserv.conf. sed
returns 0 even when it matches nothing, so when the directives were
absent (the normal case on a fresh Webmin install) the `|| echo` branch
never ran. The script printed "Webmin 2FA configuration completed" while
leaving 2FA entirely unconfigured — caught by 2fa-validation reporting
"Webmin TOTP provider not configured".
Guard each directive with grep so it is appended when absent and updated
when present.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three tests produced false failures when run on the deployed host:
- safe-download: the read-only-location assertion expects a write to
fail, but the suite runs as root and root bypasses filesystem
permissions, so the write succeeded. Skip that assertion as root.
- 2fa-validation: package presence used `dpkg -l | grep`, whose
fixed-width output wraps long names when COLUMNS is narrow (as in a
non-interactive shell), falsely reporting libpam-google-authenticator
and qrencode as missing even though they were installed. Use dpkg -s.
- https-enforcement: SSL validation passed --cert-status, which requires
OCSP stapling that many valid CDNs do not provide, flagging valid
certificates as invalid. Drop it; --ssl-reqd still enforces TLS and
certificate-chain verification.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Three bugs prevented the validation suite from running cleanly:
- run-tests.sh used `((TESTS_PASSED++))` under `set -e`. Post-increment
evaluates to the old value, so the first passing test (0 -> 1) made
`(( ))` return 1 and errexit aborted the whole run after exactly one
test. Use plain arithmetic assignment instead.
- https-enforcement.sh's comment filter ran `grep -n` (which prefixes
"linenum:") and then tried to drop comment lines with
`^[[:space:]]*#`, which never matched the line-number prefix. Every
http:// URL in a comment (deprecated curl lines, the strict-mode
attribution comment) was flagged as a violation. Match the prefix.
- 2fa-validation.sh hardcoded `/home/$user/` for the setup-instructions
check, so for root it looked in /home/root (which does not exist)
instead of /root. Resolve the home directory with getent.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-2fa enforces AuthenticationMethods publickey,keyboard-
interactive, so once setup completes no non-interactive SSH client can
authenticate (a TOTP token is required). The harness's post-deploy steps
— log fetch, repo path resolution, and the validation suite — all relied
on SSH and therefore failed after the first successful deploy, masking
the fact that setup itself had completed (rc=0).
- remote.sh: add a vm-guest mode that runs commands as root inside the
VM via the Proxmox qemu-guest-agent (qm guest exec), bypassing SSH/2FA
entirely. Output is parsed on the Proxmox host with python3.
- vm-validation.sh: resolve repo path, fetch the setup log, and run the
validation suite via vm-guest when SSH is unavailable. Detect the
setup exit marker from the always-available live stream as a fallback
to the fetched log. Make restore_vm_access 2FA-aware so a post-deploy
SSH failure is understood (not a hard error) once 2FA is in effect.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
configure_ssh_2fa only enabled the deprecated ChallengeResponseAuthentication
directive (removed as a usable knob in modern OpenSSH; it no longer controls
keyboard-interactive). The base tsys-sshd-config ships
KbdInteractiveAuthentication no, so on Debian 13 (OpenSSH 9.x/10.x)
keyboard-interactive stayed disabled. With AuthenticationMethods set to
"publickey,keyboard-interactive", sshd -t then failed:
Disabled method "keyboard-interactive" in AuthenticationMethods list ...
AuthenticationMethods cannot be satisfied by enabled authentication methods
which aborted provisioning under errexit.
Add the modern KbdInteractiveAuthentication yes directive alongside the
legacy one so 2FA works on both current and older OpenSSH.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
29 files under ProjectCode/ConfigFiles lacked a trailing newline. They
deploy via `cat file > target`, and several targets are subsequently
appended to (notably /etc/ssh/sshd_config, which configure_ssh_2fa
appends `AuthenticationMethods publickey,keyboard-interactive` to).
Without a trailing newline the append fused onto the last line,
producing `LoginGraceTime 60AuthenticationMethods ...`, which sshd -t
rejected as an invalid time value and aborted provisioning under errexit.
This is the same defect class that already broke the managed
authorized_keys files. Add the trailing newline to every config file
that was missing one so the cat-then-append pattern is always safe.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
setup_user_2fa wrote each user's 2FA-setup instructions to the quoted
path "~$user/2fa-setup-instructions.txt". Tilde expansion does not occur
inside double quotes, so the path was treated literally and the write
failed with "No such file or directory", aborting the whole 2FA module
(and thus provisioning) under errexit.
Resolve the home directory explicitly with `getent passwd` and use that
absolute path for both the instructions file and the chown. Skip the
user cleanly if no home directory exists.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-ssh intentionally replaces authorized_keys with the managed
production key set, which locks out the bootstrap/dev key the validation
harness uses to drive the VM. After the first deploy that reaches SSH
hardening, the harness could no longer connect to fetch logs or run the
test suite, breaking the iteration loop.
Add restore_vm_access(): after each deploy, if SSH is unreachable, it
re-injects the validation pubkey OUT OF BAND via the Proxmox guest agent
(qm guest exec runs as root inside the VM and does not depend on SSH).
The injected payload is prefixed with a newline to avoid key
concatenation when the managed file lacks a trailing newline.
Config: ACCESS_PUBKEY (default ~/.ssh/id_ed25519.pub), RESTORE_ACCESS=1.
Disable with RESTORE_ACCESS=0.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
secharden-wazuh.sh did a hard `systemctl start wazuh-agent`. The agent
attempts to reach its manager (tsys-nsm.knel.net) during startup; when
that host is unreachable (e.g. an isolated lab/sandbox VM, or the SIEM
being temporarily down during a fresh build), systemd's start exceeds
its timeout and the whole provisioning aborts under the framework's
errexit.
The agent is already installed and enabled, so it will keep retrying the
manager on its own. Make the start non-fatal with `|| true` so a host can
finish building even when the manager isn't reachable at deploy time.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Both managed authorized_keys files lacked a trailing newline. sshd reads
the final key fine on its own, but any subsequent append (tooling, a
follow-up key, or the validation harness) concatenated onto the last key
line, fusing two keys into one unparseable blob and silently breaking
public-key auth for both.
Add the trailing newline so the files concatenate safely.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The KexAlgorithms line in ssh-audit-hardening.conf began with a space.
In sshd_config a leading whitespace marks a line continuation, so the
entire directive was absorbed as arguments to the (non-existent)
previous directive. The effective kexalgorithms collapsed to only the
two trailing GSSAPI entries (gss-curve25519-sha256-, gss-group16-sha512-),
which no normal OpenSSH client can negotiate.
Result: after secharden-ssh deployed this file, every SSH connection to
the host died in [preauth] with no usable key exchange algorithm. sshd -t
still returned 0, so the breakage was completely silent.
Drop the leading space so the directive is parsed as intended. This
restores normal client compatibility while keeping the hardened
algorithm set.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The `apt-get --purge autoremove` after the package removal pass had no
`-y`, so once removing modemmanager/wpasupplicant orphaned eleven
dependent packages, autoremove printed its "[Y/n] Abort." confirmation
and exited 1, aborting the run under errexit.
Add `-y` so the now-orphaned dependencies are purged without prompting,
consistent with the surrounding non-interactive apt invocations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
global-installPackages installed latencytop and cockpit-tests on every
non-Kali host, but both have been dropped from Debian trixie (latencytop
is dead upstream; cockpit-tests has no candidate). Under the framework's
errexit the failing apt-get aborted the entire run with rc=100 right
after the core package install completed.
These are optional monitoring/test extras, not core requirements, so
make the install best-effort with `|| true` to match the script's
existing treatment of non-critical operations.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add an end-to-end validation loop for the sandbox VM and the single
SSH/SCP chokepoint it depends on:
- Project-Tests/remote.sh: the only place ssh/scp is invoked. Provides
prox / vm / vmroot / *-file / *-copy modes. Centralizes host/user/key
config and keeps remote access auditable and reusable.
- Project-Tests/vm-validation.sh: drives a Proxmox VM through
snapshot -> deploy -> validate with one-command rollback. Deployment is
GIT-BASED: the VM clones/pulls the public repo itself, exactly as a
fresh server would, so results are identical regardless of who runs it
(no reliance on a local working copy or rsync). Resolves the absolute
repo path on the VM before sudo to avoid the '~' -> root's home trap.
- logs/.gitignore: ignore generated validation/test logs (was a no-op
`!.gitignore` with no ignore rule; logs would have been committed).
Also fixes a `help`-branch typo (`${BASH_SOURCE[0]}`) and adds
PROX_USER (defaults to root) since the bare Proxmox hostname defaulted
to the wrong SSH user.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The framework sources ErrorHandling.sh which enables `set -o errexit`
globally, yet PrettyPrint's print_info/print_error invoked `tput bold`
and `tput sgr0` with no protection. In any TERM-less context (SSH
automation, CI, cron) `tput` fails with "unknown terminal" and, under
errexit, aborts the entire script on the very first status message.
Suppress tput stderr and add `|| true` so the color helpers degrade
gracefully to plain output instead of crashing every consumer script.
This is an internal framework consistency fix: strict mode + unguarded
external command were mutually incompatible.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Make AGENTS.md actionable for future agents by recording what was learned
while fixing the provisioning scripts:
- Add a Repository Layout section: the KNELShellFramework is vendored
under vendor/.../KNELShellFramework (not at repo root), scripts must
self-locate via BASH_SOURCE, configs are read locally (no CDN), and
some .sh agents are actually PHP
- Replace the vague "commit immediately" note with an explicit
Autonomous Git Workflow section authorizing agents to commit AND push
without being asked, grouped into coherent atomic commits
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
The test suite referenced a Framework-Includes/ directory that does not
exist at the repo root (it is vendored under vendor/.../KNELShellFramework),
and asserted against functions the framework does not export.
- Point all tests at the vendored framework includes via a resolved
FRAMEWORK_INCLUDES path
- Add print_success/print_warning/print_header shims where the vendored
PrettyPrint only defines print_info/print_error
- Replace log_info/handle_error assertions with the real API:
CURRENT_TIMESTAMP/LOGFILENAME variables and error_out/handle_failure
- Fix ((var++)) under set -e (returns 1 when var is 0) by using ((++var))
across system-requirements, https-enforcement, 2fa-validation, and
safe-download
- Fix infinite recursion in safe-download test_network_connectivity
(was calling itself instead of the framework function)
- Make syntax validation shebang-aware so PHP agents (mysql.sh) are
skipped instead of flagged as bash syntax errors
Framework unit test now passes (exit 0).
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
All provisioning scripts and modules now resolve their own location via
BASH_SOURCE and derive PROJECT_ROOT_PATH from it, removing a hard
dependency on the current working directory.
- Derive PROJECT_ROOT_PATH/CONFIGFILES_PATH/MODULES_PATH/SCRIPTS_PATH
from BASH_SOURCE in SetupNewSystem.sh and every module
- Replace all curl ${DL_ROOT}/... downloads with cat of the matching
local files under ProjectCode/ConfigFiles (the dl.knownelement.com CDN
is no longer required for a git clone)
- Invoke modules by absolute path instead of cd ./Modules/X && bash ./x
- Fix secharden-audit-agents.sh: wrong path depth (../../ vs ../../..),
wrong Project-Includes glob, and ConfigFiles/AudidD -> AuditD typo,
all of which previously crashed the script
- Remove duplicate FrameworkVars source lines
Run from anywhere with: sudo bash ProjectCode/SetupNewSystem.sh
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add agent guidelines for AI assistants working on this repository:
- Document atomic commit requirements
- Specify conventional commit format with examples
- Require verbose, formatted commit messages
- Emphasize immediate commit/push behavior
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
mrcharles
Charles N Wyble