Commit Graph
100 Commits
Author SHA1 Message Date
mrcharles 6dc8c51580 docs: organize docs into project subdirectories
Split docs/ into project-based subdirectories:
- docs/proxmox/     fleet ops, hardware, k8s (PROJECT.md, TODO.md, K8S.md)
- docs/server-build/ provisioning, security, DNS (SECURITY.md, tailscale.md,
                    DEPLOYMENT.md, TSYS-2FA-GUIDE.md, DEVELOPMENT-GUIDELINES.md)
- docs/archive/     historical AI reviews, completed todos, pre-refactor docs

docmap.md rewritten with new paths. All cross-references in AGENTS.md,
README.md, STATUS.md, and dns-cluster-setup/README.md updated.

Code directories unchanged — scripts stay where BASH_SOURCE expects them.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:34:23 -05:00
mrcharles 3d5b6c859e docs: add STATUS.md, docmap.md, encode gardening protocol
Restructure top-level to exactly three .md files:
- AGENTS.md (agent operating instructions)
- README.md (project overview, links to status + docmap)
- STATUS.md (living project status, agent-maintained, human read-only)

Add docs/docmap.md as the single documentation index/map. All docs are
categorized (active, operational guides, historical) with last-reviewed
dates. Includes the agent gardening protocol requiring agents to update
STATUS.md and docmap.md after every work session.

Rewrite AGENTS.md to be lean: points to docmap.md for doc discovery,
encodes the automatic gardening protocol (keep docs/code in sync, grep
for stale paths after renames, update STATUS.md after infrastructure
changes). All references are Gitea-renderable relative links.

Simplify README.md: header links to STATUS.md + docmap.md + AGENTS.md,
doc table replaced with pointer to docmap.md.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:32:03 -05:00
mrcharles 37e59ca310 docs: end-to-end gardening — links, stale refs, tailscale.md update
Comprehensive documentation gardening across the merged repo:

- tailscale.md: fully rewritten with current ground truth. The netinfra
  pair now runs production Technitium with all knel.net records
  replicated. Both LAN IPs resolve knel.net device names and recurse
  externally. The old "NXDOMAIN / zone is stale" findings are replaced
  with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
  to all key scripts and docs. Autonomous commit/push policy
  prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
  updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
  "PFVCluster" in actionable docs; historical AI-review docs tagged
  with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
  docs/SECURITY.md: path references fixed to current structure.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:28:52 -05:00
mrcharles 8e5b9558fe docs: unified README and AGENTS.md for merged repo
Replace the KNELServerBuild README with a unified PFVCluster README
covering both provisioning and cluster ops. Update AGENTS.md to document
the merged repo layout, key scripts, and project context. Consolidate
all documentation under docs/.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:25:16 -05:00
mrcharles 66e7843f27 refactor: reorganize merged repo into clean directory structure
Reorganize the merged KNELServerBuild + PFVCluster repo:

  provisioning/    server provisioning (was ProjectCode/ +
                   Project-Includes/ + Project-ConfigFiles/)
  tests/           test suite (was Project-Tests/)
  perf/            Proxmox perf scripts (was top-level *.sh + scripts/)
  docs/            all documentation (was ProjectDocs/ + PROJECT.md +
                   K8S.md + TODO.md)
  dns-cluster-setup/  Technitium DNS cluster (unchanged)
  netinfra/        netinfra audit scripts (unchanged)
  switches/        switch configs (unchanged)
  vendor/          vendored KNELShellFramework (unchanged)

Update all internal path references from old directory names
(ProjectCode/, Project-Includes/, Project-Tests/) to the new ones
(provisioning/, tests/) across all scripts.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:24:39 -05:00
mrcharles c14b48f39e merge: combine KNELServerBuild into PFVCluster
Merge the KNELServerBuild repository (server provisioning, security
hardening, DNS cluster setup, test suite) into PFVCluster (Proxmox
cluster ops, performance tuning, fleet audit). Both histories are
preserved via --allow-unrelated-histories.

The two repos had no source-file collisions; only AGENTS.md and
.gitignore conflicted (both resolved by merging content from both).

Directory reorganization and doc gardening will follow in subsequent
commits.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:23:22 -05:00
mrcharles b1088e8487 feat(dns-cluster): replicate Technitium production to netinfra pair
Set up a fully scripted, documented Technitium DNS cluster that
replicates the production instance from tailscale-router to
pfv-netinfra-01 (primary) and pfv-netinfra-02 (secondary).

What it does:
- EXPORT: reads the production Technitium config (auth.config with
  users + 2FA, dns.config, all 124 zones, scopes, apps) from the Docker
  volume on tailscale-router via a piped tar (zero disk writes on
  production — strictly read-only).
- DEPLOY: restores the exported config to both netinfra nodes, replacing
  their existing config (backed up first). Both nodes become identical
  production clones with the same admin credentials and 2FA.
- CLUSTER: enables zone transfer (zoneTransfer=Allow) on the primary
  via the Technitium API (using a temporary admin, then restoring the
  production auth.config). Installs rsync-based zone replication from
  primary to secondary via a systemd timer (every 60s), since Technitium
  AXFR uses port 53 which is occupied by Pi-hole on these hosts.
- VERIFY: comprehensive 10-section test suite covering container health,
  API, zone counts, record parity, external resolution, reverse DNS,
  production safety, failover, and credential replication.

Scripts:
- remote-dns.sh: SSH chokepoint for all DNS host access
- setup.sh: master orchestrator (export → deploy → cluster → verify)
- sync-zones.sh: rsync-based zone replication (installed as systemd timer)
- verify.sh: 10-section verification suite

Safety:
- tailscale-router is NEVER modified (read-only export only)
- Production auth.config is backed up before any temporary admin swap
- Each node's existing config is backed up before replacement
- The export tarball is gitignored (contains production credentials)

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 08:50:14 -05:00
mrcharles 1951667f8b fix(network): remove interface restriction that broke NTP client sync
The ntp.conf hardening used `interface ignore wildcard` +
`interface listen 127.0.0.1`, which binds ntpd to loopback only. Outbound
NTP queries to the upstream servers then carried a 127.0.0.1 source
address that the servers cannot reply to, so the daemon's peers stayed
stuck in .INIT. with reach 0 — even though the servers are reachable
(verified: ntpdate -q succeeds, ntpd does not).

Replace the interface-based restriction with restrict-based hardening:
`restrict default ignore` blocks unsolicited queries from any host (so
the box never serves time to others), while explicit allow rules for the
two upstream servers and localhost let the client sync normally.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 06:03:21 -05:00
mrcharles 9a4961d94b docs(network): analyze Tailscale vs managed DNS conflict
Add an architecture analysis for the tension between Tailscale's
default resolv.conf management (100.100.100.100) and the managed
LAN-resolver resolv.conf (.252/.253). Documents a key finding from
live-network probing: knel.net device records only resolve via the
Tailscale MagicDNS path; querying the LAN DNS servers directly returns
NXDOMAIN because their knel.net zone is stale (SOA serial 2025-06-23).

Lays out four options (Tailscale-owned, LAN-pinned, split DNS,
Tailscale-pushes-LAN-resolvers) with pros/cons, recommends leaving DNS
to Tailscale in the short term (since wazuh/postfix/syslog depend on
knel.net names that only resolve there) and fixing the Technitium/Pi-hole
knel.net zone before pinning the LAN resolvers. Confirms the NTP
(LAN-IP) change is safe regardless. Flags that the managed-resolv.conf
change will be overwritten by Tailscale and would break knel.net
resolution if it ever sticks.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 05:31:47 -05:00
mrcharles f010fa9609 feat(network): use pfv-netinfra-01/02 as redundant DNS and NTP
Route every host built by this project through the new
pfv-netinfra-01 (192.168.3.252) / pfv-netinfra-02 (192.168.3.253)
pair for both name resolution and time, with automatic failover.

- NTP: replace the single pfv-netboot.knel.net upstream with both
  netinfra servers (iburst) so time sync survives either one failing.
- DNS: add a managed static /etc/resolv.conf (new ConfigFiles/Resolv/).
  The repo previously had no resolver configuration at all. Both servers
  are listed so glibc falls through to the secondary on failure.
- DHCP: request domain-name-servers/domain-search/ntp-servers and
  supersede them to the netinfra pair, so a DHCP renew can't silently
  revert to whatever the DHCP server advertises.
- SetupNewSystem.sh: deploy resolv.conf (robustly replacing any
  systemd-resolved/NetworkManager symlink) and add pfv-netinfra to the
  NTP-server self-exclusion guard so those boxes don't client off
  themselves.

LAN IPs are used throughout (not the knel.net hostnames) because those
hostnames resolve to Tailscale CGNAT addresses, not the LAN addresses,
and NTP must come up before DNS. Add a validation test asserting the
config is present and both servers actually answer DNS and NTP queries.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 05:23:13 -05:00
mrcharles 18d57ea4fb feat: wire Pi-hole to forward knel.net to local Technitium + document setup
Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-28 05:16:01 -05:00
mrcharles 89469ff028 feat: full re-audit of all 7 hosts with fresh ground truth
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:

Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)

Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2

Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.

Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.

Added tsys9 to deploy-check.sh host list.
2026-07-27 22:04:05 -05:00
mrcharles 48cb6842c6 docs: lock in storage philosophy and PDM migration capability
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
  running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)

Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.

Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 21:46:48 -05:00
mrcharles 6814374bac fix: keep UCS on spinning disk, repurpose D3 SSD for k8s/CI workloads
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
  redundancy only, not for speed

D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.

Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:51:30 -05:00
mrcharles 019ab468bf feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday
Deep-dive on tsys4/5 storage controllers revealed:
- tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0
- tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0

Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB
bottleneck and consolidates all fast storage on tsys5: NVMe (local) +
D3 SSD + T5-SSD + 4 HDD exports.

Updated storage architecture, speed tiers, critical VM placement,
failure survival matrix, StorageClass design, and Friday plan.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:50:16 -05:00
mrcharles ea2048dc87 fix: correct storage bus types after deep-dive on tsys6/7
Verified via lsblk TRAN column and dmesg that tsys6/7 local-lvm is
backed by WD My Passport portable HDDs on USB 2.0 (~30 MB/s), not
internal SATA/SAS as previously assumed. Both R610 and R620 only have
EHCI controllers -- no USB 3.0/xHCI. tsys7 has 5 empty SATA ports but
no drives installed; tsys6's SAS controller is disabled in BIOS.

Corrected all wnode recommendations: tsys6/7 stay on NFS (faster than
their USB 2.0 local-lvm). Only tsys3/5/9/2 have local storage suitable
for VM boot disks.

Also discovered tsys3 boots from Samsung PM961 NVMe 512GB (previously
documented as HDD). Its 349 GB local-lvm is the fastest existing local
storage in the fleet.

Updated speed tier table, wnode placement tables, storage distribution
matrices, migration plan, and caching strategy across PROJECT.md and
K8S.md.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:45:06 -05:00
mrcharles db575080c9 docs: integrate tsys2 hardware findings from Windows inventory
tsys2 validated via PowerShell on Windows: 2 SSDs (Samsung 960 PRO
NVMe 512GB + Samsung 850 EVO SATA 1TB) -- the best local storage in
the fleet. wnode-tsys2 needs no NFS; boot on NVMe, bulk data on SATA
SSD. Both NICs are USB dongles (ASIX + Realtek) -- unavoidable on
this laptop form factor, no PCIe NIC expansion possible.

Updated storage tiers, wnode placement, and k8s labeling to reflect
that tsys2 is the fastest-storage worker node. Marked TODO.md items
as resolved.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:39:59 -05:00
mrcharles 11ae08f7e6 docs: add TODO.md with tsys2 Windows inventory commands
Adds the PowerShell and cmd.exe commands the user needs to run on the
Windows host (pfv-tsys2) to gather disk, network, RAM, GPU, and PCIe
details before rebuilding it as Proxmox. Also documents the Friday
maintenance window steps (tsys4 NIC+RAM, tsys5 cable+NVMe) and open
questions for the next session.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:35:33 -05:00
mrcharles a791baa085 chore: ignore python bytecode and document git workflow
Add __pycache__/*.pyc to .gitignore and remove the tracked bytecode
file. Add version control instructions to AGENTS.md for future sessions.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:31:49 -05:00
mrcharles 422999bf3c chore: initialize repo with full project state
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.

Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
  nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
  hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
  storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
  distribution, StorageClass design, and ETL/HPC workload planning

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:31:29 -05:00
mrcharles 4201f3e669 chore: ignore framework LOGFILENAME timestamp artifacts
The framework defines LOGFILENAME as "$0.<Weekday>-YYYY-MM-DD-HH:MM:SS.$$"
and PrettyPrint appends every print_info/print_error line to it, so
executing any script that sources the framework leaves a timestamped log
file beside it (e.g. run-tests.sh.Monday-2026-07-27-10:44:31.123). These
are runtime artifacts, not source, and were showing up as untracked
noise. Ignore them across the whole repo.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 11:01:15 -05:00
mrcharles 65b972e623 fix(security): actually set Webmin 2FA directives in miniserv.conf
configure_webmin_2fa used `sed -i ... || echo ... >>` to add
twofactor_provider and twofactor to /etc/webmin/miniserv.conf. sed
returns 0 even when it matches nothing, so when the directives were
absent (the normal case on a fresh Webmin install) the `|| echo` branch
never ran. The script printed "Webmin 2FA configuration completed" while
leaving 2FA entirely unconfigured — caught by 2fa-validation reporting
"Webmin TOTP provider not configured".

Guard each directive with grep so it is appended when absent and updated
when present.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:58:10 -05:00
mrcharles bd00b61047 fix(tests): correct root/SSL/package false failures in validation suite
Three tests produced false failures when run on the deployed host:

- safe-download: the read-only-location assertion expects a write to
  fail, but the suite runs as root and root bypasses filesystem
  permissions, so the write succeeded. Skip that assertion as root.
- 2fa-validation: package presence used `dpkg -l | grep`, whose
  fixed-width output wraps long names when COLUMNS is narrow (as in a
  non-interactive shell), falsely reporting libpam-google-authenticator
  and qrencode as missing even though they were installed. Use dpkg -s.
- https-enforcement: SSL validation passed --cert-status, which requires
  OCSP stapling that many valid CDNs do not provide, flagging valid
  certificates as invalid. Drop it; --ssl-reqd still enforces TLS and
  certificate-chain verification.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:53:33 -05:00
mrcharles 1fb1413f5b fix(tests): repair test-runner arithmetic and false-positive checks
Three bugs prevented the validation suite from running cleanly:

- run-tests.sh used `((TESTS_PASSED++))` under `set -e`. Post-increment
  evaluates to the old value, so the first passing test (0 -> 1) made
  `(( ))` return 1 and errexit aborted the whole run after exactly one
  test. Use plain arithmetic assignment instead.
- https-enforcement.sh's comment filter ran `grep -n` (which prefixes
  "linenum:") and then tried to drop comment lines with
  `^[[:space:]]*#`, which never matched the line-number prefix. Every
  http:// URL in a comment (deprecated curl lines, the strict-mode
  attribution comment) was flagged as a violation. Match the prefix.
- 2fa-validation.sh hardcoded `/home/$user/` for the setup-instructions
  check, so for root it looked in /home/root (which does not exist)
  instead of /root. Resolve the home directory with getent.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:48:50 -05:00
mrcharles a54da7a43a test(validation): route post-deploy ops through guest agent for 2FA
secharden-2fa enforces AuthenticationMethods publickey,keyboard-
interactive, so once setup completes no non-interactive SSH client can
authenticate (a TOTP token is required). The harness's post-deploy steps
— log fetch, repo path resolution, and the validation suite — all relied
on SSH and therefore failed after the first successful deploy, masking
the fact that setup itself had completed (rc=0).

- remote.sh: add a vm-guest mode that runs commands as root inside the
  VM via the Proxmox qemu-guest-agent (qm guest exec), bypassing SSH/2FA
  entirely. Output is parsed on the Proxmox host with python3.
- vm-validation.sh: resolve repo path, fetch the setup log, and run the
  validation suite via vm-guest when SSH is unavailable. Detect the
  setup exit marker from the always-available live stream as a fallback
  to the fetched log. Make restore_vm_access 2FA-aware so a post-deploy
  SSH failure is understood (not a hard error) once 2FA is in effect.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:42:45 -05:00
mrcharles 40dfda47f2 fix(security): enable KbdInteractiveAuthentication for SSH 2FA
configure_ssh_2fa only enabled the deprecated ChallengeResponseAuthentication
directive (removed as a usable knob in modern OpenSSH; it no longer controls
keyboard-interactive). The base tsys-sshd-config ships
KbdInteractiveAuthentication no, so on Debian 13 (OpenSSH 9.x/10.x)
keyboard-interactive stayed disabled. With AuthenticationMethods set to
"publickey,keyboard-interactive", sshd -t then failed:

  Disabled method "keyboard-interactive" in AuthenticationMethods list ...
  AuthenticationMethods cannot be satisfied by enabled authentication methods

which aborted provisioning under errexit.

Add the modern KbdInteractiveAuthentication yes directive alongside the
legacy one so 2FA works on both current and older OpenSSH.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:37:18 -05:00
mrcharles 21cc6ee54c fix(config): newline-terminate all deployed config files
29 files under ProjectCode/ConfigFiles lacked a trailing newline. They
deploy via `cat file > target`, and several targets are subsequently
appended to (notably /etc/ssh/sshd_config, which configure_ssh_2fa
appends `AuthenticationMethods publickey,keyboard-interactive` to).
Without a trailing newline the append fused onto the last line,
producing `LoginGraceTime 60AuthenticationMethods ...`, which sshd -t
rejected as an invalid time value and aborted provisioning under errexit.

This is the same defect class that already broke the managed
authorized_keys files. Add the trailing newline to every config file
that was missing one so the cat-then-append pattern is always safe.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:35:17 -05:00
mrcharles b19bc87361 fix(security): resolve user home dir for 2FA setup instructions
setup_user_2fa wrote each user's 2FA-setup instructions to the quoted
path "~$user/2fa-setup-instructions.txt". Tilde expansion does not occur
inside double quotes, so the path was treated literally and the write
failed with "No such file or directory", aborting the whole 2FA module
(and thus provisioning) under errexit.

Resolve the home directory explicitly with `getent passwd` and use that
absolute path for both the instructions file and the chown. Skip the
user cleanly if no home directory exists.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:16:48 -05:00
mrcharles 6d77775bd6 test(validation): preserve sandbox access across SSH hardening
secharden-ssh intentionally replaces authorized_keys with the managed
production key set, which locks out the bootstrap/dev key the validation
harness uses to drive the VM. After the first deploy that reaches SSH
hardening, the harness could no longer connect to fetch logs or run the
test suite, breaking the iteration loop.

Add restore_vm_access(): after each deploy, if SSH is unreachable, it
re-injects the validation pubkey OUT OF BAND via the Proxmox guest agent
(qm guest exec runs as root inside the VM and does not depend on SSH).
The injected payload is prefixed with a newline to avoid key
concatenation when the managed file lacks a trailing newline.

Config: ACCESS_PUBKEY (default ~/.ssh/id_ed25519.pub), RESTORE_ACCESS=1.
Disable with RESTORE_ACCESS=0.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:13:04 -05:00
mrcharles 5f26f7dca1 fix(provisioning): make wazuh-agent start best-effort
secharden-wazuh.sh did a hard `systemctl start wazuh-agent`. The agent
attempts to reach its manager (tsys-nsm.knel.net) during startup; when
that host is unreachable (e.g. an isolated lab/sandbox VM, or the SIEM
being temporarily down during a fresh build), systemd's start exceeds
its timeout and the whole provisioning aborts under the framework's
errexit.

The agent is already installed and enabled, so it will keep retrying the
manager on its own. Make the start non-fatal with `|| true` so a host can
finish building even when the manager isn't reachable at deploy time.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:13:04 -05:00
mrcharles 53d953e092 fix(security): newline-terminate managed authorized_keys files
Both managed authorized_keys files lacked a trailing newline. sshd reads
the final key fine on its own, but any subsequent append (tooling, a
follow-up key, or the validation harness) concatenated onto the last key
line, fusing two keys into one unparseable blob and silently breaking
public-key auth for both.

Add the trailing newline so the files concatenate safely.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:12:55 -05:00
mrcharles 163ef9de16 fix(security): repair KexAlgorithms leading-space in sshd hardening
The KexAlgorithms line in ssh-audit-hardening.conf began with a space.
In sshd_config a leading whitespace marks a line continuation, so the
entire directive was absorbed as arguments to the (non-existent)
previous directive. The effective kexalgorithms collapsed to only the
two trailing GSSAPI entries (gss-curve25519-sha256-, gss-group16-sha512-),
which no normal OpenSSH client can negotiate.

Result: after secharden-ssh deployed this file, every SSH connection to
the host died in [preauth] with no usable key exchange algorithm. sshd -t
still returned 0, so the breakage was completely silent.

Drop the leading space so the directive is parsed as intended. This
restores normal client compatibility while keeping the hardened
algorithm set.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 10:12:38 -05:00
mrcharles a4cdd2ee30 fix(provisioning): make post-purge autoremove non-interactive
The `apt-get --purge autoremove` after the package removal pass had no
`-y`, so once removing modemmanager/wpasupplicant orphaned eleven
dependent packages, autoremove printed its "[Y/n] Abort." confirmation
and exited 1, aborting the run under errexit.

Add `-y` so the now-orphaned dependencies are purged without prompting,
consistent with the surrounding non-interactive apt invocations.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 09:58:28 -05:00
mrcharles 550d2cd078 fix(provisioning): make latencytop/cockpit-tests install best-effort
global-installPackages installed latencytop and cockpit-tests on every
non-Kali host, but both have been dropped from Debian trixie (latencytop
is dead upstream; cockpit-tests has no candidate). Under the framework's
errexit the failing apt-get aborted the entire run with rc=100 right
after the core package install completed.

These are optional monitoring/test extras, not core requirements, so
make the install best-effort with `|| true` to match the script's
existing treatment of non-critical operations.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 09:56:25 -05:00
mrcharles 83e4d7e8ce test(validation): add reproducible git-based VM validation harness
Add an end-to-end validation loop for the sandbox VM and the single
SSH/SCP chokepoint it depends on:

- Project-Tests/remote.sh: the only place ssh/scp is invoked. Provides
  prox / vm / vmroot / *-file / *-copy modes. Centralizes host/user/key
  config and keeps remote access auditable and reusable.
- Project-Tests/vm-validation.sh: drives a Proxmox VM through
  snapshot -> deploy -> validate with one-command rollback. Deployment is
  GIT-BASED: the VM clones/pulls the public repo itself, exactly as a
  fresh server would, so results are identical regardless of who runs it
  (no reliance on a local working copy or rsync). Resolves the absolute
  repo path on the VM before sudo to avoid the '~' -> root's home trap.
- logs/.gitignore: ignore generated validation/test logs (was a no-op
  `!.gitignore` with no ignore rule; logs would have been committed).

Also fixes a `help`-branch typo (`${BASH_SOURCE[0]}`) and adds
PROX_USER (defaults to root) since the bare Proxmox hostname defaulted
to the wrong SSH user.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 09:32:14 -05:00
mrcharles 377c83bcf1 fix(framework): guard tput calls against missing TERM
The framework sources ErrorHandling.sh which enables `set -o errexit`
globally, yet PrettyPrint's print_info/print_error invoked `tput bold`
and `tput sgr0` with no protection. In any TERM-less context (SSH
automation, CI, cron) `tput` fails with "unknown terminal" and, under
errexit, aborts the entire script on the very first status message.

Suppress tput stderr and add `|| true` so the color helpers degrade
gracefully to plain output instead of crashing every consumer script.
This is an internal framework consistency fix: strict mode + unguarded
external command were mutually incompatible.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-27 09:32:04 -05:00
mrcharles 453f3a22ac a few minor annoyances now addressed 2025-12-29 14:17:08 -05:00
mrcharles f283f8cfb8 a bit of refactoring 2025-12-28 18:49:30 -05:00
mrcharles d33c8df277 AI project review 2025-12-28 18:46:53 -05:00
mrcharles 2930eeaf27 ssh pub key regression, need to use cat instead of curl 2025-07-29 13:54:39 -05:00
mrcharles 870540840c use the tailscale installer 2025-07-29 13:45:21 -05:00
mrcharles 5e2eaff55d typo 2025-07-29 13:42:14 -05:00
mrcharles 8f19c9fb6e kali corner case... 2025-07-29 13:32:53 -05:00
mrcharles 40ab4608e2 some minor ubuntu default cleanup 2025-07-17 23:04:40 -05:00
mrcharles 47ddb93fef Adding ansible-core to be able to run compliance as code playbooks 2025-07-16 09:37:11 -05:00
mrcharles e73b81e229 . 2025-07-14 13:08:05 -05:00
mrcharles 39e37d0f76 . 2025-07-14 13:04:31 -05:00
mrcharles 31e66864ad . 2025-07-14 13:02:42 -05:00
mrcharles 0006eefcf1 . 2025-07-14 12:58:25 -05:00
mrcharles abfaf765e6 . 2025-07-14 12:55:48 -05:00
mrcharles 1f2bd31380 . 2025-07-14 12:53:41 -05:00
mrcharles 93cea874a8 . 2025-07-14 12:50:48 -05:00
mrcharles a898ebc59d . 2025-07-14 12:49:26 -05:00
mrcharles 78cc8cbcf3 . 2025-07-14 12:47:40 -05:00
mrcharles 495d0bb03b . 2025-07-14 12:46:53 -05:00
mrcharles 7a7d23f36c . 2025-07-14 12:42:22 -05:00
mrcharles 84f3ca3b0e . 2025-07-14 12:38:07 -05:00
mrcharles f9f32612bb . 2025-07-14 12:37:04 -05:00
mrcharles 09063bfee4 case matters... 2025-07-14 12:36:03 -05:00
mrcharles 5bbaff89e9 refactored to use vendored shell framework. lets test. 2025-07-14 12:34:33 -05:00
mrcharles 5a8561ea84 Update "KnelShell" from "ssh://git@git.knownelement.com:29418/KNEL/KNELShellFramework.git@main"
git-vendor-name: KnelShell
git-vendor-dir: vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework
git-vendor-repository: ssh://git@git.knownelement.com:29418/KNEL/KNELShellFramework.git
git-vendor-ref: main
2025-07-14 12:18:27 -05:00
mrcharles 2fa32a5eb7 Squashed 'vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/' changes from 5ecde81..1fb5a06
1fb5a06 Added SafeDownload and added shebang to DebugMe

git-subtree-dir: vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework
git-subtree-split: 1fb5a061aecd61df406e5ebcdb010097f1ccbc69
2025-07-14 12:18:27 -05:00
mrcharles 83d5cf2f8d moved docs
Switching to using vendored shell framework
moved SafeDownload to vendored shell framework repo
2025-07-14 12:17:29 -05:00
mrcharles 49e57ff846 Squashed 'vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework/' content from commit 5ecde81
git-subtree-dir: vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework
git-subtree-split: 5ecde81ce441d5802fd7e7e91a441e34f327f457
2025-07-14 12:11:42 -05:00
mrcharles 47b5a976c2 Add "KnelShell" from "ssh://git@git.knownelement.com:29418/KNEL/KNELShellFramework.git@main"
git-vendor-name: KnelShell
git-vendor-dir: vendor/git@git.knownelement.com/29418/KNEL/KNELShellFramework
git-vendor-repository: ssh://git@git.knownelement.com:29418/KNEL/KNELShellFramework.git
git-vendor-ref: main
2025-07-14 12:11:42 -05:00
mrcharles a710fc7b4e removed debugging bits 2025-07-14 11:04:21 -05:00
mrcharles c6e458de8b . 2025-07-14 11:03:08 -05:00
mrcharles e31bab4162 . 2025-07-14 11:01:19 -05:00
mrcharles 86740b8c7d . 2025-07-14 10:59:32 -05:00
mrcharles f585f90b7f . 2025-07-14 10:55:54 -05:00
mrcharles 24c10b6f35 it hallucinated print_header 2025-07-14 10:50:42 -05:00
mrcharles 634a998d7e testing 2025-07-14 10:48:59 -05:00
mrcharles e3685f68ad forgot to call the function 2025-07-14 10:33:04 -05:00
mrcharles ac857c91c3 actually run the 2fa script. 2025-07-14 10:31:22 -05:00
mrcharlesandClaude a632e7d514 Implement comprehensive two-factor authentication for SSH and web services
- Complete rewrite of secharden-2fa.sh with full 2FA implementation
- SSH 2FA using Google Authenticator with publickey + TOTP authentication
- Cockpit web interface 2FA with custom PAM configuration
- Webmin 2FA support with automatic detection and configuration
- User setup automation with QR codes and backup codes generation
- Gradual rollout support using nullok for phased deployment
- Automatic configuration backup and restore procedures
- Add 2fa-validation.sh security test for comprehensive validation
- Create TSYS-2FA-GUIDE.md with complete implementation documentation
- Add DEVELOPMENT-GUIDELINES.md with coding standards and best practices
- Optimize package installation with single apt-get commands for performance

The 2FA implementation provides enterprise-grade security while maintaining
usability and proper emergency access procedures. Includes comprehensive
testing, documentation, and follows established security best practices.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-14 10:23:07 -05:00
mrcharlesandClaude f6acf660f6 Implement comprehensive testing framework and enhance documentation
- Add Project-Tests directory with complete testing infrastructure
- Create main test runner with JSON reporting and categorized tests
- Implement system validation tests (RAM, disk, network, permissions)
- Add security testing for HTTPS enforcement and deployment methods
- Create unit tests for framework functions and syntax validation
- Add ConfigValidation.sh framework for pre-flight system checks
- Enhance documentation with SECURITY.md and DEPLOYMENT.md guides
- Provide comprehensive testing README with usage instructions

The testing framework validates system compatibility, security configurations,
and deployment requirements before execution, preventing deployment failures
and providing clear error reporting for troubleshooting.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-14 09:35:27 -05:00
mrcharlesandClaude 0c736c7295 Enforce HTTPS for all downloads to eliminate security vulnerabilities
- Convert 16 HTTP URLs to HTTPS across 3 critical scripts
- Dell OMSA script: Ubuntu archive and Dell repository URLs now use HTTPS
- Proxmox legacy script: Download URLs converted to secure connections
- SSL stack script: Apache source URLs updated to official archive
- Update documentation to reflect resolved security issues
- Mark HTTPS enforcement as completed in todo lists

This addresses the second critical security concern from the security review,
eliminating man-in-the-middle attack vectors during package downloads.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-14 09:22:32 -05:00
mrcharles 273e7fe674 Claude code review of my work. 2025-07-12 00:17:52 -05:00
mrcharles 6609d7d9e3 sigh. 2025-07-11 11:52:28 -05:00
mrcharles 0588b2dd60 ifdev for dev boxes, they have less hardened ssh config because vscode remote etc 2025-07-11 11:48:53 -05:00
mrcharles f399308b2d allow root to login to cockpit 2025-07-10 10:47:21 -05:00
mrcharles 45b53efe11 working on v1.1, secrets management/bootstrap 2025-07-10 10:28:00 -05:00
mrcharles b0d1ae0a3e . 2025-07-10 10:13:23 -05:00
mrcharles a2ff47e5d2 . 2025-07-10 10:11:50 -05:00
mrcharles b5d09e64f0 we want a bit of observability here.. 2025-07-10 10:09:58 -05:00
mrcharles edc3ca26ad . 2025-07-10 10:06:52 -05:00
mrcharles a272764d66 . 2025-07-10 10:05:51 -05:00
mrcharles 97b67ea1fc . 2025-07-10 10:04:23 -05:00
mrcharles a86b2ea09b and agian... sigh 2025-07-10 10:03:12 -05:00
mrcharles 54cfcf669f fixed agian 2025-07-10 10:01:30 -05:00
mrcharles 28c18a2bda . 2025-07-10 10:00:25 -05:00
mrcharles 168456ee7f fixed 2025-07-10 09:59:35 -05:00
mrcharles d6364eac7a typo 2025-07-10 09:58:28 -05:00
mrcharles d2100d1146 dont' need vm management in vms.. 2025-07-10 09:56:18 -05:00
mrcharles 5c20f167b2 adding cockpit 2025-07-10 09:48:01 -05:00
mrcharles 3b705a23ba don't install rsyslog on librenms server
fixed some formatting
2025-07-09 11:24:20 -05:00
mrcharles 319cd61ad4 all the instrumentation/diagnostics... 2025-07-07 12:05:26 -05:00
mrcharles 1e458f0fae being able to use growpart is quite nice 2025-07-05 20:49:40 -05:00
mrcharles 0bf88e3d8c More ubuntu fixes 2025-07-05 17:48:41 -05:00
mrcharles bf4efcdf5a oops 2025-07-02 22:23:16 -05:00