Leftover unstaged state from the parallel 09-02 sessions: delete
questions-v1/v2, HomeAssistant/needfromcharles.md, night-grind-plan.md
(content lives in questions-09022206.md / questions-v3.md / git history);
drop superseded UniFi API-key ask (#619 multicast fix made it moot).
Details: https://projects.knownelement.com/issues/344#note-3911
The required-files check pinned questions-v1.md, which breaks every time
the Q&A file versions up (v2 was closed out by the parallel session,
leaving nothing matching). Now glob-accepts any version. questions-v3.md
restores the file; active question channel is ~/PTKR.md per founder.
Detail: https://projects.knownelement.com/issues/441 (latest notes)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
How to read and rename founder todo-list items via HA REST for the
Redmine [#NNN] sync-back workflow.
Detail: https://projects.knownelement.com/issues/441 (latest notes)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Discourse is the sole documentation space; Redmine wikis stay disabled
(verified 0/62 projects). Also repointed the stale redmine/discourse CLI
AGENTS.md references to ukrrs/connectors/KNEL-AIMiddleware.
Detail: https://projects.knownelement.com/issues/441 (latest note)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
ntp.conf captured (nodes byte-identical); 171 Technitium DZ zone files
captured from the primary; drift-check now covers dhcpd/pihole/ntp on
both nodes plus a zones md5-manifest check. Live run caught real drift:
node-02 dhcpd.conf was a stale primary copy missing the #614 minisplit
reservation — reconciled git->02 with dhcpd -t gate, serial restart,
failover "Both servers normal", all 7 checks in sync. Red-green unit
tests extended (gen_manifest, tracked-file invariants).
Results: https://projects.knownelement.com/issues/630#note-2
New "Cloudron" group (id 277); all 55 app monitors moved there
(canary-verified). Deleted IP-literal monitor id=8 (dup of DNS-named
id=210) and repointed appletv monitor to new DNS name
stl-appletv-livingroom.knel.net. Fleet-wide IP literals in Kuma: 0.
Sync tool default group updated to 277.
Results: https://projects.knownelement.com/issues/435#note-5
55 http monitors added under "Cloud Systems" (blue/green: canary first,
verified green); 9 root-level strays re-homed into founder's groups
(root now zero). New tools: inventory (read-only dump), cloudron-sync
(idempotent diff/add from committed app list), regroup (lib-based moves;
raw editMonitor times out on this build). fleet-sync now requires
--group-id so it can never place monitors at root again.
Results: https://projects.knownelement.com/issues/435#note-4
- cnodes 102/705/603 moved to TS5-SSD storage by founder (etcd apply 4-5s -> <1s)
- wnode-tsys5 slot retired; ultix-streaming joins in its place (5 workers)
- ultix-offstage unreachable; stays excluded until back on tailnet
[#367][#368] https://projects.knownelement.com/issues/368
Live proof: ecobee answers unicast mDNS with PTR only
(_hap._tcp.local -> "Main Floor._hap._tcp.local"), so the old
replace-on-probe wiped learned records every cycle. Merge by
(name, type) instead. CONFIG_PATH now env-overridable for tests.
Details: https://projects.knownelement.com/issues/619#note-5💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Archive the non-ported remainder of the legacy KNELServerBuild repo
under archive/KNELServerBuild/ with its original structure intact,
completing the legacy repo merge for everything except the live
LibreNMS patterns (ported in the previous commit).
Exclusions:
- .git history (superseded; legacy repo remains at its original path)
- ported files (Agents/librenms, Modules/OAM/oam-librenms.sh,
ConfigFiles/SNMP/snmp-sudo.conf)
- vendored KNELShellFramework tree (byte-identical duplicate of the
copy already vendored at vendor/ in this repo)
- SSH authorized-keys files (live access-control material; carrying
them in an archive invites drift — key policy lives elsewhere)
The whole tree is skip-listed in tests/shellcheck.sh (archived legacy
code, not maintained — same standing as vendor/); check-rules.sh
already prunes archive/. Rule 9 (conflict markers) now also excludes
archive/ staged files: preserved-verbatim legacy scripts contain
decorative "====" banners that false-positive as conflict markers
(same archive exclusion precedent as rule 11).
AGENTS.md: note archive/KNELServerBuild and oam/librenms-agent in the
Repository Layout, and fix the stale KNELIAC path to
/home/reachableceo/projects/KNEL/KNELIAC.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Port the live LibreNMS monitoring patterns from the legacy
KNELServerBuild repo into oam/librenms-agent/, joining the existing
OAM tooling (oxidized, unpoller, smokeping, netdisco):
- agent/ — upstream check_mk agent + snmp-extend scripts (dmi, dpkg,
mysql, ntp-client, ntp-server, os-updates, postfix, raspberry, smart,
ss, ups-nut), copied verbatim (md5-verified), never to be edited here
- setup.sh — deploy module ported from ProjectCode/Modules/OAM/
oam-librenms.sh; only the legacy framework bootstrap was replaced
with plain bash (path constants + print_info -> echo)
- snmp-sudo.conf — Debian snmpd sudo rule the extends require
(Debian-snmp NOPASSWD /bin/cat); carried as a file only, sudoers
install is a policy decision per AGENTS.md
Lint gates: extend the existing upstream-skip mechanism for the
verbatim agent scripts (tests/shellcheck.sh + check-rules.sh prune,
same precedent as archive/provisioning/Agents/librenms), and fix the
stale skip path there (provisioning/ moved to archive/provisioning in
6244c1c; the bash extends have been failing the whole-repo shellcheck
gate since).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
DNS/DHCP/NTP redundancy per founder ruling 2026-08-27:
- DHCP failover converted load-balance -> hot standby (split 255 on
primary; secondary answers only if primary unreachable >3s). DHCP
options now hand out BOTH nodes for DNS and NTP (active/active) —
Tailscale hosts stay on MagicDNS 100.100.100.100.
- netinfra-02 ntpsec deployed (pool.ntp.org, all interfaces); 01
repointed from debian pool zones to pool.ntp.org. NTP now redundant;
GPS stratum-1 on pfvsvrpi tracked as follow-up ticket.
- snmpd on both nodes with scoped ACLs (LibreNMS LAN+TS sources only)
and ntpq/dhcpd-lease extends — prep for DNS/DHCP/NTP graphing.
- Deployed sync-zones.sh (git version, DNS name instead of IP literal)
to netinfra-02; zone sync verified 171/171 zones both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The ticket gate required the single .crush/active-ticket file, so two
concurrent sessions overwrote each other's ticket mid-work. The gate now
accepts any non-empty .crush/active-ticket* file; each session keeps its
own (e.g. active-ticket-plant, active-ticket-core). Also prune .crush/
session scratch from both shellcheck scanners so one session's throwaway
probe scripts cannot block the other session's commits. Documented in
AGENTS.md Task Tracking.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
lm-sensors is silent on SBCs (Pis, Jetson-class), so the wrapper now falls
back to /sys/class/thermal zones rendered in lm-sensors format when native
sensors output is empty; suppressed when native output exists to avoid
duplicates. TDD: 3 new unit tests (12 green). Sensor stack deployed to
jetson + pfvsvrpi + 3 subopis (verified from poller), HA pack extended to
16 sensors across 14 hosts, high-temp automation covers the SBC fleet.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
SNMP sensors for all 9 instrumented hosts (temps in °F per founder
ruling, °C converted at display layer), ACPI node-wattage feeds for
tsys6/7, UPS alert automations (on-battery, battery-low, comm-lost,
power-restored, high-temp) targeting Pushover, and Riemann-sum kWh
helpers for the Energy dashboard. PDU and iDRAC blocks left as pinned
placeholders pending PDU community and #462 OMSA work.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The reservation carried a4:ba:db:0b:df:a0 (core-switch OUI, mis-migrated
from pfv-netboot) so tsys6's iDRAC could never lease its reserved address.
Replaced with the in-band-verified BMC MAC; deployed to both failover
nodes, validated, restarted.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Native lm-sensors output plus optional TEMPer USB probe, unified in
lm-sensors format behind one snmpd extend (lmsensors). Idempotent
deploy binds snmpd to explicit LAN+Tailscale addresses only and
source-scopes the community to the pollers (LibreNMS, Home Assistant
app-connector LAN IP, admin workstation, Cloudron) per the founder's
security ruling. TDD unit suite included; fleet verified live (6 hosts)
and negative-tested (refused source).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The live configs were edited in place (canonical renames, stale block
removal, 7 new reservations, 69 total) but the repo copies were never
updated — the exact drift class that bit pi-hole. Repo and live state
now match on both failover nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
EOF
Bring the framework's global baseline (BASELINE-PROMPT.md, PATTERNS.md,
ADOPTING.md) into the repo — AGENTS.md referenced them but the files were
missing. AGENTS.md gains three rules adopted this session: TDD & Linting
(mandatory at the Ansible/IaC transition), IaC codification shadow-tracking
(every manual fleet change same-day ticketed to #454), and Redmine tracker
discipline (Support not Bug — the CLI default created nine misfiled
tickets, now corrected). Q10 records the direct-push vs PR conflict for
the founder to rule on.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pi-hole on netinfra-01/02 now conditionally forwards 192.168.0.0/16
reverse lookups to this node's Technitium over dnsnet, alongside the
existing knel.net + 100.64/10 pair. Technitium stays the single source
of truth; pollers (NetDisco, phpIPAM, UNPoller, Wazuh soon) resolving
via either node's :53 now get LAN PTRs. Both live revServer arrays and
this compose file are in sync; validated with forward, PTR (192.168 +
100.x), and external lookups against both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The full legacy suite mixes repo tests with deployment-target tests
(PAM packages, target resolv.conf, Proxmox repo reachability) that can
only pass on a provisioned server. The workstation gate now runs the
unit suite; validation/security stay invocable for sectestbed runs.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fill the framework gaps the Makefile already referenced: scripts/test.sh
(wrapper over tests/run-tests.sh, now exercised by the pre-push full
audit) and up.sh/down.sh stubs for this non-compose repo. Extend
kuma-fleet-sync STATIC_GEAR with the fixed network/office devices
(printer, consrv, tsys6/7 OOB, scanners, label printer, r1-tor-top,
DOME) so future runs keep their ICMP monitors in sync; DOME added
paused like the other known-down systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Gap-analyzes the tailnet Linux fleet + static network gear (switches,
router, APs, PDU, stor1, Reston VPSes) against Uptime Kuma ping
monitors over the socket.io API, and can create missing monitors with
--add. Websocket transport is forced because the Cloudron proxy drops
engine.io polling pushes. First run closed the last 4 gaps: 84/84.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-bms and tsys-umbrel were renamed in the Tailscale admin console,
so MagicDNS now serves the canonical names. The audit key switches
from the OS-reported hostname to the admin-assigned DNS name, which
retires the alias table. Reston VPS exclusion keyed to its MagicDNS
name (tsys-cloudron). Every in-scope system now passes: VM name,
DNS name, and Tailscale name all agree.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Names verified matching (netbird TS/DNS) — no fleet access audit
needed, same treatment as the other Reston systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Device-side hostname, /etc/hosts, and Tailscale node renamed from
ultixfield to ultix-field to match the DNS canonical name (convention
match with ultix-streaming/ultix-offstage). Verified unprivileged path
is the ultixfield account with passwordless sudo.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
With UCS and Jetson keys in place every auditable system now passes:
68 OK / 2 by-design (Home Assistant, Umbrel). Zero failures.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
3dscanner has been offline ~7 months; sandbox is a disposable
break-fix VM rolled back via Proxmox snapshots, so a failed SSH
probe there is expected noise, not a finding.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
era stlpc uses root, pfvsvrpi is localuser not subodev [#446]
Probed root access on every host where localuser failed: all Proxmox
appliance VMs (PDM/PMG/PBS), stlpc-artroom/garage, pfv-jetson-nano-1,
and ultix-field accept root key auth; pfvsvrpi takes localuser with
passwordless sudo. Audit now models this correctly — 65 of 74 green.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Exclude the Reston prod VPS (tsys-cloudron-new) from all future audits,
mark Home Assistant and Umbrel as by-design no-SSH appliances, and query
Technitium directly (dig @netinfra-01) instead of getent so local
/etc/hosts entries cannot produce false mismatches. Drops aliases made
obsolete by the sectestbed-hfnoc-uisp rename.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Reusable audit over every Linux tailnet peer: knel.net DNS resolution vs
Tailscale IP, Proxmox VM-name match, expected-user SSH login via the
remote.sh/remote-dns.sh chokepoints, and passwordless sudo verification.
First run (2026-08-26): 54/74 peers fully green, report on Discourse #312.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
If a tool has an API, use it. Exhaust all API endpoints before
considering direct DB access. DB schemas change without warning;
APIs are the stable contract. Aligns with upcoming zero trust
proxy migration.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- UAT rule: user must accept work before done-ratio 100% or close
- check-rules.sh: prune .crush/ from Discourse pointer scan (agent
working space, not documentation)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The HUD is an ephemeral status block (active ticket, done items, current
work, queued items, user action items) shown after significant responses.
Written to .crush/hud.md on disk to save context tokens; never committed.
Redmine/Discourse/git remain the durable systems of record.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds a mandatory second gate (step 8) after the rule-summary gate. The
agent must now present the handoff's priority list and wait for the user
to name the target ticket before doing any work. Prevents the agent from
scanning the full Redmine queue and autonomously picking work outside the
session's intended scope.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Complete context for a new Crush session to pick up OAM work without
re-reading this conversation. Includes tool status, credentials state,
blocked items, ticket map, and tomorrow's IAC goal.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- oxidized user created on all 4 Dell PowerConnect switches (privilege 1)
- Configs pulled and pushed to KNEL/network-device-configs on Gitea
- SSH key generated on tsys-librenms, added to Gitea for auto-push
- Cisco router (pfv-rrinfra-rtr) blocked: fan failure floods console (#431)
- UNPoller fully operational with real UniFi credentials
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
UNPoller Docker image moved to ghcr.io/unpoller/unpoller. Container now
pulls and runs successfully (v3.4.1, Prometheus metrics on :9130).
Auth failing as expected with CHANGE_ME placeholder creds.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deploy OAM monitoring stack on tsys-librenms VM:
- Smokeping: fully operational (port 8081, 16 targets, FPing probing)
- NetDisco: backend+DB+PostgreSQL running (port 8082), all 4 Dell switches
discovered via SNMP. Web UI blocked by Dancer session_cookie_key config
issue — needs interactive netdisco-deploy run
- Oxidized: container running (port 8083), deployed with placeholder
credentials — needs switch login/password to start config backups
- UNPoller: config + docker-compose ready, blocked on Docker image
availability (image moved from all common registries)
- Weathermap: not yet started (deferred to separate commit)
All configs use DNS names only (no IP literals). All scripts pass
shellcheck via check-rules.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add NON-NEGOTIABLE rule: every remote command must be wrapped with
timeout (30s reads, 120s standard, 300s deployments). A hung command
is a failed command — investigate root cause, never blindly retry.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Replace soft "ssh may be blocked" note with hard non-negotiable rules:
ALL SSH must route through remote.sh/remote-dns.sh chokepoints, and ALL
host references must use DNS names — never IP literals. Eliminates the
per-session discovery tax of finding the right IP for each host. Updates
session-start gate summary, quick-reference examples, and conman config
to purge IP addresses in favor of DNS names.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Mechanically enforces the Agent Authority policy: blocks all modifying
operations (bash, edit, write, multiedit) until an active ticket is set
via: echo '#NNN' > .crush/active-ticket
Exempts read-only tools, ticket management (redmine-cli/discourse-cli),
repo hygiene (check-rules, shellcheck), and monitoring commands.
Also: un-ignore .crush/crush.json so the hook config is tracked in git.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Pinned every production VM to its current ARP-observed LAN IP via DHCP
reservation. No IP changes — purely converts dynamic leases to fixed.
Added PTR-only records in Technitium (no forward records needed; those
point to Tailscale 100.x addresses).
DHCP failover sync triggered manually to netinfra-02. Live dhcpd.conf
pulled into repo.
Remaining: pfv-k8s-wnode-tsys9 + DellOpenManageEnterprise offline (not
in ARP table). Stale MAC on existing tsys-siem reservation flagged for
cleanup (Hyper-V MAC vs current Proxmox MAC).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
New NON-NEGOTIABLE rule: agents may suggest closing a ticket when the
result is clearly scoped and delivered, but must never close without
explicit user permission. Added to both PFVCluster AGENTS.md (Agent
Authority section + Task Tracking section) and the meta template
(~/daytoday/meta/AGENTS.md Systems of Record section).
Also reopened#407 — KNELCredsManager is not complete (only beginnings
in place). Corrected Vault understanding: deployed on Cloudron, not SITER k8s.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
identify-outlets.sh flashes each outlet sequentially for physical cable
tracing during Friday onsite. rename-outlets.sh takes a mapping file
and rewrites powerman.conf node entries. Both shellcheck-clean.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The cross-rack trunk (ch1, 4×1G) between core-sw01 and tor3-stor is
still static mode=on — no failure detection if a cable dies. Adding
command files to convert both switches to LACP during Friday's
maintenance window. Core hash stays layer-2-3 (5448 hardware limit);
stor hash stays layer-2-3-4 (already set).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys-siem verified SSHOK + SUDOOK via access-matrix probe. With this
landing, 69/70 non-excluded systems are at intended access state; the
sole remaining gap (stlp-3dscanner) is deferred to [#417]. Strip the
now-empty active-target section from bootstrap-all.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Access validation shows 68/70 non-excluded systems at intended access
state. Strip already-bootstrapped systems (devbox-cloudron, subopi3,
subopi-dev-3/4, ultix-field, pfvsvrpi, sectestbed-cloudron) from the
active list; tsys-siem is the sole remaining actionable target.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Codify the orientation checkpoint in Quick Start so every session begins
with the agent summarizing all rules and waiting for user confirmation
before any task work. Portable (AGENTS.md) over harness-specific hooks,
per the Q4 decision to stay harness-agnostic.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The login banner on subopi/pfvsvrpi systems was bleeding into the sudo
check output, making SUDOOK results look garbled and broken. Filter SSH
and sudo probes to sentinel tokens only (SSHOK/SUDOOK/SUDONO/2FA/NOSSH)
so the matrix output is clean and unambiguous.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Previous sessions used `qm guest exec` to back-door SSH keys into ~30 VMs,
bypassing sshd's audit trail in an ITAR/CMMC environment. Wire the ban deep
so it cannot recur:
- tests/remote.sh: remove the vm-guest mode + qm-guest-exec path entirely
- scripts/check-rules.sh: rule #11 fails on any `qm guest exec` / `vm-guest`
pattern in code (scans .sh/.bash/.py; docs may describe the ban freely)
- AGENTS.md: codify "Access-channel policy: SSH only" as non-negotiable;
add "Questions" rule banning harness question tools (use questions-v1.md)
- tests/vm-validation.sh: drop guest-agent key re-injection; SSH-only
- proxmox/perf/scripts/perf-matrix.sh + deploy-tuned-guests.sh: convert
guest-agent execution to SSH (vmroot) now that VMs have key + sudo
- bootstrap-all.sh: re-target the 8 remaining locked-out systems with
correct users/methods; print a console one-liner for publickey-only Pis
Guest-agent remains installable/checkable for Proxmox state visibility —
never as an execution or key-delivery path.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- tsys-ucs-01/02 moved to EXCLUDE in access-matrix.sh
- Close [#418] — no longer need 2FA removal or SSH key push
Refs [#418]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
agent-bootstrap.sh:
- Accept AGENT_USER env var (defaults to localuser)
- Push SSH key to root + $AGENT_USER + labuser (if present)
- Grant NOPASSWD sudo to $AGENT_USER (not hardcoded localuser)
- Skip qemu-guest-agent on bare metal (systemd-detect-virt) [#419]
bootstrap-all.sh:
- Pass AGENT_USER per system group:
subopi* → subodev, ultix-field → ultixfield
- All 8 remaining NO-KEY systems covered with correct user + escalation
This closes the gap where subodev/ultixfield systems would get the key
pushed to localuser (which doesn't exist) instead of the real agent user.
Refs [#403], [#419]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Replace complex log-capture with direct scp+ssh+ssh -t calls so password
prompts work at the terminal. Same sudo/su split per system type.
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- Split into sudo vs su escalation methods:
sudo: cloudron + siem (Ubuntu Server, no root password)
su: everything else (root has password)
- Capture full output to bootstrap-logs/<name>.log for debugging
- Show last 5 lines on screen, full log for agent review
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
access-matrix.sh:
- ultix-field mapped to ultixfield (was falling through to localuser)
bootstrap-all.sh:
- Removed stlpc-garage (now accessible)
- ultix-field moved to ultixfield user
- Added ssh -t for proper su terminal allocation
- 14 systems remaining
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles