#307 row annotated RETIRED (VM 500 gone from every node, not in the k8s
roster); seed regenerated via the converter. Also carries the ultix-offstage
retirement.
kuma-notifications.py (list/delete/usage). Deleted unused Pushover provider
'KNEL Alerts' per Charles — 211/211 monitors already on the ultix-sidecar +
Ultix-mini app webhooks. fleet-sync creation payload updated to the live
notification IDs.
to-glpi.sh: initSession, creates scoped local 'cmdb' user (Technician @
root entity, random token -> ~/.creds/glpi-agent.env), then batch-imports
systems.csv as Computers (chunked, idempotent by name, --dry-run).
Fires the moment Charles drops the corrected GLPI_USER_TOKEN.
Pure bash/jq per house rules.
deploy-agent.sh now self-heals the three postinst failure modes found on
PMG/PBS hosts (MANAGER_IP placeholder, missing wazuh user, root:root
ownership). scripts/disable-ipv6.sh: live sysctl + persisted conf, with
in-file rollback notes. Rollout state in #335/#748.
Meat: https://projects.knownelement.com/issues/335#note-4008
Node retired and replaced by ultix-streaming; tailnet-side entry still
requires admin-console removal (no TS API key in store). Zone snapshot
refreshed same-session per DNS sync rule.
Ticket: https://projects.knownelement.com/issues/737
Clickable-refs mandate from Charles: ticket comments link commits,
commit bodies link the Redmine comment URL, Discourse links both —
one click between Redmine/Gitea/Discourse (GLPI/CMDB change control
will extend this). Mechanical body-URL check queued under #441.
scripts/mastodon/post.sh: zero-install curl poster; creds land in
~/.creds/mastodon.env (MASTODON_URL/MASTODON_TOKEN, write scope).
Hosts opt in via /etc/snmp/lmsensors-zone-types (one zone type per
line, e.g. Jetson GPU-therm); each type renders its own zone-<type>
chip block so HA templates anchor on the chip name. Default behavior
unchanged; no snmpd restarts (extend re-execs per poll).
Idempotent in-guest tuning: sysctl profile, fq/bbr, THP->madvise with
boot persistence (yields to tuned), fstrim.timer. First applied to
pfv-k8s-wnode-tsys3.
Ticket: https://projects.knownelement.com/issues/737
Founder rule: every Technitium/DNS/DHCP change ends with a same-session
SoR sync — zone-snapshot.sh then drift-check green, then commit. Added
the tool (tar-pulls the DZ store from the primary via the chokepoint),
wrote the rule into AGENTS.md (DNS change discipline + Key Scripts rows),
refreshed 4 stale snapshots (knel.net, 1/3.168.192 reverse, 119.70.100
— incl. the deleted ultix-offstage PTR). drift-check: ALL IN SYNC.
Also: last tsrouter mentions retired (AGENTS.md, setup.sh header).
Founder ruling 2026-09-02: ultix-offstage is retired. Removed the host
block (bc:24:11:1f:9d:83 -> 192.168.3.79) from both SoR confs and deployed
to the live pair serially with health gates; failover reports both-normal.
A record + PTR deleted from Technitium both nodes; #307 inventory rows
annotated. Tailscale device removal is a founder console action.
Founder ruling 2026-09-02: tailscale-router retired, subnet routing now
pfv-netinfra-01/02 (both advertise 192.168.0.0/22 + exit routes, verified).
Drop dead tsrouter alias from the chokepoint script; setup/verify now use
the primary as production source; refresh verify record list (tsys-nsm is
also a dead name; add tsys-wazuh CNAME + tsys-siem).
archive/KNELServerBuild copies left as read-only history by design.
Leftover unstaged state from the parallel 09-02 sessions: delete
questions-v1/v2, HomeAssistant/needfromcharles.md, night-grind-plan.md
(content lives in questions-09022206.md / questions-v3.md / git history);
drop superseded UniFi API-key ask (#619 multicast fix made it moot).
Details: https://projects.knownelement.com/issues/344#note-3911
The required-files check pinned questions-v1.md, which breaks every time
the Q&A file versions up (v2 was closed out by the parallel session,
leaving nothing matching). Now glob-accepts any version. questions-v3.md
restores the file; active question channel is ~/PTKR.md per founder.
Detail: https://projects.knownelement.com/issues/441 (latest notes)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
How to read and rename founder todo-list items via HA REST for the
Redmine [#NNN] sync-back workflow.
Detail: https://projects.knownelement.com/issues/441 (latest notes)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Discourse is the sole documentation space; Redmine wikis stay disabled
(verified 0/62 projects). Also repointed the stale redmine/discourse CLI
AGENTS.md references to ukrrs/connectors/KNEL-AIMiddleware.
Detail: https://projects.knownelement.com/issues/441 (latest note)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
ntp.conf captured (nodes byte-identical); 171 Technitium DZ zone files
captured from the primary; drift-check now covers dhcpd/pihole/ntp on
both nodes plus a zones md5-manifest check. Live run caught real drift:
node-02 dhcpd.conf was a stale primary copy missing the #614 minisplit
reservation — reconciled git->02 with dhcpd -t gate, serial restart,
failover "Both servers normal", all 7 checks in sync. Red-green unit
tests extended (gen_manifest, tracked-file invariants).
Results: https://projects.knownelement.com/issues/630#note-2
New "Cloudron" group (id 277); all 55 app monitors moved there
(canary-verified). Deleted IP-literal monitor id=8 (dup of DNS-named
id=210) and repointed appletv monitor to new DNS name
stl-appletv-livingroom.knel.net. Fleet-wide IP literals in Kuma: 0.
Sync tool default group updated to 277.
Results: https://projects.knownelement.com/issues/435#note-5
55 http monitors added under "Cloud Systems" (blue/green: canary first,
verified green); 9 root-level strays re-homed into founder's groups
(root now zero). New tools: inventory (read-only dump), cloudron-sync
(idempotent diff/add from committed app list), regroup (lib-based moves;
raw editMonitor times out on this build). fleet-sync now requires
--group-id so it can never place monitors at root again.
Results: https://projects.knownelement.com/issues/435#note-4
- cnodes 102/705/603 moved to TS5-SSD storage by founder (etcd apply 4-5s -> <1s)
- wnode-tsys5 slot retired; ultix-streaming joins in its place (5 workers)
- ultix-offstage unreachable; stays excluded until back on tailnet
[#367][#368] https://projects.knownelement.com/issues/368
Live proof: ecobee answers unicast mDNS with PTR only
(_hap._tcp.local -> "Main Floor._hap._tcp.local"), so the old
replace-on-probe wiped learned records every cycle. Merge by
(name, type) instead. CONFIG_PATH now env-overridable for tests.
Details: https://projects.knownelement.com/issues/619#note-5💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Archive the non-ported remainder of the legacy KNELServerBuild repo
under archive/KNELServerBuild/ with its original structure intact,
completing the legacy repo merge for everything except the live
LibreNMS patterns (ported in the previous commit).
Exclusions:
- .git history (superseded; legacy repo remains at its original path)
- ported files (Agents/librenms, Modules/OAM/oam-librenms.sh,
ConfigFiles/SNMP/snmp-sudo.conf)
- vendored KNELShellFramework tree (byte-identical duplicate of the
copy already vendored at vendor/ in this repo)
- SSH authorized-keys files (live access-control material; carrying
them in an archive invites drift — key policy lives elsewhere)
The whole tree is skip-listed in tests/shellcheck.sh (archived legacy
code, not maintained — same standing as vendor/); check-rules.sh
already prunes archive/. Rule 9 (conflict markers) now also excludes
archive/ staged files: preserved-verbatim legacy scripts contain
decorative "====" banners that false-positive as conflict markers
(same archive exclusion precedent as rule 11).
AGENTS.md: note archive/KNELServerBuild and oam/librenms-agent in the
Repository Layout, and fix the stale KNELIAC path to
/home/reachableceo/projects/KNEL/KNELIAC.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Port the live LibreNMS monitoring patterns from the legacy
KNELServerBuild repo into oam/librenms-agent/, joining the existing
OAM tooling (oxidized, unpoller, smokeping, netdisco):
- agent/ — upstream check_mk agent + snmp-extend scripts (dmi, dpkg,
mysql, ntp-client, ntp-server, os-updates, postfix, raspberry, smart,
ss, ups-nut), copied verbatim (md5-verified), never to be edited here
- setup.sh — deploy module ported from ProjectCode/Modules/OAM/
oam-librenms.sh; only the legacy framework bootstrap was replaced
with plain bash (path constants + print_info -> echo)
- snmp-sudo.conf — Debian snmpd sudo rule the extends require
(Debian-snmp NOPASSWD /bin/cat); carried as a file only, sudoers
install is a policy decision per AGENTS.md
Lint gates: extend the existing upstream-skip mechanism for the
verbatim agent scripts (tests/shellcheck.sh + check-rules.sh prune,
same precedent as archive/provisioning/Agents/librenms), and fix the
stale skip path there (provisioning/ moved to archive/provisioning in
6244c1c; the bash extends have been failing the whole-repo shellcheck
gate since).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
DNS/DHCP/NTP redundancy per founder ruling 2026-08-27:
- DHCP failover converted load-balance -> hot standby (split 255 on
primary; secondary answers only if primary unreachable >3s). DHCP
options now hand out BOTH nodes for DNS and NTP (active/active) —
Tailscale hosts stay on MagicDNS 100.100.100.100.
- netinfra-02 ntpsec deployed (pool.ntp.org, all interfaces); 01
repointed from debian pool zones to pool.ntp.org. NTP now redundant;
GPS stratum-1 on pfvsvrpi tracked as follow-up ticket.
- snmpd on both nodes with scoped ACLs (LibreNMS LAN+TS sources only)
and ntpq/dhcpd-lease extends — prep for DNS/DHCP/NTP graphing.
- Deployed sync-zones.sh (git version, DNS name instead of IP literal)
to netinfra-02; zone sync verified 171/171 zones both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The ticket gate required the single .crush/active-ticket file, so two
concurrent sessions overwrote each other's ticket mid-work. The gate now
accepts any non-empty .crush/active-ticket* file; each session keeps its
own (e.g. active-ticket-plant, active-ticket-core). Also prune .crush/
session scratch from both shellcheck scanners so one session's throwaway
probe scripts cannot block the other session's commits. Documented in
AGENTS.md Task Tracking.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
lm-sensors is silent on SBCs (Pis, Jetson-class), so the wrapper now falls
back to /sys/class/thermal zones rendered in lm-sensors format when native
sensors output is empty; suppressed when native output exists to avoid
duplicates. TDD: 3 new unit tests (12 green). Sensor stack deployed to
jetson + pfvsvrpi + 3 subopis (verified from poller), HA pack extended to
16 sensors across 14 hosts, high-temp automation covers the SBC fleet.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
SNMP sensors for all 9 instrumented hosts (temps in °F per founder
ruling, °C converted at display layer), ACPI node-wattage feeds for
tsys6/7, UPS alert automations (on-battery, battery-low, comm-lost,
power-restored, high-temp) targeting Pushover, and Riemann-sum kWh
helpers for the Energy dashboard. PDU and iDRAC blocks left as pinned
placeholders pending PDU community and #462 OMSA work.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The reservation carried a4:ba:db:0b:df:a0 (core-switch OUI, mis-migrated
from pfv-netboot) so tsys6's iDRAC could never lease its reserved address.
Replaced with the in-band-verified BMC MAC; deployed to both failover
nodes, validated, restarted.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Native lm-sensors output plus optional TEMPer USB probe, unified in
lm-sensors format behind one snmpd extend (lmsensors). Idempotent
deploy binds snmpd to explicit LAN+Tailscale addresses only and
source-scopes the community to the pollers (LibreNMS, Home Assistant
app-connector LAN IP, admin workstation, Cloudron) per the founder's
security ruling. TDD unit suite included; fleet verified live (6 hosts)
and negative-tested (refused source).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The live configs were edited in place (canonical renames, stale block
removal, 7 new reservations, 69 total) but the repo copies were never
updated — the exact drift class that bit pi-hole. Repo and live state
now match on both failover nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
EOF
Bring the framework's global baseline (BASELINE-PROMPT.md, PATTERNS.md,
ADOPTING.md) into the repo — AGENTS.md referenced them but the files were
missing. AGENTS.md gains three rules adopted this session: TDD & Linting
(mandatory at the Ansible/IaC transition), IaC codification shadow-tracking
(every manual fleet change same-day ticketed to #454), and Redmine tracker
discipline (Support not Bug — the CLI default created nine misfiled
tickets, now corrected). Q10 records the direct-push vs PR conflict for
the founder to rule on.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pi-hole on netinfra-01/02 now conditionally forwards 192.168.0.0/16
reverse lookups to this node's Technitium over dnsnet, alongside the
existing knel.net + 100.64/10 pair. Technitium stays the single source
of truth; pollers (NetDisco, phpIPAM, UNPoller, Wazuh soon) resolving
via either node's :53 now get LAN PTRs. Both live revServer arrays and
this compose file are in sync; validated with forward, PTR (192.168 +
100.x), and external lookups against both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The full legacy suite mixes repo tests with deployment-target tests
(PAM packages, target resolv.conf, Proxmox repo reachability) that can
only pass on a provisioned server. The workstation gate now runs the
unit suite; validation/security stay invocable for sectestbed runs.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fill the framework gaps the Makefile already referenced: scripts/test.sh
(wrapper over tests/run-tests.sh, now exercised by the pre-push full
audit) and up.sh/down.sh stubs for this non-compose repo. Extend
kuma-fleet-sync STATIC_GEAR with the fixed network/office devices
(printer, consrv, tsys6/7 OOB, scanners, label printer, r1-tor-top,
DOME) so future runs keep their ICMP monitors in sync; DOME added
paused like the other known-down systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Gap-analyzes the tailnet Linux fleet + static network gear (switches,
router, APs, PDU, stor1, Reston VPSes) against Uptime Kuma ping
monitors over the socket.io API, and can create missing monitors with
--add. Websocket transport is forced because the Cloudron proxy drops
engine.io polling pushes. First run closed the last 4 gaps: 84/84.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-bms and tsys-umbrel were renamed in the Tailscale admin console,
so MagicDNS now serves the canonical names. The audit key switches
from the OS-reported hostname to the admin-assigned DNS name, which
retires the alias table. Reston VPS exclusion keyed to its MagicDNS
name (tsys-cloudron). Every in-scope system now passes: VM name,
DNS name, and Tailscale name all agree.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Names verified matching (netbird TS/DNS) — no fleet access audit
needed, same treatment as the other Reston systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Device-side hostname, /etc/hosts, and Tailscale node renamed from
ultixfield to ultix-field to match the DNS canonical name (convention
match with ultix-streaming/ultix-offstage). Verified unprivileged path
is the ultixfield account with passwordless sudo.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
With UCS and Jetson keys in place every auditable system now passes:
68 OK / 2 by-design (Home Assistant, Umbrel). Zero failures.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
3dscanner has been offline ~7 months; sandbox is a disposable
break-fix VM rolled back via Proxmox snapshots, so a failed SSH
probe there is expected noise, not a finding.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
era stlpc uses root, pfvsvrpi is localuser not subodev [#446]
Probed root access on every host where localuser failed: all Proxmox
appliance VMs (PDM/PMG/PBS), stlpc-artroom/garage, pfv-jetson-nano-1,
and ultix-field accept root key auth; pfvsvrpi takes localuser with
passwordless sudo. Audit now models this correctly — 65 of 74 green.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Exclude the Reston prod VPS (tsys-cloudron-new) from all future audits,
mark Home Assistant and Umbrel as by-design no-SSH appliances, and query
Technitium directly (dig @netinfra-01) instead of getent so local
/etc/hosts entries cannot produce false mismatches. Drops aliases made
obsolete by the sectestbed-hfnoc-uisp rename.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Reusable audit over every Linux tailnet peer: knel.net DNS resolution vs
Tailscale IP, Proxmox VM-name match, expected-user SSH login via the
remote.sh/remote-dns.sh chokepoints, and passwordless sudo verification.
First run (2026-08-26): 54/74 peers fully green, report on Discourse #312.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
If a tool has an API, use it. Exhaust all API endpoints before
considering direct DB access. DB schemas change without warning;
APIs are the stable contract. Aligns with upcoming zero trust
proxy migration.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- UAT rule: user must accept work before done-ratio 100% or close
- check-rules.sh: prune .crush/ from Discourse pointer scan (agent
working space, not documentation)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The HUD is an ephemeral status block (active ticket, done items, current
work, queued items, user action items) shown after significant responses.
Written to .crush/hud.md on disk to save context tokens; never committed.
Redmine/Discourse/git remain the durable systems of record.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles