Commit Graph
533 Commits
Author SHA1 Message Date
mrcharles 041d311d1c docs(questions): C4 credentials + C5 scope + C6 docker rulings recorded [#767]
Answers landed 2026-09-03: Bitwarden/Vault = cred SoR (.creds bootstrap-only);
this chat = COO/C2-only (PM/policy/enablement), agent stack lives here;
docker(988) verified on all 7 TSG*/COS* officer accounts.
2026-09-03 17:27:54 -05:00
mrcharles 994959d4ad docs(questions): C4-C8 agent-stack + credentials-policy rulings [#767]
Detail: https://projects.knownelement.com/issues/767
2026-09-03 17:13:29 -05:00
mrcharles a2a45ea9d8 feat(bench): harness v1.1 — PSI latency, load, device identity context [#709]
Replaces unavailable latencytop; sysstat/iotop deployed fleet-wide
separately. Redmine: https://projects.knownelement.com/issues/709
2026-09-03 16:55:28 -05:00
mrcharles abb725be85 feat(oam): nginx vhost SoR — OAM docker UIs behind fleet TLS [#697]
4 subdomain vhosts proxying to local docker ports; SAN cert from fleet
CA; workstation chain-verified (ssl_verify=0 all four).

Detail: https://projects.knownelement.com/issues/697
2026-09-03 14:42:33 -05:00
mrcharles 8a4ee6f64f dns: oam UI subdomains — smokeping/netdisco/oxidized/unpoller → tsys-librenms [#697]
A + auto-PTR via dns-cli on primary; zone-snapshot + drift-check ALL
IN SYNC same session. Serves the fleet-TLS pass for the tsys-librenms
docker UIs.

Detail: https://projects.knownelement.com/issues/697
2026-09-03 14:39:24 -05:00
mrcharles ba4463c4d0 fix(oam): kuma inventory target-field bug + add-ping tooling [#343][#435]
kuma-inventory.py printed the junk url field (https://) instead of the
hostname for ping monitors — hid ~100 monitors from FQDN matching.
kuma-add-ping.py: idempotent ping-monitor adds (exists-check by name and
target). Matrix published to Discourse t/309 (canonical, #343).

Detail: https://projects.knownelement.com/issues/343
2026-09-03 14:34:44 -05:00
mrcharles a07d866116 fix(cmdb): canonical CI names — s7 rename map in seed generator [#705]
from-inventory.sh now applies the #307 s7 VM-name -> Tailscale-name
mapping (two-pass awk), so rr-middleware-01/preprod-proxmoxmailgw rows
emit canonical names. Seed regenerated; GLPI renames + dup purges done
in same session (78 CIs at parity).

Detail: https://projects.knownelement.com/issues/705
2026-09-03 14:22:05 -05:00
mrcharles 8666182d8b docs(cmdb): regenerate seed post-reconcile — 75 CIs, stale rows gone [#705]
Detail: https://projects.knownelement.com/issues/705
2026-09-03 14:05:04 -05:00
mrcharles 056412a311 feat(cmdb): inventory reconcile + GLPI software catalog seeder [#705]
#307 s6 rewritten post-verification: zone was ALREADY clean (zero DNS
deletes); 12 stale CIs purged from GLPI, 3 alive off-cluster systems
given proper CIs (tsys-cloudron, ultix-field/highside), 75-row seed
regenerated. software-catalog.sh: 21 products + 59 installs (wazuh
agents linked from live manager); version lookup fixed to client-side
parent filter (GLPI ignores softwares_id search param).

Detail: https://projects.knownelement.com/issues/705
2026-09-03 14:01:56 -05:00
mrcharles 4c461c14e1 refactor: move HA tooling + reference packs to KNEL/pfv-bms [#762]
HomeAssistant/ is now a redirect stub; AGENTS.md layout updated.
Zero HA impact — pfv-bms never loaded from PFVCluster.

Detail: https://projects.knownelement.com/issues/762
2026-09-03 13:34:55 -05:00
mrcharles 1620351028 feat(ca): artifact mirror live — pinned wazuh-agent debs on tsys-ca [#758]
dist/wazuh-agent/4.14.7-1/ (amd64+arm64+armhf) + SHA256SUMS, autoindex
enabled, workstation hash-verified against published manifest.
deploy-agent.sh now installs from the mirror (upstream fallback) and
auto-detects arch. #335: kali-tsys enrolled+active (30 total agents).

Detail: https://projects.knownelement.com/issues/758
2026-09-03 13:25:28 -05:00
mrcharles e27e3a06b8 feat(oam): codify garage-pdu-relay snmpd extend + timeout hardening [#733]
Deployed manually on tsys-librenms 2026-09-02 (founder relay ruling);
codified same-day rule. snmpget -t2 -r1 so a dark PDU answers in ~2s.
Deployed target: tsys-librenms:/usr/local/bin/garage-pdu-relay.

Detail: https://projects.knownelement.com/issues/733
2026-09-03 12:55:31 -05:00
mrcharles d3b9099d5f feat(mail): fleet mail-env audit + map; PMG VIP proposal [#696][#694]
Audit findings note (55/64 systems, read-only):
https://projects.knownelement.com/issues/696#note-4088

Map + deploy plan: https://community.turnsys.com/t/322
Key: relayhost empty fleet-wide (direct-to-MX today); prod PMG pair
already installed (VM 604/711, PMG 9.1, unclustered); VIP slot
proposed 192.168.3.249; open PQs in questions-v8.md.
2026-09-03 12:49:59 -05:00
mrcharles 7ca87c1200 docs(questions): C1-C3 CMDB change-control rulings needed [#705]
Detail: https://projects.knownelement.com/issues/705
2026-09-03 12:32:20 -05:00
mrcharles 412d706abd docs(questions): v7 — PMG design round PQ1-PQ7 [#696][#694]
Session-start note (ground truth + design questions):
https://projects.knownelement.com/issues/696#note-4086

No duplicate ticket: #696 (child of umbrella #694) already covers
PMG deploy + MX cut-over; session gated on #696.
2026-09-03 12:27:34 -05:00
mrcharles 07a0a3472b fix(cmdb): GLPI import fixes — input wrapper + Read-Only agent profile [#705]
Import executed live: 87 CIs seeded, scoped cmdb user (Read-Only)
verified read-ok / write-denied. Fixes: POST body needs {input:[...]}
(ERROR_BAD_ARRAY on bare array), profile swapped Technician ->
Read-Only per least-priv ruling, shape-tolerant user-exists check.

Detail: https://projects.knownelement.com/issues/705
2026-09-03 12:26:39 -05:00
mrcharles 379f7d376b docs(cmdb): regenerate seed — wnode-tsys5 ghost retired [#728]
#307 row annotated RETIRED (VM 500 gone from every node, not in the k8s
roster); seed regenerated via the converter. Also carries the ultix-offstage
retirement.
2026-09-03 11:46:36 -05:00
mrcharles b7eb9a5066 fix(bench): trixie libaio1 fallback in bench LXC installer [#709]
Redmine detail: https://projects.knownelement.com/issues/709
2026-09-03 11:43:48 -05:00
mrcharles d237e4a65d feat(monitoring): kuma notification tooling; app webhooks replace pushover [#435]
kuma-notifications.py (list/delete/usage). Deleted unused Pushover provider
'KNEL Alerts' per Charles — 211/211 monitors already on the ultix-sidecar +
Ultix-mini app webhooks. fleet-sync creation payload updated to the live
notification IDs.
2026-09-03 11:38:38 -05:00
mrcharles 4c2726b116 feat(cmdb): GLPI importer — agent user + seed import ready to fire [#705]
to-glpi.sh: initSession, creates scoped local 'cmdb' user (Technician @
root entity, random token -> ~/.creds/glpi-agent.env), then batch-imports
systems.csv as Computers (chunked, idempotent by name, --dry-run).
Fires the moment Charles drops the corrected GLPI_USER_TOKEN.
Pure bash/jq per house rules.
2026-09-03 09:59:32 -05:00
mrcharles b575a8e864 fix(perf): install libaio1t64 fallback for Debian 13 guests [#709] 2026-09-03 09:30:29 -05:00
mrcharles bf52f85ac5 docs(agents): field lessons from #684 PBS rollout + restore-drill incident
Meat: https://projects.knownelement.com/issues/684
2026-09-03 08:33:58 -05:00
mrcharles 7586605b70 fix(perf): bench jq direction bug + libaio + json note guard [#709]
- write tests read .write.* explicitly (jq // kept read=0 for writes)
- libaio1 installed everywhere (real aio engine, no sync cap)
- fio stdout notes stripped before jq parse

Ticket: https://projects.knownelement.com/issues/709
2026-09-03 07:53:37 -05:00
mrcharles 82fdb161cc feat(ca): fleet CA tooling — intermediate live on tsys-ca, first cert issued [#697]
ca-init/issue-cert/selftest (TDD loop, shellcheck clean); design doc on
Discourse t/320. Offline RSA-4096 root (Nitrokey ceremony later), 5y
intermediate, 825d SAN leaves. First cert: tsys-wazuh.knel.net (fingerprint
on the ticket note).
Meat: https://projects.knownelement.com/issues/697#note-4033
2026-09-03 07:52:07 -05:00
mrcharles 02bdeae9e7 feat(perf): standardized fleet benchmark artifact (fio/iperf3) [#709]
bench-run.sh: identical test shape everywhere (4k rand rw, 1M seq rw,
direct IO, grep-able BENCH| lines) - runs in guests, LXCs, containers.
install-bench.sh for Debian guests/LXCs; Dockerfile for registry image;
lxc-bench-setup.sh stands up the host-side bench LXC (proxmox = no docker).

Ticket: https://projects.knownelement.com/issues/709
2026-09-03 07:21:27 -05:00
mrcharles d75366edf5 feat(siem): agent fixes from fleet pass; IPv6 kill script [#335][#748]
deploy-agent.sh now self-heals the three postinst failure modes found on
PMG/PBS hosts (MANAGER_IP placeholder, missing wazuh user, root:root
ownership). scripts/disable-ipv6.sh: live sysctl + persisted conf, with
in-file rollback notes. Rollout state in #335/#748.
Meat: https://projects.knownelement.com/issues/335#note-4008
2026-09-03 07:13:28 -05:00
mrcharles 2485866ccd chore(dns): remove retired wnode-tsys5 ghost A record from knel.net [#737]
Node retired and replaced by ultix-streaming; tailnet-side entry still
requires admin-console removal (no TS API key in store). Zone snapshot
refreshed same-session per DNS sync rule.

Ticket: https://projects.knownelement.com/issues/737
2026-09-03 06:32:44 -05:00
mrcharles 06679d9a0c feat(siem): fleet rollout tooling — agents + PVE rsyslog forwarding [#335]
deploy-agent.sh (Ubuntu VMs) + deploy-rsyslog.sh (PVE hosts, TCP/514 via
Tailscale). Landed: agents on netinfra-01/02, librenms, awx (7 total
active on manager); rsyslog forwarding verified on all 7 PVE hosts
(persistent 514 sessions on the manager). Deferred: k8s nodes (k8s-chat
coordination), docker json-log caps (needs daemon restart window).
Meat + verification: https://projects.knownelement.com/issues/335#note-4006
2026-09-03 06:30:25 -05:00
mrcharles 5c5e173a1b feat(cmdb): seed tooling — inventory→CSV converter + 87-CI seed v0 [#705]
P0 of the CMDB/change-control plan (design: Discourse t/319). Regenerates
the GLPI seed dataset from the #307 inventory raw; re-runnable whenever
#307 changes. AGENTS.md: cmdb/ layout entry + Key Scripts row.

Meat + plan: https://projects.knownelement.com/issues/705#note-4004
2026-09-03 05:54:08 -05:00
mrcharles 6a38b4443c docs(agents): cross-linking house rule + mastodon post tool [#743][#441]
Clickable-refs mandate from Charles: ticket comments link commits,
commit bodies link the Redmine comment URL, Discourse links both —
one click between Redmine/Gitea/Discourse (GLPI/CMDB change control
will extend this). Mechanical body-URL check queued under #441.
scripts/mastodon/post.sh: zero-install curl poster; creds land in
~/.creds/mastodon.env (MASTODON_URL/MASTODON_TOKEN, write scope).
2026-09-03 05:28:57 -05:00
mrcharles 6c80b76b47 docs: questions v4 — pairing mechanism fork + gitea allowlist ask [#626][#734] 2026-09-02 23:10:07 -05:00
mrcharles 4a38e9bac8 docs(questions): v6 — Q5 resolved (separate roster repo, built); Q4 open (second human) [#345]
https://projects.knownelement.com/issues/345
2026-09-02 21:59:36 -05:00
mrcharles 7376763824 docs(questions): v5 — v4 answers recorded (relay hw, probe cleared, HA whitelist); PR approver questions [#345]
Founder answered Q1-Q3 2026-09-02; new Q4/Q5 ask for named PR
approvers and pfv-bms protection scope.

https://projects.knownelement.com/issues/345
2026-09-02 21:30:51 -05:00
mrcharles 04b85e9935 feat(perf): ZFS blacklist script, applied to all 7 tsys hosts [#737]
ZFS is unused on the fleet (zero pools). Blacklisted module + disabled
zfs service family + initramfs bake-in; ARC reclaimed live (tsys5 ~9G,
tsys6 ~13G, tsys7 ~16G, tsys1 ~3G). Guard aborts on zfs-rooted hosts.

Ticket: https://projects.knownelement.com/issues/737
2026-09-02 21:17:34 -05:00
mrcharles 6414fee6ec feat(sensors): labeled per-zone thermal layer in lmsensors-extend [#736]
Hosts opt in via /etc/snmp/lmsensors-zone-types (one zone type per
line, e.g. Jetson GPU-therm); each type renders its own zone-<type>
chip block so HA templates anchor on the chip name. Default behavior
unchanged; no snmpd restarts (extend re-execs per poll).
2026-09-02 21:16:34 -05:00
mrcharles c71d66e129 docs: remote-access decision question for founder [#626][#344] 2026-09-02 21:07:27 -05:00
mrcharles b958dd762f docs: ask 14 answered — dockerd bounce applied [#731]
Meat: https://projects.knownelement.com/issues/731#note-3929
2026-09-02 20:42:28 -05:00
mrcharles a4d4c1c91a feat(perf): reusable guest perf pack deploy script [#737]
Idempotent in-guest tuning: sysctl profile, fq/bbr, THP->madvise with
boot persistence (yields to tuned), fstrim.timer. First applied to
pfv-k8s-wnode-tsys3.

Ticket: https://projects.knownelement.com/issues/737
2026-09-02 20:37:03 -05:00
mrcharles 17c555adf5 docs(questions): v4 badge doorman modernization round [#345]
Opens the actuator/reader-format/whitelist questions for the badge
system modernization on #345/#355.

Findings note: https://projects.knownelement.com/issues/355#note-3955
2026-09-02 20:29:17 -05:00
mrcharles 4de1457eaa docs(agents): dhcpd AppArmor /etc/dhcp path constraint field lesson [#728] 2026-09-02 20:23:38 -05:00
mrcharles 1dbf16c9df feat(dns): zone-snapshot tooling + sync rule; refresh stale snapshots [#630][#728]
Founder rule: every Technitium/DNS/DHCP change ends with a same-session
SoR sync — zone-snapshot.sh then drift-check green, then commit. Added
the tool (tar-pulls the DZ store from the primary via the chokepoint),
wrote the rule into AGENTS.md (DNS change discipline + Key Scripts rows),
refreshed 4 stale snapshots (knel.net, 1/3.168.192 reverse, 119.70.100
— incl. the deleted ultix-offstage PTR). drift-check: ALL IN SYNC.
Also: last tsrouter mentions retired (AGENTS.md, setup.sh header).
2026-09-02 20:19:23 -05:00
mrcharles c2b7c91079 fix(dhcp): retire ultix-offstage reservation — VM 5112 deleted [#728][#420]
Founder ruling 2026-09-02: ultix-offstage is retired. Removed the host
block (bc:24:11:1f:9d:83 -> 192.168.3.79) from both SoR confs and deployed
to the live pair serially with health gates; failover reports both-normal.
A record + PTR deleted from Technitium both nodes; #307 inventory rows
annotated. Tailscale device removal is a founder console action.
2026-09-02 20:18:09 -05:00
mrcharles ffda5a68f0 feat(switches): add show-only audit cmds for 4 consoles [#732]
Fleet perf audit found only 2/6 consoles had repo cmds sets; these four
were improvised show-only and used for the 2026-09-02 conman pulls.

Report: https://community.turnsys.com/t/298/66
Ticket: https://projects.knownelement.com/issues/732
2026-09-02 20:14:14 -05:00
mrcharles d9aa989957 docs(agents): HA snmp two-layer + package dupe-key field lessons [#344] 2026-09-02 20:02:54 -05:00
mrcharles 76e50de25f fix(dns): retire tailscale-router references; netinfra pair is prod [#728]
Founder ruling 2026-09-02: tailscale-router retired, subnet routing now
pfv-netinfra-01/02 (both advertise 192.168.0.0/22 + exit routes, verified).
Drop dead tsrouter alias from the chokepoint script; setup/verify now use
the primary as production source; refresh verify record list (tsys-nsm is
also a dead name; add tsys-wazuh CNAME + tsys-siem).

archive/KNELServerBuild copies left as read-only history by design.
2026-09-02 19:42:15 -05:00
mrcharles e6f33ca32a docs: ask 13 answered — Cloudron API token received+verified [#727]
Stored 0600 in ~/.creds/cloudron.env; API verified (profile/apps 200).
Meat: https://projects.knownelement.com/issues/727#note-3928
2026-09-02 19:07:43 -05:00
mrcharles 13fcb13182 move cloudron kuma artifacts to KNEL/cloudron [#727]
cloudron-apps.txt + kuma-cloudron-sync.py now live at
KNEL/cloudron monitoring/ (path re-rooted there).
Meat: https://projects.knownelement.com/issues/727#note-3913
2026-09-02 18:51:20 -05:00
mrcharles e824a1fc72 docs: Cloudron asks 13/14 — API token + dockerd window [#727]
Bootstrap of KNEL/cloudron repo + script capture.
Meat: https://projects.knownelement.com/issues/727#note-3913
2026-09-02 18:44:50 -05:00
mrcharles 77811927e3 docs: finish questions consolidation — drop merged files [#344]
Leftover unstaged state from the parallel 09-02 sessions: delete
questions-v1/v2, HomeAssistant/needfromcharles.md, night-grind-plan.md
(content lives in questions-09022206.md / questions-v3.md / git history);
drop superseded UniFi API-key ask (#619 multicast fix made it moot).

Details: https://projects.knownelement.com/issues/344#note-3911
2026-09-02 18:26:14 -05:00
mrcharles 0c6aabbb4d fix(rules): accept any questions-v*.md version + restore questions file
The required-files check pinned questions-v1.md, which breaks every time
the Q&A file versions up (v2 was closed out by the parallel session,
leaving nothing matching). Now glob-accepts any version. questions-v3.md
restores the file; active question channel is ~/PTKR.md per founder.

Detail: https://projects.knownelement.com/issues/441 (latest notes)

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-09-02 18:12:52 -05:00