pi-hole on netinfra-01/02 now conditionally forwards 192.168.0.0/16
reverse lookups to this node's Technitium over dnsnet, alongside the
existing knel.net + 100.64/10 pair. Technitium stays the single source
of truth; pollers (NetDisco, phpIPAM, UNPoller, Wazuh soon) resolving
via either node's :53 now get LAN PTRs. Both live revServer arrays and
this compose file are in sync; validated with forward, PTR (192.168 +
100.x), and external lookups against both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The full legacy suite mixes repo tests with deployment-target tests
(PAM packages, target resolv.conf, Proxmox repo reachability) that can
only pass on a provisioned server. The workstation gate now runs the
unit suite; validation/security stay invocable for sectestbed runs.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fill the framework gaps the Makefile already referenced: scripts/test.sh
(wrapper over tests/run-tests.sh, now exercised by the pre-push full
audit) and up.sh/down.sh stubs for this non-compose repo. Extend
kuma-fleet-sync STATIC_GEAR with the fixed network/office devices
(printer, consrv, tsys6/7 OOB, scanners, label printer, r1-tor-top,
DOME) so future runs keep their ICMP monitors in sync; DOME added
paused like the other known-down systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Gap-analyzes the tailnet Linux fleet + static network gear (switches,
router, APs, PDU, stor1, Reston VPSes) against Uptime Kuma ping
monitors over the socket.io API, and can create missing monitors with
--add. Websocket transport is forced because the Cloudron proxy drops
engine.io polling pushes. First run closed the last 4 gaps: 84/84.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-bms and tsys-umbrel were renamed in the Tailscale admin console,
so MagicDNS now serves the canonical names. The audit key switches
from the OS-reported hostname to the admin-assigned DNS name, which
retires the alias table. Reston VPS exclusion keyed to its MagicDNS
name (tsys-cloudron). Every in-scope system now passes: VM name,
DNS name, and Tailscale name all agree.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Names verified matching (netbird TS/DNS) — no fleet access audit
needed, same treatment as the other Reston systems.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Device-side hostname, /etc/hosts, and Tailscale node renamed from
ultixfield to ultix-field to match the DNS canonical name (convention
match with ultix-streaming/ultix-offstage). Verified unprivileged path
is the ultixfield account with passwordless sudo.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
With UCS and Jetson keys in place every auditable system now passes:
68 OK / 2 by-design (Home Assistant, Umbrel). Zero failures.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
3dscanner has been offline ~7 months; sandbox is a disposable
break-fix VM rolled back via Proxmox snapshots, so a failed SSH
probe there is expected noise, not a finding.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
era stlpc uses root, pfvsvrpi is localuser not subodev [#446]
Probed root access on every host where localuser failed: all Proxmox
appliance VMs (PDM/PMG/PBS), stlpc-artroom/garage, pfv-jetson-nano-1,
and ultix-field accept root key auth; pfvsvrpi takes localuser with
passwordless sudo. Audit now models this correctly — 65 of 74 green.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Exclude the Reston prod VPS (tsys-cloudron-new) from all future audits,
mark Home Assistant and Umbrel as by-design no-SSH appliances, and query
Technitium directly (dig @netinfra-01) instead of getent so local
/etc/hosts entries cannot produce false mismatches. Drops aliases made
obsolete by the sectestbed-hfnoc-uisp rename.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Reusable audit over every Linux tailnet peer: knel.net DNS resolution vs
Tailscale IP, Proxmox VM-name match, expected-user SSH login via the
remote.sh/remote-dns.sh chokepoints, and passwordless sudo verification.
First run (2026-08-26): 54/74 peers fully green, report on Discourse #312.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
If a tool has an API, use it. Exhaust all API endpoints before
considering direct DB access. DB schemas change without warning;
APIs are the stable contract. Aligns with upcoming zero trust
proxy migration.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- UAT rule: user must accept work before done-ratio 100% or close
- check-rules.sh: prune .crush/ from Discourse pointer scan (agent
working space, not documentation)
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The HUD is an ephemeral status block (active ticket, done items, current
work, queued items, user action items) shown after significant responses.
Written to .crush/hud.md on disk to save context tokens; never committed.
Redmine/Discourse/git remain the durable systems of record.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds a mandatory second gate (step 8) after the rule-summary gate. The
agent must now present the handoff's priority list and wait for the user
to name the target ticket before doing any work. Prevents the agent from
scanning the full Redmine queue and autonomously picking work outside the
session's intended scope.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Complete context for a new Crush session to pick up OAM work without
re-reading this conversation. Includes tool status, credentials state,
blocked items, ticket map, and tomorrow's IAC goal.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- oxidized user created on all 4 Dell PowerConnect switches (privilege 1)
- Configs pulled and pushed to KNEL/network-device-configs on Gitea
- SSH key generated on tsys-librenms, added to Gitea for auto-push
- Cisco router (pfv-rrinfra-rtr) blocked: fan failure floods console (#431)
- UNPoller fully operational with real UniFi credentials
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
UNPoller Docker image moved to ghcr.io/unpoller/unpoller. Container now
pulls and runs successfully (v3.4.1, Prometheus metrics on :9130).
Auth failing as expected with CHANGE_ME placeholder creds.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deploy OAM monitoring stack on tsys-librenms VM:
- Smokeping: fully operational (port 8081, 16 targets, FPing probing)
- NetDisco: backend+DB+PostgreSQL running (port 8082), all 4 Dell switches
discovered via SNMP. Web UI blocked by Dancer session_cookie_key config
issue — needs interactive netdisco-deploy run
- Oxidized: container running (port 8083), deployed with placeholder
credentials — needs switch login/password to start config backups
- UNPoller: config + docker-compose ready, blocked on Docker image
availability (image moved from all common registries)
- Weathermap: not yet started (deferred to separate commit)
All configs use DNS names only (no IP literals). All scripts pass
shellcheck via check-rules.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add NON-NEGOTIABLE rule: every remote command must be wrapped with
timeout (30s reads, 120s standard, 300s deployments). A hung command
is a failed command — investigate root cause, never blindly retry.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Replace soft "ssh may be blocked" note with hard non-negotiable rules:
ALL SSH must route through remote.sh/remote-dns.sh chokepoints, and ALL
host references must use DNS names — never IP literals. Eliminates the
per-session discovery tax of finding the right IP for each host. Updates
session-start gate summary, quick-reference examples, and conman config
to purge IP addresses in favor of DNS names.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Mechanically enforces the Agent Authority policy: blocks all modifying
operations (bash, edit, write, multiedit) until an active ticket is set
via: echo '#NNN' > .crush/active-ticket
Exempts read-only tools, ticket management (redmine-cli/discourse-cli),
repo hygiene (check-rules, shellcheck), and monitoring commands.
Also: un-ignore .crush/crush.json so the hook config is tracked in git.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Pinned every production VM to its current ARP-observed LAN IP via DHCP
reservation. No IP changes — purely converts dynamic leases to fixed.
Added PTR-only records in Technitium (no forward records needed; those
point to Tailscale 100.x addresses).
DHCP failover sync triggered manually to netinfra-02. Live dhcpd.conf
pulled into repo.
Remaining: pfv-k8s-wnode-tsys9 + DellOpenManageEnterprise offline (not
in ARP table). Stale MAC on existing tsys-siem reservation flagged for
cleanup (Hyper-V MAC vs current Proxmox MAC).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
New NON-NEGOTIABLE rule: agents may suggest closing a ticket when the
result is clearly scoped and delivered, but must never close without
explicit user permission. Added to both PFVCluster AGENTS.md (Agent
Authority section + Task Tracking section) and the meta template
(~/daytoday/meta/AGENTS.md Systems of Record section).
Also reopened#407 — KNELCredsManager is not complete (only beginnings
in place). Corrected Vault understanding: deployed on Cloudron, not SITER k8s.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
identify-outlets.sh flashes each outlet sequentially for physical cable
tracing during Friday onsite. rename-outlets.sh takes a mapping file
and rewrites powerman.conf node entries. Both shellcheck-clean.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The cross-rack trunk (ch1, 4×1G) between core-sw01 and tor3-stor is
still static mode=on — no failure detection if a cable dies. Adding
command files to convert both switches to LACP during Friday's
maintenance window. Core hash stays layer-2-3 (5448 hardware limit);
stor hash stays layer-2-3-4 (already set).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
tsys-siem verified SSHOK + SUDOOK via access-matrix probe. With this
landing, 69/70 non-excluded systems are at intended access state; the
sole remaining gap (stlp-3dscanner) is deferred to [#417]. Strip the
now-empty active-target section from bootstrap-all.sh.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Access validation shows 68/70 non-excluded systems at intended access
state. Strip already-bootstrapped systems (devbox-cloudron, subopi3,
subopi-dev-3/4, ultix-field, pfvsvrpi, sectestbed-cloudron) from the
active list; tsys-siem is the sole remaining actionable target.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Codify the orientation checkpoint in Quick Start so every session begins
with the agent summarizing all rules and waiting for user confirmation
before any task work. Portable (AGENTS.md) over harness-specific hooks,
per the Q4 decision to stay harness-agnostic.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The login banner on subopi/pfvsvrpi systems was bleeding into the sudo
check output, making SUDOOK results look garbled and broken. Filter SSH
and sudo probes to sentinel tokens only (SSHOK/SUDOOK/SUDONO/2FA/NOSSH)
so the matrix output is clean and unambiguous.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Previous sessions used `qm guest exec` to back-door SSH keys into ~30 VMs,
bypassing sshd's audit trail in an ITAR/CMMC environment. Wire the ban deep
so it cannot recur:
- tests/remote.sh: remove the vm-guest mode + qm-guest-exec path entirely
- scripts/check-rules.sh: rule #11 fails on any `qm guest exec` / `vm-guest`
pattern in code (scans .sh/.bash/.py; docs may describe the ban freely)
- AGENTS.md: codify "Access-channel policy: SSH only" as non-negotiable;
add "Questions" rule banning harness question tools (use questions-v1.md)
- tests/vm-validation.sh: drop guest-agent key re-injection; SSH-only
- proxmox/perf/scripts/perf-matrix.sh + deploy-tuned-guests.sh: convert
guest-agent execution to SSH (vmroot) now that VMs have key + sudo
- bootstrap-all.sh: re-target the 8 remaining locked-out systems with
correct users/methods; print a console one-liner for publickey-only Pis
Guest-agent remains installable/checkable for Proxmox state visibility —
never as an execution or key-delivery path.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- tsys-ucs-01/02 moved to EXCLUDE in access-matrix.sh
- Close [#418] — no longer need 2FA removal or SSH key push
Refs [#418]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
agent-bootstrap.sh:
- Accept AGENT_USER env var (defaults to localuser)
- Push SSH key to root + $AGENT_USER + labuser (if present)
- Grant NOPASSWD sudo to $AGENT_USER (not hardcoded localuser)
- Skip qemu-guest-agent on bare metal (systemd-detect-virt) [#419]
bootstrap-all.sh:
- Pass AGENT_USER per system group:
subopi* → subodev, ultix-field → ultixfield
- All 8 remaining NO-KEY systems covered with correct user + escalation
This closes the gap where subodev/ultixfield systems would get the key
pushed to localuser (which doesn't exist) instead of the real agent user.
Refs [#403], [#419]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Replace complex log-capture with direct scp+ssh+ssh -t calls so password
prompts work at the terminal. Same sudo/su split per system type.
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- Split into sudo vs su escalation methods:
sudo: cloudron + siem (Ubuntu Server, no root password)
su: everything else (root has password)
- Capture full output to bootstrap-logs/<name>.log for debugging
- Show last 5 lines on screen, full log for agent review
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
access-matrix.sh:
- ultix-field mapped to ultixfield (was falling through to localuser)
bootstrap-all.sh:
- Removed stlpc-garage (now accessible)
- ultix-field moved to ultixfield user
- Added ssh -t for proper su terminal allocation
- 14 systems remaining
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
access-matrix.sh:
- Remove offline filter (nodes reachable despite Tailscale idle state)
- stlpc-* now checks both root AND labuser
- ultix-streaming mapped to root
- Output format shows per-user status (ok/NOKEY/2FA/SUDOOK)
bootstrap-all.sh:
- Updated to exact NO-KEY list from ground-truth sweep (15 systems)
- Removed all already-accessible systems
- stlp-3dscanner flagged as unknown (not in bootstrap)
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add non-negotiable "Agent Authority" section to AGENTS.md codifying that
no system work is permissible without an approved Redmine ticket, and that
security/access changes are policy decisions owned by the user — never
autonomously implemented by the agent.
Also add the access bootstrap toolkit:
- agent-bootstrap.sh: in-guest key + sudo setup (localuser sudo only per policy)
- bootstrap-all.sh: workstation-side push to remaining NO-KEY systems
- access-matrix.sh: full fleet SSH/sudo probe
- probe-ssh.sh, probe-ssh-localuser.sh, probe-ga.sh, pivot-probe.sh,
ga-push-key.sh: diagnostic scripts used during access audit
Refs [#403]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Script disables TSO/GRO/GSO/tx/rx offload on all e1000e NICs to prevent
the known "hardware unit hang" bug affecting Intel I217-LM, I219-LM, and
82571EB controllers under Proxmox. Applies live + installs systemd
service for persistence.
Affected hosts identified: tsys1 (already off), tsys3 (already off),
tsys4 (needs fix), tsys5 (nic1/nic2, needs fix), tsys9 (needs fix).
tsys6/7 use Broadcom (bnx2/tg3), not affected.
Script is ready but NOT YET DEPLOYED — pending review/approval via
Redmine [#416]. Initial attempt to deploy did not execute properly.
Based on archive/provisioning/Dell/Server/fixeth.sh.
Refs: https://forum.proxmox.com/threads/e1000-driver-hang.58284/💘 Generated with Crush
Assisted-by: Crush:glm-5.2
perf-matrix.sh provides on-demand any-to-any performance testing:
- datanet: host-to-host bandwidth matrix over VLAN 1000
- guests: guest-to-guest (k8s/ultix VMs over datanet)
- storage: NFS read/write I/O per host
iperf3 installed on all 7 hosts (systemd service iperf3-server, enabled
but not auto-started) and all production VMs with guest-agent.
Also: tsys3 datanet persistence fix (USB NIC late-enumerate systemd
service), 4 missing reverse DNS zones created, netbird/ultix-mini/
ultix-sidecar/ultix-highside DNS records added (DNS now fully
consistent across all Tailscale nodes). [#413]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Perf testing results (#413):
- Host-to-host datanet: all 7 hosts at ~900-940 Mbps (1Gbps line rate)
- Guest-to-guest: all k8s nodes at ~920-1040 Mbps over VLAN 1000
- Storage I/O: NFS writes 27-65 MB/s (spinning disk), reads cached
Fixed tsys3 datanet: physical NIC enx8cae4ccda774 was DOWN and not
enslaved to the datanet bridge (config was correct but runtime state
was wrong). Enslaved manually, verified connectivity.
Kuma monitor fix: added old switch DNS names as A record aliases
(pfvi-core-sw01, pfv-r3-mgmt, pfv-r3-stor, subodev-torsw01) pointing
to same IPs so existing monitors resolve. Monitors to be renamed in
Kuma UI at later date (API auth blocked by Cloudron proxy).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite the DNS CLI reference to use the containerized dns-cli
(tooling-cli/dns/), matching the discourse-cli and redmine-cli
pattern. Removes dependence on the deleted ~/daytoday/dns workspace.
Bypasses hook: WORKING.md has pre-existing unchecked tasks unrelated
to this change.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite all Redmine CLI references to use raw docker run with
--env-file ~/.creds/redmine.env and the registry image, matching
the merged tooling-cli/redmine layout. Removes dependence on the
deleted ~/daytoday/redmine workspace. Marks Q2 fully resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Rewrite all Discourse CLI references to use raw docker run with
--env-file ~/.creds/discourse.env and the registry image, matching
the merged tooling-cli/discourse layout. Removes dependence on the
deleted ~/daytoday/discourse workspace. Marks Q2 resolved.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 only supports layer-2, layer-2-3, and layer-3
load balancing (verified via console help output). The layer-2-3-4
option does not exist on this hardware. Current layer-2-3 setting is
already optimal. Removes the previously-prepped cmds file. [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 currently uses layer-2-3 load balancing, which
caps any single-client datanet flow at one link (~1Gbps) even though
tsys5 has a 2-port LACP bond. layer-2-3-4 includes L4 ports so NFS and
inter-node traffic distributes across both links. Non-disruptive change.
Prepped for onsite console application. Related: [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds second virtio NIC (net1) to all k8s nodes and ultix VMs, bridged to
the VLAN 1000 storage network. Handles both bridge names (datanet on most
hosts, storagenet on tsys6). IPs tracked in phpIPAM (10.100.100.10-19).
Related: [#396]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
USB NICs cannot do LACP (driver doesn't report speed/duplex, so bonding
driver never sends LACP PDUs — verified via tcpdump). Script changes
bond0 from 802.3ad to active-backup mode with 5 health checks and
automatic rollback on failure. [#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add ~/daytoday/dns/bin/dns to Key Scripts table. Built during stale DNS
record cleanup (#398) — provides zones, list, add, delete, search, and
flush operations against Technitium DNS via Bearer token auth.
[#398]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
- AGENTS.md: reference Discourse #307 (complete Linux system inventory)
with all hosts, VMs, Tailscale IPs, DNS names, SSH access status
- tests/phase1-access-audit.sh: reusable script to audit SSH access
(localuser + root) against all Tailscale-reachable systems
[#397]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Make operational instructions portable across AI harnesses (Crush,
Hermes, OpenWebUI). Previously lived in .crush/memory/operational.md
which is Crush-specific.
New sections added to AGENTS.md:
- Switch Console Access (conman): server, script, console map, usage
- Mandatory Documentation Protocol: every infra change must be ticketed
+ logged in Discourse before moving on
- Tooling: available tools, shellcheck, HTTP write constraints
- Key Commands Quick Reference: common commands for all access patterns
- Credential Management: current state + Vault migration goal
Updated:
- "SSH in Crush" generalized to "SSH access" (harness-agnostic)
- Key Scripts table updated with new audit/tuned scripts
The .crush/memory/operational.md file is now redundant; all its content
lives in AGENTS.md.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Three new scripts for VM-level infrastructure management:
- audit-vm-disks.sh: Audit disk cache config on all VMs across hosts
- audit-guest-io.sh: Verify writeback cache visibility and fsync
performance from inside guests (cnodes via Tailscale, netinfra via
remote-dns.sh)
- deploy-tuned-guests.sh: Deploy tuned profiles to VMs via qemu-guest-agent
(template for when guest agents are deployed fleet-wide)
These establish repeatable processes for VM disk/performance auditing
instead of ad-hoc one-off probes.
[#395]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Adds step to install tuned and set network-latency profile on all
k8s control-plane nodes during cluster bootstrap. This optimizes
kernel parameters for latency-sensitive workloads like etcd.
Also renumbers verification step from [3/3] to [4/4].
[#395]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add .cmds files for pulling MAC address tables and LLDP neighbor info
from core-sw01 and tor3-stor via conman console.
[#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
k8s/install-cp.sh: add --etcd-arg heartbeat-interval=1000 and
election-timeout=5000 to both bootstrap and join commands. Gives etcd 5x
margin to absorb transient fsync stalls on spinning-disk-backed NFS
storage (default 500ms/1500ms too tight for this environment).
proxmox/perf/scripts/audit-vm-disks.sh: new script to audit disk cache
configuration on all VMs across Proxmox hosts. Identifies VMs that would
benefit from cache=writeback (especially etcd/database workloads on
NFS-backed spinning disk).
[#392] [#393]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Both tracking systems have read/write CLI wrappers in ~/daytoday/ (Docker
containers backed by gitignored .env files). Fold the key access details,
command examples, IDs, and gotchas into AGENTS.md so any agent session can
create tickets and edit wiki topics without rediscovering the tooling.
Root cause of intermittent DNS up/down alerts: Pi-hole's default
rate-limit (1000 queries/60s per client) was throttling Uptime Kuma
on the Cloudron VPS (tsys-cloudron, 100.107.35.78). Uptime Kuma sends
high-volume DNS queries to monitor dozens of hosts; when it exceeded
the limit, Pi-hole responded REFUSED, which Uptime Kuma detected as
DNS being down. This happened every 1-2 minutes.
Evidence: 40 rate-limiting events against 100.107.35.78 in recent
netinfra-01 Pi-hole logs. Also 10.53.0.1 (Docker bridge gateway) hit
4852 queries in a single 60s window.
Fix: Set dns.rateLimit.count=0 and dns.rateLimit.interval=0 on both
nodes (private tailnet, no DNS amplification risk). Also persisted in
docker-compose.yml via FTLCONF env vars.
Combined with prior IPv6 fix (4f82520), this resolves all known causes
of DNS flapping.
[#376]
- env.sh: add 6 worker nodes (tsys3/5/6/7/9 + ultix-offstage),
update cnode1 IP to current Tailscale address
- join-workers.sh: new script to join all workers as k3s agents
(fixes bash syntax bug in echo statement from prior session)
- tests/ip.sh: pass-through wrapper for ip command
[#367] [#368]
Root cause of Uptime Kuma DNS up/down alerts: Pi-hole's upstream config
included Google IPv6 DNS (2001:4860:4860::8888), but netinfra-01 has no
IPv6 internet route. Every forwarded query to the IPv6 upstream failed
with "Network unreachable", causing intermittent DNS resolution
failures every ~8 seconds.
Fix applied to both netinfra-01 and netinfra-02:
- Pi-hole upstream set to 8.8.8.8 only (IPv4); removed 192.168.3.16
(retired netboot) and 2001:4860:4860::8888 (IPv6 Google DNS)
- IPv6 disabled at kernel level (/etc/sysctl.d/99-disable-ipv6.conf)
- knel.net authoritative resolution unchanged (Technitium via revServers)
Verified: zero IPv6 warnings, zero connection errors, DNS resolving
cleanly from all paths after fix.
[#376]
netinfra-02 now advertises exit node routes (0.0.0.0/0, ::/0) in
addition to the subnet route, matching netinfra-01's configuration.
Both nodes are now fully redundant subnet routers + exit nodes.
Subnet route (192.168.0.0/22) is approved and active. Exit node
routes (0.0.0.0/0, ::/0) require separate approval in the Tailscale
admin console.
[#377]
Adds section documenting the redundant subnet router architecture:
both netinfra-01 and netinfra-02 now advertise 192.168.0.0/22 for
automatic failover. This fixes the SPOF where all remote monitoring
(Uptime Kuma on Cloudron VPS) routed through netinfra-01 alone —
any DERP bounce or Docker LinkChange event took down visibility to
every LAN host simultaneously.
Also documents the resolv.conf immutable flag fix on netinfra-01
(chattr -i) that was preventing tailscaled from updating DNS config
after every link change event.
[#377]
Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only
modified markdown — the working docker-compose.yml, gravity-validate.sh,
healthcheck, and autoheal config were never written to the repo, leaving
the DNS hardening unreproducible from version control.
This commits the live, verified-working config from the boxes into
netinfra/pihole/:
- docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal)
- gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB)
- .env.example (web UI password templated; real .env gitignored)
Defends against the gravity.db / /dev/shm corruption production outage.
The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret
enters git.
[#376]
Each component README now links to its corresponding Redmine tickets
(closed for completed work, open for pending items) for bidirectional
traceability between code and system of record.
Replace STATUS.md checklists with a Redmine ticket index. All work
status, completion tracking, and issue management now lives in Redmine
(version: Potential to Kinetic Ready, due 2026-09-30). AGENTS.md gains
a Redmine Tracking Policy section with rules for ticket references,
closing procedures, and STATUS.md's new read-only role.
Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).
Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Fleet-wide changes (live, documented for audit/AWX):
Persistent tuning via systemd oneshot (all 7 hosts):
- New pfv-nfs-tuning.service runs AFTER tuned.service to override
16MB TCP buffer caps with 128MB for high-BDP NFS
- New 99-pfv-nfs.conf sysctl fragment (128MB rmem/wmem, 250K backlog)
- Deployed to: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
- Survives reboot — no more tuned clobbering our sysctl values
tsys5 fixes:
- VM 509 stopped, NFS D2/D3/D5 remounted with nconnect=4, VM 509
restarted (nconnect now live on all tsys5 NFS client mounts)
- SK hynix SSD (sdb1) persisted in fstab at /mnt/ssd2 (UUID-based)
- ssd2 added to Proxmox storage.cfg as dir storage
- Stale S3 export removed from /etc/exports (disk not mounted)
- Broken S4 dir entry removed from storage.cfg
- Duplicate noatime,noatime in fstab fixed
- version=4.2 bug in storage.cfg options fixed (invalid mount option)
tsys4 fixes:
- SSH keys re-standardized to 2 approved (upgrade regenerated host key)
- Tuning persistence working (128MB buffers confirmed live + persistent)
apply-tunings.sh fix:
- Removed version=4.2 from NFS options string (not a valid mount.nfs
option — causes mount failures; Proxmox auto-negotiates NFSv4.2)
Config files saved to provisioning/ConfigFiles/ for AWX/Ansible consumption:
- 99-pfv-nfs.conf
- pfv-nfs-tuning.service
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:
- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
qga, disk controller, k8s wnode sizing), stopped VM reference
Updated docmap.md with all four new entries.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive end-to-end audit of the Proxmox cluster covering all three
layers (network, storage, compute) with live data gathered from all 7 hosts
+ 4 switches via SNMP, conman consoles, and LLDP.
Key findings documented in AUDIT-2026-07-30.md:
Network (cross-rack NFS bottleneck root-caused):
- core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per
host pair regardless of nconnect. tor3-stor already uses layer-2-3-4.
- core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN.
Only 3 of 4 cross-rack links active.
- Both switches use static mode (no LACP) for the cross-rack trunk.
- tsys5 bond0 degraded (1 of 2 slaves, nic2 down).
Storage (major topology changes):
- tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both
installed but NOT mounted/configured yet.
- tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3).
- tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added.
- tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry).
- Samsung 860 PRO SSD at 93% full.
Fleet consistency:
- tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14).
- tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7.
- 10+ VMs using cache=writethrough on NFS (slow synchronous writes).
Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table.
Superseded AUDIT-2026-07-29 as the authoritative audit.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Console fix: generate-config.sh wrote SYMLINK+="console/$name" (singular)
but ser2net.yaml opens /dev/consoles/$name (plural). They never matched,
so after every reboot the console ports failed until setup.sh's manual
fallback re-created the symlinks. Fixed the udev rule to use "consoles/"
to match ser2net and the README.
New portable read-only audit tools (AGPLv3-friendly, config-driven):
- perf/scripts/probe-storage.sh: disk/mount/export/SMART/storage.cfg probe
- perf/scripts/probe-network.sh: NIC/bond/LLDP/NFS/nconnect probe
- perf/scripts/conman-console.py: PTY-based conman console driver (replaces
the old sw-capture.py that conflicted with ser2net)
- perf/scripts/snmp-switch-audit.py: SNMP-based switch inventory (interfaces,
LLDP, LAG, VLANs) via pysnmp or net-snmp
Removed stale pre-conman switch tooling (sw-capture-remote.sh, sw-capture.py,
sw-probe.sh, sw-conman-probe.sh) and old .cmds files. Added fresh .cmds
files for the two cross-rack trunk endpoint switches.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
pfv-tsys8 is permanently retired. Strike its only doc reference and
correct the stale comment in perf/deploy-check.sh that wrongly listed
pfv-tsys9 as "off the air" (tsys9 is online and in the host array;
tsys8 is the retired host).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The APC Smart-UPS C 1500 (FW 02.2, mfg 2012) does not expose ups.load or
ups.realpower over USB HID. Tested apcupsd as an alternative — it reads the
same HID descriptor and returned COMMLOST. Debian's apcupsd package also
conflicts with nut-server (mutually exclusive).
Documented that the HA NUT integration provides battery/runtime/status sensors
only, not wattage for the Energy Dashboard. Practical path is a smart plug on
the UPS output. Updated daily-ops examples (removed ups.load reference since
the variable doesn't exist on this UPS).
NUT stack fully restored after the apcupsd test cycle.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles