Founder ruling 2026-09-02: ultix-offstage is retired. Removed the host
block (bc:24:11:1f:9d:83 -> 192.168.3.79) from both SoR confs and deployed
to the live pair serially with health gates; failover reports both-normal.
A record + PTR deleted from Technitium both nodes; #307 inventory rows
annotated. Tailscale device removal is a founder console action.
Founder ruling 2026-09-02: tailscale-router retired, subnet routing now
pfv-netinfra-01/02 (both advertise 192.168.0.0/22 + exit routes, verified).
Drop dead tsrouter alias from the chokepoint script; setup/verify now use
the primary as production source; refresh verify record list (tsys-nsm is
also a dead name; add tsys-wazuh CNAME + tsys-siem).
archive/KNELServerBuild copies left as read-only history by design.
ntp.conf captured (nodes byte-identical); 171 Technitium DZ zone files
captured from the primary; drift-check now covers dhcpd/pihole/ntp on
both nodes plus a zones md5-manifest check. Live run caught real drift:
node-02 dhcpd.conf was a stale primary copy missing the #614 minisplit
reservation — reconciled git->02 with dhcpd -t gate, serial restart,
failover "Both servers normal", all 7 checks in sync. Red-green unit
tests extended (gen_manifest, tracked-file invariants).
Results: https://projects.knownelement.com/issues/630#note-2
Live proof: ecobee answers unicast mDNS with PTR only
(_hap._tcp.local -> "Main Floor._hap._tcp.local"), so the old
replace-on-probe wiped learned records every cycle. Merge by
(name, type) instead. CONFIG_PATH now env-overridable for tests.
Details: https://projects.knownelement.com/issues/619#note-5💘 Generated with Crush
Assisted-by: Crush:glm-5.2
DNS/DHCP/NTP redundancy per founder ruling 2026-08-27:
- DHCP failover converted load-balance -> hot standby (split 255 on
primary; secondary answers only if primary unreachable >3s). DHCP
options now hand out BOTH nodes for DNS and NTP (active/active) —
Tailscale hosts stay on MagicDNS 100.100.100.100.
- netinfra-02 ntpsec deployed (pool.ntp.org, all interfaces); 01
repointed from debian pool zones to pool.ntp.org. NTP now redundant;
GPS stratum-1 on pfvsvrpi tracked as follow-up ticket.
- snmpd on both nodes with scoped ACLs (LibreNMS LAN+TS sources only)
and ntpq/dhcpd-lease extends — prep for DNS/DHCP/NTP graphing.
- Deployed sync-zones.sh (git version, DNS name instead of IP literal)
to netinfra-02; zone sync verified 171/171 zones both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The reservation carried a4:ba:db:0b:df:a0 (core-switch OUI, mis-migrated
from pfv-netboot) so tsys6's iDRAC could never lease its reserved address.
Replaced with the in-band-verified BMC MAC; deployed to both failover
nodes, validated, restarted.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The live configs were edited in place (canonical renames, stale block
removal, 7 new reservations, 69 total) but the repo copies were never
updated — the exact drift class that bit pi-hole. Repo and live state
now match on both failover nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
EOF
pi-hole on netinfra-01/02 now conditionally forwards 192.168.0.0/16
reverse lookups to this node's Technitium over dnsnet, alongside the
existing knel.net + 100.64/10 pair. Technitium stays the single source
of truth; pollers (NetDisco, phpIPAM, UNPoller, Wazuh soon) resolving
via either node's :53 now get LAN PTRs. Both live revServer arrays and
this compose file are in sync; validated with forward, PTR (192.168 +
100.x), and external lookups against both nodes.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Pinned every production VM to its current ARP-observed LAN IP via DHCP
reservation. No IP changes — purely converts dynamic leases to fixed.
Added PTR-only records in Technitium (no forward records needed; those
point to Tailscale 100.x addresses).
DHCP failover sync triggered manually to netinfra-02. Live dhcpd.conf
pulled into repo.
Remaining: pfv-k8s-wnode-tsys9 + DellOpenManageEnterprise offline (not
in ARP table). Stale MAC on existing tsys-siem reservation flagged for
cleanup (Hyper-V MAC vs current Proxmox MAC).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The cross-rack trunk (ch1, 4×1G) between core-sw01 and tor3-stor is
still static mode=on — no failure detection if a cable dies. Adding
command files to convert both switches to LACP during Friday's
maintenance window. Core hash stays layer-2-3 (5448 hardware limit);
stor hash stays layer-2-3-4 (already set).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 only supports layer-2, layer-2-3, and layer-3
load balancing (verified via console help output). The layer-2-3-4
option does not exist on this hardware. Current layer-2-3 setting is
already optimal. Removes the previously-prepped cmds file. [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 currently uses layer-2-3 load balancing, which
caps any single-client datanet flow at one link (~1Gbps) even though
tsys5 has a 2-port LACP bond. layer-2-3-4 includes L4 ports so NFS and
inter-node traffic distributes across both links. Non-disruptive change.
Prepped for onsite console application. Related: [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add .cmds files for pulling MAC address tables and LLDP neighbor info
from core-sw01 and tor3-stor via conman console.
[#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.
37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
Root cause of intermittent DNS up/down alerts: Pi-hole's default
rate-limit (1000 queries/60s per client) was throttling Uptime Kuma
on the Cloudron VPS (tsys-cloudron, 100.107.35.78). Uptime Kuma sends
high-volume DNS queries to monitor dozens of hosts; when it exceeded
the limit, Pi-hole responded REFUSED, which Uptime Kuma detected as
DNS being down. This happened every 1-2 minutes.
Evidence: 40 rate-limiting events against 100.107.35.78 in recent
netinfra-01 Pi-hole logs. Also 10.53.0.1 (Docker bridge gateway) hit
4852 queries in a single 60s window.
Fix: Set dns.rateLimit.count=0 and dns.rateLimit.interval=0 on both
nodes (private tailnet, no DNS amplification risk). Also persisted in
docker-compose.yml via FTLCONF env vars.
Combined with prior IPv6 fix (4f82520), this resolves all known causes
of DNS flapping.
[#376]
Root cause of Uptime Kuma DNS up/down alerts: Pi-hole's upstream config
included Google IPv6 DNS (2001:4860:4860::8888), but netinfra-01 has no
IPv6 internet route. Every forwarded query to the IPv6 upstream failed
with "Network unreachable", causing intermittent DNS resolution
failures every ~8 seconds.
Fix applied to both netinfra-01 and netinfra-02:
- Pi-hole upstream set to 8.8.8.8 only (IPv4); removed 192.168.3.16
(retired netboot) and 2001:4860:4860::8888 (IPv6 Google DNS)
- IPv6 disabled at kernel level (/etc/sysctl.d/99-disable-ipv6.conf)
- knel.net authoritative resolution unchanged (Technitium via revServers)
Verified: zero IPv6 warnings, zero connection errors, DNS resolving
cleanly from all paths after fix.
[#376]
Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only
modified markdown — the working docker-compose.yml, gravity-validate.sh,
healthcheck, and autoheal config were never written to the repo, leaving
the DNS hardening unreproducible from version control.
This commits the live, verified-working config from the boxes into
netinfra/pihole/:
- docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal)
- gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB)
- .env.example (web UI password templated; real .env gitignored)
Defends against the gravity.db / /dev/shm corruption production outage.
The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret
enters git.
[#376]
Each component README now links to its corresponding Redmine tickets
(closed for completed work, open for pending items) for bidirectional
traceability between code and system of record.
ISC DHCP server with failover-peer (primary/secondary) deployed on
pfv-netinfra-01 (192.168.3.252) and pfv-netinfra-02 (192.168.3.253). Both
servers in "normal" state, load-balancing 333 active leases.
Migration details:
- Copied all 37 host reservations + subnet/pool config from pfv-netboot
- DHCP lease database copied from netboot for zero-disruption cutover
- DNS servers changed from 192.168.3.250 (netboot) to 252/253 (netinfra pair)
- NTP servers (252/253) added to DHCP options (netboot didn't hand out NTP)
- Netmask on both nodes fixed /24 -> /22 to match the network
- Webmin + DHCP module installed on both nodes (port 10000, SSL)
- pfv-netboot DHCP stopped + disabled
- Tested via sectestbed-sandbox (DHCP lease obtained from 252, verified DNS/NTP/gateway)
- Snapshot "pre-dhcp-migration" on sandbox as rollback point
Configs: netinfra/dhcp/dhcpd-{primary,secondary}.conf
Plan + results: netinfra/dhcp-migration.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles
reachableceo