Pinned every production VM to its current ARP-observed LAN IP via DHCP
reservation. No IP changes — purely converts dynamic leases to fixed.
Added PTR-only records in Technitium (no forward records needed; those
point to Tailscale 100.x addresses).
DHCP failover sync triggered manually to netinfra-02. Live dhcpd.conf
pulled into repo.
Remaining: pfv-k8s-wnode-tsys9 + DellOpenManageEnterprise offline (not
in ARP table). Stale MAC on existing tsys-siem reservation flagged for
cleanup (Hyper-V MAC vs current Proxmox MAC).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
The cross-rack trunk (ch1, 4×1G) between core-sw01 and tor3-stor is
still static mode=on — no failure detection if a cable dies. Adding
command files to convert both switches to LACP during Friday's
maintenance window. Core hash stays layer-2-3 (5448 hardware limit);
stor hash stays layer-2-3-4 (already set).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 only supports layer-2, layer-2-3, and layer-3
load balancing (verified via console help output). The layer-2-3-4
option does not exist on this hardware. Current layer-2-3 setting is
already optimal. Removes the previously-prepped cmds file. [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Dell PowerConnect 5448 currently uses layer-2-3 load balancing, which
caps any single-client datanet flow at one link (~1Gbps) even though
tsys5 has a 2-port LACP bond. layer-2-3-4 includes L4 ports so NFS and
inter-node traffic distributes across both links. Non-disruptive change.
Prepped for onsite console application. Related: [#369]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Add .cmds files for pulling MAC address tables and LLDP neighbor info
from core-sw01 and tor3-stor via conman console.
[#394]
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.
37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
Root cause of intermittent DNS up/down alerts: Pi-hole's default
rate-limit (1000 queries/60s per client) was throttling Uptime Kuma
on the Cloudron VPS (tsys-cloudron, 100.107.35.78). Uptime Kuma sends
high-volume DNS queries to monitor dozens of hosts; when it exceeded
the limit, Pi-hole responded REFUSED, which Uptime Kuma detected as
DNS being down. This happened every 1-2 minutes.
Evidence: 40 rate-limiting events against 100.107.35.78 in recent
netinfra-01 Pi-hole logs. Also 10.53.0.1 (Docker bridge gateway) hit
4852 queries in a single 60s window.
Fix: Set dns.rateLimit.count=0 and dns.rateLimit.interval=0 on both
nodes (private tailnet, no DNS amplification risk). Also persisted in
docker-compose.yml via FTLCONF env vars.
Combined with prior IPv6 fix (4f82520), this resolves all known causes
of DNS flapping.
[#376]
Root cause of Uptime Kuma DNS up/down alerts: Pi-hole's upstream config
included Google IPv6 DNS (2001:4860:4860::8888), but netinfra-01 has no
IPv6 internet route. Every forwarded query to the IPv6 upstream failed
with "Network unreachable", causing intermittent DNS resolution
failures every ~8 seconds.
Fix applied to both netinfra-01 and netinfra-02:
- Pi-hole upstream set to 8.8.8.8 only (IPv4); removed 192.168.3.16
(retired netboot) and 2001:4860:4860::8888 (IPv6 Google DNS)
- IPv6 disabled at kernel level (/etc/sysctl.d/99-disable-ipv6.conf)
- knel.net authoritative resolution unchanged (Technitium via revServers)
Verified: zero IPv6 warnings, zero connection errors, DNS resolving
cleanly from all paths after fix.
[#376]
Commit 33b5c76 claimed to harden Pi-hole on both DNS nodes but only
modified markdown — the working docker-compose.yml, gravity-validate.sh,
healthcheck, and autoheal config were never written to the repo, leaving
the DNS hardening unreproducible from version control.
This commits the live, verified-working config from the boxes into
netinfra/pihole/:
- docker-compose.yml (shm_size 1024M root-cause fix, healthcheck, autoheal)
- gravity-validate.sh (pre-start SQLite header check, auto-quarantine corrupt DB)
- .env.example (web UI password templated; real .env gitignored)
Defends against the gravity.db / /dev/shm corruption production outage.
The live password is templated as ${PIHOLE_WEB_PASSWORD} so no secret
enters git.
[#376]
Each component README now links to its corresponding Redmine tickets
(closed for completed work, open for pending items) for bidirectional
traceability between code and system of record.
ISC DHCP server with failover-peer (primary/secondary) deployed on
pfv-netinfra-01 (192.168.3.252) and pfv-netinfra-02 (192.168.3.253). Both
servers in "normal" state, load-balancing 333 active leases.
Migration details:
- Copied all 37 host reservations + subnet/pool config from pfv-netboot
- DHCP lease database copied from netboot for zero-disruption cutover
- DNS servers changed from 192.168.3.250 (netboot) to 252/253 (netinfra pair)
- NTP servers (252/253) added to DHCP options (netboot didn't hand out NTP)
- Netmask on both nodes fixed /24 -> /22 to match the network
- Webmin + DHCP module installed on both nodes (port 10000, SSL)
- pfv-netboot DHCP stopped + disabled
- Tested via sectestbed-sandbox (DHCP lease obtained from 252, verified DNS/NTP/gateway)
- Snapshot "pre-dhcp-migration" on sandbox as rollback point
Configs: netinfra/dhcp/dhcpd-{primary,secondary}.conf
Plan + results: netinfra/dhcp-migration.md
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Comprehensive documentation gardening across the merged repo:
- tailscale.md: fully rewritten with current ground truth. The netinfra
pair now runs production Technitium with all knel.net records
replicated. Both LAN IPs resolve knel.net device names and recurse
externally. The old "NXDOMAIN / zone is stale" findings are replaced
with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
to all key scripts and docs. Autonomous commit/push policy
prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
"PFVCluster" in actionable docs; historical AI-review docs tagged
with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
docs/SECURITY.md: path references fixed to current structure.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
Add a shared Docker network (dnsnet, 10.53.0.0/24) connecting Pi-hole and
Technitium containers so Pi-hole can conditionally forward knel.net and
Tailscale-reverse queries to the local authoritative Technitium instance
(10.53.0.53) instead of netboot's upstream 192.168.3.16. Also adds
end-to-end documentation for both the reference node (pfv-netboot) and
the replicated nodes (pfv-netinfra-01/02).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:
Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)
Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2
Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.
Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.
Added tsys9 to deploy-check.sh host list.
Storage philosophy (user directive):
- NVMe/SSD: k8s worker scratch + ultix-streaming (dev workstation
running "cluster of 1" pre-prod jobs before full k8s deployment)
- Spinning rust: all infrastructure VMs (UCS, netinfra, LibreNMS, SIEM)
Clarified that hosts are standalone but managed via Proxmox Datacenter
Manager (PDM), which supports VM migration between nodes through the
UI -- eliminating the need for manual disk copies in the migration plan.
Updated all migration steps to reference PDM storage migrate instead
of manual cp commands.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
mrcharles
reachableceo