mrcharles
33b5c76bec
netinfra-02 was accidentally wiped during storage moves. Rebuilt from
netinfra-01 config: Docker + Technitium (167 zones, rsync timer every 60s),
Pi-hole v6 (healthy, gravity.db validated), NTP container, ISC DHCP failover
secondary (load-balancing with primary, 99+84 active leases).
Pi-hole hardened on BOTH nodes with defense-in-depth against the
gravity.db corruption outage:
- gravity-validate.sh: pre-start SQLite header check, auto-removes corrupt DB
- Healthcheck: validates DNS resolution + gravity.db file integrity
- autoheal container: restarts unhealthy containers automatically
- Root cause (/dev/shm too small) already fixed by operator (shm_size: 1024M)
Also corrected DHCP lease count in audit (98 unique active, not 753 raw blocks).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
10 KiB
Ground-Truth Audit — 2026-08-05
Authoritative fleet audit performed by Crush (GLM-5.2) via read-only network discovery (nmap, SNMP, SSH, API queries). Supersedes
AUDIT-2026-07-30.mdfor all fleet state.Method: Tailscale
status, nmap ping sweep (192.168.0-3.0/24), SNMP walks (kn3lmgmtcommunity), SSH viatests/remote.sh+netinfra/dns-cluster-setup/remote-dns.sh, Proxmoxqm list/pvecm, Docker/container inspection.
TL;DR — What Changed Since 2026-07-30
The fleet grew significantly and the repository was restructured into a module/subproject layout. Headline deltas:
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|---|---|---|---|
| VMs (running) | 28 | 63 | +125% |
| DNS zones | 124 | 167 | +35% |
| DHCP active leases | 333 | 98 (unique active) | 431 total active binding-state entries |
| Proxmox hosts | 7 | 7 | — |
| PVE version (fleet) | mixed (9.1.5–9.2.5) | all 9.2.5 | tsys4 upgraded |
| Tailscale nodes | ~30 | 80 | full 3-tier env mirror |
| k8s cluster | LIVE (3-node HA) | DOWN | cnode wiped, no quorum |
| netinfra-02 | redundant secondary | WIPED | needs rebuild |
Repository Restructure (DONE by operator)
Top-level dirs were consolidated into modules. Old flat dirs no longer exist at the repo root:
| Was (repo root) | Now |
|---|---|
provisioning/ |
archive/provisioning/ — superseded by KNELIAC project (/home/reachableceo/projects/KNELIAC) |
powerman/ console/ ups/ |
dcinfra/{powerman,console,ups}/ |
dns-cluster-setup/ switches/ dhcp/ |
netinfra/{dns-cluster-setup,switches,dhcp}/ |
perf/ |
proxmox/perf/ (+ proxmox/perf/scripts/) |
Current repo root: archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/ + logs/ returned-logs/.
WARNING:
README.md,AGENTS.md, anddocs/docmap.mdstill reference the OLD paths. These are being updated in this same session.
Network Switches
SNMP community: kn3lmgmt (read) / kn3lwrite (write). SNMP responds from
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
| IP | sysName | Model | Status |
|---|---|---|---|
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | UP (Dell, core, rack5) |
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | UP (rack3 mgmt TOR) |
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | UP (rack3 storage TOR) |
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | UP (was "unidentified" — now named) |
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | OFFLINE — no ping, no SNMP |
Ports open on switches: TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
161 (SNMP) responds with kn3lmgmt.
LACP / Cross-Rack Trunk (core-sw01 ch1)
- Operator reports LACP is fixed (was static
mode on, hash layer-2-3). - SNMP confirms ch1 ifIndex=49 is UP; ch3 ifIndex=51 also UP.
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
aggregate bundle speed via standard ifSpeed, so member count + hash
algorithm need console verification (conman requires a PTY — run manually):
conman -d pfv-tsys4:7890 -f pfv-core-sw01 # then: show interfaces status port-channel 1 # show running-config | begin port channel
Proxmox Hosts (7 standalone, managed via PDM)
All PVE 9.2.5 (tsys4 upgraded from 9.1.5 — resolves old STATUS item). Not clustered — standalone installs (corosync Expected votes=1 each).
| Host | RAM used/total | VMs | Uptime | Notes |
|---|---|---|---|---|
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
| pfv-tsys5 | 48% (46.8/97 GB) | 32 | 6d | sectestbed+preprod host |
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | 193 GB RAM, prod workloads |
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
Memory capacity goal (≤80%): PASS fleet-wide. Highest is tsys1 at 71%.
tsys5 Storage (RESOLVED)
Old STATUS item "NVMe + SSD not configured" is FIXED. All storage active:
| Disk | Mount | Use |
|---|---|---|
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
| local-lvm (1.67 TB thin) | LVM | 6.9% |
NFS from tsys4: D2, D3, D5 all mounted and active.
DNS / NTP / DHCP (netinfra)
pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
| Service | State | Detail |
|---|---|---|
| Technitium DNS | UP (Docker tsys-dns, 5d) |
Container, NOT systemd. Config: /home/localuser/services/technitium/config → /etc/dns. 167 zones. Port 5380 via docker-proxy. |
| Pi-hole | UP (healthy, 26h) | /dev/shm = 1.0 GB (operator hotfix applied). Healthcheck = dig pi.hole (DNS-resolution based). |
| NTP | UP (ntpsec, NOT chrony) |
Clock synchronized. |
| DHCP | UP (isc-dhcp-server) |
753 active leases (was 333). |
pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
WIPED. Reachable on Tailscale (host key changed, cleared from known_hosts)
but SSH key auth fails (Permission denied (publickey,password)). Needs
full rebuild as redundant node — see Pending.
Kubernetes (pfv-k8s) — DOWN
| Node | Tailscale IP | k3s API :6443 | Location |
|---|---|---|---|
| pfv-k8s-cnode1 | 100.125.134.53 | closed | VM 102 on tsys1 |
| pfv-k8s-cnode2 | 100.109.34.72 | OPEN | VM 705 on tsys7 |
| pfv-k8s-cnode3 | 100.106.222.18 | closed | VM 603 on tsys6 |
- No etcd quorum (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
- kubeconfig stale:
~/.kube/config.pfv-k8spoints to100.97.178.106:6443— an IP matching NO current cnode. Cluster must be rebuilt. - Worker nodes (5) all online on Tailscale, not joined: pfv-k8s-wnode-tsys3/5/6/7/9.
Monitoring / Observability Stack
| System | Host | Status | Access |
|---|---|---|---|
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
| Uptime Kuma | status.knownelement.com | UP | API key in ~/.secrets/api-keys.env; /metrics needs auth |
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
| Vault | vault.knownelement.com | Deployed, UNCONFIGURED | API keys migration = HIGH priority TODO |
Console Management (pfv-tsys4)
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
All 7 /dev/consoles/<name> symlinks present:
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3
conman requires a PTY — cannot be driven via stdin pipe from Crush. Use an interactive terminal, or the conman client directly.
Tailscale Fleet (80 nodes)
Three full environment tiers now exist, each mirroring the same service set:
tsys-*(production): tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms, tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02, tsys-umbrel, tsys-voippreprod-*: preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp, preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw, preprod-rancherplatform, preprod-siem, preprod-voipsectestbed-*: sectestbed-awx, sectestbed-ca, sectestbed-cloudron, sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode, sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter, sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve, sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9, pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02, pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp, kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3, stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
Note: Tailscale reports
--accept-routes is falseon this workstation.
Identified LAN Hosts (non-switch)
| IP | Identity | Evidence |
|---|---|---|
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
| 192.168.3.1 | gateway/router | Linux + xrdp |
Divergences vs STATUS.md (2026-07-30)
- VM count 28→63 — sectestbed (16) + preprod (8) + new prod VMs added
- DNS zones 124→167, DHCP leases 333→753
- tsys4 PVE 9.1.5→9.2.5 (was "behind fleet")
- tsys5 NVMe+SSD now configured (was "not configured")
- tsys-umbrel + tsys-sipwise deployed (new VMs)
- LACP core↔r3-stor operator reports fixed (was #1 known issue)
- k8s cluster now DOWN (was LIVE)
- netinfra-02 wiped (was redundant secondary)
- Technitium now Docker container (was systemd dnsServer)
- NTP is ntpsec (STATUS assumed chrony)
- Pi-hole /dev/shm = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
- Tailscale fleet 3× larger with 3 environment tiers
Pending / Recommended Next Actions
(From nextprompt — reference only; awaiting operator direction)
- netinfra-02 rebuild as redundant secondary (SSH key deploy + Technitium zone sync + DHCP failover-peer)
- k8s cluster rebuild (fix wiped cnode, restore quorum, join 5 wnodes)
- Pi-hole add gravity.db healthcheck to compose (current check is DNS-only)
- Vault migrate all API keys (HIGH priority)
- Monitoring: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
- Home Assistant: own all environmental monitoring/reporting
- Benchmarking: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
- Repo gardening: README/AGENTS/docmap path fixes (this session)