KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d159504121c076c82ff662242cdbc439c832aa2b
PFVCluster/proxmox/docs/AUDIT-2026-08-05.md
T
mrcharles d159504121 docs(audit): full ground-truth re-orientation + sync index files to module layout 
Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).

Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-06 00:49:30 -05:00

10 KiB
Raw Blame History

Ground-Truth Audit — 2026-08-05

Authoritative fleet audit performed by Crush (GLM-5.2) via read-only network discovery (nmap, SNMP, SSH, API queries). Supersedes AUDIT-2026-07-30.md for all fleet state.

Method: Tailscale status, nmap ping sweep (192.168.0-3.0/24), SNMP walks (kn3lmgmt community), SSH via tests/remote.sh + netinfra/dns-cluster-setup/remote-dns.sh, Proxmox qm list/pvecm, Docker/container inspection.

TL;DR — What Changed Since 2026-07-30

The fleet grew significantly and the repository was restructured into a module/subproject layout. Headline deltas:

Metric 2026-07-30 2026-08-05 Δ
VMs (running) 28 63 +125%
DNS zones 124 167 +35%
DHCP leases 333 753 +126%
Proxmox hosts 7 7 —
PVE version (fleet) mixed (9.1.5–9.2.5) all 9.2.5 tsys4 upgraded
Tailscale nodes ~30 80 full 3-tier env mirror
k8s cluster LIVE (3-node HA) DOWN cnode wiped, no quorum
netinfra-02 redundant secondary WIPED needs rebuild

Repository Restructure (DONE by operator)

Top-level dirs were consolidated into modules. Old flat dirs no longer exist at the repo root:

Was (repo root) Now
provisioning/ archive/provisioning/ — superseded by KNELIAC project (/home/reachableceo/projects/KNELIAC)
powerman/ console/ ups/ dcinfra/{powerman,console,ups}/
dns-cluster-setup/ switches/ dhcp/ netinfra/{dns-cluster-setup,switches,dhcp}/
perf/ proxmox/perf/ (+ proxmox/perf/scripts/)

Current repo root: archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/ + logs/ returned-logs/.

WARNING: README.md, AGENTS.md, and docs/docmap.md still reference the OLD paths. These are being updated in this same session.

Network Switches

SNMP community: kn3lmgmt (read) / kn3lwrite (write). SNMP responds from the workstation LAN (no ACL — earlier "filtered" was just a wrong community).

IP sysName Model Status
192.168.0.12 pfv-core-sw01 PowerConnect 5448 UP (Dell, core, rack5)
192.168.0.7 pfv-r3-tor-mgmt Neyland 24T (PC5324) UP (rack3 mgmt TOR)
192.168.0.9 pfv-r3-tor-stor Neyland 24T (PC5324) UP (rack3 storage TOR)
192.168.0.8 subodev-torsw01 Neyland 24T (PC5324) UP (was "unidentified" — now named)
192.168.0.5 pfv-r2-tor-top (unknown) OFFLINE — no ping, no SNMP

Ports open on switches: TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP 161 (SNMP) responds with kn3lmgmt.

LACP / Cross-Rack Trunk (core-sw01 ch1)

  • Operator reports LACP is fixed (was static mode on, hash layer-2-3).
  • SNMP confirms ch1 ifIndex=49 is UP; ch3 ifIndex=51 also UP.
  • ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report aggregate bundle speed via standard ifSpeed, so member count + hash algorithm need console verification (conman requires a PTY — run manually): 
    conman -d pfv-tsys4:7890 -f pfv-core-sw01
# then: show interfaces status port-channel 1
#       show running-config | begin port channel

Proxmox Hosts (7 standalone, managed via PDM)

All PVE 9.2.5 (tsys4 upgraded from 9.1.5 — resolves old STATUS item). Not clustered — standalone installs (corosync Expected votes=1 each).

Host RAM used/total VMs Uptime Notes
pfv-tsys1 71% (22.7/32 GB) 8 6d Highest RAM use in fleet
pfv-tsys3 12% (3.7/32 GB) 2 9d under-utilized
pfv-tsys4 18% (2.8/16 GB) 2 5d storage server, kernel 7.0.14-8
pfv-tsys5 48% (46.8/97 GB) 32 6d sectestbed+preprod host
pfv-tsys6 14% (18/129 GB) 5 10d 129 GB RAM, wnode+cnode3
pfv-tsys7 30% (58.8/193 GB) 11 10d 193 GB RAM, prod workloads
pfv-tsys9 30% (7.2/24 GB) 3 9d netinfra-02 + wnode

Memory capacity goal (≤80%): PASS fleet-wide. Highest is tsys1 at 71%.

tsys5 Storage (RESOLVED)

Old STATUS item "NVMe + SSD not configured" is FIXED. All storage active:

Disk Mount Use
nvme0n1 (CT500P1 500GB) /mnt/nvme (PVE: NVME dir) 49% used
sdb (SK hynix 512GB SSD) /mnt/ssd2 (PVE: ssd2 dir) 14% used
sdd (Samsung 860 PRO 256GB) /mnt/pfv-tsys5/ssd (PVE: SSD dir) 1%
sde (ST1000VN002 1TB) /mnt/pfv-tsys5/S1 6%
sdf (ST1000VN002 1TB) /mnt/pfv-tsys5/S2 7%
local-lvm (1.67 TB thin) LVM 6.9%

NFS from tsys4: D2, D3, D5 all mounted and active.

DNS / NTP / DHCP (netinfra)

pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)

Service State Detail
Technitium DNS UP (Docker tsys-dns, 5d) Container, NOT systemd. Config: /home/localuser/services/technitium/config → /etc/dns. 167 zones. Port 5380 via docker-proxy.
Pi-hole UP (healthy, 26h) /dev/shm = 1.0 GB (operator hotfix applied). Healthcheck = dig pi.hole (DNS-resolution based).
NTP UP (ntpsec, NOT chrony) Clock synchronized.
DHCP UP (isc-dhcp-server) 753 active leases (was 333).

pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)

WIPED. Reachable on Tailscale (host key changed, cleared from known_hosts) but SSH key auth fails (Permission denied (publickey,password)). Needs full rebuild as redundant node — see Pending.

Kubernetes (pfv-k8s) — DOWN

Node Tailscale IP k3s API :6443 Location
pfv-k8s-cnode1 100.125.134.53 closed VM 102 on tsys1
pfv-k8s-cnode2 100.109.34.72 OPEN VM 705 on tsys7
pfv-k8s-cnode3 100.106.222.18 closed VM 603 on tsys6
  • No etcd quorum (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
  • kubeconfig stale: ~/.kube/config.pfv-k8s points to 100.97.178.106:6443 — an IP matching NO current cnode. Cluster must be rebuilt.
  • Worker nodes (5) all online on Tailscale, not joined: pfv-k8s-wnode-tsys3/5/6/7/9.

Monitoring / Observability Stack

System Host Status Access
LibreNMS tsys-librenms (100.86.204.77) UP (Tailscale) API needs token (401) — SSH in to create
Uptime Kuma status.knownelement.com UP API key in ~/.secrets/api-keys.env; /metrics needs auth
Beszel (fleet) deployed (per STATUS) metrics host TBD
Home Assistant (HAOS VM) configured (per STATUS) NUT integration live
Vault vault.knownelement.com Deployed, UNCONFIGURED API keys migration = HIGH priority TODO

Console Management (pfv-tsys4)

ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93. All 7 /dev/consoles/<name> symlinks present:

pfv-core-sw01→ttyUSB2  pfv-tor3-mgmt→ttyUSB1  pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5  pfv-r2-tor-top→ttyUSB7  subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3

conman requires a PTY — cannot be driven via stdin pipe from Crush. Use an interactive terminal, or the conman client directly.

Tailscale Fleet (80 nodes)

Three full environment tiers now exist, each mirroring the same service set:

  • tsys-* (production): tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms, tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02, tsys-umbrel, tsys-voip
  • preprod-*: preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp, preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw, preprod-rancherplatform, preprod-siem, preprod-voip
  • sectestbed-*: sectestbed-awx, sectestbed-ca, sectestbed-cloudron, sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode, sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter, sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve, sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip

Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9, pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02, pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp, kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3, stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).

Note: Tailscale reports --accept-routes is false on this workstation.

Identified LAN Hosts (non-switch)

IP Identity Evidence
192.168.0.100 stlpc-garage.knel.net Linux + Postfix + xrdp
192.168.0.250 Linux storage (NFS) SSH + rpcbind
192.168.1.166 pfv-stor1 (NAS) Samba smbd 4 + Golang HTTP + SSH
192.168.3.1 gateway/router Linux + xrdp

Divergences vs STATUS.md (2026-07-30)

  1. VM count 28→63 — sectestbed (16) + preprod (8) + new prod VMs added
  2. DNS zones 124→167, DHCP leases 333→753
  3. tsys4 PVE 9.1.5→9.2.5 (was "behind fleet")
  4. tsys5 NVMe+SSD now configured (was "not configured")
  5. tsys-umbrel + tsys-sipwise deployed (new VMs)
  6. LACP core↔r3-stor operator reports fixed (was #1 known issue)
  7. k8s cluster now DOWN (was LIVE)
  8. netinfra-02 wiped (was redundant secondary)
  9. Technitium now Docker container (was systemd dnsServer)
  10. NTP is ntpsec (STATUS assumed chrony)
  11. Pi-hole /dev/shm = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
  12. Tailscale fleet 3× larger with 3 environment tiers

Pending / Recommended Next Actions

(From nextprompt — reference only; awaiting operator direction)

  • netinfra-02 rebuild as redundant secondary (SSH key deploy + Technitium zone sync + DHCP failover-peer)
  • k8s cluster rebuild (fix wiped cnode, restore quorum, join 5 wnodes)
  • Pi-hole add gravity.db healthcheck to compose (current check is DNS-only)
  • Vault migrate all API keys (HIGH priority)
  • Monitoring: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
  • Home Assistant: own all environmental monitoring/reporting
  • Benchmarking: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
  • Repo gardening: README/AGENTS/docmap path fixes (this session)
Reference in New Issue View Git Blame Copy Permalink