KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
33b5c76bec9c89f19883f259f7b8f602c5524663
PFVCluster/proxmox/docs/AUDIT-2026-08-05.md
T
mrcharles 33b5c76bec feat(netinfra): rebuild netinfra-02 as redundant DNS/DHCP + harden Pi-hole 
netinfra-02 was accidentally wiped during storage moves. Rebuilt from
netinfra-01 config: Docker + Technitium (167 zones, rsync timer every 60s),
Pi-hole v6 (healthy, gravity.db validated), NTP container, ISC DHCP failover
secondary (load-balancing with primary, 99+84 active leases).

Pi-hole hardened on BOTH nodes with defense-in-depth against the
gravity.db corruption outage:
- gravity-validate.sh: pre-start SQLite header check, auto-removes corrupt DB
- Healthcheck: validates DNS resolution + gravity.db file integrity
- autoheal container: restarts unhealthy containers automatically
- Root cause (/dev/shm too small) already fixed by operator (shm_size: 1024M)

Also corrected DHCP lease count in audit (98 unique active, not 753 raw blocks).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-08-06 09:18:45 -05:00

221 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Ground-Truth Audit — 2026-08-05
> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
>
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
> Proxmox `qm list`/`pvecm`, Docker/container inspection.
## TL;DR — What Changed Since 2026-07-30
The fleet grew significantly and the repository was restructured into a
module/subproject layout. Headline deltas:
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|--------|-----------|-----------|---|
| VMs (running) | 28 | **63** | +125% |
| DNS zones | 124 | **167** | +35% |
| DHCP active leases | 333 | **98** (unique active) | 431 total active binding-state entries |
| Proxmox hosts | 7 | 7 | — |
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
## Repository Restructure (DONE by operator)
Top-level dirs were consolidated into modules. **Old flat dirs no longer
exist** at the repo root:
| Was (repo root) | Now |
|-----------------|-----|
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
> the OLD paths. These are being updated in this same session.
## Network Switches
SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
| IP | sysName | Model | Status |
|----|---------|-------|--------|
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
161 (SNMP) responds with `kn3lmgmt`.
### LACP / Cross-Rack Trunk (core-sw01 ch1)
- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
aggregate bundle speed via standard ifSpeed, so **member count + hash
algorithm need console verification** (conman requires a PTY — run manually):
```bash
conman -d pfv-tsys4:7890 -f pfv-core-sw01
# then: show interfaces status port-channel 1
# show running-config | begin port channel
```
## Proxmox Hosts (7 standalone, managed via PDM)
All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
**Not clustered** — standalone installs (corosync Expected votes=1 each).
| Host | RAM used/total | VMs | Uptime | Notes |
|------|---------------|-----|--------|-------|
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
### tsys5 Storage (RESOLVED)
Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
| Disk | Mount | Use |
|------|-------|-----|
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
| local-lvm (1.67 TB thin) | LVM | 6.9% |
NFS from tsys4: D2, D3, D5 all mounted and active.
## DNS / NTP / DHCP (netinfra)
### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
| Service | State | Detail |
|---------|-------|--------|
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
full rebuild as redundant node — see Pending.
## Kubernetes (pfv-k8s) — DOWN
| Node | Tailscale IP | k3s API :6443 | Location |
|------|-------------|---------------|----------|
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
— an IP matching NO current cnode. Cluster must be rebuilt.
- Worker nodes (5) all online on Tailscale, **not joined**:
pfv-k8s-wnode-tsys3/5/6/7/9.
## Monitoring / Observability Stack
| System | Host | Status | Access |
|--------|------|--------|--------|
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
## Console Management (pfv-tsys4)
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
All 7 `/dev/consoles/<name>` symlinks present:
```
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3
```
**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
interactive terminal, or the conman client directly.
## Tailscale Fleet (80 nodes)
Three full environment tiers now exist, each mirroring the same service set:
- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
tsys-umbrel, tsys-voip
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
preprod-rancherplatform, preprod-siem, preprod-voip
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
## Identified LAN Hosts (non-switch)
| IP | Identity | Evidence |
|----|----------|----------|
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
| 192.168.3.1 | gateway/router | Linux + xrdp |
## Divergences vs STATUS.md (2026-07-30)
1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
2. **DNS zones 124→167**, **DHCP leases 333→753**
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
4. **tsys5 NVMe+SSD** now configured (was "not configured")
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
7. **k8s cluster** now DOWN (was LIVE)
8. **netinfra-02** wiped (was redundant secondary)
9. **Technitium** now Docker container (was systemd dnsServer)
10. **NTP** is ntpsec (STATUS assumed chrony)
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
12. **Tailscale fleet** 3× larger with 3 environment tiers
## Pending / Recommended Next Actions
*(From `nextprompt` — reference only; awaiting operator direction)*
- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
zone sync + DHCP failover-peer)
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
- **Vault** migrate all API keys (HIGH priority)
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
- **Home Assistant**: own all environmental monitoring/reporting
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)
Reference in New Issue View Git Blame Copy Permalink