# Ground-Truth Audit — 2026-08-05

> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
>
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
> Proxmox `qm list`/`pvecm`, Docker/container inspection.

## TL;DR — What Changed Since 2026-07-30

The fleet grew significantly and the repository was restructured into a
module/subproject layout. Headline deltas:

| Metric | 2026-07-30 | 2026-08-05 | Δ |
|--------|-----------|-----------|---|
| VMs (running) | 28 | **63** | +125% |
| DNS zones | 124 | **167** | +35% |
| DHCP active leases | 333 | **98** (unique active) | 431 total active binding-state entries |
| Proxmox hosts | 7 | 7 | — |
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |

## Repository Restructure (DONE by operator)

Top-level dirs were consolidated into modules. **Old flat dirs no longer
exist** at the repo root:

| Was (repo root) | Now |
|-----------------|-----|
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |

Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.

> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
> the OLD paths. These are being updated in this same session.

## Network Switches

SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).

| IP | sysName | Model | Status |
|----|---------|-------|--------|
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |

**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
161 (SNMP) responds with `kn3lmgmt`.

### LACP / Cross-Rack Trunk (core-sw01 ch1)

- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
  aggregate bundle speed via standard ifSpeed, so **member count + hash
  algorithm need console verification** (conman requires a PTY — run manually):
  ```bash
  conman -d pfv-tsys4:7890 -f pfv-core-sw01
  # then: show interfaces status port-channel 1
  #       show running-config | begin port channel
  ```

## Proxmox Hosts (7 standalone, managed via PDM)

All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
**Not clustered** — standalone installs (corosync Expected votes=1 each).

| Host | RAM used/total | VMs | Uptime | Notes |
|------|---------------|-----|--------|-------|
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |

**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.

### tsys5 Storage (RESOLVED)

Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:

| Disk | Mount | Use |
|------|-------|-----|
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
| local-lvm (1.67 TB thin) | LVM | 6.9% |

NFS from tsys4: D2, D3, D5 all mounted and active.

## DNS / NTP / DHCP (netinfra)

### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)

| Service | State | Detail |
|---------|-------|--------|
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |

### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)

**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
full rebuild as redundant node — see Pending.

## Kubernetes (pfv-k8s) — DOWN

| Node | Tailscale IP | k3s API :6443 | Location |
|------|-------------|---------------|----------|
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |

- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
  — an IP matching NO current cnode. Cluster must be rebuilt.
- Worker nodes (5) all online on Tailscale, **not joined**:
  pfv-k8s-wnode-tsys3/5/6/7/9.

## Monitoring / Observability Stack

| System | Host | Status | Access |
|--------|------|--------|--------|
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |

## Console Management (pfv-tsys4)

ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
All 7 `/dev/consoles/<name>` symlinks present:

```
pfv-core-sw01→ttyUSB2  pfv-tor3-mgmt→ttyUSB1  pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5  pfv-r2-tor-top→ttyUSB7  subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3
```

**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
interactive terminal, or the conman client directly.

## Tailscale Fleet (80 nodes)

Three full environment tiers now exist, each mirroring the same service set:

- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
  tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
  tsys-umbrel, tsys-voip
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
  preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
  preprod-rancherplatform, preprod-siem, preprod-voip
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
  sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
  sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
  sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
  sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip

Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).

> **Note:** Tailscale reports `--accept-routes is false` on this workstation.

## Identified LAN Hosts (non-switch)

| IP | Identity | Evidence |
|----|----------|----------|
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
| 192.168.3.1 | gateway/router | Linux + xrdp |

## Divergences vs STATUS.md (2026-07-30)

1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
2. **DNS zones 124→167**, **DHCP leases 333→753**
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
4. **tsys5 NVMe+SSD** now configured (was "not configured")
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
7. **k8s cluster** now DOWN (was LIVE)
8. **netinfra-02** wiped (was redundant secondary)
9. **Technitium** now Docker container (was systemd dnsServer)
10. **NTP** is ntpsec (STATUS assumed chrony)
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
12. **Tailscale fleet** 3× larger with 3 environment tiers

## Pending / Recommended Next Actions

*(From `nextprompt` — reference only; awaiting operator direction)*

- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
  zone sync + DHCP failover-peer)
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
- **Vault** migrate all API keys (HIGH priority)
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
- **Home Assistant**: own all environmental monitoring/reporting
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)