mrcharles
d159504121
Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).
Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
221 lines
10 KiB
Markdown
221 lines
10 KiB
Markdown
# Ground-Truth Audit — 2026-08-05
|
||
|
||
> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
|
||
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
|
||
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
|
||
>
|
||
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
|
||
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
|
||
> Proxmox `qm list`/`pvecm`, Docker/container inspection.
|
||
|
||
## TL;DR — What Changed Since 2026-07-30
|
||
|
||
The fleet grew significantly and the repository was restructured into a
|
||
module/subproject layout. Headline deltas:
|
||
|
||
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|
||
|--------|-----------|-----------|---|
|
||
| VMs (running) | 28 | **63** | +125% |
|
||
| DNS zones | 124 | **167** | +35% |
|
||
| DHCP leases | 333 | **753** | +126% |
|
||
| Proxmox hosts | 7 | 7 | — |
|
||
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
|
||
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
|
||
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
|
||
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
|
||
|
||
## Repository Restructure (DONE by operator)
|
||
|
||
Top-level dirs were consolidated into modules. **Old flat dirs no longer
|
||
exist** at the repo root:
|
||
|
||
| Was (repo root) | Now |
|
||
|-----------------|-----|
|
||
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
|
||
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
|
||
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
|
||
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
|
||
|
||
Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
|
||
|
||
> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
|
||
> the OLD paths. These are being updated in this same session.
|
||
|
||
## Network Switches
|
||
|
||
SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
|
||
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
|
||
|
||
| IP | sysName | Model | Status |
|
||
|----|---------|-------|--------|
|
||
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
|
||
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
|
||
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
|
||
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
|
||
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
|
||
|
||
**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
|
||
161 (SNMP) responds with `kn3lmgmt`.
|
||
|
||
### LACP / Cross-Rack Trunk (core-sw01 ch1)
|
||
|
||
- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
|
||
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
|
||
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
|
||
aggregate bundle speed via standard ifSpeed, so **member count + hash
|
||
algorithm need console verification** (conman requires a PTY — run manually):
|
||
```bash
|
||
conman -d pfv-tsys4:7890 -f pfv-core-sw01
|
||
# then: show interfaces status port-channel 1
|
||
# show running-config | begin port channel
|
||
```
|
||
|
||
## Proxmox Hosts (7 standalone, managed via PDM)
|
||
|
||
All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
|
||
**Not clustered** — standalone installs (corosync Expected votes=1 each).
|
||
|
||
| Host | RAM used/total | VMs | Uptime | Notes |
|
||
|------|---------------|-----|--------|-------|
|
||
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
|
||
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
|
||
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
|
||
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
|
||
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
|
||
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
|
||
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
|
||
|
||
**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
|
||
|
||
### tsys5 Storage (RESOLVED)
|
||
|
||
Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
|
||
|
||
| Disk | Mount | Use |
|
||
|------|-------|-----|
|
||
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
|
||
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
|
||
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
|
||
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
|
||
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
|
||
| local-lvm (1.67 TB thin) | LVM | 6.9% |
|
||
|
||
NFS from tsys4: D2, D3, D5 all mounted and active.
|
||
|
||
## DNS / NTP / DHCP (netinfra)
|
||
|
||
### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
|
||
|
||
| Service | State | Detail |
|
||
|---------|-------|--------|
|
||
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
|
||
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
|
||
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
|
||
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
|
||
|
||
### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
|
||
|
||
**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
|
||
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
|
||
full rebuild as redundant node — see Pending.
|
||
|
||
## Kubernetes (pfv-k8s) — DOWN
|
||
|
||
| Node | Tailscale IP | k3s API :6443 | Location |
|
||
|------|-------------|---------------|----------|
|
||
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
|
||
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
|
||
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
|
||
|
||
- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
|
||
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
|
||
— an IP matching NO current cnode. Cluster must be rebuilt.
|
||
- Worker nodes (5) all online on Tailscale, **not joined**:
|
||
pfv-k8s-wnode-tsys3/5/6/7/9.
|
||
|
||
## Monitoring / Observability Stack
|
||
|
||
| System | Host | Status | Access |
|
||
|--------|------|--------|--------|
|
||
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
|
||
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
|
||
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
|
||
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
|
||
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
|
||
|
||
## Console Management (pfv-tsys4)
|
||
|
||
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
|
||
All 7 `/dev/consoles/<name>` symlinks present:
|
||
|
||
```
|
||
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
|
||
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
|
||
pfv-r2-sw→ttyUSB3
|
||
```
|
||
|
||
**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
|
||
interactive terminal, or the conman client directly.
|
||
|
||
## Tailscale Fleet (80 nodes)
|
||
|
||
Three full environment tiers now exist, each mirroring the same service set:
|
||
|
||
- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
|
||
tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
|
||
tsys-umbrel, tsys-voip
|
||
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
|
||
preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
|
||
preprod-rancherplatform, preprod-siem, preprod-voip
|
||
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
|
||
sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
|
||
sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
|
||
sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
|
||
sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
|
||
|
||
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
|
||
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
|
||
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
|
||
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
|
||
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
|
||
|
||
> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
|
||
|
||
## Identified LAN Hosts (non-switch)
|
||
|
||
| IP | Identity | Evidence |
|
||
|----|----------|----------|
|
||
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
|
||
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
|
||
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
|
||
| 192.168.3.1 | gateway/router | Linux + xrdp |
|
||
|
||
## Divergences vs STATUS.md (2026-07-30)
|
||
|
||
1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
|
||
2. **DNS zones 124→167**, **DHCP leases 333→753**
|
||
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
|
||
4. **tsys5 NVMe+SSD** now configured (was "not configured")
|
||
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
|
||
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
|
||
7. **k8s cluster** now DOWN (was LIVE)
|
||
8. **netinfra-02** wiped (was redundant secondary)
|
||
9. **Technitium** now Docker container (was systemd dnsServer)
|
||
10. **NTP** is ntpsec (STATUS assumed chrony)
|
||
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
|
||
12. **Tailscale fleet** 3× larger with 3 environment tiers
|
||
|
||
## Pending / Recommended Next Actions
|
||
|
||
*(From `nextprompt` — reference only; awaiting operator direction)*
|
||
|
||
- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
|
||
zone sync + DHCP failover-peer)
|
||
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
|
||
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
|
||
- **Vault** migrate all API keys (HIGH priority)
|
||
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
|
||
- **Home Assistant**: own all environmental monitoring/reporting
|
||
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
|
||
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)
|