KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-08-05.md
+8 -218
View File
@@ -1,220 +1,10 @@
# Ground-Truth Audit — 2026-08-05
# proxmox/docs/AUDIT-2026-08-05.md
> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
> Proxmox `qm list`/`pvecm`, Docker/container inspection.
## TL;DR — What Changed Since 2026-07-30
The fleet grew significantly and the repository was restructured into a
module/subproject layout. Headline deltas:
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|--------|-----------|-----------|---|
| VMs (running) | 28 | **63** | +125% |
| DNS zones | 124 | **167** | +35% |
| DHCP active leases | 333 | **98** (unique active) | 431 total active binding-state entries |
| Proxmox hosts | 7 | 7 | — |
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
## Repository Restructure (DONE by operator)
Top-level dirs were consolidated into modules. **Old flat dirs no longer
exist** at the repo root:
| Was (repo root) | Now |
|-----------------|-----|
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
> the OLD paths. These are being updated in this same session.
## Network Switches
SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
| IP | sysName | Model | Status |
|----|---------|-------|--------|
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
161 (SNMP) responds with `kn3lmgmt`.
### LACP / Cross-Rack Trunk (core-sw01 ch1)
- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
aggregate bundle speed via standard ifSpeed, so **member count + hash
algorithm need console verification** (conman requires a PTY — run manually):
```bash
conman -d pfv-tsys4:7890 -f pfv-core-sw01
# then: show interfaces status port-channel 1
# show running-config | begin port channel
```
## Proxmox Hosts (7 standalone, managed via PDM)
All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
**Not clustered** — standalone installs (corosync Expected votes=1 each).
| Host | RAM used/total | VMs | Uptime | Notes |
|------|---------------|-----|--------|-------|
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
### tsys5 Storage (RESOLVED)
Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
| Disk | Mount | Use |
|------|-------|-----|
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
| local-lvm (1.67 TB thin) | LVM | 6.9% |
NFS from tsys4: D2, D3, D5 all mounted and active.
## DNS / NTP / DHCP (netinfra)
### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
| Service | State | Detail |
|---------|-------|--------|
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
full rebuild as redundant node — see Pending.
## Kubernetes (pfv-k8s) — DOWN
| Node | Tailscale IP | k3s API :6443 | Location |
|------|-------------|---------------|----------|
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
— an IP matching NO current cnode. Cluster must be rebuilt.
- Worker nodes (5) all online on Tailscale, **not joined**:
pfv-k8s-wnode-tsys3/5/6/7/9.
## Monitoring / Observability Stack
| System | Host | Status | Access |
|--------|------|--------|--------|
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
## Console Management (pfv-tsys4)
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
All 7 `/dev/consoles/<name>` symlinks present:
```
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3
```
**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
interactive terminal, or the conman client directly.
## Tailscale Fleet (80 nodes)
Three full environment tiers now exist, each mirroring the same service set:
- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
tsys-umbrel, tsys-voip
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
preprod-rancherplatform, preprod-siem, preprod-voip
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
## Identified LAN Hosts (non-switch)
| IP | Identity | Evidence |
|----|----------|----------|
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
| 192.168.3.1 | gateway/router | Linux + xrdp |
## Divergences vs STATUS.md (2026-07-30)
1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
2. **DNS zones 124→167**, **DHCP leases 333→753**
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
4. **tsys5 NVMe+SSD** now configured (was "not configured")
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
7. **k8s cluster** now DOWN (was LIVE)
8. **netinfra-02** wiped (was redundant secondary)
9. **Technitium** now Docker container (was systemd dnsServer)
10. **NTP** is ntpsec (STATUS assumed chrony)
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
12. **Tailscale fleet** 3× larger with 3 environment tiers
## Pending / Recommended Next Actions
*(From `nextprompt` — reference only; awaiting operator direction)*
- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
zone sync + DHCP failover-peer)
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
- **Vault** migrate all API keys (HIGH priority)
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
- **Home Assistant**: own all environmental monitoring/reporting
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)
> **Audit snapshot 2026-08-05 (current authoritative)**
>
> **Read it here:** https://community.turnsys.com/t/298
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*