diff --git a/AGENTS.md b/AGENTS.md index ea80db3..87c8513 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,33 +8,27 @@ lives in subdirectories. ## Repository Layout ``` -provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA) -tests/ Test suite + VM validation harness -dns-cluster-setup/ Technitium DNS cluster replication -k8s/ k3s cluster setup scripts (3-node HA over Tailscale) + docs/ -powerman/ Cyclades PM10i PDU management via powerman -console/ Serial console management (ser2net + conman) for switches -ups/ UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 -perf/ Proxmox perf tuning, fleet audit, iperf -proxmox/ Proxmox fleet docs (hardware audit, capacity, k8s host planning) -netinfra/ pfv-netinfra-01/02 DNS/NTP/DHCP setup -switches/ Switch configuration captures -awx/ Ansible AWX deployment (k3s + AWX Operator) -docs/ Server-build docs, docmap index, and archive -vendor/ Vendored KNELShellFramework +dcinfra/ Data-center infra: PDU (powerman), serial console (console), UPS (ups) +netinfra/ DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup); + switch captures (switches); DHCP config (dhcp) +k8s/ k3s cluster setup scripts (HA control plane over Tailscale) + docs/ +proxmox/ Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/) +awx/ Ansible AWX deployment (k3s + AWX Operator) +tests/ Test suite + VM validation harness + remote.sh SSH chokepoint +docs/ Server-build docs, docmap index, and archive +archive/ Historical/superseded code (provisioning -> replaced by KNELIAC project) +vendor/ Vendored KNELShellFramework ``` -- **Self-locating scripts**: All provisioning scripts derive their own - location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run - from anywhere. -- **Local configs are the source of truth**: Files in - [`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with - `cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads. -- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh` +- **Server provisioning moved to KNELIAC**: The + [`archive/provisioning/`](archive/provisioning/) tree is historical. Active + server provisioning lives in the **KNELIAC** project at + `/home/reachableceo/projects/KNELIAC`. +- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh` extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks. - **SSH in Crush**: Direct ssh/scp is blocked. Use - [`tests/remote.sh`](tests/remote.sh) or - [`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh). + [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or + [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts). ## Git Policy @@ -90,20 +84,17 @@ vendor/ Vendored KNELShellFramework | Script | Purpose | |--------|---------| -| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning | +| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here | +| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) | | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM | | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite | -| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication | +| [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication | | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane | -| [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman | -| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman | -| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring | -| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings | -| [`perf/scripts/probe-storage.sh`](perf/scripts/probe-storage.sh) | Read-only disk/mount/export/SMART audit | -| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit | -| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) | -| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) | -| [`perf/scripts/probe-drift.sh`](perf/scripts/probe-drift.sh) | Read-only fleet drift probe (packages, services, tuning, SSH, security) | +| [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman | +| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman | +| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring | +| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings | +| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift | ## Key Docs diff --git a/README.md b/README.md index 9d4b456..08cfb1d 100644 --- a/README.md +++ b/README.md @@ -8,19 +8,14 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste | Directory | Description | |-----------|-------------| -| [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) | -| [`tests/`](tests/) | Test suite + VM validation harness | -| [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts | -| [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs | -| [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | -| [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 | -| [`ups/`](ups/) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale | -| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics | -| [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning | -| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts | -| [`switches/`](switches/) | Switch configuration captures | -| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | +| [`dcinfra/`](dcinfra/) | Data-center infrastructure: PDU (`powerman/`), serial console (`console/`), UPS (`ups/`) | +| [`netinfra/`](netinfra/) | DNS/NTP/DHCP setup + audit scripts, DNS cluster replication (`dns-cluster-setup/`), switch captures (`switches/`), DHCP config (`dhcp/`) | +| [`k8s/`](k8s/) | k3s cluster setup scripts (HA control plane over Tailscale) + co-located docs | +| [`proxmox/`](proxmox/) | Proxmox fleet docs (hardware audit, capacity, storage, k8s host planning) + performance tuning (`perf/`) | +| [`awx/`](awx/) | Ansible AWX deployment (k3s + AWX Operator) | +| [`tests/`](tests/) | Test suite + VM validation harness + `remote.sh` SSH chokepoint | | [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive | +| [`archive/`](archive/) | Historical/superseded code (provisioning → replaced by KNELIAC project) | | [`vendor/`](vendor/) | Vendored KNELShellFramework | ## Quick Start diff --git a/STATUS.md b/STATUS.md index 804f333..f7a9e24 100644 --- a/STATUS.md +++ b/STATUS.md @@ -3,19 +3,24 @@ > **Human read-only. Agents maintain this file automatically after each work > session.** Do not edit by hand — the next agent run will overwrite it. -> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute) +> **Last updated:** 2026-08-05 by Crush (GLM-5.2) — full read-only ground-truth +> re-orientation (nmap + SNMP + SSH + API queries). > -> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) -> — network topology, switch LAG configs, corrected storage topology, VM config -> audit, fleet consistency, cross-rack NFS root-cause analysis. +> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) +> — repo restructure, 4 switches up / 1 down, 63 VMs, 167 DNS zones, 753 DHCP +> leases, k8s cluster DOWN (cnode wiped), netinfra-02 wiped, tsys5 storage +> configured, tsys4 upgraded, 3-tier Tailscale fleet (80 nodes). > Capacity targets are codified in > [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM -> steady-state, ~50% idle CPU. +> steady-state (fleet PASSES, highest tsys1 at 71%). -## Current State: STABLE +## Current State: MIXED (infra stable; k8s + netinfra-02 need rebuild) -The merged PFVCluster repo is fully operational across provisioning, DNS -infrastructure, Proxmox cluster ops, and k8s control plane. +Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack +are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum) +and **netinfra-02 is wiped** (needs rebuild as redundant secondary). Repo was +restructured into a module/subproject layout (see +[`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)). ## Completed Work @@ -34,7 +39,16 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [x] Zone replication via rsync systemd timer (every 60s) - [x] Credentials + 2FA replicated identically to production - [x] Both LAN IPs resolve knel.net device names + recurse externally -- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes. +- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, **753 active leases** (was 333). DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes. +- [ ] **netinfra-02 WIPED 2026-08-05** (storage move accident). Reachable on + Tailscale (100.71.171.20) but SSH key auth fails — needs full rebuild as + redundant secondary (key deploy + zone sync + DHCP failover-peer). +- [x] **Technitium now runs as Docker container** (`tsys-dns`) on netinfra-01, + not systemd dnsServer. Config at `/home/localuser/services/technitium/config`. + **167 zones** (was 124). NTP = `ntpsec` (not chrony). +- [x] **Pi-hole `/dev/shm` = 1.0 GB** (operator hotfix, was too small → prod + outage). Healthcheck = `dig pi.hole` (DNS-based; gravity.db-specific check + not yet added). ### Proxmox Cluster Ops - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9) @@ -53,8 +67,11 @@ infrastructure, Proxmox cluster ops, and k8s control plane. - [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured - [ ] tsys2: pending rebuild from Win10 to Proxmox -### Kubernetes Control Plane (k3s HA — LIVE) -- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd) +### Kubernetes Control Plane (k3s HA — DOWN, needs rebuild) +- [ ] **Cluster DOWN 2026-08-05**: operator wiped a cnode + shut cnodes down + (100% RAM on one). cnode2 has k3s API up, cnode1/cnode3 do not — no etcd + quorum. Kubeconfig points to stale IP (100.97.178.106, matches no cnode). +- [x] Was: 3-node k3s HA control plane (cnode1/2/3, embedded etcd, Tailscale) - [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs - [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads) - [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, diff --git a/docs/docmap.md b/docs/docmap.md index c4775de..d5aa13a 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -3,7 +3,7 @@ > **Index of all documentation in this repo.** Agents must update this file > whenever a doc is added, removed, or substantively changed. -> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute) +> **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra) ## Kubernetes Architecture ([`k8s/`](k8s/)) @@ -28,23 +28,25 @@ Fleet operations, hardware, performance tuning, storage architecture. | [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 | | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 | +| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | Prior audit (network + storage + compute + switches). **Superseded by AUDIT-2026-08-05** | 2026-07-30 | +| [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) | **Current & authoritative.** Read-only ground-truth via nmap/SNMP/SSH/API: 4 switches up / 1 down, 63 VMs, 167 zones, 753 DHCP leases, k8s DOWN, netinfra-02 wiped, repo restructure, 3-tier Tailscale fleet | 2026-08-05 | | [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | +| [`proxmox/docs/NAMING-POLICY.md`](proxmox/docs/NAMING-POLICY.md) | Canonical naming policy: VM name = source of truth (DNS/Tailscale/monitoring must match) | 2026-07-31 | +| [`proxmox/docs/PHYSICAL-SYSTEMS.md`](proxmox/docs/PHYSICAL-SYSTEMS.md) | Physical infrastructure: Pi's, network devices, UniFi APs, SNMP-discovered hosts | 2026-07-31 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | -## Server Build ([`server-build/`](server-build/)) +## Server Build ([`docs/server-build/`](server-build/)) Server provisioning, security hardening, DNS/NTP configuration. | Document | Description | Last Reviewed | |----------|-------------|---------------| -| [`server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 | -| [`server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 | -| [`server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 | -| [`server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 | -| [`server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 | +| [`docs/server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 | +| [`docs/server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 | +| [`docs/server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 | +| [`docs/server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 | +| [`docs/server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 | ## Operational Guides (outside docs/) @@ -52,11 +54,11 @@ Server provisioning, security hardening, DNS/NTP configuration. |----------|-------------|---------------| | [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 | | [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 | -| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | -| [`../console/README.md`](../console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 | -| [`../ups/README.md`](../ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 | +| [`../dcinfra/powerman/README.md`](../dcinfra/powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | +| [`../dcinfra/console/README.md`](../dcinfra/console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 | +| [`../dcinfra/ups/README.md`](../dcinfra/ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | -| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | +| [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | | [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 | | [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 | diff --git a/proxmox/docs/AUDIT-2026-08-05.md b/proxmox/docs/AUDIT-2026-08-05.md new file mode 100644 index 0000000..7831b94 --- /dev/null +++ b/proxmox/docs/AUDIT-2026-08-05.md @@ -0,0 +1,220 @@ +# Ground-Truth Audit — 2026-08-05 + +> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only +> network discovery (nmap, SNMP, SSH, API queries). Supersedes +> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state. +> +> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP +> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`, +> Proxmox `qm list`/`pvecm`, Docker/container inspection. + +## TL;DR — What Changed Since 2026-07-30 + +The fleet grew significantly and the repository was restructured into a +module/subproject layout. Headline deltas: + +| Metric | 2026-07-30 | 2026-08-05 | Δ | +|--------|-----------|-----------|---| +| VMs (running) | 28 | **63** | +125% | +| DNS zones | 124 | **167** | +35% | +| DHCP leases | 333 | **753** | +126% | +| Proxmox hosts | 7 | 7 | — | +| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded | +| Tailscale nodes | ~30 | **80** | full 3-tier env mirror | +| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum | +| netinfra-02 | redundant secondary | **WIPED** | needs rebuild | + +## Repository Restructure (DONE by operator) + +Top-level dirs were consolidated into modules. **Old flat dirs no longer +exist** at the repo root: + +| Was (repo root) | Now | +|-----------------|-----| +| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) | +| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` | +| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` | +| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) | + +Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`. + +> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference +> the OLD paths. These are being updated in this same session. + +## Network Switches + +SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from +the workstation LAN (no ACL — earlier "filtered" was just a wrong community). + +| IP | sysName | Model | Status | +|----|---------|-------|--------| +| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) | +| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) | +| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) | +| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) | +| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP | + +**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP +161 (SNMP) responds with `kn3lmgmt`. + +### LACP / Cross-Rack Trunk (core-sw01 ch1) + +- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3). +- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP. +- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report + aggregate bundle speed via standard ifSpeed, so **member count + hash + algorithm need console verification** (conman requires a PTY — run manually): + ```bash + conman -d pfv-tsys4:7890 -f pfv-core-sw01 + # then: show interfaces status port-channel 1 + # show running-config | begin port channel + ``` + +## Proxmox Hosts (7 standalone, managed via PDM) + +All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item). +**Not clustered** — standalone installs (corosync Expected votes=1 each). + +| Host | RAM used/total | VMs | Uptime | Notes | +|------|---------------|-----|--------|-------| +| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet | +| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized | +| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 | +| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host | +| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 | +| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads | +| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode | + +**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%. + +### tsys5 Storage (RESOLVED) + +Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active: + +| Disk | Mount | Use | +|------|-------|-----| +| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used | +| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used | +| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% | +| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% | +| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% | +| local-lvm (1.67 TB thin) | LVM | 6.9% | + +NFS from tsys4: D2, D3, D5 all mounted and active. + +## DNS / NTP / DHCP (netinfra) + +### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72) + +| Service | State | Detail | +|---------|-------|--------| +| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. | +| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). | +| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. | +| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). | + +### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20) + +**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts) +but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs +full rebuild as redundant node — see Pending. + +## Kubernetes (pfv-k8s) — DOWN + +| Node | Tailscale IP | k3s API :6443 | Location | +|------|-------------|---------------|----------| +| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 | +| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 | +| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 | + +- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening. +- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443` + — an IP matching NO current cnode. Cluster must be rebuilt. +- Worker nodes (5) all online on Tailscale, **not joined**: + pfv-k8s-wnode-tsys3/5/6/7/9. + +## Monitoring / Observability Stack + +| System | Host | Status | Access | +|--------|------|--------|--------| +| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create | +| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth | +| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD | +| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live | +| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO | + +## Console Management (pfv-tsys4) + +ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93. +All 7 `/dev/consoles/` symlinks present: + +``` +pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6 +pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0 +pfv-r2-sw→ttyUSB3 +``` + +**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an +interactive terminal, or the conman client directly. + +## Tailscale Fleet (80 nodes) + +Three full environment tiers now exist, each mirroring the same service set: + +- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms, + tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02, + tsys-umbrel, tsys-voip +- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp, + preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw, + preprod-rancherplatform, preprod-siem, preprod-voip +- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron, + sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode, + sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter, + sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve, + sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip + +Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9, +pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02, +pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp, +kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3, +stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints). + +> **Note:** Tailscale reports `--accept-routes is false` on this workstation. + +## Identified LAN Hosts (non-switch) + +| IP | Identity | Evidence | +|----|----------|----------| +| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp | +| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind | +| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH | +| 192.168.3.1 | gateway/router | Linux + xrdp | + +## Divergences vs STATUS.md (2026-07-30) + +1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added +2. **DNS zones 124→167**, **DHCP leases 333→753** +3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet") +4. **tsys5 NVMe+SSD** now configured (was "not configured") +5. **tsys-umbrel + tsys-sipwise** deployed (new VMs) +6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue) +7. **k8s cluster** now DOWN (was LIVE) +8. **netinfra-02** wiped (was redundant secondary) +9. **Technitium** now Docker container (was systemd dnsServer) +10. **NTP** is ntpsec (STATUS assumed chrony) +11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based) +12. **Tailscale fleet** 3× larger with 3 environment tiers + +## Pending / Recommended Next Actions + +*(From `nextprompt` — reference only; awaiting operator direction)* + +- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium + zone sync + DHCP failover-peer) +- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes) +- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only) +- **Vault** migrate all API keys (HIGH priority) +- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping +- **Home Assistant**: own all environmental monitoring/reporting +- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster +- **Repo gardening**: README/AGENTS/docmap path fixes (this session)