KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: KNEL/PFVCluster:ea54beea703307e8dfb77af8396557a56b75f731
Branches Tags
KNEL/PFVCluster:main
KNEL/PFVCluster:1.0
..
compare: KNEL/PFVCluster:d159504121c076c82ff662242cdbc439c832aa2b
Branches Tags
KNEL/PFVCluster:main
KNEL/PFVCluster:1.0
2 Commits
ea54beea70 .. d159504121
Author SHA1 Message Date
mrcharles d159504121 docs(audit): full ground-truth re-orientation + sync index files to module layout 
Read-only audit (nmap/SNMP/SSH/API) reconciling the repo and fleet after the
operator's module/subproject restructure and infrastructure changes. New
authoritative audit doc captures: 4 switches up / 1 down (r2-tor-top offline),
63 VMs (was 28), 167 DNS zones (was 124), 753 DHCP leases (was 333), tsys5
storage now configured, tsys4 upgraded to 9.2.5, k8s cluster DOWN (cnode wiped,
no etcd quorum), netinfra-02 wiped, 3-tier Tailscale fleet (80 nodes).

Index files (README, AGENTS, docmap, STATUS) updated to the new layout
(provisioning→archive, powerman/console/ups→dcinfra, dns-cluster-setup/switches/
dhcp→netinfra, perf→proxmox/perf) and all stale root-level path references fixed.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
 2026-08-06 00:49:30 -05:00
mrcharles 04ed433e84 . 2026-08-01 20:31:13 -05:00
6 changed files with 296 additions and 73 deletions
Expand all files Collapse all files
AGENTS.md
+25 -34
View File
@@ -8,33 +8,27 @@ lives in subdirectories.
## Repository Layout ## Repository Layout
``` ```
provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA) dcinfra/ Data-center infra: PDU (powerman), serial console (console), UPS (ups)
tests/ Test suite + VM validation harness netinfra/ DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup);
dns-cluster-setup/ Technitium DNS cluster replication switch captures (switches); DHCP config (dhcp)
k8s/ k3s cluster setup scripts (3-node HA over Tailscale) + docs/ k8s/ k3s cluster setup scripts (HA control plane over Tailscale) + docs/
powerman/ Cyclades PM10i PDU management via powerman proxmox/ Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
console/ Serial console management (ser2net + conman) for switches awx/ Ansible AWX deployment (k3s + AWX Operator)
ups/ UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 tests/ Test suite + VM validation harness + remote.sh SSH chokepoint
perf/ Proxmox perf tuning, fleet audit, iperf docs/ Server-build docs, docmap index, and archive
proxmox/ Proxmox fleet docs (hardware audit, capacity, k8s host planning) archive/ Historical/superseded code (provisioning -> replaced by KNELIAC project)
netinfra/ pfv-netinfra-01/02 DNS/NTP/DHCP setup vendor/ Vendored KNELShellFramework
switches/ Switch configuration captures
awx/ Ansible AWX deployment (k3s + AWX Operator)
docs/ Server-build docs, docmap index, and archive
vendor/ Vendored KNELShellFramework
``` ```
- **Self-locating scripts**: All provisioning scripts derive their own - **Server provisioning moved to KNELIAC**: The
location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run [`archive/provisioning/`](archive/provisioning/) tree is historical. Active
from anywhere. server provisioning lives in the **KNELIAC** project at
- **Local configs are the source of truth**: Files in `/home/reachableceo/projects/KNELIAC`.
[`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with - **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
`cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks. extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
- **SSH in Crush**: Direct ssh/scp is blocked. Use - **SSH in Crush**: Direct ssh/scp is blocked. Use
[`tests/remote.sh`](tests/remote.sh) or [`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
[`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh). [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
## Git Policy ## Git Policy
@@ -90,20 +84,17 @@ vendor/ Vendored KNELShellFramework
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning | | [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM | | [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite | | [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication | | [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
| [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane | | [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
| [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman | | [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman |
| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman | | [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring | | [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings | | [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
| [`perf/scripts/probe-storage.sh`](perf/scripts/probe-storage.sh) | Read-only disk/mount/export/SMART audit | | [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift |
| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit |
| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) |
| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) |
| [`perf/scripts/probe-drift.sh`](perf/scripts/probe-drift.sh) | Read-only fleet drift probe (packages, services, tuning, SSH, security) |
## Key Docs ## Key Docs
README.md
+7 -12
View File
@@ -8,19 +8,14 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste
| Directory | Description | | Directory | Description |
|-----------|-------------| |-----------|-------------|
| [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) | | [`dcinfra/`](dcinfra/) | Data-center infrastructure: PDU (`powerman/`), serial console (`console/`), UPS (`ups/`) |
| [`tests/`](tests/) | Test suite + VM validation harness | | [`netinfra/`](netinfra/) | DNS/NTP/DHCP setup + audit scripts, DNS cluster replication (`dns-cluster-setup/`), switch captures (`switches/`), DHCP config (`dhcp/`) |
| [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts | | [`k8s/`](k8s/) | k3s cluster setup scripts (HA control plane over Tailscale) + co-located docs |
| [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs | | [`proxmox/`](proxmox/) | Proxmox fleet docs (hardware audit, capacity, storage, k8s host planning) + performance tuning (`perf/`) |
| [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | | [`awx/`](awx/) | Ansible AWX deployment (k3s + AWX Operator) |
| [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 | | [`tests/`](tests/) | Test suite + VM validation harness + `remote.sh` SSH chokepoint |
| [`ups/`](ups/) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale |
| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
| [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |
| [`switches/`](switches/) | Switch configuration captures |
| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) |
| [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive | | [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive |
| [`archive/`](archive/) | Historical/superseded code (provisioning → replaced by KNELIAC project) |
| [`vendor/`](vendor/) | Vendored KNELShellFramework | | [`vendor/`](vendor/) | Vendored KNELShellFramework |
## Quick Start ## Quick Start
STATUS.md
+28 -11
View File
@@ -3,19 +3,24 @@
> **Human read-only. Agents maintain this file automatically after each work > **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it. > session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute) > **Last updated:** 2026-08-05 by Crush (GLM-5.2) — full read-only ground-truth
> re-orientation (nmap + SNMP + SSH + API queries).
> >
> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) > **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)
> — network topology, switch LAG configs, corrected storage topology, VM config > — repo restructure, 4 switches up / 1 down, 63 VMs, 167 DNS zones, 753 DHCP
> audit, fleet consistency, cross-rack NFS root-cause analysis. > leases, k8s cluster DOWN (cnode wiped), netinfra-02 wiped, tsys5 storage
> configured, tsys4 upgraded, 3-tier Tailscale fleet (80 nodes).
> Capacity targets are codified in > Capacity targets are codified in
> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM > [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
> steady-state, ~50% idle CPU. > steady-state (fleet PASSES, highest tsys1 at 71%).
## Current State: STABLE ## Current State: MIXED (infra stable; k8s + netinfra-02 need rebuild)
The merged PFVCluster repo is fully operational across provisioning, DNS Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack
infrastructure, Proxmox cluster ops, and k8s control plane. are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum)
and **netinfra-02 is wiped** (needs rebuild as redundant secondary). Repo was
restructured into a module/subproject layout (see
[`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)).
## Completed Work ## Completed Work
@@ -34,7 +39,16 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
- [x] Zone replication via rsync systemd timer (every 60s) - [x] Zone replication via rsync systemd timer (every 60s)
- [x] Credentials + 2FA replicated identically to production - [x] Credentials + 2FA replicated identically to production
- [x] Both LAN IPs resolve knel.net device names + recurse externally - [x] Both LAN IPs resolve knel.net device names + recurse externally
- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes. - [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, **753 active leases** (was 333). DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
- [ ] **netinfra-02 WIPED 2026-08-05** (storage move accident). Reachable on
Tailscale (100.71.171.20) but SSH key auth fails — needs full rebuild as
redundant secondary (key deploy + zone sync + DHCP failover-peer).
- [x] **Technitium now runs as Docker container** (`tsys-dns`) on netinfra-01,
not systemd dnsServer. Config at `/home/localuser/services/technitium/config`.
**167 zones** (was 124). NTP = `ntpsec` (not chrony).
- [x] **Pi-hole `/dev/shm` = 1.0 GB** (operator hotfix, was too small → prod
outage). Healthcheck = `dig pi.hole` (DNS-based; gravity.db-specific check
not yet added).
### Proxmox Cluster Ops ### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9) - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
@@ -53,8 +67,11 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured - [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
- [ ] tsys2: pending rebuild from Win10 to Proxmox - [ ] tsys2: pending rebuild from Win10 to Proxmox
### Kubernetes Control Plane (k3s HA — LIVE) ### Kubernetes Control Plane (k3s HA — DOWN, needs rebuild)
- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd) - [ ] **Cluster DOWN 2026-08-05**: operator wiped a cnode + shut cnodes down
(100% RAM on one). cnode2 has k3s API up, cnode1/cnode3 do not — no etcd
quorum. Kubeconfig points to stale IP (100.97.178.106, matches no cnode).
- [x] Was: 3-node k3s HA control plane (cnode1/2/3, embedded etcd, Tailscale)
- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs - [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads) - [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum, - [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
docs/docmap.md
+15 -13
View File
@@ -3,7 +3,7 @@
> **Index of all documentation in this repo.** Agents must update this file > **Index of all documentation in this repo.** Agents must update this file
> whenever a doc is added, removed, or substantively changed. > whenever a doc is added, removed, or substantively changed.
> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute) > **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra)
## Kubernetes Architecture ([`k8s/`](k8s/)) ## Kubernetes Architecture ([`k8s/`](k8s/))
@@ -28,23 +28,25 @@ Fleet operations, hardware, performance tuning, storage architecture.
| [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 | | [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 |
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 | | [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | Prior audit (network + storage + compute + switches). **Superseded by AUDIT-2026-08-05** | 2026-07-30 |
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 | | [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) | **Current & authoritative.** Read-only ground-truth via nmap/SNMP/SSH/API: 4 switches up / 1 down, 63 VMs, 167 zones, 753 DHCP leases, k8s DOWN, netinfra-02 wiped, repo restructure, 3-tier Tailscale fleet | 2026-08-05 |
| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 | | [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 |
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`proxmox/docs/NAMING-POLICY.md`](proxmox/docs/NAMING-POLICY.md) | Canonical naming policy: VM name = source of truth (DNS/Tailscale/monitoring must match) | 2026-07-31 |
| [`proxmox/docs/PHYSICAL-SYSTEMS.md`](proxmox/docs/PHYSICAL-SYSTEMS.md) | Physical infrastructure: Pi's, network devices, UniFi APs, SNMP-discovered hosts | 2026-07-31 |
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
## Server Build ([`server-build/`](server-build/)) ## Server Build ([`docs/server-build/`](server-build/))
Server provisioning, security hardening, DNS/NTP configuration. Server provisioning, security hardening, DNS/NTP configuration.
| Document | Description | Last Reviewed | | Document | Description | Last Reviewed |
|----------|-------------|---------------| |----------|-------------|---------------|
| [`server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 | | [`docs/server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
| [`server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 | | [`docs/server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
| [`server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 | | [`docs/server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
| [`server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 | | [`docs/server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
| [`server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 | | [`docs/server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
## Operational Guides (outside docs/) ## Operational Guides (outside docs/)
@@ -52,11 +54,11 @@ Server provisioning, security hardening, DNS/NTP configuration.
|----------|-------------|---------------| |----------|-------------|---------------|
| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 | | [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 | | [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 | | [`../dcinfra/powerman/README.md`](../dcinfra/powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
| [`../console/README.md`](../console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 | | [`../dcinfra/console/README.md`](../dcinfra/console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
| [`../ups/README.md`](../ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 | | [`../dcinfra/ups/README.md`](../dcinfra/ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 | | [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 | | [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 | | [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 | | [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 | | [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
nextprompt
+1 -3
View File
@@ -95,12 +95,10 @@ I want to get you connected into promethus/promethus alert manager/grafana and h
I want home assistant to handle all environmental monitoring and reporting for the site. I want home assistant to handle all environmental monitoring and reporting for the site.
Also, any api keys must be moved into hashicorp vault as soon as possible. Track that as a very high priority todo. vault is deployed and out of box setup but nothing configured. its at https://vault.knownelement.com Also, any api keys must be moved into hashicorp vault as soon as possible. Track that as a very high priority todo. vault is deployed and out of box setup but nothing configured. its at https://vault.knownelement.com
Also, any work we do here, that perhaps should also be replicated in the KNELIAC repository/project, track it in a STATUS.md file in that project. We are doing a bit of fast/loose here at this time with bash scripts, in the interest of time. For the mid/long term, the work should be captured as ansible roles/playbooks. Also, any work we do here, that perhaps should also be replicated in the KNELIAC repository/project, track it in a STATUS.md file in that project. We are doing a bit of fast/loose here at this time with bash scripts, in the interest of time. For the mid/long term, the work should be captured as ansible roles/playbooks.
ALso start thinking about k8s. vcluster/rancher/rbac , auth to OIDC/cloudron/keycloak. we will work on that in a dedicated session. for now, ssh to all the worker nodes (not sectestbed/preprod though) and join them to the cluster. oh the cnodes... i also accidently wiped one of them out. so that will need to be fixed. I was working directly on the storage nodes, that will not happen again! I was migrating things around and I should have stayed in the proxmox interface. for now, ssh to all the worker nodes (not sectestbed/preprod though) and join them to the cluster. oh the cnodes... i also accidently wiped one of them out. so that will need to be fixed. I shut the cnodes down, because one of them was 100% memory, i'm guessing cause one of the nodes got wiped out. So youll need to power them all up in proxmox.
Oh a meta point, to wrap things up. Stop thinking so much. Get to code faster. Explore with code. Gather ground truth. You are burning tokens way too much for things that can be answered very quickly/easily with code. Also, you need to be very comfortable to ask me questions. EIther via the question tool in the crush harness, or in a questions-v(x).md file that i can review/edit inline. Dont over think/self debate/ruminate. Just gather data either from me , or via code or both. The token burn is totally unacceptable. I have a quota and cant keep burning through it.
proxmox/docs/AUDIT-2026-08-05.md
+220
View File
@@ -0,0 +1,220 @@
# Ground-Truth Audit — 2026-08-05
> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
>
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
> Proxmox `qm list`/`pvecm`, Docker/container inspection.
## TL;DR — What Changed Since 2026-07-30
The fleet grew significantly and the repository was restructured into a
module/subproject layout. Headline deltas:
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|--------|-----------|-----------|---|
| VMs (running) | 28 | **63** | +125% |
| DNS zones | 124 | **167** | +35% |
| DHCP leases | 333 | **753** | +126% |
| Proxmox hosts | 7 | 7 | — |
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
## Repository Restructure (DONE by operator)
Top-level dirs were consolidated into modules. **Old flat dirs no longer
exist** at the repo root:
| Was (repo root) | Now |
|-----------------|-----|
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
> the OLD paths. These are being updated in this same session.
## Network Switches
SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
| IP | sysName | Model | Status |
|----|---------|-------|--------|
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
161 (SNMP) responds with `kn3lmgmt`.
### LACP / Cross-Rack Trunk (core-sw01 ch1)
- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
aggregate bundle speed via standard ifSpeed, so **member count + hash
algorithm need console verification** (conman requires a PTY — run manually):
```bash
conman -d pfv-tsys4:7890 -f pfv-core-sw01
# then: show interfaces status port-channel 1
# show running-config | begin port channel
```
## Proxmox Hosts (7 standalone, managed via PDM)
All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
**Not clustered** — standalone installs (corosync Expected votes=1 each).
| Host | RAM used/total | VMs | Uptime | Notes |
|------|---------------|-----|--------|-------|
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
### tsys5 Storage (RESOLVED)
Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
| Disk | Mount | Use |
|------|-------|-----|
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
| local-lvm (1.67 TB thin) | LVM | 6.9% |
NFS from tsys4: D2, D3, D5 all mounted and active.
## DNS / NTP / DHCP (netinfra)
### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
| Service | State | Detail |
|---------|-------|--------|
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
full rebuild as redundant node — see Pending.
## Kubernetes (pfv-k8s) — DOWN
| Node | Tailscale IP | k3s API :6443 | Location |
|------|-------------|---------------|----------|
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
— an IP matching NO current cnode. Cluster must be rebuilt.
- Worker nodes (5) all online on Tailscale, **not joined**:
pfv-k8s-wnode-tsys3/5/6/7/9.
## Monitoring / Observability Stack
| System | Host | Status | Access |
|--------|------|--------|--------|
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
## Console Management (pfv-tsys4)
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
All 7 `/dev/consoles/<name>` symlinks present:
```
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
pfv-r2-sw→ttyUSB3
```
**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
interactive terminal, or the conman client directly.
## Tailscale Fleet (80 nodes)
Three full environment tiers now exist, each mirroring the same service set:
- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
tsys-umbrel, tsys-voip
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
preprod-rancherplatform, preprod-siem, preprod-voip
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
## Identified LAN Hosts (non-switch)
| IP | Identity | Evidence |
|----|----------|----------|
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
| 192.168.3.1 | gateway/router | Linux + xrdp |
## Divergences vs STATUS.md (2026-07-30)
1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
2. **DNS zones 124→167**, **DHCP leases 333→753**
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
4. **tsys5 NVMe+SSD** now configured (was "not configured")
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
7. **k8s cluster** now DOWN (was LIVE)
8. **netinfra-02** wiped (was redundant secondary)
9. **Technitium** now Docker container (was systemd dnsServer)
10. **NTP** is ntpsec (STATUS assumed chrony)
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
12. **Tailscale fleet** 3× larger with 3 environment tiers
## Pending / Recommended Next Actions
*(From `nextprompt` — reference only; awaiting operator direction)*
- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
zone sync + DHCP failover-peer)
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
- **Vault** migrate all API keys (HIGH priority)
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
- **Home Assistant**: own all environmental monitoring/reporting
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)