Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 mrcharles
|
d159504121 | ||
|
mrcharles
|
04ed433e84 |
@@ -8,33 +8,27 @@ lives in subdirectories.
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
provisioning/ Server provisioning (SetupNewSystem.sh, security, 2FA)
|
||||
tests/ Test suite + VM validation harness
|
||||
dns-cluster-setup/ Technitium DNS cluster replication
|
||||
k8s/ k3s cluster setup scripts (3-node HA over Tailscale) + docs/
|
||||
powerman/ Cyclades PM10i PDU management via powerman
|
||||
console/ Serial console management (ser2net + conman) for switches
|
||||
ups/ UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1
|
||||
perf/ Proxmox perf tuning, fleet audit, iperf
|
||||
proxmox/ Proxmox fleet docs (hardware audit, capacity, k8s host planning)
|
||||
netinfra/ pfv-netinfra-01/02 DNS/NTP/DHCP setup
|
||||
switches/ Switch configuration captures
|
||||
dcinfra/ Data-center infra: PDU (powerman), serial console (console), UPS (ups)
|
||||
netinfra/ DNS/NTP/DHCP setup + audit; DNS cluster replication (dns-cluster-setup);
|
||||
switch captures (switches); DHCP config (dhcp)
|
||||
k8s/ k3s cluster setup scripts (HA control plane over Tailscale) + docs/
|
||||
proxmox/ Proxmox fleet docs (hardware audit, capacity, storage) + perf tuning (perf/)
|
||||
awx/ Ansible AWX deployment (k3s + AWX Operator)
|
||||
tests/ Test suite + VM validation harness + remote.sh SSH chokepoint
|
||||
docs/ Server-build docs, docmap index, and archive
|
||||
archive/ Historical/superseded code (provisioning -> replaced by KNELIAC project)
|
||||
vendor/ Vendored KNELShellFramework
|
||||
```
|
||||
|
||||
- **Self-locating scripts**: All provisioning scripts derive their own
|
||||
location via `BASH_SOURCE` and compute `PROJECT_ROOT_PATH` from it. Run
|
||||
from anywhere.
|
||||
- **Local configs are the source of truth**: Files in
|
||||
[`provisioning/ConfigFiles/`](provisioning/ConfigFiles/) are read with
|
||||
`cat`/`cp`. Do NOT re-introduce `curl ${DL_ROOT}/...` downloads.
|
||||
- **Non-bash files**: Some files under `provisioning/Agents/` have `.sh`
|
||||
- **Server provisioning moved to KNELIAC**: The
|
||||
[`archive/provisioning/`](archive/provisioning/) tree is historical. Active
|
||||
server provisioning lives in the **KNELIAC** project at
|
||||
`/home/reachableceo/projects/KNELIAC`.
|
||||
- **Non-bash files**: Some files under `archive/provisioning/Agents/` have `.sh`
|
||||
extension but are PHP (shebang `#!/usr/bin/php`). Skip in syntax checks.
|
||||
- **SSH in Crush**: Direct ssh/scp is blocked. Use
|
||||
[`tests/remote.sh`](tests/remote.sh) or
|
||||
[`dns-cluster-setup/remote-dns.sh`](dns-cluster-setup/remote-dns.sh).
|
||||
[`tests/remote.sh`](tests/remote.sh) (Proxmox hosts + sandbox VM) or
|
||||
[`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) (DNS infra hosts).
|
||||
|
||||
## Git Policy
|
||||
|
||||
@@ -90,20 +84,17 @@ vendor/ Vendored KNELShellFramework
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| [`provisioning/SetupNewSystem.sh`](provisioning/SetupNewSystem.sh) | Full server provisioning |
|
||||
| [`tests/remote.sh`](tests/remote.sh) | **SSH chokepoint** — all Proxmox host + sandbox VM access routes here |
|
||||
| [`netinfra/dns-cluster-setup/remote-dns.sh`](netinfra/dns-cluster-setup/remote-dns.sh) | SSH chokepoint for DNS infra hosts (netinfra-01/02, tsrouter, netboot) |
|
||||
| [`tests/vm-validation.sh`](tests/vm-validation.sh) | Deploy + validate on sandbox VM |
|
||||
| [`tests/run-tests.sh`](tests/run-tests.sh) | Test suite |
|
||||
| [`dns-cluster-setup/setup.sh`](dns-cluster-setup/setup.sh) | DNS cluster replication |
|
||||
| [`netinfra/dns-cluster-setup/setup.sh`](netinfra/dns-cluster-setup/setup.sh) | DNS cluster replication |
|
||||
| [`k8s/install-cp.sh`](k8s/install-cp.sh) | Bootstrap k3s HA control plane |
|
||||
| [`powerman/setup.sh`](powerman/setup.sh) | Configure Cyclades PDU via powerman |
|
||||
| [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman |
|
||||
| [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
|
||||
| [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings |
|
||||
| [`perf/scripts/probe-storage.sh`](perf/scripts/probe-storage.sh) | Read-only disk/mount/export/SMART audit |
|
||||
| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit |
|
||||
| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) |
|
||||
| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) |
|
||||
| [`perf/scripts/probe-drift.sh`](perf/scripts/probe-drift.sh) | Read-only fleet drift probe (packages, services, tuning, SSH, security) |
|
||||
| [`dcinfra/powerman/setup.sh`](dcinfra/powerman/setup.sh) | Configure Cyclades PDU via powerman |
|
||||
| [`dcinfra/console/setup.sh`](dcinfra/console/setup.sh) | Configure serial console access via ser2net + conman |
|
||||
| [`dcinfra/ups/setup.sh`](dcinfra/ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring |
|
||||
| [`proxmox/perf/deploy-tuning.sh`](proxmox/perf/deploy-tuning.sh) | Deploy perf tunings |
|
||||
| [`proxmox/perf/scripts/`](proxmox/perf/scripts/) | Read-only audit: probe-storage, probe-network, conman-console, snmp-switch-audit, probe-drift |
|
||||
|
||||
## Key Docs
|
||||
|
||||
|
||||
@@ -8,19 +8,14 @@ Unified infrastructure repo for the Known Element Enterprises Proxmox R&D cluste
|
||||
|
||||
| Directory | Description |
|
||||
|-----------|-------------|
|
||||
| [`provisioning/`](provisioning/) | Server provisioning (SetupNewSystem.sh, security hardening, 2FA, NTP/DNS config, SNMP, Dell OMSA) |
|
||||
| [`tests/`](tests/) | Test suite + VM validation harness |
|
||||
| [`dns-cluster-setup/`](dns-cluster-setup/) | Technitium DNS cluster replication scripts |
|
||||
| [`k8s/`](k8s/) | k3s cluster setup scripts (3-node HA control plane over Tailscale) + co-located docs |
|
||||
| [`powerman/`](powerman/) | Cyclades PM10i PDU management via powerman on pfv-tsys1 |
|
||||
| [`console/`](console/) | Serial console management (ser2net + conman) for network switches on pfv-tsys4 |
|
||||
| [`ups/`](ups/) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale |
|
||||
| [`perf/`](perf/) | Proxmox performance tuning, fleet audit, iperf, switch diagnostics |
|
||||
| [`proxmox/`](proxmox/) | Proxmox fleet docs: hardware audit, capacity analysis, k8s host planning |
|
||||
| [`netinfra/`](netinfra/) | pfv-netinfra-01/02 DNS/NTP/DHCP setup + audit scripts |
|
||||
| [`switches/`](switches/) | Switch configuration captures |
|
||||
| [`awx/`](awx/) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) |
|
||||
| [`dcinfra/`](dcinfra/) | Data-center infrastructure: PDU (`powerman/`), serial console (`console/`), UPS (`ups/`) |
|
||||
| [`netinfra/`](netinfra/) | DNS/NTP/DHCP setup + audit scripts, DNS cluster replication (`dns-cluster-setup/`), switch captures (`switches/`), DHCP config (`dhcp/`) |
|
||||
| [`k8s/`](k8s/) | k3s cluster setup scripts (HA control plane over Tailscale) + co-located docs |
|
||||
| [`proxmox/`](proxmox/) | Proxmox fleet docs (hardware audit, capacity, storage, k8s host planning) + performance tuning (`perf/`) |
|
||||
| [`awx/`](awx/) | Ansible AWX deployment (k3s + AWX Operator) |
|
||||
| [`tests/`](tests/) | Test suite + VM validation harness + `remote.sh` SSH chokepoint |
|
||||
| [`docs/`](docs/) | Server-build docs, documentation index ([docmap](docs/docmap.md)), and archive |
|
||||
| [`archive/`](archive/) | Historical/superseded code (provisioning → replaced by KNELIAC project) |
|
||||
| [`vendor/`](vendor/) | Vendored KNELShellFramework |
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -3,19 +3,24 @@
|
||||
> **Human read-only. Agents maintain this file automatically after each work
|
||||
> session.** Do not edit by hand — the next agent run will overwrite it.
|
||||
|
||||
> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute)
|
||||
> **Last updated:** 2026-08-05 by Crush (GLM-5.2) — full read-only ground-truth
|
||||
> re-orientation (nmap + SNMP + SSH + API queries).
|
||||
>
|
||||
> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md)
|
||||
> — network topology, switch LAG configs, corrected storage topology, VM config
|
||||
> audit, fleet consistency, cross-rack NFS root-cause analysis.
|
||||
> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)
|
||||
> — repo restructure, 4 switches up / 1 down, 63 VMs, 167 DNS zones, 753 DHCP
|
||||
> leases, k8s cluster DOWN (cnode wiped), netinfra-02 wiped, tsys5 storage
|
||||
> configured, tsys4 upgraded, 3-tier Tailscale fleet (80 nodes).
|
||||
> Capacity targets are codified in
|
||||
> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
|
||||
> steady-state, ~50% idle CPU.
|
||||
> steady-state (fleet PASSES, highest tsys1 at 71%).
|
||||
|
||||
## Current State: STABLE
|
||||
## Current State: MIXED (infra stable; k8s + netinfra-02 need rebuild)
|
||||
|
||||
The merged PFVCluster repo is fully operational across provisioning, DNS
|
||||
infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
Provisioning, DNS primary (netinfra-01), Proxmox fleet, and monitoring stack
|
||||
are operational. **k8s control plane is DOWN** (cnode wiped, no etcd quorum)
|
||||
and **netinfra-02 is wiped** (needs rebuild as redundant secondary). Repo was
|
||||
restructured into a module/subproject layout (see
|
||||
[`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md)).
|
||||
|
||||
## Completed Work
|
||||
|
||||
@@ -34,7 +39,16 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
- [x] Zone replication via rsync systemd timer (every 60s)
|
||||
- [x] Credentials + 2FA replicated identically to production
|
||||
- [x] Both LAN IPs resolve knel.net device names + recurse externally
|
||||
- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, 333 active leases load-balanced. DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
|
||||
- [x] **DHCP migrated from pfv-netboot → netinfra-01/02** (ISC failover-peer, primary/secondary). Both `normal`, **753 active leases** (was 333). DNS+NTP now handed out as 252/253. Netboot DHCP stopped + disabled. Webmin DHCP module on both nodes.
|
||||
- [ ] **netinfra-02 WIPED 2026-08-05** (storage move accident). Reachable on
|
||||
Tailscale (100.71.171.20) but SSH key auth fails — needs full rebuild as
|
||||
redundant secondary (key deploy + zone sync + DHCP failover-peer).
|
||||
- [x] **Technitium now runs as Docker container** (`tsys-dns`) on netinfra-01,
|
||||
not systemd dnsServer. Config at `/home/localuser/services/technitium/config`.
|
||||
**167 zones** (was 124). NTP = `ntpsec` (not chrony).
|
||||
- [x] **Pi-hole `/dev/shm` = 1.0 GB** (operator hotfix, was too small → prod
|
||||
outage). Healthcheck = `dig pi.hole` (DNS-based; gravity.db-specific check
|
||||
not yet added).
|
||||
|
||||
### Proxmox Cluster Ops
|
||||
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
|
||||
@@ -53,8 +67,11 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured
|
||||
- [ ] tsys2: pending rebuild from Win10 to Proxmox
|
||||
|
||||
### Kubernetes Control Plane (k3s HA — LIVE)
|
||||
- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
|
||||
### Kubernetes Control Plane (k3s HA — DOWN, needs rebuild)
|
||||
- [ ] **Cluster DOWN 2026-08-05**: operator wiped a cnode + shut cnodes down
|
||||
(100% RAM on one). cnode2 has k3s API up, cnode1/cnode3 do not — no etcd
|
||||
quorum. Kubeconfig points to stale IP (100.97.178.106, matches no cnode).
|
||||
- [x] Was: 3-node k3s HA control plane (cnode1/2/3, embedded etcd, Tailscale)
|
||||
- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
|
||||
- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
|
||||
- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
|
||||
|
||||
+15
-13
@@ -3,7 +3,7 @@
|
||||
> **Index of all documentation in this repo.** Agents must update this file
|
||||
> whenever a doc is added, removed, or substantively changed.
|
||||
|
||||
> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute)
|
||||
> **Last updated:** 2026-08-05 (full read-only ground-truth audit: network+SNMP+compute+k8s+netinfra)
|
||||
|
||||
## Kubernetes Architecture ([`k8s/`](k8s/))
|
||||
|
||||
@@ -28,23 +28,25 @@ Fleet operations, hardware, performance tuning, storage architecture.
|
||||
| [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 |
|
||||
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | Prior audit (network + storage + compute + switches). **Superseded by AUDIT-2026-08-05** | 2026-07-30 |
|
||||
| [`proxmox/docs/AUDIT-2026-08-05.md`](proxmox/docs/AUDIT-2026-08-05.md) | **Current & authoritative.** Read-only ground-truth via nmap/SNMP/SSH/API: 4 switches up / 1 down, 63 VMs, 167 zones, 753 DHCP leases, k8s DOWN, netinfra-02 wiped, repo restructure, 3-tier Tailscale fleet | 2026-08-05 |
|
||||
| [`proxmox/docs/DRIFT-2026-07-30.md`](proxmox/docs/DRIFT-2026-07-30.md) | **Fleet drift report.** Executive briefing of package/service/tuning/security inconsistencies across all 7 hosts, with detailed appendixes | 2026-07-30 |
|
||||
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/docs/NAMING-POLICY.md`](proxmox/docs/NAMING-POLICY.md) | Canonical naming policy: VM name = source of truth (DNS/Tailscale/monitoring must match) | 2026-07-31 |
|
||||
| [`proxmox/docs/PHYSICAL-SYSTEMS.md`](proxmox/docs/PHYSICAL-SYSTEMS.md) | Physical infrastructure: Pi's, network devices, UniFi APs, SNMP-discovered hosts | 2026-07-31 |
|
||||
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
|
||||
## Server Build ([`server-build/`](server-build/))
|
||||
## Server Build ([`docs/server-build/`](server-build/))
|
||||
|
||||
Server provisioning, security hardening, DNS/NTP configuration.
|
||||
|
||||
| Document | Description | Last Reviewed |
|
||||
|----------|-------------|---------------|
|
||||
| [`server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
|
||||
| [`server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
|
||||
| [`server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
|
||||
| [`server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
|
||||
| [`server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
|
||||
| [`docs/server-build/SECURITY.md`](server-build/SECURITY.md) | Security architecture: SSH hardening, 2FA, SCAP-STIG, Wazuh, auditd | 2026-07-25 |
|
||||
| [`docs/server-build/tailscale.md`](server-build/tailscale.md) | Tailscale vs managed DNS analysis (RESOLVED — netinfra pair serves knel.net) | 2026-07-28 |
|
||||
| [`docs/server-build/DEPLOYMENT.md`](server-build/DEPLOYMENT.md) | Server deployment procedures, package lists, config flow | 2026-07-25 |
|
||||
| [`docs/server-build/TSYS-2FA-GUIDE.md`](server-build/TSYS-2FA-GUIDE.md) | End-user guide for 2FA setup (SSH, Cockpit, Webmin) | 2026-07-25 |
|
||||
| [`docs/server-build/DEVELOPMENT-GUIDELINES.md`](server-build/DEVELOPMENT-GUIDELINES.md) | Coding standards, commit conventions, script patterns | 2026-07-25 |
|
||||
|
||||
## Operational Guides (outside docs/)
|
||||
|
||||
@@ -52,11 +54,11 @@ Server provisioning, security hardening, DNS/NTP configuration.
|
||||
|----------|-------------|---------------|
|
||||
| [`../netinfra/dhcp-migration.md`](../netinfra/dhcp-migration.md) | DHCP migration: pfv-netboot → netinfra-01/02 (ISC failover-peer, COMPLETE) | 2026-07-29 |
|
||||
| [`../awx/README.md`](../awx/README.md) | Ansible AWX deployment on tsys-awx (k3s + AWX Operator 2.19.1, v24.6.1) | 2026-07-29 |
|
||||
| [`../powerman/README.md`](../powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
|
||||
| [`../console/README.md`](../console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
|
||||
| [`../ups/README.md`](../ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
|
||||
| [`../dcinfra/powerman/README.md`](../dcinfra/powerman/README.md) | Cyclades PM10i PDU management via powerman on pfv-tsys1 | 2026-07-28 |
|
||||
| [`../dcinfra/console/README.md`](../dcinfra/console/README.md) | Serial console management (ser2net telnet rfc2217 + conman) for 7 network switches on pfv-tsys4 | 2026-07-29 |
|
||||
| [`../dcinfra/ups/README.md`](../dcinfra/ups/README.md) | UPS management (NUT) for APC Smart-UPS C 1500 on pfv-tsys1 — upsd on Tailscale, HA integration | 2026-07-30 |
|
||||
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
|
||||
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
|
||||
| [`../netinfra/dns-cluster-setup/README.md`](../netinfra/dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
|
||||
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
|
||||
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
|
||||
| [`../netinfra/pfv-netboot-setup.md`](../netinfra/pfv-netboot-setup.md) | pfv-netboot reference node setup | 2026-07-27 |
|
||||
|
||||
+1
-3
@@ -95,12 +95,10 @@ I want to get you connected into promethus/promethus alert manager/grafana and h
|
||||
|
||||
I want home assistant to handle all environmental monitoring and reporting for the site.
|
||||
|
||||
|
||||
Also, any api keys must be moved into hashicorp vault as soon as possible. Track that as a very high priority todo. vault is deployed and out of box setup but nothing configured. its at https://vault.knownelement.com
|
||||
|
||||
Also, any work we do here, that perhaps should also be replicated in the KNELIAC repository/project, track it in a STATUS.md file in that project. We are doing a bit of fast/loose here at this time with bash scripts, in the interest of time. For the mid/long term, the work should be captured as ansible roles/playbooks.
|
||||
|
||||
ALso start thinking about k8s. vcluster/rancher/rbac , auth to OIDC/cloudron/keycloak. we will work on that in a dedicated session. for now, ssh to all the worker nodes (not sectestbed/preprod though) and join them to the cluster. oh the cnodes... i also accidently wiped one of them out. so that will need to be fixed. I was working directly on the storage nodes, that will not happen again! I was migrating things around and I should have stayed in the proxmox interface.
|
||||
for now, ssh to all the worker nodes (not sectestbed/preprod though) and join them to the cluster. oh the cnodes... i also accidently wiped one of them out. so that will need to be fixed. I shut the cnodes down, because one of them was 100% memory, i'm guessing cause one of the nodes got wiped out. So youll need to power them all up in proxmox.
|
||||
|
||||
|
||||
Oh a meta point, to wrap things up. Stop thinking so much. Get to code faster. Explore with code. Gather ground truth. You are burning tokens way too much for things that can be answered very quickly/easily with code. Also, you need to be very comfortable to ask me questions. EIther via the question tool in the crush harness, or in a questions-v(x).md file that i can review/edit inline. Dont over think/self debate/ruminate. Just gather data either from me , or via code or both. The token burn is totally unacceptable. I have a quota and cant keep burning through it.
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
# Ground-Truth Audit — 2026-08-05
|
||||
|
||||
> **Authoritative fleet audit** performed by Crush (GLM-5.2) via read-only
|
||||
> network discovery (nmap, SNMP, SSH, API queries). Supersedes
|
||||
> [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md) for all fleet state.
|
||||
>
|
||||
> **Method:** Tailscale `status`, nmap ping sweep (192.168.0-3.0/24), SNMP
|
||||
> walks (`kn3lmgmt` community), SSH via `tests/remote.sh` + `netinfra/dns-cluster-setup/remote-dns.sh`,
|
||||
> Proxmox `qm list`/`pvecm`, Docker/container inspection.
|
||||
|
||||
## TL;DR — What Changed Since 2026-07-30
|
||||
|
||||
The fleet grew significantly and the repository was restructured into a
|
||||
module/subproject layout. Headline deltas:
|
||||
|
||||
| Metric | 2026-07-30 | 2026-08-05 | Δ |
|
||||
|--------|-----------|-----------|---|
|
||||
| VMs (running) | 28 | **63** | +125% |
|
||||
| DNS zones | 124 | **167** | +35% |
|
||||
| DHCP leases | 333 | **753** | +126% |
|
||||
| Proxmox hosts | 7 | 7 | — |
|
||||
| PVE version (fleet) | mixed (9.1.5–9.2.5) | **all 9.2.5** | tsys4 upgraded |
|
||||
| Tailscale nodes | ~30 | **80** | full 3-tier env mirror |
|
||||
| k8s cluster | LIVE (3-node HA) | **DOWN** | cnode wiped, no quorum |
|
||||
| netinfra-02 | redundant secondary | **WIPED** | needs rebuild |
|
||||
|
||||
## Repository Restructure (DONE by operator)
|
||||
|
||||
Top-level dirs were consolidated into modules. **Old flat dirs no longer
|
||||
exist** at the repo root:
|
||||
|
||||
| Was (repo root) | Now |
|
||||
|-----------------|-----|
|
||||
| `provisioning/` | `archive/provisioning/` — superseded by **KNELIAC** project (`/home/reachableceo/projects/KNELIAC`) |
|
||||
| `powerman/` `console/` `ups/` | `dcinfra/{powerman,console,ups}/` |
|
||||
| `dns-cluster-setup/` `switches/` `dhcp/` | `netinfra/{dns-cluster-setup,switches,dhcp}/` |
|
||||
| `perf/` | `proxmox/perf/` (+ `proxmox/perf/scripts/`) |
|
||||
|
||||
Current repo root: `archive/ awx/ dcinfra/ docs/ k8s/ netinfra/ proxmox/ tests/ vendor/` + `logs/ returned-logs/`.
|
||||
|
||||
> **WARNING:** `README.md`, `AGENTS.md`, and `docs/docmap.md` still reference
|
||||
> the OLD paths. These are being updated in this same session.
|
||||
|
||||
## Network Switches
|
||||
|
||||
SNMP community: `kn3lmgmt` (read) / `kn3lwrite` (write). SNMP responds from
|
||||
the workstation LAN (no ACL — earlier "filtered" was just a wrong community).
|
||||
|
||||
| IP | sysName | Model | Status |
|
||||
|----|---------|-------|--------|
|
||||
| 192.168.0.12 | pfv-core-sw01 | PowerConnect 5448 | **UP** (Dell, core, rack5) |
|
||||
| 192.168.0.7 | pfv-r3-tor-mgmt | Neyland 24T (PC5324) | **UP** (rack3 mgmt TOR) |
|
||||
| 192.168.0.9 | pfv-r3-tor-stor | Neyland 24T (PC5324) | **UP** (rack3 storage TOR) |
|
||||
| 192.168.0.8 | subodev-torsw01 | Neyland 24T (PC5324) | **UP** (was "unidentified" — now named) |
|
||||
| 192.168.0.5 | pfv-r2-tor-top | (unknown) | **OFFLINE** — no ping, no SNMP |
|
||||
|
||||
**Ports open on switches:** TCP 22 (ssh), 23 (telnet), 80 (http web UI). UDP
|
||||
161 (SNMP) responds with `kn3lmgmt`.
|
||||
|
||||
### LACP / Cross-Rack Trunk (core-sw01 ch1)
|
||||
|
||||
- Operator reports **LACP is fixed** (was static `mode on`, hash layer-2-3).
|
||||
- SNMP confirms **ch1 ifIndex=49 is UP**; ch3 ifIndex=51 also UP.
|
||||
- ch1 reports ifSpeed=1000Mbps — the Dell FastPath switch does not report
|
||||
aggregate bundle speed via standard ifSpeed, so **member count + hash
|
||||
algorithm need console verification** (conman requires a PTY — run manually):
|
||||
```bash
|
||||
conman -d pfv-tsys4:7890 -f pfv-core-sw01
|
||||
# then: show interfaces status port-channel 1
|
||||
# show running-config | begin port channel
|
||||
```
|
||||
|
||||
## Proxmox Hosts (7 standalone, managed via PDM)
|
||||
|
||||
All **PVE 9.2.5** (tsys4 upgraded from 9.1.5 — resolves old STATUS item).
|
||||
**Not clustered** — standalone installs (corosync Expected votes=1 each).
|
||||
|
||||
| Host | RAM used/total | VMs | Uptime | Notes |
|
||||
|------|---------------|-----|--------|-------|
|
||||
| pfv-tsys1 | 71% (22.7/32 GB) | 8 | 6d | Highest RAM use in fleet |
|
||||
| pfv-tsys3 | 12% (3.7/32 GB) | 2 | 9d | under-utilized |
|
||||
| pfv-tsys4 | 18% (2.8/16 GB) | 2 | 5d | storage server, kernel 7.0.14-8 |
|
||||
| pfv-tsys5 | 48% (46.8/97 GB) | **32** | 6d | sectestbed+preprod host |
|
||||
| pfv-tsys6 | 14% (18/129 GB) | 5 | 10d | 129 GB RAM, wnode+cnode3 |
|
||||
| pfv-tsys7 | 30% (58.8/193 GB) | 11 | 10d | **193 GB RAM**, prod workloads |
|
||||
| pfv-tsys9 | 30% (7.2/24 GB) | 3 | 9d | netinfra-02 + wnode |
|
||||
|
||||
**Memory capacity goal (≤80%): PASS fleet-wide.** Highest is tsys1 at 71%.
|
||||
|
||||
### tsys5 Storage (RESOLVED)
|
||||
|
||||
Old STATUS item "NVMe + SSD not configured" is **FIXED**. All storage active:
|
||||
|
||||
| Disk | Mount | Use |
|
||||
|------|-------|-----|
|
||||
| nvme0n1 (CT500P1 500GB) | /mnt/nvme (PVE: NVME dir) | 49% used |
|
||||
| sdb (SK hynix 512GB SSD) | /mnt/ssd2 (PVE: ssd2 dir) | 14% used |
|
||||
| sdd (Samsung 860 PRO 256GB) | /mnt/pfv-tsys5/ssd (PVE: SSD dir) | 1% |
|
||||
| sde (ST1000VN002 1TB) | /mnt/pfv-tsys5/S1 | 6% |
|
||||
| sdf (ST1000VN002 1TB) | /mnt/pfv-tsys5/S2 | 7% |
|
||||
| local-lvm (1.67 TB thin) | LVM | 6.9% |
|
||||
|
||||
NFS from tsys4: D2, D3, D5 all mounted and active.
|
||||
|
||||
## DNS / NTP / DHCP (netinfra)
|
||||
|
||||
### pfv-netinfra-01 (PRIMARY — 192.168.3.252, TS 100.70.181.72)
|
||||
|
||||
| Service | State | Detail |
|
||||
|---------|-------|--------|
|
||||
| **Technitium DNS** | UP (Docker `tsys-dns`, 5d) | Container, NOT systemd. Config: `/home/localuser/services/technitium/config` → `/etc/dns`. **167 zones.** Port 5380 via docker-proxy. |
|
||||
| **Pi-hole** | UP (healthy, 26h) | `/dev/shm` = **1.0 GB** (operator hotfix applied). Healthcheck = `dig pi.hole` (DNS-resolution based). |
|
||||
| **NTP** | UP (`ntpsec`, NOT chrony) | Clock synchronized. |
|
||||
| **DHCP** | UP (`isc-dhcp-server`) | **753 active leases** (was 333). |
|
||||
|
||||
### pfv-netinfra-02 (SECONDARY — 192.168.3.253, TS 100.71.171.20)
|
||||
|
||||
**WIPED.** Reachable on Tailscale (host key changed, cleared from known_hosts)
|
||||
but **SSH key auth fails** (`Permission denied (publickey,password)`). Needs
|
||||
full rebuild as redundant node — see Pending.
|
||||
|
||||
## Kubernetes (pfv-k8s) — DOWN
|
||||
|
||||
| Node | Tailscale IP | k3s API :6443 | Location |
|
||||
|------|-------------|---------------|----------|
|
||||
| pfv-k8s-cnode1 | 100.125.134.53 | **closed** | VM 102 on tsys1 |
|
||||
| pfv-k8s-cnode2 | 100.109.34.72 | **OPEN** | VM 705 on tsys7 |
|
||||
| pfv-k8s-cnode3 | 100.106.222.18 | **closed** | VM 603 on tsys6 |
|
||||
|
||||
- **No etcd quorum** (≤1 of 3 members). cnode1 and cnode3 k3s not listening.
|
||||
- **kubeconfig stale**: `~/.kube/config.pfv-k8s` points to `100.97.178.106:6443`
|
||||
— an IP matching NO current cnode. Cluster must be rebuilt.
|
||||
- Worker nodes (5) all online on Tailscale, **not joined**:
|
||||
pfv-k8s-wnode-tsys3/5/6/7/9.
|
||||
|
||||
## Monitoring / Observability Stack
|
||||
|
||||
| System | Host | Status | Access |
|
||||
|--------|------|--------|--------|
|
||||
| LibreNMS | tsys-librenms (100.86.204.77) | UP (Tailscale) | API needs token (401) — SSH in to create |
|
||||
| Uptime Kuma | status.knownelement.com | UP | API key in `~/.secrets/api-keys.env`; `/metrics` needs auth |
|
||||
| Beszel | (fleet) | deployed (per STATUS) | metrics host TBD |
|
||||
| Home Assistant | (HAOS VM) | configured (per STATUS) | NUT integration live |
|
||||
| **Vault** | vault.knownelement.com | **Deployed, UNCONFIGURED** | API keys migration = HIGH priority TODO |
|
||||
|
||||
## Console Management (pfv-tsys4)
|
||||
|
||||
ser2net listening on all 7 ports (2001-2007) bound to Tailscale 100.70.77.93.
|
||||
All 7 `/dev/consoles/<name>` symlinks present:
|
||||
|
||||
```
|
||||
pfv-core-sw01→ttyUSB2 pfv-tor3-mgmt→ttyUSB1 pfv-tor3-stor→ttyUSB6
|
||||
pfv-rrinfra-rtr→ttyUSB5 pfv-r2-tor-top→ttyUSB7 subodev-torsw→ttyUSB0
|
||||
pfv-r2-sw→ttyUSB3
|
||||
```
|
||||
|
||||
**conman requires a PTY** — cannot be driven via stdin pipe from Crush. Use an
|
||||
interactive terminal, or the conman client directly.
|
||||
|
||||
## Tailscale Fleet (80 nodes)
|
||||
|
||||
Three full environment tiers now exist, each mirroring the same service set:
|
||||
|
||||
- **`tsys-*` (production):** tsys-awx, tsys-ca, tsys-cloudron, tsys-librenms,
|
||||
tsys-proxmox-datacenter, tsys-proxmox-mailgw-01/02, tsys-siem, tsys-ucs-01/02,
|
||||
tsys-umbrel, tsys-voip
|
||||
- **`preprod-*`:** preprod-awx, preprod-ca, preprod-cloudron, preprod-hfnoc-uisp,
|
||||
preprod-librenms, preprod-proxmox-datacenter, preprod-proxmox-mailgw,
|
||||
preprod-rancherplatform, preprod-siem, preprod-voip
|
||||
- **`sectestbed-*`:** sectestbed-awx, sectestbed-ca, sectestbed-cloudron,
|
||||
sectestbed-hfnoc-uisp, sectestbed-k8s-cnode, sectestbed-k8s-wnode,
|
||||
sectestbed-librenms, sectestbed-netinfra, sectestbed-proxmox-datacenter,
|
||||
sectestbed-proxmox-mailgw, sectestbed-proxmox-pbs, sectestbed-proxmox-pve,
|
||||
sectestbed-rancherplatform, sectestbed-sandbox, sectestbed-siem, sectestbed-voip
|
||||
|
||||
Plus: pfv-tsys1/3/4/5/6/7/9, pfv-k8s-cnode1/2/3, pfv-k8s-wnode-tsys3/5/6/7/9,
|
||||
pfv-netinfra-01/02, pfv-proxmox-backup-server, pfv-rr-middleware-01/02,
|
||||
pfv-bms, pfv-jetson-nano-1, pfvsvrpi, devbox-cloudron, hfnoc-uisp,
|
||||
kali-rd, kali-tsys, netbird, preprod-awx, subopi-dev-3/4, subopi3,
|
||||
stlpc-artroom/garage, siter-bizoffice, ultix-* (endpoints).
|
||||
|
||||
> **Note:** Tailscale reports `--accept-routes is false` on this workstation.
|
||||
|
||||
## Identified LAN Hosts (non-switch)
|
||||
|
||||
| IP | Identity | Evidence |
|
||||
|----|----------|----------|
|
||||
| 192.168.0.100 | stlpc-garage.knel.net | Linux + Postfix + xrdp |
|
||||
| 192.168.0.250 | Linux storage (NFS) | SSH + rpcbind |
|
||||
| 192.168.1.166 | pfv-stor1 (NAS) | Samba smbd 4 + Golang HTTP + SSH |
|
||||
| 192.168.3.1 | gateway/router | Linux + xrdp |
|
||||
|
||||
## Divergences vs STATUS.md (2026-07-30)
|
||||
|
||||
1. **VM count 28→63** — sectestbed (16) + preprod (8) + new prod VMs added
|
||||
2. **DNS zones 124→167**, **DHCP leases 333→753**
|
||||
3. **tsys4 PVE 9.1.5→9.2.5** (was "behind fleet")
|
||||
4. **tsys5 NVMe+SSD** now configured (was "not configured")
|
||||
5. **tsys-umbrel + tsys-sipwise** deployed (new VMs)
|
||||
6. **LACP core↔r3-stor** operator reports fixed (was #1 known issue)
|
||||
7. **k8s cluster** now DOWN (was LIVE)
|
||||
8. **netinfra-02** wiped (was redundant secondary)
|
||||
9. **Technitium** now Docker container (was systemd dnsServer)
|
||||
10. **NTP** is ntpsec (STATUS assumed chrony)
|
||||
11. **Pi-hole /dev/shm** = 1 GB (operator hotfix; gravity.db healthcheck still DNS-based)
|
||||
12. **Tailscale fleet** 3× larger with 3 environment tiers
|
||||
|
||||
## Pending / Recommended Next Actions
|
||||
|
||||
*(From `nextprompt` — reference only; awaiting operator direction)*
|
||||
|
||||
- **netinfra-02 rebuild** as redundant secondary (SSH key deploy + Technitium
|
||||
zone sync + DHCP failover-peer)
|
||||
- **k8s cluster rebuild** (fix wiped cnode, restore quorum, join 5 wnodes)
|
||||
- **Pi-hole** add gravity.db healthcheck to compose (current check is DNS-only)
|
||||
- **Vault** migrate all API keys (HIGH priority)
|
||||
- **Monitoring**: deploy unpoller + netdisco to LibreNMS VM; add weathermap/smokeping
|
||||
- **Home Assistant**: own all environmental monitoring/reporting
|
||||
- **Benchmarking**: iperf (1gbps-per-flow verification) + fio + CPU/RAM across cluster
|
||||
- **Repo gardening**: README/AGENTS/docmap path fixes (this session)
|
||||
Reference in New Issue
Block a user