docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts

Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM,
CPU, and storage for all 7 reachable hosts. Written to
docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in
PROJECT.md sections 4-8.

Key audit findings:
- CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4
  storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These
  migrations were the #1 recommendation from the previous audit and
  have not been done.
- CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS.
  tsys4 failure = etcd quorum lost.
- 59% of running VMs still on tsys4 storage (improved from 68%).
- cnode VMIDs have changed since PROJECT.md was written (cnode1 is now
  VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.)

Gardening fixes:
- Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already
  exist for this purpose)
- Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/
  scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md
  self-locating scripts convention)
- Updated STATUS.md Known Issues with the two critical findings
- Updated STATUS.md Pending with prioritized pre-k8s action items
- Registered AUDIT-2026-07-28.md in docmap.md

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-28 20:07:09 -05:00
parent 5ce5d2e6b7
commit 21adb89d4e
13 changed files with 281 additions and 24 deletions
+14 -12
View File
@@ -78,26 +78,28 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| Issue | Impact | Status |
|-------|--------|--------|
| **UCS + netinfra HA pairs both on tsys4** | tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark | **CRITICAL — needs PDM migration** |
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost | **CRITICAL — needs PDM migration** |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
## Pending (next session priorities)
1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical
devices (Friday onsite)
2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
- Cnode count: 3 (recommended) vs 5
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
1. **CRITICAL: Migrate HA pairs to separate storage** (PDM, 10 min):
- netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5)
- ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5)
2. **CRITICAL: Fix k3s cnode quorum risk** (PDM, 5 min):
- Move cnode1 (VMID 906) or cnode2 (VMID 705) from tsys4 to tsys5 storage
3. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
7. **k8s deferred topics** (tracked for future sessions): ETL tooling
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
per-tenant vcluster policy templates, solar-aware scale-out
7. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
solar-aware scale-out
> **See [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md)
> for the full fresh audit with VM inventory and action items.**
## Infrastructure Summary