docs(audit): fresh fleet audit + fix stale paths across 13 perf scripts
Fresh Proxmox fleet audit (2026-07-28) with current VM placements, RAM, CPU, and storage for all 7 reachable hosts. Written to docs/proxmox/AUDIT-2026-07-28.md — supersedes placement data in PROJECT.md sections 4-8. Key audit findings: - CRITICAL: UCS01/02 and netinfra01/02 HA pairs both still on tsys4 storage. tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark. These migrations were the #1 recommendation from the previous audit and have not been done. - CRITICAL: 2 of 3 active k3s cnodes (cnode1 + cnode2) on tsys4 NFS. tsys4 failure = etcd quorum lost. - 59% of running VMs still on tsys4 storage (improved from 68%). - cnode VMIDs have changed since PROJECT.md was written (cnode1 is now VMID 906 on tsys9, cnode2 is VMID 705 on tsys7, etc.) Gardening fixes: - Removed duplicate fleet-audit.sh (check.sh + deploy-check.sh already exist for this purpose) - Fixed hardcoded path /home/reachableceo/projects/perfopt in 13 perf/ scripts to use BASH_SOURCE-derived relative paths (per AGENTS.md self-locating scripts convention) - Updated STATUS.md Known Issues with the two critical findings - Updated STATUS.md Pending with prioritized pre-k8s action items - Registered AUDIT-2026-07-28.md in docmap.md 💘 Generated with Crush Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -78,26 +78,28 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
|
||||
| Issue | Impact | Status |
|
||||
|-------|--------|--------|
|
||||
| **UCS + netinfra HA pairs both on tsys4** | tsys4 failure = DNS/DHCP/NTP + LDAP/AD fully dark | **CRITICAL — needs PDM migration** |
|
||||
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost | **CRITICAL — needs PDM migration** |
|
||||
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
|
||||
| `download.proxmox.com` unreachable from sandbox VM | 2 validation tests warn (environmental) | Not a code issue |
|
||||
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Waiting on physical install |
|
||||
|
||||
## Pending (next session priorities)
|
||||
|
||||
1. **PDU: rename outlets** in `/etc/powerman/powerman.conf` to match physical
|
||||
devices (Friday onsite)
|
||||
2. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
||||
2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
|
||||
- Cnode count: 3 (recommended) vs 5
|
||||
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
|
||||
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
|
||||
3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
1. **CRITICAL: Migrate HA pairs to separate storage** (PDM, 10 min):
|
||||
- netinfra-02 (VMID 904): D2 → S3 (tsys4 → tsys5)
|
||||
- ucs-02 (VMID 902): D5 → S2 (tsys4 → tsys5)
|
||||
2. **CRITICAL: Fix k3s cnode quorum risk** (PDM, 5 min):
|
||||
- Move cnode1 (VMID 906) or cnode2 (VMID 705) from tsys4 to tsys5 storage
|
||||
3. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
||||
4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
|
||||
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
7. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
||||
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
|
||||
per-tenant vcluster policy templates, solar-aware scale-out
|
||||
7. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
|
||||
solar-aware scale-out
|
||||
|
||||
> **See [`docs/proxmox/AUDIT-2026-07-28.md`](docs/proxmox/AUDIT-2026-07-28.md)
|
||||
> for the full fresh audit with VM inventory and action items.**
|
||||
|
||||
## Infrastructure Summary
|
||||
|
||||
|
||||
Reference in New Issue
Block a user