docs(audit): consolidate to single audit with corrected storage topology

Delete AUDIT-2026-07-28.md (old audit). Rewrite AUDIT-2026-07-29.md as the
sole authoritative audit after reading live /etc/pve/storage.cfg, /etc/exports,
and /etc/fstab on every host.

Key corrections:
- local-nonprod on tsys5 IS the S1 physical disk (dir storage, not separate).
  S1 carries 16 sectestbed VMs, not 0 — it is NOT a migration target.
- storage.cfg is per-host, not cluster-wide — each node defines its own entries.
- BUG: tsys6 S1 export path typo (S21 instead of S1) — S1 broken on tsys6.
- GAP: tsys9 has no S1 defined in storage.cfg.
- Revised disk migration plan avoids S1, targets S2/S3/S4/D5 for IOPS spreading.
- D2 goes 17->5 VMs; S2/S3 absorb the load (7 each); S1 untouched (16 idle VMs).
- Refreshed ground truth: cnode5 deleted, cnode4 stopped (user actions this session).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-29 14:32:43 -05:00
parent 6199368710
commit 915111e9dd
4 changed files with 335 additions and 819 deletions
+11 -9
View File
@@ -35,10 +35,13 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit refreshed 2026-07-29 (live data, all 7 hosts)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 72%
- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73%
- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
- [ ] CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware)
- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg
- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
- [ ] **BUG: tsys6 S1 export path typo** (`S21` instead of `S1`) — S1 broken on tsys6
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
- [ ] tsys2: pending rebuild from Win10 to Proxmox
@@ -89,7 +92,7 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 72%/37%/60% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §8 |
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
| **cnode4/5 stale VMs running** | Not joined to the 3-node cluster; waste 8 GB + 8 vCPU | Stop + delete (AUDIT-2026-07-29 §10) |
| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ |
| **Fleet CPU 0-6% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
@@ -107,16 +110,15 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
3. **Stop + delete stale cnode4 (601) / cnode5 (706)** — not in the 3-node cluster
4. **Join wnodes to the k3s cluster** (after migrations free resources)
5. **Friday maintenance:** physical hardware + storage work (see
[`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5
and [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
[`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
6. **Wnode storage rebalancing** (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1
7. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
8. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
solar-aware scale-out
> **See [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md)
> for the full fresh audit with VM inventory, capacity analysis, and
> Friday migration plan.**
> **See [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md)
> for the consolidated audit with corrected storage topology, VM inventory,
> capacity analysis, and action checklist.**
## Infrastructure Summary