KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): RAM/CPU placement audit + codify capacity goals

Browse Source 
Add CAPACITY-GOALS.md (80% RAM / ~50% idle CPU targets + workload placement
model) and AUDIT-2026-07-29.md (live end-to-end analysis of all 7 hosts). Key
findings: memory goal met fleet-wide (highest 72%); CPU idles 0-6% (chase via
workloads, not hardware); tsys5 is single-CPU not dual; cnode4/5 are stale
running VMs. Answers the wnode-sizing question (grow tsys9, not tsys1), confirms
tsys6/7 for RackRental, and gives a DDR3 compatibility decision tree (RDIMM-only
servers reject consumer UDIMM). Refreshed STATUS/docmap and the returned audit
logs, and fixed an empty-array abort in deploy-check.sh.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 05:32:12 -05:00
parent 428e9ecb1d
commit 575d79f611
20 changed files with 17317 additions and 19981 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+32 -14
View File
@@ -3,7 +3,12 @@
> **Human read-only. Agents maintain this file automatically after each work
> session.** Do not edit by hand — the next agent run will overwrite it.
> **Last updated:** 2026-07-28 by Crush (GLM-5.2)
> **Last updated:** 2026-07-29 by Crush (GLM-5.2)
>
> **Capacity targets are now codified** in
> [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM
> steady-state, ~50% idle CPU. The current placement analysis lives in
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md).
## Current State: STABLE
@@ -30,9 +35,12 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit complete (PROJECT.md has ground truth)
- [x] Fleet audit refreshed 2026-07-29 (live data, all 7 hosts)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 72%
- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
- [ ] CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware)
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
- [ ] tsys2: pending rebuild from Win10 to Proxmox
### Kubernetes Control Plane (k3s HA — LIVE)
@@ -78,22 +86,32 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| Issue | Impact | Status |
|-------|--------|--------|
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode) |
| **tsys1/5/9 RAM-overcommitted** | 118%/135%/131% respectively | Friday: migrate NFS-backed VMs to tsys6/tsys7 (77+132 GB free) |
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 72%/37%/60% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §8 |
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
| **cnode4/5 stale VMs running** | Not joined to the 3-node cluster; waste 8 GB + 8 vCPU | Stop + delete (AUDIT-2026-07-29 §10) |
| **Fleet CPU 0-6% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
## Pending (next session priorities)
1. **Wnode storage rebalancing** (2 PDM storage migrations):
- wnode-tsys6 (100): D5 → S4 (tsys4 → tsys5)
- wnode-tsys7 (701): D5 → S1 (tsys4 → tsys5)
- Gives every wnode its own physical disk, balances 3/3 across servers
2. **Friday maintenance:** physical hardware + VM migrations (see
[`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5)
3. **k8s workers:** Join wnodes to the k3s cluster (after rebalancing)
4. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
5. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
> **RAM/CPU placement actions** are itemized in
> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §12.
> Storage/disk actions are deferred to post-Friday (see below).
1. **VM migrations (RAM/CPU relief, zero hardware):**
- tsys1 → tsys7: librenms(104), pdm(105), workbench(117), tailscale-router(109), kali-tsys(114)
- tsys9 → tsys6: tsys-siem(901)
2. **Grow wnode-tsys9** to 6 vCPU / 16 GB (after siem move) — cross-compile node
3. **Stop + delete stale cnode4 (601) / cnode5 (706)** — not in the 3-node cluster
4. **Join wnodes to the k3s cluster** (after migrations free resources)
5. **Friday maintenance:** physical hardware + storage work (see
[`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5
and [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
6. **Wnode storage rebalancing** (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1
7. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
8. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
solar-aware scale-out
> **See [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md)