KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): consolidate to single audit with corrected storage topology

Browse Source 
Delete AUDIT-2026-07-28.md (old audit). Rewrite AUDIT-2026-07-29.md as the
sole authoritative audit after reading live /etc/pve/storage.cfg, /etc/exports,
and /etc/fstab on every host.

Key corrections:
- local-nonprod on tsys5 IS the S1 physical disk (dir storage, not separate).
  S1 carries 16 sectestbed VMs, not 0 — it is NOT a migration target.
- storage.cfg is per-host, not cluster-wide — each node defines its own entries.
- BUG: tsys6 S1 export path typo (S21 instead of S1) — S1 broken on tsys6.
- GAP: tsys9 has no S1 defined in storage.cfg.
- Revised disk migration plan avoids S1, targets S2/S3/S4/D5 for IOPS spreading.
- D2 goes 17->5 VMs; S2/S3 absorb the load (7 each); S1 untouched (16 idle VMs).
- Refreshed ground truth: cnode5 deleted, cnode4 stopped (user actions this session).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 14:32:43 -05:00
parent 6199368710
commit 915111e9dd
4 changed files with 335 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files
STATUS.md
+11 -9
View File
@@ -35,10 +35,13 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
### Proxmox Cluster Ops
- [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9)
- [x] Fleet audit refreshed 2026-07-29 (live data, all 7 hosts)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 72%
- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected)
- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73%
- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model)
- [ ] CPU ~50%-idle goal: fleet idles 0-6% (under-utilized; chase via workloads, not hardware)
- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg
- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending)
- [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware)
- [ ] **BUG: tsys6 S1 export path typo** (`S21` instead of `S1`) — S1 broken on tsys6
- [ ] tsys4: blocked on PCIe NIC + RAM install
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty)
- [ ] tsys2: pending rebuild from Win10 to Proxmox
@@ -89,7 +92,7 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** |
| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 72%/37%/60% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §8 |
| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) |
| **cnode4/5 stale VMs running** | Not joined to the 3-node cluster; waste 8 GB + 8 vCPU | Stop + delete (AUDIT-2026-07-29 §10) |
| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ |
| **Fleet CPU 0-6% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads |
| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place |
| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install |
@@ -107,16 +110,15 @@ infrastructure, Proxmox cluster ops, and k8s control plane.
3. **Stop + delete stale cnode4 (601) / cnode5 (706)** — not in the 3-node cluster
4. **Join wnodes to the k3s cluster** (after migrations free resources)
5. **Friday maintenance:** physical hardware + storage work (see
[`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) §5
and [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
[`proxmox/docs/TODO.md`](proxmox/docs/TODO.md))
6. **Wnode storage rebalancing** (disk — deferred): wnode-tsys6 D5→S4, wnode-tsys7 D5→S1
7. **PDU:** Rename outlets in powerman.conf to match physical devices (Friday)
8. **k8s deferred topics:** ETL tooling, HPC scheduler, vcluster policy,
solar-aware scale-out
> **See [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md)
> for the full fresh audit with VM inventory, capacity analysis, and
> Friday migration plan.**
> **See [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md)
> for the consolidated audit with corrected storage topology, VM inventory,
> capacity analysis, and action checklist.**
## Infrastructure Summary
docs/docmap.md
+1 -2
View File
@@ -24,8 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
|----------|-------------|---------------|
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
| [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current.** RAM/CPU + HDD disk placement audit (current/target inventory + action checklist); live data from all 7 hosts | 2026-07-29 |
| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Prior fleet audit with VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current & only audit.** Consolidated RAM/CPU + HDD disk placement with corrected storage topology, current/target inventory, action checklist | 2026-07-29 |
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
proxmox/docs/AUDIT-2026-07-28.md
-276
View File
@@ -1,276 +0,0 @@
# Fresh Fleet Audit — 2026-07-28 (revised)
> **A newer audit exists:** [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) (RAM/CPU
> placement focus, with codified capacity goals). Placement numbers below are
> superseded where the two conflict; this doc remains authoritative for the
> **Friday storage/hardware plan (§5)** and storage-redundancy analysis.
> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**
**Audit time:** 2026-07-28 20:30 CDT
**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9
**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
---
## 1. HA Pairs — FIXED ✓
Both infrastructure pairs now split across storage servers:
| VM | VMID | Host | Disk | Server | tsys4 dies | tsys5 dies |
|----|------|------|------|--------|-----------|-----------|
| netinfra-01 | 103 | tsys1 | S2 | tsys5 | alive | dies |
| netinfra-02 | 904 | tsys9 | D2 | tsys4 | dies | alive |
| ucs-01 | 108 | tsys1 | D2 | tsys4 | dies | alive |
| ucs-02 | 902 | tsys9 | S2 | tsys5 | alive | dies |
**Any single storage server failure leaves one half of each pair alive.**
---
## 2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers)
| Cnode | VMID | Host | Disk | Server |
|-------|------|------|------|--------|
| cnode1 | 906 | tsys9 | S3 | **tsys5** |
| cnode2 | 705 | tsys7 | D2 | **tsys4** |
| cnode3 | 106 | tsys1 | D5 | **tsys4** |
**tsys4 failure:** cnode2 + cnode3 die → 1 of 3 → **quorum lost.**
**tsys5 failure:** cnode1 dies → 2 of 3 → quorum OK.
This is a structural constraint: 3 cnodes on 2 NFS servers means one side
holds 2. The only true fixes are:
- A 3rd storage target (not available until Friday hardware work)
- Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now)
- Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM)
**Decision: accept for now. Revisit after Friday when local-SSD boot disks
become an option for cnodes.**
---
## 3. Worker Nodes — NEEDS REBALANCING (2 storage migrations)
### Current state (problem: 3 wnodes share one physical disk)
| Wnode | VMID | Host | Disk | Server | Disk shared with |
|-------|------|------|------|--------|-----------------|
| wnode-tsys1 | 102 | tsys1 | S3 | tsys5 | — |
| wnode-tsys3 | 313 | tsys3 | **D5** | tsys4 | wnode-tsys6, wnode-tsys7 |
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | — |
| wnode-tsys6 | 100 | tsys6 | **D5** | tsys4 | wnode-tsys3, wnode-tsys7 |
| wnode-tsys7 | 701 | tsys7 | **D5** | tsys4 | wnode-tsys3, wnode-tsys6 |
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | — |
**D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS.**
**S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).**
### Recommended moves (2 PDM storage migrations)
| Wnode | Disk from | Disk to | Effect |
|-------|-----------|---------|--------|
| wnode-tsys6 (100) | D5 (tsys4) | **S4** (tsys5, 499 GB free) | D5: 3→2 wnodes |
| wnode-tsys7 (701) | D5 (tsys4) | **S1** (tsys5, 916 GB free) | D5: 2→1 wnode |
### After rebalancing — every wnode on its own physical disk
| Wnode | Host | Disk | Server |
|-------|------|------|--------|
| wnode-tsys1 | tsys1 | S3 | tsys5 |
| wnode-tsys3 | tsys3 | D5 | tsys4 |
| wnode-tsys5 | tsys5 | D2 | tsys4 |
| wnode-tsys6 | tsys6 | **S4** | **tsys5** |
| wnode-tsys7 | tsys7 | **S1** | **tsys5** |
| wnode-tsys9 | tsys9 | S2 | tsys5 |
**tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode
has a dedicated physical spindle.**
---
## 4. Host Capacity
| Host | RAM | VM RAM | RAM% | Threads | vCPU | CPU% | Notes |
|------|-----|--------|------|---------|------|------|-------|
| tsys1 | 31 GB | 36.6 GB | **118%** | 8 | 24 | 300% | Overcommitted. USB-passthrough VMs locked (bms, ca). |
| tsys3 | 31 GB | 28 GB | 90% | 8 | 8 | 100% | 1 wnode (28 GB). OK. |
| tsys4 | 15 GB | 2 GB | 13% | 8 | 2 | 25% | Storage server. Friday: →64 GB. |
| tsys5 | 94 GB | 127 GB | **135%** | 8 | 72 | 900% | Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate). |
| tsys6 | 125 GB | 48 GB | **38%** | 16 | 12 | 75% | **77 GB free.** Room for more. |
| tsys7 | 188 GB | 56 GB | **30%** | 24 | 22 | 92% | **132 GB free.** Most room in fleet. |
| tsys9 | 23 GB | 30.2 GB | **131%** | 12 | 16 | 133% | Overcommitted by 7 GB. |
### Hardware dependencies (locked VMs)
| Host | VM | Hardware | Can move? |
|------|-----|----------|-----------|
| tsys1 | pfv-bms (100) | Zigbee + Z-Wave USB dongles | Friday only (physical move) |
| tsys1 | tsys-ca (101) | Nitrokey HSM (2×) | Friday only (physical move) |
| tsys5 | (various) | SDR dongles | **No — do not move** |
---
## 5. Friday Maintenance Plan
### 5.1 Primary goal: Proxmox on tsys2
tsys2 is currently Windows 10. It has the **best local storage in the fleet**
(Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary
Friday deliverable is:
1. **Move DHCP off pfv-netboot** to the netinfra-01/02 redundant pair. This
must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware
currently, or depends on a service that does).
2. **Format tsys2**, install Proxmox.
3. **Agent inventory** — once Proxmox is live, the audit scripts
(`perf/deploy-check.sh`) can gather hardware and VM data directly,
replacing the Windows PowerShell inventory from
[`TODO.md`](TODO.md) §1.
4. **Run provisioning** — `provisioning/SetupNewSystem.sh` for baseline
hardening, then `perf/scripts/apply-tunings.sh`.
5. **Create wnode-tsys2** — boot disk on NVMe (960 PRO), data disk on SATA
SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage
of any worker.
6. **Join wnode-tsys2 to k3s cluster.**
tsys2 hardware (from [`TODO.md`](TODO.md) Windows inventory):
- Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t)
- 32 GB RAM
- Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD
- Both wired NICs are USB-attached (same anti-pattern as tsys4/9)
- NVIDIA Quadro M1200 4 GB (passthrough candidate)
- Service tag: GH1XZG2
### 5.2 Physical hardware work (tsys4, tsys5)
| Host | Work |
|------|------|
| tsys4 | PCIe NIC install (replace USB dongle), RAM 16→64 GB |
| tsys5 | 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB |
### 5.3 VM migrations (while cluster is down for hardware)
**Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:**
| VM | VMID | From | To | RAM freed | Storage stays |
|----|------|------|----|-----------|---------------|
| tsys-siem | 901 | tsys9 (131%) | tsys6 (38%) | 8 GB on tsys9 | D2 (tsys4) |
| tsys-librenms | 104 | tsys1 (118%) | tsys7 (30%) | 2 GB on tsys1 | D2 (tsys4) |
| tsys-pdm | 105 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
| tsys-workbench | 117 | tsys1 | tsys7 | 4 GB | D2 (tsys4) |
| tailscale-router | 109 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
| kali-tsys | 114 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
**Result:** tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from
30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily.
**Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):**
| VM | VMID | From | To | Hardware |
|----|------|------|----|---------|
| pfv-bms | 100 | tsys1 | tsys9 | Zigbee + Z-Wave dongles (physically move) |
| tsys-ca | 101 | tsys1 | tsys9 | Nitrokey HSMs (physically move) |
**Result:** tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only).
**Wnode storage rebalancing:**
| VM | VMID | Disk from | Disk to |
|----|------|-----------|---------|
| wnode-tsys6 | 100 | D5 (tsys4) | S4 (tsys5) |
| wnode-tsys7 | 701 | D5 (tsys4) | S1 (tsys5) |
### 5.4 Post-hardware cnode quorum fix (optional, if local-SSD available)
After Friday hardware work, consider moving one cnode to local-lvm:
- cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency
- cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd
This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS.
**Any single failure (host, storage server, or disk) leaves 2 of 3 alive.**
Deferred — user wants NFS-only for now.
---
## 6. Complete VM Inventory (running VMs only)
### tsys1 — 11 running (Infrastructure)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 100 | pfv-bms | 2 | 4096 | 32 GB | D2 | tsys4 |
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4096 | 32 GB | S3 | tsys5 |
| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | S2 | tsys5 |
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 | tsys4 |
| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D5 | tsys4 |
| 108 | tsys-ucs-01 | 2×2 | 8000 | 32 GB | D2 | tsys4 |
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 | tsys4 |
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 | tsys4 |
### tsys3 — 1 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28000 | 32 GB | D5 | tsys4 |
### tsys4 — 1 running (Storage)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
### tsys5 — 17 running (Storage + Preprod)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 509 | pfv-k8s-wnode-tsys5 | 2×4 | 32000 | 32 GB | D2 | tsys4 |
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
### tsys6 — 3 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 2×2 | 32000 | 32 GB | D5 | tsys4 |
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 | tsys4 |
### tsys7 — 6 running (Kubernetes)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 | tsys4 |
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 | tsys4 |
| 704 | TCTC | 4 | 6000 | 32 GB | D2 | tsys4 |
| 705 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 | tsys4 |
| 706 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | S2 | tsys5 |
### tsys9 — 6 running (Infrastructure)
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|------|------|-----|----------|------|---------|--------|
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 | tsys4 |
| 902 | tsys-ucs-02 | 2×2 | 8000 | 50 GB | S2 | tsys5 |
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 | tsys4 |
| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 | tsys4 |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S2 | tsys5 |
| 906 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | S3 | tsys5 |
proxmox/docs/AUDIT-2026-07-29.md
+323 -532
View File
@@ -1,637 +1,428 @@
# Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
# Fleet Audit — 2026-07-29 (consolidated, storage-corrected)
> **Scope:** End-to-end analysis of **memory and CPU placement** against the
> operator's capacity goals, plus **HDD disk placement** for IOPS balancing
> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally
> deferred to post-Friday work (NVMe on tsys5). See
> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets.
> **The single authoritative audit.** Supersedes all prior audits and the
> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict.
> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
capacity probe (`/proc/meminfo`, `qm config`, 5s CPU sample, `kubectl top`)
collected via [`tests/remote.sh`](../../tests/remote.sh).
**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware
dumps + ~05:40 CDT capacity probe). Storage configs read directly from
`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host.
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days)
**Supersedes:** placement numbers in [`PROJECT.md`](PROJECT.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) where they conflict.
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline)
---
## 1. Executive summary — fleet vs the two goals
## 1. Executive summary
| Goal | Status | Detail |
|------|--------|--------|
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 72%. Fleet is well within budget. |
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-6% busy**. Fleet is heavily CPU-underutilized. |
**Headline:** The cluster has **abundant RAM and CPU headroom everywhere**. The
memory goal is met with margin to spare; the CPU-utilization goal is not
remotely met because there simply isn't enough sustained workload yet. The right
response is *not* to add hardware — it is to (a) consolidate the always-on
baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and
ready for RackRental/compute bursts, and (c) let utilization rise naturally as
RackRental + ETL demand comes online.
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. |
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. |
| Finding | Severity | § |
|---------|----------|---|
| **tsys1/5/9 are over-allocated (>115%)** though actual usage is fine | Medium | 4 |
| **tsys6/7 idle at 6%/9% with 117/171 GB free** — perfect, underused RackRental capacity | Info | 6 |
| **tsys5 has only ONE CPU** (CPU2 socket unpopulated) — prior "dual E5620" record was wrong | Medium | 9 |
| **cnode4 + cnode5 are stale running VMs not in the k3s cluster** (waste 8 GB + 8 vCPU) | Low | 10 |
| **Fleet CPU ~0-6% idle** — no path to 50% without more workloads | Info | 5 |
| **DDR3 sticks: compatibility-blocked from the high-RAM hosts** (they need RDIMM) | Medium | 7 |
| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 |
| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 |
| **tsys6 S1 NFS export is broken** (typo: path says `S21` not `S1`) | Bug | 6 |
| **tsys9 has no S1 in storage.cfg** — can't target S1 from tsys9 | Config gap | 6 |
| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 |
| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 |
| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 |
---
## 2. Host capacity (live, 2026-07-29)
## 2. Host capacity (live, 2026-07-29 ~05:40 CDT)
CPU "threads" = logical processors (`nproc`). "vCPU" = sum of running-VM cores
× sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling).
"Actual%" = `(MemTotal - MemAvailable) / MemTotal` (steady-state truth).
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load |
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 |
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | vCPU | CPU busy | Load |
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|------|----------|------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **72%** | 0 | 26 | 3% | 1.3 |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 8 | 0% | 0.2 |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **13%** | 832 MB | 2 | 0% | 3.0 |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 72 | 6% | 5.1 |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 3 | 48 GB | 38% | **6%** | 0 | 12 | 0% | 0.2 |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 6 | 56 GB | 30% | **9%** | 0 | 22 | 1% | 1.2 |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **60%** | 0 | 16 | 1% | 0.2 |
**Notes on the table:**
- **tsys4 load 3.0 with 0% CPU busy** = NFS/disk I/O wait (it is the storage
server). The 832 MB of swap used is the only swap activity in the fleet —
minor, but worth watching; it predates this audit.
- **tsys5 "single CPU":** dmidecode reports `CPU2: Status Unpopulated`. The T7500
is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads).
See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and
useless until a 2nd CPU is seated.
- **tsys5 alloc 134%** looks alarming but actual usage is only **37%** — the 17
VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual
low). Still, it is the most over-committed host by allocation.
**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6,
cnode5 (706) deleted from tsys7. No host migrations performed yet.
---
## 3. Running VM inventory (RAM/CPU only)
## 3. Storage topology (CORRECTED — read from live configs)
> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its
> own storage entries. The same physical disk can appear under different storage
> IDs on different hosts. This was the source of the prior audit's errors.
### 3.1 Physical disk → Proxmox storage ID mapping
| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type |
|---------------|--------|------------|-----------------|---------------------|------|
| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD |
| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) |
| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD |
| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown |
| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown |
| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** |
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD |
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD |
| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD |
| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** |
### 3.2 Critical correction: `local-nonprod` = S1
On tsys5, the storage entry is:
```
dir: local-nonprod
path /mnt/pfv-tsys5/S1
```
**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**,
accessed locally as a directory storage instead of via NFS. The 16 sectestbed
and preprod VMs "on local-nonprod" are physically on S1. They appear locked to
tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage
than `S1` (NFS type), even though they point to the same directory.
**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan
to move production VMs onto S1 would have stacked them on top of 16 sandboxes.
### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday)
| Tier | Spindles | Speed | Current use |
|------|----------|-------|-------------|
| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs |
| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs |
| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) |
| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday |
| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only |
---
## 4. Per-spindle IOPS load (CORRECTED)
| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** |
|---------|--------|----------------|----------------|------------------------|
| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** |
| D5 (2TB) | tsys4 | 3 | 0 | 3 |
| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** |
| S2 (916GB) | tsys5 | 4 | 0 | 4 |
| S3 (916GB) | tsys5 | 3 | 0 | 3 |
| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity |
**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.**
The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real
IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI
pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to
S1.** Target S2, S3, S4, and D5 for IOPS spreading.
---
## 5. Storage bugs and config gaps found
### 5.1 BUG: tsys6 S1 export path typo (S21)
On tsys6's `/etc/pve/storage.cfg`:
```
nfs: S1
export /mnt/pfv-tsys5/S21 ← should be S1, not S21
```
**S1 is broken on tsys6.** Any attempt to move a VM disk to S1 on tsys6 will
fail. Fix: change `S21` → `S1`.
### 5.2 GAP: tsys9 has no S1 in storage.cfg
tsys9 defines S2, S3, S4, T5-SSD but **not S1**. If any VM on tsys9 ever needs
S1 storage, the entry must be added first. (Given that S1 is already loaded with
16 sectestbed VMs, this is low priority — but the gap should be noted.)
### 5.3 ORPHANED: D1 and D7 exports on tsys4
tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg.
- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped
templates only, no running VMs.
- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped.
These aren't causing problems but are underdocumented. Not actionable now.
---
## 6. Current VM inventory (host + disk, post-user-changes)
> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from
> start of session. `Disk` = NFS export or local storage holding the primary
> qcow2.
### tsys1 — Infrastructure (11 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB |
| 101 | tsys-ca | 2 | 2 GB |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB |
| 103 | pfv-netinfra-01 | 2 | 2 GB |
| 104 | tsys-librenms | 2 | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB |
| 108 | tsys-ucs-01 | 4 | 8 GB |
| 109 | tailscale-router | 2 | 2 GB |
| 114 | kali-tsys | 2 | 2 GB |
| 117 | tsys-secure-workbench | 2 | 4 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 |
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 |
| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 |
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 |
| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB |
| 5101 | sectestbed-siem | 4 | 10 GB |
| 5111 | ultix-streaming | 4 | 9 GB |
| 53100 | tsys-preprod-awx | 4 | 9 GB |
| 53101 | tsys-preprod-siem | 4 | 12 GB |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB |
| 5112 | ultix-offstage | 4 | 6 GB |
| 51010-51013,5105-5109,6000 | sectestbed suite (×10) | 4 ea | 4 GB ea |
| VMID | Name | vCPU | RAM | Disk | Server | Migratable? |
|------|------|------|-----|------|--------|-------------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) |
| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) |
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) |
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) |
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No |
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No |
| ×9 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No |
### tsys6 — RackRental-ready (3 running)
> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk
> but accessed via dir storage, not NFS — Proxmox will not migrate them. To
> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the
> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.*
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB |
| 600 | tsys-awx | 4 | 12 GB |
| 601 | pfv-k8s-cnode4 **(stale — not in cluster)** | 4 | 4 GB |
### tsys6 — RackRental-ready (2 running, 1 stopped)
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM | Disk | Server | Status |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running |
| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running |
| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** |
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB |
| 702 | hfnoc-uisp | 4 | 8 GB |
| 704 | TCTC | 4 | 6 GB |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB |
| 706 | pfv-k8s-cnode5 **(stale — not in cluster)** | 4 | 4 GB |
| 703 | rr-middleware | 2 | 2 GB |
### tsys7 — RackRental-ready (5 running)
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 |
| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 |
| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 |
| 704 | TCTC | 4 | 6 GB | D2 | tsys4 |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 901 | tsys-siem | 2 | 8 GB |
| 902 | tsys-ucs-02 | 4 | 8 GB |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB |
| 904 | pfv-netinfra-02 | 2 | 4 GB |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB |
| 903 | kali-rd | 2 | 2 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 |
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 |
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 |
### k3s control plane (ground truth via `kubectl`)
Only **3 cnodes** are actually joined to the pfv-k8s cluster:
| Node | Host | CPU | Mem | Role |
|------|------|-----|-----|------|
| cnode1 (906) | tsys9 | 2% | 1181 Mi (30%) | control-plane,etcd |
| cnode2 (705) | tsys7 | 2% | 1206 Mi (30%) | control-plane,etcd |
| cnode3 (106) | tsys1 | 2% | 1124 Mi (28%) | control-plane,etcd |
cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are **running as VMs but not
joined** — leftovers from the pre-rebuild 5-node cluster. See §10. **No wnodes
are joined yet** (all workers run standalone, pending the join step).
3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1).
All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.**
---
## 4. Memory analysis vs the 80% goal
**Verdict: every host passes actual usage.** The 80% steady-state rule is met
fleet-wide with margin. The hosts closest to the line:
| Host | Actual% | Alloc% | Risk | Action |
|------|---------|--------|------|--------|
| **tsys1** | **72%** | 115% | Closest to 80%; alloc > 100% | Shed 5 lightweight VMs → tsys7 (§8) |
| **tsys9** | 60% | 129% | Alloc > 100% but actual comfortable | Move tsys-siem (8 GB) → tsys6 (§8) |
| **tsys5** | 37% | 134% | Alloc highest, but mostly idle sandboxes | No action now; monitor if sandboxes wake |
| tsys3 | 10% | 90% | None | — |
| tsys6 | 6% | 38% | None (most empty) | Absorb migrated VMs |
| tsys7 | 9% | 30% | None | Absorb migrated VMs |
**Key insight:** actual usage (72% max) is far below allocation (134% max)
because of KSM/balloon. The memory goal is about *actual*, and the fleet is
healthy. The over-allocation on tsys1/5/9 is a *planning* concern, resolved by
the migrations in §8 — **no RAM purchase required**.
## 5. CPU analysis vs the 50%-idle goal
**Verdict: the fleet misses the target badly — in the *safe* direction.** Every
host is 0-6% busy at idle. This is not a capacity problem; it is an
*under-utilization* problem.
You cannot manufacture 50% utilization without workloads to run. The realistic
strategy, in priority order:
1. **Consolidate the always-on baseline onto the efficient hosts.** tsys9
(Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the
control plane + infra. Their baseline will rise toward the target as compute
workloads land.
2. **Keep tsys6/7 warm for bursts, not baseline.** They are the RackRental and
parallel-batch hosts. Their idle utilization will climb to ~50% *naturally*
as RackRental customers spin up labs — that is the designed use, not waste.
3. **Solar-gate the inefficient idle hosts when truly idle.** tsys5 (single
Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running
RackRental/batch, they are candidates for power-down / solar-aware scheduling
(aligns with the existing solar-aware scale-out concept). Running a Nehalem
box at 0% busy 24/7 is pure electricity cost.
4. **Grow into it.** As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7)
demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend
to chase the number.
---
## 6. Workload placement model
## 7. Workload placement model
| Workload family | Needs | Best hosts | Why |
|-----------------|-------|-----------|-----|
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + only local SSD/NVMe in fleet |
| **RackRental.net (network labs)** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe |
| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
CPU quality ranking (newest → oldest IPC): **tsys9** (Comet Lake 2020) >
**tsys3** (Skylake 2015) > **tsys7** (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell
2013) > **tsys5/6** (Westmere/Nehalem 2009-2010).
---
## 8. Answers to operator's questions
### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?"
**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB.
After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**.
**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most
RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode.
### 8.2 "Are tsys6/7 better for RackRental.net?"
**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need
more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s)
— deferred to Friday + StorageClass work.
### 8.3 "How should I deploy the DDR3?"
Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST).
tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB).
Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM
migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5.
---
## 7. Answers to the operator's questions
## 9. Hardware corrections
### 7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"
**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has
one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch —
useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity
and activate the CPU2 DIMM branch.
**tsys9 — YES.** It has the newest CPU in the fleet (i5-10500, 12 threads) and a
local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural
**cross-compile champion**. To grow it, first free RAM by moving tsys-siem
(8 GB) to tsys6 (§8), then bump wnode-tsys9 to **6 vCPU / 16 GB**. Actual usage
on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.
**tsys1 — NO (premise correction).** The i7-4770 is **Haswell 2013** — the
second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most
RAM-loaded host (72% actual, 115% alloc). It should **shed** VMs, not grow its
wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are
tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.
### 7.2 "Are tsys6/7 better for RackRental.net workloads?"
**Yes — exactly right.** RackRental (renting ephemeral network labs) is
RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls.
tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM.
Their older CPUs are irrelevant because labs sit idle between interactions.
**They do NOT need more RAM.** They idle at 9%/6% actual with 171/117 GB free.
They are over-provisioned in the right direction. Their real bottleneck is
**disk** (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls),
which is the deferred Friday + StorageClass work, not a RAM/CPU issue.
### 7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)
**First, identify the exact DIMM type.** Compatibility is decisive and the
fleet's DDR3 hosts are **not interchangeable**:
| Host | Platform | Accepts | Current | Empty slots | Notes |
|------|----------|---------|---------|-------------|-------|
| **tsys6** (R610) | PowerEdge server | **RDIMM ECC only** | 126 GB RDIMM 1333 | many (B-branch) | Consumer UDIMM → **no POST** |
| **tsys7** (R620) | PowerEdge server | **RDIMM ECC only** | 189 GB RDIMM 1600 | many (A7/A8 + B) | Consumer UDIMM → **no POST** |
| **tsys5** (T7500) | Workstation | **RDIMM ECC** | 94 GB RDIMM 1600 | 6 (CPU2 branch) | Useless without a 2nd CPU (§9) |
| **tsys4** (T1700) | Xeon E3 workstation | **ECC UDIMM** | 16 GB (4×4) | 0 (must swap) | Max 32 GB (4×8). 16GB sticks unsupported |
| **tsys1** (OptiPlex 9020) | Consumer | non-ECC UDIMM | 32 GB (4×8) | 0 | **Maxed.** Haswell caps at 8 GB/slot |
**Decision tree (run this before buying/installing anything):**
1. **Read the SPD** on each stick (labels or `decode-dimms` / `dmidecode -t
memory` once installed in *any* testable board). The binary question is:
**Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.**
2. **If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):**
- They fit **nowhere useful.** tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is
maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable
there). tsys4 requires ECC and caps at 8 GB/slot.
- The *only* possible target is **tsys4** — and only the 8 GB sticks, and only
if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4
16→32 GB. Marginal.
3. **If the sticks are RDIMM ECC (unlikely for Patriot, possible for some
Kingston server lines):**
- Deploy 16 GB RDIMM into **tsys7/tsys6** empty B-branch slots to balance
memory across both CPUs (currently CPU-B branches are partly empty —
single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
- But **value is low**: those hosts don't need more RAM (§7.2).
**Bottom line on DDR3:** Do not spend effort/money chasing RAM into tsys6/7 —
they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9
over-allocation) is solved for free by **VM migration** (§8), not by sticks. If
the sticks are free/cheap UDIMM, the realistic win is just **tsys4 16→32 GB**
(if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7
will stop them from POSTing.
**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but
monitor after the planned RAM upgrade.
---
## 8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)
## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced)
These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge
free RAM. **NFS-backed VMs migrate cleanly via PDM** (storage stays on the same
NFS export; only the compute host changes).
**From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):**
| VMID | Name | RAM freed on tsys1 |
|------|------|--------------------|
| 104 | tsys-librenms | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 GB |
| 117 | tsys-secure-workbench | 4 GB |
| 109 | tailscale-router | 2 GB |
| 114 | kali-tsys | 2 GB |
| | **Total** | **12 GB** |
Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB
trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.
**From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):**
| VMID | Name | RAM freed on tsys9 |
|------|------|--------------------|
| 901 | tsys-siem | 8 GB |
Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).
> **Note:** the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are
> hardware-locked and only move with a physical dongle relocation — defer to the
> onsite window, not part of this RAM/CPU pass.
---
## 9. Hardware corrections discovered
### tsys5 is single-CPU, not dual
dmidecode: `CPU2: Status Unpopulated`. The Precision T7500 has **one E5620**
(4 cores / 8 threads), not two. `PROJECT.md` and prior audits said "2× E5620" —
**that was wrong.** Implications:
- CPU capacity is half what was documented. tsys5 is the weakest single-socket
CPU in the fleet (tied era with tsys6's Nehalem).
- The 6 empty DIMM slots are on the **CPU2 branch** — they are useless until a
2nd CPU is seated. Adding RAM there today does nothing.
- If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU
capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth.
This is a higher-value hardware move than adding RAM sticks.
### tsys4 is using 832 MB of swap
Only host in the fleet with active swap. Predates this audit and is minor, but a
16 GB storage server running PBS should not be swapping. After the planned
16→32/64 GB upgrade, monitor that swap returns to zero.
---
## 10. Cleanup items
| Item | Impact | Fix |
|------|--------|-----|
| **cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster** | 8 GB + 8 vCPU wasted on stale pre-rebuild VMs | Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node. |
| **No wnodes joined to k3s** | Workers run standalone; cluster has no schedulable capacity | Join wnodes (post-migration) — the §8 moves free the resources to size them well |
| **rr-middleware (703) on tsys7** | RackRental middleware lives on a RackRental host — fine | Intentional; just confirming placement is consistent |
---
## 11. Deferred to post-Friday (storage — intentionally out of scope here)
Per operator direction, disk/storage optimization happens after the Friday
maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s
StorageClass design. Tracked in [`TODO.md`](TODO.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) §5. The one RAM/CPU-adjacent note:
**tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck**, not their CPU/RAM.
---
## 12. Summary of recommended actions (this pass)
| # | Action | Cost | Effect |
|---|--------|------|--------|
| 1 | Migrate 5 lightweight VMs tsys1→tsys7 | Free (PDM) | tsys1 alloc 115%→77% |
| 2 | Migrate tsys-siem tsys9→tsys6 | Free (PDM) | tsys9 alloc 129%→95%; room to grow wnode |
| 3 | Stop + delete stale cnode4/cnode5 | Free | Reclaim 8 GB + 8 vCPU |
| 4 | Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2) | Free | Cross-compile node ready |
| 5 | **Do not** grow wnode-tsys1; keep it small | Free | Corrects the CPU-quality premise |
| 6 | Identify DDR3 stick type before any install | Free | Avoids no-POST on tsys6/7 |
| 7 | Consider seating a 2nd CPU in tsys5 (if spare LGA771) | Cheap | Doubles tsys5 CPU; activates RAM branch |
| 8 | Solar-gate tsys5/6 when idle (no RackRental load) | Free | Stops wasting power on 0%-busy old cores |
> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
> hardware** — and by consolidating/solar-gating so idle silicon isn't burning
> power for nothing.
---
## 13. Current VM inventory (host + disk, as of 2026-07-29)
> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate,
> DellOpenManage, sectestbed stopped clones) are omitted — they consume no
> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in
> parens is which physical box serves it (tsys4 or tsys5).
### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) |
| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) |
| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) |
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) |
| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) |
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) |
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) |
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) |
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) |
| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) |
> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod /
> local-lvm — cannot migrate). They are idle sandboxes; leave them.
### tsys6 — RackRental-ready (3 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) |
| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) |
| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) |
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) |
| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) |
| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) |
| 704 | TCTC | 4 | 6 GB | D2 (tsys4) |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) |
| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) |
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) |
### Current per-spindle IOPS load (HDD exports only)
| Spindle | Server | # VMs | Status |
|---------|--------|-------|--------|
| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** |
| D5 (Hitachi 2TB) | tsys4 | 3 | OK |
| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** |
| S2 (Seagate 916GB) | tsys5 | 4 | OK |
| S3 (Seagate 916GB) | tsys5 | 3 | OK |
| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty |
**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.**
---
## 14. Target end-state inventory (RAM/CPU + IOPS-balanced)
> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The
> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to
> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers.
> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve
> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4
> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA
> pairs split across storage servers.
### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none |
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| VMID | Name | vCPU | RAM | Disk | Change |
|------|------|------|-----|------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
### tsys5 — Storage + sandboxes (17 running)
### tsys5 — Storage + sandboxes (17 running, unchanged)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** |
| — | (16 locked sandbox VMs) | — | — | local-nonprod | none |
All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay
locked (idle sandboxes, low priority).
### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none |
| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** |
| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** |
| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none |
| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** |
| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** |
| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** |
### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none |
| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** |
| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** |
| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** |
| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** |
| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** |
| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** |
| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none |
| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** |
| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** |
| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** |
| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** |
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) |
| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** |
| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*)
### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none |
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none |
| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** |
> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via
> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to
> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB**
> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's
> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and
> tsys2 (post-Friday).*
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none |
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none |
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none |
| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** |
### End-state per-spindle IOPS load
| Spindle | Server | # VMs | VMs on it |
|---------|--------|-------|-----------|
| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC |
| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 |
| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali |
| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench |
| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) |
| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* |
| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench |
| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... |
**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a
hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2).
> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16
> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7
> production VMs each is fine (they're 916 GB disks with light per-VM I/O).*
**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.**
HA pairs stay split:
- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓
- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓
---
## 15. Action checklist (do in this order)
## 11. Action checklist (do in this order)
> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk
> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move
> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O
> pauses. Recommended: do disk moves first, then host moves.
### Phase 0 — fix storage config bugs first
### Phase 0 — cleanup (frees resources first)
- **Fix tsys6 S1 typo:** edit `/etc/pve/storage.cfg` on tsys6, change
`export /mnt/pfv-tsys5/S21` → `export /mnt/pfv-tsys5/S1`
- **(Optional) Add S1 to tsys9 storage.cfg** (low priority — S1 is already
loaded; only needed if a tsys9 VM ever targets S1)
- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale.
- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total.
### Phase 1 — cleanup
### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves)
- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster
- **Confirm cnode5 (706) gone** — already deleted by operator ✓
> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one
> disk move per still-on-original-host VM first. All targets are **HDD** exports.
### Phase 2 — disk migrations off D2 (IOPS rebalancing)
- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5)
- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6)
- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7)
- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7)
- **TCTC (704):** move disk **D2 → S1** (host stays tsys7)
- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7)
> Do disk moves on VMs that are staying on their current host first.
### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs)
- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5)
- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6)
- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7)
- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7)
- **TCTC (704):** disk **D2 → S3** (host stays tsys7)
- **cnode2 (705):** disk **D2 → S2** (host stays tsys7)
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1**
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1**
### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs)
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4**
- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4**
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1**
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2**
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2**
> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs.
### Phase 4 — grow the compute wnode
### Phase 3 — grow the compute wnode
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required)
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9)
### Phase 5 — join workers to k3s
### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9)
- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9
- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%)
### Phase 5 — bring the workers into the cluster (after placement settles)
- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh`
- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
- Join wnodes via `k8s/join-servers.sh`
- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
### Verification after each phase
- `qm list` on each host — confirm VM count + status
- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80%
- `qm list` per host — confirm VM count + status
- `cat /proc/meminfo` — confirm actual RAM ≤ 80%
- `kubectl top nodes` — confirm cnodes healthy
- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills)
- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up
- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills