KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): consolidate to single audit with corrected storage topology

Browse Source 
Delete AUDIT-2026-07-28.md (old audit). Rewrite AUDIT-2026-07-29.md as the
sole authoritative audit after reading live /etc/pve/storage.cfg, /etc/exports,
and /etc/fstab on every host.

Key corrections:
- local-nonprod on tsys5 IS the S1 physical disk (dir storage, not separate).
  S1 carries 16 sectestbed VMs, not 0 — it is NOT a migration target.
- storage.cfg is per-host, not cluster-wide — each node defines its own entries.
- BUG: tsys6 S1 export path typo (S21 instead of S1) — S1 broken on tsys6.
- GAP: tsys9 has no S1 defined in storage.cfg.
- Revised disk migration plan avoids S1, targets S2/S3/S4/D5 for IOPS spreading.
- D2 goes 17->5 VMs; S2/S3 absorb the load (7 each); S1 untouched (16 idle VMs).
- Refreshed ground truth: cnode5 deleted, cnode4 stopped (user actions this session).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 14:32:43 -05:00
parent 6199368710
commit 915111e9dd
4 changed files with 335 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-07-29.md
+323 -532
View File
@@ -1,637 +1,428 @@
# Fleet Audit — 2026-07-29 (RAM/CPU placement focus)
# Fleet Audit — 2026-07-29 (consolidated, storage-corrected)
> **Scope:** End-to-end analysis of **memory and CPU placement** against the
> operator's capacity goals, plus **HDD disk placement** for IOPS balancing
> (§13-§15). SSD/NVMe tiering and k8s StorageClass design are intentionally
> deferred to post-Friday work (NVMe on tsys5). See
> [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) for the standing targets.
> **The single authoritative audit.** Supersedes all prior audits and the
> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict.
> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
**Audit time:** 2026-07-29, ~05:09-05:16 CDT (live data from all 7 hosts)
**Method:** `perf/deploy-check.sh` (full hardware dump per host) + a live
capacity probe (`/proc/meminfo`, `qm config`, 5s CPU sample, `kubectl top`)
collected via [`tests/remote.sh`](../../tests/remote.sh).
**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware
dumps + ~05:40 CDT capacity probe). Storage configs read directly from
`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host.
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline 5+ days)
**Supersedes:** placement numbers in [`PROJECT.md`](PROJECT.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) where they conflict.
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline)
---
## 1. Executive summary — fleet vs the two goals
## 1. Executive summary
| Goal | Status | Detail |
|------|--------|--------|
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 72%. Fleet is well within budget. |
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-6% busy**. Fleet is heavily CPU-underutilized. |
**Headline:** The cluster has **abundant RAM and CPU headroom everywhere**. The
memory goal is met with margin to spare; the CPU-utilization goal is not
remotely met because there simply isn't enough sustained workload yet. The right
response is *not* to add hardware — it is to (a) consolidate the always-on
baseline onto efficient hosts, (b) keep the big-RAM hosts (tsys6/7) warm and
ready for RackRental/compute bursts, and (c) let utilization rise naturally as
RackRental + ETL demand comes online.
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. |
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. |
| Finding | Severity | § |
|---------|----------|---|
| **tsys1/5/9 are over-allocated (>115%)** though actual usage is fine | Medium | 4 |
| **tsys6/7 idle at 6%/9% with 117/171 GB free** — perfect, underused RackRental capacity | Info | 6 |
| **tsys5 has only ONE CPU** (CPU2 socket unpopulated) — prior "dual E5620" record was wrong | Medium | 9 |
| **cnode4 + cnode5 are stale running VMs not in the k3s cluster** (waste 8 GB + 8 vCPU) | Low | 10 |
| **Fleet CPU ~0-6% idle** — no path to 50% without more workloads | Info | 5 |
| **DDR3 sticks: compatibility-blocked from the high-RAM hosts** (they need RDIMM) | Medium | 7 |
| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 |
| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 |
| **tsys6 S1 NFS export is broken** (typo: path says `S21` not `S1`) | Bug | 6 |
| **tsys9 has no S1 in storage.cfg** — can't target S1 from tsys9 | Config gap | 6 |
| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 |
| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 |
| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 |
---
## 2. Host capacity (live, 2026-07-29)
## 2. Host capacity (live, 2026-07-29 ~05:40 CDT)
CPU "threads" = logical processors (`nproc`). "vCPU" = sum of running-VM cores
× sockets. "Alloc%" = running-VM RAM ÷ host RAM (worst-case ceiling).
"Actual%" = `(MemTotal - MemAvailable) / MemTotal` (steady-state truth).
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load |
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 |
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | vCPU | CPU busy | Load |
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|------|----------|------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **72%** | 0 | 26 | 3% | 1.3 |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 8 | 0% | 0.2 |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **13%** | 832 MB | 2 | 0% | 3.0 |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 72 | 6% | 5.1 |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 3 | 48 GB | 38% | **6%** | 0 | 12 | 0% | 0.2 |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 6 | 56 GB | 30% | **9%** | 0 | 22 | 1% | 1.2 |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **60%** | 0 | 16 | 1% | 0.2 |
**Notes on the table:**
- **tsys4 load 3.0 with 0% CPU busy** = NFS/disk I/O wait (it is the storage
server). The 832 MB of swap used is the only swap activity in the fleet —
minor, but worth watching; it predates this audit.
- **tsys5 "single CPU":** dmidecode reports `CPU2: Status Unpopulated`. The T7500
is dual-capable but only one E5620 is installed (4 cores, HT on = 8 threads).
See §9. All 6 DIMMs are on CPU1's branch; CPU2's 6 DIMM slots are empty and
useless until a 2nd CPU is seated.
- **tsys5 alloc 134%** looks alarming but actual usage is only **37%** — the 17
VMs are mostly idle sectestbed/preprod sandboxes (KSM + balloon keep actual
low). Still, it is the most over-committed host by allocation.
**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6,
cnode5 (706) deleted from tsys7. No host migrations performed yet.
---
## 3. Running VM inventory (RAM/CPU only)
## 3. Storage topology (CORRECTED — read from live configs)
> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its
> own storage entries. The same physical disk can appear under different storage
> IDs on different hosts. This was the source of the prior audit's errors.
### 3.1 Physical disk → Proxmox storage ID mapping
| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type |
|---------------|--------|------------|-----------------|---------------------|------|
| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD |
| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) |
| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD |
| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown |
| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown |
| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** |
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD |
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD |
| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD |
| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** |
### 3.2 Critical correction: `local-nonprod` = S1
On tsys5, the storage entry is:
```
dir: local-nonprod
path /mnt/pfv-tsys5/S1
```
**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**,
accessed locally as a directory storage instead of via NFS. The 16 sectestbed
and preprod VMs "on local-nonprod" are physically on S1. They appear locked to
tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage
than `S1` (NFS type), even though they point to the same directory.
**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan
to move production VMs onto S1 would have stacked them on top of 16 sandboxes.
### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday)
| Tier | Spindles | Speed | Current use |
|------|----------|-------|-------------|
| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs |
| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs |
| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) |
| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday |
| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only |
---
## 4. Per-spindle IOPS load (CORRECTED)
| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** |
|---------|--------|----------------|----------------|------------------------|
| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** |
| D5 (2TB) | tsys4 | 3 | 0 | 3 |
| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** |
| S2 (916GB) | tsys5 | 4 | 0 | 4 |
| S3 (916GB) | tsys5 | 3 | 0 | 3 |
| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity |
**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.**
The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real
IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI
pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to
S1.** Target S2, S3, S4, and D5 for IOPS spreading.
---
## 5. Storage bugs and config gaps found
### 5.1 BUG: tsys6 S1 export path typo (S21)
On tsys6's `/etc/pve/storage.cfg`:
```
nfs: S1
export /mnt/pfv-tsys5/S21 ← should be S1, not S21
```
**S1 is broken on tsys6.** Any attempt to move a VM disk to S1 on tsys6 will
fail. Fix: change `S21` → `S1`.
### 5.2 GAP: tsys9 has no S1 in storage.cfg
tsys9 defines S2, S3, S4, T5-SSD but **not S1**. If any VM on tsys9 ever needs
S1 storage, the entry must be added first. (Given that S1 is already loaded with
16 sectestbed VMs, this is low priority — but the gap should be noted.)
### 5.3 ORPHANED: D1 and D7 exports on tsys4
tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg.
- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped
templates only, no running VMs.
- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped.
These aren't causing problems but are underdocumented. Not actionable now.
---
## 6. Current VM inventory (host + disk, post-user-changes)
> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from
> start of session. `Disk` = NFS export or local storage holding the primary
> qcow2.
### tsys1 — Infrastructure (11 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB |
| 101 | tsys-ca | 2 | 2 GB |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB |
| 103 | pfv-netinfra-01 | 2 | 2 GB |
| 104 | tsys-librenms | 2 | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB |
| 108 | tsys-ucs-01 | 4 | 8 GB |
| 109 | tailscale-router | 2 | 2 GB |
| 114 | kali-tsys | 2 | 2 GB |
| 117 | tsys-secure-workbench | 2 | 4 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 |
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 |
| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 |
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 |
| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 |
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB |
| 5101 | sectestbed-siem | 4 | 10 GB |
| 5111 | ultix-streaming | 4 | 9 GB |
| 53100 | tsys-preprod-awx | 4 | 9 GB |
| 53101 | tsys-preprod-siem | 4 | 12 GB |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB |
| 5112 | ultix-offstage | 4 | 6 GB |
| 51010-51013,5105-5109,6000 | sectestbed suite (×10) | 4 ea | 4 GB ea |
| VMID | Name | vCPU | RAM | Disk | Server | Migratable? |
|------|------|------|-----|------|--------|-------------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) |
| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) |
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) |
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) |
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No |
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No |
| ×9 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No |
### tsys6 — RackRental-ready (3 running)
> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk
> but accessed via dir storage, not NFS — Proxmox will not migrate them. To
> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the
> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.*
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB |
| 600 | tsys-awx | 4 | 12 GB |
| 601 | pfv-k8s-cnode4 **(stale — not in cluster)** | 4 | 4 GB |
### tsys6 — RackRental-ready (2 running, 1 stopped)
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM | Disk | Server | Status |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running |
| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running |
| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** |
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB |
| 702 | hfnoc-uisp | 4 | 8 GB |
| 704 | TCTC | 4 | 6 GB |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB |
| 706 | pfv-k8s-cnode5 **(stale — not in cluster)** | 4 | 4 GB |
| 703 | rr-middleware | 2 | 2 GB |
### tsys7 — RackRental-ready (5 running)
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 |
| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 |
| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 |
| 704 | TCTC | 4 | 6 GB | D2 | tsys4 |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM |
|------|------|------|-----|
| 901 | tsys-siem | 2 | 8 GB |
| 902 | tsys-ucs-02 | 4 | 8 GB |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB |
| 904 | pfv-netinfra-02 | 2 | 4 GB |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB |
| 903 | kali-rd | 2 | 2 GB |
| VMID | Name | vCPU | RAM | Disk | Server |
|------|------|------|-----|------|--------|
| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 |
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 |
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 |
### k3s control plane (ground truth via `kubectl`)
Only **3 cnodes** are actually joined to the pfv-k8s cluster:
| Node | Host | CPU | Mem | Role |
|------|------|-----|-----|------|
| cnode1 (906) | tsys9 | 2% | 1181 Mi (30%) | control-plane,etcd |
| cnode2 (705) | tsys7 | 2% | 1206 Mi (30%) | control-plane,etcd |
| cnode3 (106) | tsys1 | 2% | 1124 Mi (28%) | control-plane,etcd |
cnode4 (601 on tsys6) and cnode5 (706 on tsys7) are **running as VMs but not
joined** — leftovers from the pre-rebuild 5-node cluster. See §10. **No wnodes
are joined yet** (all workers run standalone, pending the join step).
3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1).
All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.**
---
## 4. Memory analysis vs the 80% goal
**Verdict: every host passes actual usage.** The 80% steady-state rule is met
fleet-wide with margin. The hosts closest to the line:
| Host | Actual% | Alloc% | Risk | Action |
|------|---------|--------|------|--------|
| **tsys1** | **72%** | 115% | Closest to 80%; alloc > 100% | Shed 5 lightweight VMs → tsys7 (§8) |
| **tsys9** | 60% | 129% | Alloc > 100% but actual comfortable | Move tsys-siem (8 GB) → tsys6 (§8) |
| **tsys5** | 37% | 134% | Alloc highest, but mostly idle sandboxes | No action now; monitor if sandboxes wake |
| tsys3 | 10% | 90% | None | — |
| tsys6 | 6% | 38% | None (most empty) | Absorb migrated VMs |
| tsys7 | 9% | 30% | None | Absorb migrated VMs |
**Key insight:** actual usage (72% max) is far below allocation (134% max)
because of KSM/balloon. The memory goal is about *actual*, and the fleet is
healthy. The over-allocation on tsys1/5/9 is a *planning* concern, resolved by
the migrations in §8 — **no RAM purchase required**.
## 5. CPU analysis vs the 50%-idle goal
**Verdict: the fleet misses the target badly — in the *safe* direction.** Every
host is 0-6% busy at idle. This is not a capacity problem; it is an
*under-utilization* problem.
You cannot manufacture 50% utilization without workloads to run. The realistic
strategy, in priority order:
1. **Consolidate the always-on baseline onto the efficient hosts.** tsys9
(Comet Lake) and tsys3 (Skylake) are cheap to idle and should host the
control plane + infra. Their baseline will rise toward the target as compute
workloads land.
2. **Keep tsys6/7 warm for bursts, not baseline.** They are the RackRental and
parallel-batch hosts. Their idle utilization will climb to ~50% *naturally*
as RackRental customers spin up labs — that is the designed use, not waste.
3. **Solar-gate the inefficient idle hosts when truly idle.** tsys5 (single
Westmere) and tsys6 (Nehalem) are the worst perf-per-watt. When not running
RackRental/batch, they are candidates for power-down / solar-aware scheduling
(aligns with the existing solar-aware scale-out concept). Running a Nehalem
box at 0% busy 24/7 is pure electricity cost.
4. **Grow into it.** As ETL/cross-compile (tsys3/9) and RackRental (tsys6/7)
demand comes online, idle CPU rises toward 50% on its own. Do not pre-spend
to chase the number.
---
## 6. Workload placement model
## 7. Workload placement model
| Workload family | Needs | Best hosts | Why |
|-----------------|-------|-----------|-----|
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + only local SSD/NVMe in fleet |
| **RackRental.net (network labs)** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe |
| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
CPU quality ranking (newest → oldest IPC): **tsys9** (Comet Lake 2020) >
**tsys3** (Skylake 2015) > **tsys7** (Ivy Bridge 2013) ≈ tsys4/tsys1 (Haswell
2013) > **tsys5/6** (Westmere/Nehalem 2009-2010).
---
## 8. Answers to operator's questions
### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?"
**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB.
After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**.
**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most
RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode.
### 8.2 "Are tsys6/7 better for RackRental.net?"
**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need
more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s)
— deferred to Friday + StorageClass work.
### 8.3 "How should I deploy the DDR3?"
Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST).
tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB).
Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM
migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5.
---
## 7. Answers to the operator's questions
## 9. Hardware corrections
### 7.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC/cross-compile?"
**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has
one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch —
useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity
and activate the CPU2 DIMM branch.
**tsys9 — YES.** It has the newest CPU in the fleet (i5-10500, 12 threads) and a
local SSD. wnode-tsys9 is currently tiny (4 vCPU / 4 GB). It is the natural
**cross-compile champion**. To grow it, first free RAM by moving tsys-siem
(8 GB) to tsys6 (§8), then bump wnode-tsys9 to **6 vCPU / 16 GB**. Actual usage
on tsys9 today is 60%, so ~16 GB of grow-room exists after the siem move.
**tsys1 — NO (premise correction).** The i7-4770 is **Haswell 2013** — the
second-oldest CPU in the fleet, not a "nice" one. tsys1 is also the most
RAM-loaded host (72% actual, 115% alloc). It should **shed** VMs, not grow its
wnode. Keep wnode-tsys1 small (4/4) or shrink it. The genuinely-nice CPUs are
tsys9 (Comet Lake) and tsys3 (Skylake) — direct those to compute, not tsys1.
### 7.2 "Are tsys6/7 better for RackRental.net workloads?"
**Yes — exactly right.** RackRental (renting ephemeral network labs) is
RAM-bound and CPU-light: many containers/idle namespaces, bursty image pulls.
tsys7 (189 GB, 24 threads) and tsys6 (126 GB, 16 threads) are unmatched for RAM.
Their older CPUs are irrelevant because labs sit idle between interactions.
**They do NOT need more RAM.** They idle at 9%/6% actual with 171/117 GB free.
They are over-provisioned in the right direction. Their real bottleneck is
**disk** (local-lvm is USB 2.0 HDD ≈ 30 MB/s — brutal for container image pulls),
which is the deferred Friday + StorageClass work, not a RAM/CPU issue.
### 7.3 "How should I deploy the cheap DDR3?" (4× Patriot 16GB, 2× Kingston 8GB, 2× Kingston 8GB diff)
**First, identify the exact DIMM type.** Compatibility is decisive and the
fleet's DDR3 hosts are **not interchangeable**:
| Host | Platform | Accepts | Current | Empty slots | Notes |
|------|----------|---------|---------|-------------|-------|
| **tsys6** (R610) | PowerEdge server | **RDIMM ECC only** | 126 GB RDIMM 1333 | many (B-branch) | Consumer UDIMM → **no POST** |
| **tsys7** (R620) | PowerEdge server | **RDIMM ECC only** | 189 GB RDIMM 1600 | many (A7/A8 + B) | Consumer UDIMM → **no POST** |
| **tsys5** (T7500) | Workstation | **RDIMM ECC** | 94 GB RDIMM 1600 | 6 (CPU2 branch) | Useless without a 2nd CPU (§9) |
| **tsys4** (T1700) | Xeon E3 workstation | **ECC UDIMM** | 16 GB (4×4) | 0 (must swap) | Max 32 GB (4×8). 16GB sticks unsupported |
| **tsys1** (OptiPlex 9020) | Consumer | non-ECC UDIMM | 32 GB (4×8) | 0 | **Maxed.** Haswell caps at 8 GB/slot |
**Decision tree (run this before buying/installing anything):**
1. **Read the SPD** on each stick (labels or `decode-dimms` / `dmidecode -t
memory` once installed in *any* testable board). The binary question is:
**Registered/Buffered (RDIMM) vs Unbuffered (UDIMM), and ECC vs non-ECC.**
2. **If the sticks are consumer UDIMM (Patriot/Kingston gaming — most likely):**
- They fit **nowhere useful.** tsys6/7/5 refuse UDIMM (RDIMM-only). tsys1 is
maxed and caps at 8 GB/slot (so the 16 GB Patriot sticks are unusable
there). tsys4 requires ECC and caps at 8 GB/slot.
- The *only* possible target is **tsys4** — and only the 8 GB sticks, and only
if the T1700 tolerates non-ECC (unreliable). Expected yield: tsys4
16→32 GB. Marginal.
3. **If the sticks are RDIMM ECC (unlikely for Patriot, possible for some
Kingston server lines):**
- Deploy 16 GB RDIMM into **tsys7/tsys6** empty B-branch slots to balance
memory across both CPUs (currently CPU-B branches are partly empty —
single-CPU-channel hurts bandwidth). 8 GB RDIMM likewise.
- But **value is low**: those hosts don't need more RAM (§7.2).
**Bottom line on DDR3:** Do not spend effort/money chasing RAM into tsys6/7 —
they are already RAM-rich and underused. The fleet's only RAM pressure (tsys1/5/9
over-allocation) is solved for free by **VM migration** (§8), not by sticks. If
the sticks are free/cheap UDIMM, the realistic win is just **tsys4 16→32 GB**
(if ECC-tolerant). Verify type before installing — putting UDIMM into tsys6/7
will stop them from POSTing.
**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but
monitor after the planned RAM upgrade.
---
## 8. Recommended VM migrations (RAM/CPU relief — zero hardware cost)
## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced)
These moves relieve the over-allocated infra hosts by exploiting tsys6/7's huge
free RAM. **NFS-backed VMs migrate cleanly via PDM** (storage stays on the same
NFS export; only the compute host changes).
**From tsys1 (72% actual / 115% alloc) → tsys7 (9% / 171 GB free):**
| VMID | Name | RAM freed on tsys1 |
|------|------|--------------------|
| 104 | tsys-librenms | 2 GB |
| 105 | tsys-proxmox-datacenter | 2 GB |
| 117 | tsys-secure-workbench | 4 GB |
| 109 | tailscale-router | 2 GB |
| 114 | kali-tsys | 2 GB |
| | **Total** | **12 GB** |
Result: tsys1 alloc 36→24 GB (77%), actual ~72%→~45%. tsys7 absorbs 12 GB
trivially. tsys1 becomes a lean cnode3 + netinfra-01 + ucs-01 + wnode host.
**From tsys9 (60% actual / 129% alloc) → tsys6 (6% / 117 GB free):**
| VMID | Name | RAM freed on tsys9 |
|------|------|--------------------|
| 901 | tsys-siem | 8 GB |
Result: tsys9 alloc 30→22 GB (95%), frees room to grow wnode-tsys9 (§7.1).
> **Note:** the USB-passthrough VMs on tsys1 (pfv-bms 100, tsys-ca 101) are
> hardware-locked and only move with a physical dongle relocation — defer to the
> onsite window, not part of this RAM/CPU pass.
---
## 9. Hardware corrections discovered
### tsys5 is single-CPU, not dual
dmidecode: `CPU2: Status Unpopulated`. The Precision T7500 has **one E5620**
(4 cores / 8 threads), not two. `PROJECT.md` and prior audits said "2× E5620" —
**that was wrong.** Implications:
- CPU capacity is half what was documented. tsys5 is the weakest single-socket
CPU in the fleet (tied era with tsys6's Nehalem).
- The 6 empty DIMM slots are on the **CPU2 branch** — they are useless until a
2nd CPU is seated. Adding RAM there today does nothing.
- If a spare LGA771 Xeon is available, seating a 2nd CPU would (a) double CPU
capacity and (b) activate the CPU2 DIMM branch and dual-channel bandwidth.
This is a higher-value hardware move than adding RAM sticks.
### tsys4 is using 832 MB of swap
Only host in the fleet with active swap. Predates this audit and is minor, but a
16 GB storage server running PBS should not be swapping. After the planned
16→32/64 GB upgrade, monitor that swap returns to zero.
---
## 10. Cleanup items
| Item | Impact | Fix |
|------|--------|-----|
| **cnode4 (601, tsys6) + cnode5 (706, tsys7) running but not in cluster** | 8 GB + 8 vCPU wasted on stale pre-rebuild VMs | Stop + delete (or re-join if a 5-node CP is actually wanted). The live cluster is 3-node. |
| **No wnodes joined to k3s** | Workers run standalone; cluster has no schedulable capacity | Join wnodes (post-migration) — the §8 moves free the resources to size them well |
| **rr-middleware (703) on tsys7** | RackRental middleware lives on a RackRental host — fine | Intentional; just confirming placement is consistent |
---
## 11. Deferred to post-Friday (storage — intentionally out of scope here)
Per operator direction, disk/storage optimization happens after the Friday
maintenance window (NVMe on tsys5, D3 SSD relocation, tsys4 NIC+RAM) and the k8s
StorageClass design. Tracked in [`TODO.md`](TODO.md) and
[`AUDIT-2026-07-28.md`](AUDIT-2026-07-28.md) §5. The one RAM/CPU-adjacent note:
**tsys6/7's USB-2.0 local-lvm is the RackRental bottleneck**, not their CPU/RAM.
---
## 12. Summary of recommended actions (this pass)
| # | Action | Cost | Effect |
|---|--------|------|--------|
| 1 | Migrate 5 lightweight VMs tsys1→tsys7 | Free (PDM) | tsys1 alloc 115%→77% |
| 2 | Migrate tsys-siem tsys9→tsys6 | Free (PDM) | tsys9 alloc 129%→95%; room to grow wnode |
| 3 | Stop + delete stale cnode4/cnode5 | Free | Reclaim 8 GB + 8 vCPU |
| 4 | Grow wnode-tsys9 to 6 vCPU / 16 GB (after #2) | Free | Cross-compile node ready |
| 5 | **Do not** grow wnode-tsys1; keep it small | Free | Corrects the CPU-quality premise |
| 6 | Identify DDR3 stick type before any install | Free | Avoids no-POST on tsys6/7 |
| 7 | Consider seating a 2nd CPU in tsys5 (if spare LGA771) | Cheap | Doubles tsys5 CPU; activates RAM branch |
| 8 | Solar-gate tsys5/6 when idle (no RackRental load) | Free | Stops wasting power on 0%-busy old cores |
> Memory goal: **already met.** CPU goal: **chase it by adding workloads, not
> hardware** — and by consolidating/solar-gating so idle silicon isn't burning
> power for nothing.
---
## 13. Current VM inventory (host + disk, as of 2026-07-29)
> Running VMs only. Stopped templates (KNELTemplate×3, RestoreTemplate,
> DellOpenManage, sectestbed stopped clones) are omitted — they consume no
> RAM/CPU. `Disk` = the NFS export holding the primary qcow2; the **server** in
> parens is which physical box serves it (tsys4 or tsys5).
### tsys1 — Infrastructure (11 running, 36 GB alloc / 72% actual)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 (tsys4) |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) |
| 104 | tsys-librenms | 2 | 2 GB | D2 (tsys4) |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 (tsys4) |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) |
| 114 | kali-tsys | 2 | 2 GB | D2 (tsys4) |
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 (tsys4) |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm (LOCAL) |
### tsys5 — Storage + sandboxes (17 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 (tsys4) |
| 5111 | ultix-streaming | 4 | 9 GB | local (LOCAL) |
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm (LOCAL) |
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (LOCAL) |
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (LOCAL) |
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (LOCAL) |
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (LOCAL) |
| ×9 | sectestbed suite (5105-5109,5106,5107,5108,51010-51013,6000) | 4 ea | 4 GB ea | local-nonprod (LOCAL) |
> The 16 LOCAL-tagged VMs are **locked to tsys5** (disks on local-nonprod /
> local-lvm — cannot migrate). They are idle sandboxes; leave them.
### tsys6 — RackRental-ready (3 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) |
| 600 | tsys-awx | 4 | 12 GB | D2 (tsys4) |
| 601 | pfv-k8s-cnode4 **(stale)** | 4 | 4 GB | D2 (tsys4) |
### tsys7 — RackRental-ready (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) |
| 702 | hfnoc-uisp | 4 | 8 GB | D2 (tsys4) |
| 703 | rr-middleware | 2 | 2 GB | D2 (tsys4) |
| 704 | TCTC | 4 | 6 GB | D2 (tsys4) |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 (tsys4) |
| 706 | pfv-k8s-cnode5 **(stale)** | 4 | 4 GB | S2 (tsys5) |
### tsys9 — Infra + control plane (6 running)
| VMID | Name | vCPU | RAM | Disk (server) |
|------|------|------|-----|---------------|
| 901 | tsys-siem | 2 | 8 GB | D2 (tsys4) |
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 (tsys5) |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) |
### Current per-spindle IOPS load (HDD exports only)
| Spindle | Server | # VMs | Status |
|---------|--------|-------|--------|
| **D2** (WDC Red 3TB) | tsys4 | **17** | **CRITICAL — 65% of all NFS VMs contend here** |
| D5 (Hitachi 2TB) | tsys4 | 3 | OK |
| S1 (Seagate 916GB) | tsys5 | **0** | **EMPTY — wasted spindle** |
| S2 (Seagate 916GB) | tsys5 | 4 | OK |
| S3 (Seagate 916GB) | tsys5 | 3 | OK |
| S4 (Toshiba 458GB) | tsys5 | 1 | Nearly empty |
**The single biggest IOPS problem is D2 carrying 17 VMs while S1 sits empty.**
---
## 14. Target end-state inventory (RAM/CPU + IOPS-balanced)
> All disks stay on **HDD** (no SSD/NVMe moves — deferred to post-Friday). The
> goals: (a) relieve tsys1/9 RAM over-allocation, (b) spread every spindle to
> ~4-5 VMs (D2: 17→5, S1: 0→5), (c) keep HA pairs split across storage servers.
> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve
> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4
> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA
> pairs split across storage servers.
### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-bms | 2 | 4 GB | D2 (tsys4) | none |
| 101 | tsys-ca | 2 | 2 GB | D2 (tsys4) | none |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 (tsys5) | none |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 (tsys5) | none |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 (tsys4) | none |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 (tsys4) | none |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none |
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none |
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none |
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none |
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none |
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none |
### tsys3 — Compute (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 (tsys4) | none |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none |
### tsys4 — Storage (1 running)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| VMID | Name | vCPU | RAM | Disk | Change |
|------|------|------|-----|------|--------|
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
### tsys5 — Storage + sandboxes (17 running)
### tsys5 — Storage + sandboxes (17 running, unchanged)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | **S4** (tsys5) | **disk D2→S4** |
| — | (16 locked sandbox VMs) | — | — | local-nonprod | none |
All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay
locked (idle sandboxes, low priority).
### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 (tsys5) | none |
| 600 | tsys-awx | 4 | 12 GB | **D5** (tsys4) | **disk D2→D5** |
| 901 | tsys-siem | 2 | 8 GB | **S1** (tsys5) | **host tsys9→tsys6, disk D2→S1** |
| ~~601~~ | ~~cnode4 (stale)~~ | — | — | — | **DELETE** |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none |
| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** |
| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** |
| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** |
### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 (tsys5) | none |
| 702 | hfnoc-uisp | 4 | 8 GB | **S1** (tsys5) | **disk D2→S1** |
| 703 | rr-middleware | 2 | 2 GB | **S4** (tsys5) | **disk D2→S4** |
| 704 | TCTC | 4 | 6 GB | **S1** (tsys5) | **disk D2→S1** |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | S2 (tsys5) | **disk D2→S2** |
| 104 | tsys-librenms | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S1** (tsys5) | **host tsys1→tsys7, disk D2→S1** |
| 109 | tailscale-router | 2 | 2 GB | D2 (tsys4) | **host tsys1→tsys7** |
| 114 | kali-tsys | 2 | 2 GB | **S3** (tsys5) | **host tsys1→tsys7, disk D2→S3** |
| 117 | tsys-secure-workbench | 2 | 4 GB | **S4** (tsys5) | **host tsys1→tsys7, disk D2→S4** |
| ~~706~~ | ~~cnode5 (stale)~~ | — | — | — | **DELETE** |
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none |
| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** |
| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** |
| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** |
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** |
| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** |
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) |
| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** |
| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
### tsys9 — Infra + control + compute (5 running, ~26 GB alloc → 108%*)
### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%)
| VMID | Name | vCPU | RAM | Disk (server) | Change |
|------|------|------|-----|---------------|--------|
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 (tsys5) | none |
| 903 | kali-rd | 2 | 2 GB | D5 (tsys4) | none |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 (tsys4) | none |
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 (tsys5) | **grow 4c/4G→6c/8G** |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 (tsys5) | none |
| ~~901~~ | ~~tsys-siem~~ | — | — | — | **move→tsys6** |
> *\*tsys9 stays over 100% by **allocation**, but actual usage stays <80% via
> KSM/balloon (it's 60% today with MORE allocated). wnode9 grows modestly to
> 8 GB / 6 cores — enough to be useful for cross-compile. The full **16 GB**
> cross-compile node from §7.1 requires also moving ucs02 (8 GB) to tsys6; that's
> the optional bullet in §15. The real heavy compute wnodes are tsys3 (now) and
> tsys2 (post-Friday).*
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|------|------|------|-----|------|--------|--------|
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none |
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none |
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none |
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** |
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none |
| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** |
### End-state per-spindle IOPS load
| Spindle | Server | # VMs | VMs on it |
|---------|--------|-------|-----------|
| D2 (WDC Red 3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
| D5 (Hitachi 2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
| S1 (Seagate 916GB) | tsys5 | **5** | librenms, pdm, siem, uisp, TCTC |
| S2 (Seagate 916GB) | tsys5 | 4 | netinfra01, cnode2, ucs02, wnode9 |
| S3 (Seagate 916GB) | tsys5 | 4 | wnode1, wnode7, cnode1, kali |
| S4 (Toshiba 458GB) | tsys5 | 4 | wnode5, wnode6, rr-middleware, workbench |
| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) |
| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* |
| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench |
| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... |
**D2 goes 17→5. S1 goes 0→5. Every spindle carries 4-5 VMs. No spindle is a
hotspot.** HA pairs stay split (netinfra: S2/D2; ucs: D2/S2).
> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16
> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7
> production VMs each is fine (they're 916 GB disks with light per-VM I/O).*
**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.**
HA pairs stay split:
- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓
- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓
---
## 15. Action checklist (do in this order)
## 11. Action checklist (do in this order)
> **Two operation types:** **host migrate** (PDM "Migrate" — moves compute, disk
> stays put) and **disk migrate** (Proxmox UI: VM → Hardware → disk → Move
> Storage, or `qm move-disk`). For NFS VMs both can be done live with brief I/O
> pauses. Recommended: do disk moves first, then host moves.
### Phase 0 — fix storage config bugs first
### Phase 0 — cleanup (frees resources first)
- **Fix tsys6 S1 typo:** edit `/etc/pve/storage.cfg` on tsys6, change
`export /mnt/pfv-tsys5/S21` → `export /mnt/pfv-tsys5/S1`
- **(Optional) Add S1 to tsys9 storage.cfg** (low priority — S1 is already
loaded; only needed if a tsys9 VM ever targets S1)
- **Stop + destroy cnode4 (VMID 601) on tsys6** — not joined to the 3-node cluster; stale.
- **Stop + destroy cnode5 (VMID 706) on tsys7** — same. Frees 8 GB + 8 vCPU total.
### Phase 1 — cleanup
### Phase 1 — disk migrations off D2 (IOPS rebalancing, ~11 moves)
- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster
- **Confirm cnode5 (706) gone** — already deleted by operator ✓
> Order: move the VMs whose HOST is also changing last (Phase 2), so you do one
> disk move per still-on-original-host VM first. All targets are **HDD** exports.
### Phase 2 — disk migrations off D2 (IOPS rebalancing)
- **wnode-tsys5 (509):** move disk **D2 → S4** (host stays tsys5)
- **tsys-awx (600):** move disk **D2 → D5** (host stays tsys6)
- **hfnoc-uisp (702):** move disk **D2 → S1** (host stays tsys7)
- **rr-middleware (703):** move disk **D2 → S4** (host stays tsys7)
- **TCTC (704):** move disk **D2 → S1** (host stays tsys7)
- **cnode2 (705):** move disk **D2 → S2** (host stays tsys7)
> Do disk moves on VMs that are staying on their current host first.
### Phase 2 — host + disk migrations (RAM/CPU relief, 6 VMs)
- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5)
- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6)
- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7)
- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7)
- **TCTC (704):** disk **D2 → S3** (host stays tsys7)
- **cnode2 (705):** disk **D2 → S2** (host stays tsys7)
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S1**
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S1**
### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs)
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4**
- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S4**
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S1**
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2**
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3**
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2**
> Result: tsys1 drops to 6 VMs (~24 GB alloc, 77%). tsys9 drops to 5 VMs.
### Phase 4 — grow the compute wnode
### Phase 3 — grow the compute wnode
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required)
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (requires stop/start; done after siem leaves tsys9)
### Phase 5 — join workers to k3s
### Phase 4 — optional (only if you want a 16 GB cross-compile node on tsys9)
- **tsys-ucs-02 (902):** host **tsys9 → tsys6** (disk stays S2) — frees 8 GB on tsys9
- Then grow **wnode-tsys9 (905):** 8 GB → **16 GB** (tsys9 alloc then ~108% but actual still <80%)
### Phase 5 — bring the workers into the cluster (after placement settles)
- **Join the wnodes** (wnode-tsys1/3/5/6/7/9) to the k3s cluster via `k8s/join-servers.sh`
- Label them by role: `workload=compute` (tsys3/9/2), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
- Join wnodes via `k8s/join-servers.sh`
- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
### Verification after each phase
- `qm list` on each host — confirm VM count + status
- `pveperf` / `cat /proc/meminfo` — confirm actual RAM ≤ 80%
- `qm list` per host — confirm VM count + status
- `cat /proc/meminfo` — confirm actual RAM ≤ 80%
- `kubectl top nodes` — confirm cnodes healthy
- After Phase 1/2: `ssh root@pfv-tsys4 'df -h /mnt/*'` and `ssh root@pfv-tsys5 'df -h /mnt/*'` — confirm disks rebalanced (D2 frees up, S1 fills)
- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up
- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills