# Fleet Audit — 2026-07-29 (consolidated, storage-corrected) > **The single authoritative audit.** Supersedes all prior audits and the > placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict. > Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md). **Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware dumps + ~05:40 CDT capacity probe). Storage configs read directly from `/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host. **Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 **Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (offline) --- ## 1. Executive summary | Goal | Status | Detail | |------|--------|--------| | **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. | | **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. | | Finding | Severity | § | |---------|----------|---| | **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 | | **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 | | ~~tsys6 S1 NFS export typo~~ (`S21`) — **FIXED by operator 2026-07-29**; all 8 stores active | Resolved | 6 | | ~~tsys9 missing S1~~ in storage.cfg — **FIXED by operator 2026-07-29**; S1 active | Resolved | 6 | | **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 | | **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 | | **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 | --- ## 2. Host capacity (live, 2026-07-29 ~05:40 CDT) | Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load | |------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------| | tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 | | tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 | | tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 | | tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 | | tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 | | tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 | | tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 | **Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6, cnode5 (706) deleted from tsys7. No host migrations performed yet. --- ## 3. Storage topology (CORRECTED — read from live configs) > **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its > own storage entries. The same physical disk can appear under different storage > IDs on different hosts. This was the source of the prior audit's errors. ### 3.1 Physical disk → Proxmox storage ID mapping | Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type | |---------------|--------|------------|-----------------|---------------------|------| | WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD | | SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) | | Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD | | *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown | | *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown | | **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** | | Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD | | Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD | | Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD | | **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** | ### 3.2 Critical correction: `local-nonprod` = S1 On tsys5, the storage entry is: ``` dir: local-nonprod path /mnt/pfv-tsys5/S1 ``` **`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**, accessed locally as a directory storage instead of via NFS. The 16 sectestbed and preprod VMs "on local-nonprod" are physically on S1. They appear locked to tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage than `S1` (NFS type), even though they point to the same directory. **Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan to move production VMs onto S1 would have stacked them on top of 16 sandboxes. ### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday) | Tier | Spindles | Speed | Current use | |------|----------|-------|-------------| | NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs | | NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs | | Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) | | SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday | | SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only | --- ## 4. Per-spindle IOPS load (CORRECTED) | Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** | |---------|--------|----------------|----------------|------------------------| | **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** | | D5 (2TB) | tsys4 | 3 | 0 | 3 | | **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** | | S2 (916GB) | tsys5 | 4 | 0 | 4 | | S3 (916GB) | tsys5 | 3 | 0 | 3 | | S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity | **D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.** The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to S1.** Target S2, S3, S4, and D5 for IOPS spreading. --- ## 5. Storage bugs and config gaps found ### 5.1 BUG: tsys6 S1 export path typo (S21) — RESOLVED ✓ **Fixed by operator 2026-07-29.** The export path was corrected from `S21` to `S1`. Verified: `pvesm status` on tsys6 shows all 8 NFS stores active, including S1 (96 GB, 10.3% used by the sectestbed VMs visible via NFS). ### 5.2 GAP: tsys9 missing S1 in storage.cfg — RESOLVED ✓ **Fixed by operator 2026-07-29.** S1 entry added to tsys9's storage.cfg. Verified: all 8 NFS stores active on tsys9, S1 listing identical to tsys6. ### 5.3 ORPHANED: D1 and D7 exports on tsys4 tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg. - **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped templates only, no running VMs. - **D7** holds a disk for DellOpenManageEnterprise (500) — stopped. These aren't causing problems but are underdocumented. Not actionable now. --- ## 6. Current VM inventory (host + disk, post-user-changes) > cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from > start of session. `Disk` = NFS export or local storage holding the primary > qcow2. ### tsys1 — Infrastructure (11 running) | VMID | Name | vCPU | RAM | Disk | Server | |------|------|------|-----|------|--------| | 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 | | 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | | 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | | 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | | 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 | | 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 | | 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | | 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | | 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | | 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 | | 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 | ### tsys3 — Compute (1 running) | VMID | Name | vCPU | RAM | Disk | Server | |------|------|------|-----|------|--------| | 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | ### tsys4 — Storage (1 running) | VMID | Name | vCPU | RAM | Disk | Server | |------|------|------|-----|------|--------| | 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL | ### tsys5 — Storage + sandboxes (17 running) | VMID | Name | vCPU | RAM | Disk | Server | Migratable? | |------|------|------|-----|------|--------|-------------| | 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) | | 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) | | 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) | | 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) | | 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No | | 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No | | 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No | | ×10 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No | > *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk > but accessed via dir storage, not NFS — Proxmox will not migrate them. To > unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the > storage ID to the NFS-shared one). Low priority — these are idle sandboxes.* ### tsys6 — RackRental-ready (2 running, 1 stopped) | VMID | Name | vCPU | RAM | Disk | Server | Status | |------|------|------|-----|------|--------|--------| | 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running | | 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running | | 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** | ### tsys7 — RackRental-ready (5 running) | VMID | Name | vCPU | RAM | Disk | Server | |------|------|------|-----|------|--------| | 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | | 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 | | 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 | | 704 | TCTC | 4 | 6 GB | D2 | tsys4 | | 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 | ### tsys9 — Infra + control plane (6 running) | VMID | Name | vCPU | RAM | Disk | Server | |------|------|------|-----|------|--------| | 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 | | 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | | 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | | 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | | 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 | | 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | ### k3s control plane (ground truth via `kubectl`) 3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1). All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.** --- ## 7. Workload placement model | Workload family | Needs | Best hosts | Why | |-----------------|-------|-----------|-----| | **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe | | **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs | | **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity | --- ## 8. Answers to operator's questions ### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?" **tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB. After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**. **tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode. ### 8.2 "Are tsys6/7 better for RackRental.net?" **Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s) — deferred to Friday + StorageClass work. ### 8.3 "How should I deploy the DDR3?" Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST). tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB). Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5. --- ## 9. Hardware corrections **tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch — useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity and activate the CPU2 DIMM branch. **tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but monitor after the planned RAM upgrade. --- ## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced) > All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve > tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4 > (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA > pairs split across storage servers. ### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%) | VMID | Name | vCPU | RAM | Disk | Server | Change | |------|------|------|-----|------|--------|--------| | 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none | | 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none | | 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none | | 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none | | 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none | | 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none | ### tsys3 — Compute (1 running) | VMID | Name | vCPU | RAM | Disk | Server | Change | |------|------|------|-----|------|--------|--------| | 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none | ### tsys4 — Storage (1 running) | VMID | Name | vCPU | RAM | Disk | Change | |------|------|------|-----|------|--------| | 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none | ### tsys5 — Storage + sandboxes (17 running, unchanged) All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay locked (idle sandboxes, low priority). ### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%) | VMID | Name | vCPU | RAM | Disk | Server | Change | |------|------|------|-----|------|--------|--------| | 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none | | 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** | | 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** | | ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** | ### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%) | VMID | Name | vCPU | RAM | Disk | Server | Change | |------|------|------|-----|------|--------|--------| | 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none | | 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** | | 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** | | 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** | | 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** | | 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | | 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** | | 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) | | 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** | | 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** | ### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%) | VMID | Name | vCPU | RAM | Disk | Server | Change | |------|------|------|-----|------|--------|--------| | 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none | | 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none | | 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none | | 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** | | 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none | | ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** | ### End-state per-spindle IOPS load | Spindle | Server | # VMs | VMs on it | |---------|--------|-------|-----------| | **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 | | D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd | | S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) | | **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* | | **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench | | **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... | > *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16 > sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7 > production VMs each is fine (they're 916 GB disks with light per-VM I/O).* **D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.** HA pairs stay split: - netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓ - ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓ --- ## 11. Action checklist (do in this order) ### Phase 0 — fix storage config bugs first — DONE ✓ - ~~Fix tsys6 S1 typo~~ — **operator fixed 2026-07-29** (verified: 8/8 stores active) - ~~Add S1 to tsys9 storage.cfg~~ — **operator fixed 2026-07-29** (verified) ### Phase 1 — cleanup - **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster - **Confirm cnode5 (706) gone** — already deleted by operator ✓ ### Phase 2 — disk migrations off D2 (IOPS rebalancing) > Do disk moves on VMs that are staying on their current host first. - **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5) - **tsys-awx (600):** disk **D2 → D5** (host stays tsys6) - **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7) - **rr-middleware (703):** disk **D2 → S4** (host stays tsys7) - **TCTC (704):** disk **D2 → S3** (host stays tsys7) - **cnode2 (705):** disk **D2 → S2** (host stays tsys7) ### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs) - **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3** - **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4** - **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**) - **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2** - **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3** - **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2** ### Phase 4 — grow the compute wnode - **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required) ### Phase 5 — join workers to k3s - Join wnodes via `k8s/join-servers.sh` - Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5) ### Verification after each phase - `qm list` per host — confirm VM count + status - `cat /proc/meminfo` — confirm actual RAM ≤ 80% - `kubectl top nodes` — confirm cnodes healthy - `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up - `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills