mrcharles
e7312ad460
pfv-tsys8 is permanently retired. Strike its only doc reference and
correct the stale comment in perf/deploy-check.sh that wrongly listed
pfv-tsys9 as "off the air" (tsys9 is online and in the host array;
tsys8 is the retired host).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
453 lines
21 KiB
Markdown
453 lines
21 KiB
Markdown
# Fleet Audit — 2026-07-29 (consolidated, storage-corrected)
|
||
|
||
> **The single authoritative audit.** Supersedes all prior audits and the
|
||
> placement data in [`PROJECT.md`](PROJECT.md) §4-§8 where they conflict.
|
||
> Capacity targets are codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
|
||
|
||
**Audit time:** 2026-07-29, fresh data from all 7 hosts (~05:09 CDT hardware
|
||
dumps + ~05:40 CDT capacity probe). Storage configs read directly from
|
||
`/etc/pve/storage.cfg`, `/etc/exports`, `/etc/fstab` on every host.
|
||
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
|
||
**Hosts offline:** tsys2 (Win10, pending rebuild), tsys8 (retired / permanently offline)
|
||
|
||
---
|
||
|
||
## 1. Executive summary
|
||
|
||
| Goal | Status | Detail |
|
||
|------|--------|--------|
|
||
| **Memory ≤ 80% actual steady-state** | **PASS (all 7 hosts)** | Highest is tsys1 at 73%. Fleet is healthy. |
|
||
| **CPU ~50% at idle** | **MISS (all 7 hosts)** | Every host idles at **0-5% busy**. Under-utilized, not over-loaded. |
|
||
|
||
| Finding | Severity | § |
|
||
|---------|----------|---|
|
||
| **D2 carries 17 VMs** — the worst IOPS hotspot in the fleet | High | 5 |
|
||
| **S1 is NOT empty — it carries 16 sectestbed VMs via `local-nonprod`** (previous audit was wrong) | High | 4, 5 |
|
||
| ~~tsys6 S1 NFS export typo~~ (`S21`) — **FIXED by operator 2026-07-29**; all 8 stores active | Resolved | 6 |
|
||
| ~~tsys9 missing S1~~ in storage.cfg — **FIXED by operator 2026-07-29**; S1 active | Resolved | 6 |
|
||
| **`local-nonprod` on tsys5 = the S1 physical disk** (dir storage, not a separate disk) | Topology | 4 |
|
||
| **tsys5 is single-CPU** (CPU2 socket unpopulated; prior "dual E5620" was wrong) | Medium | 9 |
|
||
| **cnode4 (601) stopped, cnode5 (706) deleted** (user action this session) | Resolved | 10 |
|
||
|
||
---
|
||
|
||
## 2. Host capacity (live, 2026-07-29 ~05:40 CDT)
|
||
|
||
| Host | CPU (era) | Threads | RAM | Running VMs | Alloc RAM | Alloc% | **Actual%** | Swap used | CPU busy | Load |
|
||
|------|-----------|---------|-----|-------------|-----------|--------|------------|-----------|----------|------|
|
||
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 11 | 36 GB | 115% | **73%** | 0 | 4% | 1.1 |
|
||
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 28 GB | 90% | **10%** | 0 | 0% | 0.3 |
|
||
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 2 GB | 13% | **14%** | 1.1 GB | 0% | 1.3 |
|
||
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 17 | 126 GB | 134% | **37%** | 0 | 5% | 4.0 |
|
||
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 44 GB | 35% | **4%** | 0 | 0% | 0.6 |
|
||
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 52 GB | 28% | **7%** | 0 | 1% | 0.9 |
|
||
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | 30 GB | 129% | **62%** | 0 | 1% | 0.3 |
|
||
|
||
**Changes since first collection (~05:09):** cnode4 (601) stopped on tsys6,
|
||
cnode5 (706) deleted from tsys7. No host migrations performed yet.
|
||
|
||
---
|
||
|
||
## 3. Storage topology (CORRECTED — read from live configs)
|
||
|
||
> **`storage.cfg` is per-host, not cluster-wide.** Each Proxmox node defines its
|
||
> own storage entries. The same physical disk can appear under different storage
|
||
> IDs on different hosts. This was the source of the prior audit's errors.
|
||
|
||
### 3.1 Physical disk → Proxmox storage ID mapping
|
||
|
||
| Physical disk | Server | NFS export | NFS ID (clients) | Local ID (on tsys5) | Type |
|
||
|---------------|--------|------------|-----------------|---------------------|------|
|
||
| WDC Red 3TB | tsys4 | `/mnt/tsys4/D2` | `D2` | — | HDD |
|
||
| SK hynix SC300 512GB | tsys4 | `/mnt/tsys4/D3` | `D3` | — | SSD (USB) |
|
||
| Hitachi 2TB | tsys4 | `/mnt/tsys4/D5` | `D5` | — | HDD |
|
||
| *(disk1)* | tsys4 | `/mnt/tsys4/D1` | *(not in most storage.cfg)* | — | unknown |
|
||
| *(disk7)* | tsys4 | `/mnt/tsys4/D7` | *(not in most storage.cfg)* | — | unknown |
|
||
| **Seagate 916GB** | **tsys5** | **`/mnt/pfv-tsys5/S1`** | **`S1`** | **`local-nonprod`** | **HDD** |
|
||
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S2` | `S2` | `S2` (NFS client) | HDD |
|
||
| Seagate 916GB | tsys5 | `/mnt/pfv-tsys5/S3` | `S3` | `S3` (NFS client) | HDD |
|
||
| Toshiba 458GB | tsys5 | `/mnt/pfv-tsys5/S4` | `S4` | `S4` (NFS client) | HDD |
|
||
| **Samsung 860 PRO 234GB** | **tsys5** | **`/mnt/pfv-tsys5/ssd`** | **`T5-SSD`** | **`SSD`** (dir, local-only) | **SSD** |
|
||
|
||
### 3.2 Critical correction: `local-nonprod` = S1
|
||
|
||
On tsys5, the storage entry is:
|
||
```
|
||
dir: local-nonprod
|
||
path /mnt/pfv-tsys5/S1
|
||
```
|
||
|
||
**`local-nonprod` is NOT a separate disk.** It is the **S1 Seagate 916GB HDD**,
|
||
accessed locally as a directory storage instead of via NFS. The 16 sectestbed
|
||
and preprod VMs "on local-nonprod" are physically on S1. They appear locked to
|
||
tsys5 because Proxmox treats `local-nonprod` (dir type) as a different storage
|
||
than `S1` (NFS type), even though they point to the same directory.
|
||
|
||
**Implication:** S1 has **16 VMs of IOPS load**, not 0. The prior audit's plan
|
||
to move production VMs onto S1 would have stacked them on top of 16 sandboxes.
|
||
|
||
### 3.3 Disk speed tiers (HDD only — SSD/NVMe deferred to Friday)
|
||
|
||
| Tier | Spindles | Speed | Current use |
|
||
|------|----------|-------|-------------|
|
||
| NFS-HDD (tsys4) | D2, D5 | 80-120 MB/s | Production VMs |
|
||
| NFS-HDD (tsys5) | S1, S2, S3, S4 | 80-120 MB/s | Production + sectestbed VMs |
|
||
| Local HDD (tsys5) | local-nonprod (=S1) | 80-120 MB/s | Sectestbed suite (locked to tsys5) |
|
||
| SSD (tsys4 USB) | D3 | unreliable (USB 2.0) | Empty — do not use until Friday |
|
||
| SSD (tsys5 local) | SSD dir (=Samsung 860 PRO) | 200-400 MB/s | ultix-streaming only |
|
||
|
||
---
|
||
|
||
## 4. Per-spindle IOPS load (CORRECTED)
|
||
|
||
| Spindle | Server | NFS-visible VMs | Local-only VMs | **Total physical load** |
|
||
|---------|--------|----------------|----------------|------------------------|
|
||
| **D2** (3TB) | tsys4 | **17** | 0 | **17 — worst hotspot** |
|
||
| D5 (2TB) | tsys4 | 3 | 0 | 3 |
|
||
| **S1** (916GB) | tsys5 | 0 | **16** (via local-nonprod) | **16 — second worst** |
|
||
| S2 (916GB) | tsys5 | 4 | 0 | 4 |
|
||
| S3 (916GB) | tsys5 | 3 | 0 | 3 |
|
||
| S4 (458GB) | tsys5 | 1 | 0 | 1 — most spare capacity |
|
||
|
||
**D2 and S1 are both overloaded. S2/S3/S4 and D5 have room.**
|
||
|
||
The 16 sectestbed VMs on S1 are idle sandboxes — they don't generate much real
|
||
IOPS, but they're all on one spindle. If any of them wake up simultaneously (CI
|
||
pipeline, test suite), S1 becomes a bottleneck. **Do not add production VMs to
|
||
S1.** Target S2, S3, S4, and D5 for IOPS spreading.
|
||
|
||
---
|
||
|
||
## 5. Storage bugs and config gaps found
|
||
|
||
### 5.1 BUG: tsys6 S1 export path typo (S21) — RESOLVED ✓
|
||
|
||
**Fixed by operator 2026-07-29.** The export path was corrected from `S21` to
|
||
`S1`. Verified: `pvesm status` on tsys6 shows all 8 NFS stores active, including
|
||
S1 (96 GB, 10.3% used by the sectestbed VMs visible via NFS).
|
||
|
||
### 5.2 GAP: tsys9 missing S1 in storage.cfg — RESOLVED ✓
|
||
|
||
**Fixed by operator 2026-07-29.** S1 entry added to tsys9's storage.cfg.
|
||
Verified: all 8 NFS stores active on tsys9, S1 listing identical to tsys6.
|
||
|
||
### 5.3 ORPHANED: D1 and D7 exports on tsys4
|
||
|
||
tsys4 exports D1 and D7 via NFS, but neither appears in most hosts' storage.cfg.
|
||
- **D1** holds KNELTemplate (4000) and RestoreTemplate (5500) — stopped
|
||
templates only, no running VMs.
|
||
- **D7** holds a disk for DellOpenManageEnterprise (500) — stopped.
|
||
|
||
These aren't causing problems but are underdocumented. Not actionable now.
|
||
|
||
---
|
||
|
||
## 6. Current VM inventory (host + disk, post-user-changes)
|
||
|
||
> cnode4 (601) = stopped. cnode5 (706) = deleted. All other VMs unchanged from
|
||
> start of session. `Disk` = NFS export or local storage holding the primary
|
||
> qcow2.
|
||
|
||
### tsys1 — Infrastructure (11 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server |
|
||
|------|------|------|-----|------|--------|
|
||
| 100 | pfv-bms (HomeAssistant) | 2 | 4 GB | D2 | tsys4 |
|
||
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 |
|
||
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 |
|
||
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 |
|
||
| 104 | tsys-librenms | 2 | 2 GB | D2 | tsys4 |
|
||
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | D2 | tsys4 |
|
||
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 |
|
||
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 |
|
||
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 |
|
||
| 114 | kali-tsys | 2 | 2 GB | D2 | tsys4 |
|
||
| 117 | tsys-secure-workbench | 2 | 4 GB | D2 | tsys4 |
|
||
|
||
### tsys3 — Compute (1 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server |
|
||
|------|------|------|-----|------|--------|
|
||
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 |
|
||
|
||
### tsys4 — Storage (1 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server |
|
||
|------|------|------|-----|------|--------|
|
||
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | LOCAL |
|
||
|
||
### tsys5 — Storage + sandboxes (17 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Migratable? |
|
||
|------|------|------|-----|------|--------|-------------|
|
||
| 509 | pfv-k8s-wnode-tsys5 | 8 | 32 GB | D2 | tsys4 | Yes (NFS) |
|
||
| 5111 | ultix-streaming | 4 | 9 GB | SSD (local dir) | LOCAL | No (local) |
|
||
| 5112 | ultix-offstage | 4 | 6 GB | local-lvm | LOCAL | No (local) |
|
||
| 5101 | sectestbed-siem | 4 | 10 GB | local-nonprod (=S1) | LOCAL* | No (dir storage) |
|
||
| 53100 | tsys-preprod-awx | 4 | 9 GB | local-nonprod (=S1) | LOCAL* | No |
|
||
| 53101 | tsys-preprod-siem | 4 | 12 GB | local-nonprod (=S1) | LOCAL* | No |
|
||
| 53102 | tsys-preprod-rancherplatform | 4 | 8 GB | local-nonprod (=S1) | LOCAL* | No |
|
||
| ×10 | sectestbed suite | 4 ea | 4 GB ea | local-nonprod (=S1) | LOCAL* | No |
|
||
|
||
> *\*LOCAL = locked to tsys5. The local-nonprod VMs are on the S1 physical disk
|
||
> but accessed via dir storage, not NFS — Proxmox will not migrate them. To
|
||
> unlock, move disk from `local-nonprod` → `S1` (same physical disk, changes the
|
||
> storage ID to the NFS-shared one). Low priority — these are idle sandboxes.*
|
||
|
||
### tsys6 — RackRental-ready (2 running, 1 stopped)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Status |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | running |
|
||
| 600 | tsys-awx | 4 | 12 GB | D2 | tsys4 | running |
|
||
| 601 | pfv-k8s-cnode4 | 4 | 4 GB | D2 | tsys4 | **STOPPED** |
|
||
|
||
### tsys7 — RackRental-ready (5 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server |
|
||
|------|------|------|-----|------|--------|
|
||
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 |
|
||
| 702 | hfnoc-uisp | 4 | 8 GB | D2 | tsys4 |
|
||
| 703 | rr-middleware | 2 | 2 GB | D2 | tsys4 |
|
||
| 704 | TCTC | 4 | 6 GB | D2 | tsys4 |
|
||
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | D2 | tsys4 |
|
||
|
||
### tsys9 — Infra + control plane (6 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server |
|
||
|------|------|------|-----|------|--------|
|
||
| 901 | tsys-siem | 2 | 8 GB | D2 | tsys4 |
|
||
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 |
|
||
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 |
|
||
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 |
|
||
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4 GB | S2 | tsys5 |
|
||
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 |
|
||
|
||
### k3s control plane (ground truth via `kubectl`)
|
||
|
||
3 cnodes joined: cnode1 (906/tsys9), cnode2 (705/tsys7), cnode3 (106/tsys1).
|
||
All control-plane,etcd role. CPU 2%, mem ~1.1 GB each. **No wnodes joined yet.**
|
||
|
||
---
|
||
|
||
## 7. Workload placement model
|
||
|
||
| Workload family | Needs | Best hosts | Why |
|
||
|-----------------|-------|-----------|-----|
|
||
| **ETL / HPC / cross-compile** | High IPC + fast storage | **tsys9, tsys3, tsys2(in)** | Newest CPUs + local SSD/NVMe |
|
||
| **RackRental.net** | Massive RAM, CPU-light | **tsys7, tsys6** | 189/126 GB RAM; weak CPU fine for idle labs |
|
||
| **Infra + k8s control plane** | Always-on, low resource | **tsys1, tsys9** | Efficient enough; cnodes spread for etcd diversity |
|
||
|
||
---
|
||
|
||
## 8. Answers to operator's questions
|
||
|
||
### 8.1 "Should I make the k8s wnode VMs bigger on tsys1/9 for ETL/HPC?"
|
||
|
||
**tsys9 — YES.** Newest CPU (i5-10500, 12 threads) + local SSD. Currently 4c/4GB.
|
||
After moving tsys-siem (8 GB) to tsys6, grow to **6c/8-16 GB**.
|
||
|
||
**tsys1 — NO.** The i7-4770 is Haswell-2013 (2nd-oldest). tsys1 is the most
|
||
RAM-loaded host (73% actual, 115% alloc). It should shed VMs, not grow its wnode.
|
||
|
||
### 8.2 "Are tsys6/7 better for RackRental.net?"
|
||
|
||
**Yes.** RAM-bound, CPU-light labs fit 126/189 GB perfectly. They don't need
|
||
more RAM (4%/7% actual). Their bottleneck is disk (USB 2.0 local-lvm ≈ 30 MB/s)
|
||
— deferred to Friday + StorageClass work.
|
||
|
||
### 8.3 "How should I deploy the DDR3?"
|
||
|
||
Verify type first. tsys5/6/7 require **RDIMM ECC** (consumer UDIMM won't POST).
|
||
tsys1 is maxed (32 GB, caps at 8 GB/slot). tsys4 caps at 8 GB/slot (max 32 GB).
|
||
Most likely only useful for tsys4 16→32 GB. The real RAM lever is **free VM
|
||
migration**, not sticks. See [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §5.
|
||
|
||
---
|
||
|
||
## 9. Hardware corrections
|
||
|
||
**tsys5 is single-CPU.** dmidecode: `CPU2: Status Unpopulated`. The T7500 has
|
||
one E5620 (4c/8t), not two. The 6 empty DIMM slots are on the CPU2 branch —
|
||
useless without a 2nd CPU. Seating a spare LGA771 Xeon would double CPU capacity
|
||
and activate the CPU2 DIMM branch.
|
||
|
||
**tsys4 is using ~1.1 GB of swap.** Only host with active swap. Minor, but
|
||
monitor after the planned RAM upgrade.
|
||
|
||
---
|
||
|
||
## 10. Target end-state inventory (RAM/CPU + HDD IOPS balanced)
|
||
|
||
> All disks stay on **HDD** (SSD/NVMe deferred to Friday). Goals: (a) relieve
|
||
> tsys1/9 RAM over-allocation, (b) relieve D2 (17→5 VMs), (c) spread to S2/S3/S4
|
||
> (currently 4/3/1), (d) **avoid S1** (already has 16 local VMs), (e) keep HA
|
||
> pairs split across storage servers.
|
||
|
||
### tsys1 — Infrastructure (6 running, ~24 GB alloc → 77%)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 100 | pfv-bms | 2 | 4 GB | D2 | tsys4 | none |
|
||
| 101 | tsys-ca | 2 | 2 GB | D2 | tsys4 | none |
|
||
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4 GB | S3 | tsys5 | none |
|
||
| 103 | **pfv-netinfra-01** | 2 | 2 GB | S2 | tsys5 | none |
|
||
| 106 | pfv-k8s-cnode3 | 2 | 4 GB | D5 | tsys4 | none |
|
||
| 108 | **tsys-ucs-01** | 4 | 8 GB | D2 | tsys4 | none |
|
||
|
||
### tsys3 — Compute (1 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28 GB | D5 | tsys4 | none |
|
||
|
||
### tsys4 — Storage (1 running)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Change |
|
||
|------|------|------|-----|------|--------|
|
||
| 400 | pfv-proxmox-backup-server | 2 | 2 GB | local-lvm | none |
|
||
|
||
### tsys5 — Storage + sandboxes (17 running, unchanged)
|
||
|
||
All 17 stay. wnode-tsys5 (509) disk moves D2→S4. The 16 local-nonprod VMs stay
|
||
locked (idle sandboxes, low priority).
|
||
|
||
### tsys6 — RackRental-ready (3 running, ~52 GB alloc → 41%)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 100 | pfv-k8s-wnode-tsys6 | 4 | 32 GB | S4 | tsys5 | none |
|
||
| 600 | tsys-awx | 4 | 12 GB | **D5** | tsys4 | **disk D2→D5** |
|
||
| 901 | tsys-siem | 2 | 8 GB | **S2** | tsys5 | **host tsys9→tsys6, disk D2→S2** |
|
||
| ~~601~~ | ~~cnode4~~ | — | — | — | — | **DELETE (stopped, stale)** |
|
||
|
||
### tsys7 — RackRental-ready (10 running, ~64 GB alloc → 34%)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32 GB | S3 | tsys5 | none |
|
||
| 702 | hfnoc-uisp | 4 | 8 GB | **S4** | tsys5 | **disk D2→S4** |
|
||
| 703 | rr-middleware | 2 | 2 GB | **S4** | tsys5 | **disk D2→S4** |
|
||
| 704 | TCTC | 4 | 6 GB | **S3** | tsys5 | **disk D2→S3** |
|
||
| 705 | pfv-k8s-cnode2 | 4 | 4 GB | **S2** | tsys5 | **disk D2→S2** |
|
||
| 104 | tsys-librenms | 2 | 2 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
|
||
| 105 | tsys-proxmox-datacenter | 2 | 2 GB | **S4** | tsys5 | **host tsys1→tsys7, disk D2→S4** |
|
||
| 109 | tailscale-router | 2 | 2 GB | D2 | tsys4 | **host tsys1→tsys7** (disk stays D2) |
|
||
| 114 | kali-tsys | 2 | 2 GB | **S2** | tsys5 | **host tsys1→tsys7, disk D2→S2** |
|
||
| 117 | tsys-secure-workbench | 2 | 4 GB | **S3** | tsys5 | **host tsys1→tsys7, disk D2→S3** |
|
||
|
||
### tsys9 — Infra + control + compute (5 running, ~22 GB alloc → 94%)
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Server | Change |
|
||
|------|------|------|-----|------|--------|--------|
|
||
| 902 | **tsys-ucs-02** | 4 | 8 GB | S2 | tsys5 | none |
|
||
| 903 | kali-rd | 2 | 2 GB | D5 | tsys4 | none |
|
||
| 904 | **pfv-netinfra-02** | 2 | 4 GB | D2 | tsys4 | none |
|
||
| 905 | pfv-k8s-wnode-tsys9 | **6** | **8 GB** | S2 | tsys5 | **grow 4c/4G→6c/8G** |
|
||
| 906 | pfv-k8s-cnode1 | 2 | 4 GB | S3 | tsys5 | none |
|
||
| ~~901~~ | ~~tsys-siem~~ | — | — | — | — | **move→tsys6** |
|
||
|
||
### End-state per-spindle IOPS load
|
||
|
||
| Spindle | Server | # VMs | VMs on it |
|
||
|---------|--------|-------|-----------|
|
||
| **D2** (3TB) | tsys4 | **5** | bms, ca, ucs01, tailscale, netinfra02 |
|
||
| D5 (2TB) | tsys4 | 4 | cnode3, wnode3, awx, kali-rd |
|
||
| S1 (916GB) | tsys5 | **16** (unchanged) | sectestbed suite (local-nonprod, idle) |
|
||
| **S2** (916GB) | tsys5 | **7** | netinfra01, ucs02, wnode9, cnode2, siem, kali, pdm* |
|
||
| **S3** (916GB) | tsys5 | **7** | wnode1, wnode7, cnode1, uisp, TCTC, librenms, workbench |
|
||
| **S4** (458GB) | tsys5 | **5** | wnode5, wnode6, rr-middleware, uisp*... |
|
||
|
||
> *S2/S3 carry 7 each because the sectestbed load on S1 is idle. If those 16
|
||
> sectestbed VMs ever become active, rebalance — but for now S2/S3 at 7
|
||
> production VMs each is fine (they're 916 GB disks with light per-VM I/O).*
|
||
|
||
**D2: 17→5. No VMs added to S1. S2/S3/S4 absorb the load.**
|
||
|
||
HA pairs stay split:
|
||
- netinfra: 01 on S2 (tsys5), 02 on D2 (tsys4) ✓
|
||
- ucs: 01 on D2 (tsys4), 02 on S2 (tsys5) ✓
|
||
|
||
---
|
||
|
||
## 11. Action checklist (do in this order)
|
||
|
||
### Phase 0 — fix storage config bugs first — DONE ✓
|
||
|
||
- ~~Fix tsys6 S1 typo~~ — **operator fixed 2026-07-29** (verified: 8/8 stores active)
|
||
- ~~Add S1 to tsys9 storage.cfg~~ — **operator fixed 2026-07-29** (verified)
|
||
|
||
### Phase 1 — cleanup
|
||
|
||
- **Delete cnode4 (VMID 601) on tsys6** — stopped, stale, not in cluster
|
||
- **Confirm cnode5 (706) gone** — already deleted by operator ✓
|
||
|
||
### Phase 2 — disk migrations off D2 (IOPS rebalancing)
|
||
|
||
> Do disk moves on VMs that are staying on their current host first.
|
||
|
||
- **wnode-tsys5 (509):** disk **D2 → S4** (host stays tsys5)
|
||
- **tsys-awx (600):** disk **D2 → D5** (host stays tsys6)
|
||
- **hfnoc-uisp (702):** disk **D2 → S4** (host stays tsys7)
|
||
- **rr-middleware (703):** disk **D2 → S4** (host stays tsys7)
|
||
- **TCTC (704):** disk **D2 → S3** (host stays tsys7)
|
||
- **cnode2 (705):** disk **D2 → S2** (host stays tsys7)
|
||
|
||
### Phase 3 — host + disk migrations (RAM/CPU relief, 6 VMs)
|
||
|
||
- **tsys-librenms (104):** host **tsys1 → tsys7**, disk **D2 → S3**
|
||
- **tsys-proxmox-datacenter (105):** host **tsys1 → tsys7**, disk **D2 → S4**
|
||
- **tailscale-router (109):** host **tsys1 → tsys7** (disk stays **D2**)
|
||
- **kali-tsys (114):** host **tsys1 → tsys7**, disk **D2 → S2**
|
||
- **tsys-secure-workbench (117):** host **tsys1 → tsys7**, disk **D2 → S3**
|
||
- **tsys-siem (901):** host **tsys9 → tsys6**, disk **D2 → S2**
|
||
|
||
### Phase 4 — grow the compute wnode
|
||
|
||
- **wnode-tsys9 (905):** resize **4c/4GB → 6c/8GB** (stop/start required)
|
||
|
||
### Phase 5 — join workers to k3s
|
||
|
||
- Join wnodes via `k8s/join-servers.sh`
|
||
- Label: `workload=compute` (tsys3/9), `workload=rackrental` (tsys6/7), `workload=light` (tsys1/5)
|
||
|
||
### Verification after each phase
|
||
|
||
- `qm list` per host — confirm VM count + status
|
||
- `cat /proc/meminfo` — confirm actual RAM ≤ 80%
|
||
- `kubectl top nodes` — confirm cnodes healthy
|
||
- `ssh root@pfv-tsys4 'df -h /mnt/tsys4/D2'` — confirm D2 frees up
|
||
- `ssh root@pfv-tsys5 'df -h /mnt/pfv-tsys5/S2 /mnt/pfv-tsys5/S3 /mnt/pfv-tsys5/S4'` — confirm fills
|
||
|
||
---
|
||
|
||
## 12. Future: local storage as ephemeral scratch
|
||
|
||
> Codified as standing policy in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.
|
||
|
||
The fleet has substantial **unused local storage IOPS** on several hosts. OS
|
||
drives stay on NFS (no per-node disk redundancy — see policy), but local storage
|
||
is ideal for **ephemeral scratch** (container image cache, build artifacts,
|
||
`emptyDir` volumes) where data loss on node failure is acceptable.
|
||
|
||
**Per-host opportunity:**
|
||
|
||
| Host | Local | Available | Scratch-suitable? |
|
||
|------|-------|-----------|-------------------|
|
||
| tsys3 | NVMe (PM961) | 349 GB | **Yes — best existing** |
|
||
| tsys9 | SSD (PNY CS900) | 136 GB | **Yes** |
|
||
| tsys2 | NVMe + SSD | 512 GB + 1 TB | **Yes (incoming)** |
|
||
| tsys5 | HDD (Hitachi) | ~1.7 TB | Yes (bulk) |
|
||
| tsys6 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
|
||
| tsys7 | USB 2.0 HDD | 1.7 TB | **NO (30 MB/s)** |
|
||
|
||
**The tsys6/7 problem:** they are the RackRental hosts (most in need of fast
|
||
image pulls) but their local storage is USB 2.0 — slower than NFS. The real fix
|
||
is installing internal drives (tsys6 has a disabled SAS controller; tsys7 has 5
|
||
empty SATA ports). Deferred to a physical-hardware session.
|
||
|
||
**Implementation:** post-Friday, once SSD/NVMe tiers are installed and the k8s
|
||
StorageClass design is finalized. Wnodes with local storage get a `/scratch`
|
||
mount for ephemeral data; persistent data stays on NFS. See
|
||
[`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6.4 for the StorageClass mapping.
|