docs: full documentation suite — network, storage, hosts, VMs

Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:

- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
  maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
  analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
  inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
  per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
  network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
  disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
  qga, disk controller, k8s wnode sizing), stopped VM reference

Updated docmap.md with all four new entries.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-30 23:10:38 -05:00
parent cd1c2f7332
commit 0a8865de38
5 changed files with 963 additions and 0 deletions
+173
View File
@@ -0,0 +1,173 @@
# Host Reference
> **One-page-per-host quick reference for every Proxmox node.** Hardware,
> network, storage, role, and running VMs. Data gathered 2026-07-30.
>
> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
> [`VM-INVENTORY.md`](VM-INVENTORY.md).
---
## Fleet summary
```mermaid
graph TB
subgraph "RACK 5 — Core / Data / Infra"
T1["tsys1 — Infra<br/>i7-4770 · 32GB · 9 VMs<br/>57% RAM"]
T3["tsys3 — Compute<br/>Xeon E3-1535M v5 · 32GB · 1 VM<br/>10% RAM"]
T4["tsys4 — Storage+PBS<br/>Xeon E3-1246 v3 · 16GB · 1 VM<br/>15% RAM"]
T5["tsys5 — Storage+Sandbox<br/>Xeon E5620 · 96GB · 4 VMs<br/>15% RAM"]
T9["tsys9 — Infra+Compute<br/>i5-10500 · 24GB · 6 VMs<br/>69% RAM"]
end
subgraph "RACK 3 — Compute / RackRental"
T6["tsys6 — RackRental<br/>2× Xeon E5530 · 126GB · 2 VMs<br/>12% RAM"]
T7["tsys7 — RackRental<br/>2× Xeon E5-2630 v2 · 189GB · 5 VMs<br/>7% RAM"]
end
```
| Host | CPU (era) | Threads | RAM | PVE | Kernel | Role | Running VMs | RAM actual% |
|------|-----------|---------|-----|-----|--------|------|-------------|-------------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 32 GB | 9.2.5 | 7.0.14 | Infra | 9 | 57% |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% |
| tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% |
**Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired)
---
## pfv-tsys1 — Infrastructure host
| Attribute | Value |
|-----------|-------|
| **Role** | Infrastructure: Home Assistant, CA, netinfra-01, LibreNMS, PDM, k8s cnode3, UCS-01, kali, workbench |
| **Hardware** | Dell, i7-4770 Haswell (2013), 8 threads, 32 GB RAM (maxed) |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.11 (vmbr0, DHCP) |
| **Tailscale** | 100.121.189.98 |
| **Storage IP** | 10.100.100.1 (datanet via nic1, USB dongle, no bond) |
| **NFS mounts** | S1, S2, S3, S4, D2 (all nconnect=4) |
| **Local storage** | local-lvm (~90 GB, HDD, marginal) |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | None (single USB NIC for storage) |
| **Special services** | NUT (UPS), powerman (PDU) |
| **Running VMs** | 100, 101, 103, 104, 105, 106, 108, 114, 117 |
---
## pfv-tsys3 — Compute host
| Attribute | Value |
|-----------|-------|
| **Role** | Compute: k8s wnode-tsys3 (28 GB RAM, 8 vCPU) |
| **Hardware** | Xeon E3-1535M v5 Skylake (2015), 8 threads, 32 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) |
| **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) |
| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) — **best existing local storage** |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | None (single USB NIC for storage) |
| **Running VMs** | 313 |
---
## pfv-tsys4 — Storage server + PBS
| Attribute | Value |
|-----------|-------|
| **Role** | NFS storage server (D1/D2/D3/D5) + Proxmox Backup Server + console management (conman/ser2net) |
| **Hardware** | Xeon E3-1246 v3 Haswell (2013), 8 threads, 16 GB RAM |
| **PVE** | **9.1.5** / kernel **6.17.9-1-pve** (BEHIND fleet) |
| **mgmt IP** | 192.168.3.251 (vmbr0, DHCP) |
| **Tailscale** | 100.70.77.93 |
| **Storage IP** | 10.100.100.4 (datanet via enx8cae4ccda926 USB dongle, bond0 with 1 slave) |
| **NFS mounts** | None (NFS server only) |
| **Local storage** | local-lvm (WDC WD10EZRX 1TB, ~62 GB free) |
| **NFS exports** | D1 (WD 1TB USB), D2 (WDC Red 3TB), D3 (Seagate 500GB USB), D5 (Hitachi 2TB) |
| **Other disks** | /mnt/albert (Hitachi 2TB, not exported), /mnt/backup (WD 5TB USB) |
| **Tuning** | governor=performance, swappiness=1, bbr, **rmem/wmem=16MB** (should be 128MB), **tuned=throughput-performance** (should be network-throughput) |
| **Bond** | bond0 (1 USB slave), hash=layer2+3 |
| **Special services** | ser2net + conman (7 switch consoles on :7890) |
| **Running VMs** | 400 (PBS) |
| **Notes** | USB NIC bottleneck. Needs PVE upgrade + tuning alignment. |
---
## pfv-tsys5 — Storage server + sandboxes
| Attribute | Value |
|-----------|-------|
| **Role** | NFS storage server (S1/S2) + sectestbed sandboxes + ultix streaming |
| **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** — nic2 DOWN) |
| **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) |
| **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) |
| **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) |
| **Unconfigured disks** | **NVMe CT500P1 500GB** (not mounted), **SK hynix SC300B 512GB SSD** (not mounted) |
| **Tuning** | governor=N/A (no cpufreq driver on Westmere), swappiness=1, bbr, rmem/wmem=128MB, tuned=network-throughput |
| **Bond** | bond0 (1 of 2 slaves), hash=layer2+3 |
| **Running VMs** | 509, 515, 5111, 5112 (+ 22 stopped sandboxes) |
| **Notes** | Bond degraded (needs 2nd cable). NVMe + SSD need mounting/config. SSD 93% full. Single CPU. |
---
## pfv-tsys6 — RackRental host
| Attribute | Value |
|-----------|-------|
| **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX |
| **Hardware** | 2× Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) |
| **Tailscale** | 100.73.35.111 |
| **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) |
| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
| **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) |
| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 — unusable for scratch. |
---
## pfv-tsys7 — RackRental host
| Attribute | Value |
|-----------|-------|
| **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 |
| **Hardware** | 2× Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) |
| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
| **Running VMs** | 701, 702, 703, 704, 705 |
| **Notes** | Best storage bond in fleet (2G LACP). Most RAM in fleet (189 GB). Local storage USB 2.0. |
---
## pfv-tsys9 — Infra + Compute
| Attribute | Value |
|-----------|-------|
| **Role** | Infra + Compute: k8s cnode1 + wnode-tsys9 + siem + UCS-02 + netinfra-02 + kali |
| **Hardware** | i5-10500 Comet Lake (2020), 12 threads, 24 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) |
| **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) |
| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) — **excellent for scratch** |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | None (single USB NIC for storage) |
| **Running VMs** | 901, 902, 903, 904, 905, 906 |
| **Notes** | Newest CPU (best IPC in fleet). **69% RAM — approaching 80% goal.** Target for compute growth (after siem migrates off). |
+365
View File
@@ -0,0 +1,365 @@
# Network Topology
> **Complete physical + logical network reference for the PFVCluster.**
> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct
> host probes. All data is live ground truth — not documentation inherited.
>
> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
---
## 1. Physical topology diagram
```mermaid
graph TB
subgraph "RACK 5 — Core / Data / Infra"
CORE["pfv-core-sw01<br/>PowerConnect 5448<br/>192.168.0.12<br/>FW 2.0.0.46<br/>(mgmt + storage VLANs)"]
TSYS1["pfv-tsys1<br/>i7-4770 Haswell<br/>Infra host"]
TSYS3["pfv-tsys3<br/>Xeon E3-1535M v5<br/>Compute"]
TSYS4["pfv-tsys4<br/>Xeon E3-1246 v3<br/>NFS Storage + PBS"]
TSYS5["pfv-tsys5<br/>Xeon E5620 Westmere<br/>NFS Storage + Sandboxes"]
TSYS9["pfv-tsys9<br/>i5-10500 Comet Lake<br/>Infra + Compute"]
TSYS4_CONSOLE["conman + ser2net<br/>7 switch consoles<br/>(Tailscale 100.70.77.93:7890)"]
TSYS1_PDU["Cyclades PM10i PDU<br/>powerman on tsys1<br/>(port 10101)"]
TSYS1_UPS["APC Smart-UPS C 1500<br/>NUT on tsys1<br/>(port 3493)"]
end
subgraph "RACK 3 — Compute / RackRental"
TOR3STOR["pfv-tor3-stor<br/>PowerConnect 5324 (Neyland 24T)<br/>192.168.0.9<br/>FW 2.0.1.4<br/>(storage VLAN)"]
TOR3MGMT["pfv-tor3-mgmt<br/>PowerConnect 5324 (Neyland 24T)<br/>192.168.0.7<br/>(mgmt VLAN)"]
TSYS6["pfv-tsys6<br/>2× Xeon E5530 Nehalem<br/>RackRental"]
TSYS7["pfv-tsys7<br/>2× Xeon E5-2630 v2 Ivy Bridge<br/>RackRental"]
end
%% Cross-rack trunk
CORE <--. "ch1 LAG: 3×1G active<br/>(g13,g14,g15)<br/>hash=layer-2-3<br/>mode=on (static)" .--> TOR3STOR
TOR3STOR -. "ch1: 4×1G<br/>(g20,g21,g22,g23)<br/>hash=layer-2-3-4<br/>mode=on (static)" .-> CORE
%% Rack 5 connections to core
CORE --> TSYS1
CORE --> TSYS3
CORE --> TSYS4
CORE --> TSYS5
CORE --> TSYS9
%% Rack 3 connections
TOR3STOR --> TSYS6
TOR3STOR --> TSYS7
TOR3MGMT --> TSYS6
TOR3MGMT --> TSYS7
```
### How to read this diagram
The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk**
the most critical link in the cluster. It carries all NFS storage traffic
between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5).
The trunk is a 4-port static LAG, but only 3 links are active due to a cable
mismatch (see §3 below).
---
## 2. Subnet / VLAN map
| VLAN | Subnet | Purpose | Where it lives |
|------|--------|---------|----------------|
| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) |
| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) |
| — (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs |
**Management IPs (VLAN 1, via DHCP reservations):**
| Host | mgmt IP (vmbr0) | Tailscale IP |
|------|-----------------|--------------|
| tsys1 | 192.168.3.11 | 100.121.189.98 |
| tsys3 | 192.168.2.5 | (connected) |
| tsys4 | 192.168.3.251 | 100.70.77.93 |
| tsys5 | 192.168.3.179 | (connected) |
| tsys6 | 192.168.3.169 | 100.73.35.111 |
| tsys7 | 192.168.3.55 | (connected) |
| tsys9 | 192.168.3.58 | (connected) |
**Storage IPs (VLAN 1000, static):**
| Host | Storage IP | Interface |
|------|-----------|-----------|
| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) |
| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) |
| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) |
| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) |
| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) |
| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) |
| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) |
**Switch IPs (VLAN 1, static DHCP reservations):**
| Switch | IP | Model | Console |
|--------|----|-------|---------|
| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 |
| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 |
| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 |
| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? |
**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c)
---
## 3. Cross-rack trunk detail (ch1) — the critical link
This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and
rack 5 (tsys4/5). It is the single most important network path in the cluster.
```mermaid
graph LR
subgraph "core-sw01 (rack5)"
G13["g13<br/>1G UP"]
G14["g14<br/>1G UP"]
G15["g15<br/>1G UP"]
G16["g16<br/>1G UP<br/>NOT in LAG!"]
G17["g17<br/>DOWN<br/>in config but<br/>no cable"]
end
subgraph "tor3-stor (rack3)"
G20["g20<br/>1G UP"]
G21["g21<br/>1G UP"]
G22["g22<br/>1G UP"]
G23["g23<br/>1G UP"]
end
G13 --- G20
G14 --- G21
G15 --- G22
G16 -.- G23
G17 -.->|no cable| G23
```
### Configuration comparison
| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? |
|----------|-------------------|--------------------|--------|
| Physical ports configured | g(13-15,17) | g(20-23) | **No — asymmetric** |
| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | — |
| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No — 3 vs 4** |
| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No — asymmetric** |
| Channel mode | on (static) | on (static) | Yes (both should be LACP) |
| VLAN | access vlan 1000 | access vlan 1000 | Yes |
### Problems
1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on
core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex)
but is NOT in the LAG group. g17 is configured in the LAG but has no cable
(DOWN). Result: **only 3 of 4 links are active**.
2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP).
tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With
layer-2-3 on core-sw01, **all traffic between a given host pair hashes to
one link** regardless of how many TCP connections (nconnect) are used.
3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure
detection — a dead cable stays in the LAG until manually discovered.
---
## 4. Complete switch port map
### pfv-core-sw01 (PowerConnect 5448, 48 ports) — rack 5
| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) |
| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) |
| g5 | — | UP | 1 | pfv-tsys1 (mgmt) |
| g6 | 100 | UP | 1 | pfvsvrpi (secondary) |
| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt |
| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** |
| g10 | 1000 | UP | 1 | (unknown) |
| g11 | 1000 | UP | 1 | (unknown) |
| g12 | 1000 | UP | 1 | (unknown) |
| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** |
| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** |
| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** |
| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG — should be g17)** |
| **g17** | — | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** |
| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt |
| g21 | 1000 | UP | 1 | (unknown) |
| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) |
| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** |
| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) |
| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) |
| g32 | — | — | — | pfv-tsys4-storage (configured, DOWN) |
| g33 | — | — | — | pfv-tsys5-stor (ch2, DOWN/Not Present) |
| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) |
| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** |
| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) |
| g41 | 1000 | UP | 1 | (unknown) |
| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) |
| g44 | 1000 | UP | 1 | (unknown) |
| g45-g48 | — | DOWN | — | (Combo ports, unused) |
**Port-channels on core-sw01:**
| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor |
| ch2 | — | DOWN | g32-33 (non-candidate) | — | Unused |
| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) |
**LAG hash:** `port-channel load-balance layer-2-3`
### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3
| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | (unknown — uplink to tor3-mgmt or router) |
| g2-g7 | 1000 | UP | 1 | (unused but UP) |
| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** |
| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** |
| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** |
| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** |
| **g13** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** |
| **g14** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** |
| g15-g19 | 1000 | UP | 1000 | (unused storage ports) |
| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** |
| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** |
| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** |
| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** |
| g24 | — | DOWN | 1 | (spanning-tree disabled, portfast) |
**Port-channels on tor3-stor:**
| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4×1G) |
| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2×1G LACP) |
| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2×1G LACP) |
| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE — retired host)** |
**LAG hash:** `port-channel load-balance layer-2-3-4`
### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3
| Port | Speed | Status | Connected device |
|------|-------|--------|-----------------|
| g3 | — | UP | tsys7 (mgmt nic0) |
| g7 | — | UP | tsys6 (mgmt nic0) |
(Full port map not captured — this switch carries only VLAN 1 management
traffic. Low priority for storage performance.)
---
## 5. Per-host network interface reference
| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed |
|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------|
| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | — | 1G |
| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | — | 1G |
| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G |
| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** |
| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | — | 1G |
### Key observations
- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2
active slaves). All NFS traffic from rack 3 uses these bonds.
- **tsys1/3/4/9 use USB dongles** for storage network — single 1G, no redundancy.
- **tsys5 bond0 is degraded** — nic2 is DOWN (no cable connected). Only nic1 is
active via core-sw01 ch3 (g34). This is the LACP problem the operator has been
troubleshooting.
- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS),
tsys4/5 use layer2+3 (suboptimal — same src/dst IP pair always hashes to one
link even with nconnect's multiple TCP connections).
---
## 6. NFS data flow diagram
```mermaid
graph TB
subgraph "Storage Servers (rack 5)"
T4D2["tsys4 D2<br/>WDC Red 3TB<br/>Tier 2"]
T4D5["tsys4 D5<br/>Hitachi 2TB<br/>Tier 2"]
T4D1["tsys4 D1<br/>WD 1TB USB<br/>Tier 3"]
T4D3["tsys4 D3<br/>Seagate 500GB USB<br/>Tier 3"]
T5S1["tsys5 S1<br/>IronWolf 1TB<br/>Tier 2"]
T5S2["tsys5 S2<br/>IronWolf 1TB<br/>Tier 2"]
T5SSD["tsys5 SSD<br/>Samsung 860 PRO<br/>Tier 1 (93% full)"]
end
subgraph "VLAN 1000 storage network"
NFS["NFS v4.2<br/>nconnect=4<br/>rsize/wsize=1MB"]
end
subgraph "NFS Clients"
T1["tsys1 (9 mounts)"]
T3["tsys3 (5 mounts)"]
T5["tsys5 (3 mounts, NO nconnect)"]
T6["tsys6 (9 mounts)"]
T7["tsys7 (9 mounts)"]
T9["tsys9 (5 mounts)"]
end
T4D2 --> NFS
T4D5 --> NFS
T4D1 --> NFS
T4D3 --> NFS
T5S1 --> NFS
T5S2 --> NFS
T5SSD --> NFS
NFS --> T1
NFS --> T3
NFS --> T5
NFS --> T6
NFS --> T7
NFS --> T9
```
### NFS mount inventory (live)
| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|--------|--------|-------|-------|-------|-------|-------|-------|-------|
| tsys4 | D1 | — | — | — | — | ✓ | ✓ | — |
| tsys4 | **D2** | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| tsys4 | D3 | — | — | — | ✓ | ✓ | ✓ | — |
| tsys4 | D5 | — | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| tsys5 | S1 | ✓ | — | — | — | ✓ | ✓ | — |
| tsys5 | S2 | ✓ | ✓ | — | — | ✓ | ✓ | — |
| tsys5 | S3 | ✓ | — | — | — | ✓ | ✓ | ✓ |
| tsys5 | S4 | ✓ | ✓ | — | — | ✓ | ✓ | ✓ |
| tsys5 | T5-SSD | — | ✓ | — | — | ✓ | ✓ | ✓ |
(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts.
tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.)
---
## 7. Switch management access
| Method | Details |
|--------|---------|
| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. |
| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` |
| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) |
| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) |
| **Direct telnet** | `telnet pfv-tsys4 200X` — ONLY when conmand stopped (conflicts). Use conman instead. |
### Console port assignments
| Port | Console name | Switch | ID_PATH (USB port) |
|------|-------------|--------|---------------------|
| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 |
| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 |
| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 |
| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 |
| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 |
| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 |
| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 |
+196
View File
@@ -0,0 +1,196 @@
# Storage Architecture
> **Complete storage reference for the PFVCluster.** Covers physical disks,
> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle
> VM placement. Data gathered 2026-07-30 via live host probes.
>
> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
---
## 1. Storage tier definitions
| Tier | Definition | Typical speed | Reliability | Use case |
|------|-----------|---------------|-------------|----------|
| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch |
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage |
| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. |
**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk
storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS
(tsys4/tsys5) where redundancy is achieved by spreading across physical disks
and HA pairs. Local storage is for ephemeral data where loss on node failure
is acceptable.
---
## 2. Physical disk inventory
### tsys4 — NFS storage server + PBS (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|--------|------|-------|------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
| sdd | WDC WD10EZRX | 1.0 TB | — | SATA | (pve boot/swap/lvm) | — | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% |
| sdf | WDC WD10EARS | 1.0 TB | — | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% |
**Changes from previous audit:**
- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD.
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2.
- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only.
- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone.
### tsys5 — NFS storage server + sandboxes (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|-------------|------|-------|------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | — |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | — | **Tier 1** | PASSED | **NEW, not configured** |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | — | Tier 2 | PASSED | OS disk |
| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | — | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |
**Changes from previous audit:**
- **NVMe added:** Crucial CT500P1 500GB. True NVMe — highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry.
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) — needs attention.
- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).
---
## 3. NFS export → Proxmox storage ID map
```mermaid
graph LR
subgraph "tsys4 — NFS Server"
D2["D2: /mnt/tsys4/D2<br/>WDC Red 3TB<br/>Tier 2 · 7% used"]
D5["D5: /mnt/tsys4/D5<br/>Hitachi 2TB<br/>Tier 2 · 13% used"]
D1["D1: /mnt/tsys4/D1<br/>WD 1TB USB<br/>Tier 3 · 0% used"]
D3["D3: /mnt/tsys4/D3<br/>Seagate 500GB USB<br/>Tier 3 · 0% used"]
end
subgraph "tsys5 — NFS Server"
S1["S1: /mnt/pfv-tsys5/S1<br/>IronWolf 1TB<br/>Tier 2 · 11% used"]
S2["S2: /mnt/pfv-tsys5/S2<br/>IronWolf 1TB<br/>Tier 2 · 4% used"]
S3stale["S3: (stale export)<br/>disk not mounted"]
end
subgraph "All hosts — NFS Clients"
MNT["/mnt/pve/D1 D2 D3 D5<br/>/mnt/pve/S1 S2 S3 S4<br/>/mnt/pve/T5-SSD"]
end
D2 -->|NFS v4.2| MNT
D5 -->|NFS v4.2| MNT
D1 -->|NFS v4.2| MNT
D3 -->|NFS v4.2| MNT
S1 -->|NFS v4.2| MNT
S2 -->|NFS v4.2| MNT
S3stale -.->|broken| MNT
```
### NFS export details
| Server | Export path | NFS ID | Protocol | Options (client-side) |
|--------|-------------|--------|----------|----------------------|
| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) |
| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | — | **broken** (disk gone, storage.cfg entry stale) |
> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from
> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on
> all mounts.
---
## 4. Per-spindle VM placement (running VMs only)
> Shows every running VM's primary disk and the physical spindle it lives on.
> This is the IOPS distribution map — the key to storage balancing.
```mermaid
graph TB
subgraph "tsys4 spindles"
D2["D2 — WDC Red 3TB<br/>★★★ 13 VMs — WORST HOTSPOT"]
D5["D5 — Hitachi 2TB<br/>4 VMs"]
D1["D1 — WD 1TB USB<br/>0 VMs (empty)"]
D3["D3 — Seagate 500GB USB<br/>0 VMs (empty)"]
ALB["/mnt/albert — Hitachi 2TB<br/>0 VMs (not in Proxmox)"]
end
subgraph "tsys5 spindles"
S1["S1 — IronWolf 1TB<br/>1 NFS VM + 12 local stopped"]
S2["S2 — IronWolf 1TB<br/>5 VMs"]
SSD["SSD — Samsung 860 PRO<br/>1 VM (93% full!)"]
NVME["NVMe — Crucial CT500P1<br/>NOT CONFIGURED"]
HYNIX["SK hynix SSD 512GB<br/>NOT CONFIGURED"]
end
D2 --- D2vms["100 101 104 105 108<br/>114 117 509 600 702<br/>704 901 904"]
D5 --- D5vms["106 313 600(d0) 903"]
S1 --- S1vms["702 hfnoc-uisp"]
S2 --- S2vms["103 703 705 902 905"]
SSD --- SSDvms["5111 ultix-streaming"]
```
### Detailed per-spindle load
| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes |
|---------|--------|--------|------|-------------|-------|-------|
| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** |
| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate |
| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | — | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | — | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | — | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** |
### Local-only VM disks (not on NFS)
| Host | Storage ID | Disk | VMs | Notes |
|------|-----------|------|-----|-------|
| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk |
| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage |
| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. |
| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. |
---
## 5. Storage capacity summary
| Spindle | Size | Used | Available | % Used |
|---------|------|------|-----------|--------|
| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% |
| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% |
| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% |
| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% |
| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% |
| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% |
| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** |
| NVMe (CT500P1) | 466 GB | — | — | Not configured |
| SK hynix SSD | 477 GB | — | — | Not configured |
**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2
**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup
**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD
+225
View File
@@ -0,0 +1,225 @@
# VM Inventory
> **Every VM in the cluster, with host placement, disk location, CPU/RAM
> allocation, and optimization status.** Data gathered 2026-07-30.
>
> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md),
> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md).
---
## 1. VM placement diagram
```mermaid
graph TB
subgraph tsys1["tsys1 — Infra (57% RAM)"]
VM100["100 pfv-bms<br/>2c/4G D2"]
VM101["101 tsys-ca<br/>2c/2G D2"]
VM103["103 netinfra-01<br/>2c/4G S2"]
VM104["104 librenms<br/>2c/2G D2"]
VM105["105 proxmox-dc<br/>2c/2G D2"]
VM106["106 cnode3<br/>4c/4G D5"]
VM108["108 ucs-01<br/>2c/8G D2"]
VM114["114 kali-tsys<br/>2c/2G D2"]
VM117["117 workbench<br/>2c/4G D2"]
end
subgraph tsys3["tsys3 — Compute (10% RAM)"]
VM313["313 wnode-tsys3<br/>8c/28G D5"]
end
subgraph tsys4["tsys4 — Storage (15% RAM)"]
VM400["400 PBS<br/>2c/2G local-lvm"]
end
subgraph tsys5["tsys5 — Storage+Sandbox (15% RAM)"]
VM509["509 wnode-tsys5<br/>2c/32G D2"]
VM515["515 hfnoc-uisp-preprod<br/>2c/2G local-nonprod"]
VM5111["5111 ultix-streaming<br/>4c/9G SSD"]
VM5112["5112 ultix-offstage<br/>4c/6G local-lvm"]
end
subgraph tsys6["tsys6 — RackRental (12% RAM)"]
VM100b["100 wnode-tsys6<br/>2c/64G S4"]
VM600["600 tsys-awx<br/>2c/12G D5"]
end
subgraph tsys7["tsys7 — RackRental (7% RAM)"]
VM701["701 wnode-tsys7<br/>4c/32G S3"]
VM702["702 hfnoc-uisp<br/>2c/8G S1"]
VM703["703 rr-middleware<br/>2c/2G S2"]
VM704["704 TCTC<br/>4c/6G D2"]
VM705["705 cnode2<br/>4c/4G S2"]
end
subgraph tsys9["tsys9 — Infra+Compute (69% RAM)"]
VM901["901 tsys-siem<br/>2c/8G D2"]
VM902["902 ucs-02<br/>2c/8G S2"]
VM903["903 kali-rd<br/>2c/2G D5"]
VM904["904 netinfra-02<br/>2c/4G D2"]
VM905["905 wnode-tsys9<br/>4c/4G S2"]
VM906["906 cnode1<br/>2c/4G S3"]
end
```
---
## 2. Complete running VM inventory (28 running VMs)
### Infrastructure (netinfra, UCS, CA, LibreNMS, PDM)
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | — | — | ✓ | discard=on, ssd=1. HA for power/temp monitoring. |
| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | no nested-virt flag. Certificate Authority. |
| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | — | DNS primary (Technitium + Pi-hole). |
| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | SNMP monitoring for switches. |
| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | PDM (Proxmox Datacenter Manager). |
| 108 | tsys-ucs-01 | tsys1 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Univention Corporate Server half 1. |
| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | — | ✓ | ✓ | UCS half 2 (HA pair with 108). |
| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | — | ✓ | — | DNS secondary. HA pair with 103. |
### k8s control plane (cnode1/2/3)
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
| 106 | pfv-k8s-cnode3 | tsys1 | 4 | host | 4G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. |
| 705 | pfv-k8s-cnode2 | tsys7 | 4 | host | 4G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. |
| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | — | ✓ | — | etcd member. |
### k8s workers (wnodes)
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
| 313 | pfv-k8s-wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Compute worker. |
| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | Undersized vCPU (2). |
| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | — | ✓ | — | Undersized vCPU (2), high RAM. |
| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | — | ✓ | — | RackRental worker. |
| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | ✓ | Target: grow to 6c/8G. |
### Application VMs
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------|
| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Kali Linux pentest. |
| 117 | tsys-secure-workbench | tsys1 | 2 | host | 4G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Secure workbench. |
| 400 | pfv-proxmox-backup-server | tsys4 | 2 | — | 2G | local-lvm | (local) | raw | — | — | — | PBS — all VM backups. |
| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | — | ✓ | — | Preprod. Locked to tsys5. |
| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | SSD dir | SSD | qcow2 | — | — | — | 288GB on Samsung 860 PRO (**93% full**). |
| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | (local) | raw | — | — | — | 288GB. Locked to tsys5. |
| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Ansible AWX. |
| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | — | ✓ | ✓ | 100GB disk. UISP network monitoring. |
| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | RackRental middleware. |
| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | |
| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | 132GB disk. SIEM. |
| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Kali R&D. |
---
## 3. Optimization findings
### 3.1 Cache mode audit
`cache=writethrough` forces synchronous writes to the NFS server on every write
operation. For NFS-backed qcow2, this is unnecessarily slow — `writeback` or
`none` is recommended (qcow2 already provides journaling integrity).
| Cache mode | Count | VMs |
|-----------|-------|-----|
| **writethrough** (slow) | **10** | 106, 108, 114, 117, 313, 600, 703, 705, 901, 903 |
| none/default (optimal) | 18 | all others |
**Fix:** Change `cache=writethrough``cache=writeback` or remove (defaults to
writeback for qcow2). Requires VM stop/start.
### 3.2 CPU type audit
`cpu: host` passes the host's full CPU instruction set to the VM (best
performance). `x86-64-v2-AES` is a conservative baseline (safe for migration
but lacks some host instructions).
| CPU type | Count | VMs |
|----------|-------|-----|
| **host** | 21 | most production VMs |
| **x86-64-v2-AES** | 5 | 101, 103, 509, 904, 905 |
| not set | 2 | 400, 5111, 5112 |
**Note:** Mixed CPU types prevent live migration between hosts with different
CPU generations. K8s cnodes/wnodes should ideally match within each tier.
### 3.3 Disk controller audit
| Controller | Count | Notes |
|-----------|-------|-------|
| **scsi0** (virtio-scsi) | 22 | Optimal for Proxmox |
| **virtio0** | 4 | 101, 509, 701, 704 — also good |
| raw (no virtio) | 2 | 100 (pfv-bms), 500 (stopped) |
All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks.
### 3.4 Network audit
All running VMs use **virtio-net**. Firewall enabled on most (`firewall=1`).
No legacy e1000/rtl8139 adapters.
### 3.5 Guest agent (qga) audit
| Status | Count | VMs without qga |
|--------|-------|-----------------|
| **enabled** | 20 | — |
| **not set** | 8 | 101, 103, 509, 100(tsys6), 701, 704, 904, 906 |
**Fix:** Enable `qm set <vmid> --agent 1` on the 8 VMs missing it. No VM
stop/start required (takes effect on next guest reboot, but the setting itself
is immediate).
### 3.6 k8s wnode sizing
| VMID | Name | vCPU | RAM | Tier target | Issue |
|------|------|------|-----|-------------|-------|
| 509 | wnode-tsys5 | 2 | 32G | compute | **Undersized vCPU** (2 for a compute worker) |
| 100 | wnode-tsys6 | 2 | 64G | rackrental | **Undersized vCPU** (2 for 64G RAM) |
| 701 | wnode-tsys7 | 4 | 32G | rackrental | OK for idle labs |
| 905 | wnode-tsys9 | 4 | 4G | compute | **Target: grow to 6c/8G** after siem migrates off |
| 313 | wnode-tsys3 | 8 | 28G | compute | OK (best compute worker) |
---
## 4. Stopped VMs (not running, for reference)
### Sectestbed / sandboxes (all on tsys5, all stopped)
| VMID | Name | RAM | Disk | Notes |
|------|------|-----|------|-------|
| 5100 | sectestbed-template | 4G | local-lvm | Base template |
| 5101 | sectestbed-siem | 10G | local-nonprod (S1) | 132G disk |
| 5102 | sectestbed-proxmox-pve | 4G | local-lvm | |
| 5103 | sectestbed-proxmox-datacenter | 4G | local-lvm | |
| 5104 | sectestbed-proxmox-pbs | 4G | local-lvm | |
| 5105 | sectestbed-awx | 4G | local-nonprod (S1) | 288G disk |
| 5106 | sectestbed-k8s-cnode | 4G | local-nonprod (S1) | |
| 5107 | sectestbed-k8s-wnode | 4G | local-nonprod (S1) | |
| 5108 | sectestbed-librenms | 4G | local-nonprod (S1) | |
| 5109 | sectestbed-netinfra | 4G | local-nonprod (S1) | |
| 51010 | sectestbed-tctc | 4G | local-nonprod (S1) | |
| 51011 | sectestbed-cloudron | 4G | local-nonprod (S1) | |
| 51012 | sectestbed-hfnoc | 4G | local-nonprod (S1) | |
| 51013 | sectestbed-rancherplatform | 4G | local-nonprod (S1) | |
| 51014 | sectestbed-proxmox-mailgw | 4G | local-nonprod (S1) | |
| 53100 | tsys-preprod-awx | 9G | local-nonprod (S1) | 160G disk |
| 53101 | tsys-preprod-siem | 12G | local-nonprod (S1) | |
| 53102 | tsys-preprod-rancherplatform | 8G | local-nonprod (S1) | |
| 53103 | tsys-preprod-proxmoxmailgw | 4G | local-nonprod (S1) | |
| 6000 | sectestbed-sandbox | 4G | local-nonprod (S1) | Has 3 state snapshots |
### Other stopped VMs
| VMID | Name | Host | RAM | Disk | Notes |
|------|------|------|-----|------|-------|
| 102 | pfv-k8s-wnode-tsys1 | tsys1 | 4G | (no disk) | Stopped, no boot disk |
| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail — disk export removed** |
| 5500 | RestoreTemplate | tsys5 | 2G | D2 | Template |
| 1000 | KNELTemplate | tsys1 | 2G | D2 | Base template |
| 3000 | KNELTemplate | tsys3 | 2G | D2 | Base template copy |
| 4000 | KNELTemplate | tsys4 | 2G | (local) | Base template copy |