diff --git a/docs/docmap.md b/docs/docmap.md index e3f04f3..c7b6037 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -22,6 +22,10 @@ Fleet operations, hardware, performance tuning, storage architecture. | Document | Description | Last Reviewed | |----------|-------------|---------------| +| [`proxmox/docs/NETWORK-TOPOLOGY.md`](proxmox/docs/NETWORK-TOPOLOGY.md) | **Full network reference**: Mermaid topology diagrams, complete switch port maps, VLAN/subnet map, cross-rack trunk analysis, NFS data-flow, per-host NIC reference | 2026-07-30 | +| [`proxmox/docs/STORAGE-ARCHITECTURE.md`](proxmox/docs/STORAGE-ARCHITECTURE.md) | **Full storage reference**: tier definitions, physical disk inventory (all 14 disks), NFS export/mount map, per-spindle VM placement, capacity summary | 2026-07-30 | +| [`proxmox/docs/HOST-REFERENCE.md`](proxmox/docs/HOST-REFERENCE.md) | **Per-host reference**: hardware, network, storage, tuning, role, running VMs for all 7 nodes | 2026-07-30 | +| [`proxmox/docs/VM-INVENTORY.md`](proxmox/docs/VM-INVENTORY.md) | **Complete VM catalog**: placement diagram, all 28 running VMs with disk/CPU/RAM/config, optimization findings (cache mode, CPU type, qga audit), stopped VMs | 2026-07-30 | | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | | [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 | diff --git a/proxmox/docs/HOST-REFERENCE.md b/proxmox/docs/HOST-REFERENCE.md new file mode 100644 index 0000000..d2c525b --- /dev/null +++ b/proxmox/docs/HOST-REFERENCE.md @@ -0,0 +1,173 @@ +# Host Reference + +> **One-page-per-host quick reference for every Proxmox node.** Hardware, +> network, storage, role, and running VMs. Data gathered 2026-07-30. +> +> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), +> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), +> [`VM-INVENTORY.md`](VM-INVENTORY.md). + +--- + +## Fleet summary + +```mermaid +graph TB + subgraph "RACK 5 — Core / Data / Infra" + T1["tsys1 — Infra
i7-4770 · 32GB · 9 VMs
57% RAM"] + T3["tsys3 — Compute
Xeon E3-1535M v5 · 32GB · 1 VM
10% RAM"] + T4["tsys4 — Storage+PBS
Xeon E3-1246 v3 · 16GB · 1 VM
15% RAM"] + T5["tsys5 — Storage+Sandbox
Xeon E5620 · 96GB · 4 VMs
15% RAM"] + T9["tsys9 — Infra+Compute
i5-10500 · 24GB · 6 VMs
69% RAM"] + end + + subgraph "RACK 3 — Compute / RackRental" + T6["tsys6 — RackRental
2× Xeon E5530 · 126GB · 2 VMs
12% RAM"] + T7["tsys7 — RackRental
2× Xeon E5-2630 v2 · 189GB · 5 VMs
7% RAM"] + end +``` + +| Host | CPU (era) | Threads | RAM | PVE | Kernel | Role | Running VMs | RAM actual% | +|------|-----------|---------|-----|-----|--------|------|-------------|-------------| +| tsys1 | i7-4770 Haswell (2013) | 8 | 32 GB | 9.2.5 | 7.0.14 | Infra | 9 | 57% | +| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% | +| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% | +| tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% | +| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% | +| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% | +| tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% | + +**Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired) + +--- + +## pfv-tsys1 — Infrastructure host + +| Attribute | Value | +|-----------|-------| +| **Role** | Infrastructure: Home Assistant, CA, netinfra-01, LibreNMS, PDM, k8s cnode3, UCS-01, kali, workbench | +| **Hardware** | Dell, i7-4770 Haswell (2013), 8 threads, 32 GB RAM (maxed) | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.3.11 (vmbr0, DHCP) | +| **Tailscale** | 100.121.189.98 | +| **Storage IP** | 10.100.100.1 (datanet via nic1, USB dongle, no bond) | +| **NFS mounts** | S1, S2, S3, S4, D2 (all nconnect=4) | +| **Local storage** | local-lvm (~90 GB, HDD, marginal) | +| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | +| **Bond** | None (single USB NIC for storage) | +| **Special services** | NUT (UPS), powerman (PDU) | +| **Running VMs** | 100, 101, 103, 104, 105, 106, 108, 114, 117 | + +--- + +## pfv-tsys3 — Compute host + +| Attribute | Value | +|-----------|-------| +| **Role** | Compute: k8s wnode-tsys3 (28 GB RAM, 8 vCPU) | +| **Hardware** | Xeon E3-1535M v5 Skylake (2015), 8 threads, 32 GB RAM | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) | +| **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) | +| **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) | +| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) — **best existing local storage** | +| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | +| **Bond** | None (single USB NIC for storage) | +| **Running VMs** | 313 | + +--- + +## pfv-tsys4 — Storage server + PBS + +| Attribute | Value | +|-----------|-------| +| **Role** | NFS storage server (D1/D2/D3/D5) + Proxmox Backup Server + console management (conman/ser2net) | +| **Hardware** | Xeon E3-1246 v3 Haswell (2013), 8 threads, 16 GB RAM | +| **PVE** | **9.1.5** / kernel **6.17.9-1-pve** (BEHIND fleet) | +| **mgmt IP** | 192.168.3.251 (vmbr0, DHCP) | +| **Tailscale** | 100.70.77.93 | +| **Storage IP** | 10.100.100.4 (datanet via enx8cae4ccda926 USB dongle, bond0 with 1 slave) | +| **NFS mounts** | None (NFS server only) | +| **Local storage** | local-lvm (WDC WD10EZRX 1TB, ~62 GB free) | +| **NFS exports** | D1 (WD 1TB USB), D2 (WDC Red 3TB), D3 (Seagate 500GB USB), D5 (Hitachi 2TB) | +| **Other disks** | /mnt/albert (Hitachi 2TB, not exported), /mnt/backup (WD 5TB USB) | +| **Tuning** | governor=performance, swappiness=1, bbr, **rmem/wmem=16MB** (should be 128MB), **tuned=throughput-performance** (should be network-throughput) | +| **Bond** | bond0 (1 USB slave), hash=layer2+3 | +| **Special services** | ser2net + conman (7 switch consoles on :7890) | +| **Running VMs** | 400 (PBS) | +| **Notes** | USB NIC bottleneck. Needs PVE upgrade + tuning alignment. | + +--- + +## pfv-tsys5 — Storage server + sandboxes + +| Attribute | Value | +|-----------|-------| +| **Role** | NFS storage server (S1/S2) + sectestbed sandboxes + ultix streaming | +| **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) | +| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** — nic2 DOWN) | +| **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) | +| **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) | +| **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) | +| **Unconfigured disks** | **NVMe CT500P1 500GB** (not mounted), **SK hynix SC300B 512GB SSD** (not mounted) | +| **Tuning** | governor=N/A (no cpufreq driver on Westmere), swappiness=1, bbr, rmem/wmem=128MB, tuned=network-throughput | +| **Bond** | bond0 (1 of 2 slaves), hash=layer2+3 | +| **Running VMs** | 509, 515, 5111, 5112 (+ 22 stopped sandboxes) | +| **Notes** | Bond degraded (needs 2nd cable). NVMe + SSD need mounting/config. SSD 93% full. Single CPU. | + +--- + +## pfv-tsys6 — RackRental host + +| Attribute | Value | +|-----------|-------| +| **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX | +| **Hardware** | 2× Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) | +| **Tailscale** | 100.73.35.111 | +| **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) | +| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | +| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) | +| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | +| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | +| **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) | +| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 — unusable for scratch. | + +--- + +## pfv-tsys7 — RackRental host + +| Attribute | Value | +|-----------|-------| +| **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 | +| **Hardware** | 2× Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) | +| **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) | +| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | +| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) | +| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | +| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | +| **Running VMs** | 701, 702, 703, 704, 705 | +| **Notes** | Best storage bond in fleet (2G LACP). Most RAM in fleet (189 GB). Local storage USB 2.0. | + +--- + +## pfv-tsys9 — Infra + Compute + +| Attribute | Value | +|-----------|-------| +| **Role** | Infra + Compute: k8s cnode1 + wnode-tsys9 + siem + UCS-02 + netinfra-02 + kali | +| **Hardware** | i5-10500 Comet Lake (2020), 12 threads, 24 GB RAM | +| **PVE** | 9.2.5 / kernel 7.0.14-6-pve | +| **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) | +| **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) | +| **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) | +| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) — **excellent for scratch** | +| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | +| **Bond** | None (single USB NIC for storage) | +| **Running VMs** | 901, 902, 903, 904, 905, 906 | +| **Notes** | Newest CPU (best IPC in fleet). **69% RAM — approaching 80% goal.** Target for compute growth (after siem migrates off). | diff --git a/proxmox/docs/NETWORK-TOPOLOGY.md b/proxmox/docs/NETWORK-TOPOLOGY.md new file mode 100644 index 0000000..29fedd3 --- /dev/null +++ b/proxmox/docs/NETWORK-TOPOLOGY.md @@ -0,0 +1,365 @@ +# Network Topology + +> **Complete physical + logical network reference for the PFVCluster.** +> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct +> host probes. All data is live ground truth — not documentation inherited. +> +> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), +> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). +> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). + +--- + +## 1. Physical topology diagram + +```mermaid +graph TB + subgraph "RACK 5 — Core / Data / Infra" + CORE["pfv-core-sw01
PowerConnect 5448
192.168.0.12
FW 2.0.0.46
(mgmt + storage VLANs)"] + + TSYS1["pfv-tsys1
i7-4770 Haswell
Infra host"] + TSYS3["pfv-tsys3
Xeon E3-1535M v5
Compute"] + TSYS4["pfv-tsys4
Xeon E3-1246 v3
NFS Storage + PBS"] + TSYS5["pfv-tsys5
Xeon E5620 Westmere
NFS Storage + Sandboxes"] + TSYS9["pfv-tsys9
i5-10500 Comet Lake
Infra + Compute"] + + TSYS4_CONSOLE["conman + ser2net
7 switch consoles
(Tailscale 100.70.77.93:7890)"] + TSYS1_PDU["Cyclades PM10i PDU
powerman on tsys1
(port 10101)"] + TSYS1_UPS["APC Smart-UPS C 1500
NUT on tsys1
(port 3493)"] + end + + subgraph "RACK 3 — Compute / RackRental" + TOR3STOR["pfv-tor3-stor
PowerConnect 5324 (Neyland 24T)
192.168.0.9
FW 2.0.1.4
(storage VLAN)"] + TOR3MGMT["pfv-tor3-mgmt
PowerConnect 5324 (Neyland 24T)
192.168.0.7
(mgmt VLAN)"] + + TSYS6["pfv-tsys6
2× Xeon E5530 Nehalem
RackRental"] + TSYS7["pfv-tsys7
2× Xeon E5-2630 v2 Ivy Bridge
RackRental"] + end + + %% Cross-rack trunk + CORE <--. "ch1 LAG: 3×1G active
(g13,g14,g15)
hash=layer-2-3
mode=on (static)" .--> TOR3STOR + TOR3STOR -. "ch1: 4×1G
(g20,g21,g22,g23)
hash=layer-2-3-4
mode=on (static)" .-> CORE + + %% Rack 5 connections to core + CORE --> TSYS1 + CORE --> TSYS3 + CORE --> TSYS4 + CORE --> TSYS5 + CORE --> TSYS9 + + %% Rack 3 connections + TOR3STOR --> TSYS6 + TOR3STOR --> TSYS7 + TOR3MGMT --> TSYS6 + TOR3MGMT --> TSYS7 +``` + +### How to read this diagram + +The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** — +the most critical link in the cluster. It carries all NFS storage traffic +between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). +The trunk is a 4-port static LAG, but only 3 links are active due to a cable +mismatch (see §3 below). + +--- + +## 2. Subnet / VLAN map + +| VLAN | Subnet | Purpose | Where it lives | +|------|--------|---------|----------------| +| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) | +| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) | +| — (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs | + +**Management IPs (VLAN 1, via DHCP reservations):** + +| Host | mgmt IP (vmbr0) | Tailscale IP | +|------|-----------------|--------------| +| tsys1 | 192.168.3.11 | 100.121.189.98 | +| tsys3 | 192.168.2.5 | (connected) | +| tsys4 | 192.168.3.251 | 100.70.77.93 | +| tsys5 | 192.168.3.179 | (connected) | +| tsys6 | 192.168.3.169 | 100.73.35.111 | +| tsys7 | 192.168.3.55 | (connected) | +| tsys9 | 192.168.3.58 | (connected) | + +**Storage IPs (VLAN 1000, static):** + +| Host | Storage IP | Interface | +|------|-----------|-----------| +| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) | +| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) | +| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) | +| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) | +| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) | +| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) | +| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) | + +**Switch IPs (VLAN 1, static DHCP reservations):** + +| Switch | IP | Model | Console | +|--------|----|-------|---------| +| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 | +| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 | +| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 | +| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? | + +**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c) + +--- + +## 3. Cross-rack trunk detail (ch1) — the critical link + +This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and +rack 5 (tsys4/5). It is the single most important network path in the cluster. + +```mermaid +graph LR + subgraph "core-sw01 (rack5)" + G13["g13
1G UP"] + G14["g14
1G UP"] + G15["g15
1G UP"] + G16["g16
1G UP
NOT in LAG!"] + G17["g17
DOWN
in config but
no cable"] + end + + subgraph "tor3-stor (rack3)" + G20["g20
1G UP"] + G21["g21
1G UP"] + G22["g22
1G UP"] + G23["g23
1G UP"] + end + + G13 --- G20 + G14 --- G21 + G15 --- G22 + G16 -.- G23 + G17 -.->|no cable| G23 +``` + +### Configuration comparison + +| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? | +|----------|-------------------|--------------------|--------| +| Physical ports configured | g(13-15,17) | g(20-23) | **No — asymmetric** | +| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | — | +| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No — 3 vs 4** | +| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No — asymmetric** | +| Channel mode | on (static) | on (static) | Yes (both should be LACP) | +| VLAN | access vlan 1000 | access vlan 1000 | Yes | + +### Problems + +1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on + core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex) + but is NOT in the LAG group. g17 is configured in the LAG but has no cable + (DOWN). Result: **only 3 of 4 links are active**. + +2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP). + tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With + layer-2-3 on core-sw01, **all traffic between a given host pair hashes to + one link** regardless of how many TCP connections (nconnect) are used. + +3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure + detection — a dead cable stays in the LAG until manually discovered. + +--- + +## 4. Complete switch port map + +### pfv-core-sw01 (PowerConnect 5448, 48 ports) — rack 5 + +| Port | Speed | Status | VLAN | Description / Connected device | +|------|-------|--------|------|-------------------------------| +| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) | +| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) | +| g5 | — | UP | 1 | pfv-tsys1 (mgmt) | +| g6 | 100 | UP | 1 | pfvsvrpi (secondary) | +| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt | +| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** | +| g10 | 1000 | UP | 1 | (unknown) | +| g11 | 1000 | UP | 1 | (unknown) | +| g12 | 1000 | UP | 1 | (unknown) | +| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** | +| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** | +| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** | +| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG — should be g17)** | +| **g17** | — | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** | +| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt | +| g21 | 1000 | UP | 1 | (unknown) | +| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) | +| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** | +| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) | +| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) | +| g32 | — | — | — | pfv-tsys4-storage (configured, DOWN) | +| g33 | — | — | — | pfv-tsys5-stor (ch2, DOWN/Not Present) | +| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) | +| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** | +| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) | +| g41 | 1000 | UP | 1 | (unknown) | +| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) | +| g44 | 1000 | UP | 1 | (unknown) | +| g45-g48 | — | DOWN | — | (Combo ports, unused) | + +**Port-channels on core-sw01:** + +| Channel | Speed | Status | Members | VLAN | Notes | +|---------|-------|--------|---------|------|-------| +| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor | +| ch2 | — | DOWN | g32-33 (non-candidate) | — | Unused | +| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) | + +**LAG hash:** `port-channel load-balance layer-2-3` + +### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3 + +| Port | Speed | Status | VLAN | Description / Connected device | +|------|-------|--------|------|-------------------------------| +| g1 | 1000 | UP | 1 | (unknown — uplink to tor3-mgmt or router) | +| g2-g7 | 1000 | UP | 1 | (unused but UP) | +| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** | +| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** | +| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** | +| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** | +| **g13** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** | +| **g14** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** | +| g15-g19 | 1000 | UP | 1000 | (unused storage ports) | +| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** | +| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** | +| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** | +| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** | +| g24 | — | DOWN | 1 | (spanning-tree disabled, portfast) | + +**Port-channels on tor3-stor:** + +| Channel | Speed | Status | Members | VLAN | Notes | +|---------|-------|--------|---------|------|-------| +| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4×1G) | +| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2×1G LACP) | +| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2×1G LACP) | +| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE — retired host)** | + +**LAG hash:** `port-channel load-balance layer-2-3-4` + +### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3 + +| Port | Speed | Status | Connected device | +|------|-------|--------|-----------------| +| g3 | — | UP | tsys7 (mgmt nic0) | +| g7 | — | UP | tsys6 (mgmt nic0) | + +(Full port map not captured — this switch carries only VLAN 1 management +traffic. Low priority for storage performance.) + +--- + +## 5. Per-host network interface reference + +| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed | +|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------| +| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | — | 1G | +| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | — | 1G | +| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G | +| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** | +| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | +| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | +| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | — | 1G | + +### Key observations + +- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2 + active slaves). All NFS traffic from rack 3 uses these bonds. +- **tsys1/3/4/9 use USB dongles** for storage network — single 1G, no redundancy. +- **tsys5 bond0 is degraded** — nic2 is DOWN (no cable connected). Only nic1 is + active via core-sw01 ch3 (g34). This is the LACP problem the operator has been + troubleshooting. +- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS), + tsys4/5 use layer2+3 (suboptimal — same src/dst IP pair always hashes to one + link even with nconnect's multiple TCP connections). + +--- + +## 6. NFS data flow diagram + +```mermaid +graph TB + subgraph "Storage Servers (rack 5)" + T4D2["tsys4 D2
WDC Red 3TB
Tier 2"] + T4D5["tsys4 D5
Hitachi 2TB
Tier 2"] + T4D1["tsys4 D1
WD 1TB USB
Tier 3"] + T4D3["tsys4 D3
Seagate 500GB USB
Tier 3"] + T5S1["tsys5 S1
IronWolf 1TB
Tier 2"] + T5S2["tsys5 S2
IronWolf 1TB
Tier 2"] + T5SSD["tsys5 SSD
Samsung 860 PRO
Tier 1 (93% full)"] + end + + subgraph "VLAN 1000 storage network" + NFS["NFS v4.2
nconnect=4
rsize/wsize=1MB"] + end + + subgraph "NFS Clients" + T1["tsys1 (9 mounts)"] + T3["tsys3 (5 mounts)"] + T5["tsys5 (3 mounts, NO nconnect)"] + T6["tsys6 (9 mounts)"] + T7["tsys7 (9 mounts)"] + T9["tsys9 (5 mounts)"] + end + + T4D2 --> NFS + T4D5 --> NFS + T4D1 --> NFS + T4D3 --> NFS + T5S1 --> NFS + T5S2 --> NFS + T5SSD --> NFS + + NFS --> T1 + NFS --> T3 + NFS --> T5 + NFS --> T6 + NFS --> T7 + NFS --> T9 +``` + +### NFS mount inventory (live) + +| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | +|--------|--------|-------|-------|-------|-------|-------|-------|-------| +| tsys4 | D1 | — | — | — | — | ✓ | ✓ | — | +| tsys4 | **D2** | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | +| tsys4 | D3 | — | — | — | ✓ | ✓ | ✓ | — | +| tsys4 | D5 | — | ✓ | — | ✓ | ✓ | ✓ | ✓ | +| tsys5 | S1 | ✓ | — | — | — | ✓ | ✓ | — | +| tsys5 | S2 | ✓ | ✓ | — | — | ✓ | ✓ | — | +| tsys5 | S3 | ✓ | — | — | — | ✓ | ✓ | ✓ | +| tsys5 | S4 | ✓ | ✓ | — | — | ✓ | ✓ | ✓ | +| tsys5 | T5-SSD | — | ✓ | — | — | ✓ | ✓ | ✓ | + +(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. +tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.) + +--- + +## 7. Switch management access + +| Method | Details | +|--------|---------| +| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. | +| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` | +| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) | +| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) | +| **Direct telnet** | `telnet pfv-tsys4 200X` — ONLY when conmand stopped (conflicts). Use conman instead. | + +### Console port assignments + +| Port | Console name | Switch | ID_PATH (USB port) | +|------|-------------|--------|---------------------| +| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 | +| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 | +| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 | +| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 | +| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 | +| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 | +| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 | diff --git a/proxmox/docs/STORAGE-ARCHITECTURE.md b/proxmox/docs/STORAGE-ARCHITECTURE.md new file mode 100644 index 0000000..7f80e5e --- /dev/null +++ b/proxmox/docs/STORAGE-ARCHITECTURE.md @@ -0,0 +1,196 @@ +# Storage Architecture + +> **Complete storage reference for the PFVCluster.** Covers physical disks, +> NFS exports/mounts, Proxmox storage IDs, tier definitions, and per-spindle +> VM placement. Data gathered 2026-07-30 via live host probes. +> +> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), +> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). +> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). + +--- + +## 1. Storage tier definitions + +| Tier | Definition | Typical speed | Reliability | Use case | +|------|-----------|---------------|-------------|----------| +| **Tier 1** | NVMe or SSD on internal SATA/SAS/PCIe | 200-3000 MB/s | High | OS drives, high-IOPS workloads, build scratch | +| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s | Moderate | Production VM OS drives, general-purpose storage | +| **Tier 3** | HDD/SSD on USB adapter / "toaster" | 30-60 MB/s | Low (USB) | Bulk, backup, scratch only. Never for production OS drives. | + +**Policy (from [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md) §6):** Local disk +storage is a scratch tier, not a boot/OS tier. VM OS drives stay on NFS +(tsys4/tsys5) where redundancy is achieved by spreading across physical disks +and HA pairs. Local storage is for ephemeral data where loss on node failure +is acceptable. + +--- + +## 2. Physical disk inventory + +### tsys4 — NFS storage server + PBS (7 disks) + +| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used | +|--------|-------|------|-----|-----------|-------|--------|------|-------|------| +| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% | +| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) | +| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) | +| sdd | WDC WD10EZRX | 1.0 TB | — | SATA | (pve boot/swap/lvm) | — | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk | +| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% | +| sdf | WDC WD10EARS | 1.0 TB | — | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% | +| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% | + +**Changes from previous audit:** +- **D3 changed:** was SK hynix SC300 512GB SSD → now Seagate ST500LM021 500GB HDD (still USB). Demoted from unreliable-SSD to Tier 3 HDD. +- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. Unused Tier 2. +- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only. +- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone. + +### tsys5 — NFS storage server + sandboxes (7 disks) + +| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used | +|--------|-------|------|-----|-----------|-------|-------------|------|-------|------| +| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | — | +| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | — | **Tier 1** | PASSED | **NEW, not configured** | +| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | — | Tier 2 | PASSED | OS disk | +| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** | +| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) | +| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) | +| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | — | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** | + +**Changes from previous audit:** +- **NVMe added:** Crucial CT500P1 500GB. True NVMe — highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg. +- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured. +- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports. +- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry. +- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm). +- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) — needs attention. +- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model). + +--- + +## 3. NFS export → Proxmox storage ID map + +```mermaid +graph LR + subgraph "tsys4 — NFS Server" + D2["D2: /mnt/tsys4/D2
WDC Red 3TB
Tier 2 · 7% used"] + D5["D5: /mnt/tsys4/D5
Hitachi 2TB
Tier 2 · 13% used"] + D1["D1: /mnt/tsys4/D1
WD 1TB USB
Tier 3 · 0% used"] + D3["D3: /mnt/tsys4/D3
Seagate 500GB USB
Tier 3 · 0% used"] + end + + subgraph "tsys5 — NFS Server" + S1["S1: /mnt/pfv-tsys5/S1
IronWolf 1TB
Tier 2 · 11% used"] + S2["S2: /mnt/pfv-tsys5/S2
IronWolf 1TB
Tier 2 · 4% used"] + S3stale["S3: (stale export)
disk not mounted"] + end + + subgraph "All hosts — NFS Clients" + MNT["/mnt/pve/D1 D2 D3 D5
/mnt/pve/S1 S2 S3 S4
/mnt/pve/T5-SSD"] + end + + D2 -->|NFS v4.2| MNT + D5 -->|NFS v4.2| MNT + D1 -->|NFS v4.2| MNT + D3 -->|NFS v4.2| MNT + S1 -->|NFS v4.2| MNT + S2 -->|NFS v4.2| MNT + S3stale -.->|broken| MNT +``` + +### NFS export details + +| Server | Export path | NFS ID | Protocol | Options (client-side) | +|--------|-------------|--------|----------|----------------------| +| tsys4 | /mnt/tsys4/D1 | D1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys4 | /mnt/tsys4/D2 | D2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys4 | /mnt/tsys4/D3 | D3 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys4 | /mnt/tsys4/D5 | D5 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys5 | /mnt/pfv-tsys5/S1 | S1 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) | +| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | +| tsys5 | /mnt/pfv-tsys5/S4 | S4 | — | **broken** (disk gone, storage.cfg entry stale) | + +> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from +> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on +> all mounts. + +--- + +## 4. Per-spindle VM placement (running VMs only) + +> Shows every running VM's primary disk and the physical spindle it lives on. +> This is the IOPS distribution map — the key to storage balancing. + +```mermaid +graph TB + subgraph "tsys4 spindles" + D2["D2 — WDC Red 3TB
★★★ 13 VMs — WORST HOTSPOT"] + D5["D5 — Hitachi 2TB
4 VMs"] + D1["D1 — WD 1TB USB
0 VMs (empty)"] + D3["D3 — Seagate 500GB USB
0 VMs (empty)"] + ALB["/mnt/albert — Hitachi 2TB
0 VMs (not in Proxmox)"] + end + + subgraph "tsys5 spindles" + S1["S1 — IronWolf 1TB
1 NFS VM + 12 local stopped"] + S2["S2 — IronWolf 1TB
5 VMs"] + SSD["SSD — Samsung 860 PRO
1 VM (93% full!)"] + NVME["NVMe — Crucial CT500P1
NOT CONFIGURED"] + HYNIX["SK hynix SSD 512GB
NOT CONFIGURED"] + end + + D2 --- D2vms["100 101 104 105 108
114 117 509 600 702
704 901 904"] + D5 --- D5vms["106 313 600(d0) 903"] + S1 --- S1vms["702 hfnoc-uisp"] + S2 --- S2vms["103 703 705 902 905"] + SSD --- SSDvms["5111 ultix-streaming"] +``` + +### Detailed per-spindle load + +| Spindle | Server | NFS ID | Tier | Running VMs | VMIDs | Notes | +|---------|--------|--------|------|-------------|-------|-------| +| **D2** (WDC Red 3TB) | tsys4 | D2 | Tier 2 | **13** | 100,101,104,105,108,114,117,509,600,702,704,901,904 | **Worst IOPS hotspot** | +| D5 (Hitachi 2TB) | tsys4 | D5 | Tier 2 | 4 | 106,313,600(disk0),903 | Moderate | +| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod | +| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate | +| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** | +| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | — | Empty | +| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | — | Empty | +| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | — | Not in Proxmox | +| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** | +| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** | + +### Local-only VM disks (not on NFS) + +| Host | Storage ID | Disk | VMs | Notes | +|------|-----------|------|-----|-------| +| tsys4 | local-lvm | sdd (1TB internal) | 400 (PBS) | Proxmox boot/OS disk | +| tsys5 | local-lvm | sdc (2TB internal) | 5102,5103,5104,5112 | Mixed: stopped sandboxes + ultix-offstage | +| tsys5 | local-nonprod (=S1) | sde (IronWolf 1TB) | 5101,5105,5106-5109,515,53100-53103,6000,51010-51014 | All STOPPED sandboxes. Same physical disk as S1 NFS. | +| tsys5 | SSD dir | sdd (Samsung 860 PRO) | 5111 | ultix-streaming. 93% full. | + +--- + +## 5. Storage capacity summary + +| Spindle | Size | Used | Available | % Used | +|---------|------|------|-----------|--------| +| D2 (WDC Red 3TB) | 2.7 TB | 170 GB | 2.4 TB | 7% | +| D5 (Hitachi 2TB) | 1.8 TB | 220 GB | 1.5 TB | 13% | +| D1 (WD 1TB USB) | 916 GB | 2 MB | 870 GB | 0% | +| D3 (Seagate 500GB USB) | 458 GB | 2 MB | 435 GB | 0% | +| /mnt/albert (Hitachi 2TB) | 1.8 TB | 2 MB | 1.7 TB | 0% | +| /mnt/backup (WD 5TB USB) | 4.6 TB | 2 MB | 4.3 TB | 0% | +| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% | +| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% | +| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** | +| NVMe (CT500P1) | 466 GB | — | — | Not configured | +| SK hynix SSD | 477 GB | — | — | Not configured | + +**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2 +**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup +**Total Tier 1 available (unconfigured):** ~940 GB across NVMe + SK hynix SSD diff --git a/proxmox/docs/VM-INVENTORY.md b/proxmox/docs/VM-INVENTORY.md new file mode 100644 index 0000000..6a6c664 --- /dev/null +++ b/proxmox/docs/VM-INVENTORY.md @@ -0,0 +1,225 @@ +# VM Inventory + +> **Every VM in the cluster, with host placement, disk location, CPU/RAM +> allocation, and optimization status.** Data gathered 2026-07-30. +> +> Companion docs: [`NETWORK-TOPOLOGY.md`](NETWORK-TOPOLOGY.md), +> [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), +> [`HOST-REFERENCE.md`](HOST-REFERENCE.md). + +--- + +## 1. VM placement diagram + +```mermaid +graph TB + subgraph tsys1["tsys1 — Infra (57% RAM)"] + VM100["100 pfv-bms
2c/4G D2"] + VM101["101 tsys-ca
2c/2G D2"] + VM103["103 netinfra-01
2c/4G S2"] + VM104["104 librenms
2c/2G D2"] + VM105["105 proxmox-dc
2c/2G D2"] + VM106["106 cnode3
4c/4G D5"] + VM108["108 ucs-01
2c/8G D2"] + VM114["114 kali-tsys
2c/2G D2"] + VM117["117 workbench
2c/4G D2"] + end + + subgraph tsys3["tsys3 — Compute (10% RAM)"] + VM313["313 wnode-tsys3
8c/28G D5"] + end + + subgraph tsys4["tsys4 — Storage (15% RAM)"] + VM400["400 PBS
2c/2G local-lvm"] + end + + subgraph tsys5["tsys5 — Storage+Sandbox (15% RAM)"] + VM509["509 wnode-tsys5
2c/32G D2"] + VM515["515 hfnoc-uisp-preprod
2c/2G local-nonprod"] + VM5111["5111 ultix-streaming
4c/9G SSD"] + VM5112["5112 ultix-offstage
4c/6G local-lvm"] + end + + subgraph tsys6["tsys6 — RackRental (12% RAM)"] + VM100b["100 wnode-tsys6
2c/64G S4"] + VM600["600 tsys-awx
2c/12G D5"] + end + + subgraph tsys7["tsys7 — RackRental (7% RAM)"] + VM701["701 wnode-tsys7
4c/32G S3"] + VM702["702 hfnoc-uisp
2c/8G S1"] + VM703["703 rr-middleware
2c/2G S2"] + VM704["704 TCTC
4c/6G D2"] + VM705["705 cnode2
4c/4G S2"] + end + + subgraph tsys9["tsys9 — Infra+Compute (69% RAM)"] + VM901["901 tsys-siem
2c/8G D2"] + VM902["902 ucs-02
2c/8G S2"] + VM903["903 kali-rd
2c/2G D5"] + VM904["904 netinfra-02
2c/4G D2"] + VM905["905 wnode-tsys9
4c/4G S2"] + VM906["906 cnode1
2c/4G S3"] + end +``` + +--- + +## 2. Complete running VM inventory (28 running VMs) + +### Infrastructure (netinfra, UCS, CA, LibreNMS, PDM) + +| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | +|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| +| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | — | — | ✓ | discard=on, ssd=1. HA for power/temp monitoring. | +| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | no nested-virt flag. Certificate Authority. | +| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | — | DNS primary (Technitium + Pi-hole). | +| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | SNMP monitoring for switches. | +| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | PDM (Proxmox Datacenter Manager). | +| 108 | tsys-ucs-01 | tsys1 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Univention Corporate Server half 1. | +| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | — | ✓ | ✓ | UCS half 2 (HA pair with 108). | +| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | — | ✓ | — | DNS secondary. HA pair with 103. | + +### k8s control plane (cnode1/2/3) + +| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | +|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| +| 106 | pfv-k8s-cnode3 | tsys1 | 4 | host | 4G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | +| 705 | pfv-k8s-cnode2 | tsys7 | 4 | host | 4G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | +| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | — | ✓ | — | etcd member. | + +### k8s workers (wnodes) + +| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | +|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| +| 313 | pfv-k8s-wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Compute worker. | +| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | Undersized vCPU (2). | +| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | — | ✓ | — | Undersized vCPU (2), high RAM. | +| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | — | ✓ | — | RackRental worker. | +| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | ✓ | Target: grow to 6c/8G. | + +### Application VMs + +| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | +|------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| +| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Kali Linux pentest. | +| 117 | tsys-secure-workbench | tsys1 | 2 | host | 4G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Secure workbench. | +| 400 | pfv-proxmox-backup-server | tsys4 | 2 | — | 2G | local-lvm | (local) | raw | — | — | — | PBS — all VM backups. | +| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | — | ✓ | — | Preprod. Locked to tsys5. | +| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | SSD dir | SSD | qcow2 | — | — | — | 288GB on Samsung 860 PRO (**93% full**). | +| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | (local) | raw | — | — | — | 288GB. Locked to tsys5. | +| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Ansible AWX. | +| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | — | ✓ | ✓ | 100GB disk. UISP network monitoring. | +| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | RackRental middleware. | +| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | | +| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | 132GB disk. SIEM. | +| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Kali R&D. | + +--- + +## 3. Optimization findings + +### 3.1 Cache mode audit + +`cache=writethrough` forces synchronous writes to the NFS server on every write +operation. For NFS-backed qcow2, this is unnecessarily slow — `writeback` or +`none` is recommended (qcow2 already provides journaling integrity). + +| Cache mode | Count | VMs | +|-----------|-------|-----| +| **writethrough** (slow) | **10** | 106, 108, 114, 117, 313, 600, 703, 705, 901, 903 | +| none/default (optimal) | 18 | all others | + +**Fix:** Change `cache=writethrough` → `cache=writeback` or remove (defaults to +writeback for qcow2). Requires VM stop/start. + +### 3.2 CPU type audit + +`cpu: host` passes the host's full CPU instruction set to the VM (best +performance). `x86-64-v2-AES` is a conservative baseline (safe for migration +but lacks some host instructions). + +| CPU type | Count | VMs | +|----------|-------|-----| +| **host** | 21 | most production VMs | +| **x86-64-v2-AES** | 5 | 101, 103, 509, 904, 905 | +| not set | 2 | 400, 5111, 5112 | + +**Note:** Mixed CPU types prevent live migration between hosts with different +CPU generations. K8s cnodes/wnodes should ideally match within each tier. + +### 3.3 Disk controller audit + +| Controller | Count | Notes | +|-----------|-------|-------| +| **scsi0** (virtio-scsi) | 22 | Optimal for Proxmox | +| **virtio0** | 4 | 101, 509, 701, 704 — also good | +| raw (no virtio) | 2 | 100 (pfv-bms), 500 (stopped) | + +All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks. + +### 3.4 Network audit + +All running VMs use **virtio-net**. Firewall enabled on most (`firewall=1`). +No legacy e1000/rtl8139 adapters. + +### 3.5 Guest agent (qga) audit + +| Status | Count | VMs without qga | +|--------|-------|-----------------| +| **enabled** | 20 | — | +| **not set** | 8 | 101, 103, 509, 100(tsys6), 701, 704, 904, 906 | + +**Fix:** Enable `qm set --agent 1` on the 8 VMs missing it. No VM +stop/start required (takes effect on next guest reboot, but the setting itself +is immediate). + +### 3.6 k8s wnode sizing + +| VMID | Name | vCPU | RAM | Tier target | Issue | +|------|------|------|-----|-------------|-------| +| 509 | wnode-tsys5 | 2 | 32G | compute | **Undersized vCPU** (2 for a compute worker) | +| 100 | wnode-tsys6 | 2 | 64G | rackrental | **Undersized vCPU** (2 for 64G RAM) | +| 701 | wnode-tsys7 | 4 | 32G | rackrental | OK for idle labs | +| 905 | wnode-tsys9 | 4 | 4G | compute | **Target: grow to 6c/8G** after siem migrates off | +| 313 | wnode-tsys3 | 8 | 28G | compute | OK (best compute worker) | + +--- + +## 4. Stopped VMs (not running, for reference) + +### Sectestbed / sandboxes (all on tsys5, all stopped) + +| VMID | Name | RAM | Disk | Notes | +|------|------|-----|------|-------| +| 5100 | sectestbed-template | 4G | local-lvm | Base template | +| 5101 | sectestbed-siem | 10G | local-nonprod (S1) | 132G disk | +| 5102 | sectestbed-proxmox-pve | 4G | local-lvm | | +| 5103 | sectestbed-proxmox-datacenter | 4G | local-lvm | | +| 5104 | sectestbed-proxmox-pbs | 4G | local-lvm | | +| 5105 | sectestbed-awx | 4G | local-nonprod (S1) | 288G disk | +| 5106 | sectestbed-k8s-cnode | 4G | local-nonprod (S1) | | +| 5107 | sectestbed-k8s-wnode | 4G | local-nonprod (S1) | | +| 5108 | sectestbed-librenms | 4G | local-nonprod (S1) | | +| 5109 | sectestbed-netinfra | 4G | local-nonprod (S1) | | +| 51010 | sectestbed-tctc | 4G | local-nonprod (S1) | | +| 51011 | sectestbed-cloudron | 4G | local-nonprod (S1) | | +| 51012 | sectestbed-hfnoc | 4G | local-nonprod (S1) | | +| 51013 | sectestbed-rancherplatform | 4G | local-nonprod (S1) | | +| 51014 | sectestbed-proxmox-mailgw | 4G | local-nonprod (S1) | | +| 53100 | tsys-preprod-awx | 9G | local-nonprod (S1) | 160G disk | +| 53101 | tsys-preprod-siem | 12G | local-nonprod (S1) | | +| 53102 | tsys-preprod-rancherplatform | 8G | local-nonprod (S1) | | +| 53103 | tsys-preprod-proxmoxmailgw | 4G | local-nonprod (S1) | | +| 6000 | sectestbed-sandbox | 4G | local-nonprod (S1) | Has 3 state snapshots | + +### Other stopped VMs + +| VMID | Name | Host | RAM | Disk | Notes | +|------|------|------|-----|------|-------| +| 102 | pfv-k8s-wnode-tsys1 | tsys1 | 4G | (no disk) | Stopped, no boot disk | +| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail — disk export removed** | +| 5500 | RestoreTemplate | tsys5 | 2G | D2 | Template | +| 1000 | KNELTemplate | tsys1 | 2G | D2 | Base template | +| 3000 | KNELTemplate | tsys3 | 2G | D2 | Base template copy | +| 4000 | KNELTemplate | tsys4 | 2G | (local) | Base template copy |