From 9c2882ea00bd99c9ec100b88a34f961e5e7b2ba0 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Fri, 31 Jul 2026 00:12:04 -0500 Subject: [PATCH] fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mermaid diagrams fixed for Gitea rendering: - Replaced all
→
(86 tags across 4 docs) - Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star) - Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows) - Simplified node labels for cleaner rendering Host fixes applied (live, not in repo — documented here for audit trail): - SSH keys standardized to exactly 2 approved keys on all 7 hosts (removed host-generated RSA keys, added missing VP TechOps key to tsys9) - Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5), net-tools (tsys1/6/7) - tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000, tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput - tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched; will apply on next Proxmox NFS remount/reboot) - noatime applied to root fs on tsys4 + tsys5 (was only tsys5) - tsys1 lldpd: installed but systemd/dbus issue prevents enabling ("Transport endpoint is not connected") — needs investigation/reboot Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh 💘 Generated with Crush Assisted-by: Crush:glm-5.2 --- perf/deploy-tuning.sh | 2 +- proxmox/docs/HOST-REFERENCE.md | 54 ++++----- proxmox/docs/NETWORK-TOPOLOGY.md | 162 +++++++++++++-------------- proxmox/docs/STORAGE-ARCHITECTURE.md | 82 +++++++------- proxmox/docs/VM-INVENTORY.md | 114 +++++++++---------- 5 files changed, 207 insertions(+), 207 deletions(-) diff --git a/perf/deploy-tuning.sh b/perf/deploy-tuning.sh index b1f82cb..3f21ed4 100755 --- a/perf/deploy-tuning.sh +++ b/perf/deploy-tuning.sh @@ -4,7 +4,7 @@ # Default mode is dry-run. Pass --apply to commit. Pass --no-nfs to skip NFS section. set -uo pipefail -SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/scripts/apply-tunings.sh" +SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")/scripts" && pwd)/apply-tunings.sh" SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=8 -o ServerAliveInterval=10 -o StrictHostKeyChecking=accept-new) MODE="" EXTRA_FLAGS="" diff --git a/proxmox/docs/HOST-REFERENCE.md b/proxmox/docs/HOST-REFERENCE.md index d2c525b..8ef23a2 100644 --- a/proxmox/docs/HOST-REFERENCE.md +++ b/proxmox/docs/HOST-REFERENCE.md @@ -13,17 +13,17 @@ ```mermaid graph TB - subgraph "RACK 5 — Core / Data / Infra" - T1["tsys1 — Infra
i7-4770 · 32GB · 9 VMs
57% RAM"] - T3["tsys3 — Compute
Xeon E3-1535M v5 · 32GB · 1 VM
10% RAM"] - T4["tsys4 — Storage+PBS
Xeon E3-1246 v3 · 16GB · 1 VM
15% RAM"] - T5["tsys5 — Storage+Sandbox
Xeon E5620 · 96GB · 4 VMs
15% RAM"] - T9["tsys9 — Infra+Compute
i5-10500 · 24GB · 6 VMs
69% RAM"] + subgraph "RACK 5 -- Core / Data / Infra" + T1["tsys1 -- Infra
i7-4770 | 32GB | 9 VMs
57% RAM"] + T3["tsys3 -- Compute
Xeon E3-1535M v5 | 32GB | 1 VM
10% RAM"] + T4["tsys4 -- Storage+PBS
Xeon E3-1246 v3 | 16GB | 1 VM
15% RAM"] + T5["tsys5 -- Storage+Sandbox
Xeon E5620 | 96GB | 4 VMs
15% RAM"] + T9["tsys9 -- Infra+Compute
i5-10500 | 24GB | 6 VMs
69% RAM"] end - subgraph "RACK 3 — Compute / RackRental" - T6["tsys6 — RackRental
2× Xeon E5530 · 126GB · 2 VMs
12% RAM"] - T7["tsys7 — RackRental
2× Xeon E5-2630 v2 · 189GB · 5 VMs
7% RAM"] + subgraph "RACK 3 -- Compute / RackRental" + T6["tsys6 -- RackRental
2x Xeon E5530 | 126GB | 2 VMs
12% RAM"] + T7["tsys7 -- RackRental
2x Xeon E5-2630 v2 | 189GB | 5 VMs
7% RAM"] end ``` @@ -33,15 +33,15 @@ graph TB | tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% | | tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% | | tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% | -| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% | -| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% | +| tsys6 | 2x Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% | +| tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% | | tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% | **Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired) --- -## pfv-tsys1 — Infrastructure host +## pfv-tsys1 -- Infrastructure host | Attribute | Value | |-----------|-------| @@ -60,7 +60,7 @@ graph TB --- -## pfv-tsys3 — Compute host +## pfv-tsys3 -- Compute host | Attribute | Value | |-----------|-------| @@ -70,14 +70,14 @@ graph TB | **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) | | **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) | | **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) | -| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) — **best existing local storage** | +| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) -- **best existing local storage** | | **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | | **Bond** | None (single USB NIC for storage) | | **Running VMs** | 313 | --- -## pfv-tsys4 — Storage server + PBS +## pfv-tsys4 -- Storage server + PBS | Attribute | Value | |-----------|-------| @@ -99,7 +99,7 @@ graph TB --- -## pfv-tsys5 — Storage server + sandboxes +## pfv-tsys5 -- Storage server + sandboxes | Attribute | Value | |-----------|-------| @@ -107,7 +107,7 @@ graph TB | **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM | | **PVE** | 9.2.5 / kernel 7.0.14-6-pve | | **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) | -| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** — nic2 DOWN) | +| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** -- nic2 DOWN) | | **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) | | **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) | | **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) | @@ -119,36 +119,36 @@ graph TB --- -## pfv-tsys6 — RackRental host +## pfv-tsys6 -- RackRental host | Attribute | Value | |-----------|-------| | **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX | -| **Hardware** | 2× Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM | +| **Hardware** | 2x Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM | | **PVE** | 9.2.5 / kernel 7.0.14-6-pve | | **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) | | **Tailscale** | 100.73.35.111 | | **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) | | **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | -| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) | +| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) | | **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | | **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | | **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) | -| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 — unusable for scratch. | +| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 -- unusable for scratch. | --- -## pfv-tsys7 — RackRental host +## pfv-tsys7 -- RackRental host | Attribute | Value | |-----------|-------| | **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 | -| **Hardware** | 2× Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM | +| **Hardware** | 2x Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM | | **PVE** | 9.2.5 / kernel 7.0.14-6-pve | | **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) | | **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) | | **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) | -| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s — do not use for VMs**) | +| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) | | **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | | **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) | | **Running VMs** | 701, 702, 703, 704, 705 | @@ -156,7 +156,7 @@ graph TB --- -## pfv-tsys9 — Infra + Compute +## pfv-tsys9 -- Infra + Compute | Attribute | Value | |-----------|-------| @@ -166,8 +166,8 @@ graph TB | **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) | | **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) | | **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) | -| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) — **excellent for scratch** | +| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) -- **excellent for scratch** | | **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host | | **Bond** | None (single USB NIC for storage) | | **Running VMs** | 901, 902, 903, 904, 905, 906 | -| **Notes** | Newest CPU (best IPC in fleet). **69% RAM — approaching 80% goal.** Target for compute growth (after siem migrates off). | +| **Notes** | Newest CPU (best IPC in fleet). **69% RAM -- approaching 80% goal.** Target for compute growth (after siem migrates off). | diff --git a/proxmox/docs/NETWORK-TOPOLOGY.md b/proxmox/docs/NETWORK-TOPOLOGY.md index 29fedd3..01ee9c0 100644 --- a/proxmox/docs/NETWORK-TOPOLOGY.md +++ b/proxmox/docs/NETWORK-TOPOLOGY.md @@ -2,7 +2,7 @@ > **Complete physical + logical network reference for the PFVCluster.** > Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct -> host probes. All data is live ground truth — not documentation inherited. +> host probes. All data is live ground truth -- not documentation inherited. > > Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), > [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). @@ -14,31 +14,31 @@ ```mermaid graph TB - subgraph "RACK 5 — Core / Data / Infra" - CORE["pfv-core-sw01
PowerConnect 5448
192.168.0.12
FW 2.0.0.46
(mgmt + storage VLANs)"] + subgraph "RACK 5 - Core / Data / Infra" + CORE["pfv-core-sw01
PowerConnect 5448
192.168.0.12
(mgmt + storage VLANs)"] - TSYS1["pfv-tsys1
i7-4770 Haswell
Infra host"] - TSYS3["pfv-tsys3
Xeon E3-1535M v5
Compute"] - TSYS4["pfv-tsys4
Xeon E3-1246 v3
NFS Storage + PBS"] - TSYS5["pfv-tsys5
Xeon E5620 Westmere
NFS Storage + Sandboxes"] - TSYS9["pfv-tsys9
i5-10500 Comet Lake
Infra + Compute"] + TSYS1["pfv-tsys1
i7-4770 Haswell
Infra host"] + TSYS3["pfv-tsys3
Xeon E3-1535M v5
Compute"] + TSYS4["pfv-tsys4
Xeon E3-1246 v3
NFS Storage + PBS"] + TSYS5["pfv-tsys5
Xeon E5620 Westmere
NFS Storage + Sandboxes"] + TSYS9["pfv-tsys9
i5-10500 Comet Lake
Infra + Compute"] - TSYS4_CONSOLE["conman + ser2net
7 switch consoles
(Tailscale 100.70.77.93:7890)"] - TSYS1_PDU["Cyclades PM10i PDU
powerman on tsys1
(port 10101)"] - TSYS1_UPS["APC Smart-UPS C 1500
NUT on tsys1
(port 3493)"] + TSYS4_CONSOLE["conman + ser2net
7 switch consoles
Tailscale 100.70.77.93:7890"] + TSYS1_PDU["Cyclades PM10i PDU
powerman on tsys1"] + TSYS1_UPS["APC Smart-UPS C 1500
NUT on tsys1"] end - subgraph "RACK 3 — Compute / RackRental" - TOR3STOR["pfv-tor3-stor
PowerConnect 5324 (Neyland 24T)
192.168.0.9
FW 2.0.1.4
(storage VLAN)"] - TOR3MGMT["pfv-tor3-mgmt
PowerConnect 5324 (Neyland 24T)
192.168.0.7
(mgmt VLAN)"] + subgraph "RACK 3 - Compute / RackRental" + TOR3STOR["pfv-tor3-stor
PowerConnect 5324
192.168.0.9
storage VLAN"] + TOR3MGMT["pfv-tor3-mgmt
PowerConnect 5324
192.168.0.7
mgmt VLAN"] - TSYS6["pfv-tsys6
2× Xeon E5530 Nehalem
RackRental"] - TSYS7["pfv-tsys7
2× Xeon E5-2630 v2 Ivy Bridge
RackRental"] + TSYS6["pfv-tsys6
2x Xeon E5530 Nehalem
RackRental"] + TSYS7["pfv-tsys7
2x Xeon E5-2630 v2
RackRental"] end - %% Cross-rack trunk - CORE <--. "ch1 LAG: 3×1G active
(g13,g14,g15)
hash=layer-2-3
mode=on (static)" .--> TOR3STOR - TOR3STOR -. "ch1: 4×1G
(g20,g21,g22,g23)
hash=layer-2-3-4
mode=on (static)" .-> CORE + %% Cross-rack trunk (two one-way dotted arrows with labels) + CORE -. "ch1: 3x1G active (g13-15)
hash=layer-2-3
mode=on" .-> TOR3STOR + TOR3STOR -. "ch1: 4x1G (g20-23)
hash=layer-2-3-4
mode=on" .-> CORE %% Rack 5 connections to core CORE --> TSYS1 @@ -56,7 +56,7 @@ graph TB ### How to read this diagram -The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** — +The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** -- the most critical link in the cluster. It carries all NFS storage traffic between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). The trunk is a 4-port static LAG, but only 3 links are active due to a cable @@ -70,7 +70,7 @@ mismatch (see §3 below). |------|--------|---------|----------------| | **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) | | **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) | -| — (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs | +| -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs | **Management IPs (VLAN 1, via DHCP reservations):** @@ -109,7 +109,7 @@ mismatch (see §3 below). --- -## 3. Cross-rack trunk detail (ch1) — the critical link +## 3. Cross-rack trunk detail (ch1) -- the critical link This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and rack 5 (tsys4/5). It is the single most important network path in the cluster. @@ -117,18 +117,18 @@ rack 5 (tsys4/5). It is the single most important network path in the cluster. ```mermaid graph LR subgraph "core-sw01 (rack5)" - G13["g13
1G UP"] - G14["g14
1G UP"] - G15["g15
1G UP"] - G16["g16
1G UP
NOT in LAG!"] - G17["g17
DOWN
in config but
no cable"] + G13["g13
1G UP"] + G14["g14
1G UP"] + G15["g15
1G UP"] + G16["g16
1G UP
NOT in LAG!"] + G17["g17
DOWN
in config but
no cable"] end subgraph "tor3-stor (rack3)" - G20["g20
1G UP"] - G21["g21
1G UP"] - G22["g22
1G UP"] - G23["g23
1G UP"] + G20["g20
1G UP"] + G21["g21
1G UP"] + G22["g22
1G UP"] + G23["g23
1G UP"] end G13 --- G20 @@ -142,10 +142,10 @@ graph LR | Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? | |----------|-------------------|--------------------|--------| -| Physical ports configured | g(13-15,17) | g(20-23) | **No — asymmetric** | -| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | — | -| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No — 3 vs 4** | -| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No — asymmetric** | +| Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** | +| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- | +| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** | +| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** | | Channel mode | on (static) | on (static) | Yes (both should be LACP) | | VLAN | access vlan 1000 | access vlan 1000 | Yes | @@ -162,19 +162,19 @@ graph LR one link** regardless of how many TCP connections (nconnect) are used. 3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure - detection — a dead cable stays in the LAG until manually discovered. + detection -- a dead cable stays in the LAG until manually discovered. --- ## 4. Complete switch port map -### pfv-core-sw01 (PowerConnect 5448, 48 ports) — rack 5 +### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5 | Port | Speed | Status | VLAN | Description / Connected device | |------|-------|--------|------|-------------------------------| | g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) | | g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) | -| g5 | — | UP | 1 | pfv-tsys1 (mgmt) | +| g5 | -- | UP | 1 | pfv-tsys1 (mgmt) | | g6 | 100 | UP | 1 | pfvsvrpi (secondary) | | g8 | 1000 | UP | 1 | pfv-tsys4-mgmt | | g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** | @@ -184,72 +184,72 @@ graph LR | **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** | | **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** | | **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** | -| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG — should be g17)** | -| **g17** | — | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** | +| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** | +| **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** | | g19 | 1000 | UP | 1 | pfv-tsys5-mgmt | | g21 | 1000 | UP | 1 | (unknown) | | g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) | | g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** | | g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) | | g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) | -| g32 | — | — | — | pfv-tsys4-storage (configured, DOWN) | -| g33 | — | — | — | pfv-tsys5-stor (ch2, DOWN/Not Present) | +| g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) | +| g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) | | g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) | | g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** | | g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) | | g41 | 1000 | UP | 1 | (unknown) | | g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) | | g44 | 1000 | UP | 1 | (unknown) | -| g45-g48 | — | DOWN | — | (Combo ports, unused) | +| g45-g48 | -- | DOWN | -- | (Combo ports, unused) | **Port-channels on core-sw01:** | Channel | Speed | Status | Members | VLAN | Notes | |---------|-------|--------|---------|------|-------| | ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor | -| ch2 | — | DOWN | g32-33 (non-candidate) | — | Unused | +| ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused | | ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) | **LAG hash:** `port-channel load-balance layer-2-3` -### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3 +### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 | Port | Speed | Status | VLAN | Description / Connected device | |------|-------|--------|------|-------------------------------| -| g1 | 1000 | UP | 1 | (unknown — uplink to tor3-mgmt or router) | +| g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) | | g2-g7 | 1000 | UP | 1 | (unused but UP) | | **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** | | **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** | | **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** | | **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** | -| **g13** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** | -| **g14** | **1000** | **UP** | **—** | **tsys8-stor (STALE — tsys8 retired)** | +| **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | +| **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | | g15-g19 | 1000 | UP | 1000 | (unused storage ports) | | **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** | | **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** | | **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** | | **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** | -| g24 | — | DOWN | 1 | (spanning-tree disabled, portfast) | +| g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) | **Port-channels on tor3-stor:** | Channel | Speed | Status | Members | VLAN | Notes | |---------|-------|--------|---------|------|-------| -| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4×1G) | -| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2×1G LACP) | -| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2×1G LACP) | -| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE — retired host)** | +| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) | +| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) | +| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) | +| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** | **LAG hash:** `port-channel load-balance layer-2-3-4` -### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3 +### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 | Port | Speed | Status | Connected device | |------|-------|--------|-----------------| -| g3 | — | UP | tsys7 (mgmt nic0) | -| g7 | — | UP | tsys6 (mgmt nic0) | +| g3 | -- | UP | tsys7 (mgmt nic0) | +| g7 | -- | UP | tsys6 (mgmt nic0) | -(Full port map not captured — this switch carries only VLAN 1 management +(Full port map not captured -- this switch carries only VLAN 1 management traffic. Low priority for storage performance.) --- @@ -258,24 +258,24 @@ traffic. Low priority for storage performance.) | Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed | |------|----------|-------------|-------------|-------------|-------------|-----------|-------------------| -| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | — | 1G | -| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | — | 1G | +| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G | +| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G | | tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G | | tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** | | tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | | tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | -| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | — | 1G | +| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G | ### Key observations - **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2 active slaves). All NFS traffic from rack 3 uses these bonds. -- **tsys1/3/4/9 use USB dongles** for storage network — single 1G, no redundancy. -- **tsys5 bond0 is degraded** — nic2 is DOWN (no cable connected). Only nic1 is +- **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy. +- **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is active via core-sw01 ch3 (g34). This is the LACP problem the operator has been troubleshooting. - **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS), - tsys4/5 use layer2+3 (suboptimal — same src/dst IP pair always hashes to one + tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one link even with nconnect's multiple TCP connections). --- @@ -285,17 +285,17 @@ traffic. Low priority for storage performance.) ```mermaid graph TB subgraph "Storage Servers (rack 5)" - T4D2["tsys4 D2
WDC Red 3TB
Tier 2"] - T4D5["tsys4 D5
Hitachi 2TB
Tier 2"] - T4D1["tsys4 D1
WD 1TB USB
Tier 3"] - T4D3["tsys4 D3
Seagate 500GB USB
Tier 3"] - T5S1["tsys5 S1
IronWolf 1TB
Tier 2"] - T5S2["tsys5 S2
IronWolf 1TB
Tier 2"] - T5SSD["tsys5 SSD
Samsung 860 PRO
Tier 1 (93% full)"] + T4D2["tsys4 D2
WDC Red 3TB
Tier 2"] + T4D5["tsys4 D5
Hitachi 2TB
Tier 2"] + T4D1["tsys4 D1
WD 1TB USB
Tier 3"] + T4D3["tsys4 D3
Seagate 500GB USB
Tier 3"] + T5S1["tsys5 S1
IronWolf 1TB
Tier 2"] + T5S2["tsys5 S2
IronWolf 1TB
Tier 2"] + T5SSD["tsys5 SSD
Samsung 860 PRO
Tier 1 (93% full)"] end subgraph "VLAN 1000 storage network" - NFS["NFS v4.2
nconnect=4
rsize/wsize=1MB"] + NFS["NFS v4.2
nconnect=4
rsize/wsize=1MB"] end subgraph "NFS Clients" @@ -327,15 +327,15 @@ graph TB | Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | |--------|--------|-------|-------|-------|-------|-------|-------|-------| -| tsys4 | D1 | — | — | — | — | ✓ | ✓ | — | -| tsys4 | **D2** | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | -| tsys4 | D3 | — | — | — | ✓ | ✓ | ✓ | — | -| tsys4 | D5 | — | ✓ | — | ✓ | ✓ | ✓ | ✓ | -| tsys5 | S1 | ✓ | — | — | — | ✓ | ✓ | — | -| tsys5 | S2 | ✓ | ✓ | — | — | ✓ | ✓ | — | -| tsys5 | S3 | ✓ | — | — | — | ✓ | ✓ | ✓ | -| tsys5 | S4 | ✓ | ✓ | — | — | ✓ | ✓ | ✓ | -| tsys5 | T5-SSD | — | ✓ | — | — | ✓ | ✓ | ✓ | +| tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- | +| tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ | +| tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- | +| tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ | +| tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- | +| tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- | +| tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ | +| tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ | +| tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ | (✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.) @@ -350,7 +350,7 @@ tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.) | **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` | | **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) | | **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) | -| **Direct telnet** | `telnet pfv-tsys4 200X` — ONLY when conmand stopped (conflicts). Use conman instead. | +| **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. | ### Console port assignments diff --git a/proxmox/docs/STORAGE-ARCHITECTURE.md b/proxmox/docs/STORAGE-ARCHITECTURE.md index 7f80e5e..f284099 100644 --- a/proxmox/docs/STORAGE-ARCHITECTURE.md +++ b/proxmox/docs/STORAGE-ARCHITECTURE.md @@ -28,16 +28,16 @@ is acceptable. ## 2. Physical disk inventory -### tsys4 — NFS storage server + PBS (7 disks) +### tsys4 -- NFS storage server + PBS (7 disks) | Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used | |--------|-------|------|-----|-----------|-------|--------|------|-------|------| | sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% | | sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) | | sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) | -| sdd | WDC WD10EZRX | 1.0 TB | — | SATA | (pve boot/swap/lvm) | — | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk | +| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk | | sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% | -| sdf | WDC WD10EARS | 1.0 TB | — | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% | +| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% | | sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% | **Changes from previous audit:** @@ -46,25 +46,25 @@ is acceptable. - **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only. - **D7 removed:** Previously held DellOpenManageEnterprise. Now gone. -### tsys5 — NFS storage server + sandboxes (7 disks) +### tsys5 -- NFS storage server + sandboxes (7 disks) | Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used | |--------|-------|------|-----|-----------|-------|-------------|------|-------|------| -| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | — | -| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | — | **Tier 1** | PASSED | **NEW, not configured** | -| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | — | Tier 2 | PASSED | OS disk | +| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- | +| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** | +| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk | | sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** | | sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) | | sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) | -| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | — | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** | +| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** | **Changes from previous audit:** -- **NVMe added:** Crucial CT500P1 500GB. True NVMe — highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg. +- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg. - **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured. - **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports. - **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry. - **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm). -- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) — needs attention. +- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention. - **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model). --- @@ -73,21 +73,21 @@ is acceptable. ```mermaid graph LR - subgraph "tsys4 — NFS Server" - D2["D2: /mnt/tsys4/D2
WDC Red 3TB
Tier 2 · 7% used"] - D5["D5: /mnt/tsys4/D5
Hitachi 2TB
Tier 2 · 13% used"] - D1["D1: /mnt/tsys4/D1
WD 1TB USB
Tier 3 · 0% used"] - D3["D3: /mnt/tsys4/D3
Seagate 500GB USB
Tier 3 · 0% used"] + subgraph "tsys4 -- NFS Server" + D2["D2: /mnt/tsys4/D2
WDC Red 3TB
Tier 2 | 7% used"] + D5["D5: /mnt/tsys4/D5
Hitachi 2TB
Tier 2 | 13% used"] + D1["D1: /mnt/tsys4/D1
WD 1TB USB
Tier 3 | 0% used"] + D3["D3: /mnt/tsys4/D3
Seagate 500GB USB
Tier 3 | 0% used"] end - subgraph "tsys5 — NFS Server" - S1["S1: /mnt/pfv-tsys5/S1
IronWolf 1TB
Tier 2 · 11% used"] - S2["S2: /mnt/pfv-tsys5/S2
IronWolf 1TB
Tier 2 · 4% used"] - S3stale["S3: (stale export)
disk not mounted"] + subgraph "tsys5 -- NFS Server" + S1["S1: /mnt/pfv-tsys5/S1
IronWolf 1TB
Tier 2 | 11% used"] + S2["S2: /mnt/pfv-tsys5/S2
IronWolf 1TB
Tier 2 | 4% used"] + S3stale["S3: (stale export)
disk not mounted"] end - subgraph "All hosts — NFS Clients" - MNT["/mnt/pve/D1 D2 D3 D5
/mnt/pve/S1 S2 S3 S4
/mnt/pve/T5-SSD"] + subgraph "All hosts -- NFS Clients" + MNT["/mnt/pve/D1 D2 D3 D5
/mnt/pve/S1 S2 S3 S4
/mnt/pve/T5-SSD"] end D2 -->|NFS v4.2| MNT @@ -111,7 +111,7 @@ graph LR | tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | | tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) | | tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard | -| tsys5 | /mnt/pfv-tsys5/S4 | S4 | — | **broken** (disk gone, storage.cfg entry stale) | +| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) | > **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from > tsys4) are **missing nconnect=4**. All other client hosts have nconnect on @@ -122,27 +122,27 @@ graph LR ## 4. Per-spindle VM placement (running VMs only) > Shows every running VM's primary disk and the physical spindle it lives on. -> This is the IOPS distribution map — the key to storage balancing. +> This is the IOPS distribution map -- the key to storage balancing. ```mermaid graph TB subgraph "tsys4 spindles" - D2["D2 — WDC Red 3TB
★★★ 13 VMs — WORST HOTSPOT"] - D5["D5 — Hitachi 2TB
4 VMs"] - D1["D1 — WD 1TB USB
0 VMs (empty)"] - D3["D3 — Seagate 500GB USB
0 VMs (empty)"] - ALB["/mnt/albert — Hitachi 2TB
0 VMs (not in Proxmox)"] + D2["D2 -- WDC Red 3TB
*** 13 VMs -- WORST HOTSPOT"] + D5["D5 -- Hitachi 2TB
4 VMs"] + D1["D1 -- WD 1TB USB
0 VMs (empty)"] + D3["D3 -- Seagate 500GB USB
0 VMs (empty)"] + ALB["/mnt/albert -- Hitachi 2TB
0 VMs (not in Proxmox)"] end subgraph "tsys5 spindles" - S1["S1 — IronWolf 1TB
1 NFS VM + 12 local stopped"] - S2["S2 — IronWolf 1TB
5 VMs"] - SSD["SSD — Samsung 860 PRO
1 VM (93% full!)"] - NVME["NVMe — Crucial CT500P1
NOT CONFIGURED"] - HYNIX["SK hynix SSD 512GB
NOT CONFIGURED"] + S1["S1 -- IronWolf 1TB
1 NFS VM + 12 local stopped"] + S2["S2 -- IronWolf 1TB
5 VMs"] + SSD["SSD -- Samsung 860 PRO
1 VM (93% full!)"] + NVME["NVMe -- Crucial CT500P1
NOT CONFIGURED"] + HYNIX["SK hynix SSD 512GB
NOT CONFIGURED"] end - D2 --- D2vms["100 101 104 105 108
114 117 509 600 702
704 901 904"] + D2 --- D2vms["100 101 104 105 108
114 117 509 600 702
704 901 904"] D5 --- D5vms["106 313 600(d0) 903"] S1 --- S1vms["702 hfnoc-uisp"] S2 --- S2vms["103 703 705 902 905"] @@ -158,11 +158,11 @@ graph TB | S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod | | S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate | | ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** | -| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | — | Empty | -| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | — | Empty | -| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | — | Not in Proxmox | -| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** | -| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** | +| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty | +| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty | +| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox | +| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | +| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** | ### Local-only VM disks (not on NFS) @@ -188,8 +188,8 @@ graph TB | S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% | | S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% | | SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** | -| NVMe (CT500P1) | 466 GB | — | — | Not configured | -| SK hynix SSD | 477 GB | — | — | Not configured | +| NVMe (CT500P1) | 466 GB | -- | -- | Not configured | +| SK hynix SSD | 477 GB | -- | -- | Not configured | **Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2 **Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup diff --git a/proxmox/docs/VM-INVENTORY.md b/proxmox/docs/VM-INVENTORY.md index 6a6c664..3f72551 100644 --- a/proxmox/docs/VM-INVENTORY.md +++ b/proxmox/docs/VM-INVENTORY.md @@ -13,53 +13,53 @@ ```mermaid graph TB - subgraph tsys1["tsys1 — Infra (57% RAM)"] - VM100["100 pfv-bms
2c/4G D2"] - VM101["101 tsys-ca
2c/2G D2"] - VM103["103 netinfra-01
2c/4G S2"] - VM104["104 librenms
2c/2G D2"] - VM105["105 proxmox-dc
2c/2G D2"] - VM106["106 cnode3
4c/4G D5"] - VM108["108 ucs-01
2c/8G D2"] - VM114["114 kali-tsys
2c/2G D2"] - VM117["117 workbench
2c/4G D2"] + subgraph tsys1["tsys1 -- Infra (57% RAM)"] + VM100["100 pfv-bms
2c/4G D2"] + VM101["101 tsys-ca
2c/2G D2"] + VM103["103 netinfra-01
2c/4G S2"] + VM104["104 librenms
2c/2G D2"] + VM105["105 proxmox-dc
2c/2G D2"] + VM106["106 cnode3
4c/4G D5"] + VM108["108 ucs-01
2c/8G D2"] + VM114["114 kali-tsys
2c/2G D2"] + VM117["117 workbench
2c/4G D2"] end - subgraph tsys3["tsys3 — Compute (10% RAM)"] - VM313["313 wnode-tsys3
8c/28G D5"] + subgraph tsys3["tsys3 -- Compute (10% RAM)"] + VM313["313 wnode-tsys3
8c/28G D5"] end - subgraph tsys4["tsys4 — Storage (15% RAM)"] - VM400["400 PBS
2c/2G local-lvm"] + subgraph tsys4["tsys4 -- Storage (15% RAM)"] + VM400["400 PBS
2c/2G local-lvm"] end - subgraph tsys5["tsys5 — Storage+Sandbox (15% RAM)"] - VM509["509 wnode-tsys5
2c/32G D2"] - VM515["515 hfnoc-uisp-preprod
2c/2G local-nonprod"] - VM5111["5111 ultix-streaming
4c/9G SSD"] - VM5112["5112 ultix-offstage
4c/6G local-lvm"] + subgraph tsys5["tsys5 -- Storage+Sandbox (15% RAM)"] + VM509["509 wnode-tsys5
2c/32G D2"] + VM515["515 hfnoc-uisp-preprod
2c/2G local-nonprod"] + VM5111["5111 ultix-streaming
4c/9G SSD"] + VM5112["5112 ultix-offstage
4c/6G local-lvm"] end - subgraph tsys6["tsys6 — RackRental (12% RAM)"] - VM100b["100 wnode-tsys6
2c/64G S4"] - VM600["600 tsys-awx
2c/12G D5"] + subgraph tsys6["tsys6 -- RackRental (12% RAM)"] + VM100b["100 wnode-tsys6
2c/64G S4"] + VM600["600 tsys-awx
2c/12G D5"] end - subgraph tsys7["tsys7 — RackRental (7% RAM)"] - VM701["701 wnode-tsys7
4c/32G S3"] - VM702["702 hfnoc-uisp
2c/8G S1"] - VM703["703 rr-middleware
2c/2G S2"] - VM704["704 TCTC
4c/6G D2"] - VM705["705 cnode2
4c/4G S2"] + subgraph tsys7["tsys7 -- RackRental (7% RAM)"] + VM701["701 wnode-tsys7
4c/32G S3"] + VM702["702 hfnoc-uisp
2c/8G S1"] + VM703["703 rr-middleware
2c/2G S2"] + VM704["704 TCTC
4c/6G D2"] + VM705["705 cnode2
4c/4G S2"] end - subgraph tsys9["tsys9 — Infra+Compute (69% RAM)"] - VM901["901 tsys-siem
2c/8G D2"] - VM902["902 ucs-02
2c/8G S2"] - VM903["903 kali-rd
2c/2G D5"] - VM904["904 netinfra-02
2c/4G D2"] - VM905["905 wnode-tsys9
4c/4G S2"] - VM906["906 cnode1
2c/4G S3"] + subgraph tsys9["tsys9 -- Infra+Compute (69% RAM)"] + VM901["901 tsys-siem
2c/8G D2"] + VM902["902 ucs-02
2c/8G S2"] + VM903["903 kali-rd
2c/2G D5"] + VM904["904 netinfra-02
2c/4G D2"] + VM905["905 wnode-tsys9
4c/4G S2"] + VM906["906 cnode1
2c/4G S3"] end ``` @@ -71,14 +71,14 @@ graph TB | VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | |------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| -| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | — | — | ✓ | discard=on, ssd=1. HA for power/temp monitoring. | -| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | no nested-virt flag. Certificate Authority. | -| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | — | DNS primary (Technitium + Pi-hole). | -| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | SNMP monitoring for switches. | -| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | — | ✓ | ✓ | PDM (Proxmox Datacenter Manager). | +| 100 | pfv-bms (HomeAssistant) | tsys1 | 2 | host | 4G | D2 (raw) | D2 | raw | -- | -- | ✓ | discard=on, ssd=1. HA for power/temp monitoring. | +| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | no nested-virt flag. Certificate Authority. | +| 103 | pfv-netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | -- | DNS primary (Technitium + Pi-hole). | +| 104 | tsys-librenms | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | SNMP monitoring for switches. | +| 105 | tsys-proxmox-datacenter | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | -- | ✓ | ✓ | PDM (Proxmox Datacenter Manager). | | 108 | tsys-ucs-01 | tsys1 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Univention Corporate Server half 1. | -| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | — | ✓ | ✓ | UCS half 2 (HA pair with 108). | -| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | — | ✓ | — | DNS secondary. HA pair with 103. | +| 902 | tsys-ucs-02 | tsys9 | 2 | host | 8G | S2 | S2 | qcow2 | -- | ✓ | ✓ | UCS half 2 (HA pair with 108). | +| 904 | pfv-netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | D2 | qcow2 | -- | ✓ | -- | DNS secondary. HA pair with 103. | ### k8s control plane (cnode1/2/3) @@ -86,17 +86,17 @@ graph TB |------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| | 106 | pfv-k8s-cnode3 | tsys1 | 4 | host | 4G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | | 705 | pfv-k8s-cnode2 | tsys7 | 4 | host | 4G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | etcd member. | -| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | — | ✓ | — | etcd member. | +| 906 | pfv-k8s-cnode1 | tsys9 | 2 | host | 4G | S3 | S3 | qcow2 | -- | ✓ | -- | etcd member. | ### k8s workers (wnodes) | VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Spindle | Disk type | Cache | iothread | qga | Notes | |------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| | 313 | pfv-k8s-wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Compute worker. | -| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | Undersized vCPU (2). | -| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | — | ✓ | — | Undersized vCPU (2), high RAM. | -| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | — | ✓ | — | RackRental worker. | -| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | — | ✓ | ✓ | Target: grow to 6c/8G. | +| 509 | pfv-k8s-wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | Undersized vCPU (2). | +| 100 | pfv-k8s-wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | S4 | qcow2 | -- | ✓ | -- | Undersized vCPU (2), high RAM. | +| 701 | pfv-k8s-wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | S3 | qcow2 | -- | ✓ | -- | RackRental worker. | +| 905 | pfv-k8s-wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | S2 | qcow2 | -- | ✓ | ✓ | Target: grow to 6c/8G. | ### Application VMs @@ -104,14 +104,14 @@ graph TB |------|------|------|------|----------|-----|-----------|---------|-----------|-------|----------|-----|-------| | 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Kali Linux pentest. | | 117 | tsys-secure-workbench | tsys1 | 2 | host | 4G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | Secure workbench. | -| 400 | pfv-proxmox-backup-server | tsys4 | 2 | — | 2G | local-lvm | (local) | raw | — | — | — | PBS — all VM backups. | -| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | — | ✓ | — | Preprod. Locked to tsys5. | -| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | SSD dir | SSD | qcow2 | — | — | — | 288GB on Samsung 860 PRO (**93% full**). | -| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | (local) | raw | — | — | — | 288GB. Locked to tsys5. | +| 400 | pfv-proxmox-backup-server | tsys4 | 2 | -- | 2G | local-lvm | (local) | raw | -- | -- | -- | PBS -- all VM backups. | +| 515 | hfnoc-uisp-preprod | tsys5 | 2 | host | 2G | local-nonprod | S1 | qcow2 | -- | ✓ | -- | Preprod. Locked to tsys5. | +| 5111 | ultix-streaming | tsys5 | 4 | -- | 9G | SSD dir | SSD | qcow2 | -- | -- | -- | 288GB on Samsung 860 PRO (**93% full**). | +| 5112 | ultix-offstage | tsys5 | 4 | -- | 6G | local-lvm | (local) | raw | -- | -- | -- | 288GB. Locked to tsys5. | | 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Ansible AWX. | -| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | — | ✓ | ✓ | 100GB disk. UISP network monitoring. | +| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | S1 | qcow2 | -- | ✓ | ✓ | 100GB disk. UISP network monitoring. | | 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | S2 | qcow2 | **writethrough** | ✓ | ✓ | RackRental middleware. | -| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | — | ✓ | — | | +| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | D2 | qcow2 | -- | ✓ | -- | | | 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | D2 | qcow2 | **writethrough** | ✓ | ✓ | 132GB disk. SIEM. | | 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | D5 | qcow2 | **writethrough** | ✓ | ✓ | Kali R&D. | @@ -122,7 +122,7 @@ graph TB ### 3.1 Cache mode audit `cache=writethrough` forces synchronous writes to the NFS server on every write -operation. For NFS-backed qcow2, this is unnecessarily slow — `writeback` or +operation. For NFS-backed qcow2, this is unnecessarily slow -- `writeback` or `none` is recommended (qcow2 already provides journaling integrity). | Cache mode | Count | VMs | @@ -153,7 +153,7 @@ CPU generations. K8s cnodes/wnodes should ideally match within each tier. | Controller | Count | Notes | |-----------|-------|-------| | **scsi0** (virtio-scsi) | 22 | Optimal for Proxmox | -| **virtio0** | 4 | 101, 509, 701, 704 — also good | +| **virtio0** | 4 | 101, 509, 701, 704 -- also good | | raw (no virtio) | 2 | 100 (pfv-bms), 500 (stopped) | All running VMs use virtio-based disk controllers. No IDE/SATA legacy disks. @@ -167,7 +167,7 @@ No legacy e1000/rtl8139 adapters. | Status | Count | VMs without qga | |--------|-------|-----------------| -| **enabled** | 20 | — | +| **enabled** | 20 | -- | | **not set** | 8 | 101, 103, 509, 100(tsys6), 701, 704, 904, 906 | **Fix:** Enable `qm set --agent 1` on the 8 VMs missing it. No VM @@ -218,7 +218,7 @@ is immediate). | VMID | Name | Host | RAM | Disk | Notes | |------|------|------|-----|------|-------| | 102 | pfv-k8s-wnode-tsys1 | tsys1 | 4G | (no disk) | Stopped, no boot disk | -| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail — disk export removed** | +| 500 | DellOpenManageEnterprise | tsys5 | 20G | **D7 (gone!)** | **Would fail -- disk export removed** | | 5500 | RestoreTemplate | tsys5 | 2G | D2 | Template | | 1000 | KNELTemplate | tsys1 | 2G | D2 | Base template | | 3000 | KNELTemplate | tsys3 | 2G | D2 | Base template copy |