KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes

Browse Source 
Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-31 00:12:04 -05:00
parent 290245349f
commit 9c2882ea00
5 changed files with 207 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/STORAGE-ARCHITECTURE.md
+41 -41
View File
@@ -28,16 +28,16 @@ is acceptable.
## 2. Physical disk inventory
### tsys4 — NFS storage server + PBS (7 disks)
### tsys4 -- NFS storage server + PBS (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|--------|------|-------|------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | 7200 | SATA | /mnt/albert | (not exported) | **Tier 2** | PASSED, 0 reallocated | 0% |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | 7200 | SATA | /mnt/tsys4/D5 | **D5** | Tier 2 | PASSED, 0 reallocated | 13% (220 GB) |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | 5400 | SATA | /mnt/tsys4/D2 | **D2** | Tier 2 | PASSED, 0 reallocated | 7% (170 GB) |
| sdd | WDC WD10EZRX | 1.0 TB | — | SATA | (pve boot/swap/lvm) | — | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sdd | WDC WD10EZRX | 1.0 TB | -- | SATA | (pve boot/swap/lvm) | -- | Tier 2 | PASSED, 1 reallocated (sdf has 1) | OS disk |
| sde | ST500LM021 (Seagate) | 500 GB | 7200 | **USB** | /mnt/tsys4/D3 | **D3** | **Tier 3** | PASSED | 0% |
| sdf | WDC WD10EARS | 1.0 TB | — | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
| sdf | WDC WD10EARS | 1.0 TB | -- | **USB** | /mnt/tsys4/D1 | **D1** | **Tier 3** | PASSED, 1 reallocated sector | 0% |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | 5400 | **USB** | /mnt/backup | (not exported) | **Tier 3** | PASSED | 0% |
**Changes from previous audit:**
@@ -46,25 +46,25 @@ is acceptable.
- **/mnt/backup added:** WD 5TB USB easystore. Bulk backup only.
- **D7 removed:** Previously held DellOpenManageEnterprise. Now gone.
### tsys5 — NFS storage server + sandboxes (7 disks)
### tsys5 -- NFS storage server + sandboxes (7 disks)
| Device | Model | Size | RPM | Interface | Mount | NFS/Local ID | Tier | SMART | Used |
|--------|-------|------|-----|-----------|-------|-------------|------|-------|------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | — |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | — | **Tier 1** | PASSED | **NEW, not configured** |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | — | Tier 2 | PASSED | OS disk |
| sda | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SATA | **(not mounted)** | was S3 | Tier 2 | PASSED | -- |
| sdb | SK hynix SC300B | 512 GB | SSD | SATA | **(not mounted)** | -- | **Tier 1** | PASSED | **NEW, not configured** |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | 7200 | SAS | (pve boot/swap/lvm) | -- | Tier 2 | PASSED | OS disk |
| sdd | Samsung SSD 860 PRO | 256 GB | SSD | SAS | /mnt/pfv-tsys5/ssd | `ssd` / `SSD` dir (local) | **Tier 1** | PASSED | **93% full!** |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S1 | **S1** + `local-nonprod` | Tier 2 | PASSED | 11% (96 GB) |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | 5900 | SAS | /mnt/pfv-tsys5/S2 | **S2** | Tier 2 | PASSED | 4% (32 GB) |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | — | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **PCIe** | **(not mounted)** | -- | **Tier 1** | PASSED, 0 integrity errors, 34K hrs | **NEW, not configured** |
**Changes from previous audit:**
- **NVMe added:** Crucial CT500P1 500GB. True NVMe — highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- **NVMe added:** Crucial CT500P1 500GB. True NVMe -- highest-performance local storage in the fleet. Not mounted, not in fstab, not in storage.cfg.
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal SATA. Not mounted or configured.
- **S3 unmounted:** sda (was S3) is present but fstab entry is commented out. Stale export remains in /etc/exports.
- **S4 gone:** Old Toshiba 458GB no longer present. fstab entry commented. storage.cfg has broken `dir: S4` entry.
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (5900 rpm).
- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) — needs attention.
- **SSD nearly full:** Samsung 860 PRO at 93% (206/234 GB) -- needs attention.
- **Boot disk changed:** HITACHI HUS724020ALA640 2TB (was Hitachi 2TB in old audit, now confirmed model).
---
@@ -73,21 +73,21 @@ is acceptable.
```mermaid
graph LR
subgraph "tsys4 — NFS Server"
D2["D2: /mnt/tsys4/D2<br/>WDC Red 3TB<br/>Tier 2 · 7% used"]
D5["D5: /mnt/tsys4/D5<br/>Hitachi 2TB<br/>Tier 2 · 13% used"]
D1["D1: /mnt/tsys4/D1<br/>WD 1TB USB<br/>Tier 3 · 0% used"]
D3["D3: /mnt/tsys4/D3<br/>Seagate 500GB USB<br/>Tier 3 · 0% used"]
subgraph "tsys4 -- NFS Server"
D2["D2: /mnt/tsys4/D2<br>WDC Red 3TB<br>Tier 2 | 7% used"]
D5["D5: /mnt/tsys4/D5<br>Hitachi 2TB<br>Tier 2 | 13% used"]
D1["D1: /mnt/tsys4/D1<br>WD 1TB USB<br>Tier 3 | 0% used"]
D3["D3: /mnt/tsys4/D3<br>Seagate 500GB USB<br>Tier 3 | 0% used"]
end
subgraph "tsys5 — NFS Server"
S1["S1: /mnt/pfv-tsys5/S1<br/>IronWolf 1TB<br/>Tier 2 · 11% used"]
S2["S2: /mnt/pfv-tsys5/S2<br/>IronWolf 1TB<br/>Tier 2 · 4% used"]
S3stale["S3: (stale export)<br/>disk not mounted"]
subgraph "tsys5 -- NFS Server"
S1["S1: /mnt/pfv-tsys5/S1<br>IronWolf 1TB<br>Tier 2 | 11% used"]
S2["S2: /mnt/pfv-tsys5/S2<br>IronWolf 1TB<br>Tier 2 | 4% used"]
S3stale["S3: (stale export)<br>disk not mounted"]
end
subgraph "All hosts — NFS Clients"
MNT["/mnt/pve/D1 D2 D3 D5<br/>/mnt/pve/S1 S2 S3 S4<br/>/mnt/pve/T5-SSD"]
subgraph "All hosts -- NFS Clients"
MNT["/mnt/pve/D1 D2 D3 D5<br>/mnt/pve/S1 S2 S3 S4<br>/mnt/pve/T5-SSD"]
end
D2 -->|NFS v4.2| MNT
@@ -111,7 +111,7 @@ graph LR
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | NFSv4.2 | **stale** (disk not mounted on server) |
| tsys5 | /mnt/pfv-tsys5/ssd | T5-SSD | NFSv4.2 | nconnect=4,noatime,rsize=1048576,wsize=1048576,hard |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | — | **broken** (disk gone, storage.cfg entry stale) |
| tsys5 | /mnt/pfv-tsys5/S4 | S4 | -- | **broken** (disk gone, storage.cfg entry stale) |
> **Note:** tsys5 is the only host where NFS client mounts (D2/D3/D5 from
> tsys4) are **missing nconnect=4**. All other client hosts have nconnect on
@@ -122,27 +122,27 @@ graph LR
## 4. Per-spindle VM placement (running VMs only)
> Shows every running VM's primary disk and the physical spindle it lives on.
> This is the IOPS distribution map — the key to storage balancing.
> This is the IOPS distribution map -- the key to storage balancing.
```mermaid
graph TB
subgraph "tsys4 spindles"
D2["D2 — WDC Red 3TB<br/>★★★ 13 VMs — WORST HOTSPOT"]
D5["D5 — Hitachi 2TB<br/>4 VMs"]
D1["D1 — WD 1TB USB<br/>0 VMs (empty)"]
D3["D3 — Seagate 500GB USB<br/>0 VMs (empty)"]
ALB["/mnt/albert — Hitachi 2TB<br/>0 VMs (not in Proxmox)"]
D2["D2 -- WDC Red 3TB<br>*** 13 VMs -- WORST HOTSPOT"]
D5["D5 -- Hitachi 2TB<br>4 VMs"]
D1["D1 -- WD 1TB USB<br>0 VMs (empty)"]
D3["D3 -- Seagate 500GB USB<br>0 VMs (empty)"]
ALB["/mnt/albert -- Hitachi 2TB<br>0 VMs (not in Proxmox)"]
end
subgraph "tsys5 spindles"
S1["S1 — IronWolf 1TB<br/>1 NFS VM + 12 local stopped"]
S2["S2 — IronWolf 1TB<br/>5 VMs"]
SSD["SSD — Samsung 860 PRO<br/>1 VM (93% full!)"]
NVME["NVMe — Crucial CT500P1<br/>NOT CONFIGURED"]
HYNIX["SK hynix SSD 512GB<br/>NOT CONFIGURED"]
S1["S1 -- IronWolf 1TB<br>1 NFS VM + 12 local stopped"]
S2["S2 -- IronWolf 1TB<br>5 VMs"]
SSD["SSD -- Samsung 860 PRO<br>1 VM (93% full!)"]
NVME["NVMe -- Crucial CT500P1<br>NOT CONFIGURED"]
HYNIX["SK hynix SSD 512GB<br>NOT CONFIGURED"]
end
D2 --- D2vms["100 101 104 105 108<br/>114 117 509 600 702<br/>704 901 904"]
D2 --- D2vms["100 101 104 105 108<br>114 117 509 600 702<br>704 901 904"]
D5 --- D5vms["106 313 600(d0) 903"]
S1 --- S1vms["702 hfnoc-uisp"]
S2 --- S2vms["103 703 705 902 905"]
@@ -158,11 +158,11 @@ graph TB
| S1 (IronWolf 1TB) | tsys5 | S1 + local-nonprod | Tier 2 | 1 (NFS) + 0 (local, 12 stopped) | 702 | Low; 12 stopped sandboxes on local-nonprod |
| S2 (IronWolf 1TB) | tsys5 | S2 | Tier 2 | 5 | 103,703,705,902,905 | Moderate |
| ssd (Samsung 860 PRO) | tsys5 | SSD dir | Tier 1 | 1 | 5111 | **93% full** |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | — | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | — | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | — | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | — | **Not configured** |
| D1 (WD 1TB USB) | tsys4 | D1 | Tier 3 | 0 | -- | Empty |
| D3 (Seagate 500GB USB) | tsys4 | D3 | Tier 3 | 0 | -- | Empty |
| /mnt/albert (Hitachi 2TB) | tsys4 | (none) | Tier 2 | 0 | -- | Not in Proxmox |
| NVMe (CT500P1 500GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
| SK hynix SSD (512GB) | tsys5 | (none) | Tier 1 | 0 | -- | **Not configured** |
### Local-only VM disks (not on NFS)
@@ -188,8 +188,8 @@ graph TB
| S1 (IronWolf 1TB) | 916 GB | 96 GB | 774 GB | 11% |
| S2 (IronWolf 1TB) | 916 GB | 32 GB | 838 GB | 4% |
| SSD (Samsung 860 PRO) | 234 GB | 206 GB | **16 GB** | **93%** |
| NVMe (CT500P1) | 466 GB | — | — | Not configured |
| SK hynix SSD | 477 GB | — | — | Not configured |
| NVMe (CT500P1) | 466 GB | -- | -- | Not configured |
| SK hynix SSD | 477 GB | -- | -- | Not configured |
**Total Tier 2 available (NFS production):** ~5.5 TB unused across D2/D5/S1/S2
**Total Tier 3 available (USB):** ~6 TB unused across D1/D3/backup