fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes

Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
2026-07-31 00:12:04 -05:00
parent 290245349f
commit 9c2882ea00
5 changed files with 207 additions and 207 deletions
+27 -27
View File
@@ -13,17 +13,17 @@
```mermaid
graph TB
subgraph "RACK 5 Core / Data / Infra"
T1["tsys1 Infra<br/>i7-4770 · 32GB · 9 VMs<br/>57% RAM"]
T3["tsys3 Compute<br/>Xeon E3-1535M v5 · 32GB · 1 VM<br/>10% RAM"]
T4["tsys4 Storage+PBS<br/>Xeon E3-1246 v3 · 16GB · 1 VM<br/>15% RAM"]
T5["tsys5 Storage+Sandbox<br/>Xeon E5620 · 96GB · 4 VMs<br/>15% RAM"]
T9["tsys9 Infra+Compute<br/>i5-10500 · 24GB · 6 VMs<br/>69% RAM"]
subgraph "RACK 5 -- Core / Data / Infra"
T1["tsys1 -- Infra<br>i7-4770 | 32GB | 9 VMs<br>57% RAM"]
T3["tsys3 -- Compute<br>Xeon E3-1535M v5 | 32GB | 1 VM<br>10% RAM"]
T4["tsys4 -- Storage+PBS<br>Xeon E3-1246 v3 | 16GB | 1 VM<br>15% RAM"]
T5["tsys5 -- Storage+Sandbox<br>Xeon E5620 | 96GB | 4 VMs<br>15% RAM"]
T9["tsys9 -- Infra+Compute<br>i5-10500 | 24GB | 6 VMs<br>69% RAM"]
end
subgraph "RACK 3 Compute / RackRental"
T6["tsys6 RackRental<br/>2× Xeon E5530 · 126GB · 2 VMs<br/>12% RAM"]
T7["tsys7 RackRental<br/>2× Xeon E5-2630 v2 · 189GB · 5 VMs<br/>7% RAM"]
subgraph "RACK 3 -- Compute / RackRental"
T6["tsys6 -- RackRental<br>2x Xeon E5530 | 126GB | 2 VMs<br>12% RAM"]
T7["tsys7 -- RackRental<br>2x Xeon E5-2630 v2 | 189GB | 5 VMs<br>7% RAM"]
end
```
@@ -33,15 +33,15 @@ graph TB
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 32 GB | 9.2.5 | 7.0.14 | Compute | 1 | 10% |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | **9.1.5** | **6.17.9** | Storage + PBS | 1 | 15% |
| tsys5 | Xeon E5620 Westmere (2010) | 8 | 96 GB | 9.2.5 | 7.0.14 | Storage + Sandbox | 4 | 15% |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% |
| tsys6 | 2x Xeon E5530 Nehalem (2009) | 16 | 126 GB | 9.2.5 | 7.0.14 | RackRental | 2 | 12% |
| tsys7 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 9.2.5 | 7.0.14 | RackRental | 5 | 7% |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 24 GB | 9.2.5 | 7.0.14 | Infra + Compute | 6 | 69% |
**Retired:** tsys2 (Win10, pending rebuild), tsys8 (permanently retired)
---
## pfv-tsys1 Infrastructure host
## pfv-tsys1 -- Infrastructure host
| Attribute | Value |
|-----------|-------|
@@ -60,7 +60,7 @@ graph TB
---
## pfv-tsys3 Compute host
## pfv-tsys3 -- Compute host
| Attribute | Value |
|-----------|-------|
@@ -70,14 +70,14 @@ graph TB
| **mgmt IP** | 192.168.2.5 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.3 (datanet via enx USB dongle, no bond) |
| **NFS mounts** | D2, D5, S2, S4, T5-SSD (all nconnect=4) |
| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) **best existing local storage** |
| **Local storage** | local-lvm (Samsung PM961 NVMe, 349 GB, 0% used) -- **best existing local storage** |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | None (single USB NIC for storage) |
| **Running VMs** | 313 |
---
## pfv-tsys4 Storage server + PBS
## pfv-tsys4 -- Storage server + PBS
| Attribute | Value |
|-----------|-------|
@@ -99,7 +99,7 @@ graph TB
---
## pfv-tsys5 Storage server + sandboxes
## pfv-tsys5 -- Storage server + sandboxes
| Attribute | Value |
|-----------|-------|
@@ -107,7 +107,7 @@ graph TB
| **Hardware** | Dell T7500, Xeon E5620 Westmere (2010) **single CPU** (CPU2 socket empty), 8 threads, 96 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.179 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** nic2 DOWN) |
| **Storage IP** | 10.100.100.5 (storagenet via bond0, **1 of 2 slaves active** -- nic2 DOWN) |
| **NFS mounts** | D2, D3, D5 from tsys4 (**missing nconnect**) |
| **NFS exports** | S1 (IronWolf 1TB), S2 (IronWolf 1TB), S3 (stale), SSD (Samsung 860 PRO) |
| **Local storage** | local-lvm (Hitachi 2TB), local-nonprod (=S1), SSD dir (Samsung 860 PRO 256GB) |
@@ -119,36 +119,36 @@ graph TB
---
## pfv-tsys6 RackRental host
## pfv-tsys6 -- RackRental host
| Attribute | Value |
|-----------|-------|
| **Role** | RackRental: k8s wnode-tsys6 (64 GB RAM) + AWX |
| **Hardware** | 2× Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM |
| **Hardware** | 2x Xeon E5530 Nehalem (2009), 16 threads, 126 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.169 (vmbr0, DHCP) |
| **Tailscale** | 100.73.35.111 |
| **Storage IP** | 10.100.100.6 (storagenet via bond0, **2 of 2 slaves active**, 2G LACP) |
| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s do not use for VMs**) |
| **Local storage** | local-lvm (WD My Passport USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
| **Running VMs** | 100 (wnode-tsys6, 64GB), 600 (tsys-awx) |
| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 unusable for scratch. |
| **Notes** | Best storage bond in fleet (2G LACP). Local storage is USB 2.0 -- unusable for scratch. |
---
## pfv-tsys7 RackRental host
## pfv-tsys7 -- RackRental host
| Attribute | Value |
|-----------|-------|
| **Role** | RackRental: k8s wnode-tsys7 + hfnoc-uisp + rr-middleware + TCTC + k8s cnode2 |
| **Hardware** | 2× Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM |
| **Hardware** | 2x Xeon E5-2630 v2 Ivy Bridge (2013), 24 threads, 189 GB RAM |
| **PVE** | 9.2.5 / kernel 7.0.14-6-pve |
| **mgmt IP** | 192.168.3.55 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.7 (datanet via bond0, **2 of 2 slaves active**, 2G LACP) |
| **NFS mounts** | D1, D2, D3, D5, S1, S2, S3, S4, T5-SSD (9 mounts, all nconnect=4) |
| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s do not use for VMs**) |
| **Local storage** | local-lvm (WD portable USB 2.0 HDD, 1.7 TB, **30 MB/s -- do not use for VMs**) |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | bond0 (2 of 2 slaves), **hash=layer3+4** (correct) |
| **Running VMs** | 701, 702, 703, 704, 705 |
@@ -156,7 +156,7 @@ graph TB
---
## pfv-tsys9 Infra + Compute
## pfv-tsys9 -- Infra + Compute
| Attribute | Value |
|-----------|-------|
@@ -166,8 +166,8 @@ graph TB
| **mgmt IP** | 192.168.3.58 (vmbr0, DHCP) |
| **Storage IP** | 10.100.100.9 (datanet via enx9c69d36a5b6c USB dongle, no bond) |
| **NFS mounts** | D2, D5, S2, S3, S4, T5-SSD (all nconnect=4) |
| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) **excellent for scratch** |
| **Local storage** | local-lvm (PNY CS900 SSD, 136 GB, 0% used) -- **excellent for scratch** |
| **Tuning** | governor=performance, swappiness=10, bbr, rmem/wmem=128MB, tuned=virtual-host |
| **Bond** | None (single USB NIC for storage) |
| **Running VMs** | 901, 902, 903, 904, 905, 906 |
| **Notes** | Newest CPU (best IPC in fleet). **69% RAM approaching 80% goal.** Target for compute growth (after siem migrates off). |
| **Notes** | Newest CPU (best IPC in fleet). **69% RAM -- approaching 80% goal.** Target for compute growth (after siem migrates off). |