KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9c2882ea00bd99c9ec100b88a34f961e5e7b2ba0
PFVCluster/proxmox/docs/NETWORK-TOPOLOGY.md
T
mrcharles 9c2882ea00 fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes 
Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-31 00:12:04 -05:00

366 lines
15 KiB
Markdown
Raw Blame History

# Network Topology
> **Complete physical + logical network reference for the PFVCluster.**
> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct
> host probes. All data is live ground truth -- not documentation inherited.
>
> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).
---
## 1. Physical topology diagram
```mermaid
graph TB
subgraph "RACK 5 - Core / Data / Infra"
CORE["pfv-core-sw01<br>PowerConnect 5448<br>192.168.0.12<br>(mgmt + storage VLANs)"]
TSYS1["pfv-tsys1<br>i7-4770 Haswell<br>Infra host"]
TSYS3["pfv-tsys3<br>Xeon E3-1535M v5<br>Compute"]
TSYS4["pfv-tsys4<br>Xeon E3-1246 v3<br>NFS Storage + PBS"]
TSYS5["pfv-tsys5<br>Xeon E5620 Westmere<br>NFS Storage + Sandboxes"]
TSYS9["pfv-tsys9<br>i5-10500 Comet Lake<br>Infra + Compute"]
TSYS4_CONSOLE["conman + ser2net<br>7 switch consoles<br>Tailscale 100.70.77.93:7890"]
TSYS1_PDU["Cyclades PM10i PDU<br>powerman on tsys1"]
TSYS1_UPS["APC Smart-UPS C 1500<br>NUT on tsys1"]
end
subgraph "RACK 3 - Compute / RackRental"
TOR3STOR["pfv-tor3-stor<br>PowerConnect 5324<br>192.168.0.9<br>storage VLAN"]
TOR3MGMT["pfv-tor3-mgmt<br>PowerConnect 5324<br>192.168.0.7<br>mgmt VLAN"]
TSYS6["pfv-tsys6<br>2x Xeon E5530 Nehalem<br>RackRental"]
TSYS7["pfv-tsys7<br>2x Xeon E5-2630 v2<br>RackRental"]
end
%% Cross-rack trunk (two one-way dotted arrows with labels)
CORE -. "ch1: 3x1G active (g13-15)<br>hash=layer-2-3<br>mode=on" .-> TOR3STOR
TOR3STOR -. "ch1: 4x1G (g20-23)<br>hash=layer-2-3-4<br>mode=on" .-> CORE
%% Rack 5 connections to core
CORE --> TSYS1
CORE --> TSYS3
CORE --> TSYS4
CORE --> TSYS5
CORE --> TSYS9
%% Rack 3 connections
TOR3STOR --> TSYS6
TOR3STOR --> TSYS7
TOR3MGMT --> TSYS6
TOR3MGMT --> TSYS7
```
### How to read this diagram
The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** --
the most critical link in the cluster. It carries all NFS storage traffic
between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5).
The trunk is a 4-port static LAG, but only 3 links are active due to a cable
mismatch (see §3 below).
---
## 2. Subnet / VLAN map
| VLAN | Subnet | Purpose | Where it lives |
|------|--------|---------|----------------|
| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) |
| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) |
| -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs |
**Management IPs (VLAN 1, via DHCP reservations):**
| Host | mgmt IP (vmbr0) | Tailscale IP |
|------|-----------------|--------------|
| tsys1 | 192.168.3.11 | 100.121.189.98 |
| tsys3 | 192.168.2.5 | (connected) |
| tsys4 | 192.168.3.251 | 100.70.77.93 |
| tsys5 | 192.168.3.179 | (connected) |
| tsys6 | 192.168.3.169 | 100.73.35.111 |
| tsys7 | 192.168.3.55 | (connected) |
| tsys9 | 192.168.3.58 | (connected) |
**Storage IPs (VLAN 1000, static):**
| Host | Storage IP | Interface |
|------|-----------|-----------|
| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) |
| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) |
| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) |
| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) |
| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) |
| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) |
| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) |
**Switch IPs (VLAN 1, static DHCP reservations):**
| Switch | IP | Model | Console |
|--------|----|-------|---------|
| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 |
| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 |
| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 |
| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? |
**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c)
---
## 3. Cross-rack trunk detail (ch1) -- the critical link
This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and
rack 5 (tsys4/5). It is the single most important network path in the cluster.
```mermaid
graph LR
subgraph "core-sw01 (rack5)"
G13["g13<br>1G UP"]
G14["g14<br>1G UP"]
G15["g15<br>1G UP"]
G16["g16<br>1G UP<br>NOT in LAG!"]
G17["g17<br>DOWN<br>in config but<br>no cable"]
end
subgraph "tor3-stor (rack3)"
G20["g20<br>1G UP"]
G21["g21<br>1G UP"]
G22["g22<br>1G UP"]
G23["g23<br>1G UP"]
end
G13 --- G20
G14 --- G21
G15 --- G22
G16 -.- G23
G17 -.->|no cable| G23
```
### Configuration comparison
| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? |
|----------|-------------------|--------------------|--------|
| Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** |
| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- |
| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** |
| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** |
| Channel mode | on (static) | on (static) | Yes (both should be LACP) |
| VLAN | access vlan 1000 | access vlan 1000 | Yes |
### Problems
1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on
core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex)
but is NOT in the LAG group. g17 is configured in the LAG but has no cable
(DOWN). Result: **only 3 of 4 links are active**.
2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP).
tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With
layer-2-3 on core-sw01, **all traffic between a given host pair hashes to
one link** regardless of how many TCP connections (nconnect) are used.
3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure
detection -- a dead cable stays in the LAG until manually discovered.
---
## 4. Complete switch port map
### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5
| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) |
| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) |
| g5 | -- | UP | 1 | pfv-tsys1 (mgmt) |
| g6 | 100 | UP | 1 | pfvsvrpi (secondary) |
| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt |
| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** |
| g10 | 1000 | UP | 1 | (unknown) |
| g11 | 1000 | UP | 1 | (unknown) |
| g12 | 1000 | UP | 1 | (unknown) |
| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** |
| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** |
| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** |
| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** |
| **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** |
| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt |
| g21 | 1000 | UP | 1 | (unknown) |
| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) |
| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** |
| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) |
| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) |
| g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) |
| g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) |
| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) |
| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** |
| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) |
| g41 | 1000 | UP | 1 | (unknown) |
| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) |
| g44 | 1000 | UP | 1 | (unknown) |
| g45-g48 | -- | DOWN | -- | (Combo ports, unused) |
**Port-channels on core-sw01:**
| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor |
| ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused |
| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) |
**LAG hash:** `port-channel load-balance layer-2-3`
### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3
| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) |
| g2-g7 | 1000 | UP | 1 | (unused but UP) |
| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** |
| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** |
| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** |
| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** |
| **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
| **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
| g15-g19 | 1000 | UP | 1000 | (unused storage ports) |
| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** |
| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** |
| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** |
| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** |
| g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) |
**Port-channels on tor3-stor:**
| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) |
| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) |
| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) |
| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** |
**LAG hash:** `port-channel load-balance layer-2-3-4`
### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3
| Port | Speed | Status | Connected device |
|------|-------|--------|-----------------|
| g3 | -- | UP | tsys7 (mgmt nic0) |
| g7 | -- | UP | tsys6 (mgmt nic0) |
(Full port map not captured -- this switch carries only VLAN 1 management
traffic. Low priority for storage performance.)
---
## 5. Per-host network interface reference
| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed |
|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------|
| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G |
| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G |
| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G |
| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** |
| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G |
### Key observations
- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2
active slaves). All NFS traffic from rack 3 uses these bonds.
- **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy.
- **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is
active via core-sw01 ch3 (g34). This is the LACP problem the operator has been
troubleshooting.
- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS),
tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one
link even with nconnect's multiple TCP connections).
---
## 6. NFS data flow diagram
```mermaid
graph TB
subgraph "Storage Servers (rack 5)"
T4D2["tsys4 D2<br>WDC Red 3TB<br>Tier 2"]
T4D5["tsys4 D5<br>Hitachi 2TB<br>Tier 2"]
T4D1["tsys4 D1<br>WD 1TB USB<br>Tier 3"]
T4D3["tsys4 D3<br>Seagate 500GB USB<br>Tier 3"]
T5S1["tsys5 S1<br>IronWolf 1TB<br>Tier 2"]
T5S2["tsys5 S2<br>IronWolf 1TB<br>Tier 2"]
T5SSD["tsys5 SSD<br>Samsung 860 PRO<br>Tier 1 (93% full)"]
end
subgraph "VLAN 1000 storage network"
NFS["NFS v4.2<br>nconnect=4<br>rsize/wsize=1MB"]
end
subgraph "NFS Clients"
T1["tsys1 (9 mounts)"]
T3["tsys3 (5 mounts)"]
T5["tsys5 (3 mounts, NO nconnect)"]
T6["tsys6 (9 mounts)"]
T7["tsys7 (9 mounts)"]
T9["tsys9 (5 mounts)"]
end
T4D2 --> NFS
T4D5 --> NFS
T4D1 --> NFS
T4D3 --> NFS
T5S1 --> NFS
T5S2 --> NFS
T5SSD --> NFS
NFS --> T1
NFS --> T3
NFS --> T5
NFS --> T6
NFS --> T7
NFS --> T9
```
### NFS mount inventory (live)
| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|--------|--------|-------|-------|-------|-------|-------|-------|-------|
| tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- |
| tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
| tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- |
| tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
| tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- |
| tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- |
| tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ |
| tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ |
| tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ |
(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts.
tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.)
---
## 7. Switch management access
| Method | Details |
|--------|---------|
| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. |
| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` |
| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) |
| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) |
| **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. |
### Console port assignments
| Port | Console name | Switch | ID_PATH (USB port) |
|------|-------------|--------|---------------------|
| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 |
| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 |
| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 |
| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 |
| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 |
| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 |
| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 |
Reference in New Issue View Git Blame Copy Permalink