# Network Topology > **Complete physical + logical network reference for the PFVCluster.** > Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct > host probes. All data is live ground truth -- not documentation inherited. > > Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md), > [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md). > Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md). --- ## 1. Physical topology diagram ```mermaid graph TB subgraph "RACK 5 - Core / Data / Infra" CORE["pfv-core-sw01
PowerConnect 5448
192.168.0.12
(mgmt + storage VLANs)"] TSYS1["pfv-tsys1
i7-4770 Haswell
Infra host"] TSYS3["pfv-tsys3
Xeon E3-1535M v5
Compute"] TSYS4["pfv-tsys4
Xeon E3-1246 v3
NFS Storage + PBS"] TSYS5["pfv-tsys5
Xeon E5620 Westmere
NFS Storage + Sandboxes"] TSYS9["pfv-tsys9
i5-10500 Comet Lake
Infra + Compute"] TSYS4_CONSOLE["conman + ser2net
7 switch consoles
Tailscale 100.70.77.93:7890"] TSYS1_PDU["Cyclades PM10i PDU
powerman on tsys1"] TSYS1_UPS["APC Smart-UPS C 1500
NUT on tsys1"] end subgraph "RACK 3 - Compute / RackRental" TOR3STOR["pfv-tor3-stor
PowerConnect 5324
192.168.0.9
storage VLAN"] TOR3MGMT["pfv-tor3-mgmt
PowerConnect 5324
192.168.0.7
mgmt VLAN"] TSYS6["pfv-tsys6
2x Xeon E5530 Nehalem
RackRental"] TSYS7["pfv-tsys7
2x Xeon E5-2630 v2
RackRental"] end %% Cross-rack trunk (two one-way dotted arrows with labels) CORE -. "ch1: 3x1G active (g13-15)
hash=layer-2-3
mode=on" .-> TOR3STOR TOR3STOR -. "ch1: 4x1G (g20-23)
hash=layer-2-3-4
mode=on" .-> CORE %% Rack 5 connections to core CORE --> TSYS1 CORE --> TSYS3 CORE --> TSYS4 CORE --> TSYS5 CORE --> TSYS9 %% Rack 3 connections TOR3STOR --> TSYS6 TOR3STOR --> TSYS7 TOR3MGMT --> TSYS6 TOR3MGMT --> TSYS7 ``` ### How to read this diagram The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** -- the most critical link in the cluster. It carries all NFS storage traffic between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). The trunk is a 4-port static LAG, but only 3 links are active due to a cable mismatch (see §3 below). --- ## 2. Subnet / VLAN map | VLAN | Subnet | Purpose | Where it lives | |------|--------|---------|----------------| | **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) | | **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) | | -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs | **Management IPs (VLAN 1, via DHCP reservations):** | Host | mgmt IP (vmbr0) | Tailscale IP | |------|-----------------|--------------| | tsys1 | 192.168.3.11 | 100.121.189.98 | | tsys3 | 192.168.2.5 | (connected) | | tsys4 | 192.168.3.251 | 100.70.77.93 | | tsys5 | 192.168.3.179 | (connected) | | tsys6 | 192.168.3.169 | 100.73.35.111 | | tsys7 | 192.168.3.55 | (connected) | | tsys9 | 192.168.3.58 | (connected) | **Storage IPs (VLAN 1000, static):** | Host | Storage IP | Interface | |------|-----------|-----------| | tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) | | tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) | | tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) | | tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) | | tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) | | tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) | | tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) | **Switch IPs (VLAN 1, static DHCP reservations):** | Switch | IP | Model | Console | |--------|----|-------|---------| | pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 | | pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 | | pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 | | (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? | **SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c) --- ## 3. Cross-rack trunk detail (ch1) -- the critical link This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and rack 5 (tsys4/5). It is the single most important network path in the cluster. ```mermaid graph LR subgraph "core-sw01 (rack5)" G13["g13
1G UP"] G14["g14
1G UP"] G15["g15
1G UP"] G16["g16
1G UP
NOT in LAG!"] G17["g17
DOWN
in config but
no cable"] end subgraph "tor3-stor (rack3)" G20["g20
1G UP"] G21["g21
1G UP"] G22["g22
1G UP"] G23["g23
1G UP"] end G13 --- G20 G14 --- G21 G15 --- G22 G16 -.- G23 G17 -.->|no cable| G23 ``` ### Configuration comparison | Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? | |----------|-------------------|--------------------|--------| | Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** | | Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- | | Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** | | Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** | | Channel mode | on (static) | on (static) | Yes (both should be LACP) | | VLAN | access vlan 1000 | access vlan 1000 | Yes | ### Problems 1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex) but is NOT in the LAG group. g17 is configured in the LAG but has no cable (DOWN). Result: **only 3 of 4 links are active**. 2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP). tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With layer-2-3 on core-sw01, **all traffic between a given host pair hashes to one link** regardless of how many TCP connections (nconnect) are used. 3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure detection -- a dead cable stays in the LAG until manually discovered. --- ## 4. Complete switch port map ### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5 | Port | Speed | Status | VLAN | Description / Connected device | |------|-------|--------|------|-------------------------------| | g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) | | g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) | | g5 | -- | UP | 1 | pfv-tsys1 (mgmt) | | g6 | 100 | UP | 1 | pfvsvrpi (secondary) | | g8 | 1000 | UP | 1 | pfv-tsys4-mgmt | | g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** | | g10 | 1000 | UP | 1 | (unknown) | | g11 | 1000 | UP | 1 | (unknown) | | g12 | 1000 | UP | 1 | (unknown) | | **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** | | **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** | | **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** | | **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** | | **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** | | g19 | 1000 | UP | 1 | pfv-tsys5-mgmt | | g21 | 1000 | UP | 1 | (unknown) | | g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) | | g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** | | g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) | | g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) | | g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) | | g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) | | g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) | | g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** | | g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) | | g41 | 1000 | UP | 1 | (unknown) | | g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) | | g44 | 1000 | UP | 1 | (unknown) | | g45-g48 | -- | DOWN | -- | (Combo ports, unused) | **Port-channels on core-sw01:** | Channel | Speed | Status | Members | VLAN | Notes | |---------|-------|--------|---------|------|-------| | ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor | | ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused | | ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) | **LAG hash:** `port-channel load-balance layer-2-3` ### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 | Port | Speed | Status | VLAN | Description / Connected device | |------|-------|--------|------|-------------------------------| | g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) | | g2-g7 | 1000 | UP | 1 | (unused but UP) | | **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** | | **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** | | **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** | | **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** | | **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | | **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** | | g15-g19 | 1000 | UP | 1000 | (unused storage ports) | | **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** | | **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** | | **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** | | **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** | | g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) | **Port-channels on tor3-stor:** | Channel | Speed | Status | Members | VLAN | Notes | |---------|-------|--------|---------|------|-------| | ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) | | ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) | | ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) | | ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** | **LAG hash:** `port-channel load-balance layer-2-3-4` ### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3 | Port | Speed | Status | Connected device | |------|-------|--------|-----------------| | g3 | -- | UP | tsys7 (mgmt nic0) | | g7 | -- | UP | tsys6 (mgmt nic0) | (Full port map not captured -- this switch carries only VLAN 1 management traffic. Low priority for storage performance.) --- ## 5. Per-host network interface reference | Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed | |------|----------|-------------|-------------|-------------|-------------|-----------|-------------------| | tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G | | tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G | | tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G | | tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** | | tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | | tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** | | tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G | ### Key observations - **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2 active slaves). All NFS traffic from rack 3 uses these bonds. - **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy. - **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is active via core-sw01 ch3 (g34). This is the LACP problem the operator has been troubleshooting. - **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS), tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one link even with nconnect's multiple TCP connections). --- ## 6. NFS data flow diagram ```mermaid graph TB subgraph "Storage Servers (rack 5)" T4D2["tsys4 D2
WDC Red 3TB
Tier 2"] T4D5["tsys4 D5
Hitachi 2TB
Tier 2"] T4D1["tsys4 D1
WD 1TB USB
Tier 3"] T4D3["tsys4 D3
Seagate 500GB USB
Tier 3"] T5S1["tsys5 S1
IronWolf 1TB
Tier 2"] T5S2["tsys5 S2
IronWolf 1TB
Tier 2"] T5SSD["tsys5 SSD
Samsung 860 PRO
Tier 1 (93% full)"] end subgraph "VLAN 1000 storage network" NFS["NFS v4.2
nconnect=4
rsize/wsize=1MB"] end subgraph "NFS Clients" T1["tsys1 (9 mounts)"] T3["tsys3 (5 mounts)"] T5["tsys5 (3 mounts, NO nconnect)"] T6["tsys6 (9 mounts)"] T7["tsys7 (9 mounts)"] T9["tsys9 (5 mounts)"] end T4D2 --> NFS T4D5 --> NFS T4D1 --> NFS T4D3 --> NFS T5S1 --> NFS T5S2 --> NFS T5SSD --> NFS NFS --> T1 NFS --> T3 NFS --> T5 NFS --> T6 NFS --> T7 NFS --> T9 ``` ### NFS mount inventory (live) | Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 | |--------|--------|-------|-------|-------|-------|-------|-------|-------| | tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- | | tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ | | tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- | | tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ | | tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- | | tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- | | tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ | | tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ | | tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ | (✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.) --- ## 7. Switch management access | Method | Details | |--------|---------| | **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. | | **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` | | **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) | | **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) | | **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. | ### Console port assignments | Port | Console name | Switch | ID_PATH (USB port) | |------|-------------|--------|---------------------| | 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 | | 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 | | 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 | | 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 | | 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 | | 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 | | 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 |