# Network Topology

> **Complete physical + logical network reference for the PFVCluster.**
> Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct
> host probes. All data is live ground truth -- not documentation inherited.
>
> Companion docs: [`STORAGE-ARCHITECTURE.md`](STORAGE-ARCHITECTURE.md),
> [`HOST-REFERENCE.md`](HOST-REFERENCE.md), [`VM-INVENTORY.md`](VM-INVENTORY.md).
> Findings/analysis: [`AUDIT-2026-07-30.md`](AUDIT-2026-07-30.md).

---

## 1. Physical topology diagram

```mermaid
graph TB
    subgraph "RACK 5 - Core / Data / Infra"
        CORE["pfv-core-sw01<br>PowerConnect 5448<br>192.168.0.12<br>(mgmt + storage VLANs)"]

        TSYS1["pfv-tsys1<br>i7-4770 Haswell<br>Infra host"]
        TSYS3["pfv-tsys3<br>Xeon E3-1535M v5<br>Compute"]
        TSYS4["pfv-tsys4<br>Xeon E3-1246 v3<br>NFS Storage + PBS"]
        TSYS5["pfv-tsys5<br>Xeon E5620 Westmere<br>NFS Storage + Sandboxes"]
        TSYS9["pfv-tsys9<br>i5-10500 Comet Lake<br>Infra + Compute"]

        TSYS4_CONSOLE["conman + ser2net<br>7 switch consoles<br>Tailscale 100.70.77.93:7890"]
        TSYS1_PDU["Cyclades PM10i PDU<br>powerman on tsys1"]
        TSYS1_UPS["APC Smart-UPS C 1500<br>NUT on tsys1"]
    end

    subgraph "RACK 3 - Compute / RackRental"
        TOR3STOR["pfv-tor3-stor<br>PowerConnect 5324<br>192.168.0.9<br>storage VLAN"]
        TOR3MGMT["pfv-tor3-mgmt<br>PowerConnect 5324<br>192.168.0.7<br>mgmt VLAN"]

        TSYS6["pfv-tsys6<br>2x Xeon E5530 Nehalem<br>RackRental"]
        TSYS7["pfv-tsys7<br>2x Xeon E5-2630 v2<br>RackRental"]
    end

    %% Cross-rack trunk (two one-way dotted arrows with labels)
    CORE -. "ch1: 3x1G active (g13-15)<br>hash=layer-2-3<br>mode=on" .-> TOR3STOR
    TOR3STOR -. "ch1: 4x1G (g20-23)<br>hash=layer-2-3-4<br>mode=on" .-> CORE

    %% Rack 5 connections to core
    CORE --> TSYS1
    CORE --> TSYS3
    CORE --> TSYS4
    CORE --> TSYS5
    CORE --> TSYS9

    %% Rack 3 connections
    TOR3STOR --> TSYS6
    TOR3STOR --> TSYS7
    TOR3MGMT --> TSYS6
    TOR3MGMT --> TSYS7
```

### How to read this diagram

The dashed line between core-sw01 and tor3-stor is the **cross-rack trunk** --
the most critical link in the cluster. It carries all NFS storage traffic
between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5).
The trunk is a 4-port static LAG, but only 3 links are active due to a cable
mismatch (see §3 below).

---

## 2. Subnet / VLAN map

| VLAN | Subnet | Purpose | Where it lives |
|------|--------|---------|----------------|
| **1** (default) | 192.168.0.0/22 | Management network | All switches, all hosts (vmbr0 DHCP) |
| **1000** | 10.100.100.0/24 | Storage network (NFS) | core-sw01 + tor3-stor only (trunked) |
| -- (Tailscale) | 100.x.y.z/32 (CGNAT) | Remote access / overlay | All hosts + select VMs |

**Management IPs (VLAN 1, via DHCP reservations):**

| Host | mgmt IP (vmbr0) | Tailscale IP |
|------|-----------------|--------------|
| tsys1 | 192.168.3.11 | 100.121.189.98 |
| tsys3 | 192.168.2.5 | (connected) |
| tsys4 | 192.168.3.251 | 100.70.77.93 |
| tsys5 | 192.168.3.179 | (connected) |
| tsys6 | 192.168.3.169 | 100.73.35.111 |
| tsys7 | 192.168.3.55 | (connected) |
| tsys9 | 192.168.3.58 | (connected) |

**Storage IPs (VLAN 1000, static):**

| Host | Storage IP | Interface |
|------|-----------|-----------|
| tsys1 | 10.100.100.1 | datanet (nic1, USB dongle) |
| tsys3 | 10.100.100.3 | datanet (enx8cae4ccda774, USB dongle) |
| tsys4 | 10.100.100.4 | datanet (enx8cae4ccda926, USB dongle) |
| tsys5 | 10.100.100.5 | storagenet (bond0 → nic1, single link) |
| tsys6 | 10.100.100.6 | storagenet (bond0 → nic1+nic2, 2G LACP) |
| tsys7 | 10.100.100.7 | datanet (bond0 → nic1+nic2, 2G LACP) |
| tsys9 | 10.100.100.9 | datanet (enx9c69d36a5b6c, USB dongle) |

**Switch IPs (VLAN 1, static DHCP reservations):**

| Switch | IP | Model | Console |
|--------|----|-------|---------|
| pfv-core-sw01 | 192.168.0.12 | PowerConnect 5448 | conman port 2001 |
| pfv-tor3-stor | 192.168.0.9 | PowerConnect 5324 (Neyland 24T) | conman port 2003 |
| pfv-tor3-mgmt | 192.168.0.7 | PowerConnect 5324 (Neyland 24T) | conman port 2002 |
| (unidentified) | 192.168.0.8 | PowerConnect 5324 (Neyland 24T) | conman port 2005? |

**SNMP community:** `kn3lmgmt` (read-only, all switches respond to SNMPv2c)

---

## 3. Cross-rack trunk detail (ch1) -- the critical link

This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and
rack 5 (tsys4/5). It is the single most important network path in the cluster.

```mermaid
graph LR
    subgraph "core-sw01 (rack5)"
        G13["g13<br>1G UP"]
        G14["g14<br>1G UP"]
        G15["g15<br>1G UP"]
        G16["g16<br>1G UP<br>NOT in LAG!"]
        G17["g17<br>DOWN<br>in config but<br>no cable"]
    end

    subgraph "tor3-stor (rack3)"
        G20["g20<br>1G UP"]
        G21["g21<br>1G UP"]
        G22["g22<br>1G UP"]
        G23["g23<br>1G UP"]
    end

    G13 --- G20
    G14 --- G21
    G15 --- G22
    G16 -.- G23
    G17 -.->|no cable| G23
```

### Configuration comparison

| Property | core-sw01 (rack5) | tor3-stor (rack3) | Match? |
|----------|-------------------|--------------------|--------|
| Physical ports configured | g(13-15,17) | g(20-23) | **No -- asymmetric** |
| Physical ports UP | g13, g14, g15, g16 | g20, g21, g22, g23 | -- |
| Active in LAG | g13, g14, g15 (3) | g20, g21, g22, g23 (4) | **No -- 3 vs 4** |
| Load-balance hash | layer-2-3 (MAC+IP) | layer-2-3-4 (MAC+IP+Port) | **No -- asymmetric** |
| Channel mode | on (static) | on (static) | Yes (both should be LACP) |
| VLAN | access vlan 1000 | access vlan 1000 | Yes |

### Problems

1. **g16/g17 cable mismatch.** The 4th physical cable from tor3-stor lands on
   core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex)
   but is NOT in the LAG group. g17 is configured in the LAG but has no cable
   (DOWN). Result: **only 3 of 4 links are active**.

2. **Hash policy asymmetry.** core-sw01 hashes on layer-2-3 (src/dst MAC + IP).
   tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With
   layer-2-3 on core-sw01, **all traffic between a given host pair hashes to
   one link** regardless of how many TCP connections (nconnect) are used.

3. **Static mode (`on`).** Neither switch runs LACP. There is no link-failure
   detection -- a dead cable stays in the LAG until manually discovered.

---

## 4. Complete switch port map

### pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5

| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | pfvsvrpi (Raspberry Pi) |
| g4 | 100 | UP | 1 | pfv-tsys4 (mgmt, secondary) |
| g5 | -- | UP | 1 | pfv-tsys1 (mgmt) |
| g6 | 100 | UP | 1 | pfvsvrpi (secondary) |
| g8 | 1000 | UP | 1 | pfv-tsys4-mgmt |
| g9 | 1000 | UP | 1 | AP-Wallmount (UAP-AC-LR) **⚠ 2902 inErrors** |
| g10 | 1000 | UP | 1 | (unknown) |
| g11 | 1000 | UP | 1 | (unknown) |
| g12 | 1000 | UP | 1 | (unknown) |
| **g13** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g20** |
| **g14** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g21** |
| **g15** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → tor3-stor g22** |
| **g16** | **1000** | **UP** | **1000** | **cross-rack trunk cable (NOT in LAG -- should be g17)** |
| **g17** | -- | **DOWN** | **1000 (ch1)** | **in LAG config but no cable** |
| g19 | 1000 | UP | 1 | pfv-tsys5-mgmt |
| g21 | 1000 | UP | 1 | (unknown) |
| g23 | 100 | UP | 1 | pfv-tsys9 (mgmt) |
| g26 | 1000 | UP | 1 | AP-Tabletop **⚠ 73 inErrors** |
| g27 | 1000 | UP | 1000 | pfv-tsys3-stor (datanet) |
| g31 | 1000 | UP | 1000 | pfv-tsys4-stor (datanet, USB) |
| g32 | -- | -- | -- | pfv-tsys4-storage (configured, DOWN) |
| g33 | -- | -- | -- | pfv-tsys5-stor (ch2, DOWN/Not Present) |
| g34 | 1000 | UP | 1000 (ch3) | pfv-tsys5-stor (bond0, 1 link active) |
| g38 | 1000 | UP | 1000 | pfv-tsys9-stor (datanet) **⚠ 590 inErrors** |
| g40 | 1000 | UP | 1000 | pfv-tsys1-stor (datanet) |
| g41 | 1000 | UP | 1 | (unknown) |
| g43 | 1000 | UP | 1 | pfv-tsys3 (mgmt) |
| g44 | 1000 | UP | 1 | (unknown) |
| g45-g48 | -- | DOWN | -- | (Combo ports, unused) |

**Port-channels on core-sw01:**

| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g13-15 (active), g17 (inactive) | 1000 | Cross-rack trunk to tor3-stor |
| ch2 | -- | DOWN | g32-33 (non-candidate) | -- | Unused |
| ch3 | 1000 | UP | g34 (active) | 1000 | tsys5 storage bond (degraded, 1 of 2 links) |

**LAG hash:** `port-channel load-balance layer-2-3`

### pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3

| Port | Speed | Status | VLAN | Description / Connected device |
|------|-------|--------|------|-------------------------------|
| g1 | 1000 | UP | 1 | (unknown -- uplink to tor3-mgmt or router) |
| g2-g7 | 1000 | UP | 1 | (unused but UP) |
| **g8** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic1** |
| **g9** | **1000** | **UP** | **1000 (ch2)** | **tsys6 stor nic2** |
| **g10** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic1** |
| **g11** | **1000** | **UP** | **1000 (ch3)** | **tsys7 stor nic2** |
| **g13** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
| **g14** | **1000** | **UP** | **--** | **tsys8-stor (STALE -- tsys8 retired)** |
| g15-g19 | 1000 | UP | 1000 | (unused storage ports) |
| **g20** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g13** |
| **g21** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g14** |
| **g22** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g15** |
| **g23** | **1000** | **UP** | **1000 (ch1)** | **cross-rack trunk → core-sw01 g16** |
| g24 | -- | DOWN | 1 | (spanning-tree disabled, portfast) |

**Port-channels on tor3-stor:**

| Channel | Speed | Status | Members | VLAN | Notes |
|---------|-------|--------|---------|------|-------|
| ch1 | 1000 | UP | g20-23 (`mode on`) | 1000 | Cross-rack trunk to core-sw01 (4x1G) |
| ch2 | 1000 | UP | g8-9 (`mode auto`) | 1000 | tsys6 storage bond (2x1G LACP) |
| ch3 | 1000 | UP | g10-11 (`mode auto`) | 1000 | tsys7 storage bond (2x1G LACP) |
| ch4 | 1000 | UP | g13-14 (`mode on`) | 1000 | **tsys8-stor (STALE -- retired host)** |

**LAG hash:** `port-channel load-balance layer-2-3-4`

### pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3

| Port | Speed | Status | Connected device |
|------|-------|--------|-----------------|
| g3 | -- | UP | tsys7 (mgmt nic0) |
| g7 | -- | UP | tsys6 (mgmt nic0) |

(Full port map not captured -- this switch carries only VLAN 1 management
traffic. Low priority for storage performance.)

---

## 5. Per-host network interface reference

| Host | mgmt NIC | Switch port | Storage NIC | Switch port | Storage bond | Bond hash | Storage link speed |
|------|----------|-------------|-------------|-------------|-------------|-----------|-------------------|
| tsys1 | nic0 | core g5 | nic1 (onboard) | core g40 | **No bond** (single NIC) | -- | 1G |
| tsys3 | nic0 | core g43 | enx8cae4ccda774 (USB) | core g27 | **No bond** (USB dongle) | -- | 1G |
| tsys4 | nic0 | core g8 | enx8cae4ccda926 (USB) | core g31 | bond0 (1 slave, USB) | layer2+3 | 1G |
| tsys5 | nic0 | core g19 | nic1 + nic2 | core ch3 (g34) | bond0 (1 of 2 slaves) | layer2+3 | **1G (degraded)** |
| tsys6 | nic0 | tor3-mgmt g7 | nic1 + nic2 | tor3-stor g8/g9 (ch2) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys7 | nic0 | tor3-mgmt g3 | nic1 + nic2 | tor3-stor g10/g11 (ch3) | bond0 (2 of 2 slaves) | **layer3+4** | **2G LACP** |
| tsys9 | nic0 | core g23 | enx9c69d36a5b6c (USB) | core g38 | **No bond** (USB dongle) | -- | 1G |

### Key observations

- **tsys6/7 are the only hosts with working 2G LACP bonds** (layer3+4 hash, 2
  active slaves). All NFS traffic from rack 3 uses these bonds.
- **tsys1/3/4/9 use USB dongles** for storage network -- single 1G, no redundancy.
- **tsys5 bond0 is degraded** -- nic2 is DOWN (no cable connected). Only nic1 is
  active via core-sw01 ch3 (g34). This is the LACP problem the operator has been
  troubleshooting.
- **Bond hash inconsistency:** tsys6/7 use layer3+4 (optimal for nconnect NFS),
  tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one
  link even with nconnect's multiple TCP connections).

---

## 6. NFS data flow diagram

```mermaid
graph TB
    subgraph "Storage Servers (rack 5)"
        T4D2["tsys4 D2<br>WDC Red 3TB<br>Tier 2"]
        T4D5["tsys4 D5<br>Hitachi 2TB<br>Tier 2"]
        T4D1["tsys4 D1<br>WD 1TB USB<br>Tier 3"]
        T4D3["tsys4 D3<br>Seagate 500GB USB<br>Tier 3"]
        T5S1["tsys5 S1<br>IronWolf 1TB<br>Tier 2"]
        T5S2["tsys5 S2<br>IronWolf 1TB<br>Tier 2"]
        T5SSD["tsys5 SSD<br>Samsung 860 PRO<br>Tier 1 (93% full)"]
    end

    subgraph "VLAN 1000 storage network"
        NFS["NFS v4.2<br>nconnect=4<br>rsize/wsize=1MB"]
    end

    subgraph "NFS Clients"
        T1["tsys1 (9 mounts)"]
        T3["tsys3 (5 mounts)"]
        T5["tsys5 (3 mounts, NO nconnect)"]
        T6["tsys6 (9 mounts)"]
        T7["tsys7 (9 mounts)"]
        T9["tsys9 (5 mounts)"]
    end

    T4D2 --> NFS
    T4D5 --> NFS
    T4D1 --> NFS
    T4D3 --> NFS
    T5S1 --> NFS
    T5S2 --> NFS
    T5SSD --> NFS

    NFS --> T1
    NFS --> T3
    NFS --> T5
    NFS --> T6
    NFS --> T7
    NFS --> T9
```

### NFS mount inventory (live)

| Server | Export | tsys1 | tsys3 | tsys4 | tsys5 | tsys6 | tsys7 | tsys9 |
|--------|--------|-------|-------|-------|-------|-------|-------|-------|
| tsys4 | D1 | -- | -- | -- | -- | ✓ | ✓ | -- |
| tsys4 | **D2** | ✓ | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
| tsys4 | D3 | -- | -- | -- | ✓ | ✓ | ✓ | -- |
| tsys4 | D5 | -- | ✓ | -- | ✓ | ✓ | ✓ | ✓ |
| tsys5 | S1 | ✓ | -- | -- | -- | ✓ | ✓ | -- |
| tsys5 | S2 | ✓ | ✓ | -- | -- | ✓ | ✓ | -- |
| tsys5 | S3 | ✓ | -- | -- | -- | ✓ | ✓ | ✓ |
| tsys5 | S4 | ✓ | ✓ | -- | -- | ✓ | ✓ | ✓ |
| tsys5 | T5-SSD | -- | ✓ | -- | -- | ✓ | ✓ | ✓ |

(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts.
tsys5 = NFS server + 3 client mounts from tsys4, **missing nconnect**.)

---

## 7. Switch management access

| Method | Details |
|--------|---------|
| **SNMP** | Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs. |
| **Console (conman)** | 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.` |
| **Console tool** | `perf/scripts/conman-console.py` (PTY-based driver, portable) |
| **SNMP audit tool** | `perf/scripts/snmp-switch-audit.py` (config-driven) |
| **Direct telnet** | `telnet pfv-tsys4 200X` -- ONLY when conmand stopped (conflicts). Use conman instead. |

### Console port assignments

| Port | Console name | Switch | ID_PATH (USB port) |
|------|-------------|--------|---------------------|
| 2001 | pfv-core-sw01 | Dell PowerConnect 5448 (core, rack5) | usb-0:1.5.4.4 |
| 2002 | pfv-tor3-mgmt | Neyland 24T (mgmt TOR, rack3) | usb-0:1.6.3.1 |
| 2003 | pfv-tor3-stor | Neyland 24T (storage TOR, rack3) | usb-0:1.6.3.3.2 |
| 2004 | pfv-rrinfra-rtr | Cisco router (rrinfra) | usb-0:1.6.3.3.1 |
| 2005 | pfv-r2-tor-top | Rack 2 top-of-rack switch | usb-0:1.6.3.3.3 |
| 2006 | subodev-torsw | Suborbital device TOR switch | usb-0:1.5.4.1 |
| 2007 | pfv-r2-sw | Rack 2 old Dell switch | usb-0:1.6.3.2 |
