KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
30ddb37b34302b95162e8dbbffdd7f2ab9cd8205
PFVCluster/proxmox/docs/NETWORK-TOPOLOGY.md
T
mrcharles 9c2882ea00 fix: Mermaid diagram syntax + deploy-tuning.sh path + fleet quick fixes 
Mermaid diagrams fixed for Gitea rendering:
- Replaced all <br/> → <br> (86 tags across 4 docs)
- Replaced Unicode chars with ASCII (em dash, multiply, middle dot, star)
- Fixed invalid bidirectional dotted arrow in NETWORK-TOPOLOGY.md
  (<--. "..." .--> is not valid Mermaid; replaced with two -.-> arrows)
- Simplified node labels for cleaner rendering

Host fixes applied (live, not in repo — documented here for audit trail):
- SSH keys standardized to exactly 2 approved keys on all 7 hosts
  (removed host-generated RSA keys, added missing VP TechOps key to tsys9)
- Packages installed: iperf3 (tsys9), sysstat (tsys5), nvme-cli (tsys4/5),
  net-tools (tsys1/6/7)
- tsys4 tuning: rmem/wmem 16MB→128MB, netdev_max_backlog 5000→250000,
  tcp_max_syn_backlog 1024→2048, tuned profile→network-throughput
- tsys5: nconnect options added to storage.cfg (3 NFS stanzas patched;
  will apply on next Proxmox NFS remount/reboot)
- noatime applied to root fs on tsys4 + tsys5 (was only tsys5)
- tsys1 lldpd: installed but systemd/dbus issue prevents enabling
  ("Transport endpoint is not connected") — needs investigation/reboot

Bug fix: deploy-tuning.sh computed wrong script path (scripts/ vs
perf/scripts/) — fixed to use dirname/scripts/apply-tunings.sh

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-31 00:12:04 -05:00

15 KiB
Raw Blame History

Network Topology

Complete physical + logical network reference for the PFVCluster. Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct host probes. All data is live ground truth -- not documentation inherited.

Companion docs: STORAGE-ARCHITECTURE.md, HOST-REFERENCE.md, VM-INVENTORY.md. Findings/analysis: AUDIT-2026-07-30.md.

1. Physical topology diagram

graph TB
    subgraph "RACK 5 - Core / Data / Infra"
        CORE["pfv-core-sw01<br>PowerConnect 5448<br>192.168.0.12<br>(mgmt + storage VLANs)"]

        TSYS1["pfv-tsys1<br>i7-4770 Haswell<br>Infra host"]
        TSYS3["pfv-tsys3<br>Xeon E3-1535M v5<br>Compute"]
        TSYS4["pfv-tsys4<br>Xeon E3-1246 v3<br>NFS Storage + PBS"]
        TSYS5["pfv-tsys5<br>Xeon E5620 Westmere<br>NFS Storage + Sandboxes"]
        TSYS9["pfv-tsys9<br>i5-10500 Comet Lake<br>Infra + Compute"]

        TSYS4_CONSOLE["conman + ser2net<br>7 switch consoles<br>Tailscale 100.70.77.93:7890"]
        TSYS1_PDU["Cyclades PM10i PDU<br>powerman on tsys1"]
        TSYS1_UPS["APC Smart-UPS C 1500<br>NUT on tsys1"]
    end

    subgraph "RACK 3 - Compute / RackRental"
        TOR3STOR["pfv-tor3-stor<br>PowerConnect 5324<br>192.168.0.9<br>storage VLAN"]
        TOR3MGMT["pfv-tor3-mgmt<br>PowerConnect 5324<br>192.168.0.7<br>mgmt VLAN"]

        TSYS6["pfv-tsys6<br>2x Xeon E5530 Nehalem<br>RackRental"]
        TSYS7["pfv-tsys7<br>2x Xeon E5-2630 v2<br>RackRental"]
    end

    %% Cross-rack trunk (two one-way dotted arrows with labels)
    CORE -. "ch1: 3x1G active (g13-15)<br>hash=layer-2-3<br>mode=on" .-> TOR3STOR
    TOR3STOR -. "ch1: 4x1G (g20-23)<br>hash=layer-2-3-4<br>mode=on" .-> CORE

    %% Rack 5 connections to core
    CORE --> TSYS1
    CORE --> TSYS3
    CORE --> TSYS4
    CORE --> TSYS5
    CORE --> TSYS9

    %% Rack 3 connections
    TOR3STOR --> TSYS6
    TOR3STOR --> TSYS7
    TOR3MGMT --> TSYS6
    TOR3MGMT --> TSYS7

How to read this diagram

The dashed line between core-sw01 and tor3-stor is the cross-rack trunk -- the most critical link in the cluster. It carries all NFS storage traffic between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). The trunk is a 4-port static LAG, but only 3 links are active due to a cable mismatch (see §3 below).

2. Subnet / VLAN map

VLAN Subnet Purpose Where it lives
1 (default) 192.168.0.0/22 Management network All switches, all hosts (vmbr0 DHCP)
1000 10.100.100.0/24 Storage network (NFS) core-sw01 + tor3-stor only (trunked)
-- (Tailscale) 100.x.y.z/32 (CGNAT) Remote access / overlay All hosts + select VMs

Management IPs (VLAN 1, via DHCP reservations):

Host mgmt IP (vmbr0) Tailscale IP
tsys1 192.168.3.11 100.121.189.98
tsys3 192.168.2.5 (connected)
tsys4 192.168.3.251 100.70.77.93
tsys5 192.168.3.179 (connected)
tsys6 192.168.3.169 100.73.35.111
tsys7 192.168.3.55 (connected)
tsys9 192.168.3.58 (connected)

Storage IPs (VLAN 1000, static):

Host Storage IP Interface
tsys1 10.100.100.1 datanet (nic1, USB dongle)
tsys3 10.100.100.3 datanet (enx8cae4ccda774, USB dongle)
tsys4 10.100.100.4 datanet (enx8cae4ccda926, USB dongle)
tsys5 10.100.100.5 storagenet (bond0 → nic1, single link)
tsys6 10.100.100.6 storagenet (bond0 → nic1+nic2, 2G LACP)
tsys7 10.100.100.7 datanet (bond0 → nic1+nic2, 2G LACP)
tsys9 10.100.100.9 datanet (enx9c69d36a5b6c, USB dongle)

Switch IPs (VLAN 1, static DHCP reservations):

Switch IP Model Console
pfv-core-sw01 192.168.0.12 PowerConnect 5448 conman port 2001
pfv-tor3-stor 192.168.0.9 PowerConnect 5324 (Neyland 24T) conman port 2003
pfv-tor3-mgmt 192.168.0.7 PowerConnect 5324 (Neyland 24T) conman port 2002
(unidentified) 192.168.0.8 PowerConnect 5324 (Neyland 24T) conman port 2005?

SNMP community: kn3lmgmt (read-only, all switches respond to SNMPv2c)

3. Cross-rack trunk detail (ch1) -- the critical link

This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and rack 5 (tsys4/5). It is the single most important network path in the cluster.

graph LR
    subgraph "core-sw01 (rack5)"
        G13["g13<br>1G UP"]
        G14["g14<br>1G UP"]
        G15["g15<br>1G UP"]
        G16["g16<br>1G UP<br>NOT in LAG!"]
        G17["g17<br>DOWN<br>in config but<br>no cable"]
    end

    subgraph "tor3-stor (rack3)"
        G20["g20<br>1G UP"]
        G21["g21<br>1G UP"]
        G22["g22<br>1G UP"]
        G23["g23<br>1G UP"]
    end

    G13 --- G20
    G14 --- G21
    G15 --- G22
    G16 -.- G23
    G17 -.->|no cable| G23

Configuration comparison

Property core-sw01 (rack5) tor3-stor (rack3) Match?
Physical ports configured g(13-15,17) g(20-23) No -- asymmetric
Physical ports UP g13, g14, g15, g16 g20, g21, g22, g23 --
Active in LAG g13, g14, g15 (3) g20, g21, g22, g23 (4) No -- 3 vs 4
Load-balance hash layer-2-3 (MAC+IP) layer-2-3-4 (MAC+IP+Port) No -- asymmetric
Channel mode on (static) on (static) Yes (both should be LACP)
VLAN access vlan 1000 access vlan 1000 Yes

Problems

  1. g16/g17 cable mismatch. The 4th physical cable from tor3-stor lands on core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex) but is NOT in the LAG group. g17 is configured in the LAG but has no cable (DOWN). Result: only 3 of 4 links are active.

  2. Hash policy asymmetry. core-sw01 hashes on layer-2-3 (src/dst MAC + IP). tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With layer-2-3 on core-sw01, all traffic between a given host pair hashes to one link regardless of how many TCP connections (nconnect) are used.

  3. Static mode (on). Neither switch runs LACP. There is no link-failure detection -- a dead cable stays in the LAG until manually discovered.

4. Complete switch port map

pfv-core-sw01 (PowerConnect 5448, 48 ports) -- rack 5

Port Speed Status VLAN Description / Connected device
g1 1000 UP 1 pfvsvrpi (Raspberry Pi)
g4 100 UP 1 pfv-tsys4 (mgmt, secondary)
g5 -- UP 1 pfv-tsys1 (mgmt)
g6 100 UP 1 pfvsvrpi (secondary)
g8 1000 UP 1 pfv-tsys4-mgmt
g9 1000 UP 1 AP-Wallmount (UAP-AC-LR) ⚠ 2902 inErrors
g10 1000 UP 1 (unknown)
g11 1000 UP 1 (unknown)
g12 1000 UP 1 (unknown)
g13 1000 UP 1000 (ch1) cross-rack trunk → tor3-stor g20
g14 1000 UP 1000 (ch1) cross-rack trunk → tor3-stor g21
g15 1000 UP 1000 (ch1) cross-rack trunk → tor3-stor g22
g16 1000 UP 1000 cross-rack trunk cable (NOT in LAG -- should be g17)
g17 -- DOWN 1000 (ch1) in LAG config but no cable
g19 1000 UP 1 pfv-tsys5-mgmt
g21 1000 UP 1 (unknown)
g23 100 UP 1 pfv-tsys9 (mgmt)
g26 1000 UP 1 AP-Tabletop ⚠ 73 inErrors
g27 1000 UP 1000 pfv-tsys3-stor (datanet)
g31 1000 UP 1000 pfv-tsys4-stor (datanet, USB)
g32 -- -- -- pfv-tsys4-storage (configured, DOWN)
g33 -- -- -- pfv-tsys5-stor (ch2, DOWN/Not Present)
g34 1000 UP 1000 (ch3) pfv-tsys5-stor (bond0, 1 link active)
g38 1000 UP 1000 pfv-tsys9-stor (datanet) ⚠ 590 inErrors
g40 1000 UP 1000 pfv-tsys1-stor (datanet)
g41 1000 UP 1 (unknown)
g43 1000 UP 1 pfv-tsys3 (mgmt)
g44 1000 UP 1 (unknown)
g45-g48 -- DOWN -- (Combo ports, unused)

Port-channels on core-sw01:

Channel Speed Status Members VLAN Notes
ch1 1000 UP g13-15 (active), g17 (inactive) 1000 Cross-rack trunk to tor3-stor
ch2 -- DOWN g32-33 (non-candidate) -- Unused
ch3 1000 UP g34 (active) 1000 tsys5 storage bond (degraded, 1 of 2 links)

LAG hash: port-channel load-balance layer-2-3

pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3

Port Speed Status VLAN Description / Connected device
g1 1000 UP 1 (unknown -- uplink to tor3-mgmt or router)
g2-g7 1000 UP 1 (unused but UP)
g8 1000 UP 1000 (ch2) tsys6 stor nic1
g9 1000 UP 1000 (ch2) tsys6 stor nic2
g10 1000 UP 1000 (ch3) tsys7 stor nic1
g11 1000 UP 1000 (ch3) tsys7 stor nic2
g13 1000 UP -- tsys8-stor (STALE -- tsys8 retired)
g14 1000 UP -- tsys8-stor (STALE -- tsys8 retired)
g15-g19 1000 UP 1000 (unused storage ports)
g20 1000 UP 1000 (ch1) cross-rack trunk → core-sw01 g13
g21 1000 UP 1000 (ch1) cross-rack trunk → core-sw01 g14
g22 1000 UP 1000 (ch1) cross-rack trunk → core-sw01 g15
g23 1000 UP 1000 (ch1) cross-rack trunk → core-sw01 g16
g24 -- DOWN 1 (spanning-tree disabled, portfast)

Port-channels on tor3-stor:

Channel Speed Status Members VLAN Notes
ch1 1000 UP g20-23 (mode on) 1000 Cross-rack trunk to core-sw01 (4x1G)
ch2 1000 UP g8-9 (mode auto) 1000 tsys6 storage bond (2x1G LACP)
ch3 1000 UP g10-11 (mode auto) 1000 tsys7 storage bond (2x1G LACP)
ch4 1000 UP g13-14 (mode on) 1000 tsys8-stor (STALE -- retired host)

LAG hash: port-channel load-balance layer-2-3-4

pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) -- rack 3

Port Speed Status Connected device
g3 -- UP tsys7 (mgmt nic0)
g7 -- UP tsys6 (mgmt nic0)

(Full port map not captured -- this switch carries only VLAN 1 management traffic. Low priority for storage performance.)

5. Per-host network interface reference

Host mgmt NIC Switch port Storage NIC Switch port Storage bond Bond hash Storage link speed
tsys1 nic0 core g5 nic1 (onboard) core g40 No bond (single NIC) -- 1G
tsys3 nic0 core g43 enx8cae4ccda774 (USB) core g27 No bond (USB dongle) -- 1G
tsys4 nic0 core g8 enx8cae4ccda926 (USB) core g31 bond0 (1 slave, USB) layer2+3 1G
tsys5 nic0 core g19 nic1 + nic2 core ch3 (g34) bond0 (1 of 2 slaves) layer2+3 1G (degraded)
tsys6 nic0 tor3-mgmt g7 nic1 + nic2 tor3-stor g8/g9 (ch2) bond0 (2 of 2 slaves) layer3+4 2G LACP
tsys7 nic0 tor3-mgmt g3 nic1 + nic2 tor3-stor g10/g11 (ch3) bond0 (2 of 2 slaves) layer3+4 2G LACP
tsys9 nic0 core g23 enx9c69d36a5b6c (USB) core g38 No bond (USB dongle) -- 1G

Key observations

  • tsys6/7 are the only hosts with working 2G LACP bonds (layer3+4 hash, 2 active slaves). All NFS traffic from rack 3 uses these bonds.
  • tsys1/3/4/9 use USB dongles for storage network -- single 1G, no redundancy.
  • tsys5 bond0 is degraded -- nic2 is DOWN (no cable connected). Only nic1 is active via core-sw01 ch3 (g34). This is the LACP problem the operator has been troubleshooting.
  • Bond hash inconsistency: tsys6/7 use layer3+4 (optimal for nconnect NFS), tsys4/5 use layer2+3 (suboptimal -- same src/dst IP pair always hashes to one link even with nconnect's multiple TCP connections).

6. NFS data flow diagram

graph TB
    subgraph "Storage Servers (rack 5)"
        T4D2["tsys4 D2<br>WDC Red 3TB<br>Tier 2"]
        T4D5["tsys4 D5<br>Hitachi 2TB<br>Tier 2"]
        T4D1["tsys4 D1<br>WD 1TB USB<br>Tier 3"]
        T4D3["tsys4 D3<br>Seagate 500GB USB<br>Tier 3"]
        T5S1["tsys5 S1<br>IronWolf 1TB<br>Tier 2"]
        T5S2["tsys5 S2<br>IronWolf 1TB<br>Tier 2"]
        T5SSD["tsys5 SSD<br>Samsung 860 PRO<br>Tier 1 (93% full)"]
    end

    subgraph "VLAN 1000 storage network"
        NFS["NFS v4.2<br>nconnect=4<br>rsize/wsize=1MB"]
    end

    subgraph "NFS Clients"
        T1["tsys1 (9 mounts)"]
        T3["tsys3 (5 mounts)"]
        T5["tsys5 (3 mounts, NO nconnect)"]
        T6["tsys6 (9 mounts)"]
        T7["tsys7 (9 mounts)"]
        T9["tsys9 (5 mounts)"]
    end

    T4D2 --> NFS
    T4D5 --> NFS
    T4D1 --> NFS
    T4D3 --> NFS
    T5S1 --> NFS
    T5S2 --> NFS
    T5SSD --> NFS

    NFS --> T1
    NFS --> T3
    NFS --> T5
    NFS --> T6
    NFS --> T7
    NFS --> T9

NFS mount inventory (live)

Server Export tsys1 tsys3 tsys4 tsys5 tsys6 tsys7 tsys9
tsys4 D1 -- -- -- -- ✓ ✓ --
tsys4 D2 ✓ ✓ -- ✓ ✓ ✓ ✓
tsys4 D3 -- -- -- ✓ ✓ ✓ --
tsys4 D5 -- ✓ -- ✓ ✓ ✓ ✓
tsys5 S1 ✓ -- -- -- ✓ ✓ --
tsys5 S2 ✓ ✓ -- -- ✓ ✓ --
tsys5 S3 ✓ -- -- -- ✓ ✓ ✓
tsys5 S4 ✓ ✓ -- -- ✓ ✓ ✓
tsys5 T5-SSD -- ✓ -- -- ✓ ✓ ✓

(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. tsys5 = NFS server + 3 client mounts from tsys4, missing nconnect.)

7. Switch management access

Method Details
SNMP Community kn3lmgmt (read-only). All 4 switches respond on VLAN 1 IPs.
Console (conman) 7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at pfv-tsys4:7890 (Tailscale). Escape: &.
Console tool perf/scripts/conman-console.py (PTY-based driver, portable)
SNMP audit tool perf/scripts/snmp-switch-audit.py (config-driven)
Direct telnet telnet pfv-tsys4 200X -- ONLY when conmand stopped (conflicts). Use conman instead.

Console port assignments

Port Console name Switch ID_PATH (USB port)
2001 pfv-core-sw01 Dell PowerConnect 5448 (core, rack5) usb-0:1.5.4.4
2002 pfv-tor3-mgmt Neyland 24T (mgmt TOR, rack3) usb-0:1.6.3.1
2003 pfv-tor3-stor Neyland 24T (storage TOR, rack3) usb-0:1.6.3.3.2
2004 pfv-rrinfra-rtr Cisco router (rrinfra) usb-0:1.6.3.3.1
2005 pfv-r2-tor-top Rack 2 top-of-rack switch usb-0:1.6.3.3.3
2006 subodev-torsw Suborbital device TOR switch usb-0:1.5.4.1
2007 pfv-r2-sw Rack 2 old Dell switch usb-0:1.6.3.2
Reference in New Issue View Git Blame Copy Permalink