Files

T

mrcharles 0a8865de38 docs: full documentation suite — network, storage, hosts, VMs

Four new standalone reference documents with Mermaid diagrams (renderable
on Gitea), built from live ground-truth data gathered 2026-07-30:

- NETWORK-TOPOLOGY.md: physical topology diagram, complete switch port
  maps (core-sw01 + tor3-stor), VLAN/subnet tables, cross-rack trunk
  analysis with Mermaid, NFS data-flow diagram, per-host NIC reference
- STORAGE-ARCHITECTURE.md: tier definitions, all 14 physical disks
  inventoried with SMART/size/tier, NFS export→mount map with Mermaid,
  per-spindle VM placement (IOPS hotspot analysis), capacity summary
- HOST-REFERENCE.md: one-page-per-host for all 7 nodes with hardware,
  network config, storage, tuning state, role, and running VMs
- VM-INVENTORY.md: VM placement diagram, all 28 running VMs with
  disk/CPU/RAM/config details, optimization audit (cache mode, CPU type,
  qga, disk controller, k8s wnode sizing), stopped VM reference

Updated docmap.md with all four new entries.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

2026-07-30 23:10:38 -05:00

16 KiB

Raw Blame History

Network Topology

Complete physical + logical network reference for the PFVCluster. Data gathered 2026-07-30 via SNMP, LLDP, conman console captures, and direct host probes. All data is live ground truth — not documentation inherited.

Companion docs: STORAGE-ARCHITECTURE.md, HOST-REFERENCE.md, VM-INVENTORY.md. Findings/analysis: AUDIT-2026-07-30.md.

1. Physical topology diagram

graph TB
    subgraph "RACK 5 — Core / Data / Infra"
        CORE["pfv-core-sw01<br/>PowerConnect 5448<br/>192.168.0.12<br/>FW 2.0.0.46<br/>(mgmt + storage VLANs)"]

        TSYS1["pfv-tsys1<br/>i7-4770 Haswell<br/>Infra host"]
        TSYS3["pfv-tsys3<br/>Xeon E3-1535M v5<br/>Compute"]
        TSYS4["pfv-tsys4<br/>Xeon E3-1246 v3<br/>NFS Storage + PBS"]
        TSYS5["pfv-tsys5<br/>Xeon E5620 Westmere<br/>NFS Storage + Sandboxes"]
        TSYS9["pfv-tsys9<br/>i5-10500 Comet Lake<br/>Infra + Compute"]

        TSYS4_CONSOLE["conman + ser2net<br/>7 switch consoles<br/>(Tailscale 100.70.77.93:7890)"]
        TSYS1_PDU["Cyclades PM10i PDU<br/>powerman on tsys1<br/>(port 10101)"]
        TSYS1_UPS["APC Smart-UPS C 1500<br/>NUT on tsys1<br/>(port 3493)"]
    end

    subgraph "RACK 3 — Compute / RackRental"
        TOR3STOR["pfv-tor3-stor<br/>PowerConnect 5324 (Neyland 24T)<br/>192.168.0.9<br/>FW 2.0.1.4<br/>(storage VLAN)"]
        TOR3MGMT["pfv-tor3-mgmt<br/>PowerConnect 5324 (Neyland 24T)<br/>192.168.0.7<br/>(mgmt VLAN)"]

        TSYS6["pfv-tsys6<br/>2× Xeon E5530 Nehalem<br/>RackRental"]
        TSYS7["pfv-tsys7<br/>2× Xeon E5-2630 v2 Ivy Bridge<br/>RackRental"]
    end

    %% Cross-rack trunk
    CORE <--. "ch1 LAG: 3×1G active<br/>(g13,g14,g15)<br/>hash=layer-2-3<br/>mode=on (static)" .--> TOR3STOR
    TOR3STOR -. "ch1: 4×1G<br/>(g20,g21,g22,g23)<br/>hash=layer-2-3-4<br/>mode=on (static)" .-> CORE

    %% Rack 5 connections to core
    CORE --> TSYS1
    CORE --> TSYS3
    CORE --> TSYS4
    CORE --> TSYS5
    CORE --> TSYS9

    %% Rack 3 connections
    TOR3STOR --> TSYS6
    TOR3STOR --> TSYS7
    TOR3MGMT --> TSYS6
    TOR3MGMT --> TSYS7

How to read this diagram

The dashed line between core-sw01 and tor3-stor is the cross-rack trunk — the most critical link in the cluster. It carries all NFS storage traffic between rack-3 compute hosts (tsys6/7) and rack-5 storage hosts (tsys4/5). The trunk is a 4-port static LAG, but only 3 links are active due to a cable mismatch (see §3 below).

2. Subnet / VLAN map

VLAN	Subnet	Purpose	Where it lives
1 (default)	192.168.0.0/22	Management network	All switches, all hosts (vmbr0 DHCP)
1000	10.100.100.0/24	Storage network (NFS)	core-sw01 + tor3-stor only (trunked)
— (Tailscale)	100.x.y.z/32 (CGNAT)	Remote access / overlay	All hosts + select VMs

Management IPs (VLAN 1, via DHCP reservations):

Host	mgmt IP (vmbr0)	Tailscale IP
tsys1	192.168.3.11	100.121.189.98
tsys3	192.168.2.5	(connected)
tsys4	192.168.3.251	100.70.77.93
tsys5	192.168.3.179	(connected)
tsys6	192.168.3.169	100.73.35.111
tsys7	192.168.3.55	(connected)
tsys9	192.168.3.58	(connected)

Storage IPs (VLAN 1000, static):

Host	Storage IP	Interface
tsys1	10.100.100.1	datanet (nic1, USB dongle)
tsys3	10.100.100.3	datanet (enx8cae4ccda774, USB dongle)
tsys4	10.100.100.4	datanet (enx8cae4ccda926, USB dongle)
tsys5	10.100.100.5	storagenet (bond0 → nic1, single link)
tsys6	10.100.100.6	storagenet (bond0 → nic1+nic2, 2G LACP)
tsys7	10.100.100.7	datanet (bond0 → nic1+nic2, 2G LACP)
tsys9	10.100.100.9	datanet (enx9c69d36a5b6c, USB dongle)

Switch IPs (VLAN 1, static DHCP reservations):

Switch	IP	Model	Console
pfv-core-sw01	192.168.0.12	PowerConnect 5448	conman port 2001
pfv-tor3-stor	192.168.0.9	PowerConnect 5324 (Neyland 24T)	conman port 2003
pfv-tor3-mgmt	192.168.0.7	PowerConnect 5324 (Neyland 24T)	conman port 2002
(unidentified)	192.168.0.8	PowerConnect 5324 (Neyland 24T)	conman port 2005?

SNMP community: kn3lmgmt (read-only, all switches respond to SNMPv2c)

3. Cross-rack trunk detail (ch1) — the critical link

This trunk carries ALL NFS storage traffic between rack 3 (tsys6/7) and rack 5 (tsys4/5). It is the single most important network path in the cluster.

graph LR
    subgraph "core-sw01 (rack5)"
        G13["g13<br/>1G UP"]
        G14["g14<br/>1G UP"]
        G15["g15<br/>1G UP"]
        G16["g16<br/>1G UP<br/>NOT in LAG!"]
        G17["g17<br/>DOWN<br/>in config but<br/>no cable"]
    end

    subgraph "tor3-stor (rack3)"
        G20["g20<br/>1G UP"]
        G21["g21<br/>1G UP"]
        G22["g22<br/>1G UP"]
        G23["g23<br/>1G UP"]
    end

    G13 --- G20
    G14 --- G21
    G15 --- G22
    G16 -.- G23
    G17 -.->|no cable| G23

Configuration comparison

Property	core-sw01 (rack5)	tor3-stor (rack3)	Match?
Physical ports configured	g(13-15,17)	g(20-23)	No — asymmetric
Physical ports UP	g13, g14, g15, g16	g20, g21, g22, g23	—
Active in LAG	g13, g14, g15 (3)	g20, g21, g22, g23 (4)	No — 3 vs 4
Load-balance hash	layer-2-3 (MAC+IP)	layer-2-3-4 (MAC+IP+Port)	No — asymmetric
Channel mode	on (static)	on (static)	Yes (both should be LACP)
VLAN	access vlan 1000	access vlan 1000	Yes

Problems

g16/g17 cable mismatch. The 4th physical cable from tor3-stor lands on core-sw01 g16, but the config expects it on g17. g16 is UP (1G, full duplex) but is NOT in the LAG group. g17 is configured in the LAG but has no cable (DOWN). Result: only 3 of 4 links are active.
Hash policy asymmetry. core-sw01 hashes on layer-2-3 (src/dst MAC + IP). tor3-stor hashes on layer-2-3-4 (src/dst MAC + IP + TCP/UDP port). With layer-2-3 on core-sw01, all traffic between a given host pair hashes to one link regardless of how many TCP connections (nconnect) are used.
Static mode (on). Neither switch runs LACP. There is no link-failure detection — a dead cable stays in the LAG until manually discovered.

4. Complete switch port map

pfv-core-sw01 (PowerConnect 5448, 48 ports) — rack 5

Port	Speed	Status	VLAN	Description / Connected device
g1	1000	UP	1	pfvsvrpi (Raspberry Pi)
g4	100	UP	1	pfv-tsys4 (mgmt, secondary)
g5	—	UP	1	pfv-tsys1 (mgmt)
g6	100	UP	1	pfvsvrpi (secondary)
g8	1000	UP	1	pfv-tsys4-mgmt
g9	1000	UP	1	AP-Wallmount (UAP-AC-LR) ⚠ 2902 inErrors
g10	1000	UP	1	(unknown)
g11	1000	UP	1	(unknown)
g12	1000	UP	1	(unknown)
g13	1000	UP	1000 (ch1)	cross-rack trunk → tor3-stor g20
g14	1000	UP	1000 (ch1)	cross-rack trunk → tor3-stor g21
g15	1000	UP	1000 (ch1)	cross-rack trunk → tor3-stor g22
g16	1000	UP	1000	cross-rack trunk cable (NOT in LAG — should be g17)
g17	—	DOWN	1000 (ch1)	in LAG config but no cable
g19	1000	UP	1	pfv-tsys5-mgmt
g21	1000	UP	1	(unknown)
g23	100	UP	1	pfv-tsys9 (mgmt)
g26	1000	UP	1	AP-Tabletop ⚠ 73 inErrors
g27	1000	UP	1000	pfv-tsys3-stor (datanet)
g31	1000	UP	1000	pfv-tsys4-stor (datanet, USB)
g32	—	—	—	pfv-tsys4-storage (configured, DOWN)
g33	—	—	—	pfv-tsys5-stor (ch2, DOWN/Not Present)
g34	1000	UP	1000 (ch3)	pfv-tsys5-stor (bond0, 1 link active)
g38	1000	UP	1000	pfv-tsys9-stor (datanet) ⚠ 590 inErrors
g40	1000	UP	1000	pfv-tsys1-stor (datanet)
g41	1000	UP	1	(unknown)
g43	1000	UP	1	pfv-tsys3 (mgmt)
g44	1000	UP	1	(unknown)
g45-g48	—	DOWN	—	(Combo ports, unused)

Port-channels on core-sw01:

Channel	Speed	Status	Members	VLAN	Notes
ch1	1000	UP	g13-15 (active), g17 (inactive)	1000	Cross-rack trunk to tor3-stor
ch2	—	DOWN	g32-33 (non-candidate)	—	Unused
ch3	1000	UP	g34 (active)	1000	tsys5 storage bond (degraded, 1 of 2 links)

LAG hash: port-channel load-balance layer-2-3

pfv-tor3-stor (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3

Port	Speed	Status	VLAN	Description / Connected device
g1	1000	UP	1	(unknown — uplink to tor3-mgmt or router)
g2-g7	1000	UP	1	(unused but UP)
g8	1000	UP	1000 (ch2)	tsys6 stor nic1
g9	1000	UP	1000 (ch2)	tsys6 stor nic2
g10	1000	UP	1000 (ch3)	tsys7 stor nic1
g11	1000	UP	1000 (ch3)	tsys7 stor nic2
g13	1000	UP	—	tsys8-stor (STALE — tsys8 retired)
g14	1000	UP	—	tsys8-stor (STALE — tsys8 retired)
g15-g19	1000	UP	1000	(unused storage ports)
g20	1000	UP	1000 (ch1)	cross-rack trunk → core-sw01 g13
g21	1000	UP	1000 (ch1)	cross-rack trunk → core-sw01 g14
g22	1000	UP	1000 (ch1)	cross-rack trunk → core-sw01 g15
g23	1000	UP	1000 (ch1)	cross-rack trunk → core-sw01 g16
g24	—	DOWN	1	(spanning-tree disabled, portfast)

Port-channels on tor3-stor:

Channel	Speed	Status	Members	VLAN	Notes
ch1	1000	UP	g20-23 (`mode on`)	1000	Cross-rack trunk to core-sw01 (4×1G)
ch2	1000	UP	g8-9 (`mode auto`)	1000	tsys6 storage bond (2×1G LACP)
ch3	1000	UP	g10-11 (`mode auto`)	1000	tsys7 storage bond (2×1G LACP)
ch4	1000	UP	g13-14 (`mode on`)	1000	tsys8-stor (STALE — retired host)

LAG hash: port-channel load-balance layer-2-3-4

pfv-tor3-mgmt (PowerConnect 5324 / Neyland 24T, 24 ports) — rack 3

Port	Speed	Status	Connected device
g3	—	UP	tsys7 (mgmt nic0)
g7	—	UP	tsys6 (mgmt nic0)

(Full port map not captured — this switch carries only VLAN 1 management traffic. Low priority for storage performance.)

5. Per-host network interface reference

Host	mgmt NIC	Switch port	Storage NIC	Switch port	Storage bond	Bond hash	Storage link speed
tsys1	nic0	core g5	nic1 (onboard)	core g40	No bond (single NIC)	—	1G
tsys3	nic0	core g43	enx8cae4ccda774 (USB)	core g27	No bond (USB dongle)	—	1G
tsys4	nic0	core g8	enx8cae4ccda926 (USB)	core g31	bond0 (1 slave, USB)	layer2+3	1G
tsys5	nic0	core g19	nic1 + nic2	core ch3 (g34)	bond0 (1 of 2 slaves)	layer2+3	1G (degraded)
tsys6	nic0	tor3-mgmt g7	nic1 + nic2	tor3-stor g8/g9 (ch2)	bond0 (2 of 2 slaves)	layer3+4	2G LACP
tsys7	nic0	tor3-mgmt g3	nic1 + nic2	tor3-stor g10/g11 (ch3)	bond0 (2 of 2 slaves)	layer3+4	2G LACP
tsys9	nic0	core g23	enx9c69d36a5b6c (USB)	core g38	No bond (USB dongle)	—	1G

Key observations

tsys6/7 are the only hosts with working 2G LACP bonds (layer3+4 hash, 2 active slaves). All NFS traffic from rack 3 uses these bonds.
tsys1/3/4/9 use USB dongles for storage network — single 1G, no redundancy.
tsys5 bond0 is degraded — nic2 is DOWN (no cable connected). Only nic1 is active via core-sw01 ch3 (g34). This is the LACP problem the operator has been troubleshooting.
Bond hash inconsistency: tsys6/7 use layer3+4 (optimal for nconnect NFS), tsys4/5 use layer2+3 (suboptimal — same src/dst IP pair always hashes to one link even with nconnect's multiple TCP connections).

6. NFS data flow diagram

graph TB
    subgraph "Storage Servers (rack 5)"
        T4D2["tsys4 D2<br/>WDC Red 3TB<br/>Tier 2"]
        T4D5["tsys4 D5<br/>Hitachi 2TB<br/>Tier 2"]
        T4D1["tsys4 D1<br/>WD 1TB USB<br/>Tier 3"]
        T4D3["tsys4 D3<br/>Seagate 500GB USB<br/>Tier 3"]
        T5S1["tsys5 S1<br/>IronWolf 1TB<br/>Tier 2"]
        T5S2["tsys5 S2<br/>IronWolf 1TB<br/>Tier 2"]
        T5SSD["tsys5 SSD<br/>Samsung 860 PRO<br/>Tier 1 (93% full)"]
    end

    subgraph "VLAN 1000 storage network"
        NFS["NFS v4.2<br/>nconnect=4<br/>rsize/wsize=1MB"]
    end

    subgraph "NFS Clients"
        T1["tsys1 (9 mounts)"]
        T3["tsys3 (5 mounts)"]
        T5["tsys5 (3 mounts, NO nconnect)"]
        T6["tsys6 (9 mounts)"]
        T7["tsys7 (9 mounts)"]
        T9["tsys9 (5 mounts)"]
    end

    T4D2 --> NFS
    T4D5 --> NFS
    T4D1 --> NFS
    T4D3 --> NFS
    T5S1 --> NFS
    T5S2 --> NFS
    T5SSD --> NFS

    NFS --> T1
    NFS --> T3
    NFS --> T5
    NFS --> T6
    NFS --> T7
    NFS --> T9

NFS mount inventory (live)

Server	Export	tsys1	tsys3	tsys4	tsys5	tsys6	tsys7	tsys9
tsys4	D1	—	—	—	—	✓	✓	—
tsys4	D2	✓	✓	—	✓	✓	✓	✓
tsys4	D3	—	—	—	✓	✓	✓	—
tsys4	D5	—	✓	—	✓	✓	✓	✓
tsys5	S1	✓	—	—	—	✓	✓	—
tsys5	S2	✓	✓	—	—	✓	✓	—
tsys5	S3	✓	—	—	—	✓	✓	✓
tsys5	S4	✓	✓	—	—	✓	✓	✓
tsys5	T5-SSD	—	✓	—	—	✓	✓	✓

(✓ = NFS client mount active. tsys4 = NFS server only, no client mounts. tsys5 = NFS server + 3 client mounts from tsys4, missing nconnect.)

7. Switch management access

Method	Details
SNMP	Community `kn3lmgmt` (read-only). All 4 switches respond on VLAN 1 IPs.
Console (conman)	7 serial consoles via ser2net + conman on pfv-tsys4. Conman server at `pfv-tsys4:7890` (Tailscale). Escape: `&.`
Console tool	`perf/scripts/conman-console.py` (PTY-based driver, portable)
SNMP audit tool	`perf/scripts/snmp-switch-audit.py` (config-driven)
Direct telnet	`telnet pfv-tsys4 200X` — ONLY when conmand stopped (conflicts). Use conman instead.

Console port assignments

Port	Console name	Switch	ID_PATH (USB port)
2001	pfv-core-sw01	Dell PowerConnect 5448 (core, rack5)	usb-0:1.5.4.4
2002	pfv-tor3-mgmt	Neyland 24T (mgmt TOR, rack3)	usb-0:1.6.3.1
2003	pfv-tor3-stor	Neyland 24T (storage TOR, rack3)	usb-0:1.6.3.3.2
2004	pfv-rrinfra-rtr	Cisco router (rrinfra)	usb-0:1.6.3.3.1
2005	pfv-r2-tor-top	Rack 2 top-of-rack switch	usb-0:1.6.3.3.3
2006	subodev-torsw	Suborbital device TOR switch	usb-0:1.5.4.1
2007	pfv-r2-sw	Rack 2 old Dell switch	usb-0:1.6.3.2

16 KiB Raw Blame History Unescape Escape