docs(audit): full ground-truth audit — network, storage, compute, switches

Comprehensive end-to-end audit of the Proxmox cluster covering all three layers (network, storage, compute) with live data gathered from all 7 hosts + 4 switches via SNMP, conman consoles, and LLDP. Key findings documented in AUDIT-2026-07-30.md: Network (cross-rack NFS bottleneck root-caused): - core-sw01 LAG hash = layer-2-3 (no L4 ports) — NFS caps at 1 Gbps per host pair regardless of nconnect. tor3-stor already uses layer-2-3-4. - core-sw01 ch1 member mismatch: g16 is UP but not in LAG; g17 is DOWN. Only 3 of 4 cross-rack links active. - Both switches use static mode (no LACP) for the cross-rack trunk. - tsys5 bond0 degraded (1 of 2 slaves, nic2 down). Storage (major topology changes): - tsys5 new NVMe (CT500P1 500GB) + relocated SK hynix SSD (512GB) — both installed but NOT mounted/configured yet. - tsys4 D3 changed (SK hynix SSD → Seagate HDD, still USB tier-3). - tsys4 /mnt/albert (Hitachi 2TB internal) + /mnt/backup (WD 5TB USB) added. - tsys5 S3 unmounted (stale export), S4 gone (broken storage.cfg entry). - Samsung 860 PRO SSD at 93% full. Fleet consistency: - tsys4 behind on PVE 9.1.5/kernel 6.17 (fleet: 9.2.5/7.0.14). - tsys4/5 bond hash + nconnect + rmem/wmem inconsistent with tsys6/7. - 10+ VMs using cache=writethrough on NFS (slow synchronous writes). Gardening: updated STATUS.md, docmap.md, AGENTS.md key scripts table. Superseded AUDIT-2026-07-29 as the authoritative audit. 💘 Generated with Crush Assisted-by: Crush:glm-5.2
2026-07-30 23:01:43 -05:00
parent d9b32e4eef
commit 26de312c42
6 changed files with 497 additions and 56 deletions
@@ -0,0 +1,433 @@
+# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)
+
+> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md)
+> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are
+> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
+
+**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)
+ switch SNMP/conman captures (~03:50–04:00 UTC).
+**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`,
+`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all
+read-only. No changes made to any system.
+**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
+**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild)
+
+---
+
+## 1. Physical network topology
+
+### 1.1 Diagram
+
+```
+  RACK 5 (core/data)                           RACK 3 (compute/RackRental)
+  ┌─────────────────────────┐                  ┌──────────────────────────┐
+  │                         │                  │                          │
+  │  pfv-core-sw01          │    ch1 LAG       │  pfv-tor3-stor           │
+  │  PowerConnect 5448      │◄───3×1G──────────►│  Neyland 24T (PC5324)   │
+  │  (mgmt+storage VLAN)    │   (g13-15)       │  (storage TOR)           │
+  │                         │   g16 UP, not    │                          │
+  │  VLAN1 (mgmt)           │   in LAG; g17    │  g8-9  → tsys6 bond0     │
+  │  VLAN1000 (storage)     │   DOWN           │  g10-11 → tsys7 bond0    │
+  │                         │                  │  g20-23 → ch1 (3 active) │
+  │  g8   → tsys4 nic0      │                  │                          │
+  │  g19  → tsys5 nic0      │                  │  pfv-tor3-mgmt           │
+  │  g31  → tsys4 enx (USB) │                  │  Neyland 24T (PC5324)    │
+  │  g33-34 → tsys5 ch3     │                  │  (mgmt TOR)              │
+  │  ch3 → tsys5 bond (1 ln)│                  │  g7 → tsys6 nic0         │
+  │                         │                  │  g3 → tsys7 nic0         │
+  │  tsys4 (storage+PBS)    │                  │                          │
+  │  tsys5 (storage+sectest)│                  │  tsys6 (RackRental)      │
+  │  tsys1 (infra)          │                  │  tsys7 (RackRental)      │
+  │  tsys3 (compute)        │                  │                          │
+  │  tsys9 (infra+compute)  │                  │                          │
+  └─────────────────────────┘                  └──────────────────────────┘
+```
+
+### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK
+
+| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? |
+|----------|-------------------|--------------------|----------|
+| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below |
+| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG |
+| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate |
+| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 |
+| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection |
+| **VLAN** | access vlan 1000 | access vlan 1000 | OK |
+
+**Three root causes of cross-rack underperformance:**
+
+1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on
+   `g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th
+   physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps,
+   not the intended 4).
+
+2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4`
+   (includes TCP/UDP source+dest ports — correct for NFS with nconnect).
+   core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With
+   layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one
+   physical link**. Even though the clients use `nconnect=4` (4 TCP connections
+   per mount), they share the same src/dst IP → same hash → **max 1 Gbps per
+   server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.
+
+3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the
+   cross-rack trunk. `mode on` provides no link-failure detection, no partner
+   validation, no dynamic negotiation. A dead cable stays "active" until
+   manually discovered.
+
+### 1.3 Per-host network state (ground truth)
+
+| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP |
+|------|------|----------|-------------|-------|------|---------------|-------|------------|
+| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 |
+| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 |
+| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 |
+| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 |
+| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 |
+| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 |
+| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 |
+
+**Key observations:**
+- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously.
+- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy.
+- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is
+  the LACP problem the operator has been struggling with. The switch side (core-sw01
+  ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
+- **tsys1 also uses a USB dongle** for storage.
+- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5
+  use layer2+3 (suboptimal for NFS multi-connection).
+
+### 1.4 Switch inventory (SNMP + conman ground truth)
+
+| Switch | IP | Model | Firmware | Role | Console |
+|--------|----|-------|----------|------|---------|
+| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 |
+| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 |
+| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 |
+| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? |
+
+**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c.
+**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit
+(udev symlink naming bug — singular vs plural — prevented consoles from working
+after reboot).
+
+### 1.5 Port error counters (core-sw01)
+
+| Port | In Errors | Notes |
+|------|-----------|-------|
+| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. |
+| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. |
+| **g26** | **73** | Connected to a management-network device. |
+| All others | 0 | Healthy. |
+
+tor3-stor shows **zero errors** on all active ports.
+
+---
+
+## 2. Storage topology (CORRECTED — current ground truth)
+
+### 2.1 Storage tier definitions
+
+| Tier | Definition | Performance | Use case |
+|------|-----------|-------------|----------|
+| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads |
+| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives |
+| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only |
+
+### 2.2 tsys4 disk inventory (CHANGED since last audit)
+
+| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes |
+|--------|-------|------|-----------|-------|------------|------|-------|
+| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. |
+| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. |
+| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** |
+| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
+| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. |
+| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). |
+| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. |
+
+**Changes from AUDIT-2026-07-29:**
+- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB).
+  D3 is now Tier 3 (was previously listed as unreliable USB SSD).
+- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox.
+  Potential new Tier 2 storage.
+- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup.
+- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone.
+
+### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)
+
+| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes |
+|--------|-------|------|-----------|-------|-----------|------|-------|
+| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
+| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
+| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
+| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). |
+| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. |
+| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. |
+| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. |
+
+**Changes from AUDIT-2026-07-29:**
+- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest-
+  performance local storage in the fleet (true NVMe).
+- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal
+  SATA. Not yet mounted/configured.
+- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented
+  out. /etc/exports still exports S3 (stale).
+- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented.
+  storage.cfg still has a broken `dir: S4` entry (points to root filesystem).
+- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate
+  916GB" in old audit).
+- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention.
+
+### 2.4 NFS export map (current, verified)
+
+| Server | Export | NFS ID | Disk | Tier | Used |
+|--------|--------|--------|------|------|------|
+| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% |
+| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) |
+| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% |
+| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) |
+| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) |
+| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) |
+| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — |
+
+### 2.5 NFS client mount status (nconnect)
+
+| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash |
+|------|-------------------|-------------------|-------------|------|
+| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 |
+| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 |
+| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
+| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
+
+**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4).
+**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client).
+
+### 2.6 Per-spindle IOPS load (running VMs only)
+
+| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes |
+|---------|--------|-----------------|----------------|-------|------|-------|
+| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 |
+| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 |
+| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) |
+| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 |
+| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. |
+| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
+| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
+| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) |
+
+**D2 still carries 13 running VMs** — the IOPS hotspot persists.
+
+---
+
+## 3. Host capacity (live ground truth, 2026-07-30)
+
+| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy |
+|------|-----|---------|-----|-------------|-------------|------|----------|
+| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% |
+| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% |
+| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% |
+| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% |
+| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% |
+| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% |
+| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% |
+
+**Changes since AUDIT-2026-07-29:**
+- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped.
+- **tsys9 RAM 62%→69%** — approaching 80% goal.
+- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit).
+- **All other hosts healthy** — well under 80%.
+
+---
+
+## 4. Fleet consistency audit
+
+### 4.1 PVE/kernel versions
+
+| Host | PVE | Kernel | Status |
+|------|-----|--------|--------|
+| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** |
+| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
+
+### 4.2 Performance tuning state
+
+| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect |
+|------|----------|------------|--------|---------------|---------------|-----------|----------|
+| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) |
+| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A |
+| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** |
+| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** |
+| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) |
+| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** |
+| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A |
+
+**Inconsistencies:**
+- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong
+  (`throughput-performance` should be `network-throughput`). Missing nconnect.
+- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere —
+  expected, not a bug).
+- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7).
+
+---
+
+## 5. VM inventory + config audit (running VMs only)
+
+### 5.1 Per-VM disk + CPU + network settings
+
+> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good
+> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS.
+> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled.
+
+| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes |
+|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------|
+| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 |
+| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt |
+| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | |
+| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
+| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
+| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | |
+| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | |
+| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | |
+| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | |
+| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | |
+| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server |
+| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode |
+| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) |
+| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB |
+| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM |
+| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | |
+| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | |
+| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk |
+| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | |
+| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | |
+| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | |
+| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk |
+| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk |
+| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | |
+| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | |
+| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G |
+| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | |
+
+### 5.2 VM optimization findings
+
+| Finding | VMs affected | Impact | Fix |
+|---------|-------------|--------|-----|
+| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
+| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
+| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
+| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
+| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |
+
+---
+
+## 6. Console infrastructure status (fixed during this audit)
+
+**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with
+`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
+`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
+console ports failed until `setup.sh`'s manual fallback re-created symlinks.
+
+**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
+all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
+consoles now working (verified via conman-console.py).
+
+**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
+conmand systemd service from starting.
+
+---
+
+## 7. Findings summary (prioritized)
+
+### Critical (cross-rack performance)
+
+| # | Finding | Impact | Type |
+|---|---------|--------|------|
+| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
+| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
+| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
+| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |
+
+### High (storage configuration)
+
+| # | Finding | Impact | Type |
+|---|---------|--------|------|
+| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
+| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
+| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
+| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity |
+| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config |
+
+### Medium (fleet consistency)
+
+| # | Finding | Impact | Type |
+|---|---------|--------|------|
+| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade |
+| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning |
+| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning |
+| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning |
+| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config |
+| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup |
+
+### Low (monitoring / investigation)
+
+| # | Finding | Impact | Type |
+|---|---------|--------|------|
+| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate |
+| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring |
+| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config |
+| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup |
+
+---
+
+## 8. Recommendations (for operator review — NO changes applied)
+
+### 8.1 Switch changes (require maintenance window + operator approval)
+
+1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes
+   the 4th live cable (g16) part of the LAG.
+2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor).
+   This unlocks multi-link distribution for nconnect NFS traffic.
+3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches
+   for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
+4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor.
+5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr)
+   on core-sw01.
+
+> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the
+> highest-impact changes with the lowest risk. LACP conversion (#3) is optional
+> if the static trunk is working reliably otherwise. All changes can be done
+> read-only-verified via the conman + SNMP tooling in this repo.
+
+### 8.2 Storage configuration (host-side, no switch changes needed)
+
+1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`)
+   per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact)
+   and sectestbed scratch.
+2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or
+   similar. Offloads the 93%-full Samsung 860 PRO.
+3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken
+   `dir: S4` entry in storage.cfg.
+4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or
+   keep local-only for PBS scratch.
+
+### 8.3 Fleet consistency + tuning (host-side)
+
+1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
+2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5:
+   - Fix rmem/wmem to 128MB (tsys4)
+   - Fix tuned profile (tsys4 → network-throughput)
+   - Add nconnect=4 to tsys5 NFS client mounts
+   - Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
+3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS-
+   backed qcow2 disks (requires VM stop/start).
+
+### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md))
+
+The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty
+Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading
+that didn't exist in the 07-29 audit. Detailed placement plan deferred to the
+operator review of this document.