PFVCluster/proxmox/docs/AUDIT-2026-07-30.md

# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)

> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md)
> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are
> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).

**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)
+ switch SNMP/conman captures (~03:50–04:00 UTC).
**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`,
`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all
read-only. No changes made to any system.
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild)

---

## 1. Physical network topology

### 1.1 Diagram

```
  RACK 5 (core/data)                           RACK 3 (compute/RackRental)
  ┌─────────────────────────┐                  ┌──────────────────────────┐
  │                         │                  │                          │
  │  pfv-core-sw01          │    ch1 LAG       │  pfv-tor3-stor           │
  │  PowerConnect 5448      │◄───3×1G──────────►│  Neyland 24T (PC5324)   │
  │  (mgmt+storage VLAN)    │   (g13-15)       │  (storage TOR)           │
  │                         │   g16 UP, not    │                          │
  │  VLAN1 (mgmt)           │   in LAG; g17    │  g8-9  → tsys6 bond0     │
  │  VLAN1000 (storage)     │   DOWN           │  g10-11 → tsys7 bond0    │
  │                         │                  │  g20-23 → ch1 (3 active) │
  │  g8   → tsys4 nic0      │                  │                          │
  │  g19  → tsys5 nic0      │                  │  pfv-tor3-mgmt           │
  │  g31  → tsys4 enx (USB) │                  │  Neyland 24T (PC5324)    │
  │  g33-34 → tsys5 ch3     │                  │  (mgmt TOR)              │
  │  ch3 → tsys5 bond (1 ln)│                  │  g7 → tsys6 nic0         │
  │                         │                  │  g3 → tsys7 nic0         │
  │  tsys4 (storage+PBS)    │                  │                          │
  │  tsys5 (storage+sectest)│                  │  tsys6 (RackRental)      │
  │  tsys1 (infra)          │                  │  tsys7 (RackRental)      │
  │  tsys3 (compute)        │                  │                          │
  │  tsys9 (infra+compute)  │                  │                          │
  └─────────────────────────┘                  └──────────────────────────┘
```

### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK

| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? |
|----------|-------------------|--------------------|----------|
| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below |
| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG |
| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate |
| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 |
| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection |
| **VLAN** | access vlan 1000 | access vlan 1000 | OK |

**Three root causes of cross-rack underperformance:**

1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on
   `g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th
   physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps,
   not the intended 4).

2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4`
   (includes TCP/UDP source+dest ports — correct for NFS with nconnect).
   core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With
   layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one
   physical link**. Even though the clients use `nconnect=4` (4 TCP connections
   per mount), they share the same src/dst IP → same hash → **max 1 Gbps per
   server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.

3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the
   cross-rack trunk. `mode on` provides no link-failure detection, no partner
   validation, no dynamic negotiation. A dead cable stays "active" until
   manually discovered.

### 1.3 Per-host network state (ground truth)

| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP |
|------|------|----------|-------------|-------|------|---------------|-------|------------|
| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 |
| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 |
| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 |
| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 |
| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 |
| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 |
| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 |

**Key observations:**
- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously.
- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy.
- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is
  the LACP problem the operator has been struggling with. The switch side (core-sw01
  ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
- **tsys1 also uses a USB dongle** for storage.
- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5
  use layer2+3 (suboptimal for NFS multi-connection).

### 1.4 Switch inventory (SNMP + conman ground truth)

| Switch | IP | Model | Firmware | Role | Console |
|--------|----|-------|----------|------|---------|
| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 |
| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 |
| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 |
| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? |

**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c.
**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit
(udev symlink naming bug — singular vs plural — prevented consoles from working
after reboot).

### 1.5 Port error counters (core-sw01)

| Port | In Errors | Notes |
|------|-----------|-------|
| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. |
| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. |
| **g26** | **73** | Connected to a management-network device. |
| All others | 0 | Healthy. |

tor3-stor shows **zero errors** on all active ports.

---

## 2. Storage topology (CORRECTED — current ground truth)

### 2.1 Storage tier definitions

| Tier | Definition | Performance | Use case |
|------|-----------|-------------|----------|
| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads |
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives |
| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only |

### 2.2 tsys4 disk inventory (CHANGED since last audit)

| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes |
|--------|-------|------|-----------|-------|------------|------|-------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** |
| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. |
| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. |

**Changes from AUDIT-2026-07-29:**
- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB).
  D3 is now Tier 3 (was previously listed as unreliable USB SSD).
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox.
  Potential new Tier 2 storage.
- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup.
- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone.

### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)

| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes |
|--------|-------|------|-----------|-------|-----------|------|-------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. |

**Changes from AUDIT-2026-07-29:**
- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest-
  performance local storage in the fleet (true NVMe).
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal
  SATA. Not yet mounted/configured.
- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented
  out. /etc/exports still exports S3 (stale).
- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented.
  storage.cfg still has a broken `dir: S4` entry (points to root filesystem).
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate
  916GB" in old audit).
- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention.

### 2.4 NFS export map (current, verified)

| Server | Export | NFS ID | Disk | Tier | Used |
|--------|--------|--------|------|------|------|
| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% |
| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) |
| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% |
| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — |

### 2.5 NFS client mount status (nconnect)

| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash |
|------|-------------------|-------------------|-------------|------|
| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 |
| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 |
| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |

**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4).
**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client).

### 2.6 Per-spindle IOPS load (running VMs only)

| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes |
|---------|--------|-----------------|----------------|-------|------|-------|
| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 |
| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 |
| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) |
| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 |
| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. |
| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) |

**D2 still carries 13 running VMs** — the IOPS hotspot persists.

---

## 3. Host capacity (live ground truth, 2026-07-30)

| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy |
|------|-----|---------|-----|-------------|-------------|------|----------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% |

**Changes since AUDIT-2026-07-29:**
- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped.
- **tsys9 RAM 62%→69%** — approaching 80% goal.
- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit).
- **All other hosts healthy** — well under 80%.

---

## 4. Fleet consistency audit

### 4.1 PVE/kernel versions

| Host | PVE | Kernel | Status |
|------|-----|--------|--------|
| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** |
| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ |

### 4.2 Performance tuning state

| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect |
|------|----------|------------|--------|---------------|---------------|-----------|----------|
| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) |
| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A |
| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** |
| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** |
| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) |
| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** |
| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A |

**Inconsistencies:**
- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong
  (`throughput-performance` should be `network-throughput`). Missing nconnect.
- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere —
  expected, not a bug).
- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7).

---

## 5. VM inventory + config audit (running VMs only)

### 5.1 Per-VM disk + CPU + network settings

> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good
> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS.
> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled.

| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------|
| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 |
| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt |
| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | |
| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | |
| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | |
| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | |
| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | |
| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | |
| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server |
| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode |
| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) |
| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB |
| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM |
| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | |
| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | |
| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk |
| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | |
| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | |
| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | |
| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk |
| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk |
| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | |
| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | |
| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G |
| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | |

### 5.2 VM optimization findings

| Finding | VMs affected | Impact | Fix |
|---------|-------------|--------|-----|
| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |

---

## 6. Console infrastructure status (fixed during this audit)

**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with
`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
console ports failed until `setup.sh`'s manual fallback re-created symlinks.

**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
consoles now working (verified via conman-console.py).

**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
conmand systemd service from starting.

---

## 7. Findings summary (prioritized)

### Critical (cross-rack performance)

| # | Finding | Impact | Type |
|---|---------|--------|------|
| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |

### High (storage configuration)

| # | Finding | Impact | Type |
|---|---------|--------|------|
| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity |
| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config |

### Medium (fleet consistency)

| # | Finding | Impact | Type |
|---|---------|--------|------|
| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade |
| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning |
| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning |
| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning |
| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config |
| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup |

### Low (monitoring / investigation)

| # | Finding | Impact | Type |
|---|---------|--------|------|
| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate |
| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring |
| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config |
| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup |

---

## 8. Recommendations (for operator review — NO changes applied)

### 8.1 Switch changes (require maintenance window + operator approval)

1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes
   the 4th live cable (g16) part of the LAG.
2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor).
   This unlocks multi-link distribution for nconnect NFS traffic.
3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches
   for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor.
5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr)
   on core-sw01.

> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the
> highest-impact changes with the lowest risk. LACP conversion (#3) is optional
> if the static trunk is working reliably otherwise. All changes can be done
> read-only-verified via the conman + SNMP tooling in this repo.

### 8.2 Storage configuration (host-side, no switch changes needed)

1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`)
   per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact)
   and sectestbed scratch.
2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or
   similar. Offloads the 93%-full Samsung 860 PRO.
3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken
   `dir: S4` entry in storage.cfg.
4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or
   keep local-only for PBS scratch.

### 8.3 Fleet consistency + tuning (host-side)

1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5:
   - Fix rmem/wmem to 128MB (tsys4)
   - Fix tuned profile (tsys4 → network-throughput)
   - Add nconnect=4 to tsys5 NFS client mounts
   - Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS-
   backed qcow2 disks (requires VM stop/start).

### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md))

The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty
Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading
that didn't exist in the 07-29 audit. Detailed placement plan deferred to the
operator review of this document.