diff --git a/AGENTS.md b/AGENTS.md index b1f3f5c..bfa57bb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -99,6 +99,10 @@ vendor/ Vendored KNELShellFramework | [`console/setup.sh`](console/setup.sh) | Configure serial console access via ser2net + conman | | [`ups/setup.sh`](ups/setup.sh) | Configure NUT (Network UPS Tools) for UPS monitoring | | [`perf/deploy-tuning.sh`](perf/deploy-tuning.sh) | Deploy perf tunings | +| [`perf/scripts/probe-storage.sh`](perf/scripts/probe-storage.sh) | Read-only disk/mount/export/SMART audit | +| [`perf/scripts/probe-network.sh`](perf/scripts/probe-network.sh) | Read-only NIC/bond/LLDP/NFS audit | +| [`perf/scripts/conman-console.py`](perf/scripts/conman-console.py) | Drive switch consoles via conman (PTY-based) | +| [`perf/scripts/snmp-switch-audit.py`](perf/scripts/snmp-switch-audit.py) | SNMP-based switch inventory (interfaces, LLDP, LAG, VLANs) | ## Key Docs diff --git a/STATUS.md b/STATUS.md index 099e16c..add566a 100644 --- a/STATUS.md +++ b/STATUS.md @@ -3,12 +3,14 @@ > **Human read-only. Agents maintain this file automatically after each work > session.** Do not edit by hand — the next agent run will overwrite it. -> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — HA NUT integration live +> **Last updated:** 2026-07-30 by Crush (GLM-5.2) — full ground-truth audit (network+storage+compute) > -> **Capacity targets are now codified** in +> **Current authoritative audit:** [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) +> — network topology, switch LAG configs, corrected storage topology, VM config +> audit, fleet consistency, cross-rack NFS root-cause analysis. +> Capacity targets are codified in > [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md): 80% RAM -> steady-state, ~50% idle CPU. The current placement analysis lives in -> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md). +> steady-state, ~50% idle CPU. ## Current State: STABLE @@ -36,15 +38,19 @@ infrastructure, Proxmox cluster ops, and k8s control plane. ### Proxmox Cluster Ops - [x] 5 of 7 hosts fully performance-tuned (tsys1/3/6/7/9) -- [x] Fleet audit refreshed 2026-07-29 (consolidated, storage-corrected) -- [x] **Memory goal (≤80% actual) met fleet-wide** — highest host is tsys1 at 73% -- [x] Capacity goals codified (80% RAM / ~50% idle CPU + workload model) -- [x] **Storage topology corrected:** `local-nonprod` = S1 disk (not separate); per-host storage.cfg -- [x] cnode5 (706) deleted; cnode4 (601) stopped (delete pending) -- [x] **tsys6 S1 typo + tsys9 missing S1 — both FIXED by operator** (verified: 8/8 NFS stores active on both) +- [x] **Full ground-truth audit 2026-07-30** (network + storage + compute + switches) +- [x] **Cross-rack NFS root cause identified:** core-sw01 LAG hash=layer-2-3 (no L4), + g16/g17 cable mismatch (3 of 4 links active), static mode (no LACP) +- [x] **tsys4/tsys5 storage changes documented:** new NVMe (CT500P1 500GB), relocated + SK hynix SSD, D3 changed to USB HDD, /mnt/albert + /mnt/backup added +- [x] **Console symlink bug fixed:** udev rule singular→plural mismatch broke all 7 + console ports after reboot +- [x] **Portable audit tooling built:** probe-storage.sh, probe-network.sh, + conman-console.py, snmp-switch-audit.py (AGPLv3-friendly, config-driven) +- [ ] **Memory goal (≤80% actual): PASS fleet-wide** — highest is tsys9 at 69% - [ ] CPU ~50%-idle goal: fleet idles 0-5% (under-utilized; chase via workloads, not hardware) -- [ ] tsys4: blocked on PCIe NIC + RAM install -- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install; **also single-CPU** (CPU2 socket empty) +- [ ] tsys4: PVE 9.1.5 (behind fleet 9.2.5), needs upgrade +- [ ] tsys5: NVMe + SK hynix SSD installed but NOT configured - [ ] tsys2: pending rebuild from Win10 to Proxmox ### Kubernetes Control Plane (k3s HA — LIVE) @@ -131,50 +137,51 @@ infrastructure, Proxmox cluster ops, and k8s control plane. | Issue | Impact | Status | |-------|--------|--------| -| **2 of 3 k3s cnodes on tsys4** | tsys4 failure = etcd quorum lost (1 of 3) | Accepted — fundamental constraint with 3 cnodes + 2 NFS servers. Revisit after Friday (local-SSD boot for one cnode). **Deferred storage concern.** | -| **tsys1/5/9 RAM-overcommitted (alloc)** | 115%/134%/129% allocation; **actual usage 73%/37%/62% — all under 80% goal** | Migrate NFS-backed VMs to tsys6/7 (117/171 GB free actual). See [AUDIT-2026-07-29](proxmox/docs/AUDIT-2026-07-29.md) §11 | -| **tsys5 is single-CPU** | CPU2 socket unpopulated; half the documented CPU capacity | Consider seating a spare LGA771 Xeon (also activates CPU2 DIMM branch) | -| **cnode4/5 stale VMs** | cnode4 (601) stopped; cnode5 (706) deleted | cnode4 delete pending; cnode5 done ✓ | -| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; solar-gate idle PowerEdge; grow via workloads | -| **D2 IOPS hotspot** | 17 of 26 NFS-backed VMs on one spindle; S1/S4 underused | Phase 2 disk migrations: D2→S2/S3/S4/D5 (see audit §11) | -| Technitium AXFR uses port 53 (occupied by Pi-hole) | Zone transfer via rsync instead of native AXFR | Workaround in place | -| tsys4/5 hardware pending | Perf tuning incomplete on 2 hosts | Friday physical install | +| **core-sw01 LAG hash = layer-2-3** | NFS cross-rack capped at 1 Gbps per host pair | Fix: change to layer-2-3-4 (switch change, pending approval) | +| **core-sw01 ch1 g16/g17 mismatch** | Only 3 of 4 cross-rack links active | Fix: change member list g(13-15,17)→g(13-16) | +| **Cross-rack trunk uses `mode on`** | No LACP failure detection | Fix: change to `mode active` (both switches) | +| **tsys5 bond0 degraded** (1 of 2 slaves) | Single link, no redundancy | Physical: 2nd ethernet cable needed | +| **tsys5 NVMe + SSD not configured** | Best local storage unused | Config: mount + add to Proxmox storage.cfg | +| **D2 carries 13 running VMs** | IOPS hotspot persists | Migrate to S2/S3/new disks | +| **tsys4 PVE 9.1.5** | Behind fleet (9.2.5) | Upgrade needed | +| **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Offload to new SK hynix SSD | +| **Port errors on core-sw01** | g9:2902, g38:590, g26:73 inErr | Investigate CRC/cable | +| **tsys5 is single-CPU** | CPU2 socket unpopulated | Consider seating spare LGA771 Xeon | +| **Fleet CPU 0-5% idle** | Misses ~50%-idle utilization goal | Consolidate baseline; grow via workloads | ## Pending (next session priorities) -> **Full phased action checklist with current/target inventory is in** -> [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) §11. -> Summary below. SSD/NVMe tiering is deferred to post-Friday. +> **Full audit with findings + recommendations is in** +> [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md). -**Phase 0 — Storage config fixes — DONE ✓** (tsys6 S1 typo, tsys9 S1 missing) +**Awaiting operator review of AUDIT-2026-07-30 before executing any fixes.** -**Phase 1 — Cleanup:** -- Delete cnode4 (601) on tsys6 (stopped, stale) -- cnode5 (706) already deleted ✓ +**Switch changes (need approval):** +- Fix core-sw01 ch1 member: g(13-15,17) → g(13-16) [restores 4th link] +- Fix core-sw01 LAG hash: layer-2-3 → layer-2-3-4 [unlocks multi-link NFS] +- Convert trunk to LACP: mode on → mode active [failure detection] +- Remove stale tsys8 config from tor3-stor (ch4/g13/g14) -**Phase 2 — Disk migrations off D2 (IOPS rebalancing, 6 moves):** -- wnode-tsys5 (509): D2→S4 · tsys-awx (600): D2→D5 · hfnoc-uisp (702): D2→S4 -- rr-middleware (703): D2→S4 · TCTC (704): D2→S3 · cnode2 (705): D2→S2 - -**Phase 3 — Host + disk migrations (RAM/CPU relief, 6 VMs):** -- tsys1→tsys7: librenms(104), pdm(105), tailscale-router(109), kali-tsys(114), workbench(117) -- tsys9→tsys6: tsys-siem(901) - -**Phase 4 — Grow wnode-tsys9:** 4c/4GB → 6c/8GB (after siem leaves tsys9) - -**Phase 5 — Join wnodes to k3s** (after migrations free resources) +**Host-side changes (no switch access needed):** +- Mount + configure tsys5 NVMe (CT500P1 500GB) as LVM-thin +- Mount + configure tsys5 SK hynix SSD (512GB, relocated from tsys4) +- Clean stale S3/S4 exports + storage.cfg on tsys5 +- Upgrade tsys4 to PVE 9.2.5 / kernel 7.0.14 +- Apply apply-tunings.sh to tsys4/5 (nconnect, bond hash, rmem/wmem) +- Audit VM cache modes (writethrough → writeback for NFS) **Other:** -- **Friday maintenance:** tsys4 (PCIe NIC + RAM), tsys5 (2nd CPU candidate, NVMe, 2nd ethernet) -- **PDU:** Rename outlets in powerman.conf (Friday onsite) -- **k8s deferred:** ETL tooling, HPC scheduler, vcluster policy, solar-aware scale-out -- **UPS:** APC Smart-UPS C 1500 operational via NUT on pfv-tsys1. HA NUT integration live (battery charge, status sensors). No ups.load over USB HID (firmware limitation, confirmed via apcupsd test). Tripp Lite blocked on HID report descriptor (USB hub/cable issue — try direct motherboard port). +- Investigate core-sw01 port errors (g9/g38/g26) +- k8s: join wnodes, label by workload class +- k8s deferred: ETL tooling, HPC scheduler, vcluster policy ## Infrastructure Summary | Component | Details | |-----------|---------| -| Proxmox hosts | 7 standalone PVE, managed via PDM | +| Proxmox hosts | 7 standalone PVE, managed via PDM (tsys8 retired) | +| Storage servers | tsys4 (NFS: D1/D2/D3/D5 + /mnt/albert + /mnt/backup), tsys5 (NFS: S1/S2 + local SSD + new NVMe unconfigured) | +| Switches | core-sw01 (PowerConnect 5448, rack5), tor3-stor/mgmt (PowerConnect 5324, rack3), + 1 unidentified (192.168.0.8) | | DNS primary | pfv-netinfra-01 (192.168.3.252) — Technitium + Pi-hole | | DNS secondary | pfv-netinfra-02 (192.168.3.253) — Technitium + Pi-hole | | DNS production | tailscale-router (read-only source of truth) | diff --git a/docs/docmap.md b/docs/docmap.md index 8f7c6d8..e3f04f3 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -3,7 +3,7 @@ > **Index of all documentation in this repo.** Agents must update this file > whenever a doc is added, removed, or substantively changed. -> **Last updated:** 2026-07-30 (UPS/NUT) +> **Last updated:** 2026-07-30 (full ground-truth audit: network+storage+compute) ## Kubernetes Architecture ([`k8s/`](k8s/)) @@ -24,7 +24,8 @@ Fleet operations, hardware, performance tuning, storage architecture. |----------|-------------|---------------| | [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/docs/CAPACITY-GOALS.md`](proxmox/docs/CAPACITY-GOALS.md) | Authoritative capacity targets (80% RAM, ~50% idle CPU) + workload placement model (compute / RackRental / infra) | 2026-07-29 | -| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | **Current & only audit.** Consolidated RAM/CPU + HDD disk placement with corrected storage topology, current/target inventory, action checklist | 2026-07-29 | +| [`proxmox/docs/AUDIT-2026-07-29.md`](proxmox/docs/AUDIT-2026-07-29.md) | Prior audit (consolidated RAM/CPU + HDD disk placement). **Superseded by AUDIT-2026-07-30** | 2026-07-29 | +| [`proxmox/docs/AUDIT-2026-07-30.md`](proxmox/docs/AUDIT-2026-07-30.md) | **Current & authoritative.** Full ground-truth: network topology + switch LAG configs, corrected storage topology (tsys4/5 changes + new NVMe/SSD), VM config audit, fleet consistency, cross-rack NFS root-cause analysis | 2026-07-30 | | [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | | [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | diff --git a/perf/scripts/snmp-switch-audit.py b/perf/scripts/snmp-switch-audit.py index 1075f48..26c9e99 100644 --- a/perf/scripts/snmp-switch-audit.py +++ b/perf/scripts/snmp-switch-audit.py @@ -218,8 +218,7 @@ def main(): help=f"output dir (env: OUTPUT_DIR)") args = ap.parse_args() - global COMMUNITY - COMMUNITY = args.community + community = args.community ips = args.switches if not ips: @@ -233,6 +232,7 @@ def main(): all_data = [] for ip in ips: + globals()["COMMUNITY"] = community data = audit_switch(ip.strip()) all_data.append(data) print_switch(data) diff --git a/proxmox/docs/AUDIT-2026-07-30.md b/proxmox/docs/AUDIT-2026-07-30.md new file mode 100644 index 0000000..b033070 --- /dev/null +++ b/proxmox/docs/AUDIT-2026-07-30.md @@ -0,0 +1,433 @@ +# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute) + +> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md) +> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are +> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md). + +**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC) ++ switch SNMP/conman captures (~03:50–04:00 UTC). +**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`, +`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all +read-only. No changes made to any system. +**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 +**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild) + +--- + +## 1. Physical network topology + +### 1.1 Diagram + +``` + RACK 5 (core/data) RACK 3 (compute/RackRental) + ┌─────────────────────────┐ ┌──────────────────────────┐ + │ │ │ │ + │ pfv-core-sw01 │ ch1 LAG │ pfv-tor3-stor │ + │ PowerConnect 5448 │◄───3×1G──────────►│ Neyland 24T (PC5324) │ + │ (mgmt+storage VLAN) │ (g13-15) │ (storage TOR) │ + │ │ g16 UP, not │ │ + │ VLAN1 (mgmt) │ in LAG; g17 │ g8-9 → tsys6 bond0 │ + │ VLAN1000 (storage) │ DOWN │ g10-11 → tsys7 bond0 │ + │ │ │ g20-23 → ch1 (3 active) │ + │ g8 → tsys4 nic0 │ │ │ + │ g19 → tsys5 nic0 │ │ pfv-tor3-mgmt │ + │ g31 → tsys4 enx (USB) │ │ Neyland 24T (PC5324) │ + │ g33-34 → tsys5 ch3 │ │ (mgmt TOR) │ + │ ch3 → tsys5 bond (1 ln)│ │ g7 → tsys6 nic0 │ + │ │ │ g3 → tsys7 nic0 │ + │ tsys4 (storage+PBS) │ │ │ + │ tsys5 (storage+sectest)│ │ tsys6 (RackRental) │ + │ tsys1 (infra) │ │ tsys7 (RackRental) │ + │ tsys3 (compute) │ │ │ + │ tsys9 (infra+compute) │ │ │ + └─────────────────────────┘ └──────────────────────────┘ +``` + +### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK + +| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? | +|----------|-------------------|--------------------|----------| +| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below | +| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG | +| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate | +| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 | +| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection | +| **VLAN** | access vlan 1000 | access vlan 1000 | OK | + +**Three root causes of cross-rack underperformance:** + +1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on + `g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th + physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps, + not the intended 4). + +2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4` + (includes TCP/UDP source+dest ports — correct for NFS with nconnect). + core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With + layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one + physical link**. Even though the clients use `nconnect=4` (4 TCP connections + per mount), they share the same src/dst IP → same hash → **max 1 Gbps per + server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps. + +3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the + cross-rack trunk. `mode on` provides no link-failure detection, no partner + validation, no dynamic negotiation. A dead cable stays "active" until + manually discovered. + +### 1.3 Per-host network state (ground truth) + +| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP | +|------|------|----------|-------------|-------|------|---------------|-------|------------| +| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 | +| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 | +| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 | +| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 | +| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 | +| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 | +| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 | + +**Key observations:** +- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously. +- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy. +- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is + the LACP problem the operator has been struggling with. The switch side (core-sw01 + ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave. +- **tsys1 also uses a USB dongle** for storage. +- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5 + use layer2+3 (suboptimal for NFS multi-connection). + +### 1.4 Switch inventory (SNMP + conman ground truth) + +| Switch | IP | Model | Firmware | Role | Console | +|--------|----|-------|----------|------|---------| +| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 | +| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 | +| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 | +| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? | + +**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c. +**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit +(udev symlink naming bug — singular vs plural — prevented consoles from working +after reboot). + +### 1.5 Port error counters (core-sw01) + +| Port | In Errors | Notes | +|------|-----------|-------| +| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. | +| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. | +| **g26** | **73** | Connected to a management-network device. | +| All others | 0 | Healthy. | + +tor3-stor shows **zero errors** on all active ports. + +--- + +## 2. Storage topology (CORRECTED — current ground truth) + +### 2.1 Storage tier definitions + +| Tier | Definition | Performance | Use case | +|------|-----------|-------------|----------| +| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads | +| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives | +| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only | + +### 2.2 tsys4 disk inventory (CHANGED since last audit) + +| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes | +|--------|-------|------|-----------|-------|------------|------|-------| +| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. | +| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. | +| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** | +| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. | +| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. | +| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). | +| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. | + +**Changes from AUDIT-2026-07-29:** +- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB). + D3 is now Tier 3 (was previously listed as unreliable USB SSD). +- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox. + Potential new Tier 2 storage. +- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup. +- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone. + +### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD) + +| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes | +|--------|-------|------|-----------|-------|-----------|------|-------| +| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. | +| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. | +| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. | +| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). | +| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. | +| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. | +| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. | + +**Changes from AUDIT-2026-07-29:** +- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest- + performance local storage in the fleet (true NVMe). +- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal + SATA. Not yet mounted/configured. +- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented + out. /etc/exports still exports S3 (stale). +- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented. + storage.cfg still has a broken `dir: S4` entry (points to root filesystem). +- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate + 916GB" in old audit). +- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention. + +### 2.4 NFS export map (current, verified) + +| Server | Export | NFS ID | Disk | Tier | Used | +|--------|--------|--------|------|------|------| +| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% | +| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) | +| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% | +| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) | +| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) | +| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) | +| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — | + +### 2.5 NFS client mount status (nconnect) + +| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash | +|------|-------------------|-------------------|-------------|------| +| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 | +| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 | +| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 | +| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 | + +**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4). +**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client). + +### 2.6 Per-spindle IOPS load (running VMs only) + +| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes | +|---------|--------|-----------------|----------------|-------|------|-------| +| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 | +| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 | +| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) | +| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 | +| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. | +| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty | +| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty | +| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) | + +**D2 still carries 13 running VMs** — the IOPS hotspot persists. + +--- + +## 3. Host capacity (live ground truth, 2026-07-30) + +| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy | +|------|-----|---------|-----|-------------|-------------|------|----------| +| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% | +| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% | +| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% | +| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% | +| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% | +| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% | +| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% | + +**Changes since AUDIT-2026-07-29:** +- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped. +- **tsys9 RAM 62%→69%** — approaching 80% goal. +- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit). +- **All other hosts healthy** — well under 80%. + +--- + +## 4. Fleet consistency audit + +### 4.1 PVE/kernel versions + +| Host | PVE | Kernel | Status | +|------|-----|--------|--------| +| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ | +| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ | +| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** | +| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ | +| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ | +| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ | +| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ | + +### 4.2 Performance tuning state + +| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect | +|------|----------|------------|--------|---------------|---------------|-----------|----------| +| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) | +| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A | +| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** | +| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** | +| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) | +| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** | +| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A | + +**Inconsistencies:** +- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong + (`throughput-performance` should be `network-throughput`). Missing nconnect. +- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere — + expected, not a bug). +- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7). + +--- + +## 5. VM inventory + config audit (running VMs only) + +### 5.1 Per-VM disk + CPU + network settings + +> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good +> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS. +> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled. + +| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes | +|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------| +| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 | +| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt | +| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | | +| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | | +| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | | +| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | | +| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | | +| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | | +| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | | +| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | | +| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server | +| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode | +| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) | +| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB | +| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM | +| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | | +| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | | +| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk | +| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | | +| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | | +| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | | +| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk | +| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk | +| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | | +| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | | +| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G | +| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | | + +### 5.2 VM optimization findings + +| Finding | VMs affected | Impact | Fix | +|---------|-------------|--------|-----| +| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) | +| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier | +| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM | +| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting | +| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide | + +--- + +## 6. Console infrastructure status (fixed during this audit) + +**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with +`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened +`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7 +console ports failed until `setup.sh`'s manual fallback re-created symlinks. + +**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created +all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7 +consoles now working (verified via conman-console.py). + +**Also fixed:** stale conmand process (pid 988) holding config lock, preventing +conmand systemd service from starting. + +--- + +## 7. Findings summary (prioritized) + +### Critical (cross-rack performance) + +| # | Finding | Impact | Type | +|---|---------|--------|------| +| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config | +| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config | +| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config | +| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) | + +### High (storage configuration) + +| # | Finding | Impact | Type | +|---|---------|--------|------| +| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config | +| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config | +| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement | +| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity | +| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config | + +### Medium (fleet consistency) + +| # | Finding | Impact | Type | +|---|---------|--------|------| +| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade | +| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning | +| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning | +| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning | +| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config | +| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup | + +### Low (monitoring / investigation) + +| # | Finding | Impact | Type | +|---|---------|--------|------| +| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate | +| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring | +| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config | +| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup | + +--- + +## 8. Recommendations (for operator review — NO changes applied) + +### 8.1 Switch changes (require maintenance window + operator approval) + +1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes + the 4th live cable (g16) part of the LAG. +2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor). + This unlocks multi-link distribution for nconnect NFS traffic. +3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches + for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation. +4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor. +5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr) + on core-sw01. + +> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the +> highest-impact changes with the lowest risk. LACP conversion (#3) is optional +> if the static trunk is working reliably otherwise. All changes can be done +> read-only-verified via the conman + SNMP tooling in this repo. + +### 8.2 Storage configuration (host-side, no switch changes needed) + +1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`) + per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact) + and sectestbed scratch. +2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or + similar. Offloads the 93%-full Samsung 860 PRO. +3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken + `dir: S4` entry in storage.cfg. +4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or + keep local-only for PBS scratch. + +### 8.3 Fleet consistency + tuning (host-side) + +1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet). +2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5: + - Fix rmem/wmem to 128MB (tsys4) + - Fix tuned profile (tsys4 → network-throughput) + - Add nconnect=4 to tsys5 NFS client mounts + - Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop) +3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS- + backed qcow2 disks (requires VM stop/start). + +### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md)) + +The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty +Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading +that didn't exist in the 07-29 audit. Detailed placement plan deferred to the +operator review of this document. diff --git a/switches/pfv-core-sw01.cmds b/switches/pfv-core-sw01.cmds index 5db89b2..bd73bf5 100644 --- a/switches/pfv-core-sw01.cmds +++ b/switches/pfv-core-sw01.cmds @@ -1,11 +1,7 @@ ! pfv-core-sw01 — Dell PowerConnect 5448 (core switch, rack 5) -! Read-only show commands for LAG/port-channel/VLAN audit +! Need running-config to diagnose ch1 port mismatch (g16 up but not in LAG, g17 down) terminal datadump -show version -show interfaces status -show interfaces port-channel -show port-channel summary -show lacp -show vlan -show running-config interface port-channel 1 -show running-config interface port-channel 3 +enable +show running-config +show interfaces configuration +show interfaces description