KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: migrate all documentation to Discourse wiki topics

Browse Source 
All knowledge docs (architecture, runbooks, references, audits, policies)
have been migrated to community.turnsys.com as wiki topics in the VP
TechOps category. Discourse is now the canonical source of truth for
documentation; git edit history no longer serves as the doc changelog.

37 .md files converted to short pointers linking to their Discourse
topics. AGENTS.md updated with new documentation workflow policy.
Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
reachableceo
2026-08-06 14:08:34 -05:00
parent d6988bb35f
commit 88faf35ec3
37 changed files with 378 additions and 8239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/docs/AUDIT-2026-07-30.md
+10 -478
View File
@@ -1,478 +1,10 @@
# Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)
> **The single authoritative audit.** Supersedes [`AUDIT-2026-07-29.md`](AUDIT-2026-07-29.md)
> and [`PROJECT.md`](PROJECT.md) where they conflict. Capacity targets are
> codified in [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md).
**Audit time:** 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)
+ switch SNMP/conman captures (~03:50–04:00 UTC).
**Method:** `perf/scripts/probe-storage.sh`, `perf/scripts/probe-network.sh`,
`perf/scripts/snmp-switch-audit.py`, `perf/scripts/conman-console.py` — all
read-only. No changes made to any system.
**Hosts online:** tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9
**Hosts retired:** tsys8 (permanently), tsys2 (Win10, pending rebuild)
---
## 1. Physical network topology
### 1.1 Diagram
```
RACK 5 (core/data) RACK 3 (compute/RackRental)
┌─────────────────────────┐ ┌──────────────────────────┐
│ │ │ │
│ pfv-core-sw01 │ ch1 LAG │ pfv-tor3-stor │
│ PowerConnect 5448 │◄───3×1G──────────►│ Neyland 24T (PC5324) │
│ (mgmt+storage VLAN) │ (g13-15) │ (storage TOR) │
│ │ g16 UP, not │ │
│ VLAN1 (mgmt) │ in LAG; g17 │ g8-9 → tsys6 bond0 │
│ VLAN1000 (storage) │ DOWN │ g10-11 → tsys7 bond0 │
│ │ │ g20-23 → ch1 (3 active) │
│ g8 → tsys4 nic0 │ │ │
│ g19 → tsys5 nic0 │ │ pfv-tor3-mgmt │
│ g31 → tsys4 enx (USB) │ │ Neyland 24T (PC5324) │
│ g33-34 → tsys5 ch3 │ │ (mgmt TOR) │
│ ch3 → tsys5 bond (1 ln)│ │ g7 → tsys6 nic0 │
│ │ │ g3 → tsys7 nic0 │
│ tsys4 (storage+PBS) │ │ │
│ tsys5 (storage+sectest)│ │ tsys6 (RackRental) │
│ tsys1 (infra) │ │ tsys7 (RackRental) │
│ tsys3 (compute) │ │ │
│ tsys9 (infra+compute) │ │ │
└─────────────────────────┘ └──────────────────────────┘
```
### 1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK
| Property | core-sw01 (rack5) | tor3-stor (rack3) | Problem? |
|----------|-------------------|--------------------|----------|
| **Physical ports** | g13, g14, g15 (UP) | g20, g21, g22, g23 (all UP) | Asymmetry — see below |
| **LAG config** | `g(13-15,17)` mode on | `g(20-23)` mode on | g17 DOWN; g16 UP but not in LAG |
| **Active links** | **3 of 4** | **4 of 4** | Mismatch: only 3 links negotiate |
| **Hash policy** | **layer-2-3** | **layer-2-3-4** | **ASYMMETRIC** — core lacks L4 |
| **Mode** | `on` (static) | `on` (static) | No LACP negotiation/failure detection |
| **VLAN** | access vlan 1000 | access vlan 1000 | OK |
**Three root causes of cross-rack underperformance:**
1. **g16/g17 cable mismatch.** core-sw01 config says `channel-group 1` on
`g(13-15,17)`. But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th
physical cable landed on g16, not g17. Result: **only 3 links active** (3 Gbps,
not the intended 4).
2. **LAG hash asymmetry.** tor3-stor uses `port-channel load-balance layer-2-3-4`
(includes TCP/UDP source+dest ports — correct for NFS with nconnect).
core-sw01 uses `port-channel load-balance layer-2-3` (no L4 ports). With
layer-2-3 hashing, **all NFS traffic between a given host pair hashes to one
physical link**. Even though the clients use `nconnect=4` (4 TCP connections
per mount), they share the same src/dst IP → same hash → **max 1 Gbps per
server pair**. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.
3. **Static mode (`on`) instead of LACP.** Neither switch runs LACP on the
cross-rack trunk. `mode on` provides no link-failure detection, no partner
validation, no dynamic negotiation. A dead cable stays "active" until
manually discovered.
### 1.3 Per-host network state (ground truth)
| Host | Rack | mgmt NIC | storage NIC | bond0 | Hash | Active slaves | Speed | Storage IP |
|------|------|----------|-------------|-------|------|---------------|-------|------------|
| tsys1 | 5 | nic0 → core g8 | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.1 |
| tsys3 | 5 | nic0 → core | — (no bond) | — | — | — | — | 10.100.100.3 |
| tsys4 | 5 | nic0 → core g8 | enx8cae4ccda926 (USB) → core g31 | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.4 |
| tsys5 | 5 | nic0 → core g19 | nic1 → core g34 (ch3) | bond0 | layer2+3 | **1 of 2** (nic2 DOWN) | 1G | 10.100.100.5 |
| tsys6 | 3 | nic0 → tor3-mgmt g7 | nic1+nic2 → tor3-stor g8/g9 (ch2) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.6 |
| tsys7 | 3 | nic0 → tor3-mgmt g3 | nic1+nic2 → tor3-stor g10/g11 (ch3) | bond0 | **layer3+4** | **2 of 2** | **2G** | 10.100.100.7 |
| tsys9 | 5 | nic0 → core | enx (USB dongle) | bond0 (1×USB) | layer2+3 | 1 | 1G | 10.100.100.9 |
**Key observations:**
- **tsys6/7 are the ONLY hosts with working 2G LACP** (layer3+4, 2 active slaves). Validated previously.
- **tsys4/9 use USB dongles** for storage — single 1G, no bond redundancy.
- **tsys5 bond is degraded** — nic2 is DOWN (no cable). Only nic1 active. This is
the LACP problem the operator has been struggling with. The switch side (core-sw01
ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
- **tsys1 also uses a USB dongle** for storage.
- **Hash inconsistency:** tsys6/7 use layer3+4 (correct for nconnect), tsys4/5
use layer2+3 (suboptimal for NFS multi-connection).
### 1.4 Switch inventory (SNMP + conman ground truth)
| Switch | IP | Model | Firmware | Role | Console |
|--------|----|-------|----------|------|---------|
| pfv-core-sw01 | 192.168.0.12 | Dell PowerConnect 5448 | 2.0.0.46 | Core (rack5, combined mgmt+storage) | conman port 2001 |
| pfv-tor3-stor | 192.168.0.9 | Dell PowerConnect 5324 (Neyland 24T) | 2.0.1.4 | Rack3 storage TOR | conman port 2003 |
| pfv-tor3-mgmt | 192.168.0.7 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Rack3 mgmt TOR | conman port 2002 |
| (unknown) | 192.168.0.8 | Dell PowerConnect 5324 (Neyland 24T) | (same family) | Unidentified — possibly pfv-r2-tor-top | conman port 2005? |
**Switch SNMP community:** `kn3lmgmt` (read-only). All switches respond to SNMPv2c.
**Console access:** conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit
(udev symlink naming bug — singular vs plural — prevented consoles from working
after reboot).
### 1.5 Port error counters (core-sw01)
| Port | In Errors | Notes |
|------|-----------|-------|
| **g9** | **2902** | Connected to 192.168.0.21 (device unknown). Investigate CRC/cable. |
| **g38** | **590** | On VLAN 1000 (storage). Connected to a host on the storage VLAN. |
| **g26** | **73** | Connected to a management-network device. |
| All others | 0 | Healthy. |
tor3-stor shows **zero errors** on all active ports.
---
## 2. Storage topology (CORRECTED — current ground truth)
### 2.1 Storage tier definitions
| Tier | Definition | Performance | Use case |
|------|-----------|-------------|----------|
| **Tier 1** | NVMe / SSD on internal SATA/SAS | 200-3000 MB/s, high IOPS | OS drives, high-IOPS workloads |
| **Tier 2** | HDD on internal SATA/SAS | 80-150 MB/s, moderate IOPS | Production VM OS drives |
| **Tier 3** | HDD/SSD on USB "toaster" / USB adapter | 30-60 MB/s, low IOPS, unreliable | Bulk/backup/scratch only |
### 2.2 tsys4 disk inventory (CHANGED since last audit)
| Device | Model | Size | Interface | Mount | NFS export | Tier | Notes |
|--------|-------|------|-----------|-------|------------|------|-------|
| sda | Hitachi HDS723020BLA642 | 2.0 TB | Internal SATA | /mnt/albert | (not exported) | **Tier 2** | NEW — not in Proxmox storage.cfg, not exported. 0% used. |
| sdb | HITACHI HUA723020ALA640 | 2.0 TB | Internal SATA | /mnt/tsys4/D5 | D5 | Tier 2 | 13% used (220 GB). 7200 rpm. |
| sdc | WDC WD30EFRX (Red) | 3.0 TB | Internal SATA | /mnt/tsys4/D2 | D2 | Tier 2 | 7% used (170 GB). 5400 rpm. **17 VMs on this spindle.** |
| sdd | WDC WD10EZRX | 1.0 TB | Internal SATA | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
| sde | ST500LM021 (Seagate) | 500 GB | **USB** | /mnt/tsys4/D3 | D3 | **Tier 3** | **CHANGED from SK hynix SSD to Seagate HDD.** 0% used. |
| sdf | WDC WD10EARS | 1.0 TB | **USB** | /mnt/tsys4/D1 | D1 | **Tier 3** | 0% used. 1 reallocated sector (SMART). |
| sdg | WDC WD50NDZW (easystore) | 5.0 TB | **USB** | /mnt/backup | (not exported) | **Tier 3** | NEW — bulk backup disk. 0% used. |
**Changes from AUDIT-2026-07-29:**
- **D3 changed:** SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB).
D3 is now Tier 3 (was previously listed as unreliable USB SSD).
- **/mnt/albert added:** Hitachi 2TB internal SATA. Not exported, not in Proxmox.
Potential new Tier 2 storage.
- **/mnt/backup added:** WD 5TB USB easystore. Tier 3 bulk backup.
- **D7 removed:** Previously orphaned export (held DellOpenManageEnterprise). Now gone.
### 2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)
| Device | Model | Size | Interface | Mount | NFS/Local | Tier | Notes |
|--------|-------|------|-----------|-------|-----------|------|-------|
| sda | ST1000VN002 (IronWolf) | 1.0 TB | Internal SATA | (not mounted) | — | Tier 2 | **Was S3.** Not in fstab, not mounted. Stale export remains in /etc/exports. |
| sdb | SK hynix SC300B | 512 GB | Internal SATA | (not mounted) | — | **Tier 1** | **NEW — relocated from tsys4 USB enclosure.** Not mounted, not in fstab, not in storage.cfg. |
| sdc | HITACHI HUS724020ALA640 | 2.0 TB | Internal SAS | (pve boot/swap/local-lvm) | — | Tier 2 | Proxmox OS disk. |
| sdd | Samsung SSD 860 PRO | 256 GB | Internal SAS | /mnt/pfv-tsys5/ssd | `ssd` dir (local) | **Tier 1** | **93% full!** Used by ultix-streaming (5111). |
| sde | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S1 | S1 (NFS + local-nonprod) | Tier 2 | 11% used. 5900 rpm. |
| sdf | ST1000VN002 (IronWolf) | 1.0 TB | Internal SAS | /mnt/pfv-tsys5/S2 | S2 (NFS) | Tier 2 | 4% used. |
| **nvme0n1** | **CT500P1SSD8 (Crucial)** | **500 GB** | **NVMe** | **(not mounted)** | — | **Tier 1** | **NEW — not mounted, not in fstab, not in storage.cfg.** 34K power-on hours. Healthy. |
**Changes from AUDIT-2026-07-29:**
- **NVMe added:** Crucial CT500P1 500GB. Not yet configured. This is the highest-
performance local storage in the fleet (true NVMe).
- **SK hynix SSD relocated:** Moved from tsys4 USB enclosure to tsys5 internal
SATA. Not yet mounted/configured.
- **S3 unmounted:** ST1000VN002 (was S3) is present but fstab entry is commented
out. /etc/exports still exports S3 (stale).
- **S4 gone:** Old Toshiba 458GB is no longer present. fstab entry commented.
storage.cfg still has a broken `dir: S4` entry (points to root filesystem).
- **S1/S2 drives changed:** Both are now ST1000VN002 IronWolf 1TB (were "Seagate
916GB" in old audit).
- **SSD nearly full:** Samsung 860 PRO at 93% — needs attention.
### 2.4 NFS export map (current, verified)
| Server | Export | NFS ID | Disk | Tier | Used |
|--------|--------|--------|------|------|------|
| tsys4 | /mnt/tsys4/D1 | D1 | WDC WD10EARS 1TB | Tier 3 (USB) | 0% |
| tsys4 | /mnt/tsys4/D2 | D2 | WDC Red 3TB | Tier 2 | 7% (170 GB) |
| tsys4 | /mnt/tsys4/D3 | D3 | Seagate 500GB | Tier 3 (USB) | 0% |
| tsys4 | /mnt/tsys4/D5 | D5 | Hitachi 2TB | Tier 2 | 13% (220 GB) |
| tsys5 | /mnt/pfv-tsys5/S1 | S1 | IronWolf 1TB | Tier 2 | 11% (96 GB) |
| tsys5 | /mnt/pfv-tsys5/S2 | S2 | IronWolf 1TB | Tier 2 | 4% (32 GB) |
| tsys5 | /mnt/pfv-tsys5/S3 | S3 | (stale — disk not mounted) | — | — |
### 2.5 NFS client mount status (nconnect)
| Host | Mounts from tsys4 | Mounts from tsys5 | nconnect=4? | Hash |
|------|-------------------|-------------------|-------------|------|
| tsys4 | (NFS server, no client mounts) | — | — | layer2+3 |
| tsys5 | D2, D3, D5 | (NFS server) | **NO nconnect** | layer2+3 |
| tsys6 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
| tsys7 | D1, D2, D3, D5 | S1, S2, S3, S4, T5-SSD | **YES** (9 mounts) | layer3+4 |
**tsys5 is missing nconnect** on its 3 NFS client mounts (D2/D3/D5 from tsys4).
**tsys4 is missing nconnect** entirely (but it's primarily an NFS server, not a client).
### 2.6 Per-spindle IOPS load (running VMs only)
| Spindle | Server | NFS-visible VMs | Local-only VMs | Total | Tier | Notes |
|---------|--------|-----------------|----------------|-------|------|-------|
| **D2** (3TB Red) | tsys4 | **13** | 0 | **13** | Tier 2 | Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904 |
| D5 (2TB Hitachi) | tsys4 | 4 | 0 | 4 | Tier 2 | VMs: 106,313,600(disk0),903 |
| S1 (1TB IronWolf) | tsys5 | 1 | 0 | 1 | Tier 2 | VM: 702 (hfnoc-uisp, 100GB) |
| S2 (1TB IronWolf) | tsys5 | 4 | 0 | 4 | Tier 2 | VMs: 103,703,705,902,905 |
| S4 (local-nonprod) | tsys5 | 0 | **12** (stopped sandboxes) | 12 idle | Tier 2 | All stopped. Low priority. |
| D3 (500GB Seagate USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
| D1 (1TB WD USB) | tsys4 | 0 | 0 | 0 | **Tier 3** | Empty |
| ssd (Samsung 860 PRO) | tsys5 | 0 | 1 | 1 | Tier 1 | VM 5111 (ultix-streaming, 288GB) |
**D2 still carries 13 running VMs** — the IOPS hotspot persists.
---
## 3. Host capacity (live ground truth, 2026-07-30)
| Host | CPU | Threads | RAM | Running VMs | Actual RAM% | Load | CPU busy |
|------|-----|---------|-----|-------------|-------------|------|----------|
| tsys1 | i7-4770 Haswell (2013) | 8 | 31 GB | 9 | **57%** | 1.1 | ~4% |
| tsys3 | Xeon E3-1535M v5 Skylake (2015) | 8 | 31 GB | 1 | 10% | 0.3 | 0% |
| tsys4 | Xeon E3-1246 v3 Haswell (2013) | 8 | 16 GB | 1 | 15% | 1.8 | 0% |
| tsys5 | Xeon E5620 Westmere (2010) **single CPU** | 8 | 94 GB | 4 (+22 stopped) | 15% | 3.3 | ~5% |
| tsys6 | 2× Xeon E5530 Nehalem (2009) | 16 | 126 GB | 2 | 12% | 0.7 | 0% |
| tsys7 | 2× Xeon E5-2630 v2 Ivy Bridge (2013) | 24 | 189 GB | 5 | 7% | 1.0 | ~1% |
| tsys9 | i5-10500 Comet Lake (2020) | 12 | 23 GB | 6 | **69%** | 0.3 | ~1% |
**Changes since AUDIT-2026-07-29:**
- **tsys1 RAM dropped 73%→57%** — some VMs apparently migrated/stopped.
- **tsys9 RAM 62%→69%** — approaching 80% goal.
- **tsys6 wnode-tsys6 grew to 64GB RAM** (was 32GB in old audit).
- **All other hosts healthy** — well under 80%.
---
## 4. Fleet consistency audit
### 4.1 PVE/kernel versions
| Host | PVE | Kernel | Status |
|------|-----|--------|--------|
| tsys1 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys3 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| **tsys4** | **9.1.5** | **6.17.9-1-pve** | **BEHIND — needs upgrade** |
| tsys5 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys6 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys7 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
| tsys9 | 9.2.5 | 7.0.14-6-pve | Current ✓ |
### 4.2 Performance tuning state
| Host | Governor | Swappiness | TCP CC | rmem/wmem_max | Tuned profile | Bond hash | nconnect |
|------|----------|------------|--------|---------------|---------------|-----------|----------|
| tsys1 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A (client) |
| tsys3 | performance | 10 | bbr | 128MB | virtual-host | — | N/A |
| **tsys4** | **performance** | **1** | **bbr** | **16MB** | **throughput-performance** | **layer2+3** | **0** |
| **tsys5** | **N/A** (no driver) | **1** | **bbr** | **128MB** | **network-throughput** | **layer2+3** | **0** |
| tsys6 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** (9 mounts × 4... actually 6 options lines) |
| tsys7 | performance | 10 | bbr | 128MB | virtual-host | **layer3+4** | **24** |
| tsys9 | performance | 10 | bbr | 128MB | virtual-host | layer2+3 | N/A |
**Inconsistencies:**
- **tsys4:** rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong
(`throughput-performance` should be `network-throughput`). Missing nconnect.
- **tsys5:** Missing nconnect on NFS client mounts. No cpufreq driver (Westmere —
expected, not a bug).
- **tsys4/5 bond hash = layer2+3** — should be layer3+4 (like tsys6/7).
---
## 5. VM inventory + config audit (running VMs only)
### 5.1 Per-VM disk + CPU + network settings
> `cpu: host` = best (full instruction set passthrough). `x86-64-v2-AES` = good
> (nested-virt-safe baseline). `cache=writethrough` = safe but slow for NFS.
> `iothread=1` = present on most. `agent: 1` = qemu-guest-agent enabled.
| VMID | Name | Host | vCPU | CPU type | RAM | Disk store | Cache | iothread | qga | Notes |
|------|------|------|------|----------|-----|-----------|-------|----------|-----|-------|
| 100 | pfv-bms | tsys1 | 2 | host | 4G | D2 (raw) | — | — | ✓ | discard=on,ssd=1 |
| 101 | tsys-ca | tsys1 | 2 | v2-AES | 2G | D2 (virtio) | — | ✓ | — | no nested-virt |
| 103 | netinfra-01 | tsys1 | 2 | v2-AES | 4G | S2 | — | ✓ | — | |
| 104 | librenms | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
| 105 | proxmox-dc | tsys1 | 2 | host | 2G | D2 | — | ✓ | ✓ | |
| 106 | cnode3 | tsys1 | 4 | host | 4G | D5 | **writethrough** | ✓ | ✓ | |
| 108 | ucs-01 | tsys1 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | |
| 114 | kali-tsys | tsys1 | 2 | host | 2G | D2 | **writethrough** | ✓ | ✓ | |
| 117 | workbench | tsys1 | 2 | host | 4G | D2 | **writethrough** | ✓ | ✓ | |
| 313 | wnode-tsys3 | tsys3 | 8 | host | 28G | D5 | **writethrough** | ✓ | ✓ | |
| 400 | PBS | tsys4 | 2 | — | 2G | local-lvm | — | — | — | backup server |
| 509 | wnode-tsys5 | tsys5 | 2 | host | 32G | D2 (virtio) | — | ✓ | — | low vCPU (2) for k8s wnode |
| 5111 | ultix-streaming | tsys5 | 4 | — | 9G | ssd (local) | — | — | — | 288GB on SSD (93% full) |
| 5112 | ultix-offstage | tsys5 | 4 | — | 6G | local-lvm | — | — | — | 288GB |
| 100 | wnode-tsys6 | tsys6 | 2 | host | 64G | S4 | — | ✓ | — | low vCPU (2), high RAM |
| 600 | tsys-awx | tsys6 | 2 | host | 12G | D5 | **writethrough** | ✓ | ✓ | |
| 701 | wnode-tsys7 | tsys7 | 4 | host | 32G | S3 (virtio) | — | ✓ | — | |
| 702 | hfnoc-uisp | tsys7 | 2 | host | 8G | S1 | — | ✓ | ✓ | 100GB disk |
| 703 | rr-middleware | tsys7 | 2 | host | 2G | S2 | **writethrough** | ✓ | ✓ | |
| 704 | TCTC | tsys7 | 4 | host | 6G | D2 (virtio) | — | ✓ | — | |
| 705 | cnode2 | tsys7 | 4 | host | 4G | S2 | **writethrough** | ✓ | ✓ | |
| 901 | tsys-siem | tsys9 | 2 | host | 8G | D2 | **writethrough** | ✓ | ✓ | 132GB disk |
| 902 | ucs-02 | tsys9 | 2 | host | 8G | S2 | — | ✓ | ✓ | 50GB disk |
| 903 | kali-rd | tsys9 | 2 | host | 2G | D5 | **writethrough** | ✓ | ✓ | |
| 904 | netinfra-02 | tsys9 | 2 | v2-AES | 4G | D2 | — | ✓ | — | |
| 905 | wnode-tsys9 | tsys9 | 4 | v2-AES | 4G | S2 | — | ✓ | ✓ | target: grow to 6c/8G |
| 906 | cnode1 | tsys9 | 2 | host | 4G | S3 | — | ✓ | — | |
### 5.2 VM optimization findings
| Finding | VMs affected | Impact | Fix |
|---------|-------------|--------|-----|
| **cache=writethrough on NFS** | 106,108,114,117,313,600,703,705,901,903 | Writethrough forces synchronous writes — slow on NFS. `writeback` or `none` is better for NFS-backed qcow2. | Change cache mode (requires VM stop/start) |
| **Mixed CPU types** | Some `host`, some `x86-64-v2-AES` | Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match. | Standardize per workload tier |
| **Low vCPU on k8s wnodes** | wnode-tsys5 (2c), wnode-tsys6 (2c) | K8s workers undersized for compute workloads | Grow after migrations free RAM |
| **VM 500 references D7** | DellOpenManageEnterprise (stopped) | D7 export no longer exists on tsys4 | Delete or re-disk before starting |
| **qga inconsistent** | Several VMs missing agent:1 | No guest-agent communication | Enable qga fleet-wide |
---
## 6. Console infrastructure status (fixed during this audit)
**Bug found + fixed:** `console/generate-config.sh` wrote udev rules with
`SYMLINK+="console/$name"` (singular) but ser2net.yaml opened
`/dev/consoles/$name` (plural). They never matched. After every reboot, all 7
console ports failed until `setup.sh`'s manual fallback re-created symlinks.
**Fix applied:** Changed udev rule to `SYMLINK+="consoles/$name"`. Re-created
all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7
consoles now working (verified via conman-console.py).
**Also fixed:** stale conmand process (pid 988) holding config lock, preventing
conmand systemd service from starting.
---
## 7. Findings summary (prioritized)
### Critical (cross-rack performance)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 1 | **core-sw01 LAG hash = layer-2-3** (no L4 ports) | All NFS traffic between a given host pair caps at 1 Gbps | Switch config |
| 2 | **core-sw01 ch1 cable on g16, not g17** | Only 3 of 4 cross-rack links active (3 Gbps, not 4) | Switch config |
| 3 | **Both switches use `mode on`** (static, no LACP) | No failure detection, no partner validation | Switch config |
| 4 | **tsys5 bond0 degraded** (1 of 2 slaves) | Single link to storage, no redundancy | Physical (cable) |
### High (storage configuration)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 5 | **tsys5 NVMe not configured** (CT500P1 500GB) | Best local storage in fleet sitting unused | Config |
| 6 | **tsys5 SK hynix SSD not configured** (512GB) | Relocated from tsys4, not mounted | Config |
| 7 | **D2 still carries 13 running VMs** | IOPS hotspot persists | VM placement |
| 8 | **tsys5 SSD 93% full** | Samsung 860 PRO nearly exhausted | Capacity |
| 9 | **S3/S4 stale on tsys5** | Broken exports, stale storage.cfg | Config |
### Medium (fleet consistency)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 10 | **tsys4 on PVE 9.1.5/kernel 6.17** | Behind fleet (9.2.5/7.0.14) | Upgrade |
| 11 | **tsys4 rmem/wmem = 16MB** | Suboptimal TCP buffers for NFS | Tuning |
| 12 | **tsys4/5 bond hash = layer2+3** | Suboptimal for nconnect NFS | Tuning |
| 13 | **tsys5 missing nconnect** | NFS client mounts to tsys4 lack multi-connection | Tuning |
| 14 | **cache=writethrough on 10+ VMs** | Synchronous writes slow NFS | VM config |
| 15 | **Stale tsys8 switch config** (tor3-stor ch4) | Dead LAG group for retired host | Cleanup |
### Low (monitoring / investigation)
| # | Finding | Impact | Type |
|---|---------|--------|------|
| 16 | **core-sw01 port errors** (g9:2902, g38:590, g26:73) | Possible cable/CRC issues | Investigate |
| 17 | **No continuous observability** | Only point-in-time check.sh snapshots | Monitoring |
| 18 | **/mnt/albert unused** (tsys4 Hitachi 2TB) | Potential new Tier 2 storage | Config |
| 19 | **VM 500 references non-existent D7** | Would fail on boot | Cleanup |
---
## 9. Observability stack (operator direction)
The operator has defined a clear monitoring architecture with three tools,
each serving a distinct role. Do NOT duplicate functionality across tools.
### 9.1 Current architecture
| Tool | Where it runs | Scope | Role |
|------|---------------|-------|------|
| **Beszel** | Cloudron VPS (Reston, VA) | Tailscale-connected hosts | CPU/RAM/disk/time-series metrics on all Proxmox hosts |
| **Uptime Kuma** | Cloudron VPS (Reston, VA) | All hosts + services | Up/down status + alerting via Pushover |
| **LibreNMS** (VMID 104, tsys1) | On-prem (R&D cluster) | Non-Tailscale devices only | SNMP polling for switches + devices that can't run Beszel agent |
| **Home Assistant** (VMID 100, tsys1) | On-prem (R&D cluster) | Physical infrastructure | Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.) |
### 9.2 Division of responsibility (no duplication)
| Metric class | Tool | Notes |
|-------------|------|-------|
| CPU / RAM / disk / network time-series | **Beszel** | All Proxmox hosts (Tailscale-connected) |
| Up/down alerting (hosts + services) | **Uptime Kuma** | Pushover notifications |
| SNMP polling (switches, non-Tailscale gear) | **LibreNMS** | Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent) |
| Power / temperature / physical env | **Home Assistant** | UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations |
### 9.3 Saturday "OAM Day" plan (2026-08-01)
The operator has designated Saturday as a dedicated operations/admin/management
day with the goal of achieving **100% instrumentation coverage**:
- [ ] Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover)
- [ ] Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog
relay, or Loki/Grafana stack)
- [ ] SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale
devices (community: `kn3lmgmt`, IPs: 192.168.0.7/8/9/12)
- [ ] Beszel agent coverage on all 7 Proxmox hosts
- [ ] Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU
via powerman, any temp sensors)
- [ ] Verify alerting paths end-to-end (test each alert fires correctly)
> Much of the groundwork is already done: NUT is live on tsys1, powerman is
> live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live
> on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring
> continuous monitoring + alerting on top of these.
---
## 8. Recommendations (for operator review — NO changes applied)
### 8.1 Switch changes (require maintenance window + operator approval)
1. **Fix core-sw01 ch1 member list:** `g(13-15,17)` → `g(13-16)`. This makes
the 4th live cable (g16) part of the LAG.
2. **Fix core-sw01 LAG hash:** `layer-2-3` → `layer-2-3-4` (match tor3-stor).
This unlocks multi-link distribution for nconnect NFS traffic.
3. **Convert trunk to LACP:** Change `mode on` → `mode active` on both switches
for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
4. **Remove stale tsys8 config:** Delete ch4 + g13/g14 channel-group on tor3-stor.
5. **Investigate port errors:** g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr)
on core-sw01.
> **Recommended order:** Fix #2 (hash) + #1 (member) first — these are the
> highest-impact changes with the lowest risk. LACP conversion (#3) is optional
> if the static trunk is working reliably otherwise. All changes can be done
> read-only-verified via the conman + SNMP tooling in this repo.
### 8.2 Storage configuration (host-side, no switch changes needed)
1. **Mount + configure tsys5 NVMe** (CT500P1 500GB) as LVM-thin (`nvme-local`)
per [`TODO.md`](TODO.md) §3. Use for wnode-tsys5 boot disk (highest impact)
and sectestbed scratch.
2. **Mount + configure tsys5 SK hynix SSD** (512GB). Add as `dir: ssd2` or
similar. Offloads the 93%-full Samsung 860 PRO.
3. **Clean S3/S4:** Remove stale S3 export from /etc/exports. Fix/remove broken
`dir: S4` entry in storage.cfg.
4. **Decide on /mnt/albert** (tsys4 Hitachi 2TB): export as new NFS storage or
keep local-only for PBS scratch.
### 8.3 Fleet consistency + tuning (host-side)
1. **Upgrade tsys4** to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
2. **Apply apply-tunings.sh --apply** to tsys4 and tsys5:
- Fix rmem/wmem to 128MB (tsys4)
- Fix tuned profile (tsys4 → network-throughput)
- Add nconnect=4 to tsys5 NFS client mounts
- Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
3. **Audit VM cache modes:** Change writethrough → writeback or none for NFS-
backed qcow2 disks (requires VM stop/start).
### 8.4 VM placement rebalancing (refer to [`CAPACITY-GOALS.md`](CAPACITY-GOALS.md))
The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty
Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading
that didn't exist in the 07-29 audit. Detailed placement plan deferred to the
operator review of this document.
# proxmox/docs/AUDIT-2026-07-30.md
> **Documentation moved to Discourse — the canonical source of truth.**
>
> **Audit snapshot 2026-07-30**
>
> **Read it here:** https://community.turnsys.com/t/298
>
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
> Do not update content here — edit the Discourse wiki topic instead.*