PFVCluster/proxmox/docs/AUDIT-2026-07-30.md

Fleet Audit — 2026-07-30 (full ground-truth: network + storage + compute)

The single authoritative audit. Supersedes AUDIT-2026-07-29.md and PROJECT.md where they conflict. Capacity targets are codified in CAPACITY-GOALS.md.

Audit time: 2026-07-30, fresh data from all 7 online hosts (~03:30–03:45 UTC)

• switch SNMP/conman captures (~03:50–04:00 UTC). Method: perf/scripts/probe-storage.sh, perf/scripts/probe-network.sh, perf/scripts/snmp-switch-audit.py, perf/scripts/conman-console.py — all read-only. No changes made to any system. Hosts online: tsys1, tsys3, tsys4, tsys5, tsys6, tsys7, tsys9 Hosts retired: tsys8 (permanently), tsys2 (Win10, pending rebuild)

---

1. Physical network topology

1.1 Diagram

  RACK 5 (core/data)                           RACK 3 (compute/RackRental)
  ┌─────────────────────────┐                  ┌──────────────────────────┐
  │                         │                  │                          │
  │  pfv-core-sw01          │    ch1 LAG       │  pfv-tor3-stor           │
  │  PowerConnect 5448      │◄───3×1G──────────►│  Neyland 24T (PC5324)   │
  │  (mgmt+storage VLAN)    │   (g13-15)       │  (storage TOR)           │
  │                         │   g16 UP, not    │                          │
  │  VLAN1 (mgmt)           │   in LAG; g17    │  g8-9  → tsys6 bond0     │
  │  VLAN1000 (storage)     │   DOWN           │  g10-11 → tsys7 bond0    │
  │                         │                  │  g20-23 → ch1 (3 active) │
  │  g8   → tsys4 nic0      │                  │                          │
  │  g19  → tsys5 nic0      │                  │  pfv-tor3-mgmt           │
  │  g31  → tsys4 enx (USB) │                  │  Neyland 24T (PC5324)    │
  │  g33-34 → tsys5 ch3     │                  │  (mgmt TOR)              │
  │  ch3 → tsys5 bond (1 ln)│                  │  g7 → tsys6 nic0         │
  │                         │                  │  g3 → tsys7 nic0         │
  │  tsys4 (storage+PBS)    │                  │                          │
  │  tsys5 (storage+sectest)│                  │  tsys6 (RackRental)      │
  │  tsys1 (infra)          │                  │  tsys7 (RackRental)      │
  │  tsys3 (compute)        │                  │                          │
  │  tsys9 (infra+compute)  │                  │                          │
  └─────────────────────────┘                  └──────────────────────────┘

1.2 Cross-rack trunk detail (ch1) — THE BOTTLENECK

Property	core-sw01 (rack5)	tor3-stor (rack3)	Problem?
Physical ports	g13, g14, g15 (UP)	g20, g21, g22, g23 (all UP)	Asymmetry — see below
LAG config	g(13-15,17) mode on	g(20-23) mode on	g17 DOWN; g16 UP but not in LAG
Active links	3 of 4	4 of 4	Mismatch: only 3 links negotiate
Hash policy	layer-2-3	layer-2-3-4	ASYMMETRIC — core lacks L4
Mode	on (static)	on (static)	No LACP negotiation/failure detection
VLAN	access vlan 1000	access vlan 1000	OK

Three root causes of cross-rack underperformance:

1. g16/g17 cable mismatch. core-sw01 config says channel-group 1 on g(13-15,17). But g17 is DOWN and g16 is UP (live, 1000/Full). The 4th physical cable landed on g16, not g17. Result: only 3 links active (3 Gbps, not the intended 4).

2. LAG hash asymmetry. tor3-stor uses port-channel load-balance layer-2-3-4 (includes TCP/UDP source+dest ports — correct for NFS with nconnect). core-sw01 uses port-channel load-balance layer-2-3 (no L4 ports). With layer-2-3 hashing, all NFS traffic between a given host pair hashes to one physical link. Even though the clients use nconnect=4 (4 TCP connections per mount), they share the same src/dst IP → same hash → max 1 Gbps per server pair. This is why tsys6/7 read from tsys4/5 at ~1 Gbps, not 3-4 Gbps.

3. Static mode (on) instead of LACP. Neither switch runs LACP on the cross-rack trunk. mode on provides no link-failure detection, no partner validation, no dynamic negotiation. A dead cable stays "active" until manually discovered.

1.3 Per-host network state (ground truth)

Host	Rack	mgmt NIC	storage NIC	bond0	Hash	Active slaves	Speed	Storage IP
tsys1	5	nic0 → core g8	enx (USB dongle)	bond0 (1×USB)	layer2+3	1	1G	10.100.100.1
tsys3	5	nic0 → core	— (no bond)	—	—	—	—	10.100.100.3
tsys4	5	nic0 → core g8	enx8cae4ccda926 (USB) → core g31	bond0 (1×USB)	layer2+3	1	1G	10.100.100.4
tsys5	5	nic0 → core g19	nic1 → core g34 (ch3)	bond0	layer2+3	1 of 2 (nic2 DOWN)	1G	10.100.100.5
tsys6	3	nic0 → tor3-mgmt g7	nic1+nic2 → tor3-stor g8/g9 (ch2)	bond0	layer3+4	2 of 2	2G	10.100.100.6
tsys7	3	nic0 → tor3-mgmt g3	nic1+nic2 → tor3-stor g10/g11 (ch3)	bond0	layer3+4	2 of 2	2G	10.100.100.7
tsys9	5	nic0 → core	enx (USB dongle)	bond0 (1×USB)	layer2+3	1	1G	10.100.100.9

Key observations:

• tsys6/7 are the ONLY hosts with working 2G LACP (layer3+4, 2 active slaves). Validated previously.
• tsys4/9 use USB dongles for storage — single 1G, no bond redundancy.
• tsys5 bond is degraded — nic2 is DOWN (no cable). Only nic1 active. This is the LACP problem the operator has been struggling with. The switch side (core-sw01 ch3) shows 1 active member (g34), and tsys5's bond0 reports 1 active slave.
• tsys1 also uses a USB dongle for storage.
• Hash inconsistency: tsys6/7 use layer3+4 (correct for nconnect), tsys4/5 use layer2+3 (suboptimal for NFS multi-connection).

1.4 Switch inventory (SNMP + conman ground truth)

Switch	IP	Model	Firmware	Role	Console
pfv-core-sw01	192.168.0.12	Dell PowerConnect 5448	2.0.0.46	Core (rack5, combined mgmt+storage)	conman port 2001
pfv-tor3-stor	192.168.0.9	Dell PowerConnect 5324 (Neyland 24T)	2.0.1.4	Rack3 storage TOR	conman port 2003
pfv-tor3-mgmt	192.168.0.7	Dell PowerConnect 5324 (Neyland 24T)	(same family)	Rack3 mgmt TOR	conman port 2002
(unknown)	192.168.0.8	Dell PowerConnect 5324 (Neyland 24T)	(same family)	Unidentified — possibly pfv-r2-tor-top	conman port 2005?

Switch SNMP community: kn3lmgmt (read-only). All switches respond to SNMPv2c. Console access: conman on pfv-tsys4:7890 (7 consoles). Fixed during this audit (udev symlink naming bug — singular vs plural — prevented consoles from working after reboot).

1.5 Port error counters (core-sw01)

Port	In Errors	Notes
g9	2902	Connected to 192.168.0.21 (device unknown). Investigate CRC/cable.
g38	590	On VLAN 1000 (storage). Connected to a host on the storage VLAN.
g26	73	Connected to a management-network device.
All others	0	Healthy.

tor3-stor shows zero errors on all active ports.

---

2. Storage topology (CORRECTED — current ground truth)

2.1 Storage tier definitions

Tier	Definition	Performance	Use case
Tier 1	NVMe / SSD on internal SATA/SAS	200-3000 MB/s, high IOPS	OS drives, high-IOPS workloads
Tier 2	HDD on internal SATA/SAS	80-150 MB/s, moderate IOPS	Production VM OS drives
Tier 3	HDD/SSD on USB "toaster" / USB adapter	30-60 MB/s, low IOPS, unreliable	Bulk/backup/scratch only

2.2 tsys4 disk inventory (CHANGED since last audit)

Device	Model	Size	Interface	Mount	NFS export	Tier	Notes
sda	Hitachi HDS723020BLA642	2.0 TB	Internal SATA	/mnt/albert	(not exported)	Tier 2	NEW — not in Proxmox storage.cfg, not exported. 0% used.
sdb	HITACHI HUA723020ALA640	2.0 TB	Internal SATA	/mnt/tsys4/D5	D5	Tier 2	13% used (220 GB). 7200 rpm.
sdc	WDC WD30EFRX (Red)	3.0 TB	Internal SATA	/mnt/tsys4/D2	D2	Tier 2	7% used (170 GB). 5400 rpm. 17 VMs on this spindle.
sdd	WDC WD10EZRX	1.0 TB	Internal SATA	(pve boot/swap/local-lvm)	—	Tier 2	Proxmox OS disk.
sde	ST500LM021 (Seagate)	500 GB	USB	/mnt/tsys4/D3	D3	Tier 3	CHANGED from SK hynix SSD to Seagate HDD. 0% used.
sdf	WDC WD10EARS	1.0 TB	USB	/mnt/tsys4/D1	D1	Tier 3	0% used. 1 reallocated sector (SMART).
sdg	WDC WD50NDZW (easystore)	5.0 TB	USB	/mnt/backup	(not exported)	Tier 3	NEW — bulk backup disk. 0% used.

Changes from AUDIT-2026-07-29:

• D3 changed: SK hynix SC300 512GB SSD → Seagate ST500LM021 500GB HDD (still on USB). D3 is now Tier 3 (was previously listed as unreliable USB SSD).
• /mnt/albert added: Hitachi 2TB internal SATA. Not exported, not in Proxmox. Potential new Tier 2 storage.
• /mnt/backup added: WD 5TB USB easystore. Tier 3 bulk backup.
• D7 removed: Previously orphaned export (held DellOpenManageEnterprise). Now gone.

2.3 tsys5 disk inventory (MAJOR CHANGES — new NVMe + SSD)

Device	Model	Size	Interface	Mount	NFS/Local	Tier	Notes
sda	ST1000VN002 (IronWolf)	1.0 TB	Internal SATA	(not mounted)	—	Tier 2	Was S3. Not in fstab, not mounted. Stale export remains in /etc/exports.
sdb	SK hynix SC300B	512 GB	Internal SATA	(not mounted)	—	Tier 1	NEW — relocated from tsys4 USB enclosure. Not mounted, not in fstab, not in storage.cfg.
sdc	HITACHI HUS724020ALA640	2.0 TB	Internal SAS	(pve boot/swap/local-lvm)	—	Tier 2	Proxmox OS disk.
sdd	Samsung SSD 860 PRO	256 GB	Internal SAS	/mnt/pfv-tsys5/ssd	ssd dir (local)	Tier 1	93% full! Used by ultix-streaming (5111).
sde	ST1000VN002 (IronWolf)	1.0 TB	Internal SAS	/mnt/pfv-tsys5/S1	S1 (NFS + local-nonprod)	Tier 2	11% used. 5900 rpm.
sdf	ST1000VN002 (IronWolf)	1.0 TB	Internal SAS	/mnt/pfv-tsys5/S2	S2 (NFS)	Tier 2	4% used.
nvme0n1	CT500P1SSD8 (Crucial)	500 GB	NVMe	(not mounted)	—	Tier 1	NEW — not mounted, not in fstab, not in storage.cfg. 34K power-on hours. Healthy.

Changes from AUDIT-2026-07-29:

• NVMe added: Crucial CT500P1 500GB. Not yet configured. This is the highest- performance local storage in the fleet (true NVMe).
• SK hynix SSD relocated: Moved from tsys4 USB enclosure to tsys5 internal SATA. Not yet mounted/configured.
• S3 unmounted: ST1000VN002 (was S3) is present but fstab entry is commented out. /etc/exports still exports S3 (stale).
• S4 gone: Old Toshiba 458GB is no longer present. fstab entry commented. storage.cfg still has a broken dir: S4 entry (points to root filesystem).
• S1/S2 drives changed: Both are now ST1000VN002 IronWolf 1TB (were "Seagate 916GB" in old audit).
• SSD nearly full: Samsung 860 PRO at 93% — needs attention.

2.4 NFS export map (current, verified)

Server	Export	NFS ID	Disk	Tier	Used
tsys4	/mnt/tsys4/D1	D1	WDC WD10EARS 1TB	Tier 3 (USB)	0%
tsys4	/mnt/tsys4/D2	D2	WDC Red 3TB	Tier 2	7% (170 GB)
tsys4	/mnt/tsys4/D3	D3	Seagate 500GB	Tier 3 (USB)	0%
tsys4	/mnt/tsys4/D5	D5	Hitachi 2TB	Tier 2	13% (220 GB)
tsys5	/mnt/pfv-tsys5/S1	S1	IronWolf 1TB	Tier 2	11% (96 GB)
tsys5	/mnt/pfv-tsys5/S2	S2	IronWolf 1TB	Tier 2	4% (32 GB)
tsys5	/mnt/pfv-tsys5/S3	S3	(stale — disk not mounted)	—	—

2.5 NFS client mount status (nconnect)

Host	Mounts from tsys4	Mounts from tsys5	nconnect=4?	Hash
tsys4	(NFS server, no client mounts)	—	—	layer2+3
tsys5	D2, D3, D5	(NFS server)	NO nconnect	layer2+3
tsys6	D1, D2, D3, D5	S1, S2, S3, S4, T5-SSD	YES (9 mounts)	layer3+4
tsys7	D1, D2, D3, D5	S1, S2, S3, S4, T5-SSD	YES (9 mounts)	layer3+4

tsys5 is missing nconnect on its 3 NFS client mounts (D2/D3/D5 from tsys4). tsys4 is missing nconnect entirely (but it's primarily an NFS server, not a client).

2.6 Per-spindle IOPS load (running VMs only)

Spindle	Server	NFS-visible VMs	Local-only VMs	Total	Tier	Notes
D2 (3TB Red)	tsys4	13	0	13	Tier 2	Still the worst hotspot. VMs: 100,101,104,105,108,114,117,509,600,702,704,901,904
D5 (2TB Hitachi)	tsys4	4	0	4	Tier 2	VMs: 106,313,600(disk0),903
S1 (1TB IronWolf)	tsys5	1	0	1	Tier 2	VM: 702 (hfnoc-uisp, 100GB)
S2 (1TB IronWolf)	tsys5	4	0	4	Tier 2	VMs: 103,703,705,902,905
S4 (local-nonprod)	tsys5	0	12 (stopped sandboxes)	12 idle	Tier 2	All stopped. Low priority.
D3 (500GB Seagate USB)	tsys4	0	0	0	Tier 3	Empty
D1 (1TB WD USB)	tsys4	0	0	0	Tier 3	Empty
ssd (Samsung 860 PRO)	tsys5	0	1	1	Tier 1	VM 5111 (ultix-streaming, 288GB)

D2 still carries 13 running VMs — the IOPS hotspot persists.

---

3. Host capacity (live ground truth, 2026-07-30)

Host	CPU	Threads	RAM	Running VMs	Actual RAM%	Load	CPU busy
tsys1	i7-4770 Haswell (2013)	8	31 GB	9	57%	1.1	~4%
tsys3	Xeon E3-1535M v5 Skylake (2015)	8	31 GB	1	10%	0.3	0%
tsys4	Xeon E3-1246 v3 Haswell (2013)	8	16 GB	1	15%	1.8	0%
tsys5	Xeon E5620 Westmere (2010) single CPU	8	94 GB	4 (+22 stopped)	15%	3.3	~5%
tsys6	2× Xeon E5530 Nehalem (2009)	16	126 GB	2	12%	0.7	0%
tsys7	2× Xeon E5-2630 v2 Ivy Bridge (2013)	24	189 GB	5	7%	1.0	~1%
tsys9	i5-10500 Comet Lake (2020)	12	23 GB	6	69%	0.3	~1%

Changes since AUDIT-2026-07-29:

• tsys1 RAM dropped 73%→57% — some VMs apparently migrated/stopped.
• tsys9 RAM 62%→69% — approaching 80% goal.
• tsys6 wnode-tsys6 grew to 64GB RAM (was 32GB in old audit).
• All other hosts healthy — well under 80%.

---

4. Fleet consistency audit

4.1 PVE/kernel versions

Host	PVE	Kernel	Status
tsys1	9.2.5	7.0.14-6-pve	Current ✓
tsys3	9.2.5	7.0.14-6-pve	Current ✓
tsys4	9.1.5	6.17.9-1-pve	BEHIND — needs upgrade
tsys5	9.2.5	7.0.14-6-pve	Current ✓
tsys6	9.2.5	7.0.14-6-pve	Current ✓
tsys7	9.2.5	7.0.14-6-pve	Current ✓
tsys9	9.2.5	7.0.14-6-pve	Current ✓

4.2 Performance tuning state

Host	Governor	Swappiness	TCP CC	rmem/wmem_max	Tuned profile	Bond hash	nconnect
tsys1	performance	10	bbr	128MB	virtual-host	layer2+3	N/A (client)
tsys3	performance	10	bbr	128MB	virtual-host	—	N/A
tsys4	performance	1	bbr	16MB	throughput-performance	layer2+3	0
tsys5	N/A (no driver)	1	bbr	128MB	network-throughput	layer2+3	0
tsys6	performance	10	bbr	128MB	virtual-host	layer3+4	24 (9 mounts × 4... actually 6 options lines)
tsys7	performance	10	bbr	128MB	virtual-host	layer3+4	24
tsys9	performance	10	bbr	128MB	virtual-host	layer2+3	N/A

Inconsistencies:

• tsys4: rmem/wmem = 16 MB (fleet standard: 128 MB). Tuned profile wrong (throughput-performance should be network-throughput). Missing nconnect.
• tsys5: Missing nconnect on NFS client mounts. No cpufreq driver (Westmere — expected, not a bug).
• tsys4/5 bond hash = layer2+3 — should be layer3+4 (like tsys6/7).

---

5. VM inventory + config audit (running VMs only)

5.1 Per-VM disk + CPU + network settings

cpu: host = best (full instruction set passthrough). x86-64-v2-AES = good (nested-virt-safe baseline). cache=writethrough = safe but slow for NFS. iothread=1 = present on most. agent: 1 = qemu-guest-agent enabled.

VMID	Name	Host	vCPU	CPU type	RAM	Disk store	Cache	iothread	qga	Notes
100	pfv-bms	tsys1	2	host	4G	D2 (raw)	—	—	✓	discard=on,ssd=1
101	tsys-ca	tsys1	2	v2-AES	2G	D2 (virtio)	—	✓	—	no nested-virt
103	netinfra-01	tsys1	2	v2-AES	4G	S2	—	✓	—
104	librenms	tsys1	2	host	2G	D2	—	✓	✓
105	proxmox-dc	tsys1	2	host	2G	D2	—	✓	✓
106	cnode3	tsys1	4	host	4G	D5	writethrough	✓	✓
108	ucs-01	tsys1	2	host	8G	D2	writethrough	✓	✓
114	kali-tsys	tsys1	2	host	2G	D2	writethrough	✓	✓
117	workbench	tsys1	2	host	4G	D2	writethrough	✓	✓
313	wnode-tsys3	tsys3	8	host	28G	D5	writethrough	✓	✓
400	PBS	tsys4	2	—	2G	local-lvm	—	—	—	backup server
509	wnode-tsys5	tsys5	2	host	32G	D2 (virtio)	—	✓	—	low vCPU (2) for k8s wnode
5111	ultix-streaming	tsys5	4	—	9G	ssd (local)	—	—	—	288GB on SSD (93% full)
5112	ultix-offstage	tsys5	4	—	6G	local-lvm	—	—	—	288GB
100	wnode-tsys6	tsys6	2	host	64G	S4	—	✓	—	low vCPU (2), high RAM
600	tsys-awx	tsys6	2	host	12G	D5	writethrough	✓	✓
701	wnode-tsys7	tsys7	4	host	32G	S3 (virtio)	—	✓	—
702	hfnoc-uisp	tsys7	2	host	8G	S1	—	✓	✓	100GB disk
703	rr-middleware	tsys7	2	host	2G	S2	writethrough	✓	✓
704	TCTC	tsys7	4	host	6G	D2 (virtio)	—	✓	—
705	cnode2	tsys7	4	host	4G	S2	writethrough	✓	✓
901	tsys-siem	tsys9	2	host	8G	D2	writethrough	✓	✓	132GB disk
902	ucs-02	tsys9	2	host	8G	S2	—	✓	✓	50GB disk
903	kali-rd	tsys9	2	host	2G	D5	writethrough	✓	✓
904	netinfra-02	tsys9	2	v2-AES	4G	D2	—	✓	—
905	wnode-tsys9	tsys9	4	v2-AES	4G	S2	—	✓	✓	target: grow to 6c/8G
906	cnode1	tsys9	2	host	4G	S3	—	✓	—

5.2 VM optimization findings

Finding	VMs affected	Impact	Fix
cache=writethrough on NFS	106,108,114,117,313,600,703,705,901,903	Writethrough forces synchronous writes — slow on NFS. writeback or none is better for NFS-backed qcow2.	Change cache mode (requires VM stop/start)
Mixed CPU types	Some host, some x86-64-v2-AES	Prevents migration between hosts with different CPU types. K8s cnodes/wnodes should match.	Standardize per workload tier
Low vCPU on k8s wnodes	wnode-tsys5 (2c), wnode-tsys6 (2c)	K8s workers undersized for compute workloads	Grow after migrations free RAM
VM 500 references D7	DellOpenManageEnterprise (stopped)	D7 export no longer exists on tsys4	Delete or re-disk before starting
qga inconsistent	Several VMs missing agent:1	No guest-agent communication	Enable qga fleet-wide

---

6. Console infrastructure status (fixed during this audit)

Bug found + fixed: console/generate-config.sh wrote udev rules with SYMLINK+="console/$name" (singular) but ser2net.yaml opened /dev/consoles/$name (plural). They never matched. After every reboot, all 7 console ports failed until setup.sh's manual fallback re-created symlinks.

Fix applied: Changed udev rule to SYMLINK+="consoles/$name". Re-created all 7 symlinks manually on pfv-tsys4. Restarted ser2net + conmand. All 7 consoles now working (verified via conman-console.py).

Also fixed: stale conmand process (pid 988) holding config lock, preventing conmand systemd service from starting.

---

7. Findings summary (prioritized)

Critical (cross-rack performance)

#	Finding	Impact	Type
1	core-sw01 LAG hash = layer-2-3 (no L4 ports)	All NFS traffic between a given host pair caps at 1 Gbps	Switch config
2	core-sw01 ch1 cable on g16, not g17	Only 3 of 4 cross-rack links active (3 Gbps, not 4)	Switch config
3	Both switches use mode on (static, no LACP)	No failure detection, no partner validation	Switch config
4	tsys5 bond0 degraded (1 of 2 slaves)	Single link to storage, no redundancy	Physical (cable)

High (storage configuration)

#	Finding	Impact	Type
5	tsys5 NVMe not configured (CT500P1 500GB)	Best local storage in fleet sitting unused	Config
6	tsys5 SK hynix SSD not configured (512GB)	Relocated from tsys4, not mounted	Config
7	D2 still carries 13 running VMs	IOPS hotspot persists	VM placement
8	tsys5 SSD 93% full	Samsung 860 PRO nearly exhausted	Capacity
9	S3/S4 stale on tsys5	Broken exports, stale storage.cfg	Config

Medium (fleet consistency)

#	Finding	Impact	Type
10	tsys4 on PVE 9.1.5/kernel 6.17	Behind fleet (9.2.5/7.0.14)	Upgrade
11	tsys4 rmem/wmem = 16MB	Suboptimal TCP buffers for NFS	Tuning
12	tsys4/5 bond hash = layer2+3	Suboptimal for nconnect NFS	Tuning
13	tsys5 missing nconnect	NFS client mounts to tsys4 lack multi-connection	Tuning
14	cache=writethrough on 10+ VMs	Synchronous writes slow NFS	VM config
15	Stale tsys8 switch config (tor3-stor ch4)	Dead LAG group for retired host	Cleanup

Low (monitoring / investigation)

#	Finding	Impact	Type
16	core-sw01 port errors (g9:2902, g38:590, g26:73)	Possible cable/CRC issues	Investigate
17	No continuous observability	Only point-in-time check.sh snapshots	Monitoring
18	/mnt/albert unused (tsys4 Hitachi 2TB)	Potential new Tier 2 storage	Config
19	VM 500 references non-existent D7	Would fail on boot	Cleanup

---

9. Observability stack (operator direction)

The operator has defined a clear monitoring architecture with three tools, each serving a distinct role. Do NOT duplicate functionality across tools.

9.1 Current architecture

Tool	Where it runs	Scope	Role
Beszel	Cloudron VPS (Reston, VA)	Tailscale-connected hosts	CPU/RAM/disk/time-series metrics on all Proxmox hosts
Uptime Kuma	Cloudron VPS (Reston, VA)	All hosts + services	Up/down status + alerting via Pushover
LibreNMS (VMID 104, tsys1)	On-prem (R&D cluster)	Non-Tailscale devices only	SNMP polling for switches + devices that can't run Beszel agent
Home Assistant (VMID 100, tsys1)	On-prem (R&D cluster)	Physical infrastructure	Power/temperature instrumentation + alerting (UPS, PDU outlets, etc.)

9.2 Division of responsibility (no duplication)

Metric class	Tool	Notes
CPU / RAM / disk / network time-series	Beszel	All Proxmox hosts (Tailscale-connected)
Up/down alerting (hosts + services)	Uptime Kuma	Pushover notifications
SNMP polling (switches, non-Tailscale gear)	LibreNMS	Port counters, LAG status, interface errors, switch health. Fill the gap Beszel can't cover (switches have no Beszel agent)
Power / temperature / physical env	Home Assistant	UPS (NUT), PDU outlets, temperature sensors. Alerting via HA automations

9.3 Saturday "OAM Day" plan (2026-08-01)

The operator has designated Saturday as a dedicated operations/admin/management day with the goal of achieving 100% instrumentation coverage:

• Up/down alerting on ALL hosts + key services (Uptime Kuma → Pushover)
• Centralized syslog (tool TBD — candidates: dedicated syslog VM, rsyslog relay, or Loki/Grafana stack)
• SNMP instrumentation: wire LibreNMS for all 4 switches + any non-Tailscale devices (community: kn3lmgmt, IPs: 192.168.0.7/8/9/12)
• Beszel agent coverage on all 7 Proxmox hosts
• Home Assistant: wire power/temp alerting (UPS via NUT already live, PDU via powerman, any temp sensors)
• Verify alerting paths end-to-end (test each alert fires correctly)

Much of the groundwork is already done: NUT is live on tsys1, powerman is live on tsys1, LibreNMS (VMID 104) is running, conman console logging is live on tsys4, check.sh provides point-in-time snapshots. Saturday is about wiring continuous monitoring + alerting on top of these.

---

8. Recommendations (for operator review — NO changes applied)

8.1 Switch changes (require maintenance window + operator approval)

1. Fix core-sw01 ch1 member list: g(13-15,17) → g(13-16). This makes the 4th live cable (g16) part of the LAG.
2. Fix core-sw01 LAG hash: layer-2-3 → layer-2-3-4 (match tor3-stor). This unlocks multi-link distribution for nconnect NFS traffic.
3. Convert trunk to LACP: Change mode on → mode active on both switches for the cross-rack trunk (ch1). Enables failure detection + auto-negotiation.
4. Remove stale tsys8 config: Delete ch4 + g13/g14 channel-group on tor3-stor.
5. Investigate port errors: g9 (2902 inErr), g38 (590 inErr), g26 (73 inErr) on core-sw01.

Recommended order: Fix #2 (hash) + #1 (member) first — these are the highest-impact changes with the lowest risk. LACP conversion (#3) is optional if the static trunk is working reliably otherwise. All changes can be done read-only-verified via the conman + SNMP tooling in this repo.

8.2 Storage configuration (host-side, no switch changes needed)

1. Mount + configure tsys5 NVMe (CT500P1 500GB) as LVM-thin (nvme-local) per TODO.md §3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed scratch.
2. Mount + configure tsys5 SK hynix SSD (512GB). Add as dir: ssd2 or similar. Offloads the 93%-full Samsung 860 PRO.
3. Clean S3/S4: Remove stale S3 export from /etc/exports. Fix/remove broken dir: S4 entry in storage.cfg.
4. Decide on /mnt/albert (tsys4 Hitachi 2TB): export as new NFS storage or keep local-only for PBS scratch.

8.3 Fleet consistency + tuning (host-side)

1. Upgrade tsys4 to PVE 9.2.5 / kernel 7.0.14 (matches fleet).
2. Apply apply-tunings.sh --apply to tsys4 and tsys5:
   • Fix rmem/wmem to 128MB (tsys4)
   • Fix tuned profile (tsys4 → network-throughput)
   • Add nconnect=4 to tsys5 NFS client mounts
   • Fix bond hash to layer3+4 (tsys4/5 — requires ifreload, brief network drop)
3. Audit VM cache modes: Change writethrough → writeback or none for NFS- backed qcow2 disks (requires VM stop/start).

8.4 VM placement rebalancing (refer to CAPACITY-GOALS.md)

The new NVMe + SSD on tsys5, the unmounted /mnt/albert on tsys4, and the 3 empty Tier 3 USB disks (D1, D3, /mnt/backup) create opportunities for IOPS spreading that didn't exist in the 07-29 audit. Detailed placement plan deferred to the operator review of this document.