KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
PFVCluster/PROJECT.md
T
mrcharles 6814374bac fix: keep UCS on spinning disk, repurpose D3 SSD for k8s/CI workloads 
UCS (LDAP/AD) does not need SSD. Corrected placement:
- ucs-01 stays on D2 (tsys4 HDD) -- no change needed
- ucs-02 moves from D5 (tsys4) to S2 (tsys5 HDD) for cross-server
  redundancy only, not for speed

D3 SSD (moving to tsys5 Friday) is now freed for workloads that
actually benefit: sectestbed k8s nodes, CI/CD artifact cache, or a
container image registry mirror.

Added storage tier characterization per host (tsys4 = bulk/spinning
only, tsys5 = hybrid fast-tier hub, tsys2/3 = SSD/NVMe only). Added
UCS migration TODO item for today.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:51:30 -05:00

37 KiB
Raw Permalink Blame History

Proxmox Cluster Project Report

Date: 2026-07-27 Prepared by: Performance Optimization Engagement Status: Comprehensive fleet assessment with VM placement and redundancy analysis

Table of Contents

  1. Executive Summary
  2. Host Fleet
  3. Storage Architecture
  4. VM Fleet Inventory
  5. Kubernetes Node Distribution
  6. Storage Redundancy Analysis
  7. Local SSD/NVMe Opportunity
  8. Role Alignment Audit
  9. Network Findings
  10. Recommendations
  11. Hardware End-of-Support Exposure
  12. Open Items

1. Executive Summary

The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2), running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked on physical hardware work scheduled for Friday.

The VM-layer assessment reveals three structural issues:

# Finding Severity Fix Cost
1 Every pfv-k8s node stores its disk on tsys4 NFS (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. CRITICAL $0 (rebalance)
2 No k8s node uses SSD or NVMe. All boot from NFS on spinning disks, adding latency to every etcd write and container pull. HIGH $0 (move to local SSD where available)
3 Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only. tsys4 failure takes down both halves of each pair. HIGH $0 (move -02 halves to tsys5 NFS)

Additionally, tsys9 was validated this session and revealed two new findings: its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has a 250 GB local SSD that no VM is using.

2. Host Fleet

2.1 Inventory

Host Model CPU (year) Cores RAM Local Disk Role (intended) Tuning
pfv-tsys1 OptiPlex 9020 i7-4770 Haswell (2013) 4c/8t 32 GB DDR3 HDD (LVM-thin) Infrastructure Done
pfv-tsys2 Precision 5520 i7-7820HQ Kaby Lake (2017) 4c/8t 32 GB (max) NVMe 512GB + SATA SSD 1TB Kubernetes Incoming (Win10)
pfv-tsys3 Precision 7510 Xeon E3-1535M v5 Skylake (2015) 4c/8t 32 GB DDR4 HDD (LVM-thin) Kubernetes Done
pfv-tsys4 Precision T1700 Xeon E3-1246 v3 Haswell (2013) 4c/8t 16 GB DDR3 6 disks (HDD+SSD+SMR) Storage (NFS+PBS) Blocked (NIC+RAM)
pfv-tsys5 Precision T7500 Xeon E5620 Westmere (2010) 4c/8t 96 GB DDR3 6 disks (HDD+SSD) Storage (NFS+VMs) Blocked (cable)
pfv-tsys6 PowerEdge R610 2x Xeon E5530 Nehalem (2009) 8c/16t 128 GB DDR3 HDD (LVM-thin) Kubernetes Done
pfv-tsys7 PowerEdge R620 2x Xeon E5-2630 v2 Ivy Bridge (2013) 12c/24t 192 GB DDR3 HDD (LVM-thin) Kubernetes Done
pfv-tsys9 OptiPlex 7080 i5-10500 Comet Lake (2020) 6c/12t 24 GB DDR4 250 GB SSD (PNY CS900) Infrastructure Done

2.2 Role taxonomy (per user directive)

Role Hosts Workload
Infrastructure + k8s control tsys1, tsys9 Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes
Kubernetes workers tsys2, tsys3, tsys6, tsys7 pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC
Storage tsys4, tsys5 NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs

Design rationale: cnodes (control plane) are lightweight (4 cores, 4 GB RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts (tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes run one per hypervisor host across the fleet to maximize total cluster capacity.

pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA running Cloudron -- not in this cluster.

3. Storage Architecture

3.1 NFS exports from tsys4 (primary storage server)

Export Disk model Type Bus Total Used Free Use%
D2 WDC WD30EFRX Red HDD (7200rpm) SATA 2.7 TB 187 GB 2.4 TB 8%
D3 SK hynix SC300 SSD USB — — — moving to tsys5 Friday
D5 Hitachi HDS72302 HDD (7200rpm) SATA 1.8 TB 233 GB 1.5 TB 14%

Non-exported disks on tsys4:

  • sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
  • sdd (WDC 1T) -- idle, unmounted, removable (free up for other use)
  • sdf (WDC 4.5T SMR) at /mnt/backup -- PBS backup target, 4.3 TB free

D3 migration (Friday): The SK hynix SC300 SSD is currently USB-attached on tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port, eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb /sdc/sdd), so tsys5 is the better target. See section 3.2.

3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)

Export Disk model Type Bus Total Used Free Use%
S1 Seagate ST1000VN HDD SAS 916 GB 53 GB 817 GB 7%
S2 Seagate ST1000VN HDD SAS 916 GB 2 MB 870 GB 0%
S3 Seagate ST1000VN HDD SAS 916 GB 3 MB 870 GB 0%
S4 Toshiba DT01ACA050 HDD SAS 458 GB 2 MB 435 GB 0%
T5-SSD Samsung 860 PRO SSD SAS 234 GB 83 GB 140 GB 35%
D3 (Friday) SK hynix SC300 SSD SAS 469 GB 2 MB 445 GB 0%

tsys5 storage controllers (plenty of free ports):

  • LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate), 5 free
  • Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), 2 free
  • Intel ICH10 SATA #2 (2-port): status unknown, likely free
  • 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)

Key finding: S2, S3, and S4 are all 99% empty. The secondary storage server has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to reduce tsys4's blast radius.

Friday additions:

  1. D3 (SK hynix SSD) moves from tsys4 USB to tsys5 SAS port. Eliminates USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
  2. PCI NVMe drive (local-only, not NFS-exported). Used for wnode-tsys5 boot disk and HPC scratch. The fastest tier in the fleet.

After Friday, tsys5 consolidates all fast storage: NVMe (local) + 2 SSD NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the natural home for latency-sensitive workloads and the k8s StorageClass design center.

3.3 Local storage tiers (per host, with utilization)

Host Storage ID Disk type Bus Total Used Free Used by VMs?
tsys1 local-lvm HDD SATA ~90 GB low ~90 GB No (all VMs on NFS)
tsys3 local-lvm NVMe (Samsung PM961) NVMe 349 GB 0 GB 349 GB No (all VMs on NFS)
tsys4 local-lvm HDD SATA ~94 GB PBS VM ~62 GB Yes (PBS VM 400)
tsys5 local-lvm HDD (Hitachi 1.8T) SATA 1.7 TB 40 MB 1.7 TB No
tsys5 local-nonprod HDD (Seagate 1T, =S1) SATA 916 GB 53 GB 856 GB Yes (sectestbed suite)
tsys6 local-lvm HDD (WD My Passport) USB 2.0 1.7 TB 0 GB 1.7 TB No -- DO NOT USE for VM storage
tsys7 local-lvm HDD (WD portable) USB 2.0 1.7 TB 0 GB 1.7 TB No -- DO NOT USE for VM storage
tsys9 local-lvm SSD (PNY CS900) SATA 136 GB 0 GB 136 GB No (all VMs on NFS)
tsys2 NVMe (Samsung 960 PRO) NVMe NVMe 512 GB (Win10) 512 GB Incoming -- fastest boot tier after tsys5 NVMe
tsys2 SATA SSD (Samsung 850 EVO) SSD SATA 1 TB (Win10) 1 TB Incoming -- bulk data on SSD, not rust

CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.

The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a single USB 2.0-attached WD My Passport portable HDD (tsys6: "My Passport 260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI (USB 2.0, ~480 Mbps). There is no USB 3.0/xHCI on either host.

USB 2.0 practical throughput is ~30-35 MB/s. This is 3-4x SLOWER than NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these hosts would decrease performance. local-lvm on tsys6/7 must NOT be used for VM storage.

Additionally, both servers have completely empty internal drive bays:

  • tsys6: SAS controller present but DISABLED in BIOS. No internal drives.
  • tsys7: 6-port SATA AHCI controller present, 5 ports EMPTY (only DVD-ROM on port 5). No internal drives.

This is a reliability risk beyond performance: the entire host OS boots from a consumer-grade portable USB drive not designed for 24/7 server use.

tsys3 correction: Previously documented as HDD. Actually boots from a Samsung PM961 NVMe 512GB SSD -- the fastest existing local storage in the fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.

Critical observation: every k8s host has 0% used local-lvm. tsys3 (Samsung PM961 NVMe, 349 GB), tsys6 (WD My Passport USB 2.0 HDD, 1.7 TB), tsys7 (WD portable USB 2.0 HDD, 1.7 TB) all have unused local storage.

However, only tsys3's local-lvm is suitable for VM storage. tsys6 and tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than NFS-over-HDD and unsuitable for wnode boot disks.

3.4 Disk speed tiers summary

Tier Where Speed class Best for
NVMe tsys3 (Samsung PM961), tsys5 (Friday addition), tsys2 (Samsung 960 PRO 512GB) 2000-3500 MB/s HPC scratch, ETL staging, container runtime, wnode boot, etcd
Local SSD tsys9 (PNY CS900, 136 GB), tsys2 (Samsung 850 EVO 1TB) 500 MB/s wnode boot disk, etcd
NFS-SSD tsys5 D3 (SK hynix, SAS post-Friday), tsys5 T5-SSD (Samsung) 200-400 MB/s over NFS UCS/LDAP, latency-sensitive VMs
NFS-HDD tsys4 D2/D5, tsys5 S1-S4 80-120 MB/s over NFS Bulk data, large disks, non-critical VMs, wnode boot on tsys6/7
Local SATA HDD tsys1 local-lvm 100-150 MB/s Host OS only
USB 2.0 HDD tsys6/7 local-lvm (WD My Passport) ~30-35 MB/s NOTHING -- slower than NFS, do not use for VMs

3.5 Storage tier characterization per host

Host Storage profile Detail
tsys2 SSD/NVMe only 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk.
tsys3 NVMe only Samsung PM961 NVMe 512GB. No spinning disk.
tsys5 Hybrid (fast-tier hub) NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD
tsys4 Bulk/spinning disk only D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD.
tsys9 Local SSD + NFS PNY CS900 SSD 136GB local + NFS client
tsys1 Local HDD + NFS Small local-lvm + NFS client
tsys6/7 NFS only local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS.

3.6 All exports are single-disk with no redundancy

Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS mirror, no mdraid. A single disk failure takes down every VM whose disk lives on that export. This applies to both storage servers and to the PBS backup target (a single 4.5T SMR drive).

4. VM Fleet Inventory

4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)

tsys1 (Infrastructure) -- 11 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
100 pfv-bms (HomeAssistant) 2 4096 32 GB D2 (tsys4 HDD) NFS
101 tsys-ca 2 2048 32 GB D2 (tsys4 HDD) NFS
103 pfv-netinfra-01 2 2048 32 GB D5 (tsys4 HDD) NFS
104 tsys-librenms 2 2048 50 GB D2 (tsys4 HDD) NFS
105 tsys-proxmox-datacenter 2 2048 32 GB D2 (tsys4 HDD) NFS
106 pfv-k8s-cnode3 2 4096 32 GB D2 (tsys4 HDD) NFS
107 pfv-k8s-cnode1 2 4096 32 GB D5 (tsys4 HDD) NFS
108 tsys-ucs-01 2x2 8000 32 GB D2 (tsys4 HDD) NFS
109 tailscale-router 2 2048 25 GB D2 (tsys4 HDD) NFS
114 kali-tsys 2 2048 32 GB D2 (tsys4 HDD) NFS
117 tsys-secure-workbench 2 4000 32 GB D2 (tsys4 HDD) NFS

tsys3 (Kubernetes) -- 1 running VM

VMID Name Cores RAM (MB) Disk Storage Tier
313 pfv-k8s-wnode-tsys3 4x2 20000 32 GB D5 (tsys4 HDD) NFS

tsys4 (Storage) -- 1 running VM

VMID Name Cores RAM (MB) Disk Storage Tier
400 pfv-proxmox-backup-server 2 2048 32 GB local-lvm LOCAL

tsys5 (Storage) -- 15 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
509 pfv-k8s-wnode-tsys5 2x4 32000 32 GB D2 (tsys4 HDD) NFS
5101 sectestbed-siem 2x2 10000 132 GB local-nonprod LOCAL
5105 sectestbed-awx 2x2 4096 288 GB local-nonprod LOCAL
5106 sectestbed-k8s-cnode 2x2 4096 32 GB local-nonprod LOCAL
5107 sectestbed-k8s-wnode 2x2 4096 32 GB local-nonprod LOCAL
5108 sectestbed-librenms 2x2 4096 32 GB local-nonprod LOCAL
5109 sectestbed-netinfra 2x2 4096 32 GB local-nonprod LOCAL
5111 ultix-streaming 2x2 9000 288 GB T5-SSD (tsys5 SSD) NFS-SSD
5112 ultix-offstage 2x2 6000 288 GB local-lvm LOCAL
6000 sectestbed-sandbox 2x2 4096 32 GB local-nonprod LOCAL
51010 sectestbed-tctc 2x2 4096 32 GB local-nonprod LOCAL
51011 sectestbed-cloudron 2x2 4096 32 GB local-nonprod LOCAL
51012 sectestbed-hfnoc 2x2 4096 32 GB local-nonprod LOCAL
51013 sectestbed-rancherplatform 2x2 4096 32 GB local-nonprod LOCAL
53100 tsys-preprod-awx 2x2 9000 160 GB local-nonprod LOCAL
53101 tsys-preprod-siem 2x2 12000 32 GB local-nonprod LOCAL
53102 tsys-preprod-rancherplatform 2x2 8000 32 GB local-nonprod LOCAL

tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)

VMID Name Cores RAM (MB) Disk Storage Tier
100 pfv-k8s-wnode-tsys6 2x2 32000 32 GB D5 (tsys4 HDD) NFS -- STOPPED
600 tsys-awx 2x2 12000 32 GB D2 (tsys4 HDD) NFS -- STOPPED
601 pfv-k8s-cnode4 4 4096 32 GB D2 (tsys4 HDD) NFS
602 pfv-k8s-cnode5 4 4096 32 GB D5 (tsys4 HDD) NFS
603 pfv-k8s-cnode2 4 4096 32 GB D2 (tsys4 HDD) NFS

tsys7 (Kubernetes) -- 4 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
701 pfv-k8s-wnode-tsys7 4 32000 32 GB D5 (tsys4 HDD) NFS
702 hfnoc-uisp 2x2 8000 100 GB D2 (tsys4 HDD) NFS
703 rr-middleware 2 2048 32 GB D2 (tsys4 HDD) NFS
704 TCTC 4 6000 32 GB D2 (tsys4 HDD) NFS

tsys9 (Infrastructure) -- 5 running VMs

VMID Name Cores RAM (MB) Disk Storage Tier
901 tsys-siem 2 8000 132 GB D2 (tsys4 HDD) NFS
902 tsys-ucs-02 2x2 8000 50 GB D5 (tsys4 HDD) NFS
903 kali-rd 2 2048 32 GB D5 (tsys4 HDD) NFS
904 pfv-netinfra-02 2 4000 32 GB D2 (tsys4 HDD) NFS
905 pfv-k8s-wnode-tsys9 4 4096 32 GB S3 (tsys5 HDD) NFS

4.2 Storage concentration summary

Storage target # of running VMs % of fleet
D2 (tsys4 WDC Red 3TB HDD) 18 45%
D5 (tsys4 Hitachi 2TB HDD) 9 23%
local-nonprod (tsys5 local HDD) 10 25%
S3 (tsys5 Seagate 1TB HDD) 1 3%
T5-SSD (tsys5 Samsung SSD) 1 3%
local-lvm (tsys4 local) 1 3%

27 of 40 running VMs (68%) store their disks on tsys4 NFS exports. If tsys4 goes offline, two-thirds of the fleet loses its storage.

5. Kubernetes Node Distribution

5.1 pfv-k8s cnode (control plane) placement

VMID Name Hypervisor Storage Disk Type Host Role Compliant?
107 cnode1 tsys1 D5 (tsys4) HDD NO (infra host)
106 cnode3 tsys1 D2 (tsys4) HDD NO (infra host)
603 cnode2 tsys6 D2 (tsys4) HDD YES
601 cnode4 tsys6 D2 (tsys4) HDD YES
602 cnode5 tsys6 D5 (tsys4) HDD YES

Findings:

  • 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
  • All 5 cnodes store disks on tsys4 NFS. Zero use tsys5 storage.
  • 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
  • All cnodes on spinning HDDs. No SSD/NVMe use.
  • etcd quorum risk: if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes inoperable.

5.2 pfv-k8s wnode (worker) placement

VMID Name Hypervisor Storage Disk Type Status Host Role Compliant?
313 wnode-tsys3 tsys3 D5 (tsys4) HDD running YES
509 wnode-tsys5 tsys5 D2 (tsys4) HDD running NO (storage host)
100 wnode-tsys6 tsys6 D5 (tsys4) HDD STOPPED YES (but offline)
701 wnode-tsys7 tsys7 D5 (tsys4) HDD running YES
905 wnode-tsys9 tsys9 S3 (tsys5) HDD running NO (infra host)

Findings:

  • 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
  • wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) -- it depends on tsys4 for its disk despite running on tsys5.
  • wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
  • wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it is the ONLY k8s node using tsys5 storage (which is good for redundancy).
  • All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.

5.3 Summary: k8s node storage distribution

Storage server cnodes wnodes Total k8s nodes
tsys4 NFS 5 (100%) 4 (80%) 9 (90%)
tsys5 NFS 0 1 (20%) 1 (10%)
Local SSD/NVMe 0 0 0 (0%)

90% of the kubernetes cluster's storage depends on a single host (tsys4). This is the single most critical risk in the VM layer.

5.4 sectestbed k8s nodes (separate from pfv-k8s)

tsys5 also hosts a separate sectestbed kubernetes stack using local storage:

VMID Name Storage
5106 sectestbed-k8s-cnode local-nonprod (local HDD)
5107 sectestbed-k8s-wnode local-nonprod (local HDD)

These are on local storage (good -- no NFS dependency) but on a single host's single local disk (no redundancy). They are isolated from the pfv-k8s cluster.

6. Storage Redundancy Analysis

6.1 -01/-02 infrastructure pair audit

Two -01/-02 pairs exist in the fleet:

Pair 1: pfv-netinfra (network infrastructure)

Role VMID Host Storage NFS Server
-01 103 tsys1 D5 tsys4
-02 904 tsys9 D2 tsys4

Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED. Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their disks. The -02 half should be on an S2/S3/S4 export from tsys5.

Pair 2: tsys-ucs (Univention Corporate Server)

Role VMID Host Storage NFS Server
-01 108 tsys1 D2 tsys4
-02 902 tsys9 D5 tsys4

Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED. Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.

6.2 Redundancy principle for paired VMs

For any -01/-02 pair to survive a single storage server failure:

-01 VM disk -> tsys4 NFS export (D2/D3/D5)
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)

This ensures that losing either tsys4 or tsys5 takes down only one half of the pair. Currently, both pairs fail this test because both halves are on tsys4.

6.3 NFS server failure blast radius

If tsys4 goes offline (USB NIC failure, disk failure, reboot):

Impact Count
k8s cnodes that lose storage 5 of 5 (etcd quorum lost)
k8s wnodes that lose storage 4 of 5
Infrastructure VMs that lose storage 11 of 12 on tsys1 (all on D2/D5)
Total VMs that lose storage 27 of 40 (68%)

If tsys5 goes offline:

Impact Count
k8s cnodes that lose storage 0 of 5
k8s wnodes that lose storage 1 of 5
Total VMs that lose storage 1 of 40 (3%)

tsys4 is a massive blast-radius liability. tsys5 is barely used. Rebalancing VM storage across both servers dramatically reduces risk.

7. Local SSD/NVMe Opportunity

7.1 Available fast tiers (currently unused by k8s)

Host Device Type Size Available for VMs? Currently used by k8s?
tsys4 D3 (SK hynix SC300, USB) SSD 512 GB Yes (via NFS) No
tsys5 T5-SSD (Samsung 860 PRO) SSD 256 GB Yes (via NFS) No (used by ultix-streaming)
tsys5 New NVMe (Friday) NVMe TBD Yes (local or NFS) No
tsys9 local-lvm (PNY CS900) SSD 137 GB free Yes (local) No

7.2 Why local storage matters for k8s nodes

Kubernetes nodes are latency-sensitive in two specific areas:

  1. etcd (control plane): etcd writes are synchronous and latency-critical. On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms. This directly affects k8s API responsiveness and pod scheduling speed.

  2. Container image pulls: Worker nodes pull container images frequently. On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD eliminates the network hop and reduces seek time. This matters most for RackRental/containerlab workloads that spin up containers rapidly.

7.3 Current waste: tsys9 local SSD

tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available. Zero VMs use it. All 5 VMs on tsys9 boot from NFS. The local SSD sits idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD -- its disk is currently on S3 (tsys5 NFS over a Seagate HDD).

7.4 Upcoming opportunity: tsys5 NVMe (Friday)

The PCI NVMe being added to tsys5 will be the fastest storage tier in the fleet. Two placement options:

Option A: NFS-export the NVMe (shared). All hosts can use it. Good for VMs that might need migration. Adds the NFS/network overhead back.

Option B: Local-only on tsys5. VMs on tsys5 get full NVMe speed with no network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be accessed from other hosts.

Recommendation: Option B (local-only). k8s worker nodes do not need shared storage -- pods are ephemeral and reschedule on failure. The NVMe should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5 and used for local VM images.

8. Role Alignment Audit

Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is correct by design (keeps heavy hosts free for workers).

8.1 VMs that need to move

VMID Name Current host Issue Target
509 pfv-k8s-wnode-tsys5 tsys5 (storage) Worker on storage host tsys7 or tsys2 (when online)
905 pfv-k8s-wnode-tsys9 tsys9 (infra) Can stay if small; user decides tsys9 OK if small wnode

8.2 Host capacity for k8s nodes

Host Role Current k8s nodes k8s RAM used RAM total Headroom
tsys1 Infra+k8s ctrl 2 cnodes 8 GB 32 GB ~12 GB (after 11 infra VMs)
tsys3 K8s worker 1 wnode 20 GB 32 GB ~12 GB
tsys6 K8s worker 3 cnodes + 1 wnode (stopped) 12 GB 128 GB ~116 GB
tsys7 K8s worker 1 wnode 32 GB 192 GB ~160 GB
tsys9 Infra+k8s ctrl 1 wnode 4 GB 24 GB ~12 GB (after 4 infra VMs)
tsys2 K8s worker 0 (incoming) 0 32 GB ~32 GB

tsys6 and tsys7 are dramatically underutilized -- 116 GB and 160 GB of free RAM respectively. They should be the primary targets for heavy worker nodes and ETL/HPC workloads.

8.3 tsys6 wnode-tsys6 is stopped

VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4 NFS). This wnode should be restarted (or recreated on local-lvm) to restore cluster capacity.

9. Network Findings

9.1 tsys9 storage NIC is a USB dongle (new finding)

Validating tsys9 revealed that its storage network interface (enx9c69d36a5b6c) is USB-attached (parentbus usb). This is the same anti-pattern as tsys4. The onboard Intel NIC (enp0s31f6) is used for management; storage uses the USB adapter.

Impact: Same as tsys4 -- achieves line rate but is susceptible to cable wobble, ESD, and USB controller resets. For an infrastructure host with 5 VMs, this is a reliability risk.

Mitigation: tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A PCIe NIC would eliminate this risk (same recommendation as tsys4).

9.2 tsys4 and tsys5 still blocked (Friday hardware work)

Host Blocker Staged fix
tsys4 USB cdc_ncm storage NIC PCIe NIC install + RAM upgrade (16 to 64 GB)
tsys5 bond0 broken (1 of 2 slaves) Plug 2nd ethernet cable + apply layer3+4 hash

9.3 LACP resolved on tsys6/tsys7

tsys6 to tsys7 storage path now measures 1.83 Gbps (was 943 Mbps). The switch LACP hash change took effect after renegotiation. The 56-106K retransmits on this path are confirmed to be non-lossy multi-flow TCP-over- LACP overhead. See RESULTS.md for the full analysis.

9.4 NFS nconnect=4 + noatime confirmed active

All hosts (including tsys9) show nconnect=4,noatime in their NFS mount options. Each host maintains 4 TCP connections per NFS mount to each storage server. This was the Tier 0 tuning item from the performance optimization engagement and is confirmed working cluster-wide.

10. Recommendations

No changes have been made. These are assessment-only recommendations. See K8S.md for the detailed kubernetes architecture deep-dive.

10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)

These are the most critical production VMs in the fleet. They must survive any single-point failure (host, storage server, or disk).

Design principle for -01/-02 HA pairs:

  • Different hypervisors (already satisfied: tsys1 vs tsys9)
  • Different storage servers (currently FAILED: all on tsys4)
  • Prefer SSD for latency-sensitive services

Recommended placement:

VM Host Storage Tier Free space Rationale
netinfra-01 (103) tsys1 D5 (tsys4 HDD) NFS-HDD 1.5 TB DNS/DHCP/NTP = minimal I/O. Stays put.
netinfra-02 (904) tsys9 S3 (tsys5 HDD) NFS-HDD 870 GB Move from D2. Cross-server redundancy. Minimal I/O.
ucs-01 (108) tsys1 D2 (tsys4 HDD) NFS-HDD 2.4 TB Stays put. LDAP/AD does not need SSD.
ucs-02 (902) tsys9 S2 (tsys5 HDD) NFS-HDD 870 GB Move from D5. Cross-server redundancy. No SSD needed.

Failure survival matrix (all single-point failures):

Failure netinfra-01 netinfra-02 ucs-01 ucs-02 Result
tsys4 dies dies (D5) alive (S3) dies (D2) alive (S2) netinfra-02 + ucs-02 alive
tsys5 dies alive (D5) dies (S3) alive (D2) dies (S2) netinfra-01 + ucs-01 alive
tsys1 dies dies alive dies alive -02 pair survives
tsys9 dies alive dies alive dies -01 pair survives
Any single disk all 4 on different disks/servers all 4 alive

Why this works: Every row has at least one netinfra and one UCS alive. The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.

D3 SSD repurposed: With UCS staying on HDD, the D3 SSD (moving to tsys5 Friday) is freed for latency-sensitive workloads that actually benefit from SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image registry mirror. Not infrastructure VMs.

10.2 CRITICAL: Cnode (control plane) storage split

Problem: All 5 cnodes store disks on tsys4. D2 disk failure loses etcd quorum (3 of 5 cnodes share D2).

Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.

cnode Current host Target host Current storage Target storage Rationale
cnode1 (107) tsys1 tsys1 (stays) D5 (tsys4) D5 (tsys4) -- no change Already correct
cnode3 (106) tsys1 tsys1 (stays) D2 (tsys4) S2 (tsys5) -- MOVE disk Split storage to tsys5
cnode2 (603) tsys6 tsys9 D2 (tsys4) D2 (tsys4) -- no disk change Free tsys6 for heavy workers
cnode4 (601) tsys6 tsys9 D2 (tsys4) D5 (tsys4) -- spread disk Free tsys6; spread off D2
cnode5 (602) tsys6 tsys3 D5 (tsys4) S3 (tsys5) -- MOVE disk Free tsys6; split storage to tsys5

Result after changes:

Host cnodes Storage server
tsys1 cnode1 (D5), cnode3 (S2) tsys4 + tsys5
tsys9 cnode2 (D2), cnode4 (D5) tsys4
tsys3 cnode5 (S3) tsys5
  • 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and etcd keeps quorum (3 of 5 survive).
  • D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk holds more than 2 cnodes.
  • Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves at least 3 cnodes alive.
  • tsys6 and tsys7 are fully freed for heavy worker nodes.

Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2 cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.

10.3 HIGH: Wnode distribution -- one per host, tiered storage

Target: one wnode per hypervisor host, using local storage where possible and NFS-HDD for bulk data.

wnode Host Boot disk (OS+containers) Data disk (bulk/ETL) Rationale
wnode-tsys1 tsys1 D5 (tsys4 NFS) -- Small wnode on infra host. Minimal capacity.
wnode-tsys3 tsys3 local-lvm (349 GB NVMe) S3 (NFS) Move from NFS to NVMe (Samsung PM961). Fastest boot disk after tsys5/2.
wnode-tsys6 tsys6 D2 (tsys4 NFS) -- Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS.
wnode-tsys7 tsys7 D5 (tsys4 NFS) -- Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS.
wnode-tsys9 tsys9 local-lvm (136 GB SSD) S2 (NFS) Move from NFS to local SSD. Fast boot, NFS for bulk.
wnode-tsys2 tsys2 NVMe (Samsung 960 PRO 512GB) SATA SSD (Samsung 850 EVO 1TB) Best storage of any wnode. No NFS needed -- 1.5 TB local SSD.
wnode-tsys5 tsys5 new NVMe (local) local-nonprod (HDD) Fastest wnode in fleet. HPC/ETL workloads land here.

Storage tiering strategy per wnode:

Each wnode gets two disk tiers mapped to k8s StorageClasses:

  1. Boot + container runtime (local-lvm or NVMe): OS, kubelet, container images, ephemeral storage. This is where local SSD/NVMe shines -- container image pulls and layer extraction are seek-bound and benefit enormously from low-latency storage.

  2. Bulk data (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging areas, large files that do not fit on local storage. Mounted as a second disk in the VM and exposed to k8s as a StorageClass.

This maps to two k8s StorageClasses:

  • local-storage: bound to the wnode's boot/local disk (fast, ephemeral)
  • nfs-bulk: bound to NFS exports (slow, persistent, large capacity)

10.4 HIGH: Use D3 SSD and T5-SSD for latency-sensitive VMs

Currently wasted:

  • D3 (tsys4 SK hynix SSD, 445 GB free, 0% used) -- nobody using it
  • T5-SSD (tsys5 Samsung 860 PRO, 140 GB free) -- only ultix-streaming

Recommended uses for D3 (after moving ucs-01 there, 32 GB):

  • Remaining D3 space (413 GB): sectestbed-k8s-cnode, sectestbed-k8s-wnode (move from local-nonprod HDD to NFS-SSD for faster test runs)

Recommended uses for T5-SSD (after moving nothing new, ultix-streaming uses 83 GB):

  • Remaining T5-SSD space (140 GB): tsys-siem (VM 901, 132 GB disk currently on D2 HDD) would benefit from SSD for SIEM database writes.

10.5 MEDIUM: Restart wnode-tsys6

VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm (1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves the boot disk to local storage simultaneously.

10.6 MEDIUM: tsys5 NVMe placement (Friday)

Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or directory storage.

Use for:

  • wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
  • sectestbed VMs that need fast scratch space
  • Not NFS-exported (avoid adding network overhead to the fastest tier)

10.7 LOW: Add PCIe NIC to tsys9

tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe slots. A $150 PCIe NIC eliminates the USB reliability risk.

10.8 LOW: Standardize PVE/kernel versions

tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary (9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.

11. Hardware End-of-Support Exposure

Host EOS date Years past Form factor
pfv-tsys6 (R610) 2013-05 13.2 1U server
pfv-tsys5 (T7500) 2014-12 11.7 Workstation
pfv-tsys4 (T1700) 2018-03 8.4 Workstation
pfv-tsys7 (R620) 2019-03 7.4 1U server
pfv-tsys1 (9020) 2019-07 7.0 SFF desktop
pfv-tsys3 (7510) 2020-07 6.0 Laptop
pfv-tsys2 (5520) TBD -- Laptop
pfv-tsys9 (7080) 2024-02 2.4 SFF desktop (only supported)

6 of 8 hosts are past vendor end-of-support. Only tsys9 is still covered. The two actual rack servers (R610, R620) are the most overdue for replacement. This belongs in the next budget cycle.

12. Open Items

12.1 Friday maintenance window (user action)

  1. tsys5: Plug 2nd ethernet cable into storage NIC, verify bond0 recovers, apply layer3+4 hash, reboot to activate NFS nconnect.
  2. tsys4: Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB), reconfigure /etc/network/interfaces, reboot.
  3. tsys5: Install PCI NVMe drive, format and mount (recommend local directory storage for VM images).

12.2 Post-hardware validation

  1. Re-run iperf-full-matrix.sh to confirm tsys4/tsys5 network recovery.
  2. Run validate-fixes.sh on tsys4 and tsys5.

12.3 Data gaps

Gap How to close
tsys5 SDR/parallel-port workload dependency Confirm what uses the SDR before planning tsys5 role changes
tsys2 post-rebuild baseline Run check.sh once it is converted to Proxmox
tsys5 NVMe size and model Confirm after Friday installation
Whether hosts form a Proxmox cluster or are standalone Check pvecm status on each host
tsys3 thermal state (laptop in rack) Check sensors on next maintenance

12.4 Scripts and tools available

Script Purpose
scripts/check.sh Read-only data collector (run on hosts)
scripts/apply-tunings.sh Apply all Tier 0 tunings (dry-run/apply/rollback)
scripts/fix-bond-nfs.sh Fix NFS options + bond hash (live)
validate-fixes.sh Read-only validation of all applied changes
iperf-full-matrix.sh Full iperf test suite (mgmt + storage)
deploy-check.sh Deploy check.sh to all hosts via SSH
Reference in New Issue View Git Blame Copy Permalink