KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: initialize repo with full project state

Browse Source 
Performance optimization engagement for a 7-host Proxmox R&D cluster.
Captures the accumulated work across host tuning, network analysis,
fleet assessment, and kubernetes architecture planning.

Contents:
- Host-side tunings (scripts/): CPU governor, swappiness, BBR, NFS
  nconnect, tuned profiles -- complete on 5 of 7 hosts
- Validation + benchmarking scripts: iperf matrix, bond/NFS fixes
- Collected host data (returned-logs/): check.sh output from all 7
  hosts + iperf results, including newly-validated pfv-tsys9
- AGENTS.md: operating context for AI agents
- PROJECT.md: board-ready fleet assessment with VM placement and
  storage redundancy analysis (40 VMs across 7 hosts)
- K8S.md: kubernetes architecture deep-dive covering cnode/wnode
  distribution, StorageClass design, and ETL/HPC workload planning

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-27 11:31:29 -05:00
commit 422999bf3c
106 changed files with 114159 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT.md
+724
View File
@@ -0,0 +1,724 @@
# Proxmox Cluster Project Report
**Date:** 2026-07-27
**Prepared by:** Performance Optimization Engagement
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
---
## Table of Contents
1. [Executive Summary](#1-executive-summary)
2. [Host Fleet](#2-host-fleet)
3. [Storage Architecture](#3-storage-architecture)
4. [VM Fleet Inventory](#4-vm-fleet-inventory)
5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
8. [Role Alignment Audit](#8-role-alignment-audit)
9. [Network Findings](#9-network-findings)
10. [Recommendations](#10-recommendations)
11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
12. [Open Items](#12-open-items)
---
## 1. Executive Summary
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side
performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned
profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked
on physical hardware work scheduled for Friday.
The VM-layer assessment reveals three structural issues:
| # | Finding | Severity | Fix Cost |
|---|---------|----------|----------|
| 1 | **Every pfv-k8s node stores its disk on tsys4 NFS** (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. | **CRITICAL** | $0 (rebalance) |
| 2 | **No k8s node uses SSD or NVMe.** All boot from NFS on spinning disks, adding latency to every etcd write and container pull. | **HIGH** | $0 (move to local SSD where available) |
| 3 | **Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only.** tsys4 failure takes down both halves of each pair. | **HIGH** | $0 (move -02 halves to tsys5 NFS) |
Additionally, tsys9 was validated this session and revealed two new findings:
its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has
a 250 GB local SSD that no VM is using.
---
## 2. Host Fleet
### 2.1 Inventory
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|------|-------|-----------|-------|-----|-----------|-----------------|--------|
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | TBD | **Kubernetes** | Incoming (Win10) |
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
### 2.2 Role taxonomy (per user directive)
| Role | Hosts | Workload |
|------|-------|----------|
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
run one per hypervisor host across the fleet to maximize total cluster capacity.
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
running Cloudron -- not in this cluster.
---
## 3. Storage Architecture
### 3.1 NFS exports from tsys4 (primary storage server)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------|
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
| D3 | SK hynix SC300 | **SSD** | **USB** | 469 GB | 2 MB | **445 GB** | **0%** |
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
Non-exported disks on tsys4:
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
- sdd (WDC 1T) -- not mounted, idle
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
**Key finding: D3 (SSD, 445 GB) is 99% empty.** Nobody is using the only SSD
export on the primary storage server. This is the fastest available NFS tier
from tsys4 and should be used for latency-sensitive VMs (UCS/LDAP, etcd).
**Warning: D3 is USB-attached.** The SK hynix SSD is connected via a USB adapter
(`usb-SSD_ThinkPlus`). It achieves SSD speeds but carries the same USB
reliability risk as the cdc_ncm NIC dongles. After tsys4 gets its PCIe NIC
(Friday), consider adding a proper SATA/SAS SSD internally to replace the
USB-SSD for D3.
### 3.2 NFS exports from tsys5 (secondary storage server)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------|
| S1 | Seagate ST1000VN | HDD | SATA | 916 GB | 53 GB | 817 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SATA | 916 GB | 2 MB | **870 GB** | **0%** |
| S3 | Seagate ST1000VN | HDD | SATA | 916 GB | 3 MB | **870 GB** | **0%** |
| S4 | Toshiba DT01ACA050 | HDD | SATA | 458 GB | 2 MB | **435 GB** | **0%** |
| T5-SSD | Samsung 860 PRO | **SSD** | SATA | 234 GB | 83 GB | **140 GB** | 35% |
**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
has 2.2 TB of unused HDD capacity plus 140 GB of unused SSD. This is where
VMs should be relocated to reduce tsys4's blast radius.
T5-SSD (Samsung 860 PRO) is 35% used by ultix-streaming (VM 5111, 288 GB disk).
It has 140 GB free for additional SSD-tier VMs.
**New Friday:** PCI NVMe drive being added to tsys5. This will be the fastest
storage tier in the fleet. See section 7.4 for placement recommendation.
### 3.3 Local storage tiers (per host, with utilization)
| Host | Storage ID | Disk type | Total | Used | Free | Used by VMs? |
|------|-----------|-----------|-------|------|------|-------------|
| tsys1 | local-lvm | HDD | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
| tsys3 | local-lvm | HDD | 349 GB | 0 GB | **349 GB** | No (all VMs on NFS) |
| tsys4 | local-lvm | HDD | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | 1.7 TB | 40 MB | **1.7 TB** | No |
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
| tsys6 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
| tsys7 | local-lvm | HDD | 1.7 TB | 0 GB | **1.7 TB** | No (all VMs on NFS) |
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
**Critical observation: every k8s host has 0% used local-lvm.** tsys3 (349 GB),
tsys6 (1.7 TB), tsys7 (1.7 TB) all have massive unused local storage. All their
VMs boot from NFS instead. Moving wnode boot disks to local-lvm would eliminate
NFS dependency and network latency for every k8s worker -- at zero cost.
### 3.4 Disk speed tiers summary
| Tier | Where | Speed class | Best for |
|------|-------|------------|----------|
| **NVMe** | tsys5 (Friday addition) | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime |
| **Local SSD** | tsys9 (PNY CS900, 136 GB) | 500 MB/s | wnode boot disk, etcd |
| **NFS-SSD** | tsys4 D3 (SK hynix, USB), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | UCS/LDAP, latency-sensitive VMs |
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs |
| **Local HDD** | tsys3/6/7 local-lvm | 100-150 MB/s | wnode boot disk (better than NFS-HDD due to no network hop) |
### 3.5 All exports are single-disk with no redundancy
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
on that export. This applies to **both storage servers** and to the **PBS
backup target** (a single 4.5T SMR drive).
---
## 4. VM Fleet Inventory
### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
#### tsys1 (Infrastructure) -- 11 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys3 (Kubernetes) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
#### tsys4 (Storage) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
#### tsys5 (Storage) -- 15 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys7 (Kubernetes) -- 4 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys9 (Infrastructure) -- 5 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
### 4.2 Storage concentration summary
| Storage target | # of running VMs | % of fleet |
|---------------|-----------------|------------|
| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
| local-nonprod (tsys5 local HDD) | 10 | 25% |
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
| local-lvm (tsys4 local) | 1 | 3% |
**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
If tsys4 goes offline, two-thirds of the fleet loses its storage.
---
## 5. Kubernetes Node Distribution
### 5.1 pfv-k8s cnode (control plane) placement
| VMID | Name | Hypervisor | Storage | Disk Type | Host Role Compliant? |
|------|------|------------|---------|-----------|---------------------|
| 107 | cnode1 | tsys1 | D5 (tsys4) | HDD | **NO** (infra host) |
| 106 | cnode3 | tsys1 | D2 (tsys4) | HDD | **NO** (infra host) |
| 603 | cnode2 | tsys6 | D2 (tsys4) | HDD | YES |
| 601 | cnode4 | tsys6 | D2 (tsys4) | HDD | YES |
| 602 | cnode5 | tsys6 | D5 (tsys4) | HDD | YES |
**Findings:**
- 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
- **All 5 cnodes store disks on tsys4 NFS.** Zero use tsys5 storage.
- 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
- All cnodes on spinning HDDs. No SSD/NVMe use.
- **etcd quorum risk:** if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their
disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is
LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes
inoperable.
### 5.2 pfv-k8s wnode (worker) placement
| VMID | Name | Hypervisor | Storage | Disk Type | Status | Host Role Compliant? |
|------|------|------------|---------|-----------|--------|---------------------|
| 313 | wnode-tsys3 | tsys3 | D5 (tsys4) | HDD | running | YES |
| 509 | wnode-tsys5 | tsys5 | D2 (tsys4) | HDD | running | **NO** (storage host) |
| 100 | wnode-tsys6 | tsys6 | D5 (tsys4) | HDD | **STOPPED** | YES (but offline) |
| 701 | wnode-tsys7 | tsys7 | D5 (tsys4) | HDD | running | YES |
| 905 | wnode-tsys9 | tsys9 | S3 (tsys5) | HDD | running | **NO** (infra host) |
**Findings:**
- 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
- wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) --
it depends on tsys4 for its disk despite running on tsys5.
- wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
- wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it
is the ONLY k8s node using tsys5 storage (which is good for redundancy).
- All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.
### 5.3 Summary: k8s node storage distribution
| Storage server | cnodes | wnodes | Total k8s nodes |
|---------------|--------|--------|-----------------|
| **tsys4 NFS** | **5 (100%)** | **4 (80%)** | **9 (90%)** |
| tsys5 NFS | 0 | 1 (20%) | 1 (10%) |
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
**90% of the kubernetes cluster's storage depends on a single host (tsys4).**
This is the single most critical risk in the VM layer.
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
| VMID | Name | Storage |
|------|------|---------|
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
These are on local storage (good -- no NFS dependency) but on a single host's
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
---
## 6. Storage Redundancy Analysis
### 6.1 -01/-02 infrastructure pair audit
Two -01/-02 pairs exist in the fleet:
**Pair 1: pfv-netinfra (network infrastructure)**
| Role | VMID | Host | Storage | NFS Server |
|------|------|------|---------|-----------|
| -01 | 103 | tsys1 | D5 | **tsys4** |
| -02 | 904 | tsys9 | D2 | **tsys4** |
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
disks. The -02 half should be on an S2/S3/S4 export from tsys5.
**Pair 2: tsys-ucs (Univention Corporate Server)**
| Role | VMID | Host | Storage | NFS Server |
|------|------|------|---------|-----------|
| -01 | 108 | tsys1 | D2 | **tsys4** |
| -02 | 902 | tsys9 | D5 | **tsys4** |
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
### 6.2 Redundancy principle for paired VMs
For any -01/-02 pair to survive a single storage server failure:
```
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
```
This ensures that losing either tsys4 or tsys5 takes down only one half of
the pair. Currently, **both pairs fail this test** because both halves are on
tsys4.
### 6.3 NFS server failure blast radius
If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
| Impact | Count |
|--------|-------|
| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
| k8s wnodes that lose storage | 4 of 5 |
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
| Total VMs that lose storage | **27 of 40 (68%)** |
If **tsys5** goes offline:
| Impact | Count |
|--------|-------|
| k8s cnodes that lose storage | 0 of 5 |
| k8s wnodes that lose storage | 1 of 5 |
| Total VMs that lose storage | 1 of 40 (3%) |
**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
Rebalancing VM storage across both servers dramatically reduces risk.
---
## 7. Local SSD/NVMe Opportunity
### 7.1 Available fast tiers (currently unused by k8s)
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|------|--------|------|------|--------------------|-----------------------|
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
### 7.2 Why local storage matters for k8s nodes
Kubernetes nodes are latency-sensitive in two specific areas:
1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
This directly affects k8s API responsiveness and pod scheduling speed.
2. **Container image pulls:** Worker nodes pull container images frequently.
On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
eliminates the network hop and reduces seek time. This matters most for
RackRental/containerlab workloads that spin up containers rapidly.
### 7.3 Current waste: tsys9 local SSD
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
The PCI NVMe being added to tsys5 will be the fastest storage tier in the
fleet. Two placement options:
**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
VMs that might need migration. Adds the NFS/network overhead back.
**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
accessed from other hosts.
**Recommendation:** Option B (local-only). k8s worker nodes do not need
shared storage -- pods are ephemeral and reschedule on failure. The NVMe
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
and used for local VM images.
---
## 8. Role Alignment Audit
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
**correct by design** (keeps heavy hosts free for workers).
### 8.1 VMs that need to move
| VMID | Name | Current host | Issue | Target |
|------|------|-------------|-------|--------|
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
### 8.2 Host capacity for k8s nodes
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|------|------|-------------------|-------------|-----------|----------|
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
free RAM respectively. They should be the primary targets for heavy worker
nodes and ETL/HPC workloads.
### 8.3 tsys6 wnode-tsys6 is stopped
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
cluster capacity.
---
## 9. Network Findings
### 9.1 tsys9 storage NIC is a USB dongle (new finding)
Validating tsys9 revealed that its storage network interface
(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
management; storage uses the USB adapter.
**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
wobble, ESD, and USB controller resets. For an infrastructure host with 5
VMs, this is a reliability risk.
**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
PCIe NIC would eliminate this risk (same recommendation as tsys4).
### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
| Host | Blocker | Staged fix |
|------|---------|-----------|
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
### 9.3 LACP resolved on tsys6/tsys7
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
The switch LACP hash change took effect after renegotiation. The 56-106K
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
LACP overhead. See `RESULTS.md` for the full analysis.
### 9.4 NFS nconnect=4 + noatime confirmed active
All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
options. Each host maintains 4 TCP connections per NFS mount to each storage
server. This was the Tier 0 tuning item from the performance optimization
engagement and is confirmed working cluster-wide.
---
## 10. Recommendations
**No changes have been made. These are assessment-only recommendations.**
See `K8S.md` for the detailed kubernetes architecture deep-dive.
### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
These are the most critical production VMs in the fleet. They must survive
any single-point failure (host, storage server, or disk).
**Design principle for -01/-02 HA pairs:**
- Different hypervisors (already satisfied: tsys1 vs tsys9)
- Different storage servers (currently FAILED: all on tsys4)
- Prefer SSD for latency-sensitive services
**Recommended placement:**
| VM | Host | Storage | Tier | Free space | Rationale |
|----|------|---------|------|-----------|-----------|
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O, HDD is fine. Stays put. |
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
| **ucs-01** (108) | tsys1 | **D3 (tsys4 SSD)** | NFS-SSD | 445 GB | Move from D2. LDAP/Kerberos benefits from SSD. D3 is 99% empty. |
| **ucs-02** (902) | tsys9 | **local-lvm (SSD)** | LOCAL-SSD | 136 GB | Move from D5. Local SSD = lowest latency + no NFS dependency. |
**Failure survival matrix (all single-point failures):**
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|---------|-------------|-------------|--------|--------|--------|
| tsys4 dies | dies (D5) | **alive** (S3) | **alive** (D3 down) | **alive** (local) | 3 of 4 alive |
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D3) | **alive** (local) | 3 of 4 alive |
| tsys1 dies | dies | **alive** | dies | **alive** | netinfra-02 + ucs-02 alive |
| tsys9 dies | **alive** | dies | **alive** | dies | netinfra-01 + ucs-01 alive |
| D2 disk fails | **alive** (D5) | **alive** (S3) | **alive** (D3) | **alive** (local) | **all 4 alive** |
Note: ucs-01 on D3 goes down if tsys4 dies (D3 is on tsys4), but ucs-02 on
tsys9 local SSD survives. That is the HA pair working as designed.
**Why this works:** Every row has at least one netinfra and one UCS alive.
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
### 10.2 CRITICAL: Cnode (control plane) storage split
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
quorum (3 of 5 cnodes share D2).
**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
|-------|-------------|----------------|----------------|-------------------|-----------|
| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
**Result after changes:**
| Host | cnodes | Storage server |
|------|--------|---------------|
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
| tsys3 | cnode5 (S3) | tsys5 |
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
etcd keeps quorum (3 of 5 survive).
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
holds more than 2 cnodes.
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
at least 3 cnodes alive.
- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
**Target: one wnode per hypervisor host, using local storage where possible
and NFS-HDD for bulk data.**
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|-------|------|--------------------------|---------------------|-----------|
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
| wnode-tsys3 | tsys3 | **local-lvm (349 GB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 349 GB local is plenty for boot. |
| wnode-tsys6 | tsys6 | **local-lvm (1.7 TB HDD)** | D2 or S2 (NFS) | Move from NFS to local. 1.7 TB local is massive. Restart (currently stopped). |
| wnode-tsys7 | tsys7 | **local-lvm (1.7 TB HDD)** | D5 or S3 (NFS) | Move from NFS to local. 1.7 TB local is massive. |
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 or D2 (NFS) | Move from NFS to local SSD. **Fastest boot disk for any wnode.** |
| wnode-tsys2 | tsys2 | local (TBD) | NFS | Create when tsys2 is Proxmox. |
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
**Storage tiering strategy per wnode:**
Each wnode gets two disk tiers mapped to k8s StorageClasses:
1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
images, ephemeral storage. This is where local SSD/NVMe shines -- container
image pulls and layer extraction are seek-bound and benefit enormously from
low-latency storage.
2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
areas, large files that do not fit on local storage. Mounted as a second
disk in the VM and exposed to k8s as a StorageClass.
This maps to two k8s StorageClasses:
- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
### 10.4 HIGH: Use D3 SSD and T5-SSD for latency-sensitive VMs
Currently wasted:
- **D3 (tsys4 SK hynix SSD, 445 GB free, 0% used)** -- nobody using it
- **T5-SSD (tsys5 Samsung 860 PRO, 140 GB free)** -- only ultix-streaming
Recommended uses for D3 (after moving ucs-01 there, 32 GB):
- Remaining D3 space (413 GB): sectestbed-k8s-cnode, sectestbed-k8s-wnode
(move from local-nonprod HDD to NFS-SSD for faster test runs)
Recommended uses for T5-SSD (after moving nothing new, ultix-streaming uses 83 GB):
- Remaining T5-SSD space (140 GB): tsys-siem (VM 901, 132 GB disk currently
on D2 HDD) would benefit from SSD for SIEM database writes.
### 10.5 MEDIUM: Restart wnode-tsys6
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
the boot disk to local storage simultaneously.
### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
directory storage.**
Use for:
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
- sectestbed VMs that need fast scratch space
- Not NFS-exported (avoid adding network overhead to the fastest tier)
### 10.7 LOW: Add PCIe NIC to tsys9
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
slots. A $150 PCIe NIC eliminates the USB reliability risk.
### 10.8 LOW: Standardize PVE/kernel versions
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
---
## 11. Hardware End-of-Support Exposure
| Host | EOS date | Years past | Form factor |
|------|----------|-----------|-------------|
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
| pfv-tsys2 (5520) | TBD | -- | Laptop |
| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
The two actual rack servers (R610, R620) are the most overdue for replacement.
This belongs in the next budget cycle.
---
## 12. Open Items
### 12.1 Friday maintenance window (user action)
1. **tsys5:** Plug 2nd ethernet cable into storage NIC, verify bond0 recovers,
apply layer3+4 hash, reboot to activate NFS nconnect.
2. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
reconfigure `/etc/network/interfaces`, reboot.
3. **tsys5:** Install PCI NVMe drive, format and mount (recommend local
directory storage for VM images).
### 12.2 Post-hardware validation
1. Re-run `iperf-full-matrix.sh` to confirm tsys4/tsys5 network recovery.
2. Run `validate-fixes.sh` on tsys4 and tsys5.
### 12.3 Data gaps
| Gap | How to close |
|-----|--------------|
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR before planning tsys5 role changes |
| tsys2 post-rebuild baseline | Run `check.sh` once it is converted to Proxmox |
| tsys5 NVMe size and model | Confirm after Friday installation |
| Whether hosts form a Proxmox cluster or are standalone | Check `pvecm status` on each host |
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
### 12.4 Scripts and tools available
| Script | Purpose |
|--------|---------|
| `scripts/check.sh` | Read-only data collector (run on hosts) |
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash (live) |
| `validate-fixes.sh` | Read-only validation of all applied changes |
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH |