mrcharles
6b6a8ff021
Split docs/ into project-based subdirectories:
- docs/proxmox/ fleet ops, hardware, k8s (PROJECT.md, TODO.md, K8S.md)
- docs/server-build/ provisioning, security, DNS (SECURITY.md, tailscale.md,
DEPLOYMENT.md, TSYS-2FA-GUIDE.md, DEVELOPMENT-GUIDELINES.md)
- docs/archive/ historical AI reviews, completed todos, pre-refactor docs
docmap.md rewritten with new paths. All cross-references in AGENTS.md,
README.md, STATUS.md, and dns-cluster-setup/README.md updated.
Code directories unchanged — scripts stay where BASH_SOURCE expects them.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
40 KiB
Proxmox Cluster Project Report
Date: 2026-07-27 (re-audited) Prepared by: Performance Optimization Engagement Status: Comprehensive fleet assessment with VM placement and redundancy analysis
Data freshness: All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
deploy-check.sh. VM placements reflect live state after user's PDM
migrations. This is ground truth.
Table of Contents
- Executive Summary
- Host Fleet
- Storage Architecture
- VM Fleet Inventory
- Kubernetes Node Distribution
- Storage Redundancy Analysis
- Local SSD/NVMe Opportunity
- Role Alignment Audit
- Network Findings
- Recommendations
- Hardware End-of-Support Exposure
- Open Items
1. Executive Summary
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2), running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked on physical hardware work scheduled for Friday.
Progress since initial audit: The user has been actively rebalancing k8s nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to 73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration needed for etcd quorum survival.
The VM-layer assessment reveals:
| # | Finding | Severity | Status |
|---|---|---|---|
| 1 | 4 of 5 cnodes still store disks on tsys4 NFS. cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | CRITICAL | Improving |
| 2 | Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only. | HIGH | TODO today |
| 3 | No k8s node uses SSD or NVMe yet. tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | HIGH | Deferred to k8s session |
| 4 | D3 SSD (tsys4, USB) is 99% empty (445 GB free). Moving to tsys5 SAS Friday. | MEDIUM | Friday |
2. Host Fleet
2.1 Inventory
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|---|---|---|---|---|---|---|---|
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | Infrastructure | Done |
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | NVMe 512GB + SATA SSD 1TB | Kubernetes | Incoming (Win10) |
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | Kubernetes | Done |
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | Storage (NFS+PBS) | Blocked (NIC+RAM) |
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | Storage (NFS+VMs) | Blocked (cable) |
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | Kubernetes | Done |
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | Kubernetes | Done |
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | 250 GB SSD (PNY CS900) | Infrastructure | Done |
2.2 Role taxonomy (per user directive)
| Role | Hosts | Workload |
|---|---|---|
| Infrastructure + k8s control | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
| Kubernetes workers | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
| Storage | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
Design rationale: cnodes (control plane) are lightweight (4 cores, 4 GB RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts (tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes run one per hypervisor host across the fleet to maximize total cluster capacity.
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA running Cloudron -- not in this cluster.
3. Storage Architecture
3.1 NFS exports from tsys4 (primary storage server)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|---|---|---|---|---|---|---|---|
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | 2.4 TB | 8% |
| — | — | — | moving to tsys5 Friday | ||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | 1.5 TB | 14% |
Non-exported disks on tsys4:
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
- sdd (WDC 1T) -- idle, unmounted, removable (free up for other use)
- sdf (WDC 4.5T SMR) at /mnt/backup -- PBS backup target, 4.3 TB free
D3 migration (Friday): The SK hynix SC300 SSD is currently USB-attached on tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port, eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb /sdc/sdd), so tsys5 is the better target. See section 3.2.
3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|---|---|---|---|---|---|---|---|
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 6.9 GB | 863 GB | 1% |
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | 863 GB | 1% |
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | 435 GB | 0% |
| T5-SSD | Samsung 860 PRO | SSD | SAS | 234 GB | 122 GB | 101 GB | 55% |
| D3 (Friday) | SK hynix SC300 | SSD | SAS | 469 GB | 2 MB | 445 GB | 0% |
tsys5 storage controllers (plenty of free ports):
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate), 5 free
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), 2 free
- Intel ICH10 SATA #2 (2-port): status unknown, likely free
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
Key finding: S2 and S3 now have k8s node disks. S2 holds cnode5 + wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk (unused, 7 GB). S4 still 99% empty (435 GB free).
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
Friday additions:
- D3 (SK hynix SSD) moves from tsys4 USB to tsys5 SAS port. Eliminates USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
- PCI NVMe drive (local-only, not NFS-exported). Used for wnode-tsys5 boot disk and HPC scratch. The fastest tier in the fleet.
After Friday, tsys5 consolidates all fast storage: NVMe (local) + 2 SSD NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the natural home for latency-sensitive workloads and the k8s StorageClass design center.
3.3 Local storage tiers (per host, with utilization)
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|---|---|---|---|---|---|---|---|
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
| tsys3 | local-lvm | NVMe (Samsung PM961) | NVMe | 349 GB | 0 GB | 349 GB | No (all VMs on NFS) |
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | 1.7 TB | No |
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | 856 GB | Yes (sectestbed suite) |
| tsys6 | local-lvm | HDD (WD My Passport) | USB 2.0 | 1.7 TB | 0 GB | 1.7 TB | No -- DO NOT USE for VM storage |
| tsys7 | local-lvm | HDD (WD portable) | USB 2.0 | 1.7 TB | 0 GB | 1.7 TB | No -- DO NOT USE for VM storage |
| tsys9 | local-lvm | SSD (PNY CS900) | SATA | 136 GB | 0 GB | 136 GB | No (all VMs on NFS) |
| tsys2 | NVMe (Samsung 960 PRO) | NVMe | NVMe | 512 GB | (Win10) | 512 GB | Incoming -- fastest boot tier after tsys5 NVMe |
| tsys2 | SATA SSD (Samsung 850 EVO) | SSD | SATA | 1 TB | (Win10) | 1 TB | Incoming -- bulk data on SSD, not rust |
CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a single USB 2.0-attached WD My Passport portable HDD (tsys6: "My Passport 260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI (USB 2.0, ~480 Mbps). There is no USB 3.0/xHCI on either host.
USB 2.0 practical throughput is ~30-35 MB/s. This is 3-4x SLOWER than NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these hosts would decrease performance. local-lvm on tsys6/7 must NOT be used for VM storage.
Additionally, both servers have completely empty internal drive bays:
- tsys6: SAS controller present but DISABLED in BIOS. No internal drives.
- tsys7: 6-port SATA AHCI controller present, 5 ports EMPTY (only DVD-ROM on port 5). No internal drives.
This is a reliability risk beyond performance: the entire host OS boots from a consumer-grade portable USB drive not designed for 24/7 server use.
tsys3 correction: Previously documented as HDD. Actually boots from a Samsung PM961 NVMe 512GB SSD -- the fastest existing local storage in the fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
Critical observation: every k8s host has 0% used local-lvm. tsys3 (Samsung PM961 NVMe, 349 GB), tsys6 (WD My Passport USB 2.0 HDD, 1.7 TB), tsys7 (WD portable USB 2.0 HDD, 1.7 TB) all have unused local storage.
However, only tsys3's local-lvm is suitable for VM storage. tsys6 and tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than NFS-over-HDD and unsuitable for wnode boot disks.
3.4 Disk speed tiers summary
| Tier | Where | Speed class | Best for |
|---|---|---|---|
| NVMe | tsys3 (Samsung PM961), tsys5 (Friday addition), tsys2 (Samsung 960 PRO 512GB) | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
| Local SSD | tsys9 (PNY CS900, 136 GB), tsys2 (Samsung 850 EVO 1TB) | 500 MB/s | wnode boot disk, etcd |
| NFS-SSD | tsys5 D3 (SK hynix, SAS post-Friday), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | k8s worker scratch only (etcd, container cache, ephemeral volumes) |
| NFS-HDD | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, wnode boot on tsys6/7 |
| Local SATA HDD | tsys1 local-lvm | 100-150 MB/s | Host OS only |
| USB 2.0 HDD | tsys6/7 local-lvm (WD My Passport) | ~30-35 MB/s | NOTHING -- slower than NFS, do not use for VMs |
3.5 Storage tier characterization per host
| Host | Storage profile | Detail |
|---|---|---|
| tsys2 | SSD/NVMe only | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
| tsys3 | NVMe only | Samsung PM961 NVMe 512GB. No spinning disk. |
| tsys5 | Hybrid (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
| tsys4 | Bulk/spinning disk only | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
| tsys9 | Local SSD + NFS | PNY CS900 SSD 136GB local + NFS client |
| tsys1 | Local HDD + NFS | Small local-lvm + NFS client |
| tsys6/7 | NFS only | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
3.6 All exports are single-disk with no redundancy
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS mirror, no mdraid. A single disk failure takes down every VM whose disk lives on that export. This applies to both storage servers and to the PBS backup target (a single 4.5T SMR drive).
4. VM Fleet Inventory
4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
tsys1 (Infrastructure) -- 11 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 107 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 108 | tsys-ucs-01 | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
tsys3 (Kubernetes) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 313 | pfv-k8s-wnode-tsys3 | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
tsys4 (Storage) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
tsys5 (Storage) -- 15 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 509 | pfv-k8s-wnode-tsys5 | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- STOPPED |
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- STOPPED |
| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 602 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 603 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
tsys7 (Kubernetes) -- 4 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
tsys9 (Infrastructure) -- 5 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|---|---|---|---|---|---|---|
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
| 902 | tsys-ucs-02 | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
4.2 Storage concentration summary
| Storage target | # of running VMs | % of fleet |
|---|---|---|
| D2 (tsys4 WDC Red 3TB HDD) | 18 | 45% |
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
| local-nonprod (tsys5 local HDD) | 10 | 25% |
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
| local-lvm (tsys4 local) | 1 | 3% |
27 of 40 running VMs (68%) store their disks on tsys4 NFS exports. If tsys4 goes offline, two-thirds of the fleet loses its storage.
5. Kubernetes Node Distribution (re-audited 21:50 CDT)
5.1 pfv-k8s cnode (control plane) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|---|---|---|---|---|---|
| 906 | cnode1 | tsys9 | D5 | tsys4 | MOVED from tsys1 |
| 705 | cnode2 | tsys7 | D2 | tsys4 | MOVED from tsys6 |
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
| 706 | cnode5 | tsys7 | S2 | tsys5 | MOVED from tsys6, storage moved D5→S2 |
Storage distribution:
| Storage server | cnodes | Quorum impact if it fails |
|---|---|---|
| tsys4 (D2+D5) | 4 (cnode1,2,3,4) | Only cnode5 survives = QUORUM LOST |
| tsys5 (S2) | 1 (cnode5) | 4 survive = quorum OK |
Progress: cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4 still means a tsys4 failure loses quorum. Need 2 more cnodes on tsys5.
Host distribution: cnodes spread across 4 hosts (tsys1, tsys6, tsys7, tsys9) -- good host diversity.
5.2 pfv-k8s wnode (worker) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|---|---|---|---|---|---|---|---|
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | STOPPED | NEW |
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | 28 GB | running | RAM bumped 20→28 |
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | NOW RUNNING |
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
| 905 | wnode-tsys9 | tsys9 | S2 | tsys5 | 4 GB | running | Storage moved S3→S2 |
One wnode per host achieved (tsys1,3,5,6,7,9). wnode-tsys1 is created but stopped. wnode-tsys6 restarted.
Storage distribution:
| Storage server | wnodes | Notes |
|---|---|---|
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
5.3 Summary: k8s node storage distribution
| Storage server | cnodes | wnodes | Total k8s nodes |
|---|---|---|---|
| tsys4 NFS | 4 (80%) | 4 (67%) | 8 (73%) |
| tsys5 NFS | 1 (20%) | 2 (33%) | 3 (27%) |
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
Was 90%/10%. Now 73%/27%. Improving but still tsys4-heavy. Target: 3 cnodes on each storage server (60/40 or better) so either server failing leaves quorum intact.
5.4 Remaining cnode migration needed for etcd quorum survival
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must be on tsys5:
| Action | Effect |
|---|---|
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
Simplest path: migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
5.5 Future k8s architecture (next session -- see K8S.md)
The k8s layer will be tackled soon. Key requirements from user:
- Platform: vcluster + Rancher for multi-tenant management
- Auth: OIDC to Keycloak (running on Cloudron in Reston, VA production)
- Workload isolation (vcluster per tenant):
- RackRental workloads (containerlab network labs)
- Suborbital ITAR (compliance-restricted)
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- Solar-aware scale-out: PowerEdge 19xx and 2950 systems (older hardware) will be brought online during peak solar production for burst capacity. These older cores/ram supplement the main fleet when power is abundant.
- WNode sizing: every Proxmox node will have a wnode. Some nodes will host both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into leftover host capacity) to large (28-32 GB, consuming most of a host).
- Friday final audit: tsys2 will be loaded with Proxmox on Friday, and a full final audit will be performed at that time (post-NVMe install on tsys5, post-D3 SSD relocation, post-tsys4 NIC+RAM).
5.4 sectestbed k8s nodes (separate from pfv-k8s)
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
| VMID | Name | Storage |
|---|---|---|
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
These are on local storage (good -- no NFS dependency) but on a single host's single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
6. Storage Redundancy Analysis
6.1 -01/-02 infrastructure pair audit
Two -01/-02 pairs exist in the fleet:
Pair 1: pfv-netinfra (network infrastructure)
| Role | VMID | Host | Storage | NFS Server |
|---|---|---|---|---|
| -01 | 103 | tsys1 | D5 | tsys4 |
| -02 | 904 | tsys9 | D2 | tsys4 |
Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED. Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their disks. The -02 half should be on an S2/S3/S4 export from tsys5.
Pair 2: tsys-ucs (Univention Corporate Server)
| Role | VMID | Host | Storage | NFS Server |
|---|---|---|---|---|
| -01 | 108 | tsys1 | D2 | tsys4 |
| -02 | 902 | tsys9 | D5 | tsys4 |
Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED. Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
6.2 Redundancy principle for paired VMs
For any -01/-02 pair to survive a single storage server failure:
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
This ensures that losing either tsys4 or tsys5 takes down only one half of the pair. Currently, both pairs fail this test because both halves are on tsys4.
6.3 NFS server failure blast radius
If tsys4 goes offline (USB NIC failure, disk failure, reboot):
| Impact | Count |
|---|---|
| k8s cnodes that lose storage | 5 of 5 (etcd quorum lost) |
| k8s wnodes that lose storage | 4 of 5 |
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
| Total VMs that lose storage | 27 of 40 (68%) |
If tsys5 goes offline:
| Impact | Count |
|---|---|
| k8s cnodes that lose storage | 0 of 5 |
| k8s wnodes that lose storage | 1 of 5 |
| Total VMs that lose storage | 1 of 40 (3%) |
tsys4 is a massive blast-radius liability. tsys5 is barely used. Rebalancing VM storage across both servers dramatically reduces risk.
7. Local SSD/NVMe Opportunity
7.1 Available fast tiers (currently unused by k8s)
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|---|---|---|---|---|---|
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | No |
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | No (used by ultix-streaming) |
| tsys5 | New NVMe (Friday) | NVMe | TBD | Yes (local or NFS) | No |
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | No |
7.2 Why local storage matters for k8s nodes
Kubernetes nodes are latency-sensitive in two specific areas:
-
etcd (control plane): etcd writes are synchronous and latency-critical. On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms. This directly affects k8s API responsiveness and pod scheduling speed.
-
Container image pulls: Worker nodes pull container images frequently. On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD eliminates the network hop and reduces seek time. This matters most for RackRental/containerlab workloads that spin up containers rapidly.
7.3 Current waste: tsys9 local SSD
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available. Zero VMs use it. All 5 VMs on tsys9 boot from NFS. The local SSD sits idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD -- its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
7.4 Upcoming opportunity: tsys5 NVMe (Friday)
The PCI NVMe being added to tsys5 will be the fastest storage tier in the fleet. Two placement options:
Option A: NFS-export the NVMe (shared). All hosts can use it. Good for VMs that might need migration. Adds the NFS/network overhead back.
Option B: Local-only on tsys5. VMs on tsys5 get full NVMe speed with no network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be accessed from other hosts.
Recommendation: Option B (local-only). k8s worker nodes do not need shared storage -- pods are ephemeral and reschedule on failure. The NVMe should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5 and used for local VM images.
8. Role Alignment Audit
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is correct by design (keeps heavy hosts free for workers).
8.1 VMs that need to move
| VMID | Name | Current host | Issue | Target |
|---|---|---|---|---|
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
8.2 Host capacity for k8s nodes
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|---|---|---|---|---|---|
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | ~116 GB |
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | ~160 GB |
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
tsys6 and tsys7 are dramatically underutilized -- 116 GB and 160 GB of free RAM respectively. They should be the primary targets for heavy worker nodes and ETL/HPC workloads.
8.3 tsys6 wnode-tsys6 is stopped
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4 NFS). This wnode should be restarted (or recreated on local-lvm) to restore cluster capacity.
9. Network Findings
9.1 tsys9 storage NIC is a USB dongle (new finding)
Validating tsys9 revealed that its storage network interface
(enx9c69d36a5b6c) is USB-attached (parentbus usb). This is the same
anti-pattern as tsys4. The onboard Intel NIC (enp0s31f6) is used for
management; storage uses the USB adapter.
Impact: Same as tsys4 -- achieves line rate but is susceptible to cable wobble, ESD, and USB controller resets. For an infrastructure host with 5 VMs, this is a reliability risk.
Mitigation: tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A PCIe NIC would eliminate this risk (same recommendation as tsys4).
9.2 tsys4 and tsys5 still blocked (Friday hardware work)
| Host | Blocker | Staged fix |
|---|---|---|
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
9.3 LACP resolved on tsys6/tsys7
tsys6 to tsys7 storage path now measures 1.83 Gbps (was 943 Mbps).
The switch LACP hash change took effect after renegotiation. The 56-106K
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
LACP overhead. See RESULTS.md (not yet created) for the full analysis.
9.4 NFS nconnect=4 + noatime confirmed active
All hosts (including tsys9) show nconnect=4,noatime in their NFS mount
options. Each host maintains 4 TCP connections per NFS mount to each storage
server. This was the Tier 0 tuning item from the performance optimization
engagement and is confirmed working cluster-wide.
10. Recommendations
No changes have been made. These are assessment-only recommendations.
See K8S.md for the detailed kubernetes architecture deep-dive.
10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
These are the most critical production VMs in the fleet. They must survive any single-point failure (host, storage server, or disk).
Design principle for -01/-02 HA pairs:
- Different hypervisors (already satisfied: tsys1 vs tsys9)
- Different storage servers (currently FAILED: all on tsys4)
- Prefer SSD for latency-sensitive services
Recommended placement:
| VM | Host | Storage | Tier | Free space | Rationale |
|---|---|---|---|---|---|
| netinfra-01 (103) | tsys1 | D5 (tsys4 HDD) | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
| netinfra-02 (904) | tsys9 | S3 (tsys5 HDD) | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
| ucs-01 (108) | tsys1 | D2 (tsys4 HDD) | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
| ucs-02 (902) | tsys9 | S2 (tsys5 HDD) | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
Failure survival matrix (all single-point failures):
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|---|---|---|---|---|---|
| tsys4 dies | dies (D5) | alive (S3) | dies (D2) | alive (S2) | netinfra-02 + ucs-02 alive |
| tsys5 dies | alive (D5) | dies (S3) | alive (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
| tsys1 dies | dies | alive | dies | alive | -02 pair survives |
| tsys9 dies | alive | dies | alive | dies | -01 pair survives |
| Any single disk | all 4 on different disks/servers | all 4 alive |
Why this works: Every row has at least one netinfra and one UCS alive. The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
D3 SSD repurposed: With UCS staying on HDD, the D3 SSD (moving to tsys5 Friday) is freed for latency-sensitive workloads that actually benefit from SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image registry mirror. Not infrastructure VMs.
10.2 CRITICAL: Cnode (control plane) storage split
Problem: All 5 cnodes store disks on tsys4. D2 disk failure loses etcd quorum (3 of 5 cnodes share D2).
Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.
| cnode | Current host | Target host | Current storage | Target storage | Rationale |
|---|---|---|---|---|---|
| cnode1 (107) | tsys1 | tsys1 (stays) | D5 (tsys4) | D5 (tsys4) -- no change | Already correct |
| cnode3 (106) | tsys1 | tsys1 (stays) | D2 (tsys4) | S2 (tsys5) -- MOVE disk | Split storage to tsys5 |
| cnode2 (603) | tsys6 | tsys9 | D2 (tsys4) | D2 (tsys4) -- no disk change | Free tsys6 for heavy workers |
| cnode4 (601) | tsys6 | tsys9 | D2 (tsys4) | D5 (tsys4) -- spread disk | Free tsys6; spread off D2 |
| cnode5 (602) | tsys6 | tsys3 | D5 (tsys4) | S3 (tsys5) -- MOVE disk | Free tsys6; split storage to tsys5 |
Result after changes:
| Host | cnodes | Storage server |
|---|---|---|
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
| tsys3 | cnode5 (S3) | tsys5 |
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and etcd keeps quorum (3 of 5 survive).
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk holds more than 2 cnodes.
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves at least 3 cnodes alive.
- tsys6 and tsys7 are fully freed for heavy worker nodes.
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2 cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
10.3 HIGH: Wnode distribution -- one per host, tiered storage
Target: one wnode per hypervisor host, using local storage where possible and NFS-HDD for bulk data.
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|---|---|---|---|---|
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
| wnode-tsys3 | tsys3 | local-lvm (349 GB NVMe) | S3 (NFS) | Move from NFS to NVMe (Samsung PM961). Fastest boot disk after tsys5/2. |
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | Stays on NFS. local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys9 | tsys9 | local-lvm (136 GB SSD) | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
| wnode-tsys2 | tsys2 | NVMe (Samsung 960 PRO 512GB) | SATA SSD (Samsung 850 EVO 1TB) | Best storage of any wnode. No NFS needed -- 1.5 TB local SSD. |
| wnode-tsys5 | tsys5 | new NVMe (local) | local-nonprod (HDD) | Fastest wnode in fleet. HPC/ETL workloads land here. |
Storage tiering strategy per wnode:
Each wnode gets two disk tiers mapped to k8s StorageClasses:
-
Boot + container runtime (local-lvm or NVMe): OS, kubelet, container images, ephemeral storage. This is where local SSD/NVMe shines -- container image pulls and layer extraction are seek-bound and benefit enormously from low-latency storage.
-
Bulk data (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging areas, large files that do not fit on local storage. Mounted as a second disk in the VM and exposed to k8s as a StorageClass.
This maps to two k8s StorageClasses:
local-storage: bound to the wnode's boot/local disk (fast, ephemeral)nfs-bulk: bound to NFS exports (slow, persistent, large capacity)
10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch space and ultix-streaming (developer workstation running "cluster of 1" pre-production jobs). Spinning rust hosts all other infrastructure VMs (UCS, netinfra, LibreNMS, SIEM, etc.).
SSD allocation after Friday:
- D3 (tsys5 SAS, 445 GB free, 0% used) -- dedicated to k8s scratch via
the
nfs-ssdStorageClass (etcd, container cache, ephemeral volumes). - T5-SSD (tsys5 SAS, 140 GB free) -- ultix-streaming (VM 5111) stays here (developer workstation, runs single-node test jobs before k8s). Remaining 140 GB available for k8s.
Deep-dive on exact k8s scratch allocation is deferred to the next session (K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
10.5 MEDIUM: Restart wnode-tsys6
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm (1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves the boot disk to local storage simultaneously.
10.6 MEDIUM: tsys5 NVMe placement (Friday)
Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or directory storage.
Use for:
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
- sectestbed VMs that need fast scratch space
- Not NFS-exported (avoid adding network overhead to the fastest tier)
10.7 LOW: Add PCIe NIC to tsys9
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe slots. A $150 PCIe NIC eliminates the USB reliability risk.
10.8 LOW: Standardize PVE/kernel versions
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary (9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
11. Hardware End-of-Support Exposure
| Host | EOS date | Years past | Form factor |
|---|---|---|---|
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
| pfv-tsys2 (5520) | TBD | -- | Laptop |
| pfv-tsys9 (7080) | 2024-02 | 2.4 | SFF desktop (only supported) |
6 of 8 hosts are past vendor end-of-support. Only tsys9 is still covered. The two actual rack servers (R610, R620) are the most overdue for replacement. This belongs in the next budget cycle.
12. Open Items
12.1 Immediate (do today via PDM)
- Migrate ucs-02 (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server redundancy. UCS stays on HDD.
- Migrate netinfra-02 (VM 904) from D2 (tsys4) to S3 (tsys5).
- Migrate cnode3 (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
- Migrate cnode4 (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
- Start wnode-tsys1 (VM 102) if the cluster needs the capacity.
12.2 Friday maintenance window (user action)
- tsys4: Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
reconfigure
/etc/network/interfaces, reboot. - tsys5: Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash, install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port, format NVMe as local storage, reboot.
- tsys2: Load Proxmox (replacing Windows 10).
- Final audit: Re-run
deploy-check.shacross all hosts including tsys2.
12.3 Post-Friday validation
- Re-run iperf matrix:
./iperf-full-matrix.sh - Validate tsys4 and tsys5:
./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5 - Run
scripts/check.shon tsys2 once Proxmox is loaded. - Update PROJECT.md with post-hardware numbers.
12.4 Future: Kubernetes deep-dive (see K8S.md)
Next major workstream. Requirements captured:
- vcluster + Rancher for multi-tenant k8s management
- OIDC auth to Keycloak (on Cloudron, Reston VA production)
- Workload isolation via separate vclusters:
- RackRental (containerlab)
- Suborbital ITAR
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- Solar-aware scale-out: PowerEdge 19xx + 2950 systems brought online during peak solar production for burst capacity
- Every host gets a wnode (variable sizing: small 4 GB to large 32 GB)
- SSD/NVMe reserved for k8s scratch (plus ultix-streaming exception)
- Spinning rust for all infrastructure VMs
12.5 Data gaps
| Gap | How to close |
|---|---|
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
| tsys2 post-Proxmox baseline | Run check.sh after Friday install |
| tsys5 NVMe size and model | Confirm after Friday installation |
| tsys3 thermal state (laptop in rack) | Check sensors on next maintenance |
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
12.6 Scripts and tools available
| Script | Purpose |
|---|---|
scripts/check.sh |
Read-only data collector (run on hosts) |
scripts/apply-tunings.sh |
Apply all Tier 0 tunings (dry-run/apply/rollback) |
scripts/fix-bond-nfs.sh |
Fix NFS options + bond hash |
validate-fixes.sh |
Read-only validation of all applied changes |
iperf-full-matrix.sh |
Full iperf test suite (mgmt + storage) |
deploy-check.sh |
Deploy check.sh to all hosts via SSH (now includes tsys9) |