KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4f2bfe18a5cf12f96d15e4e3b42fc00fff5aa06b
PFVCluster/docs/PROJECT.md
T
mrcharles a7fbad60ab docs: end-to-end gardening — links, stale refs, tailscale.md update 
Comprehensive documentation gardening across the merged repo:

- tailscale.md: fully rewritten with current ground truth. The netinfra
  pair now runs production Technitium with all knel.net records
  replicated. Both LAN IPs resolve knel.net device names and recurse
  externally. The old "NXDOMAIN / zone is stale" findings are replaced
  with the resolved state and current recommendations.
- AGENTS.md: rewritten with Gitea-compatible clickable relative links
  to all key scripts and docs. Autonomous commit/push policy
  prominently documented. SSH user corrected to localuser.
- README.md: directory table and docs table now use clickable links.
- All .md cross-references converted to Gitea-renderable relative links.
- Stale path references (ProjectCode/, Project-Tests/, ProjectDocs/)
  updated to current names (provisioning/, tests/) across all docs.
- Stale repo name "FetchApply" / "KNELServerBuild" updated to
  "PFVCluster" in actionable docs; historical AI-review docs tagged
  with an HTML comment notice.
- REFACTORING-EXAMPLES.md: tagged as historical (pre-refactor patterns).
- tests/README.md, dns-cluster-setup/README.md, docs/DEPLOYMENT.md,
  docs/SECURITY.md: path references fixed to current structure.

🤖 Generated with [Crush](https://github.com/charmassociates/crush)

Assisted-by: GLM-5 via Crush <crush@charm.land>
2026-07-28 11:28:52 -05:00

858 lines
40 KiB
Markdown
Raw Blame History

# Proxmox Cluster Project Report
**Date:** 2026-07-27 (re-audited)
**Prepared by:** Performance Optimization Engagement
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
`deploy-check.sh`. VM placements reflect live state after user's PDM
migrations. This is ground truth.
---
## Table of Contents
1. [Executive Summary](#1-executive-summary)
2. [Host Fleet](#2-host-fleet)
3. [Storage Architecture](#3-storage-architecture)
4. [VM Fleet Inventory](#4-vm-fleet-inventory)
5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
8. [Role Alignment Audit](#8-role-alignment-audit)
9. [Network Findings](#9-network-findings)
10. [Recommendations](#10-recommendations)
11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
12. [Open Items](#12-open-items)
---
## 1. Executive Summary
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
are blocked on physical hardware work scheduled for Friday.
**Progress since initial audit:** The user has been actively rebalancing k8s
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
needed for etcd quorum survival.
The VM-layer assessment reveals:
| # | Finding | Severity | Status |
|---|---------|----------|--------|
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
---
## 2. Host Fleet
### 2.1 Inventory
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|------|-------|-----------|-------|-----|-----------|-----------------|--------|
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
### 2.2 Role taxonomy (per user directive)
| Role | Hosts | Workload |
|------|-------|----------|
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
run one per hypervisor host across the fleet to maximize total cluster capacity.
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
running Cloudron -- not in this cluster.
---
## 3. Storage Architecture
### 3.1 NFS exports from tsys4 (primary storage server)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------|
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
Non-exported disks on tsys4:
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
/sdc/sdd), so tsys5 is the better target. See section 3.2.
### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------|
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
**tsys5 storage controllers (plenty of free ports):**
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
**5 free**
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
(unused, 7 GB). S4 still 99% empty (435 GB free).
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
**Friday additions:**
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
boot disk and HPC scratch. The fastest tier in the fleet.
After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
natural home for latency-sensitive workloads and the k8s StorageClass design
center.
### 3.3 Local storage tiers (per host, with utilization)
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|------|-----------|-----------|-----|-------|------|------|-------------|
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
for VM storage.
Additionally, both servers have completely empty internal drive bays:
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
drives.
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
DVD-ROM on port 5). No internal drives.
This is a reliability risk beyond performance: the entire host OS boots
from a consumer-grade portable USB drive not designed for 24/7 server use.
**tsys3 correction:** Previously documented as HDD. Actually boots from a
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
storage.
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
NFS-over-HDD and unsuitable for wnode boot disks.
### 3.4 Disk speed tiers summary
| Tier | Where | Speed class | Best for |
|------|-------|------------|----------|
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
### 3.5 Storage tier characterization per host
| Host | Storage profile | Detail |
|------|----------------|--------|
| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
### 3.6 All exports are single-disk with no redundancy
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
on that export. This applies to **both storage servers** and to the **PBS
backup target** (a single 4.5T SMR drive).
---
## 4. VM Fleet Inventory
### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
#### tsys1 (Infrastructure) -- 11 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys3 (Kubernetes) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
#### tsys4 (Storage) -- 1 running VM
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
#### tsys5 (Storage) -- 15 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys7 (Kubernetes) -- 4 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
#### tsys9 (Infrastructure) -- 5 running VMs
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|------|------|-------|----------|------|---------|------|
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
### 4.2 Storage concentration summary
| Storage target | # of running VMs | % of fleet |
|---------------|-----------------|------------|
| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
| local-nonprod (tsys5 local HDD) | 10 | 25% |
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
| local-lvm (tsys4 local) | 1 | 3% |
**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
If tsys4 goes offline, two-thirds of the fleet loses its storage.
---
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|------|------|------------|---------|-----------|----------|
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
**Storage distribution:**
| Storage server | cnodes | Quorum impact if it fails |
|---------------|--------|--------------------------|
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
tsys9) -- good host diversity.
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|------|------|------------|---------|-----------|-----|--------|----------|
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
stopped. wnode-tsys6 restarted.
**Storage distribution:**
| Storage server | wnodes | Notes |
|---------------|--------|-------|
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
### 5.3 Summary: k8s node storage distribution
| Storage server | cnodes | wnodes | Total k8s nodes |
|---------------|--------|--------|-----------------|
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
cnodes on each storage server (60/40 or better) so either server failing
leaves quorum intact.
### 5.4 Remaining cnode migration needed for etcd quorum survival
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
be on tsys5:
| Action | Effect |
|--------|--------|
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
The k8s layer will be tackled soon. Key requirements from user:
- **Platform:** vcluster + Rancher for multi-tenant management
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
- **Workload isolation (vcluster per tenant):**
- RackRental workloads (containerlab network labs)
- Suborbital ITAR (compliance-restricted)
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
will be brought online during peak solar production for burst capacity.
These older cores/ram supplement the main fleet when power is abundant.
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
leftover host capacity) to large (28-32 GB, consuming most of a host).
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
full final audit will be performed at that time (post-NVMe install on tsys5,
post-D3 SSD relocation, post-tsys4 NIC+RAM).
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
| VMID | Name | Storage |
|------|------|---------|
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
These are on local storage (good -- no NFS dependency) but on a single host's
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
---
## 6. Storage Redundancy Analysis
### 6.1 -01/-02 infrastructure pair audit
Two -01/-02 pairs exist in the fleet:
**Pair 1: pfv-netinfra (network infrastructure)**
| Role | VMID | Host | Storage | NFS Server |
|------|------|------|---------|-----------|
| -01 | 103 | tsys1 | D5 | **tsys4** |
| -02 | 904 | tsys9 | D2 | **tsys4** |
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
disks. The -02 half should be on an S2/S3/S4 export from tsys5.
**Pair 2: tsys-ucs (Univention Corporate Server)**
| Role | VMID | Host | Storage | NFS Server |
|------|------|------|---------|-----------|
| -01 | 108 | tsys1 | D2 | **tsys4** |
| -02 | 902 | tsys9 | D5 | **tsys4** |
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
### 6.2 Redundancy principle for paired VMs
For any -01/-02 pair to survive a single storage server failure:
```
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
```
This ensures that losing either tsys4 or tsys5 takes down only one half of
the pair. Currently, **both pairs fail this test** because both halves are on
tsys4.
### 6.3 NFS server failure blast radius
If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
| Impact | Count |
|--------|-------|
| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
| k8s wnodes that lose storage | 4 of 5 |
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
| Total VMs that lose storage | **27 of 40 (68%)** |
If **tsys5** goes offline:
| Impact | Count |
|--------|-------|
| k8s cnodes that lose storage | 0 of 5 |
| k8s wnodes that lose storage | 1 of 5 |
| Total VMs that lose storage | 1 of 40 (3%) |
**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
Rebalancing VM storage across both servers dramatically reduces risk.
---
## 7. Local SSD/NVMe Opportunity
### 7.1 Available fast tiers (currently unused by k8s)
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|------|--------|------|------|--------------------|-----------------------|
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
### 7.2 Why local storage matters for k8s nodes
Kubernetes nodes are latency-sensitive in two specific areas:
1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
This directly affects k8s API responsiveness and pod scheduling speed.
2. **Container image pulls:** Worker nodes pull container images frequently.
On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
eliminates the network hop and reduces seek time. This matters most for
RackRental/containerlab workloads that spin up containers rapidly.
### 7.3 Current waste: tsys9 local SSD
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
The PCI NVMe being added to tsys5 will be the fastest storage tier in the
fleet. Two placement options:
**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
VMs that might need migration. Adds the NFS/network overhead back.
**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
accessed from other hosts.
**Recommendation:** Option B (local-only). k8s worker nodes do not need
shared storage -- pods are ephemeral and reschedule on failure. The NVMe
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
and used for local VM images.
---
## 8. Role Alignment Audit
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
**correct by design** (keeps heavy hosts free for workers).
### 8.1 VMs that need to move
| VMID | Name | Current host | Issue | Target |
|------|------|-------------|-------|--------|
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
### 8.2 Host capacity for k8s nodes
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|------|------|-------------------|-------------|-----------|----------|
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
free RAM respectively. They should be the primary targets for heavy worker
nodes and ETL/HPC workloads.
### 8.3 tsys6 wnode-tsys6 is stopped
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
cluster capacity.
---
## 9. Network Findings
### 9.1 tsys9 storage NIC is a USB dongle (new finding)
Validating tsys9 revealed that its storage network interface
(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
management; storage uses the USB adapter.
**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
wobble, ESD, and USB controller resets. For an infrastructure host with 5
VMs, this is a reliability risk.
**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
PCIe NIC would eliminate this risk (same recommendation as tsys4).
### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
| Host | Blocker | Staged fix |
|------|---------|-----------|
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
### 9.3 LACP resolved on tsys6/tsys7
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
The switch LACP hash change took effect after renegotiation. The 56-106K
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
### 9.4 NFS nconnect=4 + noatime confirmed active
All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
options. Each host maintains 4 TCP connections per NFS mount to each storage
server. This was the Tier 0 tuning item from the performance optimization
engagement and is confirmed working cluster-wide.
---
## 10. Recommendations
**No changes have been made. These are assessment-only recommendations.**
See `K8S.md` for the detailed kubernetes architecture deep-dive.
### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
These are the most critical production VMs in the fleet. They must survive
any single-point failure (host, storage server, or disk).
**Design principle for -01/-02 HA pairs:**
- Different hypervisors (already satisfied: tsys1 vs tsys9)
- Different storage servers (currently FAILED: all on tsys4)
- Prefer SSD for latency-sensitive services
**Recommended placement:**
| VM | Host | Storage | Tier | Free space | Rationale |
|----|------|---------|------|-----------|-----------|
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
**Failure survival matrix (all single-point failures):**
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|---------|-------------|-------------|--------|--------|--------|
| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
**Why this works:** Every row has at least one netinfra and one UCS alive.
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
Friday) is freed for latency-sensitive workloads that actually benefit from
SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
registry mirror. Not infrastructure VMs.
### 10.2 CRITICAL: Cnode (control plane) storage split
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
quorum (3 of 5 cnodes share D2).
**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
|-------|-------------|----------------|----------------|-------------------|-----------|
| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
**Result after changes:**
| Host | cnodes | Storage server |
|------|--------|---------------|
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
| tsys3 | cnode5 (S3) | tsys5 |
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
etcd keeps quorum (3 of 5 survive).
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
holds more than 2 cnodes.
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
at least 3 cnodes alive.
- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
**Target: one wnode per hypervisor host, using local storage where possible
and NFS-HDD for bulk data.**
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|-------|------|--------------------------|---------------------|-----------|
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
**Storage tiering strategy per wnode:**
Each wnode gets two disk tiers mapped to k8s StorageClasses:
1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
images, ephemeral storage. This is where local SSD/NVMe shines -- container
image pulls and layer extraction are seek-bound and benefit enormously from
low-latency storage.
2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
areas, large files that do not fit on local storage. Mounted as a second
disk in the VM and exposed to k8s as a StorageClass.
This maps to two k8s StorageClasses:
- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
space and ultix-streaming (developer workstation running "cluster of 1"
pre-production jobs). Spinning rust hosts all other infrastructure VMs**
(UCS, netinfra, LibreNMS, SIEM, etc.).
SSD allocation after Friday:
- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
(developer workstation, runs single-node test jobs before k8s). Remaining
140 GB available for k8s.
**Deep-dive on exact k8s scratch allocation is deferred to the next session**
(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
### 10.5 MEDIUM: Restart wnode-tsys6
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
the boot disk to local storage simultaneously.
### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
directory storage.**
Use for:
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
- sectestbed VMs that need fast scratch space
- Not NFS-exported (avoid adding network overhead to the fastest tier)
### 10.7 LOW: Add PCIe NIC to tsys9
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
slots. A $150 PCIe NIC eliminates the USB reliability risk.
### 10.8 LOW: Standardize PVE/kernel versions
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
---
## 11. Hardware End-of-Support Exposure
| Host | EOS date | Years past | Form factor |
|------|----------|-----------|-------------|
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
| pfv-tsys2 (5520) | TBD | -- | Laptop |
| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
The two actual rack servers (R610, R620) are the most overdue for replacement.
This belongs in the next budget cycle.
---
## 12. Open Items
### 12.1 Immediate (do today via PDM)
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
redundancy. UCS stays on HDD.
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
### 12.2 Friday maintenance window (user action)
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
reconfigure `/etc/network/interfaces`, reboot.
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
format NVMe as local storage, reboot.
3. **tsys2:** Load Proxmox (replacing Windows 10).
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
### 12.3 Post-Friday validation
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
4. Update PROJECT.md with post-hardware numbers.
### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
Next major workstream. Requirements captured:
- **vcluster + Rancher** for multi-tenant k8s management
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
- **Workload isolation** via separate vclusters:
- RackRental (containerlab)
- Suborbital ITAR
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
during peak solar production for burst capacity
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
- **Spinning rust for all infrastructure VMs**
### 12.5 Data gaps
| Gap | How to close |
|-----|--------------|
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
| tsys5 NVMe size and model | Confirm after Friday installation |
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
### 12.6 Scripts and tools available
| Script | Purpose |
|--------|---------|
| `scripts/check.sh` | Read-only data collector (run on hosts) |
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
| `validate-fixes.sh` | Read-only validation of all applied changes |
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
Reference in New Issue View Git Blame Copy Permalink