KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: full re-audit of all 7 hosts with fresh ground truth

Browse Source 
Deployed check.sh to all 7 hosts at 21:50 CDT. Captures the live state
after the user's PDM migrations:

Cnode movements since last audit:
- cnode1: tsys1 -> tsys9
- cnode2: tsys6 -> tsys7
- cnode5: tsys6 -> tsys7, storage D5(tsys4) -> S2(tsys5)

Wnode changes:
- wnode-tsys1 (102): new VM on S2, stopped
- wnode-tsys3: RAM bumped 20 -> 28 GB
- wnode-tsys6: now running (was stopped)
- wnode-tsys9: storage moved S3 -> S2

Storage distribution improved from 90/10 to 73/27 (tsys4/tsys5).
Still need 2 more cnode moves for etcd quorum survival.

Updated executive summary, k8s distribution tables, storage
utilization, and open items with the fresh data. Captured future k8s
requirements: vcluster + Rancher, OIDC to Keycloak, workload isolation
(RackRental/Suborbital ITAR/non-ITAR/SLP), and solar-aware scale-out
with PowerEdge 19xx/2950 systems.

Added tsys9 to deploy-check.sh host list.
This commit is contained in:
mrcharles
2026-07-27 22:04:05 -05:00
parent 53954d8494
commit 291a7ebc27
21 changed files with 29261 additions and 21654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT.md
+152 -76
View File
@@ -1,9 +1,13 @@
# Proxmox Cluster Project Report
**Date:** 2026-07-27
**Date:** 2026-07-27 (re-audited)
**Prepared by:** Performance Optimization Engagement
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
`deploy-check.sh`. VM placements reflect live state after user's PDM
migrations. This is ground truth.
---
## Table of Contents
@@ -26,22 +30,23 @@
## 1. Executive Summary
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
running 40+ VMs across two NFS storage servers (tsys4, tsys5). Host-side
performance tunings (CPU governor, swappiness, BBR, NFS nconnect, tuned
profiles) are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5) are blocked
on physical hardware work scheduled for Friday.
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
are blocked on physical hardware work scheduled for Friday.
The VM-layer assessment reveals three structural issues:
**Progress since initial audit:** The user has been actively rebalancing k8s
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
needed for etcd quorum survival.
| # | Finding | Severity | Fix Cost |
|---|---------|----------|----------|
| 1 | **Every pfv-k8s node stores its disk on tsys4 NFS** (10 of 10 nodes). tsys4 failure kills the entire kubernetes cluster. | **CRITICAL** | $0 (rebalance) |
| 2 | **No k8s node uses SSD or NVMe.** All boot from NFS on spinning disks, adding latency to every etcd write and container pull. | **HIGH** | $0 (move to local SSD where available) |
| 3 | **Both -01/-02 infrastructure pairs (netinfra, UCS) are on tsys4 NFS only.** tsys4 failure takes down both halves of each pair. | **HIGH** | $0 (move -02 halves to tsys5 NFS) |
The VM-layer assessment reveals:
Additionally, tsys9 was validated this session and revealed two new findings:
its storage NIC is also a USB dongle (same anti-pattern as tsys4), and it has
a 250 GB local SSD that no VM is using.
| # | Finding | Severity | Status |
|---|---------|----------|--------|
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
---
@@ -87,7 +92,7 @@ running Cloudron -- not in this cluster.
|--------|-----------|------|-----|-------|------|------|------|
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 233 GB | **1.5 TB** | 14% |
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
Non-exported disks on tsys4:
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
@@ -103,11 +108,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|--------|-----------|------|-----|-------|------|------|------|
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 53 GB | 817 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | 2 MB | **870 GB** | **0%** |
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 3 MB | **870 GB** | **0%** |
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | 83 GB | **140 GB** | 35% |
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
**tsys5 storage controllers (plenty of free ports):**
@@ -117,9 +122,11 @@ eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
**Key finding: S2, S3, and S4 are all 99% empty.** The secondary storage server
has 2.2 TB of unused HDD capacity. This is where VMs should be relocated to
reduce tsys4's blast radius.
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
(unused, 7 GB). S4 still 99% empty (435 GB free).
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
**Friday additions:**
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
@@ -312,57 +319,98 @@ If tsys4 goes offline, two-thirds of the fleet loses its storage.
---
## 5. Kubernetes Node Distribution
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
### 5.1 pfv-k8s cnode (control plane) placement
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | Disk Type | Host Role Compliant? |
|------|------|------------|---------|-----------|---------------------|
| 107 | cnode1 | tsys1 | D5 (tsys4) | HDD | **NO** (infra host) |
| 106 | cnode3 | tsys1 | D2 (tsys4) | HDD | **NO** (infra host) |
| 603 | cnode2 | tsys6 | D2 (tsys4) | HDD | YES |
| 601 | cnode4 | tsys6 | D2 (tsys4) | HDD | YES |
| 602 | cnode5 | tsys6 | D5 (tsys4) | HDD | YES |
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|------|------|------------|---------|-----------|----------|
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
**Findings:**
- 2 of 5 cnodes are on tsys1 (an infrastructure host, should be k8s-only).
- **All 5 cnodes store disks on tsys4 NFS.** Zero use tsys5 storage.
- 3 of 5 cnodes share the same physical disk (D2 = WDC Red 3TB).
- All cnodes on spinning HDDs. No SSD/NVMe use.
- **etcd quorum risk:** if D2 (tsys4) fails, cnode2/cnode3/cnode4 lose their
disks simultaneously. That is 3 of 5 control plane nodes -- etcd quorum is
LOST (need 3 of 5 alive; only 2 survive). The kubernetes cluster becomes
inoperable.
**Storage distribution:**
### 5.2 pfv-k8s wnode (worker) placement
| Storage server | cnodes | Quorum impact if it fails |
|---------------|--------|--------------------------|
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
| VMID | Name | Hypervisor | Storage | Disk Type | Status | Host Role Compliant? |
|------|------|------------|---------|-----------|--------|---------------------|
| 313 | wnode-tsys3 | tsys3 | D5 (tsys4) | HDD | running | YES |
| 509 | wnode-tsys5 | tsys5 | D2 (tsys4) | HDD | running | **NO** (storage host) |
| 100 | wnode-tsys6 | tsys6 | D5 (tsys4) | HDD | **STOPPED** | YES (but offline) |
| 701 | wnode-tsys7 | tsys7 | D5 (tsys4) | HDD | running | YES |
| 905 | wnode-tsys9 | tsys9 | S3 (tsys5) | HDD | running | **NO** (infra host) |
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
**Findings:**
- 4 of 5 wnodes are on tsys4 NFS. Only wnode-tsys9 uses tsys5 (S3).
- wnode-tsys5 is on a storage host AND its disk is on tsys4 NFS (not local) --
it depends on tsys4 for its disk despite running on tsys5.
- wnode-tsys6 is STOPPED. The cluster is running without a worker on tsys6.
- wnode-tsys9 is on an infrastructure host (should be on a k8s host), but it
is the ONLY k8s node using tsys5 storage (which is good for redundancy).
- All wnodes on spinning HDDs. No local SSD/NVMe use anywhere.
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
tsys9) -- good host diversity.
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|------|------|------------|---------|-----------|-----|--------|----------|
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
stopped. wnode-tsys6 restarted.
**Storage distribution:**
| Storage server | wnodes | Notes |
|---------------|--------|-------|
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
### 5.3 Summary: k8s node storage distribution
| Storage server | cnodes | wnodes | Total k8s nodes |
|---------------|--------|--------|-----------------|
| **tsys4 NFS** | **5 (100%)** | **4 (80%)** | **9 (90%)** |
| tsys5 NFS | 0 | 1 (20%) | 1 (10%) |
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
**90% of the kubernetes cluster's storage depends on a single host (tsys4).**
This is the single most critical risk in the VM layer.
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
cnodes on each storage server (60/40 or better) so either server failing
leaves quorum intact.
### 5.4 Remaining cnode migration needed for etcd quorum survival
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
be on tsys5:
| Action | Effect |
|--------|--------|
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
### 5.5 Future k8s architecture (next session -- see K8S.md)
The k8s layer will be tackled soon. Key requirements from user:
- **Platform:** vcluster + Rancher for multi-tenant management
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
- **Workload isolation (vcluster per tenant):**
- RackRental workloads (containerlab network labs)
- Suborbital ITAR (compliance-restricted)
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
will be brought online during peak solar production for burst capacity.
These older cores/ram supplement the main fleet when power is abundant.
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
leftover host capacity) to large (28-32 GB, consuming most of a host).
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
full final audit will be performed at that time (post-NVMe install on tsys5,
post-D3 SSD relocation, post-tsys4 NIC+RAM).
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
@@ -745,37 +793,65 @@ This belongs in the next budget cycle.
## 12. Open Items
### 12.1 Friday maintenance window (user action)
### 12.1 Immediate (do today via PDM)
1. **tsys5:** Plug 2nd ethernet cable into storage NIC, verify bond0 recovers,
apply layer3+4 hash, reboot to activate NFS nconnect.
2. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
redundancy. UCS stays on HDD.
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
### 12.2 Friday maintenance window (user action)
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
reconfigure `/etc/network/interfaces`, reboot.
3. **tsys5:** Install PCI NVMe drive, format and mount (recommend local
directory storage for VM images).
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
format NVMe as local storage, reboot.
3. **tsys2:** Load Proxmox (replacing Windows 10).
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
### 12.2 Post-hardware validation
### 12.3 Post-Friday validation
1. Re-run `iperf-full-matrix.sh` to confirm tsys4/tsys5 network recovery.
2. Run `validate-fixes.sh` on tsys4 and tsys5.
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
4. Update PROJECT.md with post-hardware numbers.
### 12.3 Data gaps
### 12.4 Future: Kubernetes deep-dive (see K8S.md)
Next major workstream. Requirements captured:
- **vcluster + Rancher** for multi-tenant k8s management
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
- **Workload isolation** via separate vclusters:
- RackRental (containerlab)
- Suborbital ITAR
- Suborbital non-ITAR
- Starting Line Productions customer workloads
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
during peak solar production for burst capacity
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
- **Spinning rust for all infrastructure VMs**
### 12.5 Data gaps
| Gap | How to close |
|-----|--------------|
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR before planning tsys5 role changes |
| tsys2 post-rebuild baseline | Run `check.sh` once it is converted to Proxmox |
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
| tsys5 NVMe size and model | Confirm after Friday installation |
| Whether hosts form a Proxmox cluster or are standalone | Check `pvecm status` on each host |
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
### 12.4 Scripts and tools available
### 12.6 Scripts and tools available
| Script | Purpose |
|--------|---------|
| `scripts/check.sh` | Read-only data collector (run on hosts) |
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash (live) |
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
| `validate-fixes.sh` | Read-only validation of all applied changes |
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH |
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |