docs: organize docs into project subdirectories
Split docs/ into project-based subdirectories:
- docs/proxmox/ fleet ops, hardware, k8s (PROJECT.md, TODO.md, K8S.md)
- docs/server-build/ provisioning, security, DNS (SECURITY.md, tailscale.md,
DEPLOYMENT.md, TSYS-2FA-GUIDE.md, DEVELOPMENT-GUIDELINES.md)
- docs/archive/ historical AI reviews, completed todos, pre-refactor docs
docmap.md rewritten with new paths. All cross-references in AGENTS.md,
README.md, STATUS.md, and dns-cluster-setup/README.md updated.
Code directories unchanged — scripts stay where BASH_SOURCE expects them.
🤖 Generated with [Crush](https://github.com/charmassociates/crush)
Assisted-by: GLM-5 via Crush <crush@charm.land>
This commit is contained in:
@@ -0,0 +1,485 @@
|
||||
# K8S.md -- Kubernetes Architecture Deep-Dive
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
|
||||
Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment).
|
||||
**Status:** For discussion in a future session. No changes made.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Workload Profile](#1-workload-profile)
|
||||
2. [Current State](#2-current-state)
|
||||
3. [Target Architecture](#3-target-architecture)
|
||||
4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
|
||||
5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
|
||||
6. [Storage Class Design](#6-storage-class-design)
|
||||
7. [ETL/HPC Considerations](#7-etlhpc-considerations)
|
||||
8. [Migration Plan](#8-migration-plan)
|
||||
|
||||
---
|
||||
|
||||
## 1. Workload Profile
|
||||
|
||||
This cluster runs **R&D and RackRental (containerlab) workloads** via
|
||||
Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
|
||||
Reston, VA running Cloudron.
|
||||
|
||||
**Workload types expected:**
|
||||
|
||||
| Type | Description | Storage need | RAM need | Examples |
|
||||
|------|------------|-------------|----------|---------|
|
||||
| **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
|
||||
| **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
|
||||
| **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
|
||||
|
||||
**Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
|
||||
sequential I/O). HPC and containerlab need low-latency random I/O (local
|
||||
SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current State
|
||||
|
||||
### 2.1 pfv-k8s nodes and their storage
|
||||
|
||||
| Node | Type | Host | Storage | Disk type | Status |
|
||||
|------|------|------|---------|-----------|--------|
|
||||
| cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
|
||||
| cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
|
||||
| wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
|
||||
|
||||
### 2.2 Problems
|
||||
|
||||
1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
|
||||
etcd quorum lost.
|
||||
2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
|
||||
3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
|
||||
4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
|
||||
(349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
|
||||
5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
|
||||
6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
|
||||
|
||||
---
|
||||
|
||||
## 3. Target Architecture
|
||||
|
||||
### 3.1 Design principles
|
||||
|
||||
1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
|
||||
heavy workers.
|
||||
2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
|
||||
storage server failing.
|
||||
3. **Wnode boot disks on local storage** -- eliminates NFS latency for
|
||||
container runtime and kubelet.
|
||||
4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
|
||||
5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
|
||||
6. **One wnode per hypervisor host** -- maximize total cluster capacity.
|
||||
|
||||
### 3.2 Target node-host-storage matrix
|
||||
|
||||
| Node | Type | Host | Boot disk | Data disk | Disk type |
|
||||
|------|------|------|-----------|-----------|-----------|
|
||||
| cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
|
||||
| cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
|
||||
| cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
|
||||
| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
|
||||
| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
|
||||
| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
|
||||
| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
|
||||
| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
|
||||
|
||||
### 3.3 Storage server distribution after changes
|
||||
|
||||
| Storage server | cnodes | wnodes (boot) | wnodes (data) |
|
||||
|---------------|--------|---------------|---------------|
|
||||
| tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
|
||||
| tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
|
||||
| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
|
||||
| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
|
||||
|
||||
**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
|
||||
NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
|
||||
than NFS-HDD, and the user has chosen not to install internal drives.
|
||||
|
||||
**No single disk or server is a quorum-losing failure point.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Control Plane (Cnodes)
|
||||
|
||||
### 4.1 Cnode sizing
|
||||
|
||||
Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
|
||||
kubernetes control plane components (API server, scheduler, controller-manager).
|
||||
|
||||
### 4.2 Cnode host placement rationale
|
||||
|
||||
| Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
|
||||
|------|--------|---------------|---------------|------------------------|
|
||||
| tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
|
||||
| tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
|
||||
| tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
|
||||
|
||||
**tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
|
||||
|
||||
### 4.3 Cnode storage placement rationale
|
||||
|
||||
The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
|
||||
|
||||
| Disk | cnodes | Rationale |
|
||||
|------|--------|-----------|
|
||||
| D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
|
||||
| D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
|
||||
| S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
|
||||
| S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
|
||||
|
||||
**If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
|
||||
**If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
|
||||
**If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
|
||||
tsys5. **Only 2 of 5 -- QUORUM LOST.**
|
||||
|
||||
Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
|
||||
cnodes and only have 2 on tsys5. That loses quorum (need 3).
|
||||
|
||||
**Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
|
||||
3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
|
||||
tsys4. Same problem inverted.
|
||||
|
||||
The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
|
||||
3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
|
||||
holding 2 cnodes is safe (3 survive).
|
||||
|
||||
**Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
|
||||
other.** After Friday's hardware work:
|
||||
- tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
|
||||
- tsys5 will have bond0 fixed + NVMe -- more reliable
|
||||
|
||||
Either way, 3/2 split means one server failure could lose quorum. **To truly
|
||||
solve this, use a 3rd storage target.** Options:
|
||||
- Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
|
||||
independent failure domain)
|
||||
- Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
|
||||
no NFS dependency at all)
|
||||
|
||||
**Best option: put cnode boot disks on local-lvm where available.** This
|
||||
eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
|
||||
its own host's local disk -- no shared dependency.
|
||||
|
||||
| cnode | Host | **Recommended storage** | Type |
|
||||
|-------|------|------------------------|------|
|
||||
| cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
|
||||
| cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
|
||||
| cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
|
||||
|
||||
With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
|
||||
down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
|
||||
survives.
|
||||
|
||||
**This is the recommended approach.** Local storage for cnodes wherever
|
||||
possible. NFS only as fallback.
|
||||
|
||||
### 4.4 etcd performance on local vs NFS
|
||||
|
||||
| Storage | Typical fsync latency | etcd commit latency | Impact |
|
||||
|---------|----------------------|--------------------|--------|
|
||||
| NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
|
||||
| NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
|
||||
| Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
|
||||
| Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
|
||||
| NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
|
||||
|
||||
etcd is the heartbeat of the kubernetes control plane. Every API call, every
|
||||
pod schedule, every controller reconciliation involves an etcd write. Cutting
|
||||
etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
|
||||
responsive. **This is the single highest-impact change for k8s performance.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Worker Nodes (Wnodes)
|
||||
|
||||
### 5.1 One wnode per hypervisor host
|
||||
|
||||
| Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
|
||||
|------|-------|-----------|-----------|-----------|-----------|------|
|
||||
| tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
|
||||
| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
|
||||
| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
|
||||
| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
|
||||
| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
|
||||
|
||||
### 5.2 Why boot disks on local-lvm
|
||||
|
||||
Current: all wnodes boot from NFS. Every container image pull, every kubelet
|
||||
log write, every ephemeral volume traverses the NFS network path.
|
||||
|
||||
With local-lvm boot disks:
|
||||
- **Container image pulls** write to local disk (100-150 MB/s HDD, no network
|
||||
hop) instead of NFS-HDD (80-120 MB/s with network latency)
|
||||
- **kubelet logs** stay local (no NFS writes for log rotation)
|
||||
- **ephemeral storage** (emptyDir volumes) uses local disk by default
|
||||
- **NFS server failure does not kill the wnode** -- the VM stays running, only
|
||||
the data disk (if mounted) goes away
|
||||
|
||||
### 5.3 Wnode sizing guidance
|
||||
|
||||
| Host | Recommended wnode config | Rationale |
|
||||
|------|------------------------|-----------|
|
||||
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
|
||||
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
|
||||
| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
|
||||
| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
|
||||
| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
|
||||
| tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
|
||||
| tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
|
||||
|
||||
### 5.4 Tainting and labeling strategy
|
||||
|
||||
Label wnodes by capability so the k8s scheduler can target them:
|
||||
|
||||
```yaml
|
||||
# Heavy RAM hosts (ETL/HPC)
|
||||
wnode-tsys6: workload=heavy, ram=128g
|
||||
wnode-tsys7: workload=heavy, ram=192g
|
||||
|
||||
# NVMe host (I/O-intensive HPC)
|
||||
wnode-tsys5: workload=hpc, storage=nvme
|
||||
|
||||
# SSD boot host (low-latency)
|
||||
wnode-tsys9: workload=light, storage=ssd
|
||||
|
||||
# General workers
|
||||
wnode-tsys3: workload=general
|
||||
wnode-tsys2: workload=storage-fast, storage=nvme
|
||||
wnode-tsys1: workload=light
|
||||
```
|
||||
|
||||
Then use nodeSelector or nodeAffinity in job specs:
|
||||
|
||||
```yaml
|
||||
# Weather/GIS ETL job -- needs lots of RAM
|
||||
spec:
|
||||
nodeSelector:
|
||||
workload: heavy
|
||||
|
||||
# Firmware build -- needs fast storage
|
||||
spec:
|
||||
nodeSelector:
|
||||
storage: nvme
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Class Design
|
||||
|
||||
### 6.1 Proposed StorageClasses
|
||||
|
||||
| StorageClass | Provisioner | Where | Speed | Use case |
|
||||
|-------------|------------|-------|-------|----------|
|
||||
| `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
|
||||
| `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
|
||||
| `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
|
||||
|
||||
### 6.2 How this maps to wnode disk topology
|
||||
|
||||
Each wnode has:
|
||||
- **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
|
||||
runtime. k8s `local-fast` StorageClass provisioner points here.
|
||||
- **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
|
||||
second block device or filesystem. k8s `nfs-hdd` provisioner points here.
|
||||
|
||||
Inside k8s, pods request storage via PVC:
|
||||
|
||||
```yaml
|
||||
# ETL job: needs bulk storage for weather data
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
spec:
|
||||
storageClassName: nfs-hdd
|
||||
accessModes: [ReadWriteMany] # NFS allows RWX
|
||||
resources:
|
||||
requests:
|
||||
storage: 500Gi
|
||||
|
||||
# HPC job: needs fast scratch
|
||||
spec:
|
||||
storageClassName: local-fast
|
||||
accessModes: [ReadWriteOnce]
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
```
|
||||
|
||||
### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
|
||||
scratch space, with the exception of ultix-streaming which stays on T5-SSD.
|
||||
Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
|
||||
SIEM, etc.).
|
||||
|
||||
The SSD NFS exports:
|
||||
- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
|
||||
cache, ephemeral volumes). Currently 99% empty.
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
|
||||
occupies 83 GB. Remaining 140 GB available for k8s use.
|
||||
|
||||
**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
|
||||
This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
|
||||
tsys5 SSD exports, bulk PVCs target either server.
|
||||
|
||||
### 6.4 NFS data distribution across storage servers
|
||||
|
||||
To avoid re-creating the "everything on tsys4" problem, distribute NFS data
|
||||
disks across both servers:
|
||||
|
||||
| wnode | Boot (local) | Bulk data (NFS) | NFS server |
|
||||
|-------|-------------|-----------------|------------|
|
||||
| wnode-tsys3 | local-lvm | S3 | tsys5 |
|
||||
| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
|
||||
| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
|
||||
|
||||
This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
|
||||
|
||||
---
|
||||
|
||||
## 7. ETL/HPC Considerations
|
||||
|
||||
### 7.1 Weather/GIS ETL pipeline
|
||||
|
||||
Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
|
||||
store results.
|
||||
|
||||
| Stage | Storage class | Why |
|
||||
|-------|-------------|-----|
|
||||
| Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
|
||||
| Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
|
||||
| Store results | `nfs-hdd` | Large sequential writes. Persistent. |
|
||||
|
||||
**Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
|
||||
wnode. ETL jobs use `/scratch` for intermediate processing and write final
|
||||
output to the NFS-mounted `/data`.
|
||||
|
||||
### 7.2 HPC workloads (hardware startup)
|
||||
|
||||
Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
|
||||
|
||||
| Workload | Best wnode | Why |
|
||||
|----------|-----------|-----|
|
||||
| RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
|
||||
| Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
|
||||
| Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
|
||||
| Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
|
||||
|
||||
### 7.3 RackRental/containerlab
|
||||
|
||||
Rapid container deployment. Key need: fast container image pulls.
|
||||
|
||||
This is where **local-lvm boot disks** shine. Currently, every container image
|
||||
pull writes through NFS to a spinning disk -- slow. With local-lvm, images
|
||||
cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
|
||||
On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
|
||||
|
||||
### 7.4 Data locality for ETL
|
||||
|
||||
For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
|
||||
it on local-lvm of the heavy hosts:
|
||||
|
||||
```
|
||||
tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier
|
||||
tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier
|
||||
```
|
||||
|
||||
**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
|
||||
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
|
||||
|
||||
This avoids re-reading the same data from the same NFS export on every job
|
||||
if the data is already cached in the page cache.
|
||||
|
||||
---
|
||||
|
||||
## 8. Migration Plan
|
||||
|
||||
**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
|
||||
Datacenter Manager (PDM)** manages them collectively and supports VM migration
|
||||
between nodes. Storage migration can be done via the PDM/Proxmox UI rather
|
||||
than manual disk copies -- the destination node just needs access to the target
|
||||
storage (which all nodes have for NFS exports, and local storage can be
|
||||
migrated through the UI's "Storage Migrate" function).
|
||||
|
||||
### 8.1 Phase 1: Friday (after hardware work)
|
||||
|
||||
After tsys5 cable + NVMe and tsys4 NIC + RAM:
|
||||
|
||||
1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
|
||||
2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
|
||||
do not use for VM storage. Recreate on D2 or D5 NFS.
|
||||
3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
|
||||
|
||||
### 8.2 Phase 2: Cnode rebalance (maintenance window)
|
||||
|
||||
These changes require creating new VMs on target hosts and migrating disks.
|
||||
Plan for a maintenance window with the k8s cluster briefly down.
|
||||
|
||||
1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
|
||||
2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
|
||||
3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
|
||||
4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
|
||||
5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
|
||||
|
||||
### 8.3 Phase 3: Wnode local storage migration (maintenance window)
|
||||
|
||||
1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
|
||||
2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
|
||||
3. wnode-tsys7 stays on NFS (same reason).
|
||||
4. Recreate wnode-tsys5 with boot disk on NVMe.
|
||||
5. Add data disks (NFS) as second SCSI devices where applicable.
|
||||
|
||||
### 8.4 Phase 4: tsys2 integration (when rebuilt)
|
||||
|
||||
1. Install Proxmox on tsys2.
|
||||
2. Run `scripts/check.sh` to inventory.
|
||||
3. Run `scripts/apply-tunings.sh --apply`.
|
||||
4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
|
||||
5. Join to k8s cluster.
|
||||
|
||||
### 8.5 Phase 5: Critical VM relocation
|
||||
|
||||
1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
|
||||
2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
|
||||
3. (No change to T5-SSD -- ultix-streaming stays.)
|
||||
|
||||
---
|
||||
|
||||
## Open questions for next session
|
||||
|
||||
1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
|
||||
whether live migration is available (huge simplification) or we need manual
|
||||
disk migration. Check `pvecm status` on each host.
|
||||
2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
|
||||
nodes are joined/drain and how StorageClasses are configured.
|
||||
3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
|
||||
4. **Is there a container image registry mirror in the cluster?** Or do all
|
||||
pulls go to Docker Hub / external? A local registry on D3 SSD would speed
|
||||
up all pulls.
|
||||
5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
|
||||
whether jobs need shared (RWX) or exclusive (RWO) storage.
|
||||
6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
|
||||
how we label and taint nodes.
|
||||
@@ -0,0 +1,857 @@
|
||||
# Proxmox Cluster Project Report
|
||||
|
||||
**Date:** 2026-07-27 (re-audited)
|
||||
**Prepared by:** Performance Optimization Engagement
|
||||
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
|
||||
|
||||
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
|
||||
`deploy-check.sh`. VM placements reflect live state after user's PDM
|
||||
migrations. This is ground truth.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#1-executive-summary)
|
||||
2. [Host Fleet](#2-host-fleet)
|
||||
3. [Storage Architecture](#3-storage-architecture)
|
||||
4. [VM Fleet Inventory](#4-vm-fleet-inventory)
|
||||
5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
|
||||
6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
|
||||
7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
|
||||
8. [Role Alignment Audit](#8-role-alignment-audit)
|
||||
9. [Network Findings](#9-network-findings)
|
||||
10. [Recommendations](#10-recommendations)
|
||||
11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
|
||||
12. [Open Items](#12-open-items)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
|
||||
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
|
||||
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
|
||||
are blocked on physical hardware work scheduled for Friday.
|
||||
|
||||
**Progress since initial audit:** The user has been actively rebalancing k8s
|
||||
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
|
||||
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
|
||||
needed for etcd quorum survival.
|
||||
|
||||
The VM-layer assessment reveals:
|
||||
|
||||
| # | Finding | Severity | Status |
|
||||
|---|---------|----------|--------|
|
||||
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
|
||||
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
|
||||
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
|
||||
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
|
||||
|
||||
---
|
||||
|
||||
## 2. Host Fleet
|
||||
|
||||
### 2.1 Inventory
|
||||
|
||||
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|
||||
|------|-------|-----------|-------|-----|-----------|-----------------|--------|
|
||||
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
|
||||
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
|
||||
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
|
||||
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
|
||||
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
|
||||
|
||||
### 2.2 Role taxonomy (per user directive)
|
||||
|
||||
| Role | Hosts | Workload |
|
||||
|------|-------|----------|
|
||||
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
|
||||
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
|
||||
| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
|
||||
|
||||
**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
|
||||
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
|
||||
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
|
||||
run one per hypervisor host across the fleet to maximize total cluster capacity.
|
||||
|
||||
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
|
||||
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
|
||||
running Cloudron -- not in this cluster.
|
||||
|
||||
---
|
||||
|
||||
## 3. Storage Architecture
|
||||
|
||||
### 3.1 NFS exports from tsys4 (primary storage server)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
|
||||
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
|
||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
|
||||
|
||||
Non-exported disks on tsys4:
|
||||
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
|
||||
- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
|
||||
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
|
||||
|
||||
**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
|
||||
tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
|
||||
eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
|
||||
/sdc/sdd), so tsys5 is the better target. See section 3.2.
|
||||
|
||||
### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
|
||||
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
|
||||
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
|
||||
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
|
||||
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
|
||||
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
|
||||
|
||||
**tsys5 storage controllers (plenty of free ports):**
|
||||
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
|
||||
**5 free**
|
||||
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
|
||||
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
|
||||
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
|
||||
|
||||
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
|
||||
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
|
||||
(unused, 7 GB). S4 still 99% empty (435 GB free).
|
||||
|
||||
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
|
||||
|
||||
**Friday additions:**
|
||||
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
|
||||
USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
|
||||
2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
|
||||
boot disk and HPC scratch. The fastest tier in the fleet.
|
||||
|
||||
After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
|
||||
NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
|
||||
natural home for latency-sensitive workloads and the k8s StorageClass design
|
||||
center.
|
||||
|
||||
### 3.3 Local storage tiers (per host, with utilization)
|
||||
|
||||
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|
||||
|------|-----------|-----------|-----|-------|------|------|-------------|
|
||||
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
|
||||
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
|
||||
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
|
||||
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
|
||||
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
|
||||
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
|
||||
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
|
||||
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
|
||||
|
||||
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
|
||||
|
||||
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
|
||||
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
|
||||
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
|
||||
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
|
||||
|
||||
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
|
||||
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
|
||||
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
|
||||
for VM storage.
|
||||
|
||||
Additionally, both servers have completely empty internal drive bays:
|
||||
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
|
||||
drives.
|
||||
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
|
||||
DVD-ROM on port 5). No internal drives.
|
||||
|
||||
This is a reliability risk beyond performance: the entire host OS boots
|
||||
from a consumer-grade portable USB drive not designed for 24/7 server use.
|
||||
|
||||
**tsys3 correction:** Previously documented as HDD. Actually boots from a
|
||||
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
|
||||
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
|
||||
|
||||
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
|
||||
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
|
||||
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
|
||||
storage.
|
||||
|
||||
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
|
||||
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
|
||||
NFS-over-HDD and unsuitable for wnode boot disks.
|
||||
|
||||
### 3.4 Disk speed tiers summary
|
||||
|
||||
| Tier | Where | Speed class | Best for |
|
||||
|------|-------|------------|----------|
|
||||
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
|
||||
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
|
||||
| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
|
||||
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
|
||||
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
|
||||
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
|
||||
|
||||
### 3.5 Storage tier characterization per host
|
||||
|
||||
| Host | Storage profile | Detail |
|
||||
|------|----------------|--------|
|
||||
| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
|
||||
| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
|
||||
| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
|
||||
| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
|
||||
| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
|
||||
| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
|
||||
| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
|
||||
|
||||
### 3.6 All exports are single-disk with no redundancy
|
||||
|
||||
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
|
||||
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
|
||||
on that export. This applies to **both storage servers** and to the **PBS
|
||||
backup target** (a single 4.5T SMR drive).
|
||||
|
||||
---
|
||||
|
||||
## 4. VM Fleet Inventory
|
||||
|
||||
### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
|
||||
|
||||
#### tsys1 (Infrastructure) -- 11 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys3 (Kubernetes) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys4 (Storage) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
|
||||
|
||||
#### tsys5 (Storage) -- 15 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
|
||||
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
|
||||
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
|
||||
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
|
||||
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
|
||||
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
|
||||
|
||||
#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys7 (Kubernetes) -- 4 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys9 (Infrastructure) -- 5 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
|
||||
|
||||
### 4.2 Storage concentration summary
|
||||
|
||||
| Storage target | # of running VMs | % of fleet |
|
||||
|---------------|-----------------|------------|
|
||||
| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
|
||||
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
|
||||
| local-nonprod (tsys5 local HDD) | 10 | 25% |
|
||||
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
|
||||
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
|
||||
| local-lvm (tsys4 local) | 1 | 3% |
|
||||
|
||||
**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
|
||||
If tsys4 goes offline, two-thirds of the fleet loses its storage.
|
||||
|
||||
---
|
||||
|
||||
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
|
||||
|
||||
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|
||||
|------|------|------------|---------|-----------|----------|
|
||||
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
|
||||
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
|
||||
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
|
||||
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
|
||||
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | cnodes | Quorum impact if it fails |
|
||||
|---------------|--------|--------------------------|
|
||||
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
|
||||
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
|
||||
|
||||
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
|
||||
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
|
||||
|
||||
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
|
||||
tsys9) -- good host diversity.
|
||||
|
||||
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|
||||
|------|------|------------|---------|-----------|-----|--------|----------|
|
||||
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
|
||||
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
|
||||
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
|
||||
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
|
||||
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
|
||||
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
|
||||
|
||||
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
|
||||
stopped. wnode-tsys6 restarted.
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | wnodes | Notes |
|
||||
|---------------|--------|-------|
|
||||
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
|
||||
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
|
||||
|
||||
### 5.3 Summary: k8s node storage distribution
|
||||
|
||||
| Storage server | cnodes | wnodes | Total k8s nodes |
|
||||
|---------------|--------|--------|-----------------|
|
||||
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
|
||||
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
|
||||
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
|
||||
|
||||
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
|
||||
cnodes on each storage server (60/40 or better) so either server failing
|
||||
leaves quorum intact.
|
||||
|
||||
### 5.4 Remaining cnode migration needed for etcd quorum survival
|
||||
|
||||
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
|
||||
be on tsys5:
|
||||
|
||||
| Action | Effect |
|
||||
|--------|--------|
|
||||
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
|
||||
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
|
||||
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
|
||||
|
||||
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
|
||||
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
|
||||
|
||||
### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
|
||||
|
||||
The k8s layer will be tackled soon. Key requirements from user:
|
||||
|
||||
- **Platform:** vcluster + Rancher for multi-tenant management
|
||||
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
|
||||
- **Workload isolation (vcluster per tenant):**
|
||||
- RackRental workloads (containerlab network labs)
|
||||
- Suborbital ITAR (compliance-restricted)
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
|
||||
will be brought online during peak solar production for burst capacity.
|
||||
These older cores/ram supplement the main fleet when power is abundant.
|
||||
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
|
||||
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
|
||||
leftover host capacity) to large (28-32 GB, consuming most of a host).
|
||||
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
|
||||
full final audit will be performed at that time (post-NVMe install on tsys5,
|
||||
post-D3 SSD relocation, post-tsys4 NIC+RAM).
|
||||
|
||||
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
|
||||
|
||||
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
|
||||
|
||||
| VMID | Name | Storage |
|
||||
|------|------|---------|
|
||||
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
|
||||
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
|
||||
|
||||
These are on local storage (good -- no NFS dependency) but on a single host's
|
||||
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Redundancy Analysis
|
||||
|
||||
### 6.1 -01/-02 infrastructure pair audit
|
||||
|
||||
Two -01/-02 pairs exist in the fleet:
|
||||
|
||||
**Pair 1: pfv-netinfra (network infrastructure)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 103 | tsys1 | D5 | **tsys4** |
|
||||
| -02 | 904 | tsys9 | D2 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
|
||||
disks. The -02 half should be on an S2/S3/S4 export from tsys5.
|
||||
|
||||
**Pair 2: tsys-ucs (Univention Corporate Server)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 108 | tsys1 | D2 | **tsys4** |
|
||||
| -02 | 902 | tsys9 | D5 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
|
||||
|
||||
### 6.2 Redundancy principle for paired VMs
|
||||
|
||||
For any -01/-02 pair to survive a single storage server failure:
|
||||
|
||||
```
|
||||
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
|
||||
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
|
||||
```
|
||||
|
||||
This ensures that losing either tsys4 or tsys5 takes down only one half of
|
||||
the pair. Currently, **both pairs fail this test** because both halves are on
|
||||
tsys4.
|
||||
|
||||
### 6.3 NFS server failure blast radius
|
||||
|
||||
If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
|
||||
| k8s wnodes that lose storage | 4 of 5 |
|
||||
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
|
||||
| Total VMs that lose storage | **27 of 40 (68%)** |
|
||||
|
||||
If **tsys5** goes offline:
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 0 of 5 |
|
||||
| k8s wnodes that lose storage | 1 of 5 |
|
||||
| Total VMs that lose storage | 1 of 40 (3%) |
|
||||
|
||||
**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
|
||||
Rebalancing VM storage across both servers dramatically reduces risk.
|
||||
|
||||
---
|
||||
|
||||
## 7. Local SSD/NVMe Opportunity
|
||||
|
||||
### 7.1 Available fast tiers (currently unused by k8s)
|
||||
|
||||
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|
||||
|------|--------|------|------|--------------------|-----------------------|
|
||||
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
|
||||
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
|
||||
| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
|
||||
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
|
||||
|
||||
### 7.2 Why local storage matters for k8s nodes
|
||||
|
||||
Kubernetes nodes are latency-sensitive in two specific areas:
|
||||
|
||||
1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
|
||||
On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
|
||||
client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
|
||||
latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
|
||||
This directly affects k8s API responsiveness and pod scheduling speed.
|
||||
|
||||
2. **Container image pulls:** Worker nodes pull container images frequently.
|
||||
On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
|
||||
eliminates the network hop and reduces seek time. This matters most for
|
||||
RackRental/containerlab workloads that spin up containers rapidly.
|
||||
|
||||
### 7.3 Current waste: tsys9 local SSD
|
||||
|
||||
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
|
||||
**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
|
||||
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
|
||||
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
|
||||
|
||||
### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
|
||||
|
||||
The PCI NVMe being added to tsys5 will be the fastest storage tier in the
|
||||
fleet. Two placement options:
|
||||
|
||||
**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
|
||||
VMs that might need migration. Adds the NFS/network overhead back.
|
||||
|
||||
**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
|
||||
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
|
||||
accessed from other hosts.
|
||||
|
||||
**Recommendation:** Option B (local-only). k8s worker nodes do not need
|
||||
shared storage -- pods are ephemeral and reschedule on failure. The NVMe
|
||||
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
|
||||
and used for local VM images.
|
||||
|
||||
---
|
||||
|
||||
## 8. Role Alignment Audit
|
||||
|
||||
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
|
||||
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
|
||||
**correct by design** (keeps heavy hosts free for workers).
|
||||
|
||||
### 8.1 VMs that need to move
|
||||
|
||||
| VMID | Name | Current host | Issue | Target |
|
||||
|------|------|-------------|-------|--------|
|
||||
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
|
||||
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
|
||||
|
||||
### 8.2 Host capacity for k8s nodes
|
||||
|
||||
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|
||||
|------|------|-------------------|-------------|-----------|----------|
|
||||
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
|
||||
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
|
||||
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
|
||||
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
|
||||
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
|
||||
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
|
||||
|
||||
**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
|
||||
free RAM respectively. They should be the primary targets for heavy worker
|
||||
nodes and ETL/HPC workloads.
|
||||
|
||||
### 8.3 tsys6 wnode-tsys6 is stopped
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
|
||||
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
|
||||
cluster capacity.
|
||||
|
||||
---
|
||||
|
||||
## 9. Network Findings
|
||||
|
||||
### 9.1 tsys9 storage NIC is a USB dongle (new finding)
|
||||
|
||||
Validating tsys9 revealed that its storage network interface
|
||||
(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
|
||||
anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
|
||||
management; storage uses the USB adapter.
|
||||
|
||||
**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
|
||||
wobble, ESD, and USB controller resets. For an infrastructure host with 5
|
||||
VMs, this is a reliability risk.
|
||||
|
||||
**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
|
||||
PCIe NIC would eliminate this risk (same recommendation as tsys4).
|
||||
|
||||
### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
|
||||
|
||||
| Host | Blocker | Staged fix |
|
||||
|------|---------|-----------|
|
||||
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
|
||||
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
|
||||
|
||||
### 9.3 LACP resolved on tsys6/tsys7
|
||||
|
||||
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
|
||||
The switch LACP hash change took effect after renegotiation. The 56-106K
|
||||
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
|
||||
LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
|
||||
|
||||
### 9.4 NFS nconnect=4 + noatime confirmed active
|
||||
|
||||
All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
|
||||
options. Each host maintains 4 TCP connections per NFS mount to each storage
|
||||
server. This was the Tier 0 tuning item from the performance optimization
|
||||
engagement and is confirmed working cluster-wide.
|
||||
|
||||
---
|
||||
|
||||
## 10. Recommendations
|
||||
|
||||
**No changes have been made. These are assessment-only recommendations.**
|
||||
See `K8S.md` for the detailed kubernetes architecture deep-dive.
|
||||
|
||||
### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
|
||||
|
||||
These are the most critical production VMs in the fleet. They must survive
|
||||
any single-point failure (host, storage server, or disk).
|
||||
|
||||
**Design principle for -01/-02 HA pairs:**
|
||||
- Different hypervisors (already satisfied: tsys1 vs tsys9)
|
||||
- Different storage servers (currently FAILED: all on tsys4)
|
||||
- Prefer SSD for latency-sensitive services
|
||||
|
||||
**Recommended placement:**
|
||||
|
||||
| VM | Host | Storage | Tier | Free space | Rationale |
|
||||
|----|------|---------|------|-----------|-----------|
|
||||
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
|
||||
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
|
||||
| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
|
||||
| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
|
||||
|
||||
**Failure survival matrix (all single-point failures):**
|
||||
|
||||
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|
||||
|---------|-------------|-------------|--------|--------|--------|
|
||||
| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
|
||||
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
|
||||
| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
|
||||
| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
|
||||
| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
|
||||
|
||||
**Why this works:** Every row has at least one netinfra and one UCS alive.
|
||||
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
|
||||
|
||||
**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
|
||||
Friday) is freed for latency-sensitive workloads that actually benefit from
|
||||
SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
|
||||
registry mirror. Not infrastructure VMs.
|
||||
|
||||
### 10.2 CRITICAL: Cnode (control plane) storage split
|
||||
|
||||
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
|
||||
quorum (3 of 5 cnodes share D2).
|
||||
|
||||
**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
|
||||
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
|
||||
|
||||
| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
|
||||
|-------|-------------|----------------|----------------|-------------------|-----------|
|
||||
| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
|
||||
| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
|
||||
| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
|
||||
| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
|
||||
| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
|
||||
|
||||
**Result after changes:**
|
||||
|
||||
| Host | cnodes | Storage server |
|
||||
|------|--------|---------------|
|
||||
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
|
||||
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
|
||||
| tsys3 | cnode5 (S3) | tsys5 |
|
||||
|
||||
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
|
||||
etcd keeps quorum (3 of 5 survive).
|
||||
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
|
||||
holds more than 2 cnodes.
|
||||
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
|
||||
at least 3 cnodes alive.
|
||||
- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
|
||||
|
||||
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
|
||||
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
|
||||
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
|
||||
|
||||
### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
|
||||
|
||||
**Target: one wnode per hypervisor host, using local storage where possible
|
||||
and NFS-HDD for bulk data.**
|
||||
|
||||
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|
||||
|-------|------|--------------------------|---------------------|-----------|
|
||||
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
|
||||
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
|
||||
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
|
||||
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
|
||||
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
|
||||
|
||||
**Storage tiering strategy per wnode:**
|
||||
|
||||
Each wnode gets two disk tiers mapped to k8s StorageClasses:
|
||||
|
||||
1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
|
||||
images, ephemeral storage. This is where local SSD/NVMe shines -- container
|
||||
image pulls and layer extraction are seek-bound and benefit enormously from
|
||||
low-latency storage.
|
||||
|
||||
2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
|
||||
areas, large files that do not fit on local storage. Mounted as a second
|
||||
disk in the VM and exposed to k8s as a StorageClass.
|
||||
|
||||
This maps to two k8s StorageClasses:
|
||||
- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
|
||||
- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
|
||||
|
||||
### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
|
||||
space and ultix-streaming (developer workstation running "cluster of 1"
|
||||
pre-production jobs). Spinning rust hosts all other infrastructure VMs**
|
||||
(UCS, netinfra, LibreNMS, SIEM, etc.).
|
||||
|
||||
SSD allocation after Friday:
|
||||
- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
|
||||
the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
|
||||
(developer workstation, runs single-node test jobs before k8s). Remaining
|
||||
140 GB available for k8s.
|
||||
|
||||
**Deep-dive on exact k8s scratch allocation is deferred to the next session**
|
||||
(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
|
||||
|
||||
### 10.5 MEDIUM: Restart wnode-tsys6
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
|
||||
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
|
||||
the boot disk to local storage simultaneously.
|
||||
|
||||
### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
|
||||
|
||||
**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
|
||||
directory storage.**
|
||||
|
||||
Use for:
|
||||
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
|
||||
- sectestbed VMs that need fast scratch space
|
||||
- Not NFS-exported (avoid adding network overhead to the fastest tier)
|
||||
|
||||
### 10.7 LOW: Add PCIe NIC to tsys9
|
||||
|
||||
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
|
||||
slots. A $150 PCIe NIC eliminates the USB reliability risk.
|
||||
|
||||
### 10.8 LOW: Standardize PVE/kernel versions
|
||||
|
||||
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
|
||||
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
|
||||
|
||||
---
|
||||
|
||||
## 11. Hardware End-of-Support Exposure
|
||||
|
||||
| Host | EOS date | Years past | Form factor |
|
||||
|------|----------|-----------|-------------|
|
||||
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
|
||||
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
|
||||
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
|
||||
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
|
||||
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
|
||||
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
|
||||
| pfv-tsys2 (5520) | TBD | -- | Laptop |
|
||||
| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
|
||||
|
||||
**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
|
||||
The two actual rack servers (R610, R620) are the most overdue for replacement.
|
||||
This belongs in the next budget cycle.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open Items
|
||||
|
||||
### 12.1 Immediate (do today via PDM)
|
||||
|
||||
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
|
||||
redundancy. UCS stays on HDD.
|
||||
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
|
||||
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
|
||||
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
|
||||
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
|
||||
|
||||
### 12.2 Friday maintenance window (user action)
|
||||
|
||||
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
|
||||
reconfigure `/etc/network/interfaces`, reboot.
|
||||
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
|
||||
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
|
||||
format NVMe as local storage, reboot.
|
||||
3. **tsys2:** Load Proxmox (replacing Windows 10).
|
||||
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
|
||||
|
||||
### 12.3 Post-Friday validation
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
|
||||
4. Update PROJECT.md with post-hardware numbers.
|
||||
|
||||
### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
|
||||
|
||||
Next major workstream. Requirements captured:
|
||||
- **vcluster + Rancher** for multi-tenant k8s management
|
||||
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
|
||||
- **Workload isolation** via separate vclusters:
|
||||
- RackRental (containerlab)
|
||||
- Suborbital ITAR
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
|
||||
during peak solar production for burst capacity
|
||||
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
|
||||
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
|
||||
- **Spinning rust for all infrastructure VMs**
|
||||
|
||||
### 12.5 Data gaps
|
||||
|
||||
| Gap | How to close |
|
||||
|-----|--------------|
|
||||
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
|
||||
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
|
||||
| tsys5 NVMe size and model | Confirm after Friday installation |
|
||||
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
|
||||
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
|
||||
|
||||
### 12.6 Scripts and tools available
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/check.sh` | Read-only data collector (run on hosts) |
|
||||
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
||||
| `validate-fixes.sh` | Read-only validation of all applied changes |
|
||||
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
|
||||
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
|
||||
@@ -0,0 +1,242 @@
|
||||
# TODO.md — Pending User Actions
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Items needing user input or physical action.**
|
||||
|
||||
---
|
||||
|
||||
## 1. tsys2 Windows hardware inventory (run on the Windows host)
|
||||
|
||||
pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
|
||||
hardware data so the architecture plan can account for it.
|
||||
|
||||
### Option A: PowerShell (recommended — single command, copy-paste output)
|
||||
|
||||
Open **PowerShell as Administrator** and run:
|
||||
|
||||
```powershell
|
||||
# Full hardware inventory in one shot
|
||||
Write-Output "=== COMPUTER ==="
|
||||
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
|
||||
|
||||
Write-Output "`n=== CPU ==="
|
||||
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
|
||||
|
||||
Write-Output "`n=== MEMORY STICKS ==="
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISKS ==="
|
||||
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISK PARTITIONS ==="
|
||||
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== NETWORK ADAPTERS ==="
|
||||
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== GPU(s) ==="
|
||||
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
|
||||
|
||||
Write-Output "`n=== PCIe SLOTS ==="
|
||||
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== USB DEVICES (storage + network only) ==="
|
||||
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
|
||||
directly into the chat.
|
||||
|
||||
### Option B: Command Prompt (cmd.exe) fallbacks
|
||||
|
||||
If PowerShell is unavailable for some reason, these cmd commands give a
|
||||
subset:
|
||||
|
||||
```cmd
|
||||
:: Computer model and serial
|
||||
wmic computersystem get manufacturer,model
|
||||
wmic bios get serialnumber
|
||||
|
||||
:: CPU
|
||||
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
|
||||
|
||||
:: RAM (total)
|
||||
wmic computersystem get totalphysicalmemory
|
||||
|
||||
:: RAM sticks (per-slot detail)
|
||||
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
|
||||
|
||||
:: Disks
|
||||
wmic diskdrive get model,size,interfacetype,mediatype
|
||||
|
||||
:: Network adapters
|
||||
wmic nic where netenabled=true get name,speed,macaddress
|
||||
|
||||
:: GPU
|
||||
wmic path win32_videocontroller get name,adapterram,driverversion
|
||||
```
|
||||
|
||||
### What I'm looking for
|
||||
|
||||
- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
|
||||
whether wnode-tsys2 can use local storage like the other wnodes.)
|
||||
- **Network adapters**: How many onboard NICs? Model? (Determines whether
|
||||
tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
|
||||
onboard NIC available.)
|
||||
- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
|
||||
memory bandwidth for HPC workloads — dual-channel matters.)
|
||||
- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
|
||||
planning.
|
||||
- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
|
||||
- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
|
||||
|
||||
### STATUS: Collected 2026-07-27
|
||||
|
||||
**Disk inventory — RESOLVED:**
|
||||
- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
|
||||
- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
|
||||
- Both SSDs, no spinning rust. 1.5 TB total local SSD.
|
||||
|
||||
**Network adapters — RESOLVED (concerning):**
|
||||
- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
|
||||
- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
|
||||
- Wi-Fi: Intel 8265 (disconnected)
|
||||
- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
|
||||
Unavoidable on this laptop form factor — no onboard PCIe NIC available.
|
||||
|
||||
**GPU — RESOLVED:**
|
||||
- Intel HD Graphics 630 (integrated, 1 GB)
|
||||
- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
|
||||
|
||||
**PCIe slots — RESOLVED:**
|
||||
- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
|
||||
user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
|
||||
by 960 PRO.
|
||||
|
||||
**RAM — PARTIAL:**
|
||||
- Total 32 GB confirmed (34,097,573,888 bytes).
|
||||
- Per-stick detail failed to run (PowerShell line-break split
|
||||
`Format-T` + `able`). Re-run the command below if bandwidth planning
|
||||
needs stick-level detail:
|
||||
```powershell
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
|
||||
|
||||
---
|
||||
|
||||
## 2. Friday maintenance window (physical hardware)
|
||||
|
||||
### tsys4 — install PCIe NIC + add RAM
|
||||
|
||||
1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
|
||||
2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
|
||||
3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
|
||||
4. Power on, then update `/etc/network/interfaces` to replace
|
||||
`enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
|
||||
5. Reboot to activate new NIC and NFS nconnect.
|
||||
6. Run `validate-fixes.sh pfv-tsys4` to confirm.
|
||||
|
||||
### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
|
||||
|
||||
1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
|
||||
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
|
||||
"Number of ports: 2" and a real partner MAC (not all zeros).
|
||||
3. Apply bond hash fix (same as tsys6/7):
|
||||
```bash
|
||||
echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
|
||||
```
|
||||
4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
|
||||
- Power down tsys4
|
||||
- Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
|
||||
- Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
|
||||
- On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
|
||||
- Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
|
||||
`pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
|
||||
- Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
|
||||
empty, so minimal migration)
|
||||
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
|
||||
6. Format NVMe as local directory storage (see TODO section 3 below).
|
||||
7. Reboot tsys5 to activate NFS nconnect.
|
||||
8. Run `validate-fixes.sh pfv-tsys5` to confirm.
|
||||
|
||||
---
|
||||
|
||||
## 3. tsys5 NVMe format/mount decision (after Friday install)
|
||||
|
||||
**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
|
||||
storage so it shows up as a VM image target in the Proxmox UI.
|
||||
|
||||
After the NVMe is physically installed and visible in Proxmox:
|
||||
|
||||
1. Identify the device: `lsblk` or `ls /dev/nvme*`
|
||||
2. Format and add to Proxmox:
|
||||
```bash
|
||||
# Option A: LVM-thin (thin provisioning, snapshots)
|
||||
pvcreate /dev/nvme0n1
|
||||
vgcreate nvme-pool /dev/nvme0n1
|
||||
lvcreate -l 100%FREE -T nvme-pool/data
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
|
||||
# ID: nvme-local
|
||||
# Volume Group: nvme-pool
|
||||
# Thin Pool: data
|
||||
# Content: Disk image, Container template
|
||||
|
||||
# Option B: Directory (simpler, no thin provisioning)
|
||||
mkfs.ext4 /dev/nvme0n1
|
||||
mkdir -p /mnt/nvme
|
||||
mount /dev/nvme0n1 /mnt/nvme
|
||||
# Add to /etc/fstab for persistence
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > Directory
|
||||
# ID: nvme-local
|
||||
# Directory: /mnt/nvme
|
||||
# Content: Disk image, Container template
|
||||
```
|
||||
|
||||
3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
|
||||
|
||||
---
|
||||
|
||||
## 4. Post-hardware validation (run after Friday work)
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Update PROJECT.md with post-hardware iperf numbers.
|
||||
|
||||
---
|
||||
|
||||
## 5. UCS storage migration to spinning disk (do today)
|
||||
|
||||
UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
|
||||
UCS VMs should stay on spinning disk (HDD) and be split across storage
|
||||
servers for redundancy.
|
||||
|
||||
| VM | Current | Target | Action |
|
||||
|----|---------|--------|--------|
|
||||
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
|
||||
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
|
||||
|
||||
**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
|
||||
1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
|
||||
2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
|
||||
(both are NFS exports visible to tsys9, so this is a storage-only migration)
|
||||
3. Verify VM 902 boots and LDAP/AD services are healthy after migration
|
||||
|
||||
Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
|
||||
ucs-02 needs to move -- it should be on a different storage server than
|
||||
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions for next session
|
||||
|
||||
- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
|
||||
Determines whether live migration is available.
|
||||
- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
|
||||
- Container runtime? (containerd, cri-o?)
|
||||
- Is there a local container image registry mirror?
|
||||
- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
|
||||
- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
|
||||
- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
|
||||
- tsys3 thermal state (laptop in rack for years) — check `sensors`.
|
||||
Reference in New Issue
Block a user