KNEL/PFVCluster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6814374bac67e889cbe5bd2673e25f1baaaad45d
PFVCluster/K8S.md
T
mrcharles 019ab468bf feat: relocate D3 SSD from tsys4 USB to tsys5 SAS on Friday 
Deep-dive on tsys4/5 storage controllers revealed:
- tsys4: 4-port SATA, all occupied; D3 SSD is USB 2.0
- tsys5: LSI SAS1068E (8-port, 5 free) + ICH10 SATA (2 free) + USB 3.0

Decision: move D3 (SK hynix SSD) to a tsys5 SAS port. Eliminates USB
bottleneck and consolidates all fast storage on tsys5: NVMe (local) +
D3 SSD + T5-SSD + 4 HDD exports.

Updated storage architecture, speed tiers, critical VM placement,
failure survival matrix, StorageClass design, and Friday plan.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
2026-07-27 11:50:16 -05:00

20 KiB
Raw Blame History

K8S.md -- Kubernetes Architecture Deep-Dive

Date: 2026-07-27 Purpose: Detailed kubernetes architecture plan for the pfv-k8s cluster. Companion to PROJECT.md (which has the fleet-wide assessment). Status: For discussion in a future session. No changes made.

Table of Contents

  1. Workload Profile
  2. Current State
  3. Target Architecture
  4. Control Plane (Cnodes)
  5. Worker Nodes (Wnodes)
  6. Storage Class Design
  7. ETL/HPC Considerations
  8. Migration Plan

1. Workload Profile

This cluster runs R&D and RackRental (containerlab) workloads via Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA running Cloudron.

Workload types expected:

Type Description Storage need RAM need Examples
ETL (weather/GIS) Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. High capacity (100s of GB), moderate IOPS Medium (8-32 GB per job) GRIB/NetCDF processing, raster reprojection
HPC (hardware startup) Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. Low capacity, moderate IOPS High (32-128 GB per job) RTL simulation, PCB thermal analysis
RackRental (containerlab) Rapid deployment/teardown of network lab topologies. Many containers, short-lived. Low capacity, high IOPS (container image pulls) Low-Medium (4-16 GB) Network topology testing, protocol validation

Key storage insight: ETL workloads need bulk capacity (NFS-HDD is fine -- sequential I/O). HPC and containerlab need low-latency random I/O (local SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.

2. Current State

2.1 pfv-k8s nodes and their storage

Node Type Host Storage Disk type Status
cnode1 (107) control tsys1 D5 (tsys4) NFS-HDD running
cnode2 (603) control tsys6 D2 (tsys4) NFS-HDD running
cnode3 (106) control tsys1 D2 (tsys4) NFS-HDD running
cnode4 (601) control tsys6 D2 (tsys4) NFS-HDD running
cnode5 (602) control tsys6 D5 (tsys4) NFS-HDD running
wnode-tsys3 (313) worker tsys3 D5 (tsys4) NFS-HDD running
wnode-tsys5 (509) worker tsys5 D2 (tsys4) NFS-HDD running
wnode-tsys6 (100) worker tsys6 D5 (tsys4) NFS-HDD STOPPED
wnode-tsys7 (701) worker tsys7 D5 (tsys4) NFS-HDD running
wnode-tsys9 (905) worker tsys9 S3 (tsys5) NFS-HDD running

2.2 Problems

  1. 100% of cnodes on tsys4 NFS. D2 disk failure loses 3 of 5 cnodes = etcd quorum lost.
  2. 90% of all k8s nodes on tsys4 NFS. tsys4 failure kills the cluster.
  3. Zero nodes use SSD or NVMe. All on NFS-over-HDD.
  4. Zero nodes use local-lvm. tsys3/6/7/9 all have empty local storage (349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
  5. wnode-tsys6 is stopped. Reduces cluster capacity.
  6. 3 cnodes on tsys6 -- should be on lighter hosts to free tsys6 for workers.

3. Target Architecture

3.1 Design principles

  1. Cnodes on lightweight hosts (tsys1, tsys9, tsys3) -- frees tsys6/7 for heavy workers.
  2. Cnode storage split across tsys4 and tsys5 -- etcd survives either storage server failing.
  3. Wnode boot disks on local storage -- eliminates NFS latency for container runtime and kubelet.
  4. Wnode data disks on NFS-HDD -- bulk capacity for ETL/weather/GIS.
  5. tsys5 NVMe dedicated to wnode-tsys5 -- fastest tier for HPC jobs.
  6. One wnode per hypervisor host -- maximize total cluster capacity.

3.2 Target node-host-storage matrix

Node Type Host Boot disk Data disk Disk type
cnode1 control tsys1 D5 (tsys4) -- NFS-HDD
cnode2 control tsys9 D2 (tsys4) -- NFS-HDD
cnode3 control tsys1 S2 (tsys5) -- NFS-HDD
cnode4 control tsys9 D5 (tsys4) -- NFS-HDD
cnode5 control tsys3 S3 (tsys5) -- NFS-HDD
wnode-tsys1 worker tsys1 D5 (tsys4) -- NFS-HDD (small)
wnode-tsys2 worker tsys2 NVMe (960 PRO 512GB) SATA SSD (850 EVO 1TB) NVMe + SSD -- no NFS needed
wnode-tsys3 worker tsys3 local-lvm (NVMe PM961) S3 (NFS) LOCAL-NVMe
wnode-tsys5 worker tsys5 NVMe (local, Friday) local-nonprod (HDD) NVMe
wnode-tsys6 worker tsys6 D2 (tsys4 NFS) -- NFS-HDD (local-lvm is USB 2.0 -- do not use)
wnode-tsys7 worker tsys7 D5 (tsys4 NFS) -- NFS-HDD (local-lvm is USB 2.0 -- do not use)
wnode-tsys9 worker tsys9 local-lvm (SSD) S2 (NFS) LOCAL-SSD

3.3 Storage server distribution after changes

Storage server cnodes wnodes (boot) wnodes (data)
tsys4 (D2) cnode2 wnode-tsys6 wnode-tsys7
tsys4 (D5) cnode1, cnode4 wnode-tsys1 --
tsys5 (S2) cnode3 wnode-tsys9 --
tsys5 (S3) cnode5 -- wnode-tsys3

Note: wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower than NFS-HDD, and the user has chosen not to install internal drives.

No single disk or server is a quorum-losing failure point.

4. Control Plane (Cnodes)

4.1 Cnode sizing

Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd + kubernetes control plane components (API server, scheduler, controller-manager).

4.2 Cnode host placement rationale

Host cnodes RAM for cnodes Total host RAM Remaining for other VMs
tsys1 2 (cnode1, cnode3) 8 GB 32 GB ~24 GB (but 11 infra VMs consume most)
tsys9 2 (cnode2, cnode4) 8 GB 24 GB ~16 GB (4 infra VMs + 1 wnode)
tsys3 1 (cnode5) 4 GB 32 GB ~28 GB (1 wnode at 20 GB = 8 GB headroom)

tsys6 and tsys7 have ZERO cnodes -- fully dedicated to heavy worker nodes.

4.3 Cnode storage placement rationale

The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:

Disk cnodes Rationale
D5 (tsys4 HDD) cnode1, cnode4 Spread load across 2 disks on tsys4
D2 (tsys4 HDD) cnode2 Only 1 cnode on D2 (was 3 -- reduces blast radius)
S2 (tsys5 HDD) cnode3 tsys5 storage for quorum diversity
S3 (tsys5 HDD) cnode5 tsys5 storage, different disk than S2

If D2 fails: cnode2 dies. 4 of 5 survive. Quorum OK. If D5 fails: cnode1 + cnode4 die. 3 of 5 survive. Quorum OK. If tsys4 fails: cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on tsys5. Only 2 of 5 -- QUORUM LOST.

Wait -- that is a problem. If tsys4 goes completely offline, we lose 3 cnodes and only have 2 on tsys5. That loses quorum (need 3).

Revision needed: Move 1 more cnode to tsys5 storage. Target: 2 on tsys4, 3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on tsys4. Same problem inverted.

The fundamental issue: with 5 cnodes and 2 storage servers, the best split is 3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server holding 2 cnodes is safe (3 survive).

Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the other. After Friday's hardware work:

  • tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
  • tsys5 will have bond0 fixed + NVMe -- more reliable

Either way, 3/2 split means one server failure could lose quorum. To truly solve this, use a 3rd storage target. Options:

  • Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd independent failure domain)
  • Use local-lvm on the cnode's own host (etcd data is local to the VM's host, no NFS dependency at all)

Best option: put cnode boot disks on local-lvm where available. This eliminates NFS entirely for the control plane. Each cnode's etcd data lives on its own host's local disk -- no shared dependency.

cnode Host Recommended storage Type
cnode1 tsys1 local-lvm (if space) or D5 (tsys4) LOCAL-HDD or NFS-HDD
cnode2 tsys9 local-lvm (SSD) LOCAL-SSD
cnode3 tsys1 S2 (tsys5) NFS-HDD
cnode4 tsys9 local-lvm (SSD) LOCAL-SSD
cnode5 tsys3 local-lvm LOCAL-HDD

With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always survives.

This is the recommended approach. Local storage for cnodes wherever possible. NFS only as fallback.

4.4 etcd performance on local vs NFS

Storage Typical fsync latency etcd commit latency Impact
NFS-HDD (via USB dongle on tsys4) 5-15 ms 10-30 ms Slow API responses, sluggish pod scheduling
NFS-HDD (via PCIe NIC, post-Friday) 2-8 ms 5-15 ms Better but still network-bound
Local HDD (tsys1/3/6/7 local-lvm) 1-5 ms 3-10 ms No network hop, moderate improvement
Local SSD (tsys9 PNY CS900) 0.1-0.5 ms 0.5-2 ms 10-30x faster than NFS-HDD
NVMe (tsys5, Friday) 0.02-0.1 ms 0.1-0.5 ms 100x faster than NFS-HDD

etcd is the heartbeat of the kubernetes control plane. Every API call, every pod schedule, every controller reconciliation involves an etcd write. Cutting etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more responsive. This is the single highest-impact change for k8s performance.

5. Worker Nodes (Wnodes)

5.1 One wnode per hypervisor host

Host wnode Boot disk Data disk Total RAM wnode RAM Role
tsys1 wnode-tsys1 D5 (tsys4 NFS) -- 32 GB 4-8 GB Small worker, infra co-tenant
tsys2 wnode-tsys2 NVMe (960 PRO 512GB) SATA SSD (850 EVO 1TB) 32 GB 16-24 GB Best storage of any worker -- 1.5TB local SSD, no NFS needed
tsys3 wnode-tsys3 local-lvm (349 GB) S3 (NFS) 32 GB 20 GB General worker
tsys5 wnode-tsys5 NVMe (local) local-nonprod (HDD) 96 GB 32-64 GB HPC/ETL powerhouse
tsys6 wnode-tsys6 D2 (tsys4 NFS) -- 128 GB 64-96 GB Heavy worker, max RAM. local-lvm is USB 2.0 -- stays on NFS
tsys7 wnode-tsys7 D5 (tsys4 NFS) -- 192 GB 96-128 GB Heavy worker, max RAM. local-lvm is USB 2.0 -- stays on NFS
tsys9 wnode-tsys9 local-lvm SSD (136 GB) S2 (NFS) 24 GB 4-8 GB Small worker, SSD boot

5.2 Why boot disks on local-lvm

Current: all wnodes boot from NFS. Every container image pull, every kubelet log write, every ephemeral volume traverses the NFS network path.

With local-lvm boot disks:

  • Container image pulls write to local disk (100-150 MB/s HDD, no network hop) instead of NFS-HDD (80-120 MB/s with network latency)
  • kubelet logs stay local (no NFS writes for log rotation)
  • ephemeral storage (emptyDir volumes) uses local disk by default
  • NFS server failure does not kill the wnode -- the VM stays running, only the data disk (if mounted) goes away

5.3 Wnode sizing guidance

Host Recommended wnode config Rationale
tsys7 (192 GB) 8-12 cores, 96-128 GB RAM, NFS boot Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0
tsys6 (128 GB) 8 cores, 64-96 GB RAM, NFS boot Second-largest -- parallel heavy jobs. local-lvm is USB 2.0
tsys5 (96 GB + NVMe) 4 cores, 32-64 GB RAM, NVMe boot + HDD data NVMe makes this fastest for I/O-bound HPC
tsys3 (32 GB) 4 cores, 20 GB RAM, local-lvm boot General-purpose worker
tsys2 (32 GB, NVMe+SSD, incoming) 4 cores, 16-24 GB RAM, NVMe boot + SSD data Fastest storage worker -- HPC with I/O bounds
tsys1 (32 GB) 2 cores, 4-8 GB RAM Small worker, don't starve infra VMs
tsys9 (24 GB) 2-4 cores, 4-8 GB RAM Small worker, SSD boot is the advantage

5.4 Tainting and labeling strategy

Label wnodes by capability so the k8s scheduler can target them:

# Heavy RAM hosts (ETL/HPC)
wnode-tsys6: workload=heavy, ram=128g
wnode-tsys7: workload=heavy, ram=192g

# NVMe host (I/O-intensive HPC)
wnode-tsys5: workload=hpc, storage=nvme

# SSD boot host (low-latency)
wnode-tsys9: workload=light, storage=ssd

# General workers
wnode-tsys3: workload=general
wnode-tsys2: workload=storage-fast, storage=nvme
wnode-tsys1: workload=light

Then use nodeSelector or nodeAffinity in job specs:

# Weather/GIS ETL job -- needs lots of RAM
spec:
  nodeSelector:
    workload: heavy

# Firmware build -- needs fast storage
spec:
  nodeSelector:
    storage: nvme

6. Storage Class Design

6.1 Proposed StorageClasses

StorageClass Provisioner Where Speed Use case
local-fast local-path (k8s) wnode local-lvm / NVMe 100-3500 MB/s Container runtime, scratch, databases
nfs-hdd nfs-subdir-external-provisioner tsys4 D2/D5, tsys5 S1-S4 80-120 MB/s Bulk data, weather/GIS datasets
nfs-ssd nfs-subdir-external-provisioner tsys4 D3, tsys5 T5-SSD 200-400 MB/s Latency-sensitive persistent data

6.2 How this maps to wnode disk topology

Each wnode has:

  • Disk 1 (boot/OS): local-lvm or NVMe. Contains the OS, kubelet, container runtime. k8s local-fast StorageClass provisioner points here.
  • Disk 2 (bulk data, optional): NFS mount. Mounted inside the VM as a second block device or filesystem. k8s nfs-hdd provisioner points here.

Inside k8s, pods request storage via PVC:

# ETL job: needs bulk storage for weather data
apiVersion: v1
kind: PersistentVolumeClaim
spec:
  storageClassName: nfs-hdd
  accessModes: [ReadWriteMany]  # NFS allows RWX
  resources:
    requests:
      storage: 500Gi

# HPC job: needs fast scratch
spec:
  storageClassName: local-fast
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi

6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)

The SSD NFS exports are a middle tier. Both consolidated on tsys5 after the Friday D3 relocation:

  • D3 (tsys5 SAS, 445 GB free): CI/CD artifacts, container image cache, UCS/LDAP. Currently 99% empty. Fastest NFS tier after the move off USB.
  • T5-SSD (tsys5 SAS, 140 GB free): SIEM databases, monitoring data. Currently 35% used by ultix-streaming.

tsys5 is the fast-tier hub: NVMe (local) + D3 SSD + T5-SSD all on one host. This simplifies the StorageClass design -- latency-sensitive PVCs target tsys5 exports, bulk PVCs target either server.

6.4 NFS data distribution across storage servers

To avoid re-creating the "everything on tsys4" problem, distribute NFS data disks across both servers:

wnode Boot (local) Bulk data (NFS) NFS server
wnode-tsys3 local-lvm S3 tsys5
wnode-tsys5 NVMe local-nonprod local (no NFS)
wnode-tsys6 D2 (tsys4 NFS) -- tsys4
wnode-tsys7 D5 (tsys4 NFS) -- tsys4
wnode-tsys9 local-lvm (SSD) S2 tsys5

This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.

7. ETL/HPC Considerations

7.1 Weather/GIS ETL pipeline

Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) -> store results.

Stage Storage class Why
Download raw data nfs-hdd Large sequential writes. NFS-HDD handles this well.
Processing scratch local-fast Random access during transform. Local disk avoids NFS latency.
Store results nfs-hdd Large sequential writes. Persistent.

Recommendation: Deploy a local-fast PV mount as /scratch on every wnode. ETL jobs use /scratch for intermediate processing and write final output to the NFS-mounted /data.

7.2 HPC workloads (hardware startup)

Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.

Workload Best wnode Why
RTL simulation (CPU-bound, high RAM) tsys7 (192 GB) Most RAM, most cores (24t)
Firmware builds (I/O-bound, moderate RAM) tsys5 (NVMe) Fastest storage for compile I/O
Hardware-in-the-loop (latency-sensitive) tsys9 (local SSD) Lowest latency storage
Parallel batch jobs tsys6 + tsys7 Distribute across both heavy hosts

7.3 RackRental/containerlab

Rapid container deployment. Key need: fast container image pulls.

This is where local-lvm boot disks shine. Currently, every container image pull writes through NFS to a spinning disk -- slow. With local-lvm, images cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O). On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.

7.4 Data locality for ETL

For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache it on local-lvm of the heavy hosts:

tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/  -- fastest cache tier
tsys5 NVMe (local): /data/cache/gis/                 -- fastest cache tier

Note: tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.

This avoids re-reading the same data from the same NFS export on every job if the data is already cached in the page cache.

8. Migration Plan

8.1 Phase 1: Friday (after hardware work)

After tsys5 cable + NVMe and tsys4 NIC + RAM:

  1. Format tsys5 NVMe as local directory storage (e.g., nvme-local)
  2. Restart wnode-tsys6 (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 -- do not use for VM storage. Recreate on D2 or D5 NFS.
  3. Move wnode-tsys9 (VM 905) disk from S3 (NFS) to local-lvm (SSD).

8.2 Phase 2: Cnode rebalance (maintenance window)

These changes require creating new VMs on target hosts and migrating disks. Plan for a maintenance window with the k8s cluster briefly down.

  1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
  2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
  3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
  4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
  5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.

8.3 Phase 3: Wnode local storage migration (maintenance window)

  1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
  2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
  3. wnode-tsys7 stays on NFS (same reason).
  4. Recreate wnode-tsys5 with boot disk on NVMe.
  5. Add data disks (NFS) as second SCSI devices where applicable.

8.4 Phase 4: tsys2 integration (when rebuilt)

  1. Install Proxmox on tsys2.
  2. Run scripts/check.sh to inventory.
  3. Run scripts/apply-tunings.sh --apply.
  4. Create wnode-tsys2 with boot disk on NVMe (960 PRO) and data disk on SATA SSD (850 EVO). No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
  5. Join to k8s cluster.

8.5 Phase 5: Critical VM relocation

  1. Move netinfra-02 (VM 904) from D2 to S3.
  2. Move ucs-01 (VM 108) from D2 to D3 (SSD).
  3. Move ucs-02 (VM 902) from D5 to tsys9 local-lvm (SSD).

Open questions for next session

  1. Are the hosts a Proxmox cluster (pvecm) or standalone? This determines whether live migration is available (huge simplification) or we need manual disk migration. Check pvecm status on each host.
  2. What k8s distribution is in use? (k3s, kubeadm, RKE2?) This affects how nodes are joined/drain and how StorageClasses are configured.
  3. Container runtime? (containerd, cri-o?) Affects local storage layout.
  4. Is there a container image registry mirror in the cluster? Or do all pulls go to Docker Hub / external? A local registry on D3 SSD would speed up all pulls.
  5. What specific ETL tools? (GDAL, PostGIS, xarray, Dask?) This affects whether jobs need shared (RWX) or exclusive (RWO) storage.
  6. HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?) Affects how we label and taint nodes.
Reference in New Issue View Git Blame Copy Permalink