refactor(docs): co-locate k8s/proxmox docs with their code + fix broken perf paths
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
+7
-7
@@ -12,9 +12,9 @@ procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
|
||||
|
||||
| Document | Description | Last Reviewed |
|
||||
|----------|-------------|---------------|
|
||||
| [`k8s/README.md`](k8s/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
|
||||
| [`k8s/DISTRO-DECISION.md`](k8s/DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: Talos, driven by ITAR/classified requirement | 2026-07-28 |
|
||||
| [`k8s/ARCHITECTURE.md`](k8s/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
|
||||
| [`k8s/docs/README.md`](k8s/docs/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
|
||||
| [`k8s/docs/DISTRO-DECISION.md`](k8s/docs/DISTRO-DECISION.md) | Talos vs k3s analysis. Decision: k3s for regular R&D cluster; Talos preserved for future ITAR/classified | 2026-07-28 |
|
||||
| [`k8s/docs/ARCHITECTURE.md`](k8s/docs/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
|
||||
|
||||
## Proxmox Cluster ([`proxmox/`](proxmox/))
|
||||
|
||||
@@ -22,10 +22,10 @@ Fleet operations, hardware, performance tuning, storage architecture.
|
||||
|
||||
| Document | Description | Last Reviewed |
|
||||
|----------|-------------|---------------|
|
||||
| [`proxmox/PROJECT.md`](proxmox/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/AUDIT-2026-07-28.md`](proxmox/AUDIT-2026-07-28.md) | Fresh fleet audit with current VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
|
||||
| [`proxmox/TODO.md`](proxmox/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
| [`proxmox/docs/PROJECT.md`](proxmox/docs/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/docs/AUDIT-2026-07-28.md`](proxmox/docs/AUDIT-2026-07-28.md) | Fresh fleet audit with current VM placements, storage redundancy analysis, pre-k8s action items | 2026-07-28 |
|
||||
| [`proxmox/docs/TODO.md`](proxmox/docs/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/docs/K8S.md`](proxmox/docs/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
|
||||
## Server Build ([`server-build/`](server-build/))
|
||||
|
||||
|
||||
@@ -1,673 +0,0 @@
|
||||
# pfv-k8s Target Architecture (Talos)
|
||||
|
||||
> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
|
||||
> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
|
||||
> Proxmox audit).
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [High-Level Architecture](#1-high-level-architecture)
|
||||
2. [Control Plane](#2-control-plane)
|
||||
3. [Network Topology](#3-network-topology)
|
||||
4. [CNI: Cilium](#4-cni-cilium)
|
||||
5. [Identity and Trust](#5-identity-and-trust)
|
||||
6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
|
||||
7. [Storage Integration](#7-storage-integration)
|
||||
8. [Local Image Registry](#8-local-image-registry)
|
||||
9. [Bootstrap Procedure](#9-bootstrap-procedure)
|
||||
10. [Disaster Recovery](#10-disaster-recovery)
|
||||
11. [Migration from Current State](#11-migration-from-current-state)
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph RESIDENCE["Residence — Proxmox LAN"]
|
||||
subgraph CP["Talos Control Plane (3 cnodes)"]
|
||||
C1[cnode1<br/>tsys9 · local-SSD]
|
||||
C2[cnode2<br/>tsys9 · local-SSD]
|
||||
C3[cnode3<br/>tsys1 · local-HDD]
|
||||
end
|
||||
subgraph WP["Talos Worker Plane"]
|
||||
W3[wnode-tsys3<br/>NVMe · 28GB]
|
||||
W5[wnode-tsys5<br/>NVMe · 32-64GB]
|
||||
W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
|
||||
W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
|
||||
W9[wnode-tsys9<br/>local-SSD · 4-8GB]
|
||||
end
|
||||
ETCD[(etcd<br/>raft, mTLS)]
|
||||
REG[(Harbor registry<br/>on D3 SSD · tsys5)]
|
||||
BASTION[tailscale-router VM<br/>subnet router]
|
||||
end
|
||||
|
||||
subgraph TAILNET["Tailscale overlay"]
|
||||
OP[Operator devices]
|
||||
end
|
||||
|
||||
subgraph CLOUDRON["Cloudron production — Reston VA"]
|
||||
KC[Keycloak OIDC IdP]
|
||||
end
|
||||
|
||||
C1 ---|mTLS LAN| ETCD
|
||||
C2 ---|mTLS LAN| ETCD
|
||||
C3 ---|mTLS LAN| ETCD
|
||||
CP -->|pull images| REG
|
||||
WP -->|pull images| REG
|
||||
|
||||
OP -->|Talos API :50000<br/>via subnet route| BASTION
|
||||
BASTION -.->|LAN| CP
|
||||
CP -->|OIDC| KC
|
||||
WP -->|OIDC| KC
|
||||
|
||||
classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
|
||||
classDef infra fill:#0f3460,stroke:#e94560,color:#fff
|
||||
classDef external fill:#16213e,stroke:#533483,color:#fff
|
||||
class CP,WP,ETCD talos
|
||||
class REG,BASTION infra
|
||||
class OP,KC,EXTERNAL external
|
||||
```
|
||||
|
||||
### Design principles
|
||||
|
||||
1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
|
||||
Strongest posture for ITAR/classified.
|
||||
2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
|
||||
advertises the cluster LAN subnet. Operator reaches Talos API from
|
||||
anywhere.
|
||||
3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
|
||||
for etcd). Workers boot from local disk where available; NFS for bulk
|
||||
data only.
|
||||
4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
|
||||
of the Talos host cluster.
|
||||
5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
|
||||
tokens. No long-lived static credentials for humans.
|
||||
|
||||
---
|
||||
|
||||
## 2. Control Plane
|
||||
|
||||
### 2.1 Recommendation: 3 cnodes (down from 5)
|
||||
|
||||
| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
|
||||
|--------|--------|-------------------|-----------------|---------------|
|
||||
| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
|
||||
| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
|
||||
|
||||
For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
|
||||
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
|
||||
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
|
||||
host capacity) are better spent on tenant worker allocations.
|
||||
|
||||
**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
|
||||
tolerance on the control plane, keep 5. Otherwise 3.
|
||||
|
||||
### 2.2 Cnode placement
|
||||
|
||||
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
|
||||
**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
|
||||
|
||||
| cnode | Host | Boot disk | Type | Why |
|
||||
|-------|------|-----------|------|-----|
|
||||
| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
|
||||
| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
|
||||
| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
|
||||
|
||||
**Quorum survival:**
|
||||
|
||||
| Failure | cnodes lost | Quorum OK? |
|
||||
|---------|-------------|------------|
|
||||
| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
|
||||
| tsys1 host dies | cnode3 | YES (2 of 3) |
|
||||
| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
|
||||
|
||||
**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
|
||||
loses quorum. Alternative: spread cnodes across 3 different hosts. See
|
||||
"open question" at end of this section.
|
||||
|
||||
### 2.3 Machine config strategy
|
||||
|
||||
Talos nodes are configured by **machine configs** (YAML). Two flavors:
|
||||
|
||||
- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
|
||||
controller-manager, API server.
|
||||
- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
|
||||
|
||||
Strategy for this cluster:
|
||||
|
||||
1. **One shared `talosconfig`** (client identity) — stored in 1Password
|
||||
and in the Proxmox Backup Server (PBS) encrypted backup target.
|
||||
2. **Per-node machine config patches** — small patches on top of the base
|
||||
`controlplane.yaml` / `worker.yaml` for node-specific settings:
|
||||
- Hostname
|
||||
- Network interface + IP (DHCP or static — recommend static for cnodes)
|
||||
- Schematic image digest (pinned Talos version)
|
||||
- System extensions (e.g., `tailscale` — only if running Pattern A
|
||||
instead of recommended Pattern C)
|
||||
3. **All machine configs in Git** under a future `k8s/talos-configs/`
|
||||
directory. Secrets are templated in at apply-time from 1Password / sops.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
|
||||
BASE --> PATCH2[patch: cnode2]
|
||||
BASE --> PATCH3[patch: cnode3]
|
||||
BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
|
||||
PATCH1 --> APPLY1[talosctl apply]
|
||||
PATCH2 --> APPLY2[talosctl apply]
|
||||
PATCH3 --> APPLY3[talosctl apply]
|
||||
PATCHW --> APPLYW[talosctl apply]
|
||||
```
|
||||
|
||||
### 2.4 Open question: cnode host spread
|
||||
|
||||
If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
|
||||
cluster, tsys9 is brand-new hardware, single digit annual failure
|
||||
probability), the layout in §2.2 is fine.
|
||||
|
||||
If not, alternative spread across 3 hosts:
|
||||
|
||||
| cnode | Host | Boot disk |
|
||||
|-------|------|-----------|
|
||||
| cnode1 | tsys9 | local-lvm SSD |
|
||||
| cnode2 | tsys1 | local-lvm HDD |
|
||||
| cnode3 | tsys3 | local-lvm NVMe |
|
||||
|
||||
tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
|
||||
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
|
||||
would make an excellent etcd disk.
|
||||
|
||||
**This 3-host spread survives any single host failure with quorum intact.
|
||||
Recommended.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Network Topology
|
||||
|
||||
### 3.1 Zones
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph INTERNET["Internet"]
|
||||
FIBER[Gigabit symmetric fiber]
|
||||
end
|
||||
|
||||
subgraph RESLAN["Residence LAN 192.168.x.x/24"]
|
||||
subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
|
||||
CNODES[Cnodes 192.168.3.x]
|
||||
WNODES[Wnodes 192.168.3.x]
|
||||
end
|
||||
BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
|
||||
REG[Harbor registry<br/>192.168.3.x]
|
||||
STORAGE[NFS servers<br/>tsys4, tsys5]
|
||||
end
|
||||
|
||||
subgraph TSNET["Tailscale 100.x.x.x/8"]
|
||||
OPS[Operator devices]
|
||||
KC[Keycloak<br/>via Cloudron prod]
|
||||
end
|
||||
|
||||
FIBER --> BASTION
|
||||
BASTION <-. subnet route .-> CLUSTERNET
|
||||
OPS -->|TCP 50000 talos API| BASTION
|
||||
BASTION -->|LAN forward| CNODES
|
||||
CNODES -->|LAN mTLS| WNODES
|
||||
CNODES -->|OIDC HTTPS| KC
|
||||
WNODES -->|pull images| REG
|
||||
WNODES -->|bulk data IO| STORAGE
|
||||
CNODES -->|pull images| REG
|
||||
```
|
||||
|
||||
### 3.2 Address plan (suggested)
|
||||
|
||||
Reserve a small contiguous block in the residence LAN for cluster nodes:
|
||||
|
||||
| Role | Range | Count |
|
||||
|------|-------|-------|
|
||||
| Cnodes | `192.168.3.31-33` | 3 |
|
||||
| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
|
||||
| Bastion | existing `tailscale-router` | 1 |
|
||||
| Registry | `192.168.3.50` | 1 (Harbor) |
|
||||
|
||||
**Static IPs are strongly recommended for cnodes** (etcd cluster membership
|
||||
is hostname-based; stable IPs make `talosctl` targeting simple). Workers
|
||||
can DHCP.
|
||||
|
||||
### 3.3 Firewall posture
|
||||
|
||||
Each cnode/wnode has:
|
||||
|
||||
- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
|
||||
cnodes only), plus CNI ports (varies by CNI — see §4).
|
||||
- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
|
||||
- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
|
||||
firewall for these IPs. ITAR workloads must not be able to phone home.
|
||||
|
||||
The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
|
||||
via the subnet route.
|
||||
|
||||
---
|
||||
|
||||
## 4. CNI: Cilium
|
||||
|
||||
**Recommendation: Cilium** (eBPF-based CNI).
|
||||
|
||||
| Property | Why it matters here |
|
||||
|----------|---------------------|
|
||||
| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
|
||||
| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
|
||||
| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
|
||||
| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
|
||||
| **Talos integration** | First-class. Talos docs document the install path. |
|
||||
|
||||
Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
|
||||
enabled. Default-deny NetworkPolicy applied per namespace.
|
||||
|
||||
---
|
||||
|
||||
## 5. Identity and Trust
|
||||
|
||||
### 5.1 Trust flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Human as Operator
|
||||
participant TAIL as Tailscale
|
||||
participant BAST as Bastion
|
||||
participant TALOS as Talos API :50000
|
||||
participant KC as Keycloak (Cloudron)
|
||||
participant K8S as Kubernetes API :6443
|
||||
|
||||
Human->>TAIL: Authenticate (device + SSO)
|
||||
TAIL-->>Human: Tailnet IP
|
||||
Human->>BAST: Reach bastion via tailnet
|
||||
BAST->>TALOS: Forward to LAN node :50000
|
||||
Human->>KC: OIDC login (browser)
|
||||
KC-->>Human: Bearer token (short-lived)
|
||||
Human->>TALOS: talosctl (mTLS with client cert)
|
||||
Human->>K8S: kubectl --oidc (Keycloak token)
|
||||
K8S->>KC: Validate token (introspection)
|
||||
KC-->>K8S: Valid + claims
|
||||
K8S-->>Human: Authorized response
|
||||
```
|
||||
|
||||
### 5.2 Two distinct identity layers
|
||||
|
||||
| Layer | Mechanism | Audience |
|
||||
|-------|-----------|----------|
|
||||
| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
|
||||
| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
|
||||
|
||||
The **Talos secrets bundle** is the root of trust for the cluster. Lose it
|
||||
and you cannot operate the cluster; an attacker with it owns the cluster.
|
||||
Storage:
|
||||
|
||||
1. **Primary:** 1Password (or equivalent) — operator-accessible.
|
||||
2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
|
||||
3. **NOT in Git.** Machine configs go in Git; secrets stay out.
|
||||
|
||||
### 5.3 Keycloak client configuration
|
||||
|
||||
On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
|
||||
|
||||
- **Authorization Code + PKCE flow** (no implicit, no password).
|
||||
- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
|
||||
Omni/Rancher URLs if/when those are added.
|
||||
- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
|
||||
`k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tenant Isolation (vcluster)
|
||||
|
||||
### 6.1 Why vcluster
|
||||
|
||||
[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
|
||||
plane** (API server, scheduler, controller-manager, etcd) inside a namespace
|
||||
of the host cluster. Tenant workloads run on the host's worker nodes but
|
||||
are isolated by:
|
||||
|
||||
- Separate API server (tenant cannot see host cluster objects).
|
||||
- Separate RBAC and admission control.
|
||||
- Separate network policies (per-namespace).
|
||||
- Separate resource quotas.
|
||||
|
||||
This aligns with the user's per-tenant plan from
|
||||
[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
|
||||
|
||||
### 6.2 Tenant registry
|
||||
|
||||
| Tenant | Compliance | Workload example | vcluster name |
|
||||
|--------|-----------|------------------|---------------|
|
||||
| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
|
||||
| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
|
||||
| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
|
||||
| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph HOST["Talos host cluster"]
|
||||
CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
|
||||
subgraph NS["Host cluster namespaces"]
|
||||
NS_RR[ns: vc-rackrental]
|
||||
NS_SO[ns: vc-suborbital-open]
|
||||
NS_SI[ns: vc-suborbital-itar]
|
||||
NS_SLP[ns: vc-slp]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph VRR["vcluster: vc-rackrental"]
|
||||
API_RR[k8s API + etcd]
|
||||
end
|
||||
subgraph VSO["vcluster: vc-suborbital-open"]
|
||||
API_SO[k8s API + etcd]
|
||||
end
|
||||
subgraph VSI["vcluster: vc-suborbital-itar"]
|
||||
API_SI[k8s API + etcd]
|
||||
end
|
||||
subgraph VSLP["vcluster: vc-slp"]
|
||||
API_SLP[k8s API + etcd]
|
||||
end
|
||||
|
||||
CP --> NS_RR & NS_SO & NS_SI & NS_SLP
|
||||
NS_RR --> API_RR
|
||||
NS_SO --> API_SO
|
||||
NS_SI --> API_SI
|
||||
NS_SLP --> API_SLP
|
||||
|
||||
classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
|
||||
class NS_SI,API_SI itar
|
||||
```
|
||||
|
||||
### 6.3 ITAR enforcement at host layer
|
||||
|
||||
For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
|
||||
controls:
|
||||
|
||||
- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
|
||||
ITAR workloads (subset of wnodes, marked in node labels).
|
||||
- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
|
||||
namespace. Allow only explicit destinations (registry, NFS for ITAR
|
||||
data tier, Keycloak).
|
||||
- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
|
||||
`D3-itar` on tsys5) that no other tenant can mount.
|
||||
- **Audit:** Hubble flows + auditd on the host worker nodes capture all
|
||||
access to ITAR data.
|
||||
|
||||
Rancher (or Sidero Omni) sits above this, presenting each tenant's
|
||||
vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
|
||||
access per tenant group claim.
|
||||
|
||||
---
|
||||
|
||||
## 7. Storage Integration
|
||||
|
||||
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
|
||||
|
||||
| StorageClass | Provisioner | Backing | Speed | Use |
|
||||
|--------------|------------|---------|-------|-----|
|
||||
| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
|
||||
| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
|
||||
| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
|
||||
|
||||
### 7.1 CSI driver notes
|
||||
|
||||
- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
|
||||
(CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
|
||||
NFS server + base export path.
|
||||
- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
|
||||
with one manifest. Uses wnode's kubelet root dir.
|
||||
|
||||
### 7.2 ITAR data isolation
|
||||
|
||||
The ITAR tenant should target a dedicated NFS export, not shared
|
||||
`nfs-hdd`. Recommended:
|
||||
|
||||
- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
|
||||
`nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
|
||||
namespace via RBAC + NetworkPolicy.
|
||||
|
||||
---
|
||||
|
||||
## 8. Local Image Registry
|
||||
|
||||
### 8.1 Recommendation: Harbor on D3 SSD
|
||||
|
||||
D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
|
||||
for a **Harbor** instance:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
|
||||
| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
|
||||
| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
|
||||
| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
|
||||
|
||||
### 8.2 Pull-through cache benefit
|
||||
|
||||
Cluster nodes have zero internet egress (per §3.3). Without a local cache,
|
||||
image pulls fail. With Harbor as a pull-through cache:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
WNODE->>HARBOR: docker pull nginx:1.25
|
||||
alt cache hit
|
||||
HARBOR-->>WNODE: layer bytes (LAN-speed)
|
||||
else cache miss
|
||||
HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
|
||||
DOCKERHUB-->>HARBOR: layer bytes
|
||||
HARBOR-->>WNODE: layer bytes (cached for next time)
|
||||
end
|
||||
```
|
||||
|
||||
Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
|
||||
machine in the cluster with container-registry internet egress, and that
|
||||
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
|
||||
ghcr.io).
|
||||
|
||||
### 8.3 Supply-chain integrity (future)
|
||||
|
||||
Harbor + Cosign lets you require that all images deployed to the ITAR
|
||||
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
|
||||
control. Implementation deferred to a later session.
|
||||
|
||||
---
|
||||
|
||||
## 9. Bootstrap Procedure
|
||||
|
||||
### 9.1 One-time setup
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant GIT as Git repo
|
||||
participant ONEPW as 1Password
|
||||
participant PBS as PBS (tsys4)
|
||||
OP->>GIT: Clone PFVCluster repo
|
||||
OP->>ONEPW: Generate Talos secrets bundle (offline)
|
||||
ONEPW-->>OP: secrets.yaml
|
||||
OP->>PBS: Backup secrets.yaml (encrypted)
|
||||
OP->>GIT: Write machine configs (no secrets)
|
||||
```
|
||||
|
||||
### 9.2 Provision first cnode (bootstrap)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant PX as Proxmox host
|
||||
participant C1 as cnode1
|
||||
participant ETCD as etcd (new)
|
||||
|
||||
OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
|
||||
OP->>PX: qm start VMID
|
||||
C1->>C1: Boots Talos (no config yet, "maintenance mode")
|
||||
OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
|
||||
C1->>C1: Applies config, restarts services
|
||||
OP->>C1: talosctl bootstrap
|
||||
C1->>ETCD: Initialize single-node raft
|
||||
ETCD-->>C1: ready
|
||||
OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
|
||||
OP->>C1: talosctl etcd snapshot (initial backup → PBS)
|
||||
```
|
||||
|
||||
### 9.3 Add second and third cnodes
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant PX as Proxmox host
|
||||
participant C2 as cnode2
|
||||
participant C3 as cnode3
|
||||
participant C1 as cnode1 (existing)
|
||||
|
||||
OP->>PX: qm create + start cnode2 VM
|
||||
C2->>C2: Boots Talos maintenance mode
|
||||
OP->>C2: talosctl apply --patch cnode2.yaml
|
||||
C2->>C1: Join etcd cluster
|
||||
OP->>PX: qm create + start cnode3 VM
|
||||
C3->>C3: Boots Talos maintenance mode
|
||||
OP->>C3: talosctl apply --patch cnode3.yaml
|
||||
C3->>C1: Join etcd cluster
|
||||
Note over C1,C3: etcd now has 3/3 members → HA quorum
|
||||
```
|
||||
|
||||
### 9.4 Post-bootstrap cluster configuration
|
||||
|
||||
Once 3 cnodes are up and joined:
|
||||
|
||||
1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
|
||||
2. **Install CSI drivers** — nfs-csi + local-path provisioner.
|
||||
3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
|
||||
4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
|
||||
5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
|
||||
6. **Apply default-deny NetworkPolicy** in all namespaces.
|
||||
7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
|
||||
8. **First etcd snapshot** + automated daily snapshot cron → PBS.
|
||||
|
||||
### 9.5 Add workers
|
||||
|
||||
Workers are simpler (no etcd):
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
|
||||
WNODE->>WNODE: Boots maintenance mode
|
||||
OP->>WNODE: talosctl apply --patch worker-X.yaml
|
||||
WNODE->>C1: Kubelet registers with API server
|
||||
C1-->>WNODE: Approved (auto via bootstrap token)
|
||||
Note over WNODE: Joins cluster, becomes Ready
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Disaster Recovery
|
||||
|
||||
### 10.1 Backup strategy
|
||||
|
||||
| Artifact | Frequency | Storage | Tool |
|
||||
|----------|-----------|---------|------|
|
||||
| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
|
||||
| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
|
||||
| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
|
||||
| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
|
||||
| **Harbor metadata** | Daily | PBS | Harbor built-in backup |
|
||||
|
||||
### 10.2 Restore scenarios
|
||||
|
||||
**Lost 1 cnode (e.g., tsys9 disk failure):**
|
||||
|
||||
1. Provision new VM on tsys9 (or other host with local SSD).
|
||||
2. Apply cnode2 machine config patch.
|
||||
3. New cnode joins etcd, syncs state from survivors.
|
||||
4. Quorum was never lost (2 of 3 alive throughout).
|
||||
|
||||
**Lost 2 cnodes simultaneously (quorum lost):**
|
||||
|
||||
1. Use surviving cnode's etcd snapshot.
|
||||
2. Provision 3 new cnode VMs.
|
||||
3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
|
||||
4. Join other 2 cnodes.
|
||||
5. Workers reconnect automatically once API server is back.
|
||||
|
||||
**Total cluster loss (all 3 cnodes):**
|
||||
|
||||
1. Restore from latest PBS etcd snapshot.
|
||||
2. Provision new cnode VMs.
|
||||
3. `talosctl bootstrap --recover-from=snapshot.db`.
|
||||
4. Re-join workers.
|
||||
5. Verify tenant vclusters restored.
|
||||
|
||||
### 10.3 Recovery time objectives
|
||||
|
||||
| Scenario | RTO | RPO |
|
||||
|----------|-----|-----|
|
||||
| Single cnode failure | < 30 min | 0 (no data loss) |
|
||||
| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
|
||||
| Total cluster loss | < 4 hours | ≤ 24 hours |
|
||||
|
||||
---
|
||||
|
||||
## 11. Migration from Current State
|
||||
|
||||
### 11.1 Current state
|
||||
|
||||
- 5 cnode VMs exist (Debian stock + Tailscale).
|
||||
- **No k3s deployed yet.** Cluster was never bootstrapped.
|
||||
- 6 wnode VMs exist (some stopped).
|
||||
- No workloads running in k8s.
|
||||
|
||||
### 11.2 Migration: clean cutover (not a migration)
|
||||
|
||||
Since there is no etcd data and no workloads to preserve, the path is a
|
||||
**clean rebuild**:
|
||||
|
||||
| Phase | Action | Risk |
|
||||
|-------|--------|------|
|
||||
| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
|
||||
| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
|
||||
| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
|
||||
| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
|
||||
| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
|
||||
| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
|
||||
|
||||
### 11.3 Open dependency: Friday hardware work
|
||||
|
||||
Phases 1-2 require:
|
||||
|
||||
- **tsys3 local-lvm available.** Per
|
||||
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
|
||||
free NVMe local-lvm. Currently unused. **Ready.**
|
||||
- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
|
||||
- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
|
||||
[`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
|
||||
Harbor depends on D3 being available on tsys5.
|
||||
|
||||
Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
|
||||
Harbor registry does.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Open questions for next session
|
||||
|
||||
1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
|
||||
2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
|
||||
(tsys9, tsys1, tsys3).
|
||||
3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
|
||||
4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
|
||||
Defer until cluster is up.
|
||||
5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
|
||||
approval of `192.168.3.0/24` route advertisement.
|
||||
6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
|
||||
Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
|
||||
general capacity; tsys3 + tsys5 (local fast storage) reserved for
|
||||
non-ITAR HPC.
|
||||
@@ -1,188 +0,0 @@
|
||||
# Distro Decision: Talos Linux vs k3s
|
||||
|
||||
> **Recommendation: Talos Linux.**
|
||||
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
|
||||
> entered scope. Once classified workloads are on the table, Talos's
|
||||
> immutable, API-only, measured-boot-capable posture is materially easier
|
||||
> to certify and defend.
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision context
|
||||
|
||||
| Factor | Constraint |
|
||||
|--------|-----------|
|
||||
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
|
||||
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
|
||||
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
|
||||
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
|
||||
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
|
||||
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Head-to-head comparison
|
||||
|
||||
### 2.1 ITAR / classified posture
|
||||
|
||||
| Property | Talos Linux | k3s on Debian |
|
||||
|----------|-------------|---------------|
|
||||
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
|
||||
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
|
||||
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
|
||||
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
|
||||
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
|
||||
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
|
||||
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
|
||||
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
|
||||
|
||||
**Bottom line:** For classified workloads, an auditor's first question is
|
||||
"how do you prevent unauthorized changes to a node?" Talos's answer is
|
||||
"the OS is immutable and the only path is a signed API call." k3s's answer
|
||||
is"SSH is locked down and we scan with STIG." The first is structurally
|
||||
stronger; the second is operationally maintained.
|
||||
|
||||
### 2.2 Operational considerations
|
||||
|
||||
| Property | Talos | k3s |
|
||||
|----------|-------|-----|
|
||||
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
|
||||
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
|
||||
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
|
||||
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
|
||||
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
|
||||
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
|
||||
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
|
||||
|
||||
### 2.3 Cost of choosing Talos over the existing k3s plan
|
||||
|
||||
The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
|
||||
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
|
||||
|
||||
- **No etcd data to migrate.** Clean cutover, not a migration.
|
||||
- **No workloads to drain.** The cluster is empty.
|
||||
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
|
||||
either way it's a `qm` script, not a stateful migration).
|
||||
- **Tailscale config shifts** from "installed via apt" to "Talos system
|
||||
extension." (Or, per our recommendation in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
|
||||
nodes entirely** and onto the existing `tailscale-router` bastion as a
|
||||
subnet router. Cluster nodes become LAN-only.)
|
||||
|
||||
**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
|
||||
machine config YAML. The hardening investment already encoded in
|
||||
`provisioning/Modules/Security/` is **not wasted** — it still applies to
|
||||
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
|
||||
the cnodes/wnodes move to Talos.
|
||||
|
||||
---
|
||||
|
||||
## 3. Tailscale compatibility (deep-dive)
|
||||
|
||||
Tailscale on Talos is well-supported but introduces a configuration dimension
|
||||
worth being explicit about. Three patterns exist:
|
||||
|
||||
### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
|
||||
|
||||
Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
|
||||
internet egress (to Tailscale DERP servers and for coordinate).
|
||||
|
||||
- **Talos implementation:** add `siderolabs/tailscale` system extension to
|
||||
each machine config, configure `machine.network.interfaces`.
|
||||
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
|
||||
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
|
||||
finding (data exfiltration path).
|
||||
|
||||
### Pattern B — Tailscale on bastion only, SSH/API jump
|
||||
|
||||
Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
|
||||
`tailscale-router` VM), then runs `talosctl` from the bastion.
|
||||
|
||||
- **Pros:** Zero internet egress from cluster nodes.
|
||||
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
|
||||
action originates from the bastion (auditable but clunky).
|
||||
|
||||
### Pattern C — Tailscale subnet router on bastion (recommended)
|
||||
|
||||
The existing `tailscale-router` VM advertises the cluster LAN subnet
|
||||
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
|
||||
Tailscale client transparently routes cluster-bound traffic through the
|
||||
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
|
||||
"just works."
|
||||
|
||||
- **Pros:**
|
||||
- Cluster nodes have **zero internet egress** (strongest ITAR posture).
|
||||
- Operator UX is unchanged from direct LAN access.
|
||||
- All access is mediated by Tailscale's identity + ACLs (already integrated
|
||||
with your env).
|
||||
- Audit trail lives in Tailscale + bastion logs.
|
||||
- **Cons:**
|
||||
- Bastion becomes a dependency for remote admin (LAN-local admin still
|
||||
works without it).
|
||||
- Must enable IP forwarding + subnet route approval in Tailscale ACLs.
|
||||
|
||||
**Recommendation: Pattern C.** Documented in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
|
||||
|
||||
### Justification
|
||||
|
||||
1. **Compliance posture is structural, not operational.** "Immutable,
|
||||
API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
|
||||
is a property of how Debian is operated. The first is dramatically
|
||||
easier to argue to an ITAR counsel or classified accreditation officer
|
||||
(DSS, DCSA) than the second.
|
||||
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
|
||||
3 cnodes with Talos is a `qm` script invocation, not a stateful
|
||||
migration. The sunk cost of "we planned k3s" is **zero deployed state**.
|
||||
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
|
||||
`apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
|
||||
learn but more error-prone to operate at HA.
|
||||
|
||||
### Acknowledged tradeoffs
|
||||
|
||||
- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
|
||||
replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
|
||||
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
|
||||
in. Mitigation: `talosctl logs/support` produces a support bundle
|
||||
equivalent to a sosreport.
|
||||
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
|
||||
beyond virtio + common NIC drivers needs a system extension. On Proxmox
|
||||
VMs this is **not expected to be a problem** — virtio is the path.
|
||||
- **Tailscale via system extension.** Adds one config dimension per node.
|
||||
Mitigated by Pattern C (above), which removes Tailscale from cluster
|
||||
nodes entirely.
|
||||
|
||||
### What we keep from the k3s mental model
|
||||
|
||||
- **Single binary on each node** semantics (Talos is conceptually similar).
|
||||
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
|
||||
API. `kubectl`, `helm`, `kustomize` all work as-is.
|
||||
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
|
||||
distro-independent.
|
||||
|
||||
---
|
||||
|
||||
## 5. What we are NOT deciding here
|
||||
|
||||
| Topic | Deferred to |
|
||||
|--------|-------------|
|
||||
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
|
||||
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
|
||||
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
|
||||
| Solar-aware scale-out hosts | Future capacity planning session |
|
||||
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Next step
|
||||
|
||||
Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
|
||||
design, network topology, identity flow, and bootstrap procedure.
|
||||
@@ -1,44 +0,0 @@
|
||||
# Kubernetes Architecture & Build Plan
|
||||
|
||||
> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
|
||||
> (which captured the storage/host analysis from the Proxmox audit).
|
||||
> This directory takes the next step: **which distro, how to build it,
|
||||
> how to operate it.**
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
## Documents in this directory
|
||||
|
||||
| Document | Purpose |
|
||||
|----------|---------|
|
||||
| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
|
||||
| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Decision | Recommendation | Why |
|
||||
|----------|----------------|-----|
|
||||
| **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. |
|
||||
| **Runtime** | **containerd** | Talos/k3s default. |
|
||||
| **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. |
|
||||
| **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. |
|
||||
| **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. |
|
||||
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. |
|
||||
| **Multi-tenancy** | **vcluster** (per tenant) | Future work. |
|
||||
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. |
|
||||
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. |
|
||||
|
||||
## What this directory does NOT cover (deferred)
|
||||
|
||||
- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
|
||||
- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
|
||||
- vcluster per-tenant policy templates.
|
||||
- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
|
||||
|
||||
These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
|
||||
|
||||
## Open question for the user
|
||||
|
||||
1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
|
||||
2-failure tolerance on the control plane, keep 5.
|
||||
@@ -1,271 +0,0 @@
|
||||
# Fresh Fleet Audit — 2026-07-28 (revised)
|
||||
|
||||
> **Supersedes placement data in [`PROJECT.md`](PROJECT.md) §4-§8.**
|
||||
|
||||
**Audit time:** 2026-07-28 20:30 CDT
|
||||
**Hosts audited:** pfv-tsys1, pfv-tsys3, pfv-tsys4, pfv-tsys5, pfv-tsys6, pfv-tsys7, pfv-tsys9
|
||||
**Hosts offline:** pfv-tsys2 (Win10, pending rebuild), pfv-tsys8 (offline 5+ days)
|
||||
|
||||
---
|
||||
|
||||
## 1. HA Pairs — FIXED ✓
|
||||
|
||||
Both infrastructure pairs now split across storage servers:
|
||||
|
||||
| VM | VMID | Host | Disk | Server | tsys4 dies | tsys5 dies |
|
||||
|----|------|------|------|--------|-----------|-----------|
|
||||
| netinfra-01 | 103 | tsys1 | S2 | tsys5 | alive | dies |
|
||||
| netinfra-02 | 904 | tsys9 | D2 | tsys4 | dies | alive |
|
||||
| ucs-01 | 108 | tsys1 | D2 | tsys4 | dies | alive |
|
||||
| ucs-02 | 902 | tsys9 | S2 | tsys5 | alive | dies |
|
||||
|
||||
**Any single storage server failure leaves one half of each pair alive.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Active k3s Cnodes — fundamental constraint (3 nodes, 2 NFS servers)
|
||||
|
||||
| Cnode | VMID | Host | Disk | Server |
|
||||
|-------|------|------|------|--------|
|
||||
| cnode1 | 906 | tsys9 | S3 | **tsys5** |
|
||||
| cnode2 | 705 | tsys7 | D2 | **tsys4** |
|
||||
| cnode3 | 106 | tsys1 | D5 | **tsys4** |
|
||||
|
||||
**tsys4 failure:** cnode2 + cnode3 die → 1 of 3 → **quorum lost.**
|
||||
**tsys5 failure:** cnode1 dies → 2 of 3 → quorum OK.
|
||||
|
||||
This is a structural constraint: 3 cnodes on 2 NFS servers means one side
|
||||
holds 2. The only true fixes are:
|
||||
- A 3rd storage target (not available until Friday hardware work)
|
||||
- Local-lvm on a cnode's host (deferred to Friday — user wants NFS-only for now)
|
||||
- Accept the risk (tsys4 reliability improves after Friday PCIe NIC + 64 GB RAM)
|
||||
|
||||
**Decision: accept for now. Revisit after Friday when local-SSD boot disks
|
||||
become an option for cnodes.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Worker Nodes — NEEDS REBALANCING (2 storage migrations)
|
||||
|
||||
### Current state (problem: 3 wnodes share one physical disk)
|
||||
|
||||
| Wnode | VMID | Host | Disk | Server | Disk shared with |
|
||||
|-------|------|------|------|--------|-----------------|
|
||||
| wnode-tsys1 | 102 | tsys1 | S3 | tsys5 | — |
|
||||
| wnode-tsys3 | 313 | tsys3 | **D5** | tsys4 | wnode-tsys6, wnode-tsys7 |
|
||||
| wnode-tsys5 | 509 | tsys5 | D2 | tsys4 | — |
|
||||
| wnode-tsys6 | 100 | tsys6 | **D5** | tsys4 | wnode-tsys3, wnode-tsys7 |
|
||||
| wnode-tsys7 | 701 | tsys7 | **D5** | tsys4 | wnode-tsys3, wnode-tsys6 |
|
||||
| wnode-tsys9 | 905 | tsys9 | S2 | tsys5 | — |
|
||||
|
||||
**D5 (Hitachi 2TB HDD) has 3 wnodes contending for IOPS.**
|
||||
**S1 and S4 on tsys5 are completely empty (916 GB + 499 GB free).**
|
||||
|
||||
### Recommended moves (2 PDM storage migrations)
|
||||
|
||||
| Wnode | Disk from | Disk to | Effect |
|
||||
|-------|-----------|---------|--------|
|
||||
| wnode-tsys6 (100) | D5 (tsys4) | **S4** (tsys5, 499 GB free) | D5: 3→2 wnodes |
|
||||
| wnode-tsys7 (701) | D5 (tsys4) | **S1** (tsys5, 916 GB free) | D5: 2→1 wnode |
|
||||
|
||||
### After rebalancing — every wnode on its own physical disk
|
||||
|
||||
| Wnode | Host | Disk | Server |
|
||||
|-------|------|------|--------|
|
||||
| wnode-tsys1 | tsys1 | S3 | tsys5 |
|
||||
| wnode-tsys3 | tsys3 | D5 | tsys4 |
|
||||
| wnode-tsys5 | tsys5 | D2 | tsys4 |
|
||||
| wnode-tsys6 | tsys6 | **S4** | **tsys5** |
|
||||
| wnode-tsys7 | tsys7 | **S1** | **tsys5** |
|
||||
| wnode-tsys9 | tsys9 | S2 | tsys5 |
|
||||
|
||||
**tsys4: 2 wnodes. tsys5: 4 wnodes. Zero IOPS contention — each wnode
|
||||
has a dedicated physical spindle.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Host Capacity
|
||||
|
||||
| Host | RAM | VM RAM | RAM% | Threads | vCPU | CPU% | Notes |
|
||||
|------|-----|--------|------|---------|------|------|-------|
|
||||
| tsys1 | 31 GB | 36.6 GB | **118%** | 8 | 24 | 300% | Overcommitted. USB-passthrough VMs locked (bms, ca). |
|
||||
| tsys3 | 31 GB | 28 GB | 90% | 8 | 8 | 100% | 1 wnode (28 GB). OK. |
|
||||
| tsys4 | 15 GB | 2 GB | 13% | 8 | 2 | 25% | Storage server. Friday: →64 GB. |
|
||||
| tsys5 | 94 GB | 127 GB | **135%** | 8 | 72 | 900% | Overcommitted by 33 GB. SDR dongles locked. Most VMs on local-nonprod (can't migrate). |
|
||||
| tsys6 | 125 GB | 48 GB | **38%** | 16 | 12 | 75% | **77 GB free.** Room for more. |
|
||||
| tsys7 | 188 GB | 56 GB | **30%** | 24 | 22 | 92% | **132 GB free.** Most room in fleet. |
|
||||
| tsys9 | 23 GB | 30.2 GB | **131%** | 12 | 16 | 133% | Overcommitted by 7 GB. |
|
||||
|
||||
### Hardware dependencies (locked VMs)
|
||||
|
||||
| Host | VM | Hardware | Can move? |
|
||||
|------|-----|----------|-----------|
|
||||
| tsys1 | pfv-bms (100) | Zigbee + Z-Wave USB dongles | Friday only (physical move) |
|
||||
| tsys1 | tsys-ca (101) | Nitrokey HSM (2×) | Friday only (physical move) |
|
||||
| tsys5 | (various) | SDR dongles | **No — do not move** |
|
||||
|
||||
---
|
||||
|
||||
## 5. Friday Maintenance Plan
|
||||
|
||||
### 5.1 Primary goal: Proxmox on tsys2
|
||||
|
||||
tsys2 is currently Windows 10. It has the **best local storage in the fleet**
|
||||
(Samsung 960 PRO NVMe 512 GB + Samsung 850 EVO SATA SSD 1 TB). The primary
|
||||
Friday deliverable is:
|
||||
|
||||
1. **Move DHCP off pfv-netboot** to the netinfra-01/02 redundant pair. This
|
||||
must happen BEFORE tsys2 is formatted (pfv-netboot runs on tsys2 hardware
|
||||
currently, or depends on a service that does).
|
||||
2. **Format tsys2**, install Proxmox.
|
||||
3. **Agent inventory** — once Proxmox is live, the audit scripts
|
||||
(`perf/deploy-check.sh`) can gather hardware and VM data directly,
|
||||
replacing the Windows PowerShell inventory from
|
||||
[`TODO.md`](TODO.md) §1.
|
||||
4. **Run provisioning** — `provisioning/SetupNewSystem.sh` for baseline
|
||||
hardening, then `perf/scripts/apply-tunings.sh`.
|
||||
5. **Create wnode-tsys2** — boot disk on NVMe (960 PRO), data disk on SATA
|
||||
SSD (850 EVO). No NFS needed — 1.5 TB local SSD is the most local storage
|
||||
of any worker.
|
||||
6. **Join wnode-tsys2 to k3s cluster.**
|
||||
|
||||
tsys2 hardware (from [`TODO.md`](TODO.md) Windows inventory):
|
||||
- Dell Precision 5520, i7-7820HQ Kaby Lake (4c/8t)
|
||||
- 32 GB RAM
|
||||
- Samsung 960 PRO 512 GB NVMe + Samsung 850 EVO 1 TB SATA SSD
|
||||
- Both wired NICs are USB-attached (same anti-pattern as tsys4/9)
|
||||
- NVIDIA Quadro M1200 4 GB (passthrough candidate)
|
||||
- Service tag: GH1XZG2
|
||||
|
||||
### 5.2 Physical hardware work (tsys4, tsys5)
|
||||
|
||||
| Host | Work |
|
||||
|------|------|
|
||||
| tsys4 | PCIe NIC install (replace USB dongle), RAM 16→64 GB |
|
||||
| tsys5 | 2nd ethernet cable (fix bond0), NVMe install, D3 SSD relocate from tsys4 USB |
|
||||
|
||||
### 5.3 VM migrations (while cluster is down for hardware)
|
||||
|
||||
**Relieve overloaded hosts — move NFS-backed VMs to tsys6/tsys7:**
|
||||
|
||||
| VM | VMID | From | To | RAM freed | Storage stays |
|
||||
|----|------|------|----|-----------|---------------|
|
||||
| tsys-siem | 901 | tsys9 (131%) | tsys6 (38%) | 8 GB on tsys9 | D2 (tsys4) |
|
||||
| tsys-librenms | 104 | tsys1 (118%) | tsys7 (30%) | 2 GB on tsys1 | D2 (tsys4) |
|
||||
| tsys-pdm | 105 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
|
||||
| tsys-workbench | 117 | tsys1 | tsys7 | 4 GB | D2 (tsys4) |
|
||||
| tailscale-router | 109 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
|
||||
| kali-tsys | 114 | tsys1 | tsys7 | 2 GB | D2 (tsys4) |
|
||||
|
||||
**Result:** tsys1 drops from 36.6 GB → 24.6 GB (79%). tsys9 drops from
|
||||
30.2 GB → 22.2 GB (96%). tsys6/tsys7 absorb easily.
|
||||
|
||||
**Move USB-passthrough VMs from tsys1 to tsys9 (physical USB relocation):**
|
||||
|
||||
| VM | VMID | From | To | Hardware |
|
||||
|----|------|------|----|---------|
|
||||
| pfv-bms | 100 | tsys1 | tsys9 | Zigbee + Z-Wave dongles (physically move) |
|
||||
| tsys-ca | 101 | tsys1 | tsys9 | Nitrokey HSMs (physically move) |
|
||||
|
||||
**Result:** tsys1 becomes a lightweight k8s-control host (cnode3 + netinfra-01 + ucs-01 only).
|
||||
|
||||
**Wnode storage rebalancing:**
|
||||
|
||||
| VM | VMID | Disk from | Disk to |
|
||||
|----|------|-----------|---------|
|
||||
| wnode-tsys6 | 100 | D5 (tsys4) | S4 (tsys5) |
|
||||
| wnode-tsys7 | 701 | D5 (tsys4) | S1 (tsys5) |
|
||||
|
||||
### 5.4 Post-hardware cnode quorum fix (optional, if local-SSD available)
|
||||
|
||||
After Friday hardware work, consider moving one cnode to local-lvm:
|
||||
- cnode3 (106) on tsys1: local-lvm is HDD (~90 GB) — viable, no NFS dependency
|
||||
- cnode1 (906) on tsys9: local-lvm is SSD (136 GB) — ideal for etcd
|
||||
|
||||
This would give: 1 cnode on local-SSD, 1 on tsys4 NFS, 1 on tsys5 NFS.
|
||||
**Any single failure (host, storage server, or disk) leaves 2 of 3 alive.**
|
||||
Deferred — user wants NFS-only for now.
|
||||
|
||||
---
|
||||
|
||||
## 6. Complete VM Inventory (running VMs only)
|
||||
|
||||
### tsys1 — 11 running (Infrastructure)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 100 | pfv-bms | 2 | 4096 | 32 GB | D2 | tsys4 |
|
||||
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 102 | pfv-k8s-wnode-tsys1 | 4 | 4096 | 32 GB | S3 | tsys5 |
|
||||
| 103 | pfv-netinfra-01 | 2 | 2048 | 32 GB | S2 | tsys5 |
|
||||
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 | tsys4 |
|
||||
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 106 | pfv-k8s-cnode3 | 2 | 4096 | 32 GB | D5 | tsys4 |
|
||||
| 108 | tsys-ucs-01 | 2×2 | 8000 | 32 GB | D2 | tsys4 |
|
||||
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 | tsys4 |
|
||||
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 | tsys4 |
|
||||
|
||||
### tsys3 — 1 running (Kubernetes)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 313 | pfv-k8s-wnode-tsys3 | 8 | 28000 | 32 GB | D5 | tsys4 |
|
||||
|
||||
### tsys4 — 1 running (Storage)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
|
||||
|
||||
### tsys5 — 17 running (Storage + Preprod)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 509 | pfv-k8s-wnode-tsys5 | 2×4 | 32000 | 32 GB | D2 | tsys4 |
|
||||
| 5101 | sectestbed-siem | 2×2 | 10000 | 132 GB | local-nonprod | LOCAL |
|
||||
| 5105 | sectestbed-awx | 2×2 | 4096 | 288 GB | local-nonprod | LOCAL |
|
||||
| 5106 | sectestbed-k8s-cnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5107 | sectestbed-k8s-wnode | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5108 | sectestbed-librenms | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5109 | sectestbed-netinfra | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5111 | ultix-streaming | 2×2 | 9000 | 288 GB | T5-SSD | tsys5 (SSD) |
|
||||
| 5112 | ultix-offstage | 2×2 | 6000 | 288 GB | local-lvm | LOCAL |
|
||||
| 6000 | sectestbed-sandbox | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51010 | sectestbed-tctc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51011 | sectestbed-cloudron | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51012 | sectestbed-hfnoc | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51013 | sectestbed-rancherplatform | 2×2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53100 | tsys-preprod-awx | 2×2 | 9000 | 160 GB | local-nonprod | LOCAL |
|
||||
| 53101 | tsys-preprod-siem | 2×2 | 12000 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53102 | tsys-preprod-rancherplatform | 2×2 | 8000 | 32 GB | local-nonprod | LOCAL |
|
||||
|
||||
### tsys6 — 3 running (Kubernetes)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 100 | pfv-k8s-wnode-tsys6 | 2×2 | 32000 | 32 GB | D5 | tsys4 |
|
||||
| 600 | tsys-awx | 2×2 | 12000 | 32 GB | D2 | tsys4 |
|
||||
| 601 | pfv-k8s-cnode4 | 4 | 4096 | 32 GB | D2 | tsys4 |
|
||||
|
||||
### tsys7 — 6 running (Kubernetes)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 701 | pfv-k8s-wnode-tsys7 | 4 | 32000 | 32 GB | D5 | tsys4 |
|
||||
| 702 | hfnoc-uisp | 2×2 | 8000 | 100 GB | D2 | tsys4 |
|
||||
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 | tsys4 |
|
||||
| 704 | TCTC | 4 | 6000 | 32 GB | D2 | tsys4 |
|
||||
| 705 | pfv-k8s-cnode2 | 4 | 4096 | 32 GB | D2 | tsys4 |
|
||||
| 706 | pfv-k8s-cnode5 | 4 | 4096 | 32 GB | S2 | tsys5 |
|
||||
|
||||
### tsys9 — 6 running (Infrastructure)
|
||||
|
||||
| VMID | Name | CPU | RAM (MB) | Disk | Storage | Server |
|
||||
|------|------|-----|----------|------|---------|--------|
|
||||
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 | tsys4 |
|
||||
| 902 | tsys-ucs-02 | 2×2 | 8000 | 50 GB | S2 | tsys5 |
|
||||
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 | tsys4 |
|
||||
| 904 | pfv-netinfra-02 | 2 | 4000 | 32 GB | D2 | tsys4 |
|
||||
| 905 | pfv-k8s-wnode-tsys9 | 4 | 4096 | 32 GB | S2 | tsys5 |
|
||||
| 906 | pfv-k8s-cnode1 | 2 | 4096 | 32 GB | S3 | tsys5 |
|
||||
@@ -1,485 +0,0 @@
|
||||
# K8S.md -- Kubernetes Architecture Deep-Dive
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Purpose:** Detailed kubernetes architecture plan for the pfv-k8s cluster.
|
||||
Companion to [`PROJECT.md`](PROJECT.md) (which has the fleet-wide assessment).
|
||||
**Status:** For discussion in a future session. No changes made.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Workload Profile](#1-workload-profile)
|
||||
2. [Current State](#2-current-state)
|
||||
3. [Target Architecture](#3-target-architecture)
|
||||
4. [Control Plane (Cnodes)](#4-control-plane-cnodes)
|
||||
5. [Worker Nodes (Wnodes)](#5-worker-nodes-wnodes)
|
||||
6. [Storage Class Design](#6-storage-class-design)
|
||||
7. [ETL/HPC Considerations](#7-etlhpc-considerations)
|
||||
8. [Migration Plan](#8-migration-plan)
|
||||
|
||||
---
|
||||
|
||||
## 1. Workload Profile
|
||||
|
||||
This cluster runs **R&D and RackRental (containerlab) workloads** via
|
||||
Kubernetes. Production (Gitea, RustFS, Redmine, websites) lives on a VPS in
|
||||
Reston, VA running Cloudron.
|
||||
|
||||
**Workload types expected:**
|
||||
|
||||
| Type | Description | Storage need | RAM need | Examples |
|
||||
|------|------------|-------------|----------|---------|
|
||||
| **ETL (weather/GIS)** | Batch processing of large geospatial datasets. Sequential reads, transform, sequential writes. | High capacity (100s of GB), moderate IOPS | Medium (8-32 GB per job) | GRIB/NetCDF processing, raster reprojection |
|
||||
| **HPC (hardware startup)** | Compute-intensive simulations, firmware build pipelines, hardware-in-the-loop testing. | Low capacity, moderate IOPS | High (32-128 GB per job) | RTL simulation, PCB thermal analysis |
|
||||
| **RackRental (containerlab)** | Rapid deployment/teardown of network lab topologies. Many containers, short-lived. | Low capacity, high IOPS (container image pulls) | Low-Medium (4-16 GB) | Network topology testing, protocol validation |
|
||||
|
||||
**Key storage insight:** ETL workloads need bulk capacity (NFS-HDD is fine --
|
||||
sequential I/O). HPC and containerlab need low-latency random I/O (local
|
||||
SSD/NVMe is essential). The tiered StorageClass design (section 6) serves both.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current State
|
||||
|
||||
### 2.1 pfv-k8s nodes and their storage
|
||||
|
||||
| Node | Type | Host | Storage | Disk type | Status |
|
||||
|------|------|------|---------|-----------|--------|
|
||||
| cnode1 (107) | control | tsys1 | D5 (tsys4) | NFS-HDD | running |
|
||||
| cnode2 (603) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode3 (106) | control | tsys1 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode4 (601) | control | tsys6 | D2 (tsys4) | NFS-HDD | running |
|
||||
| cnode5 (602) | control | tsys6 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys3 (313) | worker | tsys3 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys5 (509) | worker | tsys5 | D2 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys6 (100) | worker | tsys6 | D5 (tsys4) | NFS-HDD | **STOPPED** |
|
||||
| wnode-tsys7 (701) | worker | tsys7 | D5 (tsys4) | NFS-HDD | running |
|
||||
| wnode-tsys9 (905) | worker | tsys9 | S3 (tsys5) | NFS-HDD | running |
|
||||
|
||||
### 2.2 Problems
|
||||
|
||||
1. **100% of cnodes on tsys4 NFS.** D2 disk failure loses 3 of 5 cnodes =
|
||||
etcd quorum lost.
|
||||
2. **90% of all k8s nodes on tsys4 NFS.** tsys4 failure kills the cluster.
|
||||
3. **Zero nodes use SSD or NVMe.** All on NFS-over-HDD.
|
||||
4. **Zero nodes use local-lvm.** tsys3/6/7/9 all have empty local storage
|
||||
(349 GB / 1.7 TB / 1.7 TB / 136 GB SSD respectively).
|
||||
5. **wnode-tsys6 is stopped.** Reduces cluster capacity.
|
||||
6. **3 cnodes on tsys6** -- should be on lighter hosts to free tsys6 for workers.
|
||||
|
||||
---
|
||||
|
||||
## 3. Target Architecture
|
||||
|
||||
### 3.1 Design principles
|
||||
|
||||
1. **Cnodes on lightweight hosts** (tsys1, tsys9, tsys3) -- frees tsys6/7 for
|
||||
heavy workers.
|
||||
2. **Cnode storage split across tsys4 and tsys5** -- etcd survives either
|
||||
storage server failing.
|
||||
3. **Wnode boot disks on local storage** -- eliminates NFS latency for
|
||||
container runtime and kubelet.
|
||||
4. **Wnode data disks on NFS-HDD** -- bulk capacity for ETL/weather/GIS.
|
||||
5. **tsys5 NVMe dedicated to wnode-tsys5** -- fastest tier for HPC jobs.
|
||||
6. **One wnode per hypervisor host** -- maximize total cluster capacity.
|
||||
|
||||
### 3.2 Target node-host-storage matrix
|
||||
|
||||
| Node | Type | Host | Boot disk | Data disk | Disk type |
|
||||
|------|------|------|-----------|-----------|-----------|
|
||||
| cnode1 | control | tsys1 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode2 | control | tsys9 | D2 (tsys4) | -- | NFS-HDD |
|
||||
| cnode3 | control | tsys1 | S2 (tsys5) | -- | NFS-HDD |
|
||||
| cnode4 | control | tsys9 | D5 (tsys4) | -- | NFS-HDD |
|
||||
| cnode5 | control | tsys3 | S3 (tsys5) | -- | NFS-HDD |
|
||||
| wnode-tsys1 | worker | tsys1 | D5 (tsys4) | -- | NFS-HDD (small) |
|
||||
| wnode-tsys2 | worker | tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | **NVMe + SSD -- no NFS needed** |
|
||||
| wnode-tsys3 | worker | tsys3 | **local-lvm (NVMe PM961)** | S3 (NFS) | **LOCAL-NVMe** |
|
||||
| wnode-tsys5 | worker | tsys5 | **NVMe (local, Friday)** | local-nonprod (HDD) | **NVMe** |
|
||||
| wnode-tsys6 | worker | tsys6 | D2 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys7 | worker | tsys7 | D5 (tsys4 NFS) | -- | NFS-HDD (local-lvm is USB 2.0 -- do not use) |
|
||||
| wnode-tsys9 | worker | tsys9 | **local-lvm (SSD)** | S2 (NFS) | **LOCAL-SSD** |
|
||||
|
||||
### 3.3 Storage server distribution after changes
|
||||
|
||||
| Storage server | cnodes | wnodes (boot) | wnodes (data) |
|
||||
|---------------|--------|---------------|---------------|
|
||||
| tsys4 (D2) | cnode2 | wnode-tsys6 | wnode-tsys7 |
|
||||
| tsys4 (D5) | cnode1, cnode4 | wnode-tsys1 | -- |
|
||||
| tsys5 (S2) | cnode3 | wnode-tsys9 | -- |
|
||||
| tsys5 (S3) | cnode5 | -- | wnode-tsys3 |
|
||||
|
||||
**Note:** wnode-tsys2 needs no NFS (1.5 TB local SSD). wnode-tsys6/7 stay on
|
||||
NFS by design -- their local-lvm is USB 2.0 portable HDD (~30 MB/s), slower
|
||||
than NFS-HDD, and the user has chosen not to install internal drives.
|
||||
|
||||
**No single disk or server is a quorum-losing failure point.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Control Plane (Cnodes)
|
||||
|
||||
### 4.1 Cnode sizing
|
||||
|
||||
Each cnode: 4 cores, 4 GB RAM, 32 GB disk. This is sufficient for etcd +
|
||||
kubernetes control plane components (API server, scheduler, controller-manager).
|
||||
|
||||
### 4.2 Cnode host placement rationale
|
||||
|
||||
| Host | cnodes | RAM for cnodes | Total host RAM | Remaining for other VMs |
|
||||
|------|--------|---------------|---------------|------------------------|
|
||||
| tsys1 | 2 (cnode1, cnode3) | 8 GB | 32 GB | ~24 GB (but 11 infra VMs consume most) |
|
||||
| tsys9 | 2 (cnode2, cnode4) | 8 GB | 24 GB | ~16 GB (4 infra VMs + 1 wnode) |
|
||||
| tsys3 | 1 (cnode5) | 4 GB | 32 GB | ~28 GB (1 wnode at 20 GB = 8 GB headroom) |
|
||||
|
||||
**tsys6 and tsys7 have ZERO cnodes** -- fully dedicated to heavy worker nodes.
|
||||
|
||||
### 4.3 Cnode storage placement rationale
|
||||
|
||||
The 5 cnodes are split 3-on-tsys4 / 2-on-tsys5:
|
||||
|
||||
| Disk | cnodes | Rationale |
|
||||
|------|--------|-----------|
|
||||
| D5 (tsys4 HDD) | cnode1, cnode4 | Spread load across 2 disks on tsys4 |
|
||||
| D2 (tsys4 HDD) | cnode2 | Only 1 cnode on D2 (was 3 -- reduces blast radius) |
|
||||
| S2 (tsys5 HDD) | cnode3 | tsys5 storage for quorum diversity |
|
||||
| S3 (tsys5 HDD) | cnode5 | tsys5 storage, different disk than S2 |
|
||||
|
||||
**If D2 fails:** cnode2 dies. 4 of 5 survive. Quorum OK.
|
||||
**If D5 fails:** cnode1 + cnode4 die. 3 of 5 survive. Quorum OK.
|
||||
**If tsys4 fails:** cnode1, cnode2, cnode4 die. cnode3 + cnode5 survive on
|
||||
tsys5. **Only 2 of 5 -- QUORUM LOST.**
|
||||
|
||||
Wait -- that is a problem. If tsys4 goes completely offline, we lose 3
|
||||
cnodes and only have 2 on tsys5. That loses quorum (need 3).
|
||||
|
||||
**Revision needed:** Move 1 more cnode to tsys5 storage. Target: 2 on tsys4,
|
||||
3 on tsys5. But that means tsys5 failure (3 cnodes die) leaves only 2 on
|
||||
tsys4. Same problem inverted.
|
||||
|
||||
The fundamental issue: with 5 cnodes and 2 storage servers, the best split is
|
||||
3/2. The server holding 3 cnodes is a quorum-loss risk if it fails. The server
|
||||
holding 2 cnodes is safe (3 survive).
|
||||
|
||||
**Proper solution: 3 cnodes on the "less likely to fail" server, 2 on the
|
||||
other.** After Friday's hardware work:
|
||||
- tsys4 will have a new PCIe NIC + 64 GB RAM -- more reliable
|
||||
- tsys5 will have bond0 fixed + NVMe -- more reliable
|
||||
|
||||
Either way, 3/2 split means one server failure could lose quorum. **To truly
|
||||
solve this, use a 3rd storage target.** Options:
|
||||
- Use tsys9 local SSD for 1 cnode (breaks the 2-server model, adds a 3rd
|
||||
independent failure domain)
|
||||
- Use local-lvm on the cnode's own host (etcd data is local to the VM's host,
|
||||
no NFS dependency at all)
|
||||
|
||||
**Best option: put cnode boot disks on local-lvm where available.** This
|
||||
eliminates NFS entirely for the control plane. Each cnode's etcd data lives on
|
||||
its own host's local disk -- no shared dependency.
|
||||
|
||||
| cnode | Host | **Recommended storage** | Type |
|
||||
|-------|------|------------------------|------|
|
||||
| cnode1 | tsys1 | **local-lvm** (if space) or D5 (tsys4) | LOCAL-HDD or NFS-HDD |
|
||||
| cnode2 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode3 | tsys1 | **S2 (tsys5)** | NFS-HDD |
|
||||
| cnode4 | tsys9 | **local-lvm (SSD)** | **LOCAL-SSD** |
|
||||
| cnode5 | tsys3 | **local-lvm** | LOCAL-HDD |
|
||||
|
||||
With this layout, a tsys4 failure takes down 0 cnodes. A tsys5 failure takes
|
||||
down 1 (cnode3). A host failure takes down at most 2 cnodes. Quorum always
|
||||
survives.
|
||||
|
||||
**This is the recommended approach.** Local storage for cnodes wherever
|
||||
possible. NFS only as fallback.
|
||||
|
||||
### 4.4 etcd performance on local vs NFS
|
||||
|
||||
| Storage | Typical fsync latency | etcd commit latency | Impact |
|
||||
|---------|----------------------|--------------------|--------|
|
||||
| NFS-HDD (via USB dongle on tsys4) | 5-15 ms | 10-30 ms | Slow API responses, sluggish pod scheduling |
|
||||
| NFS-HDD (via PCIe NIC, post-Friday) | 2-8 ms | 5-15 ms | Better but still network-bound |
|
||||
| Local HDD (tsys1/3/6/7 local-lvm) | 1-5 ms | 3-10 ms | No network hop, moderate improvement |
|
||||
| Local SSD (tsys9 PNY CS900) | 0.1-0.5 ms | 0.5-2 ms | **10-30x faster than NFS-HDD** |
|
||||
| NVMe (tsys5, Friday) | 0.02-0.1 ms | 0.1-0.5 ms | **100x faster than NFS-HDD** |
|
||||
|
||||
etcd is the heartbeat of the kubernetes control plane. Every API call, every
|
||||
pod schedule, every controller reconciliation involves an etcd write. Cutting
|
||||
etcd commit latency from 15 ms to 1 ms makes the entire cluster feel 15x more
|
||||
responsive. **This is the single highest-impact change for k8s performance.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Worker Nodes (Wnodes)
|
||||
|
||||
### 5.1 One wnode per hypervisor host
|
||||
|
||||
| Host | wnode | Boot disk | Data disk | Total RAM | wnode RAM | Role |
|
||||
|------|-------|-----------|-----------|-----------|-----------|------|
|
||||
| tsys1 | wnode-tsys1 | D5 (tsys4 NFS) | -- | 32 GB | 4-8 GB | Small worker, infra co-tenant |
|
||||
| tsys2 | wnode-tsys2 | **NVMe (960 PRO 512GB)** | **SATA SSD (850 EVO 1TB)** | 32 GB | 16-24 GB | **Best storage of any worker -- 1.5TB local SSD, no NFS needed** |
|
||||
| tsys3 | wnode-tsys3 | **local-lvm (349 GB)** | S3 (NFS) | 32 GB | 20 GB | General worker |
|
||||
| tsys5 | wnode-tsys5 | **NVMe (local)** | local-nonprod (HDD) | 96 GB | 32-64 GB | **HPC/ETL powerhouse** |
|
||||
| tsys6 | wnode-tsys6 | D2 (tsys4 NFS) | -- | 128 GB | 64-96 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys7 | wnode-tsys7 | D5 (tsys4 NFS) | -- | 192 GB | 96-128 GB | **Heavy worker, max RAM.** local-lvm is USB 2.0 -- stays on NFS |
|
||||
| tsys9 | wnode-tsys9 | **local-lvm SSD (136 GB)** | S2 (NFS) | 24 GB | 4-8 GB | Small worker, SSD boot |
|
||||
|
||||
### 5.2 Why boot disks on local-lvm
|
||||
|
||||
Current: all wnodes boot from NFS. Every container image pull, every kubelet
|
||||
log write, every ephemeral volume traverses the NFS network path.
|
||||
|
||||
With local-lvm boot disks:
|
||||
- **Container image pulls** write to local disk (100-150 MB/s HDD, no network
|
||||
hop) instead of NFS-HDD (80-120 MB/s with network latency)
|
||||
- **kubelet logs** stay local (no NFS writes for log rotation)
|
||||
- **ephemeral storage** (emptyDir volumes) uses local disk by default
|
||||
- **NFS server failure does not kill the wnode** -- the VM stays running, only
|
||||
the data disk (if mounted) goes away
|
||||
|
||||
### 5.3 Wnode sizing guidance
|
||||
|
||||
| Host | Recommended wnode config | Rationale |
|
||||
|------|------------------------|-----------|
|
||||
| tsys7 (192 GB) | 8-12 cores, 96-128 GB RAM, NFS boot | Largest host -- run the heaviest ETL/HPC jobs here. local-lvm is USB 2.0 |
|
||||
| tsys6 (128 GB) | 8 cores, 64-96 GB RAM, NFS boot | Second-largest -- parallel heavy jobs. local-lvm is USB 2.0 |
|
||||
| tsys5 (96 GB + NVMe) | 4 cores, 32-64 GB RAM, NVMe boot + HDD data | NVMe makes this fastest for I/O-bound HPC |
|
||||
| tsys3 (32 GB) | 4 cores, 20 GB RAM, local-lvm boot | General-purpose worker |
|
||||
| tsys2 (32 GB, NVMe+SSD, incoming) | 4 cores, 16-24 GB RAM, **NVMe boot + SSD data** | **Fastest storage worker** -- HPC with I/O bounds |
|
||||
| tsys1 (32 GB) | 2 cores, 4-8 GB RAM | Small worker, don't starve infra VMs |
|
||||
| tsys9 (24 GB) | 2-4 cores, 4-8 GB RAM | Small worker, SSD boot is the advantage |
|
||||
|
||||
### 5.4 Tainting and labeling strategy
|
||||
|
||||
Label wnodes by capability so the k8s scheduler can target them:
|
||||
|
||||
```yaml
|
||||
# Heavy RAM hosts (ETL/HPC)
|
||||
wnode-tsys6: workload=heavy, ram=128g
|
||||
wnode-tsys7: workload=heavy, ram=192g
|
||||
|
||||
# NVMe host (I/O-intensive HPC)
|
||||
wnode-tsys5: workload=hpc, storage=nvme
|
||||
|
||||
# SSD boot host (low-latency)
|
||||
wnode-tsys9: workload=light, storage=ssd
|
||||
|
||||
# General workers
|
||||
wnode-tsys3: workload=general
|
||||
wnode-tsys2: workload=storage-fast, storage=nvme
|
||||
wnode-tsys1: workload=light
|
||||
```
|
||||
|
||||
Then use nodeSelector or nodeAffinity in job specs:
|
||||
|
||||
```yaml
|
||||
# Weather/GIS ETL job -- needs lots of RAM
|
||||
spec:
|
||||
nodeSelector:
|
||||
workload: heavy
|
||||
|
||||
# Firmware build -- needs fast storage
|
||||
spec:
|
||||
nodeSelector:
|
||||
storage: nvme
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Class Design
|
||||
|
||||
### 6.1 Proposed StorageClasses
|
||||
|
||||
| StorageClass | Provisioner | Where | Speed | Use case |
|
||||
|-------------|------------|-------|-------|----------|
|
||||
| `local-fast` | local-path (k8s) | wnode local-lvm / NVMe | 100-3500 MB/s | Container runtime, scratch, databases |
|
||||
| `nfs-hdd` | nfs-subdir-external-provisioner | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
|
||||
| `nfs-ssd` | nfs-subdir-external-provisioner | tsys4 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
|
||||
|
||||
### 6.2 How this maps to wnode disk topology
|
||||
|
||||
Each wnode has:
|
||||
- **Disk 1 (boot/OS):** local-lvm or NVMe. Contains the OS, kubelet, container
|
||||
runtime. k8s `local-fast` StorageClass provisioner points here.
|
||||
- **Disk 2 (bulk data, optional):** NFS mount. Mounted inside the VM as a
|
||||
second block device or filesystem. k8s `nfs-hdd` provisioner points here.
|
||||
|
||||
Inside k8s, pods request storage via PVC:
|
||||
|
||||
```yaml
|
||||
# ETL job: needs bulk storage for weather data
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
spec:
|
||||
storageClassName: nfs-hdd
|
||||
accessModes: [ReadWriteMany] # NFS allows RWX
|
||||
resources:
|
||||
requests:
|
||||
storage: 500Gi
|
||||
|
||||
# HPC job: needs fast scratch
|
||||
spec:
|
||||
storageClassName: local-fast
|
||||
accessModes: [ReadWriteOnce]
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
```
|
||||
|
||||
### 6.3 NFS-SSD tier (D3 and T5-SSD -- both on tsys5 after Friday)
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is EXCLUSIVELY for k8s worker
|
||||
scratch space, with the exception of ultix-streaming which stays on T5-SSD.
|
||||
Spinning rust hosts all other infrastructure VMs** (UCS, netinfra, LibreNMS,
|
||||
SIEM, etc.).
|
||||
|
||||
The SSD NFS exports:
|
||||
- **D3 (tsys5 SAS, 445 GB free):** k8s scratch exclusively (etcd, container
|
||||
cache, ephemeral volumes). Currently 99% empty.
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free after ultix-streaming):** ultix-streaming
|
||||
occupies 83 GB. Remaining 140 GB available for k8s use.
|
||||
|
||||
**tsys5 is the fast-tier hub:** NVMe (local) + D3 SSD + T5-SSD all on one host.
|
||||
This simplifies the StorageClass design -- latency-sensitive k8s PVCs target
|
||||
tsys5 SSD exports, bulk PVCs target either server.
|
||||
|
||||
### 6.4 NFS data distribution across storage servers
|
||||
|
||||
To avoid re-creating the "everything on tsys4" problem, distribute NFS data
|
||||
disks across both servers:
|
||||
|
||||
| wnode | Boot (local) | Bulk data (NFS) | NFS server |
|
||||
|-------|-------------|-----------------|------------|
|
||||
| wnode-tsys3 | local-lvm | S3 | tsys5 |
|
||||
| wnode-tsys5 | NVMe | local-nonprod | local (no NFS) |
|
||||
| wnode-tsys6 | D2 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys7 | D5 (tsys4 NFS) | -- | tsys4 |
|
||||
| wnode-tsys9 | local-lvm (SSD) | S2 | tsys5 |
|
||||
|
||||
This balances: 2 wnodes using tsys4 for bulk data, 2 using tsys5.
|
||||
|
||||
---
|
||||
|
||||
## 7. ETL/HPC Considerations
|
||||
|
||||
### 7.1 Weather/GIS ETL pipeline
|
||||
|
||||
Typical flow: download GRIB/NetCDF files -> process (reproject, aggregate) ->
|
||||
store results.
|
||||
|
||||
| Stage | Storage class | Why |
|
||||
|-------|-------------|-----|
|
||||
| Download raw data | `nfs-hdd` | Large sequential writes. NFS-HDD handles this well. |
|
||||
| Processing scratch | `local-fast` | Random access during transform. Local disk avoids NFS latency. |
|
||||
| Store results | `nfs-hdd` | Large sequential writes. Persistent. |
|
||||
|
||||
**Recommendation:** Deploy a `local-fast` PV mount as `/scratch` on every
|
||||
wnode. ETL jobs use `/scratch` for intermediate processing and write final
|
||||
output to the NFS-mounted `/data`.
|
||||
|
||||
### 7.2 HPC workloads (hardware startup)
|
||||
|
||||
Use cases: RTL simulation, PCB thermal analysis, firmware build pipelines.
|
||||
|
||||
| Workload | Best wnode | Why |
|
||||
|----------|-----------|-----|
|
||||
| RTL simulation (CPU-bound, high RAM) | tsys7 (192 GB) | Most RAM, most cores (24t) |
|
||||
| Firmware builds (I/O-bound, moderate RAM) | tsys5 (NVMe) | Fastest storage for compile I/O |
|
||||
| Hardware-in-the-loop (latency-sensitive) | tsys9 (local SSD) | Lowest latency storage |
|
||||
| Parallel batch jobs | tsys6 + tsys7 | Distribute across both heavy hosts |
|
||||
|
||||
### 7.3 RackRental/containerlab
|
||||
|
||||
Rapid container deployment. Key need: fast container image pulls.
|
||||
|
||||
This is where **local-lvm boot disks** shine. Currently, every container image
|
||||
pull writes through NFS to a spinning disk -- slow. With local-lvm, images
|
||||
cache on local disk (even HDD is 2-3x faster than NFS-HDD for random I/O).
|
||||
On tsys9 (SSD) and tsys5 (NVMe), image pulls are near-instant.
|
||||
|
||||
### 7.4 Data locality for ETL
|
||||
|
||||
For weather/GIS data that is read repeatedly (e.g., climate reanalysis), cache
|
||||
it on local-lvm of the heavy hosts:
|
||||
|
||||
```
|
||||
tsys3 local-lvm (NVMe 349 GB): /data/cache/weather/ -- fastest cache tier
|
||||
tsys5 NVMe (local): /data/cache/gis/ -- fastest cache tier
|
||||
```
|
||||
|
||||
**Note:** tsys6/7 local-lvm is USB 2.0 portable HDD (~30 MB/s) -- cannot
|
||||
be used for caching. Pre-populate weather/GIS data on D2/D5 (NFS) instead.
|
||||
|
||||
This avoids re-reading the same data from the same NFS export on every job
|
||||
if the data is already cached in the page cache.
|
||||
|
||||
---
|
||||
|
||||
## 8. Migration Plan
|
||||
|
||||
**Key enabler:** The hosts are standalone Proxmox installs, but **Proxmox
|
||||
Datacenter Manager (PDM)** manages them collectively and supports VM migration
|
||||
between nodes. Storage migration can be done via the PDM/Proxmox UI rather
|
||||
than manual disk copies -- the destination node just needs access to the target
|
||||
storage (which all nodes have for NFS exports, and local storage can be
|
||||
migrated through the UI's "Storage Migrate" function).
|
||||
|
||||
### 8.1 Phase 1: Friday (after hardware work)
|
||||
|
||||
After tsys5 cable + NVMe and tsys4 NIC + RAM:
|
||||
|
||||
1. **Format tsys5 NVMe** as local directory storage (e.g., `nvme-local`)
|
||||
2. **Restart wnode-tsys6** (VM 100). Keep on NFS (D5). local-lvm is USB 2.0 --
|
||||
do not use for VM storage. Recreate on D2 or D5 NFS.
|
||||
3. **Move wnode-tsys9** (VM 905) disk from S3 (NFS) to local-lvm (SSD).
|
||||
|
||||
### 8.2 Phase 2: Cnode rebalance (maintenance window)
|
||||
|
||||
These changes require creating new VMs on target hosts and migrating disks.
|
||||
Plan for a maintenance window with the k8s cluster briefly down.
|
||||
|
||||
1. Create cnode2 on tsys9 (local-lvm SSD if possible, or D2 NFS).
|
||||
2. Create cnode4 on tsys9 (D5 NFS or local-lvm SSD).
|
||||
3. Create cnode5 on tsys3 (S3 NFS or local-lvm).
|
||||
4. Move cnode3 disk from D2 to S2 (tsys4 to tsys5).
|
||||
5. Join new cnodes to etcd cluster, drain old cnode2/4/5, remove.
|
||||
|
||||
### 8.3 Phase 3: Wnode local storage migration (maintenance window)
|
||||
|
||||
1. Recreate wnode-tsys3 with boot disk on local-lvm (349 GB).
|
||||
2. wnode-tsys6 stays on NFS (local-lvm is USB 2.0 HDD -- not suitable).
|
||||
3. wnode-tsys7 stays on NFS (same reason).
|
||||
4. Recreate wnode-tsys5 with boot disk on NVMe.
|
||||
5. Add data disks (NFS) as second SCSI devices where applicable.
|
||||
|
||||
### 8.4 Phase 4: tsys2 integration (when rebuilt)
|
||||
|
||||
1. Install Proxmox on tsys2.
|
||||
2. Run `scripts/check.sh` to inventory.
|
||||
3. Run `scripts/apply-tunings.sh --apply`.
|
||||
4. Create wnode-tsys2 with **boot disk on NVMe (960 PRO)** and **data disk on SATA SSD (850 EVO)**. No NFS needed -- 1.5 TB local SSD is the most local storage of any worker.
|
||||
5. Join to k8s cluster.
|
||||
|
||||
### 8.5 Phase 5: Critical VM relocation
|
||||
|
||||
1. Move netinfra-02 (VM 904) from D2 to S3 (tsys5 HDD).
|
||||
2. Move ucs-02 (VM 902) from D5 to S2 (tsys5 HDD).
|
||||
3. (No change to T5-SSD -- ultix-streaming stays.)
|
||||
|
||||
---
|
||||
|
||||
## Open questions for next session
|
||||
|
||||
1. **Are the hosts a Proxmox cluster (pvecm) or standalone?** This determines
|
||||
whether live migration is available (huge simplification) or we need manual
|
||||
disk migration. Check `pvecm status` on each host.
|
||||
2. **What k8s distribution is in use?** (k3s, kubeadm, RKE2?) This affects how
|
||||
nodes are joined/drain and how StorageClasses are configured.
|
||||
3. **Container runtime?** (containerd, cri-o?) Affects local storage layout.
|
||||
4. **Is there a container image registry mirror in the cluster?** Or do all
|
||||
pulls go to Docker Hub / external? A local registry on D3 SSD would speed
|
||||
up all pulls.
|
||||
5. **What specific ETL tools?** (GDAL, PostGIS, xarray, Dask?) This affects
|
||||
whether jobs need shared (RWX) or exclusive (RWO) storage.
|
||||
6. **HPC job scheduler?** (plain k8s Jobs, Argo Workflows, Volcano?) Affects
|
||||
how we label and taint nodes.
|
||||
@@ -1,857 +0,0 @@
|
||||
# Proxmox Cluster Project Report
|
||||
|
||||
**Date:** 2026-07-27 (re-audited)
|
||||
**Prepared by:** Performance Optimization Engagement
|
||||
**Status:** Comprehensive fleet assessment with VM placement and redundancy analysis
|
||||
|
||||
**Data freshness:** All 7 hosts re-audited at 21:50 CDT 2026-07-27 via
|
||||
`deploy-check.sh`. VM placements reflect live state after user's PDM
|
||||
migrations. This is ground truth.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#1-executive-summary)
|
||||
2. [Host Fleet](#2-host-fleet)
|
||||
3. [Storage Architecture](#3-storage-architecture)
|
||||
4. [VM Fleet Inventory](#4-vm-fleet-inventory)
|
||||
5. [Kubernetes Node Distribution](#5-kubernetes-node-distribution)
|
||||
6. [Storage Redundancy Analysis](#6-storage-redundancy-analysis)
|
||||
7. [Local SSD/NVMe Opportunity](#7-local-ssdnvme-opportunity)
|
||||
8. [Role Alignment Audit](#8-role-alignment-audit)
|
||||
9. [Network Findings](#9-network-findings)
|
||||
10. [Recommendations](#10-recommendations)
|
||||
11. [Hardware End-of-Support Exposure](#11-hardware-end-of-support-exposure)
|
||||
12. [Open Items](#12-open-items)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
The cluster consists of 7 active Proxmox hosts and 1 incoming (pfv-tsys2),
|
||||
running 43 VMs across two NFS storage servers (tsys4, tsys5). Host-side
|
||||
performance tunings are complete on 5 of 7 hosts. Two hosts (tsys4, tsys5)
|
||||
are blocked on physical hardware work scheduled for Friday.
|
||||
|
||||
**Progress since initial audit:** The user has been actively rebalancing k8s
|
||||
nodes via PDM. Storage distribution improved from 90%/10% (tsys4/tsys5) to
|
||||
73%/27%. One cnode now uses tsys5 storage (cnode5 on S2). More migration
|
||||
needed for etcd quorum survival.
|
||||
|
||||
The VM-layer assessment reveals:
|
||||
|
||||
| # | Finding | Severity | Status |
|
||||
|---|---------|----------|--------|
|
||||
| 1 | **4 of 5 cnodes still store disks on tsys4 NFS.** cnode5 moved to tsys5. Still need 1-2 more moves for quorum survival. | **CRITICAL** | Improving |
|
||||
| 2 | **Both -01/-02 infrastructure pairs (netinfra, UCS) on tsys4 NFS only.** | **HIGH** | TODO today |
|
||||
| 3 | **No k8s node uses SSD or NVMe yet.** tsys3 has 349 GB unused local NVMe; tsys9 has 136 GB local SSD. | **HIGH** | Deferred to k8s session |
|
||||
| 4 | **D3 SSD (tsys4, USB) is 99% empty (445 GB free).** Moving to tsys5 SAS Friday. | **MEDIUM** | Friday |
|
||||
|
||||
---
|
||||
|
||||
## 2. Host Fleet
|
||||
|
||||
### 2.1 Inventory
|
||||
|
||||
| Host | Model | CPU (year) | Cores | RAM | Local Disk | Role (intended) | Tuning |
|
||||
|------|-------|-----------|-------|-----|-----------|-----------------|--------|
|
||||
| pfv-tsys1 | OptiPlex 9020 | i7-4770 Haswell (2013) | 4c/8t | 32 GB DDR3 | HDD (LVM-thin) | **Infrastructure** | Done |
|
||||
| pfv-tsys2 | Precision 5520 | i7-7820HQ Kaby Lake (2017) | 4c/8t | 32 GB (max) | **NVMe 512GB + SATA SSD 1TB** | **Kubernetes** | Incoming (Win10) |
|
||||
| pfv-tsys3 | Precision 7510 | Xeon E3-1535M v5 Skylake (2015) | 4c/8t | 32 GB DDR4 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys4 | Precision T1700 | Xeon E3-1246 v3 Haswell (2013) | 4c/8t | 16 GB DDR3 | 6 disks (HDD+SSD+SMR) | **Storage (NFS+PBS)** | Blocked (NIC+RAM) |
|
||||
| pfv-tsys5 | Precision T7500 | Xeon E5620 Westmere (2010) | 4c/8t | 96 GB DDR3 | 6 disks (HDD+SSD) | **Storage (NFS+VMs)** | Blocked (cable) |
|
||||
| pfv-tsys6 | PowerEdge R610 | 2x Xeon E5530 Nehalem (2009) | 8c/16t | 128 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys7 | PowerEdge R620 | 2x Xeon E5-2630 v2 Ivy Bridge (2013) | 12c/24t | 192 GB DDR3 | HDD (LVM-thin) | **Kubernetes** | Done |
|
||||
| pfv-tsys9 | OptiPlex 7080 | i5-10500 Comet Lake (2020) | 6c/12t | 24 GB DDR4 | **250 GB SSD** (PNY CS900) | **Infrastructure** | Done |
|
||||
|
||||
### 2.2 Role taxonomy (per user directive)
|
||||
|
||||
| Role | Hosts | Workload |
|
||||
|------|-------|----------|
|
||||
| **Infrastructure + k8s control** | tsys1, tsys9 | Infra VMs (netinfra, UCS, PBS, CA, HA) + pfv-k8s cnodes (control plane) + small wnodes |
|
||||
| **Kubernetes workers** | tsys2, tsys3, tsys6, tsys7 | pfv-k8s wnodes (heavy workers) -- these hosts have the RAM (32-192 GB) for ETL/HPC |
|
||||
| **Storage** | tsys4, tsys5 | NFS server + PBS backup target. tsys5 also runs sectestbed/preprod VMs |
|
||||
|
||||
**Design rationale:** cnodes (control plane) are lightweight (4 cores, 4 GB
|
||||
RAM each) and are weighted toward tsys1/tsys9 to keep the heavy RAM/CPU hosts
|
||||
(tsys6 with 128 GB, tsys7 with 192 GB) free for large worker nodes. wnodes
|
||||
run one per hypervisor host across the fleet to maximize total cluster capacity.
|
||||
|
||||
pfv-k8s runs all R&D and RackRental (containerlab) workloads via Kubernetes.
|
||||
Production (Gitea, RustFS, Redmine, websites) lives on a VPS in Reston, VA
|
||||
running Cloudron -- not in this cluster.
|
||||
|
||||
---
|
||||
|
||||
## 3. Storage Architecture
|
||||
|
||||
### 3.1 NFS exports from tsys4 (primary storage server)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| D2 | WDC WD30EFRX Red | HDD (7200rpm) | SATA | 2.7 TB | 187 GB | **2.4 TB** | 8% |
|
||||
| ~~D3~~ | ~~SK hynix SC300~~ | ~~SSD~~ | ~~USB~~ | — | — | — | **moving to tsys5 Friday** |
|
||||
| D5 | Hitachi HDS72302 | HDD (7200rpm) | SATA | 1.8 TB | 236 GB | **1.5 TB** | 14% |
|
||||
|
||||
Non-exported disks on tsys4:
|
||||
- sda (Hitachi 1.8T) at /mnt/albert -- not NFS shared, 1.7 TB free
|
||||
- sdd (WDC 1T) -- **idle, unmounted, removable** (free up for other use)
|
||||
- sdf (WDC 4.5T SMR) at /mnt/backup -- **PBS backup target**, 4.3 TB free
|
||||
|
||||
**D3 migration (Friday):** The SK hynix SC300 SSD is currently USB-attached on
|
||||
tsys4 (via a "ThinkPad SSD" USB adapter). It is moving to a tsys5 SAS port,
|
||||
eliminating the USB bottleneck. tsys4's 4 SATA ports are all occupied (sda/sdb
|
||||
/sdc/sdd), so tsys5 is the better target. See section 3.2.
|
||||
|
||||
### 3.2 NFS exports from tsys5 (secondary storage -- becoming the fast-tier hub)
|
||||
|
||||
| Export | Disk model | Type | Bus | Total | Used | Free | Use% |
|
||||
|--------|-----------|------|-----|-------|------|------|------|
|
||||
| S1 | Seagate ST1000VN | HDD | SAS | 916 GB | 60 GB | 810 GB | 7% |
|
||||
| S2 | Seagate ST1000VN | HDD | SAS | 916 GB | **6.9 GB** | **863 GB** | **1%** |
|
||||
| S3 | Seagate ST1000VN | HDD | SAS | 916 GB | 7.0 GB | **863 GB** | **1%** |
|
||||
| S4 | Toshiba DT01ACA050 | HDD | SAS | 458 GB | 2 MB | **435 GB** | **0%** |
|
||||
| T5-SSD | Samsung 860 PRO | **SSD** | SAS | 234 GB | **122 GB** | **101 GB** | **55%** |
|
||||
| **D3** (Friday) | SK hynix SC300 | **SSD** | **SAS** | **469 GB** | **2 MB** | **445 GB** | **0%** |
|
||||
|
||||
**tsys5 storage controllers (plenty of free ports):**
|
||||
- LSI SAS1068E (SAS 6/iR): 8 ports, 3 used (Samsung SSD, Hitachi, Seagate),
|
||||
**5 free**
|
||||
- Intel ICH10 SATA #1 (4-port): 2 used (Seagate S3, Toshiba S4), **2 free**
|
||||
- Intel ICH10 SATA #2 (2-port): **status unknown, likely free**
|
||||
- 2x Renesas USB 3.0 xHCI controllers (real USB 3.0, unlike tsys6/7)
|
||||
|
||||
**Key finding: S2 and S3 now have k8s node disks.** S2 holds cnode5 +
|
||||
wnode-tsys1 + wnode-tsys9 (6.9 GB used). S3 has wnode-tsys9's old disk
|
||||
(unused, 7 GB). S4 still 99% empty (435 GB free).
|
||||
|
||||
T5-SSD grew to 55% used (122 GB) -- ultix-streaming is the primary consumer.
|
||||
|
||||
**Friday additions:**
|
||||
1. **D3 (SK hynix SSD)** moves from tsys4 USB to tsys5 SAS port. Eliminates
|
||||
USB 2.0 bottleneck. Becomes the second SSD-tier NFS export.
|
||||
2. **PCI NVMe drive** (local-only, not NFS-exported). Used for wnode-tsys5
|
||||
boot disk and HPC scratch. The fastest tier in the fleet.
|
||||
|
||||
After Friday, **tsys5 consolidates all fast storage**: NVMe (local) + 2 SSD
|
||||
NFS exports (D3 + T5-SSD) + 4 HDD NFS exports (S1-S4). This makes tsys5 the
|
||||
natural home for latency-sensitive workloads and the k8s StorageClass design
|
||||
center.
|
||||
|
||||
### 3.3 Local storage tiers (per host, with utilization)
|
||||
|
||||
| Host | Storage ID | Disk type | Bus | Total | Used | Free | Used by VMs? |
|
||||
|------|-----------|-----------|-----|-------|------|------|-------------|
|
||||
| tsys1 | local-lvm | HDD | SATA | ~90 GB | low | ~90 GB | No (all VMs on NFS) |
|
||||
| **tsys3** | **local-lvm** | **NVMe (Samsung PM961)** | **NVMe** | **349 GB** | **0 GB** | **349 GB** | **No (all VMs on NFS)** |
|
||||
| tsys4 | local-lvm | HDD | SATA | ~94 GB | PBS VM | ~62 GB | Yes (PBS VM 400) |
|
||||
| tsys5 | local-lvm | HDD (Hitachi 1.8T) | SATA | 1.7 TB | 40 MB | **1.7 TB** | No |
|
||||
| tsys5 | local-nonprod | HDD (Seagate 1T, =S1) | SATA | 916 GB | 53 GB | **856 GB** | Yes (sectestbed suite) |
|
||||
| **tsys6** | **local-lvm** | **HDD (WD My Passport)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys7** | **local-lvm** | **HDD (WD portable)** | **USB 2.0** | **1.7 TB** | **0 GB** | **1.7 TB** | **No -- DO NOT USE for VM storage** |
|
||||
| **tsys9** | **local-lvm** | **SSD (PNY CS900)** | **SATA** | **136 GB** | **0 GB** | **136 GB** | **No (all VMs on NFS)** |
|
||||
| **tsys2** | **NVMe** (Samsung 960 PRO) | **NVMe** | **NVMe** | **512 GB** | (Win10) | **512 GB** | **Incoming -- fastest boot tier after tsys5 NVMe** |
|
||||
| **tsys2** | **SATA SSD** (Samsung 850 EVO) | **SSD** | **SATA** | **1 TB** | (Win10) | **1 TB** | **Incoming -- bulk data on SSD, not rust** |
|
||||
|
||||
**CRITICAL WARNING: tsys6 and tsys7 local-lvm is USB 2.0 portable HDD.**
|
||||
|
||||
The entire Proxmox OS, swap, and local-lvm on both R610 and R620 run on a
|
||||
single **USB 2.0-attached WD My Passport portable HDD** (tsys6: "My Passport
|
||||
260D"; tsys7: "Drive 2657"). Both servers' only USB controllers are EHCI
|
||||
(USB 2.0, ~480 Mbps). There is **no USB 3.0/xHCI** on either host.
|
||||
|
||||
**USB 2.0 practical throughput is ~30-35 MB/s.** This is 3-4x SLOWER than
|
||||
NFS-over-HDD (~80-120 MB/s). Moving wnode boot disks to local-lvm on these
|
||||
hosts would **decrease** performance. local-lvm on tsys6/7 must NOT be used
|
||||
for VM storage.
|
||||
|
||||
Additionally, both servers have completely empty internal drive bays:
|
||||
- **tsys6**: SAS controller present but **DISABLED** in BIOS. No internal
|
||||
drives.
|
||||
- **tsys7**: 6-port SATA AHCI controller present, **5 ports EMPTY** (only
|
||||
DVD-ROM on port 5). No internal drives.
|
||||
|
||||
This is a reliability risk beyond performance: the entire host OS boots
|
||||
from a consumer-grade portable USB drive not designed for 24/7 server use.
|
||||
|
||||
**tsys3 correction:** Previously documented as HDD. Actually boots from a
|
||||
**Samsung PM961 NVMe 512GB SSD** -- the fastest existing local storage in the
|
||||
fleet. Its 349 GB of local-lvm is excellent for wnode boot disk use.
|
||||
|
||||
**Critical observation: every k8s host has 0% used local-lvm.** tsys3
|
||||
(Samsung PM961 **NVMe**, 349 GB), tsys6 (WD My Passport **USB 2.0** HDD,
|
||||
1.7 TB), tsys7 (WD portable **USB 2.0** HDD, 1.7 TB) all have unused local
|
||||
storage.
|
||||
|
||||
**However, only tsys3's local-lvm is suitable for VM storage.** tsys6 and
|
||||
tsys7 local-lvm is USB 2.0 portable HDD (~30-35 MB/s) -- slower than
|
||||
NFS-over-HDD and unsuitable for wnode boot disks.
|
||||
|
||||
### 3.4 Disk speed tiers summary
|
||||
|
||||
| Tier | Where | Speed class | Best for |
|
||||
|------|-------|------------|----------|
|
||||
| **NVMe** | tsys3 (Samsung PM961), tsys5 (Friday addition), **tsys2 (Samsung 960 PRO 512GB)** | 2000-3500 MB/s | HPC scratch, ETL staging, container runtime, wnode boot, etcd |
|
||||
| **Local SSD** | tsys9 (PNY CS900, 136 GB), **tsys2 (Samsung 850 EVO 1TB)** | 500 MB/s | wnode boot disk, etcd |
|
||||
| **NFS-SSD** | tsys5 D3 (SK hynix, **SAS post-Friday**), tsys5 T5-SSD (Samsung) | 200-400 MB/s over NFS | **k8s worker scratch only** (etcd, container cache, ephemeral volumes) |
|
||||
| **NFS-HDD** | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s over NFS | Bulk data, large disks, non-critical VMs, **wnode boot on tsys6/7** |
|
||||
| **Local SATA HDD** | tsys1 local-lvm | 100-150 MB/s | Host OS only |
|
||||
| **USB 2.0 HDD** | tsys6/7 local-lvm (WD My Passport) | **~30-35 MB/s** | **NOTHING -- slower than NFS, do not use for VMs** |
|
||||
|
||||
### 3.5 Storage tier characterization per host
|
||||
|
||||
| Host | Storage profile | Detail |
|
||||
|------|----------------|--------|
|
||||
| **tsys2** | **SSD/NVMe only** | 960 PRO NVMe 512GB + 850 EVO SATA SSD 1TB. No spinning disk. |
|
||||
| **tsys3** | **NVMe only** | Samsung PM961 NVMe 512GB. No spinning disk. |
|
||||
| **tsys5** | **Hybrid** (fast-tier hub) | NVMe (local, Friday) + D3 SSD + T5-SSD + S1-S4 HDD |
|
||||
| **tsys4** | **Bulk/spinning disk only** | D2 HDD 3TB + D5 HDD 2TB. D3 SSD leaving Friday. PBS target on SMR HDD. |
|
||||
| **tsys9** | **Local SSD + NFS** | PNY CS900 SSD 136GB local + NFS client |
|
||||
| **tsys1** | **Local HDD + NFS** | Small local-lvm + NFS client |
|
||||
| **tsys6/7** | **NFS only** | local-lvm is USB 2.0 HDD (unusable for VMs). All VMs on NFS. |
|
||||
|
||||
### 3.6 All exports are single-disk with no redundancy
|
||||
|
||||
Every NFS export is a single physical disk formatted ext4. No RAID, no ZFS
|
||||
mirror, no mdraid. A single disk failure takes down every VM whose disk lives
|
||||
on that export. This applies to **both storage servers** and to the **PBS
|
||||
backup target** (a single 4.5T SMR drive).
|
||||
|
||||
---
|
||||
|
||||
## 4. VM Fleet Inventory
|
||||
|
||||
### 4.1 Complete VM roster (running VMs only, 40 VMs across 7 hosts)
|
||||
|
||||
#### tsys1 (Infrastructure) -- 11 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-bms (HomeAssistant) | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 101 | tsys-ca | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 103 | **pfv-netinfra-01** | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 104 | tsys-librenms | 2 | 2048 | 50 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 105 | tsys-proxmox-datacenter | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 106 | **pfv-k8s-cnode3** | 2 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 107 | **pfv-k8s-cnode1** | 2 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 108 | **tsys-ucs-01** | 2x2 | 8000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 109 | tailscale-router | 2 | 2048 | 25 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 114 | kali-tsys | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 117 | tsys-secure-workbench | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys3 (Kubernetes) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 313 | **pfv-k8s-wnode-tsys3** | 4x2 | 20000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys4 (Storage) -- 1 running VM
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 400 | pfv-proxmox-backup-server | 2 | 2048 | 32 GB | local-lvm | LOCAL |
|
||||
|
||||
#### tsys5 (Storage) -- 15 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 509 | **pfv-k8s-wnode-tsys5** | 2x4 | 32000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 5101 | sectestbed-siem | 2x2 | 10000 | 132 GB | local-nonprod | LOCAL |
|
||||
| 5105 | sectestbed-awx | 2x2 | 4096 | 288 GB | local-nonprod | LOCAL |
|
||||
| 5106 | sectestbed-k8s-cnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5107 | sectestbed-k8s-wnode | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5108 | sectestbed-librenms | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5109 | sectestbed-netinfra | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 5111 | ultix-streaming | 2x2 | 9000 | 288 GB | T5-SSD (tsys5 SSD) | NFS-SSD |
|
||||
| 5112 | ultix-offstage | 2x2 | 6000 | 288 GB | local-lvm | LOCAL |
|
||||
| 6000 | sectestbed-sandbox | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51010 | sectestbed-tctc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51011 | sectestbed-cloudron | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51012 | sectestbed-hfnoc | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 51013 | sectestbed-rancherplatform | 2x2 | 4096 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53100 | tsys-preprod-awx | 2x2 | 9000 | 160 GB | local-nonprod | LOCAL |
|
||||
| 53101 | tsys-preprod-siem | 2x2 | 12000 | 32 GB | local-nonprod | LOCAL |
|
||||
| 53102 | tsys-preprod-rancherplatform | 2x2 | 8000 | 32 GB | local-nonprod | LOCAL |
|
||||
|
||||
#### tsys6 (Kubernetes) -- 3 running VMs (1 wnode stopped)
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 100 | pfv-k8s-wnode-tsys6 | 2x2 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 600 | tsys-awx | 2x2 | 12000 | 32 GB | D2 (tsys4 HDD) | NFS -- **STOPPED** |
|
||||
| 601 | **pfv-k8s-cnode4** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 602 | **pfv-k8s-cnode5** | 4 | 4096 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 603 | **pfv-k8s-cnode2** | 4 | 4096 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys7 (Kubernetes) -- 4 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 701 | **pfv-k8s-wnode-tsys7** | 4 | 32000 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 702 | hfnoc-uisp | 2x2 | 8000 | 100 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 703 | rr-middleware | 2 | 2048 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 704 | TCTC | 4 | 6000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
|
||||
#### tsys9 (Infrastructure) -- 5 running VMs
|
||||
|
||||
| VMID | Name | Cores | RAM (MB) | Disk | Storage | Tier |
|
||||
|------|------|-------|----------|------|---------|------|
|
||||
| 901 | tsys-siem | 2 | 8000 | 132 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 902 | **tsys-ucs-02** | 2x2 | 8000 | 50 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 903 | kali-rd | 2 | 2048 | 32 GB | D5 (tsys4 HDD) | NFS |
|
||||
| 904 | **pfv-netinfra-02** | 2 | 4000 | 32 GB | D2 (tsys4 HDD) | NFS |
|
||||
| 905 | **pfv-k8s-wnode-tsys9** | 4 | 4096 | 32 GB | S3 (tsys5 HDD) | NFS |
|
||||
|
||||
### 4.2 Storage concentration summary
|
||||
|
||||
| Storage target | # of running VMs | % of fleet |
|
||||
|---------------|-----------------|------------|
|
||||
| **D2 (tsys4 WDC Red 3TB HDD)** | **18** | **45%** |
|
||||
| D5 (tsys4 Hitachi 2TB HDD) | 9 | 23% |
|
||||
| local-nonprod (tsys5 local HDD) | 10 | 25% |
|
||||
| S3 (tsys5 Seagate 1TB HDD) | 1 | 3% |
|
||||
| T5-SSD (tsys5 Samsung SSD) | 1 | 3% |
|
||||
| local-lvm (tsys4 local) | 1 | 3% |
|
||||
|
||||
**27 of 40 running VMs (68%) store their disks on tsys4 NFS exports.**
|
||||
If tsys4 goes offline, two-thirds of the fleet loses its storage.
|
||||
|
||||
---
|
||||
|
||||
## 5. Kubernetes Node Distribution (re-audited 21:50 CDT)
|
||||
|
||||
### 5.1 pfv-k8s cnode (control plane) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | Changed? |
|
||||
|------|------|------------|---------|-----------|----------|
|
||||
| 906 | cnode1 | **tsys9** | D5 | tsys4 | **MOVED from tsys1** |
|
||||
| 705 | cnode2 | **tsys7** | D2 | tsys4 | **MOVED from tsys6** |
|
||||
| 106 | cnode3 | tsys1 | D2 | tsys4 | no change |
|
||||
| 601 | cnode4 | tsys6 | D2 | tsys4 | no change |
|
||||
| 706 | cnode5 | **tsys7** | **S2** | **tsys5** | **MOVED from tsys6, storage moved D5→S2** |
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | cnodes | Quorum impact if it fails |
|
||||
|---------------|--------|--------------------------|
|
||||
| tsys4 (D2+D5) | **4** (cnode1,2,3,4) | Only cnode5 survives = **QUORUM LOST** |
|
||||
| tsys5 (S2) | **1** (cnode5) | 4 survive = quorum OK |
|
||||
|
||||
**Progress:** cnode5 is now on tsys5 (was all 5 on tsys4). But 4-of-5 on tsys4
|
||||
still means a tsys4 failure loses quorum. **Need 2 more cnodes on tsys5.**
|
||||
|
||||
**Host distribution:** cnodes spread across 4 hosts (tsys1, tsys6, tsys7,
|
||||
tsys9) -- good host diversity.
|
||||
|
||||
### 5.2 pfv-k8s wnode (worker) placement -- CURRENT
|
||||
|
||||
| VMID | Name | Hypervisor | Storage | NFS Server | RAM | Status | Changed? |
|
||||
|------|------|------------|---------|-----------|-----|--------|----------|
|
||||
| 102 | wnode-tsys1 | tsys1 | S2 | tsys5 | 4 GB | **STOPPED** | **NEW** |
|
||||
| 313 | wnode-tsys3 | tsys3 | D5 | tsys4 | **28 GB** | running | **RAM bumped 20→28** |
|
||||
| 509 | wnode-tsys5 | tsys5 | D2 | tsys4 | 32 GB | running | no change |
|
||||
| 100 | wnode-tsys6 | tsys6 | D5 | tsys4 | 32 GB | running | **NOW RUNNING** |
|
||||
| 701 | wnode-tsys7 | tsys7 | D5 | tsys4 | 32 GB | running | no change |
|
||||
| 905 | wnode-tsys9 | tsys9 | **S2** | tsys5 | 4 GB | running | **Storage moved S3→S2** |
|
||||
|
||||
**One wnode per host achieved** (tsys1,3,5,6,7,9). wnode-tsys1 is created but
|
||||
stopped. wnode-tsys6 restarted.
|
||||
|
||||
**Storage distribution:**
|
||||
|
||||
| Storage server | wnodes | Notes |
|
||||
|---------------|--------|-------|
|
||||
| tsys4 (D2+D5) | 4 (tsys3,5,6,7) | Still concentrated |
|
||||
| tsys5 (S2) | 2 (tsys1,tsys9) | Improving |
|
||||
|
||||
### 5.3 Summary: k8s node storage distribution
|
||||
|
||||
| Storage server | cnodes | wnodes | Total k8s nodes |
|
||||
|---------------|--------|--------|-----------------|
|
||||
| **tsys4 NFS** | **4 (80%)** | **4 (67%)** | **8 (73%)** |
|
||||
| **tsys5 NFS** | **1 (20%)** | **2 (33%)** | **3 (27%)** |
|
||||
| Local SSD/NVMe | 0 | 0 | 0 (0%) |
|
||||
|
||||
**Was 90%/10%. Now 73%/27%.** Improving but still tsys4-heavy. Target: 3
|
||||
cnodes on each storage server (60/40 or better) so either server failing
|
||||
leaves quorum intact.
|
||||
|
||||
### 5.4 Remaining cnode migration needed for etcd quorum survival
|
||||
|
||||
To survive a tsys4 failure with quorum (3 of 5 alive), at least 3 cnodes must
|
||||
be on tsys5:
|
||||
|
||||
| Action | Effect |
|
||||
|--------|--------|
|
||||
| Move cnode3 (D2→S3 on tsys5) | 3 cnodes on tsys5, 2 on tsys4. tsys4 fail = 3 survive |
|
||||
| Move cnode4 (D2→S2 on tsys5) | Same result, different disk |
|
||||
| Leave cnode1 and one other on tsys4 | tsys5 fail = 4 survive (OK) |
|
||||
|
||||
**Simplest path:** migrate cnode3 and cnode4 storage to tsys5 (S3 and S2) via
|
||||
PDM. Then tsys4 failure leaves cnode5 + cnode3 + cnode4 = 3 of 5 = quorum OK.
|
||||
|
||||
### 5.5 Future k8s architecture (next session -- see [K8S.md](K8S.md))
|
||||
|
||||
The k8s layer will be tackled soon. Key requirements from user:
|
||||
|
||||
- **Platform:** vcluster + Rancher for multi-tenant management
|
||||
- **Auth:** OIDC to Keycloak (running on Cloudron in Reston, VA production)
|
||||
- **Workload isolation (vcluster per tenant):**
|
||||
- RackRental workloads (containerlab network labs)
|
||||
- Suborbital ITAR (compliance-restricted)
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx and 2950 systems (older hardware)
|
||||
will be brought online during peak solar production for burst capacity.
|
||||
These older cores/ram supplement the main fleet when power is abundant.
|
||||
- **WNode sizing:** every Proxmox node will have a wnode. Some nodes will host
|
||||
both cnodes + wnodes. Worker sizes will vary from small (4 GB, fitting into
|
||||
leftover host capacity) to large (28-32 GB, consuming most of a host).
|
||||
- **Friday final audit:** tsys2 will be loaded with Proxmox on Friday, and a
|
||||
full final audit will be performed at that time (post-NVMe install on tsys5,
|
||||
post-D3 SSD relocation, post-tsys4 NIC+RAM).
|
||||
|
||||
### 5.4 sectestbed k8s nodes (separate from pfv-k8s)
|
||||
|
||||
tsys5 also hosts a separate sectestbed kubernetes stack using local storage:
|
||||
|
||||
| VMID | Name | Storage |
|
||||
|------|------|---------|
|
||||
| 5106 | sectestbed-k8s-cnode | local-nonprod (local HDD) |
|
||||
| 5107 | sectestbed-k8s-wnode | local-nonprod (local HDD) |
|
||||
|
||||
These are on local storage (good -- no NFS dependency) but on a single host's
|
||||
single local disk (no redundancy). They are isolated from the pfv-k8s cluster.
|
||||
|
||||
---
|
||||
|
||||
## 6. Storage Redundancy Analysis
|
||||
|
||||
### 6.1 -01/-02 infrastructure pair audit
|
||||
|
||||
Two -01/-02 pairs exist in the fleet:
|
||||
|
||||
**Pair 1: pfv-netinfra (network infrastructure)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 103 | tsys1 | D5 | **tsys4** |
|
||||
| -02 | 904 | tsys9 | D2 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Both halves depend on tsys4. If tsys4 goes down, both netinfra VMs lose their
|
||||
disks. The -02 half should be on an S2/S3/S4 export from tsys5.
|
||||
|
||||
**Pair 2: tsys-ucs (Univention Corporate Server)**
|
||||
|
||||
| Role | VMID | Host | Storage | NFS Server |
|
||||
|------|------|------|---------|-----------|
|
||||
| -01 | 108 | tsys1 | D2 | **tsys4** |
|
||||
| -02 | 902 | tsys9 | D5 | **tsys4** |
|
||||
|
||||
**Verdict: HOST redundancy OK (different hosts), STORAGE redundancy FAILED.**
|
||||
Same issue. Both halves on tsys4. The -02 half should be on tsys5 storage.
|
||||
|
||||
### 6.2 Redundancy principle for paired VMs
|
||||
|
||||
For any -01/-02 pair to survive a single storage server failure:
|
||||
|
||||
```
|
||||
-01 VM disk -> tsys4 NFS export (D2/D3/D5)
|
||||
-02 VM disk -> tsys5 NFS export (S2/S3/S4/T5-SSD)
|
||||
```
|
||||
|
||||
This ensures that losing either tsys4 or tsys5 takes down only one half of
|
||||
the pair. Currently, **both pairs fail this test** because both halves are on
|
||||
tsys4.
|
||||
|
||||
### 6.3 NFS server failure blast radius
|
||||
|
||||
If **tsys4** goes offline (USB NIC failure, disk failure, reboot):
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 5 of 5 (**etcd quorum lost**) |
|
||||
| k8s wnodes that lose storage | 4 of 5 |
|
||||
| Infrastructure VMs that lose storage | 11 of 12 on tsys1 (all on D2/D5) |
|
||||
| Total VMs that lose storage | **27 of 40 (68%)** |
|
||||
|
||||
If **tsys5** goes offline:
|
||||
|
||||
| Impact | Count |
|
||||
|--------|-------|
|
||||
| k8s cnodes that lose storage | 0 of 5 |
|
||||
| k8s wnodes that lose storage | 1 of 5 |
|
||||
| Total VMs that lose storage | 1 of 40 (3%) |
|
||||
|
||||
**tsys4 is a massive blast-radius liability. tsys5 is barely used.**
|
||||
Rebalancing VM storage across both servers dramatically reduces risk.
|
||||
|
||||
---
|
||||
|
||||
## 7. Local SSD/NVMe Opportunity
|
||||
|
||||
### 7.1 Available fast tiers (currently unused by k8s)
|
||||
|
||||
| Host | Device | Type | Size | Available for VMs? | Currently used by k8s? |
|
||||
|------|--------|------|------|--------------------|-----------------------|
|
||||
| tsys4 | D3 (SK hynix SC300, USB) | SSD | 512 GB | Yes (via NFS) | **No** |
|
||||
| tsys5 | T5-SSD (Samsung 860 PRO) | SSD | 256 GB | Yes (via NFS) | **No** (used by ultix-streaming) |
|
||||
| tsys5 | **New NVMe (Friday)** | **NVMe** | TBD | **Yes (local or NFS)** | **No** |
|
||||
| tsys9 | local-lvm (PNY CS900) | SSD | 137 GB free | Yes (local) | **No** |
|
||||
|
||||
### 7.2 Why local storage matters for k8s nodes
|
||||
|
||||
Kubernetes nodes are latency-sensitive in two specific areas:
|
||||
|
||||
1. **etcd (control plane):** etcd writes are synchronous and latency-critical.
|
||||
On NFS over HDD, every etcd write traverses: VM -> virtio-scsi -> NFS
|
||||
client -> TCP -> USB dongle (on tsys4) -> ext4 -> spinning disk. Typical
|
||||
latency: 2-10 ms per write. On local SSD: 0.1-0.5 ms. On NVMe: 0.02-0.1 ms.
|
||||
This directly affects k8s API responsiveness and pod scheduling speed.
|
||||
|
||||
2. **Container image pulls:** Worker nodes pull container images frequently.
|
||||
On NFS-over-HDD, image layer extraction is seek-bound and slow. Local SSD
|
||||
eliminates the network hop and reduces seek time. This matters most for
|
||||
RackRental/containerlab workloads that spin up containers rapidly.
|
||||
|
||||
### 7.3 Current waste: tsys9 local SSD
|
||||
|
||||
tsys9 has a 250 GB PNY CS900 SSD with 137 GB of LVM-thin space available.
|
||||
**Zero VMs use it.** All 5 VMs on tsys9 boot from NFS. The local SSD sits
|
||||
idle. wnode-tsys9 (VM 905) would benefit significantly from local SSD --
|
||||
its disk is currently on S3 (tsys5 NFS over a Seagate HDD).
|
||||
|
||||
### 7.4 Upcoming opportunity: tsys5 NVMe (Friday)
|
||||
|
||||
The PCI NVMe being added to tsys5 will be the fastest storage tier in the
|
||||
fleet. Two placement options:
|
||||
|
||||
**Option A: NFS-export the NVMe (shared).** All hosts can use it. Good for
|
||||
VMs that might need migration. Adds the NFS/network overhead back.
|
||||
|
||||
**Option B: Local-only on tsys5.** VMs on tsys5 get full NVMe speed with no
|
||||
network overhead. Best for k8s wnode-tsys5 and sectestbed VMs. Cannot be
|
||||
accessed from other hosts.
|
||||
|
||||
**Recommendation:** Option B (local-only). k8s worker nodes do not need
|
||||
shared storage -- pods are ephemeral and reschedule on failure. The NVMe
|
||||
should be formatted as a Proxmox directory storage (or LVM-thin) on tsys5
|
||||
and used for local VM images.
|
||||
|
||||
---
|
||||
|
||||
## 8. Role Alignment Audit
|
||||
|
||||
Per the user's intended role taxonomy: tsys1/9 = infrastructure + k8s control
|
||||
plane; tsys2/3/6/7 = k8s workers; tsys4/5 = storage. Cnodes on tsys1/9 is
|
||||
**correct by design** (keeps heavy hosts free for workers).
|
||||
|
||||
### 8.1 VMs that need to move
|
||||
|
||||
| VMID | Name | Current host | Issue | Target |
|
||||
|------|------|-------------|-------|--------|
|
||||
| 509 | pfv-k8s-wnode-tsys5 | tsys5 (storage) | Worker on storage host | tsys7 or tsys2 (when online) |
|
||||
| 905 | pfv-k8s-wnode-tsys9 | tsys9 (infra) | Can stay if small; user decides | tsys9 OK if small wnode |
|
||||
|
||||
### 8.2 Host capacity for k8s nodes
|
||||
|
||||
| Host | Role | Current k8s nodes | k8s RAM used | RAM total | Headroom |
|
||||
|------|------|-------------------|-------------|-----------|----------|
|
||||
| tsys1 | Infra+k8s ctrl | 2 cnodes | 8 GB | 32 GB | ~12 GB (after 11 infra VMs) |
|
||||
| tsys3 | K8s worker | 1 wnode | 20 GB | 32 GB | ~12 GB |
|
||||
| tsys6 | K8s worker | 3 cnodes + 1 wnode (stopped) | 12 GB | 128 GB | **~116 GB** |
|
||||
| tsys7 | K8s worker | 1 wnode | 32 GB | 192 GB | **~160 GB** |
|
||||
| tsys9 | Infra+k8s ctrl | 1 wnode | 4 GB | 24 GB | ~12 GB (after 4 infra VMs) |
|
||||
| tsys2 | K8s worker | 0 (incoming) | 0 | 32 GB | ~32 GB |
|
||||
|
||||
**tsys6 and tsys7 are dramatically underutilized** -- 116 GB and 160 GB of
|
||||
free RAM respectively. They should be the primary targets for heavy worker
|
||||
nodes and ETL/HPC workloads.
|
||||
|
||||
### 8.3 tsys6 wnode-tsys6 is stopped
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Its disk is on D5 (tsys4
|
||||
NFS). This wnode should be restarted (or recreated on local-lvm) to restore
|
||||
cluster capacity.
|
||||
|
||||
---
|
||||
|
||||
## 9. Network Findings
|
||||
|
||||
### 9.1 tsys9 storage NIC is a USB dongle (new finding)
|
||||
|
||||
Validating tsys9 revealed that its storage network interface
|
||||
(`enx9c69d36a5b6c`) is USB-attached (`parentbus usb`). This is the same
|
||||
anti-pattern as tsys4. The onboard Intel NIC (`enp0s31f6`) is used for
|
||||
management; storage uses the USB adapter.
|
||||
|
||||
**Impact:** Same as tsys4 -- achieves line rate but is susceptible to cable
|
||||
wobble, ESD, and USB controller resets. For an infrastructure host with 5
|
||||
VMs, this is a reliability risk.
|
||||
|
||||
**Mitigation:** tsys9 is an OptiPlex 7080 SFF -- it has PCIe slots. A
|
||||
PCIe NIC would eliminate this risk (same recommendation as tsys4).
|
||||
|
||||
### 9.2 tsys4 and tsys5 still blocked (Friday hardware work)
|
||||
|
||||
| Host | Blocker | Staged fix |
|
||||
|------|---------|-----------|
|
||||
| tsys4 | USB cdc_ncm storage NIC | PCIe NIC install + RAM upgrade (16 to 64 GB) |
|
||||
| tsys5 | bond0 broken (1 of 2 slaves) | Plug 2nd ethernet cable + apply layer3+4 hash |
|
||||
|
||||
### 9.3 LACP resolved on tsys6/tsys7
|
||||
|
||||
tsys6 to tsys7 storage path now measures **1.83 Gbps** (was 943 Mbps).
|
||||
The switch LACP hash change took effect after renegotiation. The 56-106K
|
||||
retransmits on this path are confirmed to be non-lossy multi-flow TCP-over-
|
||||
LACP overhead. See `RESULTS.md` (not yet created) for the full analysis.
|
||||
|
||||
### 9.4 NFS nconnect=4 + noatime confirmed active
|
||||
|
||||
All hosts (including tsys9) show `nconnect=4,noatime` in their NFS mount
|
||||
options. Each host maintains 4 TCP connections per NFS mount to each storage
|
||||
server. This was the Tier 0 tuning item from the performance optimization
|
||||
engagement and is confirmed working cluster-wide.
|
||||
|
||||
---
|
||||
|
||||
## 10. Recommendations
|
||||
|
||||
**No changes have been made. These are assessment-only recommendations.**
|
||||
See `K8S.md` for the detailed kubernetes architecture deep-dive.
|
||||
|
||||
### 10.1 CRITICAL: Critical infrastructure VM placement (netinfra, UCS)
|
||||
|
||||
These are the most critical production VMs in the fleet. They must survive
|
||||
any single-point failure (host, storage server, or disk).
|
||||
|
||||
**Design principle for -01/-02 HA pairs:**
|
||||
- Different hypervisors (already satisfied: tsys1 vs tsys9)
|
||||
- Different storage servers (currently FAILED: all on tsys4)
|
||||
- Prefer SSD for latency-sensitive services
|
||||
|
||||
**Recommended placement:**
|
||||
|
||||
| VM | Host | Storage | Tier | Free space | Rationale |
|
||||
|----|------|---------|------|-----------|-----------|
|
||||
| **netinfra-01** (103) | tsys1 | **D5 (tsys4 HDD)** | NFS-HDD | 1.5 TB | DNS/DHCP/NTP = minimal I/O. Stays put. |
|
||||
| **netinfra-02** (904) | tsys9 | **S3 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D2. Cross-server redundancy. Minimal I/O. |
|
||||
| **ucs-01** (108) | tsys1 | **D2 (tsys4 HDD)** | NFS-HDD | 2.4 TB | Stays put. LDAP/AD does not need SSD. |
|
||||
| **ucs-02** (902) | tsys9 | **S2 (tsys5 HDD)** | NFS-HDD | 870 GB | Move from D5. Cross-server redundancy. No SSD needed. |
|
||||
|
||||
**Failure survival matrix (all single-point failures):**
|
||||
|
||||
| Failure | netinfra-01 | netinfra-02 | ucs-01 | ucs-02 | Result |
|
||||
|---------|-------------|-------------|--------|--------|--------|
|
||||
| tsys4 dies | dies (D5) | **alive** (S3) | dies (D2) | **alive** (S2) | netinfra-02 + ucs-02 alive |
|
||||
| tsys5 dies | **alive** (D5) | dies (S3) | **alive** (D2) | dies (S2) | netinfra-01 + ucs-01 alive |
|
||||
| tsys1 dies | dies | **alive** | dies | **alive** | -02 pair survives |
|
||||
| tsys9 dies | **alive** | dies | **alive** | dies | -01 pair survives |
|
||||
| Any single disk | **all 4 on different disks/servers** | **all 4 alive** | | | |
|
||||
|
||||
**Why this works:** Every row has at least one netinfra and one UCS alive.
|
||||
The network (DNS/DHCP) and directory (AD/LDAP) services never go fully dark.
|
||||
|
||||
**D3 SSD repurposed:** With UCS staying on HDD, the D3 SSD (moving to tsys5
|
||||
Friday) is freed for latency-sensitive workloads that actually benefit from
|
||||
SSD -- sectestbed k8s nodes, CI/CD artifact cache, or a container image
|
||||
registry mirror. Not infrastructure VMs.
|
||||
|
||||
### 10.2 CRITICAL: Cnode (control plane) storage split
|
||||
|
||||
**Problem:** All 5 cnodes store disks on tsys4. D2 disk failure loses etcd
|
||||
quorum (3 of 5 cnodes share D2).
|
||||
|
||||
**Target: cnodes weighted toward tsys1/tsys9 (lightweight hosts), freeing
|
||||
tsys6/tsys7 for heavy workers. Storage splits across tsys4 and tsys5.**
|
||||
|
||||
| cnode | Current host | **Target host** | Current storage | **Target storage** | Rationale |
|
||||
|-------|-------------|----------------|----------------|-------------------|-----------|
|
||||
| cnode1 (107) | tsys1 | **tsys1** (stays) | D5 (tsys4) | **D5 (tsys4)** -- no change | Already correct |
|
||||
| cnode3 (106) | tsys1 | **tsys1** (stays) | D2 (tsys4) | **S2 (tsys5)** -- **MOVE disk** | Split storage to tsys5 |
|
||||
| cnode2 (603) | tsys6 | **tsys9** | D2 (tsys4) | **D2 (tsys4)** -- no disk change | Free tsys6 for heavy workers |
|
||||
| cnode4 (601) | tsys6 | **tsys9** | D2 (tsys4) | **D5 (tsys4)** -- spread disk | Free tsys6; spread off D2 |
|
||||
| cnode5 (602) | tsys6 | **tsys3** | D5 (tsys4) | **S3 (tsys5)** -- **MOVE disk** | Free tsys6; split storage to tsys5 |
|
||||
|
||||
**Result after changes:**
|
||||
|
||||
| Host | cnodes | Storage server |
|
||||
|------|--------|---------------|
|
||||
| tsys1 | cnode1 (D5), cnode3 (S2) | tsys4 + tsys5 |
|
||||
| tsys9 | cnode2 (D2), cnode4 (D5) | tsys4 |
|
||||
| tsys3 | cnode5 (S3) | tsys5 |
|
||||
|
||||
- 3 cnodes on tsys4 storage, 2 on tsys5. Either storage server can fail and
|
||||
etcd keeps quorum (3 of 5 survive).
|
||||
- D2 has 1 cnode (was 3). D5 has 2. S2 and S3 have 1 each. No single disk
|
||||
holds more than 2 cnodes.
|
||||
- Cnodes now on 3 hosts (tsys1, tsys9, tsys3). Any single host failure leaves
|
||||
at least 3 cnodes alive.
|
||||
- **tsys6 and tsys7 are fully freed** for heavy worker nodes.
|
||||
|
||||
Note: tsys1 RAM is tight (32 GB, 11 infra VMs). Adding 0 new cnodes (keeping
|
||||
the 2 already there) is feasible with KSM. tsys9 (24 GB) has room for 2
|
||||
cnodes (8 GB). tsys3 (32 GB) has room for 1 cnode (4 GB) alongside its wnode.
|
||||
|
||||
### 10.3 HIGH: Wnode distribution -- one per host, tiered storage
|
||||
|
||||
**Target: one wnode per hypervisor host, using local storage where possible
|
||||
and NFS-HDD for bulk data.**
|
||||
|
||||
| wnode | Host | Boot disk (OS+containers) | Data disk (bulk/ETL) | Rationale |
|
||||
|-------|------|--------------------------|---------------------|-----------|
|
||||
| wnode-tsys1 | tsys1 | D5 (tsys4 NFS) | -- | Small wnode on infra host. Minimal capacity. |
|
||||
| wnode-tsys3 | tsys3 | **local-lvm (349 GB NVMe)** | S3 (NFS) | Move from NFS to **NVMe** (Samsung PM961). Fastest boot disk after tsys5/2. |
|
||||
| wnode-tsys6 | tsys6 | D2 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys7 | tsys7 | D5 (tsys4 NFS) | -- | **Stays on NFS.** local-lvm is USB 2.0 HDD (~30 MB/s) -- slower than NFS. |
|
||||
| wnode-tsys9 | tsys9 | **local-lvm (136 GB SSD)** | S2 (NFS) | Move from NFS to local SSD. Fast boot, NFS for bulk. |
|
||||
| wnode-tsys2 | tsys2 | **NVMe (Samsung 960 PRO 512GB)** | **SATA SSD (Samsung 850 EVO 1TB)** | **Best storage of any wnode.** No NFS needed -- 1.5 TB local SSD. |
|
||||
| wnode-tsys5 | tsys5 | **new NVMe (local)** | local-nonprod (HDD) | **Fastest wnode in fleet.** HPC/ETL workloads land here. |
|
||||
|
||||
**Storage tiering strategy per wnode:**
|
||||
|
||||
Each wnode gets two disk tiers mapped to k8s StorageClasses:
|
||||
|
||||
1. **Boot + container runtime** (local-lvm or NVMe): OS, kubelet, container
|
||||
images, ephemeral storage. This is where local SSD/NVMe shines -- container
|
||||
image pulls and layer extraction are seek-bound and benefit enormously from
|
||||
low-latency storage.
|
||||
|
||||
2. **Bulk data** (NFS-HDD via D2/D5/S2/S3): weather/GIS datasets, ETL staging
|
||||
areas, large files that do not fit on local storage. Mounted as a second
|
||||
disk in the VM and exposed to k8s as a StorageClass.
|
||||
|
||||
This maps to two k8s StorageClasses:
|
||||
- `local-storage`: bound to the wnode's boot/local disk (fast, ephemeral)
|
||||
- `nfs-bulk`: bound to NFS exports (slow, persistent, large capacity)
|
||||
|
||||
### 10.4 HIGH: Dedicate D3 SSD exclusively to k8s scratch
|
||||
|
||||
**Storage philosophy (user directive): NVMe/SSD is for k8s worker scratch
|
||||
space and ultix-streaming (developer workstation running "cluster of 1"
|
||||
pre-production jobs). Spinning rust hosts all other infrastructure VMs**
|
||||
(UCS, netinfra, LibreNMS, SIEM, etc.).
|
||||
|
||||
SSD allocation after Friday:
|
||||
- **D3 (tsys5 SAS, 445 GB free, 0% used)** -- dedicated to k8s scratch via
|
||||
the `nfs-ssd` StorageClass (etcd, container cache, ephemeral volumes).
|
||||
- **T5-SSD (tsys5 SAS, 140 GB free)** -- ultix-streaming (VM 5111) stays here
|
||||
(developer workstation, runs single-node test jobs before k8s). Remaining
|
||||
140 GB available for k8s.
|
||||
|
||||
**Deep-dive on exact k8s scratch allocation is deferred to the next session**
|
||||
(K8S.md) once we know the k8s distribution, job scheduler, and workload mix.
|
||||
|
||||
### 10.5 MEDIUM: Restart wnode-tsys6
|
||||
|
||||
VM 100 (pfv-k8s-wnode-tsys6) is stopped on tsys6. Recreate on local-lvm
|
||||
(1.7 TB free) instead of D5 NFS. This restores cluster capacity and moves
|
||||
the boot disk to local storage simultaneously.
|
||||
|
||||
### 10.6 MEDIUM: tsys5 NVMe placement (Friday)
|
||||
|
||||
**Recommendation: local-only on tsys5, formatted as Proxmox LVM-thin or
|
||||
directory storage.**
|
||||
|
||||
Use for:
|
||||
- wnode-tsys5 boot disk (primary beneficiary -- HPC/ETL workloads)
|
||||
- sectestbed VMs that need fast scratch space
|
||||
- Not NFS-exported (avoid adding network overhead to the fastest tier)
|
||||
|
||||
### 10.7 LOW: Add PCIe NIC to tsys9
|
||||
|
||||
tsys9's storage NIC is a USB dongle. tsys9 is an OptiPlex 7080 with PCIe
|
||||
slots. A $150 PCIe NIC eliminates the USB reliability risk.
|
||||
|
||||
### 10.8 LOW: Standardize PVE/kernel versions
|
||||
|
||||
tsys3 is on PVE kernel 7.0.14; others on 6.17.x. PVE-manager versions vary
|
||||
(9.1.1 / 9.1.5 / 9.2.5). Standardize in a maintenance window.
|
||||
|
||||
---
|
||||
|
||||
## 11. Hardware End-of-Support Exposure
|
||||
|
||||
| Host | EOS date | Years past | Form factor |
|
||||
|------|----------|-----------|-------------|
|
||||
| pfv-tsys6 (R610) | 2013-05 | 13.2 | 1U server |
|
||||
| pfv-tsys5 (T7500) | 2014-12 | 11.7 | Workstation |
|
||||
| pfv-tsys4 (T1700) | 2018-03 | 8.4 | Workstation |
|
||||
| pfv-tsys7 (R620) | 2019-03 | 7.4 | 1U server |
|
||||
| pfv-tsys1 (9020) | 2019-07 | 7.0 | SFF desktop |
|
||||
| pfv-tsys3 (7510) | 2020-07 | 6.0 | Laptop |
|
||||
| pfv-tsys2 (5520) | TBD | -- | Laptop |
|
||||
| **pfv-tsys9 (7080)** | **2024-02** | **2.4** | **SFF desktop (only supported)** |
|
||||
|
||||
**6 of 8 hosts are past vendor end-of-support.** Only tsys9 is still covered.
|
||||
The two actual rack servers (R610, R620) are the most overdue for replacement.
|
||||
This belongs in the next budget cycle.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open Items
|
||||
|
||||
### 12.1 Immediate (do today via PDM)
|
||||
|
||||
1. **Migrate ucs-02** (VM 902) from D5 (tsys4) to S2 (tsys5) for cross-server
|
||||
redundancy. UCS stays on HDD.
|
||||
2. **Migrate netinfra-02** (VM 904) from D2 (tsys4) to S3 (tsys5).
|
||||
3. **Migrate cnode3** (VM 106) from D2 (tsys4) to S3 (tsys5) -- etcd quorum.
|
||||
4. **Migrate cnode4** (VM 601) from D2 (tsys4) to S2 (tsys5) -- etcd quorum.
|
||||
5. **Start wnode-tsys1** (VM 102) if the cluster needs the capacity.
|
||||
|
||||
### 12.2 Friday maintenance window (user action)
|
||||
|
||||
1. **tsys4:** Install PCIe NIC (replace USB dongle), add RAM (16 to 64 GB),
|
||||
reconfigure `/etc/network/interfaces`, reboot.
|
||||
2. **tsys5:** Plug 2nd ethernet cable, verify bond0, apply layer3+4 hash,
|
||||
install PCI NVMe, relocate D3 SSD from tsys4 USB to tsys5 SAS port,
|
||||
format NVMe as local storage, reboot.
|
||||
3. **tsys2:** Load Proxmox (replacing Windows 10).
|
||||
4. **Final audit:** Re-run `deploy-check.sh` across all hosts including tsys2.
|
||||
|
||||
### 12.3 Post-Friday validation
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Run `scripts/check.sh` on tsys2 once Proxmox is loaded.
|
||||
4. Update PROJECT.md with post-hardware numbers.
|
||||
|
||||
### 12.4 Future: Kubernetes deep-dive (see [K8S.md](K8S.md))
|
||||
|
||||
Next major workstream. Requirements captured:
|
||||
- **vcluster + Rancher** for multi-tenant k8s management
|
||||
- **OIDC auth** to Keycloak (on Cloudron, Reston VA production)
|
||||
- **Workload isolation** via separate vclusters:
|
||||
- RackRental (containerlab)
|
||||
- Suborbital ITAR
|
||||
- Suborbital non-ITAR
|
||||
- Starting Line Productions customer workloads
|
||||
- **Solar-aware scale-out:** PowerEdge 19xx + 2950 systems brought online
|
||||
during peak solar production for burst capacity
|
||||
- **Every host gets a wnode** (variable sizing: small 4 GB to large 32 GB)
|
||||
- **SSD/NVMe reserved for k8s scratch** (plus ultix-streaming exception)
|
||||
- **Spinning rust for all infrastructure VMs**
|
||||
|
||||
### 12.5 Data gaps
|
||||
|
||||
| Gap | How to close |
|
||||
|-----|--------------|
|
||||
| tsys5 SDR/parallel-port workload dependency | Confirm what uses the SDR |
|
||||
| tsys2 post-Proxmox baseline | Run `check.sh` after Friday install |
|
||||
| tsys5 NVMe size and model | Confirm after Friday installation |
|
||||
| tsys3 thermal state (laptop in rack) | Check `sensors` on next maintenance |
|
||||
| PowerEdge 19xx/2950 inventory | When solar scale-out is planned |
|
||||
|
||||
### 12.6 Scripts and tools available
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/check.sh` | Read-only data collector (run on hosts) |
|
||||
| `scripts/apply-tunings.sh` | Apply all Tier 0 tunings (dry-run/apply/rollback) |
|
||||
| `scripts/fix-bond-nfs.sh` | Fix NFS options + bond hash |
|
||||
| `validate-fixes.sh` | Read-only validation of all applied changes |
|
||||
| `iperf-full-matrix.sh` | Full iperf test suite (mgmt + storage) |
|
||||
| `deploy-check.sh` | Deploy check.sh to all hosts via SSH (now includes tsys9) |
|
||||
@@ -1,242 +0,0 @@
|
||||
# TODO.md — Pending User Actions
|
||||
|
||||
**Date:** 2026-07-27
|
||||
**Items needing user input or physical action.**
|
||||
|
||||
---
|
||||
|
||||
## 1. tsys2 Windows hardware inventory (run on the Windows host)
|
||||
|
||||
pfv-tsys2 is currently Windows 10. Before rebuilding it as Proxmox, gather
|
||||
hardware data so the architecture plan can account for it.
|
||||
|
||||
### Option A: PowerShell (recommended — single command, copy-paste output)
|
||||
|
||||
Open **PowerShell as Administrator** and run:
|
||||
|
||||
```powershell
|
||||
# Full hardware inventory in one shot
|
||||
Write-Output "=== COMPUTER ==="
|
||||
Get-CimInstance Win32_ComputerSystem | Select-Object Manufacturer, Model, SystemType, TotalPhysicalMemory | Format-List
|
||||
|
||||
Write-Output "`n=== CPU ==="
|
||||
Get-CimInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors, MaxClockSpeed | Format-List
|
||||
|
||||
Write-Output "`n=== MEMORY STICKS ==="
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISKS ==="
|
||||
Get-PhysicalDisk | Select-Object FriendlyName, MediaType, BusType, Size, SpindleSpeed | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== DISK PARTITIONS ==="
|
||||
Get-Disk | Select-Object Number, FriendlyName, Size, PartitionStyle, OperationalStatus | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== NETWORK ADAPTERS ==="
|
||||
Get-NetAdapter | Select-Object Name, InterfaceDescription, Status, LinkSpeed, MacAddress | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== GPU(s) ==="
|
||||
Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM, DriverVersion, VideoProcessor | Format-List
|
||||
|
||||
Write-Output "`n=== PCIe SLOTS ==="
|
||||
Get-CimInstance Win32_SystemSlot | Select-Object SlotDesignation, CurrentUsage, Status | Format-Table -AutoSize
|
||||
|
||||
Write-Output "`n=== USB DEVICES (storage + network only) ==="
|
||||
Get-PnpDevice -PresentOnly | Where-Object { $_.Class -in @('DiskDrive','Net','USB') } | Select-Object Class, FriendlyName, Status | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
Copy the full output into a file (e.g., `tsys2-hardware.txt`) or paste it
|
||||
directly into the chat.
|
||||
|
||||
### Option B: Command Prompt (cmd.exe) fallbacks
|
||||
|
||||
If PowerShell is unavailable for some reason, these cmd commands give a
|
||||
subset:
|
||||
|
||||
```cmd
|
||||
:: Computer model and serial
|
||||
wmic computersystem get manufacturer,model
|
||||
wmic bios get serialnumber
|
||||
|
||||
:: CPU
|
||||
wmic cpu get name,numberofcores,numberoflogicalprocessors,maxclockspeed
|
||||
|
||||
:: RAM (total)
|
||||
wmic computersystem get totalphysicalmemory
|
||||
|
||||
:: RAM sticks (per-slot detail)
|
||||
wmic memorychip get manufacturer,capacity,speed,partnumber,devicelocator
|
||||
|
||||
:: Disks
|
||||
wmic diskdrive get model,size,interfacetype,mediatype
|
||||
|
||||
:: Network adapters
|
||||
wmic nic where netenabled=true get name,speed,macaddress
|
||||
|
||||
:: GPU
|
||||
wmic path win32_videocontroller get name,adapterram,driverversion
|
||||
```
|
||||
|
||||
### What I'm looking for
|
||||
|
||||
- **Disk inventory**: Are there any SSDs/NVMe available locally? (Determines
|
||||
whether wnode-tsys2 can use local storage like the other wnodes.)
|
||||
- **Network adapters**: How many onboard NICs? Model? (Determines whether
|
||||
tsys2 needs a USB dongle for storage network like tsys4/9, or has a real
|
||||
onboard NIC available.)
|
||||
- **RAM layout**: Is all 32 GB in 1 stick, 2 sticks, or 4 sticks? (Affects
|
||||
memory bandwidth for HPC workloads — dual-channel matters.)
|
||||
- **GPU detail**: Confirm the Quadro M1200 model and VRAM for passthrough
|
||||
planning.
|
||||
- **PCIe slots**: Is there a free PCIe slot for adding a NIC or HBA?
|
||||
- **Service tag confirmation**: `GH1XZG2` (already on file from spreadsheet).
|
||||
|
||||
### STATUS: Collected 2026-07-27
|
||||
|
||||
**Disk inventory — RESOLVED:**
|
||||
- Disk 0: Samsung SSD 960 PRO **512 GB NVMe** (best local storage in fleet)
|
||||
- Disk 1: Samsung SSD 850 EVO **1 TB SATA SSD**
|
||||
- Both SSDs, no spinning rust. 1.5 TB total local SSD.
|
||||
|
||||
**Network adapters — RESOLVED (concerning):**
|
||||
- StorageNetwork: **ASIX USB to Gigabit Ethernet** (dongle, D4-81-D7-3E-0D-5E)
|
||||
- Ethernet: **Realtek USB GbE Family Controller** (also USB, 18-FD-CB-00-D2-CA)
|
||||
- Wi-Fi: Intel 8265 (disconnected)
|
||||
- **Both wired NICs are USB-attached.** Same anti-pattern as tsys4/9.
|
||||
Unavoidable on this laptop form factor — no onboard PCIe NIC available.
|
||||
|
||||
**GPU — RESOLVED:**
|
||||
- Intel HD Graphics 630 (integrated, 1 GB)
|
||||
- NVIDIA Quadro M1200 (4 GB, confirmed for passthrough)
|
||||
|
||||
**PCIe slots — RESOLVED:**
|
||||
- Slots 3/6/7/8 report "Available" but these are laptop M.2/WWAN slots, not
|
||||
user-accessible full PCIe. **Cannot add a PCIe NIC.** NVMe slot occupied
|
||||
by 960 PRO.
|
||||
|
||||
**RAM — PARTIAL:**
|
||||
- Total 32 GB confirmed (34,097,573,888 bytes).
|
||||
- Per-stick detail failed to run (PowerShell line-break split
|
||||
`Format-T` + `able`). Re-run the command below if bandwidth planning
|
||||
needs stick-level detail:
|
||||
```powershell
|
||||
Get-CimInstance Win32_PhysicalMemory | Select-Object Manufacturer, PartNumber, Capacity, Speed, ConfiguredClockSpeed, DeviceLocator, FormFactor | Format-Table -AutoSize
|
||||
```
|
||||
|
||||
**Service tag — CONFIRMED:** `GH1XZG2` (Precision 5520).
|
||||
|
||||
---
|
||||
|
||||
## 2. Friday maintenance window (physical hardware)
|
||||
|
||||
### tsys4 — install PCIe NIC + add RAM
|
||||
|
||||
1. Power down tsys4 (graceful shutdown via Proxmox UI or `shutdown -h now`).
|
||||
2. Install the **PCIe NIC** (Intel i350-T2 or similar 1 GbE dual-port).
|
||||
3. Add **RAM**: 16 GB → 64 GB DDR3 ECC.
|
||||
4. Power on, then update `/etc/network/interfaces` to replace
|
||||
`enx8cae4ccda926` (USB dongle) with the new PCIe NIC device name.
|
||||
5. Reboot to activate new NIC and NFS nconnect.
|
||||
6. Run `validate-fixes.sh pfv-tsys4` to confirm.
|
||||
|
||||
### tsys5 — plug storage cable + install NVMe + relocate D3 SSD
|
||||
|
||||
1. Plug the **second ethernet cable** into tsys5's dedicated storage NIC.
|
||||
2. Verify bond0 recovery: `cat /proc/net/bonding/bond0` — look for
|
||||
"Number of ports: 2" and a real partner MAC (not all zeros).
|
||||
3. Apply bond hash fix (same as tsys6/7):
|
||||
```bash
|
||||
echo "layer3+4" > /sys/class/net/bond0/bonding/xmit_hash_policy
|
||||
```
|
||||
4. **Relocate D3 SSD** from tsys4 USB to tsys5 SAS port:
|
||||
- Power down tsys4
|
||||
- Remove the SK hynix SC300 SSD from its USB enclosure on tsys4
|
||||
- Install it on a free SAS port on tsys5 (5 ports free on LSI SAS1068E)
|
||||
- On tsys5: mount as `/mnt/pfv-tsys5/D3`, add to `/etc/exports`
|
||||
- Update `/etc/pve/storage.cfg` cluster-wide: repoint D3 `server` from
|
||||
`pfv-tsys4-nfs-stor` to `pfv-tsys5-nfs-stor`, update `export` path
|
||||
- Copy any existing D3 data from tsys4 first (currently ~2 MB, essentially
|
||||
empty, so minimal migration)
|
||||
5. Install the **PCI NVMe drive** (uses a PCI slot, not a SATA/SAS port).
|
||||
6. Format NVMe as local directory storage (see TODO section 3 below).
|
||||
7. Reboot tsys5 to activate NFS nconnect.
|
||||
8. Run `validate-fixes.sh pfv-tsys5` to confirm.
|
||||
|
||||
---
|
||||
|
||||
## 3. tsys5 NVMe format/mount decision (after Friday install)
|
||||
|
||||
**Recommendation: local-only, not NFS-exported.** Format as Proxmox directory
|
||||
storage so it shows up as a VM image target in the Proxmox UI.
|
||||
|
||||
After the NVMe is physically installed and visible in Proxmox:
|
||||
|
||||
1. Identify the device: `lsblk` or `ls /dev/nvme*`
|
||||
2. Format and add to Proxmox:
|
||||
```bash
|
||||
# Option A: LVM-thin (thin provisioning, snapshots)
|
||||
pvcreate /dev/nvme0n1
|
||||
vgcreate nvme-pool /dev/nvme0n1
|
||||
lvcreate -l 100%FREE -T nvme-pool/data
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > LVM-Thin
|
||||
# ID: nvme-local
|
||||
# Volume Group: nvme-pool
|
||||
# Thin Pool: data
|
||||
# Content: Disk image, Container template
|
||||
|
||||
# Option B: Directory (simpler, no thin provisioning)
|
||||
mkfs.ext4 /dev/nvme0n1
|
||||
mkdir -p /mnt/nvme
|
||||
mount /dev/nvme0n1 /mnt/nvme
|
||||
# Add to /etc/fstab for persistence
|
||||
# Then in Proxmox UI: Datacenter > Storage > Add > Directory
|
||||
# ID: nvme-local
|
||||
# Directory: /mnt/nvme
|
||||
# Content: Disk image, Container template
|
||||
```
|
||||
|
||||
3. Use for wnode-tsys5 boot disk (highest impact) and sectestbed VM scratch.
|
||||
|
||||
---
|
||||
|
||||
## 4. Post-hardware validation (run after Friday work)
|
||||
|
||||
1. Re-run iperf matrix: `./iperf-full-matrix.sh`
|
||||
2. Validate tsys4 and tsys5: `./validate-fixes.sh pfv-tsys4 && ./validate-fixes.sh pfv-tsys5`
|
||||
3. Update PROJECT.md with post-hardware iperf numbers.
|
||||
|
||||
---
|
||||
|
||||
## 5. UCS storage migration to spinning disk (do today)
|
||||
|
||||
UCS (Univention Corporate Server / open-source AD) does not need SSD. Both
|
||||
UCS VMs should stay on spinning disk (HDD) and be split across storage
|
||||
servers for redundancy.
|
||||
|
||||
| VM | Current | Target | Action |
|
||||
|----|---------|--------|--------|
|
||||
| ucs-01 (108) | D2 (tsys4 HDD) | **D2 (tsys4 HDD) -- no change** | Already correct |
|
||||
| ucs-02 (902) | D5 (tsys4 HDD) | **S2 (tsys5 HDD)** | Move for cross-server redundancy |
|
||||
|
||||
**To migrate ucs-02 to S2 (use PDM/Proxmox UI):**
|
||||
1. In Proxmox Datacenter or the node UI, select VM 902 on tsys9
|
||||
2. Use "Migrate" or "Storage Migrate" to move the disk from D5 to S2
|
||||
(both are NFS exports visible to tsys9, so this is a storage-only migration)
|
||||
3. Verify VM 902 boots and LDAP/AD services are healthy after migration
|
||||
|
||||
Note: both VMs are currently on tsys4 HDD, which is fine for UCS. Only
|
||||
ucs-02 needs to move -- it should be on a different storage server than
|
||||
ucs-01 so a tsys4 failure doesn't take down both halves of the AD pair.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions for next session
|
||||
|
||||
- Are the hosts a Proxmox cluster (`pvecm status`) or standalone installs?
|
||||
Determines whether live migration is available.
|
||||
- What k8s distribution is in use? (k3s, kubeadm, RKE2?)
|
||||
- Container runtime? (containerd, cri-o?)
|
||||
- Is there a local container image registry mirror?
|
||||
- What specific ETL tools? (GDAL, PostGIS, xarray, Dask?)
|
||||
- HPC job scheduler? (plain k8s Jobs, Argo Workflows, Volcano?)
|
||||
- What uses tsys5's SDR + parallel port before planning tsys5 role changes?
|
||||
- tsys3 thermal state (laptop in rack for years) — check `sensors`.
|
||||
Reference in New Issue
Block a user