KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(k8s): add Talos architecture, distro decision, and bootstrap plan

Browse Source 
Author the docs/k8s/ directory capturing the pfv-k8s control-plane design:

- README.md: TL;DR of all decisions (distro, runtime, cnode count, admin
  access, identity, tenancy, registry, storage)
- DISTRO-DECISION.md: Talos vs k3s analysis. Recommend Talos because the
  ITAR/classified requirement makes its immutable, API-only, measured-boot
  posture structurally easier to certify than SCAP-hardened Debian. k3s was
  only ever a plan (no cluster deployed), so cutover cost is ~zero.
- ARCHITECTURE.md: target arch with mermaid diagrams covering control
  plane, LAN-only network with Tailscale subnet-router admin, Cilium CNI,
  OIDC to Keycloak, per-tenant vcluster isolation (incl. ITAR tenant),
  Harbor pull-through cache on D3 SSD, bootstrap sequence, and DR.

Gardening: register docs/k8s/ in docmap.md, update STATUS.md with the new
k8s section and the three pending user decisions (cnode count, host spread,
Tailscale pattern).

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-28 11:50:56 -05:00
parent 99270c344d
commit 174deea2a9
5 changed files with 938 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/k8s/ARCHITECTURE.md
+673
View File
@@ -0,0 +1,673 @@
# pfv-k8s Target Architecture (Talos)
> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
> Proxmox audit).
**Last updated:** 2026-07-28
---
## Table of Contents
1. [High-Level Architecture](#1-high-level-architecture)
2. [Control Plane](#2-control-plane)
3. [Network Topology](#3-network-topology)
4. [CNI: Cilium](#4-cni-cilium)
5. [Identity and Trust](#5-identity-and-trust)
6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
7. [Storage Integration](#7-storage-integration)
8. [Local Image Registry](#8-local-image-registry)
9. [Bootstrap Procedure](#9-bootstrap-procedure)
10. [Disaster Recovery](#10-disaster-recovery)
11. [Migration from Current State](#11-migration-from-current-state)
---
## 1. High-Level Architecture
```mermaid
flowchart TB
subgraph RESIDENCE["Residence — Proxmox LAN"]
subgraph CP["Talos Control Plane (3 cnodes)"]
C1[cnode1<br/>tsys9 · local-SSD]
C2[cnode2<br/>tsys9 · local-SSD]
C3[cnode3<br/>tsys1 · local-HDD]
end
subgraph WP["Talos Worker Plane"]
W3[wnode-tsys3<br/>NVMe · 28GB]
W5[wnode-tsys5<br/>NVMe · 32-64GB]
W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
W9[wnode-tsys9<br/>local-SSD · 4-8GB]
end
ETCD[(etcd<br/>raft, mTLS)]
REG[(Harbor registry<br/>on D3 SSD · tsys5)]
BASTION[tailscale-router VM<br/>subnet router]
end
subgraph TAILNET["Tailscale overlay"]
OP[Operator devices]
end
subgraph CLOUDRON["Cloudron production — Reston VA"]
KC[Keycloak OIDC IdP]
end
C1 ---|mTLS LAN| ETCD
C2 ---|mTLS LAN| ETCD
C3 ---|mTLS LAN| ETCD
CP -->|pull images| REG
WP -->|pull images| REG
OP -->|Talos API :50000<br/>via subnet route| BASTION
BASTION -.->|LAN| CP
CP -->|OIDC| KC
WP -->|OIDC| KC
classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
classDef infra fill:#0f3460,stroke:#e94560,color:#fff
classDef external fill:#16213e,stroke:#533483,color:#fff
class CP,WP,ETCD talos
class REG,BASTION infra
class OP,KC,EXTERNAL external
```
### Design principles
1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
Strongest posture for ITAR/classified.
2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
advertises the cluster LAN subnet. Operator reaches Talos API from
anywhere.
3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
for etcd). Workers boot from local disk where available; NFS for bulk
data only.
4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
of the Talos host cluster.
5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
tokens. No long-lived static credentials for humans.
---
## 2. Control Plane
### 2.1 Recommendation: 3 cnodes (down from 5)
| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
|--------|--------|-------------------|-----------------|---------------|
| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
host capacity) are better spent on tenant worker allocations.
**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
tolerance on the control plane, keep 5. Otherwise 3.
### 2.2 Cnode placement
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
| cnode | Host | Boot disk | Type | Why |
|-------|------|-----------|------|-----|
| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
**Quorum survival:**
| Failure | cnodes lost | Quorum OK? |
|---------|-------------|------------|
| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
| tsys1 host dies | cnode3 | YES (2 of 3) |
| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
loses quorum. Alternative: spread cnodes across 3 different hosts. See
"open question" at end of this section.
### 2.3 Machine config strategy
Talos nodes are configured by **machine configs** (YAML). Two flavors:
- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
controller-manager, API server.
- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
Strategy for this cluster:
1. **One shared `talosconfig`** (client identity) — stored in 1Password
and in the Proxmox Backup Server (PBS) encrypted backup target.
2. **Per-node machine config patches** — small patches on top of the base
`controlplane.yaml` / `worker.yaml` for node-specific settings:
- Hostname
- Network interface + IP (DHCP or static — recommend static for cnodes)
- Schematic image digest (pinned Talos version)
- System extensions (e.g., `tailscale` — only if running Pattern A
instead of recommended Pattern C)
3. **All machine configs in Git** under a future `k8s/talos-configs/`
directory. Secrets are templated in at apply-time from 1Password / sops.
```mermaid
flowchart LR
BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
BASE --> PATCH2[patch: cnode2]
BASE --> PATCH3[patch: cnode3]
BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
PATCH1 --> APPLY1[talosctl apply]
PATCH2 --> APPLY2[talosctl apply]
PATCH3 --> APPLY3[talosctl apply]
PATCHW --> APPLYW[talosctl apply]
```
### 2.4 Open question: cnode host spread
If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
cluster, tsys9 is brand-new hardware, single digit annual failure
probability), the layout in §2.2 is fine.
If not, alternative spread across 3 hosts:
| cnode | Host | Boot disk |
|-------|------|-----------|
| cnode1 | tsys9 | local-lvm SSD |
| cnode2 | tsys1 | local-lvm HDD |
| cnode3 | tsys3 | local-lvm NVMe |
tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
would make an excellent etcd disk.
**This 3-host spread survives any single host failure with quorum intact.
Recommended.**
---
## 3. Network Topology
### 3.1 Zones
```mermaid
flowchart TB
subgraph INTERNET["Internet"]
FIBER[Gigabit symmetric fiber]
end
subgraph RESLAN["Residence LAN 192.168.x.x/24"]
subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
CNODES[Cnodes 192.168.3.x]
WNODES[Wnodes 192.168.3.x]
end
BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
REG[Harbor registry<br/>192.168.3.x]
STORAGE[NFS servers<br/>tsys4, tsys5]
end
subgraph TSNET["Tailscale 100.x.x.x/8"]
OPS[Operator devices]
KC[Keycloak<br/>via Cloudron prod]
end
FIBER --> BASTION
BASTION <-. subnet route .-> CLUSTERNET
OPS -->|TCP 50000 talos API| BASTION
BASTION -->|LAN forward| CNODES
CNODES -->|LAN mTLS| WNODES
CNODES -->|OIDC HTTPS| KC
WNODES -->|pull images| REG
WNODES -->|bulk data IO| STORAGE
CNODES -->|pull images| REG
```
### 3.2 Address plan (suggested)
Reserve a small contiguous block in the residence LAN for cluster nodes:
| Role | Range | Count |
|------|-------|-------|
| Cnodes | `192.168.3.31-33` | 3 |
| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
| Bastion | existing `tailscale-router` | 1 |
| Registry | `192.168.3.50` | 1 (Harbor) |
**Static IPs are strongly recommended for cnodes** (etcd cluster membership
is hostname-based; stable IPs make `talosctl` targeting simple). Workers
can DHCP.
### 3.3 Firewall posture
Each cnode/wnode has:
- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
cnodes only), plus CNI ports (varies by CNI — see §4).
- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
firewall for these IPs. ITAR workloads must not be able to phone home.
The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
via the subnet route.
---
## 4. CNI: Cilium
**Recommendation: Cilium** (eBPF-based CNI).
| Property | Why it matters here |
|----------|---------------------|
| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
| **Talos integration** | First-class. Talos docs document the install path. |
Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
enabled. Default-deny NetworkPolicy applied per namespace.
---
## 5. Identity and Trust
### 5.1 Trust flow
```mermaid
sequenceDiagram
autonumber
participant Human as Operator
participant TAIL as Tailscale
participant BAST as Bastion
participant TALOS as Talos API :50000
participant KC as Keycloak (Cloudron)
participant K8S as Kubernetes API :6443
Human->>TAIL: Authenticate (device + SSO)
TAIL-->>Human: Tailnet IP
Human->>BAST: Reach bastion via tailnet
BAST->>TALOS: Forward to LAN node :50000
Human->>KC: OIDC login (browser)
KC-->>Human: Bearer token (short-lived)
Human->>TALOS: talosctl (mTLS with client cert)
Human->>K8S: kubectl --oidc (Keycloak token)
K8S->>KC: Validate token (introspection)
KC-->>K8S: Valid + claims
K8S-->>Human: Authorized response
```
### 5.2 Two distinct identity layers
| Layer | Mechanism | Audience |
|-------|-----------|----------|
| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
The **Talos secrets bundle** is the root of trust for the cluster. Lose it
and you cannot operate the cluster; an attacker with it owns the cluster.
Storage:
1. **Primary:** 1Password (or equivalent) — operator-accessible.
2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
3. **NOT in Git.** Machine configs go in Git; secrets stay out.
### 5.3 Keycloak client configuration
On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
- **Authorization Code + PKCE flow** (no implicit, no password).
- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
Omni/Rancher URLs if/when those are added.
- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
`k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
---
## 6. Tenant Isolation (vcluster)
### 6.1 Why vcluster
[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
plane** (API server, scheduler, controller-manager, etcd) inside a namespace
of the host cluster. Tenant workloads run on the host's worker nodes but
are isolated by:
- Separate API server (tenant cannot see host cluster objects).
- Separate RBAC and admission control.
- Separate network policies (per-namespace).
- Separate resource quotas.
This aligns with the user's per-tenant plan from
[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
### 6.2 Tenant registry
| Tenant | Compliance | Workload example | vcluster name |
|--------|-----------|------------------|---------------|
| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
```mermaid
flowchart TB
subgraph HOST["Talos host cluster"]
CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
subgraph NS["Host cluster namespaces"]
NS_RR[ns: vc-rackrental]
NS_SO[ns: vc-suborbital-open]
NS_SI[ns: vc-suborbital-itar]
NS_SLP[ns: vc-slp]
end
end
subgraph VRR["vcluster: vc-rackrental"]
API_RR[k8s API + etcd]
end
subgraph VSO["vcluster: vc-suborbital-open"]
API_SO[k8s API + etcd]
end
subgraph VSI["vcluster: vc-suborbital-itar"]
API_SI[k8s API + etcd]
end
subgraph VSLP["vcluster: vc-slp"]
API_SLP[k8s API + etcd]
end
CP --> NS_RR & NS_SO & NS_SI & NS_SLP
NS_RR --> API_RR
NS_SO --> API_SO
NS_SI --> API_SI
NS_SLP --> API_SLP
classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
class NS_SI,API_SI itar
```
### 6.3 ITAR enforcement at host layer
For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
controls:
- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
ITAR workloads (subset of wnodes, marked in node labels).
- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
namespace. Allow only explicit destinations (registry, NFS for ITAR
data tier, Keycloak).
- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
`D3-itar` on tsys5) that no other tenant can mount.
- **Audit:** Hubble flows + auditd on the host worker nodes capture all
access to ITAR data.
Rancher (or Sidero Omni) sits above this, presenting each tenant's
vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
access per tenant group claim.
---
## 7. Storage Integration
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
| StorageClass | Provisioner | Backing | Speed | Use |
|--------------|------------|---------|-------|-----|
| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
### 7.1 CSI driver notes
- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
(CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
NFS server + base export path.
- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
with one manifest. Uses wnode's kubelet root dir.
### 7.2 ITAR data isolation
The ITAR tenant should target a dedicated NFS export, not shared
`nfs-hdd`. Recommended:
- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
`nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
namespace via RBAC + NetworkPolicy.
---
## 8. Local Image Registry
### 8.1 Recommendation: Harbor on D3 SSD
D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
for a **Harbor** instance:
| Property | Value |
|----------|-------|
| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
### 8.2 Pull-through cache benefit
Cluster nodes have zero internet egress (per §3.3). Without a local cache,
image pulls fail. With Harbor as a pull-through cache:
```mermaid
sequenceDiagram
WNODE->>HARBOR: docker pull nginx:1.25
alt cache hit
HARBOR-->>WNODE: layer bytes (LAN-speed)
else cache miss
HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
DOCKERHUB-->>HARBOR: layer bytes
HARBOR-->>WNODE: layer bytes (cached for next time)
end
```
Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
machine in the cluster with container-registry internet egress, and that
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
ghcr.io).
### 8.3 Supply-chain integrity (future)
Harbor + Cosign lets you require that all images deployed to the ITAR
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
control. Implementation deferred to a later session.
---
## 9. Bootstrap Procedure
### 9.1 One-time setup
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant GIT as Git repo
participant ONEPW as 1Password
participant PBS as PBS (tsys4)
OP->>GIT: Clone PFVCluster repo
OP->>ONEPW: Generate Talos secrets bundle (offline)
ONEPW-->>OP: secrets.yaml
OP->>PBS: Backup secrets.yaml (encrypted)
OP->>GIT: Write machine configs (no secrets)
```
### 9.2 Provision first cnode (bootstrap)
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant PX as Proxmox host
participant C1 as cnode1
participant ETCD as etcd (new)
OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
OP->>PX: qm start VMID
C1->>C1: Boots Talos (no config yet, "maintenance mode")
OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
C1->>C1: Applies config, restarts services
OP->>C1: talosctl bootstrap
C1->>ETCD: Initialize single-node raft
ETCD-->>C1: ready
OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
OP->>C1: talosctl etcd snapshot (initial backup → PBS)
```
### 9.3 Add second and third cnodes
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant PX as Proxmox host
participant C2 as cnode2
participant C3 as cnode3
participant C1 as cnode1 (existing)
OP->>PX: qm create + start cnode2 VM
C2->>C2: Boots Talos maintenance mode
OP->>C2: talosctl apply --patch cnode2.yaml
C2->>C1: Join etcd cluster
OP->>PX: qm create + start cnode3 VM
C3->>C3: Boots Talos maintenance mode
OP->>C3: talosctl apply --patch cnode3.yaml
C3->>C1: Join etcd cluster
Note over C1,C3: etcd now has 3/3 members → HA quorum
```
### 9.4 Post-bootstrap cluster configuration
Once 3 cnodes are up and joined:
1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
2. **Install CSI drivers** — nfs-csi + local-path provisioner.
3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
6. **Apply default-deny NetworkPolicy** in all namespaces.
7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
8. **First etcd snapshot** + automated daily snapshot cron → PBS.
### 9.5 Add workers
Workers are simpler (no etcd):
```mermaid
sequenceDiagram
OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
WNODE->>WNODE: Boots maintenance mode
OP->>WNODE: talosctl apply --patch worker-X.yaml
WNODE->>C1: Kubelet registers with API server
C1-->>WNODE: Approved (auto via bootstrap token)
Note over WNODE: Joins cluster, becomes Ready
```
---
## 10. Disaster Recovery
### 10.1 Backup strategy
| Artifact | Frequency | Storage | Tool |
|----------|-----------|---------|------|
| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
| **Harbor metadata** | Daily | PBS | Harbor built-in backup |
### 10.2 Restore scenarios
**Lost 1 cnode (e.g., tsys9 disk failure):**
1. Provision new VM on tsys9 (or other host with local SSD).
2. Apply cnode2 machine config patch.
3. New cnode joins etcd, syncs state from survivors.
4. Quorum was never lost (2 of 3 alive throughout).
**Lost 2 cnodes simultaneously (quorum lost):**
1. Use surviving cnode's etcd snapshot.
2. Provision 3 new cnode VMs.
3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
4. Join other 2 cnodes.
5. Workers reconnect automatically once API server is back.
**Total cluster loss (all 3 cnodes):**
1. Restore from latest PBS etcd snapshot.
2. Provision new cnode VMs.
3. `talosctl bootstrap --recover-from=snapshot.db`.
4. Re-join workers.
5. Verify tenant vclusters restored.
### 10.3 Recovery time objectives
| Scenario | RTO | RPO |
|----------|-----|-----|
| Single cnode failure | < 30 min | 0 (no data loss) |
| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
| Total cluster loss | < 4 hours | ≤ 24 hours |
---
## 11. Migration from Current State
### 11.1 Current state
- 5 cnode VMs exist (Debian stock + Tailscale).
- **No k3s deployed yet.** Cluster was never bootstrapped.
- 6 wnode VMs exist (some stopped).
- No workloads running in k8s.
### 11.2 Migration: clean cutover (not a migration)
Since there is no etcd data and no workloads to preserve, the path is a
**clean rebuild**:
| Phase | Action | Risk |
|-------|--------|------|
| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
### 11.3 Open dependency: Friday hardware work
Phases 1-2 require:
- **tsys3 local-lvm available.** Per
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
free NVMe local-lvm. Currently unused. **Ready.**
- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
[`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
Harbor depends on D3 being available on tsys5.
Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
Harbor registry does.
---
## Appendix: Open questions for next session
1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
(tsys9, tsys1, tsys3).
3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
Defer until cluster is up.
5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
approval of `192.168.3.0/24` route advertisement.
6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
general capacity; tsys3 + tsys5 (local fast storage) reserved for
non-ITAR HPC.
docs/k8s/DISTRO-DECISION.md
+188
View File
@@ -0,0 +1,188 @@
# Distro Decision: Talos Linux vs k3s
> **Recommendation: Talos Linux.**
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
> entered scope. Once classified workloads are on the table, Talos's
> immutable, API-only, measured-boot-capable posture is materially easier
> to certify and defend.
**Last updated:** 2026-07-28
---
## 1. Decision context
| Factor | Constraint |
|--------|-----------|
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
---
## 2. Head-to-head comparison
### 2.1 ITAR / classified posture
| Property | Talos Linux | k3s on Debian |
|----------|-------------|---------------|
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
**Bottom line:** For classified workloads, an auditor's first question is
"how do you prevent unauthorized changes to a node?" Talos's answer is
"the OS is immutable and the only path is a signed API call." k3s's answer
is"SSH is locked down and we scan with STIG." The first is structurally
stronger; the second is operationally maintained.
### 2.2 Operational considerations
| Property | Talos | k3s |
|----------|-------|-----|
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
### 2.3 Cost of choosing Talos over the existing k3s plan
The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
- **No etcd data to migrate.** Clean cutover, not a migration.
- **No workloads to drain.** The cluster is empty.
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
either way it's a `qm` script, not a stateful migration).
- **Tailscale config shifts** from "installed via apt" to "Talos system
extension." (Or, per our recommendation in
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
nodes entirely** and onto the existing `tailscale-router` bastion as a
subnet router. Cluster nodes become LAN-only.)
**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
machine config YAML. The hardening investment already encoded in
`provisioning/Modules/Security/` is **not wasted** — it still applies to
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
the cnodes/wnodes move to Talos.
---
## 3. Tailscale compatibility (deep-dive)
Tailscale on Talos is well-supported but introduces a configuration dimension
worth being explicit about. Three patterns exist:
### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
internet egress (to Tailscale DERP servers and for coordinate).
- **Talos implementation:** add `siderolabs/tailscale` system extension to
each machine config, configure `machine.network.interfaces`.
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
finding (data exfiltration path).
### Pattern B — Tailscale on bastion only, SSH/API jump
Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
`tailscale-router` VM), then runs `talosctl` from the bastion.
- **Pros:** Zero internet egress from cluster nodes.
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
action originates from the bastion (auditable but clunky).
### Pattern C — Tailscale subnet router on bastion (recommended)
The existing `tailscale-router` VM advertises the cluster LAN subnet
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
Tailscale client transparently routes cluster-bound traffic through the
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
"just works."
- **Pros:**
- Cluster nodes have **zero internet egress** (strongest ITAR posture).
- Operator UX is unchanged from direct LAN access.
- All access is mediated by Tailscale's identity + ACLs (already integrated
with your env).
- Audit trail lives in Tailscale + bastion logs.
- **Cons:**
- Bastion becomes a dependency for remote admin (LAN-local admin still
works without it).
- Must enable IP forwarding + subnet route approval in Tailscale ACLs.
**Recommendation: Pattern C.** Documented in
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
---
## 4. Recommendation
**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
### Justification
1. **Compliance posture is structural, not operational.** "Immutable,
API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
is a property of how Debian is operated. The first is dramatically
easier to argue to an ITAR counsel or classified accreditation officer
(DSS, DCSA) than the second.
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
3 cnodes with Talos is a `qm` script invocation, not a stateful
migration. The sunk cost of "we planned k3s" is **zero deployed state**.
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
`apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
learn but more error-prone to operate at HA.
### Acknowledged tradeoffs
- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
in. Mitigation: `talosctl logs/support` produces a support bundle
equivalent to a sosreport.
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
beyond virtio + common NIC drivers needs a system extension. On Proxmox
VMs this is **not expected to be a problem** — virtio is the path.
- **Tailscale via system extension.** Adds one config dimension per node.
Mitigated by Pattern C (above), which removes Tailscale from cluster
nodes entirely.
### What we keep from the k3s mental model
- **Single binary on each node** semantics (Talos is conceptually similar).
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
API. `kubectl`, `helm`, `kustomize` all work as-is.
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
distro-independent.
---
## 5. What we are NOT deciding here
| Topic | Deferred to |
|--------|-------------|
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
| Solar-aware scale-out hosts | Future capacity planning session |
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
---
## 6. Next step
Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
design, network topology, identity flow, and bootstrap procedure.
docs/k8s/README.md
+44
View File
@@ -0,0 +1,44 @@
# Kubernetes Architecture & Build Plan
> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
> (which captured the storage/host analysis from the Proxmox audit).
> This directory takes the next step: **which distro, how to build it,
> how to operate it.**
**Last updated:** 2026-07-28
## Documents in this directory
| Document | Purpose |
|----------|---------|
| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
## TL;DR
| Decision | Recommendation | Why |
|----------|----------------|-----|
| **Distro** | **Talos Linux** (not k3s) | Immutable, API-only, measured-boot-capable. Intrinsically close to a STIG/CIS posture, where k3s-on-Debian requires constant re-certification. The k3s plan was reasonable before the ITAR/classified requirement was on the table. |
| **Runtime** | **containerd** (Talos default) | Only runtime Talos supports; also the secure default. Non-decision. |
| **Cnode count** | **3** (down from 5) | Standard HA. Tolerates 1 failure. Frees 2 VMs for tenant worker capacity. Re-evaluate if a regulator mandates 5. |
| **Admin access** | **Tailscale subnet router** on existing `tailscale-router` VM | Cluster nodes stay LAN-only (no internet egress = stronger ITAR posture). Operator reaches Talos API from anywhere via Tailscale. |
| **Cluster network** | LAN mTLS only | All cnode/wnode traffic on Proxmox LAN. No Tailscale on cluster nodes themselves. |
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Already planned. Talos API + Kubernetes API both trust Keycloak tokens. |
| **Multi-tenancy** | **vcluster** (per tenant) inside Talos host cluster | Strong workload isolation. One vcluster per tenant: ITAR, non-ITAR, SLP, RackRental. |
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Avoids needless round-trips to Docker Hub / Cloudron. Adds image-signing (Cosign) capability for supply-chain integrity. |
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Unchanged. |
## What this directory does NOT cover (deferred)
- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
- vcluster per-tenant policy templates.
- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
## Open question for the user
1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
[`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
2-failure tolerance on the control plane, keep 5.