diff --git a/STATUS.md b/STATUS.md index f2bcfdd..9d6e8e2 100644 --- a/STATUS.md +++ b/STATUS.md @@ -35,6 +35,16 @@ infrastructure, and Proxmox cluster ops. - [ ] tsys5: blocked on 2nd ethernet cable + NVMe install - [ ] tsys2: pending rebuild from Win10 to Proxmox +### Kubernetes Architecture (docs authored, pending implementation) +- [x] `docs/k8s/` created: README, DISTRO-DECISION, ARCHITECTURE +- [x] Distro decision: **Talos Linux** (over k3s), driven by ITAR/classified + requirement. Immutable + API-only + measured-boot = structurally stronger + compliance posture than SCAP-hardened Debian. +- [x] Target arch documented: 3 cnodes (local-lvm boot), Cilium CNI, vcluster + per tenant, Harbor pull-through cache on D3 SSD, OIDC→Keycloak (Cloudron), + Tailscale subnet-router for admin (cluster nodes LAN-only, zero egress). +- [ ] 3 decisions pending user confirm (see Pending below) + ### Repo Merge - [x] KNELServerBuild merged into PFVCluster (history preserved) - [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/) @@ -50,10 +60,17 @@ infrastructure, and Proxmox cluster ops. ## Pending (next session priorities) -1. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy -2. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes -3. Perf: complete tsys4/5 tuning after hardware install (Friday) -4. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host) +1. **k8s: confirm 3 decisions** before bootstrap: + - Cnode count: 3 (recommended) vs 5 + - Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm) + - Tailscale pattern: subnet router on `tailscale-router` (recommended) +2. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy +3. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes +4. Perf: complete tsys4/5 tuning after hardware install (Friday) +5. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host) +6. **k8s deferred topics** (tracked for future sessions): ETL tooling + (GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano), + per-tenant vcluster policy templates, solar-aware scale-out ## Infrastructure Summary diff --git a/docs/docmap.md b/docs/docmap.md index 2c6f55a..c49633e 100644 --- a/docs/docmap.md +++ b/docs/docmap.md @@ -5,6 +5,17 @@ > **Last updated:** 2026-07-28 +## Kubernetes Architecture ([`k8s/`](k8s/)) + +Distro decision, target architecture, control-plane design, bootstrap and DR +procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC). + +| Document | Description | Last Reviewed | +|----------|-------------|---------------| +| [`k8s/README.md`](k8s/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 | +| [`k8s/DISTRO-DECISION.md`](k8s/DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: Talos, driven by ITAR/classified requirement | 2026-07-28 | +| [`k8s/ARCHITECTURE.md`](k8s/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 | + ## Proxmox Cluster ([`proxmox/`](proxmox/)) Fleet operations, hardware, performance tuning, storage architecture. @@ -13,7 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture. |----------|-------------|---------------| | [`proxmox/PROJECT.md`](proxmox/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 | | [`proxmox/TODO.md`](proxmox/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 | -| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes architecture deep-dive (for future k8s session) | 2026-07-27 | +| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 | ## Server Build ([`server-build/`](server-build/)) diff --git a/docs/k8s/ARCHITECTURE.md b/docs/k8s/ARCHITECTURE.md new file mode 100644 index 0000000..9df9c3f --- /dev/null +++ b/docs/k8s/ARCHITECTURE.md @@ -0,0 +1,673 @@ +# pfv-k8s Target Architecture (Talos) + +> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos), +> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the +> Proxmox audit). + +**Last updated:** 2026-07-28 + +--- + +## Table of Contents + +1. [High-Level Architecture](#1-high-level-architecture) +2. [Control Plane](#2-control-plane) +3. [Network Topology](#3-network-topology) +4. [CNI: Cilium](#4-cni-cilium) +5. [Identity and Trust](#5-identity-and-trust) +6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster) +7. [Storage Integration](#7-storage-integration) +8. [Local Image Registry](#8-local-image-registry) +9. [Bootstrap Procedure](#9-bootstrap-procedure) +10. [Disaster Recovery](#10-disaster-recovery) +11. [Migration from Current State](#11-migration-from-current-state) + +--- + +## 1. High-Level Architecture + +```mermaid +flowchart TB + subgraph RESIDENCE["Residence — Proxmox LAN"] + subgraph CP["Talos Control Plane (3 cnodes)"] + C1[cnode1
tsys9 · local-SSD] + C2[cnode2
tsys9 · local-SSD] + C3[cnode3
tsys1 · local-HDD] + end + subgraph WP["Talos Worker Plane"] + W3[wnode-tsys3
NVMe · 28GB] + W5[wnode-tsys5
NVMe · 32-64GB] + W6[wnode-tsys6
NFS-HDD · 64-96GB] + W7[wnode-tsys7
NFS-HDD · 96-128GB] + W9[wnode-tsys9
local-SSD · 4-8GB] + end + ETCD[(etcd
raft, mTLS)] + REG[(Harbor registry
on D3 SSD · tsys5)] + BASTION[tailscale-router VM
subnet router] + end + + subgraph TAILNET["Tailscale overlay"] + OP[Operator devices] + end + + subgraph CLOUDRON["Cloudron production — Reston VA"] + KC[Keycloak OIDC IdP] + end + + C1 ---|mTLS LAN| ETCD + C2 ---|mTLS LAN| ETCD + C3 ---|mTLS LAN| ETCD + CP -->|pull images| REG + WP -->|pull images| REG + + OP -->|Talos API :50000
via subnet route| BASTION + BASTION -.->|LAN| CP + CP -->|OIDC| KC + WP -->|OIDC| KC + + classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff + classDef infra fill:#0f3460,stroke:#e94560,color:#fff + classDef external fill:#16213e,stroke:#533483,color:#fff + class CP,WP,ETCD talos + class REG,BASTION infra + class OP,KC,EXTERNAL external +``` + +### Design principles + +1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes. + Strongest posture for ITAR/classified. +2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM + advertises the cluster LAN subnet. Operator reaches Talos API from + anywhere. +3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency + for etcd). Workers boot from local disk where available; NFS for bulk + data only. +4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top + of the Talos host cluster. +5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak + tokens. No long-lived static credentials for humans. + +--- + +## 2. Control Plane + +### 2.1 Recommendation: 3 cnodes (down from 5) + +| Option | Quorum | Failure tolerance | etcd write cost | Resource cost | +|--------|--------|-------------------|-----------------|---------------| +| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB | +| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB | + +For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The +failure-tolerance jump from 1→2 rarely justifies the doubled etcd write +quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their +host capacity) are better spent on tenant worker allocations. + +**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure +tolerance on the control plane, keep 5. Otherwise 3. + +### 2.2 Cnode placement + +Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use +**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement: + +| cnode | Host | Boot disk | Type | Why | +|-------|------|-----------|------|-----| +| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. | +| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). | +| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. | + +**Quorum survival:** + +| Failure | cnodes lost | Quorum OK? | +|---------|-------------|------------| +| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. | +| tsys1 host dies | cnode3 | YES (2 of 3) | +| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected | + +**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure +loses quorum. Alternative: spread cnodes across 3 different hosts. See +"open question" at end of this section. + +### 2.3 Machine config strategy + +Talos nodes are configured by **machine configs** (YAML). Two flavors: + +- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler, + controller-manager, API server. +- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd. + +Strategy for this cluster: + +1. **One shared `talosconfig`** (client identity) — stored in 1Password + and in the Proxmox Backup Server (PBS) encrypted backup target. +2. **Per-node machine config patches** — small patches on top of the base + `controlplane.yaml` / `worker.yaml` for node-specific settings: + - Hostname + - Network interface + IP (DHCP or static — recommend static for cnodes) + - Schematic image digest (pinned Talos version) + - System extensions (e.g., `tailscale` — only if running Pattern A + instead of recommended Pattern C) +3. **All machine configs in Git** under a future `k8s/talos-configs/` + directory. Secrets are templated in at apply-time from 1Password / sops. + +```mermaid +flowchart LR + BASE[base controlplane.yaml] --> PATCH1[patch: cnode1] + BASE --> PATCH2[patch: cnode2] + BASE --> PATCH3[patch: cnode3] + BASEW[base worker.yaml] --> PATCHW[patch: per-wnode] + PATCH1 --> APPLY1[talosctl apply] + PATCH2 --> APPLY2[talosctl apply] + PATCH3 --> APPLY3[talosctl apply] + PATCHW --> APPLYW[talosctl apply] +``` + +### 2.4 Open question: cnode host spread + +If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D +cluster, tsys9 is brand-new hardware, single digit annual failure +probability), the layout in §2.2 is fine. + +If not, alternative spread across 3 hosts: + +| cnode | Host | Boot disk | +|-------|------|-----------| +| cnode1 | tsys9 | local-lvm SSD | +| cnode2 | tsys1 | local-lvm HDD | +| cnode3 | tsys3 | local-lvm NVMe | + +tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per +[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused, +would make an excellent etcd disk. + +**This 3-host spread survives any single host failure with quorum intact. +Recommended.** + +--- + +## 3. Network Topology + +### 3.1 Zones + +```mermaid +flowchart TB + subgraph INTERNET["Internet"] + FIBER[Gigabit symmetric fiber] + end + + subgraph RESLAN["Residence LAN 192.168.x.x/24"] + subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"] + CNODES[Cnodes 192.168.3.x] + WNODES[Wnodes 192.168.3.x] + end + BASTION[tailscale-router
192.168.3.x + 100.x.x.x] + REG[Harbor registry
192.168.3.x] + STORAGE[NFS servers
tsys4, tsys5] + end + + subgraph TSNET["Tailscale 100.x.x.x/8"] + OPS[Operator devices] + KC[Keycloak
via Cloudron prod] + end + + FIBER --> BASTION + BASTION <-. subnet route .-> CLUSTERNET + OPS -->|TCP 50000 talos API| BASTION + BASTION -->|LAN forward| CNODES + CNODES -->|LAN mTLS| WNODES + CNODES -->|OIDC HTTPS| KC + WNODES -->|pull images| REG + WNODES -->|bulk data IO| STORAGE + CNODES -->|pull images| REG +``` + +### 3.2 Address plan (suggested) + +Reserve a small contiguous block in the residence LAN for cluster nodes: + +| Role | Range | Count | +|------|-------|-------| +| Cnodes | `192.168.3.31-33` | 3 | +| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) | +| Bastion | existing `tailscale-router` | 1 | +| Registry | `192.168.3.50` | 1 (Harbor) | + +**Static IPs are strongly recommended for cnodes** (etcd cluster membership +is hostname-based; stable IPs make `talosctl` targeting simple). Workers +can DHCP. + +### 3.3 Firewall posture + +Each cnode/wnode has: + +- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on + cnodes only), plus CNI ports (varies by CNI — see §4). +- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale). +- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter + firewall for these IPs. ITAR workloads must not be able to phone home. + +The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes +via the subnet route. + +--- + +## 4. CNI: Cilium + +**Recommendation: Cilium** (eBPF-based CNI). + +| Property | Why it matters here | +|----------|---------------------| +| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. | +| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** | +| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. | +| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. | +| **Talos integration** | First-class. Talos docs document the install path. | + +Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption +enabled. Default-deny NetworkPolicy applied per namespace. + +--- + +## 5. Identity and Trust + +### 5.1 Trust flow + +```mermaid +sequenceDiagram + autonumber + participant Human as Operator + participant TAIL as Tailscale + participant BAST as Bastion + participant TALOS as Talos API :50000 + participant KC as Keycloak (Cloudron) + participant K8S as Kubernetes API :6443 + + Human->>TAIL: Authenticate (device + SSO) + TAIL-->>Human: Tailnet IP + Human->>BAST: Reach bastion via tailnet + BAST->>TALOS: Forward to LAN node :50000 + Human->>KC: OIDC login (browser) + KC-->>Human: Bearer token (short-lived) + Human->>TALOS: talosctl (mTLS with client cert) + Human->>K8S: kubectl --oidc (Keycloak token) + K8S->>KC: Validate token (introspection) + KC-->>K8S: Valid + claims + K8S-->>Human: Authorized response +``` + +### 5.2 Two distinct identity layers + +| Layer | Mechanism | Audience | +|-------|-----------|----------| +| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). | +| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). | + +The **Talos secrets bundle** is the root of trust for the cluster. Lose it +and you cannot operate the cluster; an attacker with it owns the cluster. +Storage: + +1. **Primary:** 1Password (or equivalent) — operator-accessible. +2. **Backup:** PBS encrypted backup target on tsys4 (existing infra). +3. **NOT in Git.** Machine configs go in Git; secrets stay out. + +### 5.3 Keycloak client configuration + +On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`: + +- **Authorization Code + PKCE flow** (no implicit, no password). +- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero + Omni/Rancher URLs if/when those are added. +- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`, + `k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`. + +--- + +## 6. Tenant Isolation (vcluster) + +### 6.1 Why vcluster + +[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control +plane** (API server, scheduler, controller-manager, etcd) inside a namespace +of the host cluster. Tenant workloads run on the host's worker nodes but +are isolated by: + +- Separate API server (tenant cannot see host cluster objects). +- Separate RBAC and admission control. +- Separate network policies (per-namespace). +- Separate resource quotas. + +This aligns with the user's per-tenant plan from +[`../proxmox/K8S.md`](../proxmox/K8S.md) §1. + +### 6.2 Tenant registry + +| Tenant | Compliance | Workload example | vcluster name | +|--------|-----------|------------------|---------------| +| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` | +| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` | +| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` | +| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` | + +```mermaid +flowchart TB + subgraph HOST["Talos host cluster"] + CP[Host control plane
3 cnodes · etcd · Keycloak RBAC] + subgraph NS["Host cluster namespaces"] + NS_RR[ns: vc-rackrental] + NS_SO[ns: vc-suborbital-open] + NS_SI[ns: vc-suborbital-itar] + NS_SLP[ns: vc-slp] + end + end + + subgraph VRR["vcluster: vc-rackrental"] + API_RR[k8s API + etcd] + end + subgraph VSO["vcluster: vc-suborbital-open"] + API_SO[k8s API + etcd] + end + subgraph VSI["vcluster: vc-suborbital-itar"] + API_SI[k8s API + etcd] + end + subgraph VSLP["vcluster: vc-slp"] + API_SLP[k8s API + etcd] + end + + CP --> NS_RR & NS_SO & NS_SI & NS_SLP + NS_RR --> API_RR + NS_SO --> API_SO + NS_SI --> API_SI + NS_SLP --> API_SLP + + classDef itar fill:#3a0000,stroke:#ff0000,color:#fff + class NS_SI,API_SI itar +``` + +### 6.3 ITAR enforcement at host layer + +For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer +controls: + +- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to + ITAR workloads (subset of wnodes, marked in node labels). +- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar` + namespace. Allow only explicit destinations (registry, NFS for ITAR + data tier, Keycloak). +- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g., + `D3-itar` on tsys5) that no other tenant can mount. +- **Audit:** Hubble flows + auditd on the host worker nodes capture all + access to ITAR data. + +Rancher (or Sidero Omni) sits above this, presenting each tenant's +vcluster as a separate "cluster" in its UI, with Keycloak SSO gating +access per tenant group claim. + +--- + +## 7. Storage Integration + +Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses: + +| StorageClass | Provisioner | Backing | Speed | Use | +|--------------|------------|---------|-------|-----| +| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral | +| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets | +| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data | + +### 7.1 CSI driver notes + +- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs) + (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific + NFS server + base export path. +- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys + with one manifest. Uses wnode's kubelet root dir. + +### 7.2 ITAR data isolation + +The ITAR tenant should target a dedicated NFS export, not shared +`nfs-hdd`. Recommended: + +- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as + `nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar` + namespace via RBAC + NetworkPolicy. + +--- + +## 8. Local Image Registry + +### 8.1 Recommendation: Harbor on D3 SSD + +D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it +for a **Harbor** instance: + +| Property | Value | +|----------|-------| +| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) | +| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data | +| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io
(b) Host private images
(c) Cosign image signing verification | +| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. | + +### 8.2 Pull-through cache benefit + +Cluster nodes have zero internet egress (per §3.3). Without a local cache, +image pulls fail. With Harbor as a pull-through cache: + +```mermaid +sequenceDiagram + WNODE->>HARBOR: docker pull nginx:1.25 + alt cache hit + HARBOR-->>WNODE: layer bytes (LAN-speed) + else cache miss + HARBOR->>DOCKERHUB: pull nginx:1.25 (egress) + DOCKERHUB-->>HARBOR: layer bytes + HARBOR-->>WNODE: layer bytes (cached for next time) + end +``` + +Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only +machine in the cluster with container-registry internet egress, and that +egress can be locked to specific upstreams (docker.io, quay.io, gcr.io, +ghcr.io). + +### 8.3 Supply-chain integrity (future) + +Harbor + Cosign lets you require that all images deployed to the ITAR +tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation +control. Implementation deferred to a later session. + +--- + +## 9. Bootstrap Procedure + +### 9.1 One-time setup + +```mermaid +sequenceDiagram + autonumber + participant OP as Operator + participant GIT as Git repo + participant ONEPW as 1Password + participant PBS as PBS (tsys4) + OP->>GIT: Clone PFVCluster repo + OP->>ONEPW: Generate Talos secrets bundle (offline) + ONEPW-->>OP: secrets.yaml + OP->>PBS: Backup secrets.yaml (encrypted) + OP->>GIT: Write machine configs (no secrets) +``` + +### 9.2 Provision first cnode (bootstrap) + +```mermaid +sequenceDiagram + autonumber + participant OP as Operator + participant PX as Proxmox host + participant C1 as cnode1 + participant ETCD as etcd (new) + + OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm) + OP->>PX: qm start VMID + C1->>C1: Boots Talos (no config yet, "maintenance mode") + OP->>C1: talosctl apply --patch cnode1.yaml (with secrets) + C1->>C1: Applies config, restarts services + OP->>C1: talosctl bootstrap + C1->>ETCD: Initialize single-node raft + ETCD-->>C1: ready + OP->>C1: talosctl kubeconfig (fetch admin kubeconfig) + OP->>C1: talosctl etcd snapshot (initial backup → PBS) +``` + +### 9.3 Add second and third cnodes + +```mermaid +sequenceDiagram + autonumber + participant OP as Operator + participant PX as Proxmox host + participant C2 as cnode2 + participant C3 as cnode3 + participant C1 as cnode1 (existing) + + OP->>PX: qm create + start cnode2 VM + C2->>C2: Boots Talos maintenance mode + OP->>C2: talosctl apply --patch cnode2.yaml + C2->>C1: Join etcd cluster + OP->>PX: qm create + start cnode3 VM + C3->>C3: Boots Talos maintenance mode + OP->>C3: talosctl apply --patch cnode3.yaml + C3->>C1: Join etcd cluster + Note over C1,C3: etcd now has 3/3 members → HA quorum +``` + +### 9.4 Post-bootstrap cluster configuration + +Once 3 cnodes are up and joined: + +1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption. +2. **Install CSI drivers** — nfs-csi + local-path provisioner. +3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`. +4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`. +5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3). +6. **Apply default-deny NetworkPolicy** in all namespaces. +7. **Install vcluster CLI** + create 4 tenant vclusters (§6). +8. **First etcd snapshot** + automated daily snapshot cron → PBS. + +### 9.5 Add workers + +Workers are simpler (no etcd): + +```mermaid +sequenceDiagram + OP->>PX: qm create + start wnode-X VM (Talos QCOW2) + WNODE->>WNODE: Boots maintenance mode + OP->>WNODE: talosctl apply --patch worker-X.yaml + WNODE->>C1: Kubelet registers with API server + C1-->>WNODE: Approved (auto via bootstrap token) + Note over WNODE: Joins cluster, becomes Ready +``` + +--- + +## 10. Disaster Recovery + +### 10.1 Backup strategy + +| Artifact | Frequency | Storage | Tool | +|----------|-----------|---------|------| +| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual | +| **Machine configs** | Continuous (Git) | Git remote + PBS | Git | +| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` | +| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` | +| **Harbor metadata** | Daily | PBS | Harbor built-in backup | + +### 10.2 Restore scenarios + +**Lost 1 cnode (e.g., tsys9 disk failure):** + +1. Provision new VM on tsys9 (or other host with local SSD). +2. Apply cnode2 machine config patch. +3. New cnode joins etcd, syncs state from survivors. +4. Quorum was never lost (2 of 3 alive throughout). + +**Lost 2 cnodes simultaneously (quorum lost):** + +1. Use surviving cnode's etcd snapshot. +2. Provision 3 new cnode VMs. +3. On first: `talosctl bootstrap --recover-from=snapshot.db`. +4. Join other 2 cnodes. +5. Workers reconnect automatically once API server is back. + +**Total cluster loss (all 3 cnodes):** + +1. Restore from latest PBS etcd snapshot. +2. Provision new cnode VMs. +3. `talosctl bootstrap --recover-from=snapshot.db`. +4. Re-join workers. +5. Verify tenant vclusters restored. + +### 10.3 Recovery time objectives + +| Scenario | RTO | RPO | +|----------|-----|-----| +| Single cnode failure | < 30 min | 0 (no data loss) | +| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) | +| Total cluster loss | < 4 hours | ≤ 24 hours | + +--- + +## 11. Migration from Current State + +### 11.1 Current state + +- 5 cnode VMs exist (Debian stock + Tailscale). +- **No k3s deployed yet.** Cluster was never bootstrapped. +- 6 wnode VMs exist (some stopped). +- No workloads running in k8s. + +### 11.2 Migration: clean cutover (not a migration) + +Since there is no etcd data and no workloads to preserve, the path is a +**clean rebuild**: + +| Phase | Action | Risk | +|-------|--------|------| +| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. | +| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. | +| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. | +| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. | +| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. | +| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). | + +### 11.3 Open dependency: Friday hardware work + +Phases 1-2 require: + +- **tsys3 local-lvm available.** Per + [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB + free NVMe local-lvm. Currently unused. **Ready.** +- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.** +- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per + [`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday. + Harbor depends on D3 being available on tsys5. + +Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the +Harbor registry does. + +--- + +## Appendix: Open questions for next session + +1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3. +2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread + (tsys9, tsys1, tsys3). +3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`. +4. **Rancher vs Sidero Omni** for cluster management UI. Both viable. + Defer until cluster is up. +5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs + approval of `192.168.3.0/24` route advertisement. +6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR? + Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as + general capacity; tsys3 + tsys5 (local fast storage) reserved for + non-ITAR HPC. diff --git a/docs/k8s/DISTRO-DECISION.md b/docs/k8s/DISTRO-DECISION.md new file mode 100644 index 0000000..f04d5eb --- /dev/null +++ b/docs/k8s/DISTRO-DECISION.md @@ -0,0 +1,188 @@ +# Distro Decision: Talos Linux vs k3s + +> **Recommendation: Talos Linux.** +> The k3s-on-Debian plan was sound before the ITAR/classified requirement +> entered scope. Once classified workloads are on the table, Talos's +> immutable, API-only, measured-boot-capable posture is materially easier +> to certify and defend. + +**Last updated:** 2026-07-28 + +--- + +## 1. Decision context + +| Factor | Constraint | +|--------|-----------| +| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) | +| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) | +| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. | +| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). | +| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. | +| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. | + +--- + +## 2. Head-to-head comparison + +### 2.1 ITAR / classified posture + +| Property | Talos Linux | k3s on Debian | +|----------|-------------|---------------| +| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. | +| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. | +| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. | +| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. | +| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. | +| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). | +| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. | +| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. | + +**Bottom line:** For classified workloads, an auditor's first question is +"how do you prevent unauthorized changes to a node?" Talos's answer is +"the OS is immutable and the only path is a signed API call." k3s's answer +is"SSH is locked down and we scan with STIG." The first is structurally +stronger; the second is operationally maintained. + +### 2.2 Operational considerations + +| Property | Talos | k3s | +|----------|-------|-----| +| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. | +| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. | +| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. | +| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). | +| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. | +| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. | +| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. | + +### 2.3 Cost of choosing Talos over the existing k3s plan + +The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically, +**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore: + +- **No etcd data to migrate.** Clean cutover, not a migration. +- **No workloads to drain.** The cluster is empty. +- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch — + either way it's a `qm` script, not a stateful migration). +- **Tailscale config shifts** from "installed via apt" to "Talos system + extension." (Or, per our recommendation in + [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster + nodes entirely** and onto the existing `tailscale-router` bastion as a + subnet router. Cluster nodes become LAN-only.) + +**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of +machine config YAML. The hardening investment already encoded in +`provisioning/Modules/Security/` is **not wasted** — it still applies to +every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only +the cnodes/wnodes move to Talos. + +--- + +## 3. Tailscale compatibility (deep-dive) + +Tailscale on Talos is well-supported but introduces a configuration dimension +worth being explicit about. Three patterns exist: + +### Pattern A — Tailscale on every cluster node (what you have now, on Debian) + +Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have +internet egress (to Tailscale DERP servers and for coordinate). + +- **Talos implementation:** add `siderolabs/tailscale` system extension to + each machine config, configure `machine.network.interfaces`. +- **Pros:** Operator can hit any node's Talos API from any Tailscale device. +- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a + finding (data exfiltration path). + +### Pattern B — Tailscale on bastion only, SSH/API jump + +Cluster nodes are LAN-only. Operator Tunnels to bastion (existing +`tailscale-router` VM), then runs `talosctl` from the bastion. + +- **Pros:** Zero internet egress from cluster nodes. +- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator + action originates from the bastion (auditable but clunky). + +### Pattern C — Tailscale subnet router on bastion (recommended) + +The existing `tailscale-router` VM advertises the cluster LAN subnet +(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's +Tailscale client transparently routes cluster-bound traffic through the +bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x` +"just works." + +- **Pros:** + - Cluster nodes have **zero internet egress** (strongest ITAR posture). + - Operator UX is unchanged from direct LAN access. + - All access is mediated by Tailscale's identity + ACLs (already integrated + with your env). + - Audit trail lives in Tailscale + bastion logs. +- **Cons:** + - Bastion becomes a dependency for remote admin (LAN-local admin still + works without it). + - Must enable IP forwarding + subnet route approval in Tailscale ACLs. + +**Recommendation: Pattern C.** Documented in +[`ARCHITECTURE.md`](ARCHITECTURE.md) §3. + +--- + +## 4. Recommendation + +**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`. + +### Justification + +1. **Compliance posture is structural, not operational.** "Immutable, + API-only, measured-boot" is a property of Talos itself; "STIG-hardened" + is a property of how Debian is operated. The first is dramatically + easier to argue to an ITAR counsel or classified accreditation officer + (DSS, DCSA) than the second. +2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging + 3 cnodes with Talos is a `qm` script invocation, not a stateful + migration. The sunk cost of "we planned k3s" is **zero deployed state**. +3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` + + `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to + learn but more error-prone to operate at HA. + +### Acknowledged tradeoffs + +- **Learning curve.** The Talos mental model (`machine config` + `talosctl`) + replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops. +- **No shell debugging.** When something breaks on a node, you cannot `ssh` + in. Mitigation: `talosctl logs/support` produces a support bundle + equivalent to a sosreport. +- **Hardware/module surprises.** Talos ships a curated kernel. Anything + beyond virtio + common NIC drivers needs a system extension. On Proxmox + VMs this is **not expected to be a problem** — virtio is the path. +- **Tailscale via system extension.** Adds one config dimension per node. + Mitigated by Pattern C (above), which removes Tailscale from cluster + nodes entirely. + +### What we keep from the k3s mental model + +- **Single binary on each node** semantics (Talos is conceptually similar). +- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes + API. `kubectl`, `helm`, `kustomize` all work as-is. +- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are + distro-independent. + +--- + +## 5. What we are NOT deciding here + +| Topic | Deferred to | +|--------|-------------| +| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design | +| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy | +| Per-tenant vcluster policy templates | Future session, post-bootstrap | +| Solar-aware scale-out hosts | Future capacity planning session | +| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) | + +--- + +## 6. Next step + +Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane +design, network topology, identity flow, and bootstrap procedure. diff --git a/docs/k8s/README.md b/docs/k8s/README.md new file mode 100644 index 0000000..16d4d07 --- /dev/null +++ b/docs/k8s/README.md @@ -0,0 +1,44 @@ +# Kubernetes Architecture & Build Plan + +> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md) +> (which captured the storage/host analysis from the Proxmox audit). +> This directory takes the next step: **which distro, how to build it, +> how to operate it.** + +**Last updated:** 2026-07-28 + +## Documents in this directory + +| Document | Purpose | +|----------|---------| +| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. | +| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. | + +## TL;DR + +| Decision | Recommendation | Why | +|----------|----------------|-----| +| **Distro** | **Talos Linux** (not k3s) | Immutable, API-only, measured-boot-capable. Intrinsically close to a STIG/CIS posture, where k3s-on-Debian requires constant re-certification. The k3s plan was reasonable before the ITAR/classified requirement was on the table. | +| **Runtime** | **containerd** (Talos default) | Only runtime Talos supports; also the secure default. Non-decision. | +| **Cnode count** | **3** (down from 5) | Standard HA. Tolerates 1 failure. Frees 2 VMs for tenant worker capacity. Re-evaluate if a regulator mandates 5. | +| **Admin access** | **Tailscale subnet router** on existing `tailscale-router` VM | Cluster nodes stay LAN-only (no internet egress = stronger ITAR posture). Operator reaches Talos API from anywhere via Tailscale. | +| **Cluster network** | LAN mTLS only | All cnode/wnode traffic on Proxmox LAN. No Tailscale on cluster nodes themselves. | +| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Already planned. Talos API + Kubernetes API both trust Keycloak tokens. | +| **Multi-tenancy** | **vcluster** (per tenant) inside Talos host cluster | Strong workload isolation. One vcluster per tenant: ITAR, non-ITAR, SLP, RackRental. | +| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Avoids needless round-trips to Docker Hub / Cloudron. Adds image-signing (Cosign) capability for supply-chain integrity. | +| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Unchanged. | + +## What this directory does NOT cover (deferred) + +- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design. +- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy. +- vcluster per-tenant policy templates. +- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only. + +These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md). + +## Open question for the user + +1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in + [`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires + 2-failure tolerance on the control plane, keep 5.