Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 mrcharles
|
2d01a9f6be | ||
|
mrcharles
|
d65a5fa34c |
@@ -32,8 +32,13 @@ vendor/ Vendored KNELShellFramework
|
||||
|
||||
## Git Policy
|
||||
|
||||
1. **Autonomous commit + push.** Do not wait to be asked. After each logical
|
||||
unit: stage, commit (conventional format), push to `origin/main`.
|
||||
1. **ALWAYS commit + push. Never wait. Ever.** After each logical unit of
|
||||
work, immediately stage, commit (conventional format), and push to
|
||||
`origin/main`. Do NOT pause for review. Do NOT ask permission. Do NOT
|
||||
stop to "let the user read it first." The user reviews rendered markdown
|
||||
on Gitea AFTER it is pushed. **This rule overrides any default
|
||||
conservative commit-and-hold behavior** — if a default instruction says
|
||||
"never commit unless asked," that instruction is superseded here.
|
||||
2. **Atomic commits.** Each commit coherent on its own.
|
||||
3. **Conventional format**: `feat(scope): desc`, `fix(scope): desc`,
|
||||
`docs: desc`, `refactor(scope): desc`, `test(scope): desc`.
|
||||
|
||||
@@ -35,6 +35,16 @@ infrastructure, and Proxmox cluster ops.
|
||||
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
|
||||
- [ ] tsys2: pending rebuild from Win10 to Proxmox
|
||||
|
||||
### Kubernetes Architecture (docs authored, pending implementation)
|
||||
- [x] `docs/k8s/` created: README, DISTRO-DECISION, ARCHITECTURE
|
||||
- [x] Distro decision: **Talos Linux** (over k3s), driven by ITAR/classified
|
||||
requirement. Immutable + API-only + measured-boot = structurally stronger
|
||||
compliance posture than SCAP-hardened Debian.
|
||||
- [x] Target arch documented: 3 cnodes (local-lvm boot), Cilium CNI, vcluster
|
||||
per tenant, Harbor pull-through cache on D3 SSD, OIDC→Keycloak (Cloudron),
|
||||
Tailscale subnet-router for admin (cluster nodes LAN-only, zero egress).
|
||||
- [ ] 3 decisions pending user confirm (see Pending below)
|
||||
|
||||
### Repo Merge
|
||||
- [x] KNELServerBuild merged into PFVCluster (history preserved)
|
||||
- [x] Directory structure reorganized (provisioning/, tests/, perf/, docs/)
|
||||
@@ -50,10 +60,17 @@ infrastructure, and Proxmox cluster ops.
|
||||
|
||||
## Pending (next session priorities)
|
||||
|
||||
1. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
2. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
3. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
4. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
1. **k8s: confirm 3 decisions** before bootstrap:
|
||||
- Cnode count: 3 (recommended) vs 5
|
||||
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
|
||||
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
|
||||
2. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
3. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
4. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
5. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
6. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
||||
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
|
||||
per-tenant vcluster policy templates, solar-aware scale-out
|
||||
|
||||
## Infrastructure Summary
|
||||
|
||||
|
||||
+12
-1
@@ -5,6 +5,17 @@
|
||||
|
||||
> **Last updated:** 2026-07-28
|
||||
|
||||
## Kubernetes Architecture ([`k8s/`](k8s/))
|
||||
|
||||
Distro decision, target architecture, control-plane design, bootstrap and DR
|
||||
procedures for the pfv-k8s cluster (Talos + vcluster + Keycloak OIDC).
|
||||
|
||||
| Document | Description | Last Reviewed |
|
||||
|----------|-------------|---------------|
|
||||
| [`k8s/README.md`](k8s/README.md) | Index + TL;DR of all k8s decisions | 2026-07-28 |
|
||||
| [`k8s/DISTRO-DECISION.md`](k8s/DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: Talos, driven by ITAR/classified requirement | 2026-07-28 |
|
||||
| [`k8s/ARCHITECTURE.md`](k8s/ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation, bootstrap, DR. Mermaid diagrams | 2026-07-28 |
|
||||
|
||||
## Proxmox Cluster ([`proxmox/`](proxmox/))
|
||||
|
||||
Fleet operations, hardware, performance tuning, storage architecture.
|
||||
@@ -13,7 +24,7 @@ Fleet operations, hardware, performance tuning, storage architecture.
|
||||
|----------|-------------|---------------|
|
||||
| [`proxmox/PROJECT.md`](proxmox/PROJECT.md) | Comprehensive fleet report: 7 hosts, VM inventory, storage, recommendations | 2026-07-27 |
|
||||
| [`proxmox/TODO.md`](proxmox/TODO.md) | Pending physical hardware work (tsys2/4/5 Friday plan) | 2026-07-27 |
|
||||
| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes architecture deep-dive (for future k8s session) | 2026-07-27 |
|
||||
| [`proxmox/K8S.md`](proxmox/K8S.md) | Kubernetes storage/host analysis (predecessor to [`k8s/`](k8s/)) | 2026-07-27 |
|
||||
|
||||
## Server Build ([`server-build/`](server-build/))
|
||||
|
||||
|
||||
@@ -0,0 +1,673 @@
|
||||
# pfv-k8s Target Architecture (Talos)
|
||||
|
||||
> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
|
||||
> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
|
||||
> Proxmox audit).
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [High-Level Architecture](#1-high-level-architecture)
|
||||
2. [Control Plane](#2-control-plane)
|
||||
3. [Network Topology](#3-network-topology)
|
||||
4. [CNI: Cilium](#4-cni-cilium)
|
||||
5. [Identity and Trust](#5-identity-and-trust)
|
||||
6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
|
||||
7. [Storage Integration](#7-storage-integration)
|
||||
8. [Local Image Registry](#8-local-image-registry)
|
||||
9. [Bootstrap Procedure](#9-bootstrap-procedure)
|
||||
10. [Disaster Recovery](#10-disaster-recovery)
|
||||
11. [Migration from Current State](#11-migration-from-current-state)
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph RESIDENCE["Residence — Proxmox LAN"]
|
||||
subgraph CP["Talos Control Plane (3 cnodes)"]
|
||||
C1[cnode1<br/>tsys9 · local-SSD]
|
||||
C2[cnode2<br/>tsys9 · local-SSD]
|
||||
C3[cnode3<br/>tsys1 · local-HDD]
|
||||
end
|
||||
subgraph WP["Talos Worker Plane"]
|
||||
W3[wnode-tsys3<br/>NVMe · 28GB]
|
||||
W5[wnode-tsys5<br/>NVMe · 32-64GB]
|
||||
W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
|
||||
W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
|
||||
W9[wnode-tsys9<br/>local-SSD · 4-8GB]
|
||||
end
|
||||
ETCD[(etcd<br/>raft, mTLS)]
|
||||
REG[(Harbor registry<br/>on D3 SSD · tsys5)]
|
||||
BASTION[tailscale-router VM<br/>subnet router]
|
||||
end
|
||||
|
||||
subgraph TAILNET["Tailscale overlay"]
|
||||
OP[Operator devices]
|
||||
end
|
||||
|
||||
subgraph CLOUDRON["Cloudron production — Reston VA"]
|
||||
KC[Keycloak OIDC IdP]
|
||||
end
|
||||
|
||||
C1 ---|mTLS LAN| ETCD
|
||||
C2 ---|mTLS LAN| ETCD
|
||||
C3 ---|mTLS LAN| ETCD
|
||||
CP -->|pull images| REG
|
||||
WP -->|pull images| REG
|
||||
|
||||
OP -->|Talos API :50000<br/>via subnet route| BASTION
|
||||
BASTION -.->|LAN| CP
|
||||
CP -->|OIDC| KC
|
||||
WP -->|OIDC| KC
|
||||
|
||||
classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
|
||||
classDef infra fill:#0f3460,stroke:#e94560,color:#fff
|
||||
classDef external fill:#16213e,stroke:#533483,color:#fff
|
||||
class CP,WP,ETCD talos
|
||||
class REG,BASTION infra
|
||||
class OP,KC,EXTERNAL external
|
||||
```
|
||||
|
||||
### Design principles
|
||||
|
||||
1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
|
||||
Strongest posture for ITAR/classified.
|
||||
2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
|
||||
advertises the cluster LAN subnet. Operator reaches Talos API from
|
||||
anywhere.
|
||||
3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
|
||||
for etcd). Workers boot from local disk where available; NFS for bulk
|
||||
data only.
|
||||
4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
|
||||
of the Talos host cluster.
|
||||
5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
|
||||
tokens. No long-lived static credentials for humans.
|
||||
|
||||
---
|
||||
|
||||
## 2. Control Plane
|
||||
|
||||
### 2.1 Recommendation: 3 cnodes (down from 5)
|
||||
|
||||
| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
|
||||
|--------|--------|-------------------|-----------------|---------------|
|
||||
| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
|
||||
| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
|
||||
|
||||
For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
|
||||
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
|
||||
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
|
||||
host capacity) are better spent on tenant worker allocations.
|
||||
|
||||
**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
|
||||
tolerance on the control plane, keep 5. Otherwise 3.
|
||||
|
||||
### 2.2 Cnode placement
|
||||
|
||||
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
|
||||
**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
|
||||
|
||||
| cnode | Host | Boot disk | Type | Why |
|
||||
|-------|------|-----------|------|-----|
|
||||
| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
|
||||
| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
|
||||
| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
|
||||
|
||||
**Quorum survival:**
|
||||
|
||||
| Failure | cnodes lost | Quorum OK? |
|
||||
|---------|-------------|------------|
|
||||
| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
|
||||
| tsys1 host dies | cnode3 | YES (2 of 3) |
|
||||
| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
|
||||
|
||||
**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
|
||||
loses quorum. Alternative: spread cnodes across 3 different hosts. See
|
||||
"open question" at end of this section.
|
||||
|
||||
### 2.3 Machine config strategy
|
||||
|
||||
Talos nodes are configured by **machine configs** (YAML). Two flavors:
|
||||
|
||||
- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
|
||||
controller-manager, API server.
|
||||
- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
|
||||
|
||||
Strategy for this cluster:
|
||||
|
||||
1. **One shared `talosconfig`** (client identity) — stored in 1Password
|
||||
and in the Proxmox Backup Server (PBS) encrypted backup target.
|
||||
2. **Per-node machine config patches** — small patches on top of the base
|
||||
`controlplane.yaml` / `worker.yaml` for node-specific settings:
|
||||
- Hostname
|
||||
- Network interface + IP (DHCP or static — recommend static for cnodes)
|
||||
- Schematic image digest (pinned Talos version)
|
||||
- System extensions (e.g., `tailscale` — only if running Pattern A
|
||||
instead of recommended Pattern C)
|
||||
3. **All machine configs in Git** under a future `k8s/talos-configs/`
|
||||
directory. Secrets are templated in at apply-time from 1Password / sops.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
|
||||
BASE --> PATCH2[patch: cnode2]
|
||||
BASE --> PATCH3[patch: cnode3]
|
||||
BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
|
||||
PATCH1 --> APPLY1[talosctl apply]
|
||||
PATCH2 --> APPLY2[talosctl apply]
|
||||
PATCH3 --> APPLY3[talosctl apply]
|
||||
PATCHW --> APPLYW[talosctl apply]
|
||||
```
|
||||
|
||||
### 2.4 Open question: cnode host spread
|
||||
|
||||
If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
|
||||
cluster, tsys9 is brand-new hardware, single digit annual failure
|
||||
probability), the layout in §2.2 is fine.
|
||||
|
||||
If not, alternative spread across 3 hosts:
|
||||
|
||||
| cnode | Host | Boot disk |
|
||||
|-------|------|-----------|
|
||||
| cnode1 | tsys9 | local-lvm SSD |
|
||||
| cnode2 | tsys1 | local-lvm HDD |
|
||||
| cnode3 | tsys3 | local-lvm NVMe |
|
||||
|
||||
tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
|
||||
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
|
||||
would make an excellent etcd disk.
|
||||
|
||||
**This 3-host spread survives any single host failure with quorum intact.
|
||||
Recommended.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Network Topology
|
||||
|
||||
### 3.1 Zones
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph INTERNET["Internet"]
|
||||
FIBER[Gigabit symmetric fiber]
|
||||
end
|
||||
|
||||
subgraph RESLAN["Residence LAN 192.168.x.x/24"]
|
||||
subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
|
||||
CNODES[Cnodes 192.168.3.x]
|
||||
WNODES[Wnodes 192.168.3.x]
|
||||
end
|
||||
BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
|
||||
REG[Harbor registry<br/>192.168.3.x]
|
||||
STORAGE[NFS servers<br/>tsys4, tsys5]
|
||||
end
|
||||
|
||||
subgraph TSNET["Tailscale 100.x.x.x/8"]
|
||||
OPS[Operator devices]
|
||||
KC[Keycloak<br/>via Cloudron prod]
|
||||
end
|
||||
|
||||
FIBER --> BASTION
|
||||
BASTION <-. subnet route .-> CLUSTERNET
|
||||
OPS -->|TCP 50000 talos API| BASTION
|
||||
BASTION -->|LAN forward| CNODES
|
||||
CNODES -->|LAN mTLS| WNODES
|
||||
CNODES -->|OIDC HTTPS| KC
|
||||
WNODES -->|pull images| REG
|
||||
WNODES -->|bulk data IO| STORAGE
|
||||
CNODES -->|pull images| REG
|
||||
```
|
||||
|
||||
### 3.2 Address plan (suggested)
|
||||
|
||||
Reserve a small contiguous block in the residence LAN for cluster nodes:
|
||||
|
||||
| Role | Range | Count |
|
||||
|------|-------|-------|
|
||||
| Cnodes | `192.168.3.31-33` | 3 |
|
||||
| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
|
||||
| Bastion | existing `tailscale-router` | 1 |
|
||||
| Registry | `192.168.3.50` | 1 (Harbor) |
|
||||
|
||||
**Static IPs are strongly recommended for cnodes** (etcd cluster membership
|
||||
is hostname-based; stable IPs make `talosctl` targeting simple). Workers
|
||||
can DHCP.
|
||||
|
||||
### 3.3 Firewall posture
|
||||
|
||||
Each cnode/wnode has:
|
||||
|
||||
- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
|
||||
cnodes only), plus CNI ports (varies by CNI — see §4).
|
||||
- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
|
||||
- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
|
||||
firewall for these IPs. ITAR workloads must not be able to phone home.
|
||||
|
||||
The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
|
||||
via the subnet route.
|
||||
|
||||
---
|
||||
|
||||
## 4. CNI: Cilium
|
||||
|
||||
**Recommendation: Cilium** (eBPF-based CNI).
|
||||
|
||||
| Property | Why it matters here |
|
||||
|----------|---------------------|
|
||||
| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
|
||||
| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
|
||||
| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
|
||||
| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
|
||||
| **Talos integration** | First-class. Talos docs document the install path. |
|
||||
|
||||
Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
|
||||
enabled. Default-deny NetworkPolicy applied per namespace.
|
||||
|
||||
---
|
||||
|
||||
## 5. Identity and Trust
|
||||
|
||||
### 5.1 Trust flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Human as Operator
|
||||
participant TAIL as Tailscale
|
||||
participant BAST as Bastion
|
||||
participant TALOS as Talos API :50000
|
||||
participant KC as Keycloak (Cloudron)
|
||||
participant K8S as Kubernetes API :6443
|
||||
|
||||
Human->>TAIL: Authenticate (device + SSO)
|
||||
TAIL-->>Human: Tailnet IP
|
||||
Human->>BAST: Reach bastion via tailnet
|
||||
BAST->>TALOS: Forward to LAN node :50000
|
||||
Human->>KC: OIDC login (browser)
|
||||
KC-->>Human: Bearer token (short-lived)
|
||||
Human->>TALOS: talosctl (mTLS with client cert)
|
||||
Human->>K8S: kubectl --oidc (Keycloak token)
|
||||
K8S->>KC: Validate token (introspection)
|
||||
KC-->>K8S: Valid + claims
|
||||
K8S-->>Human: Authorized response
|
||||
```
|
||||
|
||||
### 5.2 Two distinct identity layers
|
||||
|
||||
| Layer | Mechanism | Audience |
|
||||
|-------|-----------|----------|
|
||||
| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
|
||||
| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
|
||||
|
||||
The **Talos secrets bundle** is the root of trust for the cluster. Lose it
|
||||
and you cannot operate the cluster; an attacker with it owns the cluster.
|
||||
Storage:
|
||||
|
||||
1. **Primary:** 1Password (or equivalent) — operator-accessible.
|
||||
2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
|
||||
3. **NOT in Git.** Machine configs go in Git; secrets stay out.
|
||||
|
||||
### 5.3 Keycloak client configuration
|
||||
|
||||
On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
|
||||
|
||||
- **Authorization Code + PKCE flow** (no implicit, no password).
|
||||
- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
|
||||
Omni/Rancher URLs if/when those are added.
|
||||
- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
|
||||
`k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tenant Isolation (vcluster)
|
||||
|
||||
### 6.1 Why vcluster
|
||||
|
||||
[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
|
||||
plane** (API server, scheduler, controller-manager, etcd) inside a namespace
|
||||
of the host cluster. Tenant workloads run on the host's worker nodes but
|
||||
are isolated by:
|
||||
|
||||
- Separate API server (tenant cannot see host cluster objects).
|
||||
- Separate RBAC and admission control.
|
||||
- Separate network policies (per-namespace).
|
||||
- Separate resource quotas.
|
||||
|
||||
This aligns with the user's per-tenant plan from
|
||||
[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
|
||||
|
||||
### 6.2 Tenant registry
|
||||
|
||||
| Tenant | Compliance | Workload example | vcluster name |
|
||||
|--------|-----------|------------------|---------------|
|
||||
| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
|
||||
| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
|
||||
| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
|
||||
| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph HOST["Talos host cluster"]
|
||||
CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
|
||||
subgraph NS["Host cluster namespaces"]
|
||||
NS_RR[ns: vc-rackrental]
|
||||
NS_SO[ns: vc-suborbital-open]
|
||||
NS_SI[ns: vc-suborbital-itar]
|
||||
NS_SLP[ns: vc-slp]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph VRR["vcluster: vc-rackrental"]
|
||||
API_RR[k8s API + etcd]
|
||||
end
|
||||
subgraph VSO["vcluster: vc-suborbital-open"]
|
||||
API_SO[k8s API + etcd]
|
||||
end
|
||||
subgraph VSI["vcluster: vc-suborbital-itar"]
|
||||
API_SI[k8s API + etcd]
|
||||
end
|
||||
subgraph VSLP["vcluster: vc-slp"]
|
||||
API_SLP[k8s API + etcd]
|
||||
end
|
||||
|
||||
CP --> NS_RR & NS_SO & NS_SI & NS_SLP
|
||||
NS_RR --> API_RR
|
||||
NS_SO --> API_SO
|
||||
NS_SI --> API_SI
|
||||
NS_SLP --> API_SLP
|
||||
|
||||
classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
|
||||
class NS_SI,API_SI itar
|
||||
```
|
||||
|
||||
### 6.3 ITAR enforcement at host layer
|
||||
|
||||
For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
|
||||
controls:
|
||||
|
||||
- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
|
||||
ITAR workloads (subset of wnodes, marked in node labels).
|
||||
- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
|
||||
namespace. Allow only explicit destinations (registry, NFS for ITAR
|
||||
data tier, Keycloak).
|
||||
- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
|
||||
`D3-itar` on tsys5) that no other tenant can mount.
|
||||
- **Audit:** Hubble flows + auditd on the host worker nodes capture all
|
||||
access to ITAR data.
|
||||
|
||||
Rancher (or Sidero Omni) sits above this, presenting each tenant's
|
||||
vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
|
||||
access per tenant group claim.
|
||||
|
||||
---
|
||||
|
||||
## 7. Storage Integration
|
||||
|
||||
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
|
||||
|
||||
| StorageClass | Provisioner | Backing | Speed | Use |
|
||||
|--------------|------------|---------|-------|-----|
|
||||
| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
|
||||
| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
|
||||
| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
|
||||
|
||||
### 7.1 CSI driver notes
|
||||
|
||||
- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
|
||||
(CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
|
||||
NFS server + base export path.
|
||||
- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
|
||||
with one manifest. Uses wnode's kubelet root dir.
|
||||
|
||||
### 7.2 ITAR data isolation
|
||||
|
||||
The ITAR tenant should target a dedicated NFS export, not shared
|
||||
`nfs-hdd`. Recommended:
|
||||
|
||||
- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
|
||||
`nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
|
||||
namespace via RBAC + NetworkPolicy.
|
||||
|
||||
---
|
||||
|
||||
## 8. Local Image Registry
|
||||
|
||||
### 8.1 Recommendation: Harbor on D3 SSD
|
||||
|
||||
D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
|
||||
for a **Harbor** instance:
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
|
||||
| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
|
||||
| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
|
||||
| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
|
||||
|
||||
### 8.2 Pull-through cache benefit
|
||||
|
||||
Cluster nodes have zero internet egress (per §3.3). Without a local cache,
|
||||
image pulls fail. With Harbor as a pull-through cache:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
WNODE->>HARBOR: docker pull nginx:1.25
|
||||
alt cache hit
|
||||
HARBOR-->>WNODE: layer bytes (LAN-speed)
|
||||
else cache miss
|
||||
HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
|
||||
DOCKERHUB-->>HARBOR: layer bytes
|
||||
HARBOR-->>WNODE: layer bytes (cached for next time)
|
||||
end
|
||||
```
|
||||
|
||||
Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
|
||||
machine in the cluster with container-registry internet egress, and that
|
||||
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
|
||||
ghcr.io).
|
||||
|
||||
### 8.3 Supply-chain integrity (future)
|
||||
|
||||
Harbor + Cosign lets you require that all images deployed to the ITAR
|
||||
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
|
||||
control. Implementation deferred to a later session.
|
||||
|
||||
---
|
||||
|
||||
## 9. Bootstrap Procedure
|
||||
|
||||
### 9.1 One-time setup
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant GIT as Git repo
|
||||
participant ONEPW as 1Password
|
||||
participant PBS as PBS (tsys4)
|
||||
OP->>GIT: Clone PFVCluster repo
|
||||
OP->>ONEPW: Generate Talos secrets bundle (offline)
|
||||
ONEPW-->>OP: secrets.yaml
|
||||
OP->>PBS: Backup secrets.yaml (encrypted)
|
||||
OP->>GIT: Write machine configs (no secrets)
|
||||
```
|
||||
|
||||
### 9.2 Provision first cnode (bootstrap)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant PX as Proxmox host
|
||||
participant C1 as cnode1
|
||||
participant ETCD as etcd (new)
|
||||
|
||||
OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
|
||||
OP->>PX: qm start VMID
|
||||
C1->>C1: Boots Talos (no config yet, "maintenance mode")
|
||||
OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
|
||||
C1->>C1: Applies config, restarts services
|
||||
OP->>C1: talosctl bootstrap
|
||||
C1->>ETCD: Initialize single-node raft
|
||||
ETCD-->>C1: ready
|
||||
OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
|
||||
OP->>C1: talosctl etcd snapshot (initial backup → PBS)
|
||||
```
|
||||
|
||||
### 9.3 Add second and third cnodes
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant OP as Operator
|
||||
participant PX as Proxmox host
|
||||
participant C2 as cnode2
|
||||
participant C3 as cnode3
|
||||
participant C1 as cnode1 (existing)
|
||||
|
||||
OP->>PX: qm create + start cnode2 VM
|
||||
C2->>C2: Boots Talos maintenance mode
|
||||
OP->>C2: talosctl apply --patch cnode2.yaml
|
||||
C2->>C1: Join etcd cluster
|
||||
OP->>PX: qm create + start cnode3 VM
|
||||
C3->>C3: Boots Talos maintenance mode
|
||||
OP->>C3: talosctl apply --patch cnode3.yaml
|
||||
C3->>C1: Join etcd cluster
|
||||
Note over C1,C3: etcd now has 3/3 members → HA quorum
|
||||
```
|
||||
|
||||
### 9.4 Post-bootstrap cluster configuration
|
||||
|
||||
Once 3 cnodes are up and joined:
|
||||
|
||||
1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
|
||||
2. **Install CSI drivers** — nfs-csi + local-path provisioner.
|
||||
3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
|
||||
4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
|
||||
5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
|
||||
6. **Apply default-deny NetworkPolicy** in all namespaces.
|
||||
7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
|
||||
8. **First etcd snapshot** + automated daily snapshot cron → PBS.
|
||||
|
||||
### 9.5 Add workers
|
||||
|
||||
Workers are simpler (no etcd):
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
|
||||
WNODE->>WNODE: Boots maintenance mode
|
||||
OP->>WNODE: talosctl apply --patch worker-X.yaml
|
||||
WNODE->>C1: Kubelet registers with API server
|
||||
C1-->>WNODE: Approved (auto via bootstrap token)
|
||||
Note over WNODE: Joins cluster, becomes Ready
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Disaster Recovery
|
||||
|
||||
### 10.1 Backup strategy
|
||||
|
||||
| Artifact | Frequency | Storage | Tool |
|
||||
|----------|-----------|---------|------|
|
||||
| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
|
||||
| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
|
||||
| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
|
||||
| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
|
||||
| **Harbor metadata** | Daily | PBS | Harbor built-in backup |
|
||||
|
||||
### 10.2 Restore scenarios
|
||||
|
||||
**Lost 1 cnode (e.g., tsys9 disk failure):**
|
||||
|
||||
1. Provision new VM on tsys9 (or other host with local SSD).
|
||||
2. Apply cnode2 machine config patch.
|
||||
3. New cnode joins etcd, syncs state from survivors.
|
||||
4. Quorum was never lost (2 of 3 alive throughout).
|
||||
|
||||
**Lost 2 cnodes simultaneously (quorum lost):**
|
||||
|
||||
1. Use surviving cnode's etcd snapshot.
|
||||
2. Provision 3 new cnode VMs.
|
||||
3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
|
||||
4. Join other 2 cnodes.
|
||||
5. Workers reconnect automatically once API server is back.
|
||||
|
||||
**Total cluster loss (all 3 cnodes):**
|
||||
|
||||
1. Restore from latest PBS etcd snapshot.
|
||||
2. Provision new cnode VMs.
|
||||
3. `talosctl bootstrap --recover-from=snapshot.db`.
|
||||
4. Re-join workers.
|
||||
5. Verify tenant vclusters restored.
|
||||
|
||||
### 10.3 Recovery time objectives
|
||||
|
||||
| Scenario | RTO | RPO |
|
||||
|----------|-----|-----|
|
||||
| Single cnode failure | < 30 min | 0 (no data loss) |
|
||||
| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
|
||||
| Total cluster loss | < 4 hours | ≤ 24 hours |
|
||||
|
||||
---
|
||||
|
||||
## 11. Migration from Current State
|
||||
|
||||
### 11.1 Current state
|
||||
|
||||
- 5 cnode VMs exist (Debian stock + Tailscale).
|
||||
- **No k3s deployed yet.** Cluster was never bootstrapped.
|
||||
- 6 wnode VMs exist (some stopped).
|
||||
- No workloads running in k8s.
|
||||
|
||||
### 11.2 Migration: clean cutover (not a migration)
|
||||
|
||||
Since there is no etcd data and no workloads to preserve, the path is a
|
||||
**clean rebuild**:
|
||||
|
||||
| Phase | Action | Risk |
|
||||
|-------|--------|------|
|
||||
| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
|
||||
| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
|
||||
| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
|
||||
| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
|
||||
| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
|
||||
| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
|
||||
|
||||
### 11.3 Open dependency: Friday hardware work
|
||||
|
||||
Phases 1-2 require:
|
||||
|
||||
- **tsys3 local-lvm available.** Per
|
||||
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
|
||||
free NVMe local-lvm. Currently unused. **Ready.**
|
||||
- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
|
||||
- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
|
||||
[`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
|
||||
Harbor depends on D3 being available on tsys5.
|
||||
|
||||
Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
|
||||
Harbor registry does.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Open questions for next session
|
||||
|
||||
1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
|
||||
2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
|
||||
(tsys9, tsys1, tsys3).
|
||||
3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
|
||||
4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
|
||||
Defer until cluster is up.
|
||||
5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
|
||||
approval of `192.168.3.0/24` route advertisement.
|
||||
6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
|
||||
Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
|
||||
general capacity; tsys3 + tsys5 (local fast storage) reserved for
|
||||
non-ITAR HPC.
|
||||
@@ -0,0 +1,188 @@
|
||||
# Distro Decision: Talos Linux vs k3s
|
||||
|
||||
> **Recommendation: Talos Linux.**
|
||||
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
|
||||
> entered scope. Once classified workloads are on the table, Talos's
|
||||
> immutable, API-only, measured-boot-capable posture is materially easier
|
||||
> to certify and defend.
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision context
|
||||
|
||||
| Factor | Constraint |
|
||||
|--------|-----------|
|
||||
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
|
||||
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
|
||||
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
|
||||
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
|
||||
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
|
||||
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Head-to-head comparison
|
||||
|
||||
### 2.1 ITAR / classified posture
|
||||
|
||||
| Property | Talos Linux | k3s on Debian |
|
||||
|----------|-------------|---------------|
|
||||
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
|
||||
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
|
||||
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
|
||||
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
|
||||
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
|
||||
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
|
||||
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
|
||||
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
|
||||
|
||||
**Bottom line:** For classified workloads, an auditor's first question is
|
||||
"how do you prevent unauthorized changes to a node?" Talos's answer is
|
||||
"the OS is immutable and the only path is a signed API call." k3s's answer
|
||||
is"SSH is locked down and we scan with STIG." The first is structurally
|
||||
stronger; the second is operationally maintained.
|
||||
|
||||
### 2.2 Operational considerations
|
||||
|
||||
| Property | Talos | k3s |
|
||||
|----------|-------|-----|
|
||||
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
|
||||
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
|
||||
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
|
||||
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
|
||||
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
|
||||
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
|
||||
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
|
||||
|
||||
### 2.3 Cost of choosing Talos over the existing k3s plan
|
||||
|
||||
The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
|
||||
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
|
||||
|
||||
- **No etcd data to migrate.** Clean cutover, not a migration.
|
||||
- **No workloads to drain.** The cluster is empty.
|
||||
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
|
||||
either way it's a `qm` script, not a stateful migration).
|
||||
- **Tailscale config shifts** from "installed via apt" to "Talos system
|
||||
extension." (Or, per our recommendation in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
|
||||
nodes entirely** and onto the existing `tailscale-router` bastion as a
|
||||
subnet router. Cluster nodes become LAN-only.)
|
||||
|
||||
**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
|
||||
machine config YAML. The hardening investment already encoded in
|
||||
`provisioning/Modules/Security/` is **not wasted** — it still applies to
|
||||
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
|
||||
the cnodes/wnodes move to Talos.
|
||||
|
||||
---
|
||||
|
||||
## 3. Tailscale compatibility (deep-dive)
|
||||
|
||||
Tailscale on Talos is well-supported but introduces a configuration dimension
|
||||
worth being explicit about. Three patterns exist:
|
||||
|
||||
### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
|
||||
|
||||
Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
|
||||
internet egress (to Tailscale DERP servers and for coordinate).
|
||||
|
||||
- **Talos implementation:** add `siderolabs/tailscale` system extension to
|
||||
each machine config, configure `machine.network.interfaces`.
|
||||
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
|
||||
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
|
||||
finding (data exfiltration path).
|
||||
|
||||
### Pattern B — Tailscale on bastion only, SSH/API jump
|
||||
|
||||
Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
|
||||
`tailscale-router` VM), then runs `talosctl` from the bastion.
|
||||
|
||||
- **Pros:** Zero internet egress from cluster nodes.
|
||||
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
|
||||
action originates from the bastion (auditable but clunky).
|
||||
|
||||
### Pattern C — Tailscale subnet router on bastion (recommended)
|
||||
|
||||
The existing `tailscale-router` VM advertises the cluster LAN subnet
|
||||
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
|
||||
Tailscale client transparently routes cluster-bound traffic through the
|
||||
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
|
||||
"just works."
|
||||
|
||||
- **Pros:**
|
||||
- Cluster nodes have **zero internet egress** (strongest ITAR posture).
|
||||
- Operator UX is unchanged from direct LAN access.
|
||||
- All access is mediated by Tailscale's identity + ACLs (already integrated
|
||||
with your env).
|
||||
- Audit trail lives in Tailscale + bastion logs.
|
||||
- **Cons:**
|
||||
- Bastion becomes a dependency for remote admin (LAN-local admin still
|
||||
works without it).
|
||||
- Must enable IP forwarding + subnet route approval in Tailscale ACLs.
|
||||
|
||||
**Recommendation: Pattern C.** Documented in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
|
||||
|
||||
### Justification
|
||||
|
||||
1. **Compliance posture is structural, not operational.** "Immutable,
|
||||
API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
|
||||
is a property of how Debian is operated. The first is dramatically
|
||||
easier to argue to an ITAR counsel or classified accreditation officer
|
||||
(DSS, DCSA) than the second.
|
||||
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
|
||||
3 cnodes with Talos is a `qm` script invocation, not a stateful
|
||||
migration. The sunk cost of "we planned k3s" is **zero deployed state**.
|
||||
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
|
||||
`apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
|
||||
learn but more error-prone to operate at HA.
|
||||
|
||||
### Acknowledged tradeoffs
|
||||
|
||||
- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
|
||||
replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
|
||||
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
|
||||
in. Mitigation: `talosctl logs/support` produces a support bundle
|
||||
equivalent to a sosreport.
|
||||
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
|
||||
beyond virtio + common NIC drivers needs a system extension. On Proxmox
|
||||
VMs this is **not expected to be a problem** — virtio is the path.
|
||||
- **Tailscale via system extension.** Adds one config dimension per node.
|
||||
Mitigated by Pattern C (above), which removes Tailscale from cluster
|
||||
nodes entirely.
|
||||
|
||||
### What we keep from the k3s mental model
|
||||
|
||||
- **Single binary on each node** semantics (Talos is conceptually similar).
|
||||
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
|
||||
API. `kubectl`, `helm`, `kustomize` all work as-is.
|
||||
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
|
||||
distro-independent.
|
||||
|
||||
---
|
||||
|
||||
## 5. What we are NOT deciding here
|
||||
|
||||
| Topic | Deferred to |
|
||||
|--------|-------------|
|
||||
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
|
||||
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
|
||||
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
|
||||
| Solar-aware scale-out hosts | Future capacity planning session |
|
||||
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Next step
|
||||
|
||||
Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
|
||||
design, network topology, identity flow, and bootstrap procedure.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Kubernetes Architecture & Build Plan
|
||||
|
||||
> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
|
||||
> (which captured the storage/host analysis from the Proxmox audit).
|
||||
> This directory takes the next step: **which distro, how to build it,
|
||||
> how to operate it.**
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
## Documents in this directory
|
||||
|
||||
| Document | Purpose |
|
||||
|----------|---------|
|
||||
| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
|
||||
| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Decision | Recommendation | Why |
|
||||
|----------|----------------|-----|
|
||||
| **Distro** | **Talos Linux** (not k3s) | Immutable, API-only, measured-boot-capable. Intrinsically close to a STIG/CIS posture, where k3s-on-Debian requires constant re-certification. The k3s plan was reasonable before the ITAR/classified requirement was on the table. |
|
||||
| **Runtime** | **containerd** (Talos default) | Only runtime Talos supports; also the secure default. Non-decision. |
|
||||
| **Cnode count** | **3** (down from 5) | Standard HA. Tolerates 1 failure. Frees 2 VMs for tenant worker capacity. Re-evaluate if a regulator mandates 5. |
|
||||
| **Admin access** | **Tailscale subnet router** on existing `tailscale-router` VM | Cluster nodes stay LAN-only (no internet egress = stronger ITAR posture). Operator reaches Talos API from anywhere via Tailscale. |
|
||||
| **Cluster network** | LAN mTLS only | All cnode/wnode traffic on Proxmox LAN. No Tailscale on cluster nodes themselves. |
|
||||
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Already planned. Talos API + Kubernetes API both trust Keycloak tokens. |
|
||||
| **Multi-tenancy** | **vcluster** (per tenant) inside Talos host cluster | Strong workload isolation. One vcluster per tenant: ITAR, non-ITAR, SLP, RackRental. |
|
||||
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Avoids needless round-trips to Docker Hub / Cloudron. Adds image-signing (Cosign) capability for supply-chain integrity. |
|
||||
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Unchanged. |
|
||||
|
||||
## What this directory does NOT cover (deferred)
|
||||
|
||||
- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
|
||||
- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
|
||||
- vcluster per-tenant policy templates.
|
||||
- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
|
||||
|
||||
These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
|
||||
|
||||
## Open question for the user
|
||||
|
||||
1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
|
||||
2-failure tolerance on the control plane, keep 5.
|
||||
Reference in New Issue
Block a user