KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(docs): co-locate k8s/proxmox docs with their code + fix broken perf paths

Browse Source 
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-29 05:28:07 -05:00
parent b860cb116d
commit 428e9ecb1d
15 changed files with 25 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/docs/ARCHITECTURE.md
+673
View File
@@ -0,0 +1,673 @@
# pfv-k8s Target Architecture (Talos)
> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
> Proxmox audit).
**Last updated:** 2026-07-28
---
## Table of Contents
1. [High-Level Architecture](#1-high-level-architecture)
2. [Control Plane](#2-control-plane)
3. [Network Topology](#3-network-topology)
4. [CNI: Cilium](#4-cni-cilium)
5. [Identity and Trust](#5-identity-and-trust)
6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
7. [Storage Integration](#7-storage-integration)
8. [Local Image Registry](#8-local-image-registry)
9. [Bootstrap Procedure](#9-bootstrap-procedure)
10. [Disaster Recovery](#10-disaster-recovery)
11. [Migration from Current State](#11-migration-from-current-state)
---
## 1. High-Level Architecture
```mermaid
flowchart TB
subgraph RESIDENCE["Residence — Proxmox LAN"]
subgraph CP["Talos Control Plane (3 cnodes)"]
C1[cnode1<br/>tsys9 · local-SSD]
C2[cnode2<br/>tsys9 · local-SSD]
C3[cnode3<br/>tsys1 · local-HDD]
end
subgraph WP["Talos Worker Plane"]
W3[wnode-tsys3<br/>NVMe · 28GB]
W5[wnode-tsys5<br/>NVMe · 32-64GB]
W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
W9[wnode-tsys9<br/>local-SSD · 4-8GB]
end
ETCD[(etcd<br/>raft, mTLS)]
REG[(Harbor registry<br/>on D3 SSD · tsys5)]
BASTION[tailscale-router VM<br/>subnet router]
end
subgraph TAILNET["Tailscale overlay"]
OP[Operator devices]
end
subgraph CLOUDRON["Cloudron production — Reston VA"]
KC[Keycloak OIDC IdP]
end
C1 ---|mTLS LAN| ETCD
C2 ---|mTLS LAN| ETCD
C3 ---|mTLS LAN| ETCD
CP -->|pull images| REG
WP -->|pull images| REG
OP -->|Talos API :50000<br/>via subnet route| BASTION
BASTION -.->|LAN| CP
CP -->|OIDC| KC
WP -->|OIDC| KC
classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
classDef infra fill:#0f3460,stroke:#e94560,color:#fff
classDef external fill:#16213e,stroke:#533483,color:#fff
class CP,WP,ETCD talos
class REG,BASTION infra
class OP,KC,EXTERNAL external
```
### Design principles
1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
Strongest posture for ITAR/classified.
2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
advertises the cluster LAN subnet. Operator reaches Talos API from
anywhere.
3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
for etcd). Workers boot from local disk where available; NFS for bulk
data only.
4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
of the Talos host cluster.
5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
tokens. No long-lived static credentials for humans.
---
## 2. Control Plane
### 2.1 Recommendation: 3 cnodes (down from 5)
| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
|--------|--------|-------------------|-----------------|---------------|
| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |
For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
host capacity) are better spent on tenant worker allocations.
**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
tolerance on the control plane, keep 5. Otherwise 3.
### 2.2 Cnode placement
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:
| cnode | Host | Boot disk | Type | Why |
|-------|------|-----------|------|-----|
| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |
**Quorum survival:**
| Failure | cnodes lost | Quorum OK? |
|---------|-------------|------------|
| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
| tsys1 host dies | cnode3 | YES (2 of 3) |
| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |
**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
loses quorum. Alternative: spread cnodes across 3 different hosts. See
"open question" at end of this section.
### 2.3 Machine config strategy
Talos nodes are configured by **machine configs** (YAML). Two flavors:
- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
controller-manager, API server.
- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.
Strategy for this cluster:
1. **One shared `talosconfig`** (client identity) — stored in 1Password
and in the Proxmox Backup Server (PBS) encrypted backup target.
2. **Per-node machine config patches** — small patches on top of the base
`controlplane.yaml` / `worker.yaml` for node-specific settings:
- Hostname
- Network interface + IP (DHCP or static — recommend static for cnodes)
- Schematic image digest (pinned Talos version)
- System extensions (e.g., `tailscale` — only if running Pattern A
instead of recommended Pattern C)
3. **All machine configs in Git** under a future `k8s/talos-configs/`
directory. Secrets are templated in at apply-time from 1Password / sops.
```mermaid
flowchart LR
BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
BASE --> PATCH2[patch: cnode2]
BASE --> PATCH3[patch: cnode3]
BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
PATCH1 --> APPLY1[talosctl apply]
PATCH2 --> APPLY2[talosctl apply]
PATCH3 --> APPLY3[talosctl apply]
PATCHW --> APPLYW[talosctl apply]
```
### 2.4 Open question: cnode host spread
If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
cluster, tsys9 is brand-new hardware, single digit annual failure
probability), the layout in §2.2 is fine.
If not, alternative spread across 3 hosts:
| cnode | Host | Boot disk |
|-------|------|-----------|
| cnode1 | tsys9 | local-lvm SSD |
| cnode2 | tsys1 | local-lvm HDD |
| cnode3 | tsys3 | local-lvm NVMe |
tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
would make an excellent etcd disk.
**This 3-host spread survives any single host failure with quorum intact.
Recommended.**
---
## 3. Network Topology
### 3.1 Zones
```mermaid
flowchart TB
subgraph INTERNET["Internet"]
FIBER[Gigabit symmetric fiber]
end
subgraph RESLAN["Residence LAN 192.168.x.x/24"]
subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
CNODES[Cnodes 192.168.3.x]
WNODES[Wnodes 192.168.3.x]
end
BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
REG[Harbor registry<br/>192.168.3.x]
STORAGE[NFS servers<br/>tsys4, tsys5]
end
subgraph TSNET["Tailscale 100.x.x.x/8"]
OPS[Operator devices]
KC[Keycloak<br/>via Cloudron prod]
end
FIBER --> BASTION
BASTION <-. subnet route .-> CLUSTERNET
OPS -->|TCP 50000 talos API| BASTION
BASTION -->|LAN forward| CNODES
CNODES -->|LAN mTLS| WNODES
CNODES -->|OIDC HTTPS| KC
WNODES -->|pull images| REG
WNODES -->|bulk data IO| STORAGE
CNODES -->|pull images| REG
```
### 3.2 Address plan (suggested)
Reserve a small contiguous block in the residence LAN for cluster nodes:
| Role | Range | Count |
|------|-------|-------|
| Cnodes | `192.168.3.31-33` | 3 |
| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
| Bastion | existing `tailscale-router` | 1 |
| Registry | `192.168.3.50` | 1 (Harbor) |
**Static IPs are strongly recommended for cnodes** (etcd cluster membership
is hostname-based; stable IPs make `talosctl` targeting simple). Workers
can DHCP.
### 3.3 Firewall posture
Each cnode/wnode has:
- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
cnodes only), plus CNI ports (varies by CNI — see §4).
- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
firewall for these IPs. ITAR workloads must not be able to phone home.
The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
via the subnet route.
---
## 4. CNI: Cilium
**Recommendation: Cilium** (eBPF-based CNI).
| Property | Why it matters here |
|----------|---------------------|
| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
| **Talos integration** | First-class. Talos docs document the install path. |
Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
enabled. Default-deny NetworkPolicy applied per namespace.
---
## 5. Identity and Trust
### 5.1 Trust flow
```mermaid
sequenceDiagram
autonumber
participant Human as Operator
participant TAIL as Tailscale
participant BAST as Bastion
participant TALOS as Talos API :50000
participant KC as Keycloak (Cloudron)
participant K8S as Kubernetes API :6443
Human->>TAIL: Authenticate (device + SSO)
TAIL-->>Human: Tailnet IP
Human->>BAST: Reach bastion via tailnet
BAST->>TALOS: Forward to LAN node :50000
Human->>KC: OIDC login (browser)
KC-->>Human: Bearer token (short-lived)
Human->>TALOS: talosctl (mTLS with client cert)
Human->>K8S: kubectl --oidc (Keycloak token)
K8S->>KC: Validate token (introspection)
KC-->>K8S: Valid + claims
K8S-->>Human: Authorized response
```
### 5.2 Two distinct identity layers
| Layer | Mechanism | Audience |
|-------|-----------|----------|
| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |
The **Talos secrets bundle** is the root of trust for the cluster. Lose it
and you cannot operate the cluster; an attacker with it owns the cluster.
Storage:
1. **Primary:** 1Password (or equivalent) — operator-accessible.
2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
3. **NOT in Git.** Machine configs go in Git; secrets stay out.
### 5.3 Keycloak client configuration
On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:
- **Authorization Code + PKCE flow** (no implicit, no password).
- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
Omni/Rancher URLs if/when those are added.
- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
`k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.
---
## 6. Tenant Isolation (vcluster)
### 6.1 Why vcluster
[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
plane** (API server, scheduler, controller-manager, etcd) inside a namespace
of the host cluster. Tenant workloads run on the host's worker nodes but
are isolated by:
- Separate API server (tenant cannot see host cluster objects).
- Separate RBAC and admission control.
- Separate network policies (per-namespace).
- Separate resource quotas.
This aligns with the user's per-tenant plan from
[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.
### 6.2 Tenant registry
| Tenant | Compliance | Workload example | vcluster name |
|--------|-----------|------------------|---------------|
| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |
```mermaid
flowchart TB
subgraph HOST["Talos host cluster"]
CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
subgraph NS["Host cluster namespaces"]
NS_RR[ns: vc-rackrental]
NS_SO[ns: vc-suborbital-open]
NS_SI[ns: vc-suborbital-itar]
NS_SLP[ns: vc-slp]
end
end
subgraph VRR["vcluster: vc-rackrental"]
API_RR[k8s API + etcd]
end
subgraph VSO["vcluster: vc-suborbital-open"]
API_SO[k8s API + etcd]
end
subgraph VSI["vcluster: vc-suborbital-itar"]
API_SI[k8s API + etcd]
end
subgraph VSLP["vcluster: vc-slp"]
API_SLP[k8s API + etcd]
end
CP --> NS_RR & NS_SO & NS_SI & NS_SLP
NS_RR --> API_RR
NS_SO --> API_SO
NS_SI --> API_SI
NS_SLP --> API_SLP
classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
class NS_SI,API_SI itar
```
### 6.3 ITAR enforcement at host layer
For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
controls:
- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
ITAR workloads (subset of wnodes, marked in node labels).
- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
namespace. Allow only explicit destinations (registry, NFS for ITAR
data tier, Keycloak).
- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
`D3-itar` on tsys5) that no other tenant can mount.
- **Audit:** Hubble flows + auditd on the host worker nodes capture all
access to ITAR data.
Rancher (or Sidero Omni) sits above this, presenting each tenant's
vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
access per tenant group claim.
---
## 7. Storage Integration
Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:
| StorageClass | Provisioner | Backing | Speed | Use |
|--------------|------------|---------|-------|-----|
| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |
### 7.1 CSI driver notes
- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
(CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
NFS server + base export path.
- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
with one manifest. Uses wnode's kubelet root dir.
### 7.2 ITAR data isolation
The ITAR tenant should target a dedicated NFS export, not shared
`nfs-hdd`. Recommended:
- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
`nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
namespace via RBAC + NetworkPolicy.
---
## 8. Local Image Registry
### 8.1 Recommendation: Harbor on D3 SSD
D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
for a **Harbor** instance:
| Property | Value |
|----------|-------|
| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |
### 8.2 Pull-through cache benefit
Cluster nodes have zero internet egress (per §3.3). Without a local cache,
image pulls fail. With Harbor as a pull-through cache:
```mermaid
sequenceDiagram
WNODE->>HARBOR: docker pull nginx:1.25
alt cache hit
HARBOR-->>WNODE: layer bytes (LAN-speed)
else cache miss
HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
DOCKERHUB-->>HARBOR: layer bytes
HARBOR-->>WNODE: layer bytes (cached for next time)
end
```
Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
machine in the cluster with container-registry internet egress, and that
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
ghcr.io).
### 8.3 Supply-chain integrity (future)
Harbor + Cosign lets you require that all images deployed to the ITAR
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
control. Implementation deferred to a later session.
---
## 9. Bootstrap Procedure
### 9.1 One-time setup
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant GIT as Git repo
participant ONEPW as 1Password
participant PBS as PBS (tsys4)
OP->>GIT: Clone PFVCluster repo
OP->>ONEPW: Generate Talos secrets bundle (offline)
ONEPW-->>OP: secrets.yaml
OP->>PBS: Backup secrets.yaml (encrypted)
OP->>GIT: Write machine configs (no secrets)
```
### 9.2 Provision first cnode (bootstrap)
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant PX as Proxmox host
participant C1 as cnode1
participant ETCD as etcd (new)
OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
OP->>PX: qm start VMID
C1->>C1: Boots Talos (no config yet, "maintenance mode")
OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
C1->>C1: Applies config, restarts services
OP->>C1: talosctl bootstrap
C1->>ETCD: Initialize single-node raft
ETCD-->>C1: ready
OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
OP->>C1: talosctl etcd snapshot (initial backup → PBS)
```
### 9.3 Add second and third cnodes
```mermaid
sequenceDiagram
autonumber
participant OP as Operator
participant PX as Proxmox host
participant C2 as cnode2
participant C3 as cnode3
participant C1 as cnode1 (existing)
OP->>PX: qm create + start cnode2 VM
C2->>C2: Boots Talos maintenance mode
OP->>C2: talosctl apply --patch cnode2.yaml
C2->>C1: Join etcd cluster
OP->>PX: qm create + start cnode3 VM
C3->>C3: Boots Talos maintenance mode
OP->>C3: talosctl apply --patch cnode3.yaml
C3->>C1: Join etcd cluster
Note over C1,C3: etcd now has 3/3 members → HA quorum
```
### 9.4 Post-bootstrap cluster configuration
Once 3 cnodes are up and joined:
1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
2. **Install CSI drivers** — nfs-csi + local-path provisioner.
3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
6. **Apply default-deny NetworkPolicy** in all namespaces.
7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
8. **First etcd snapshot** + automated daily snapshot cron → PBS.
### 9.5 Add workers
Workers are simpler (no etcd):
```mermaid
sequenceDiagram
OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
WNODE->>WNODE: Boots maintenance mode
OP->>WNODE: talosctl apply --patch worker-X.yaml
WNODE->>C1: Kubelet registers with API server
C1-->>WNODE: Approved (auto via bootstrap token)
Note over WNODE: Joins cluster, becomes Ready
```
---
## 10. Disaster Recovery
### 10.1 Backup strategy
| Artifact | Frequency | Storage | Tool |
|----------|-----------|---------|------|
| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
| **Harbor metadata** | Daily | PBS | Harbor built-in backup |
### 10.2 Restore scenarios
**Lost 1 cnode (e.g., tsys9 disk failure):**
1. Provision new VM on tsys9 (or other host with local SSD).
2. Apply cnode2 machine config patch.
3. New cnode joins etcd, syncs state from survivors.
4. Quorum was never lost (2 of 3 alive throughout).
**Lost 2 cnodes simultaneously (quorum lost):**
1. Use surviving cnode's etcd snapshot.
2. Provision 3 new cnode VMs.
3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
4. Join other 2 cnodes.
5. Workers reconnect automatically once API server is back.
**Total cluster loss (all 3 cnodes):**
1. Restore from latest PBS etcd snapshot.
2. Provision new cnode VMs.
3. `talosctl bootstrap --recover-from=snapshot.db`.
4. Re-join workers.
5. Verify tenant vclusters restored.
### 10.3 Recovery time objectives
| Scenario | RTO | RPO |
|----------|-----|-----|
| Single cnode failure | < 30 min | 0 (no data loss) |
| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
| Total cluster loss | < 4 hours | ≤ 24 hours |
---
## 11. Migration from Current State
### 11.1 Current state
- 5 cnode VMs exist (Debian stock + Tailscale).
- **No k3s deployed yet.** Cluster was never bootstrapped.
- 6 wnode VMs exist (some stopped).
- No workloads running in k8s.
### 11.2 Migration: clean cutover (not a migration)
Since there is no etcd data and no workloads to preserve, the path is a
**clean rebuild**:
| Phase | Action | Risk |
|-------|--------|------|
| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |
### 11.3 Open dependency: Friday hardware work
Phases 1-2 require:
- **tsys3 local-lvm available.** Per
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
free NVMe local-lvm. Currently unused. **Ready.**
- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
[`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
Harbor depends on D3 being available on tsys5.
Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
Harbor registry does.
---
## Appendix: Open questions for next session
1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
(tsys9, tsys1, tsys3).
3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
Defer until cluster is up.
5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
approval of `192.168.3.0/24` route advertisement.
6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
general capacity; tsys3 + tsys5 (local fast storage) reserved for
non-ITAR HPC.