PFVCluster/docs/k8s/ARCHITECTURE.md

# pfv-k8s Target Architecture (Talos)

> **Companion to:** [`DISTRO-DECISION.md`](DISTRO-DECISION.md) (why Talos),
> [`../proxmox/K8S.md`](../proxmox/K8S.md) (storage/host analysis from the
> Proxmox audit).

**Last updated:** 2026-07-28

---

## Table of Contents

1. [High-Level Architecture](#1-high-level-architecture)
2. [Control Plane](#2-control-plane)
3. [Network Topology](#3-network-topology)
4. [CNI: Cilium](#4-cni-cilium)
5. [Identity and Trust](#5-identity-and-trust)
6. [Tenant Isolation (vcluster)](#6-tenant-isolation-vcluster)
7. [Storage Integration](#7-storage-integration)
8. [Local Image Registry](#8-local-image-registry)
9. [Bootstrap Procedure](#9-bootstrap-procedure)
10. [Disaster Recovery](#10-disaster-recovery)
11. [Migration from Current State](#11-migration-from-current-state)

---

## 1. High-Level Architecture

```mermaid
flowchart TB
    subgraph RESIDENCE["Residence — Proxmox LAN"]
        subgraph CP["Talos Control Plane (3 cnodes)"]
            C1[cnode1<br/>tsys9 · local-SSD]
            C2[cnode2<br/>tsys9 · local-SSD]
            C3[cnode3<br/>tsys1 · local-HDD]
        end
        subgraph WP["Talos Worker Plane"]
            W3[wnode-tsys3<br/>NVMe · 28GB]
            W5[wnode-tsys5<br/>NVMe · 32-64GB]
            W6[wnode-tsys6<br/>NFS-HDD · 64-96GB]
            W7[wnode-tsys7<br/>NFS-HDD · 96-128GB]
            W9[wnode-tsys9<br/>local-SSD · 4-8GB]
        end
        ETCD[(etcd<br/>raft, mTLS)]
        REG[(Harbor registry<br/>on D3 SSD · tsys5)]
        BASTION[tailscale-router VM<br/>subnet router]
    end

    subgraph TAILNET["Tailscale overlay"]
        OP[Operator devices]
    end

    subgraph CLOUDRON["Cloudron production — Reston VA"]
        KC[Keycloak OIDC IdP]
    end

    C1 ---|mTLS LAN| ETCD
    C2 ---|mTLS LAN| ETCD
    C3 ---|mTLS LAN| ETCD
    CP -->|pull images| REG
    WP -->|pull images| REG

    OP -->|Talos API :50000<br/>via subnet route| BASTION
    BASTION -.->|LAN| CP
    CP -->|OIDC| KC
    WP -->|OIDC| KC

    classDef talos fill:#1a1a2e,stroke:#e94560,color:#fff
    classDef infra fill:#0f3460,stroke:#e94560,color:#fff
    classDef external fill:#16213e,stroke:#533483,color:#fff
    class CP,WP,ETCD talos
    class REG,BASTION infra
    class OP,KC,EXTERNAL external
```

### Design principles

1. **LAN-only cluster nodes.** Zero internet egress from cnodes/wnodes.
   Strongest posture for ITAR/classified.
2. **Admin via Tailscale subnet router.** Existing `tailscale-router` VM
   advertises the cluster LAN subnet. Operator reaches Talos API from
   anywhere.
3. **Local-first storage.** Cnodes boot from local disk (no NFS dependency
   for etcd). Workers boot from local disk where available; NFS for bulk
   data only.
4. **Per-tenant vcluster.** Workload isolation via virtual clusters on top
   of the Talos host cluster.
5. **OIDC everywhere.** Talos API and Kubernetes API both trust Keycloak
   tokens. No long-lived static credentials for humans.

---

## 2. Control Plane

### 2.1 Recommendation: 3 cnodes (down from 5)

| Option | Quorum | Failure tolerance | etcd write cost | Resource cost |
|--------|--------|-------------------|-----------------|---------------|
| **3 cnodes** (recommended) | 2 of 3 | Tolerates **1** failure | Lower (faster commits) | 3 × (2c/4GB/32GB) = 6c / 12GB |
| 5 cnodes (current plan) | 3 of 5 | Tolerates **2** failures | Higher | 5 × (2c/4GB/32GB) = 10c / 20GB |

For a solo-operated R&D cluster, **3 cnodes is the HA standard**. The
failure-tolerance jump from 1→2 rarely justifies the doubled etcd write
quorum and the extra 4GB/2c per cnode. The 2 freed VM slots (and their
host capacity) are better spent on tenant worker allocations.

**Caveat:** if your ITAR/classified accreditation counsel mandates 2-failure
tolerance on the control plane, keep 5. Otherwise 3.

### 2.2 Cnode placement

Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §4.3, cnodes should use
**local-lvm boot disks** so etcd has no NFS dependency. Concrete placement:

| cnode | Host | Boot disk | Type | Why |
|-------|------|-----------|------|-----|
| cnode1 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Fastest available for etcd. |
| cnode2 | tsys9 | local-lvm (PNY CS900 SSD) | LOCAL-SSD | Same host, different disk OK (host failure is the failure domain, not disk). |
| cnode3 | tsys1 | local-lvm (HDD) | LOCAL-HDD | Host diversity. Slower than SSD but no NFS hop. |

**Quorum survival:**

| Failure | cnodes lost | Quorum OK? |
|---------|-------------|------------|
| tsys9 host dies | cnode1 + cnode2 | NO (1 of 3) — would need 4th cnode elsewhere, or accept this risk. |
| tsys1 host dies | cnode3 | YES (2 of 3) |
| Any storage server dies | 0 | YES (3 of 3) — local disks unaffected |

**Refinement:** putting both SSD cnodes on tsys9 means tsys9 host failure
loses quorum. Alternative: spread cnodes across 3 different hosts. See
"open question" at end of this section.

### 2.3 Machine config strategy

Talos nodes are configured by **machine configs** (YAML). Two flavors:

- **`controlplane.yaml`** — for cnodes. Enables etcd, scheduler,
  controller-manager, API server.
- **`worker.yaml`** — for wnodes. Joins cluster, runs kubelet + containerd.

Strategy for this cluster:

1. **One shared `talosconfig`** (client identity) — stored in 1Password
   and in the Proxmox Backup Server (PBS) encrypted backup target.
2. **Per-node machine config patches** — small patches on top of the base
   `controlplane.yaml` / `worker.yaml` for node-specific settings:
   - Hostname
   - Network interface + IP (DHCP or static — recommend static for cnodes)
   - Schematic image digest (pinned Talos version)
   - System extensions (e.g., `tailscale` — only if running Pattern A
     instead of recommended Pattern C)
3. **All machine configs in Git** under a future `k8s/talos-configs/`
   directory. Secrets are templated in at apply-time from 1Password / sops.

```mermaid
flowchart LR
    BASE[base controlplane.yaml] --> PATCH1[patch: cnode1]
    BASE --> PATCH2[patch: cnode2]
    BASE --> PATCH3[patch: cnode3]
    BASEW[base worker.yaml] --> PATCHW[patch: per-wnode]
    PATCH1 --> APPLY1[talosctl apply]
    PATCH2 --> APPLY2[talosctl apply]
    PATCH3 --> APPLY3[talosctl apply]
    PATCHW --> APPLYW[talosctl apply]
```

### 2.4 Open question: cnode host spread

If you accept "tsys9 failure = quorum loss" as a tolerable risk (solo R&D
cluster, tsys9 is brand-new hardware, single digit annual failure
probability), the layout in §2.2 is fine.

If not, alternative spread across 3 hosts:

| cnode | Host | Boot disk |
|-------|------|-----------|
| cnode1 | tsys9 | local-lvm SSD |
| cnode2 | tsys1 | local-lvm HDD |
| cnode3 | tsys3 | local-lvm NVMe |

tsys3's local-lvm is **349 GB Samsung PM961 NVMe** (per
[`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3) — currently unused,
would make an excellent etcd disk.

**This 3-host spread survives any single host failure with quorum intact.
Recommended.**

---

## 3. Network Topology

### 3.1 Zones

```mermaid
flowchart TB
    subgraph INTERNET["Internet"]
        FIBER[Gigabit symmetric fiber]
    end

    subgraph RESLAN["Residence LAN 192.168.x.x/24"]
        subgraph CLUSTERNET["Cluster nodes — LAN only, no egress"]
            CNODES[Cnodes 192.168.3.x]
            WNODES[Wnodes 192.168.3.x]
        end
        BASTION[tailscale-router<br/>192.168.3.x + 100.x.x.x]
        REG[Harbor registry<br/>192.168.3.x]
        STORAGE[NFS servers<br/>tsys4, tsys5]
    end

    subgraph TSNET["Tailscale 100.x.x.x/8"]
        OPS[Operator devices]
        KC[Keycloak<br/>via Cloudron prod]
    end

    FIBER --> BASTION
    BASTION <-. subnet route .-> CLUSTERNET
    OPS -->|TCP 50000 talos API| BASTION
    BASTION -->|LAN forward| CNODES
    CNODES -->|LAN mTLS| WNODES
    CNODES -->|OIDC HTTPS| KC
    WNODES -->|pull images| REG
    WNODES -->|bulk data IO| STORAGE
    CNODES -->|pull images| REG
```

### 3.2 Address plan (suggested)

Reserve a small contiguous block in the residence LAN for cluster nodes:

| Role | Range | Count |
|------|-------|-------|
| Cnodes | `192.168.3.31-33` | 3 |
| Wnodes | `192.168.3.41-49` | up to 9 (1 per Proxmox host + spare) |
| Bastion | existing `tailscale-router` | 1 |
| Registry | `192.168.3.50` | 1 (Harbor) |

**Static IPs are strongly recommended for cnodes** (etcd cluster membership
is hostname-based; stable IPs make `talosctl` targeting simple). Workers
can DHCP.

### 3.3 Firewall posture

Each cnode/wnode has:

- **Ingress** from LAN: TCP 50000 (Talos API), TCP 6443 (Kubernetes API on
  cnodes only), plus CNI ports (varies by CNI — see §4).
- **Ingress** from Tailscale: none (cluster nodes are not on Tailscale).
- **Egress:** LAN-only. Block all RFC1918-external traffic at the perimeter
  firewall for these IPs. ITAR workloads must not be able to phone home.

The bastion runs Tailscale and forwards TCP 50000/6443 to cluster nodes
via the subnet route.

---

## 4. CNI: Cilium

**Recommendation: Cilium** (eBPF-based CNI).

| Property | Why it matters here |
|----------|---------------------|
| **NetworkPolicy** (incl. L7) | Per-tenant isolation rules in vclusters. |
| **Node-to-node encryption** | WireGuard-based IPSec replacement. All inter-node pod traffic is encrypted on the wire. **Important for ITAR tenants.** |
| **Hubble** | Observable flows — forensic record of which pod talked to which. Useful for compliance evidence. |
| **No kube-proxy** | Cilium replaces kube-proxy with eBPF. Smaller attack surface on each node. |
| **Talos integration** | First-class. Talos docs document the install path. |

Cilium is deployed via Helm after cluster bootstrap. Node-to-node encryption
enabled. Default-deny NetworkPolicy applied per namespace.

---

## 5. Identity and Trust

### 5.1 Trust flow

```mermaid
sequenceDiagram
    autonumber
    participant Human as Operator
    participant TAIL as Tailscale
    participant BAST as Bastion
    participant TALOS as Talos API :50000
    participant KC as Keycloak (Cloudron)
    participant K8S as Kubernetes API :6443

    Human->>TAIL: Authenticate (device + SSO)
    TAIL-->>Human: Tailnet IP
    Human->>BAST: Reach bastion via tailnet
    BAST->>TALOS: Forward to LAN node :50000
    Human->>KC: OIDC login (browser)
    KC-->>Human: Bearer token (short-lived)
    Human->>TALOS: talosctl (mTLS with client cert)
    Human->>K8S: kubectl --oidc (Keycloak token)
    K8S->>KC: Validate token (introspection)
    KC-->>K8S: Valid + claims
    K8S-->>Human: Authorized response
```

### 5.2 Two distinct identity layers

| Layer | Mechanism | Audience |
|-------|-----------|----------|
| **Talos API** (node ops) | Mutual TLS with client certificate generated from the Talos secrets bundle. | Operators (automation + humans). |
| **Kubernetes API** (kubectl) | OIDC bearer token from Keycloak. RBAC maps group claims → ClusterRole. | Humans. Service accounts use projected tokens (no OIDC). |

The **Talos secrets bundle** is the root of trust for the cluster. Lose it
and you cannot operate the cluster; an attacker with it owns the cluster.
Storage:

1. **Primary:** 1Password (or equivalent) — operator-accessible.
2. **Backup:** PBS encrypted backup target on tsys4 (existing infra).
3. **NOT in Git.** Machine configs go in Git; secrets stay out.

### 5.3 Keycloak client configuration

On Cloudron-hosted Keycloak, register a client `pfv-k8s-talos`:

- **Authorization Code + PKCE flow** (no implicit, no password).
- **Redirect URIs:** `http://localhost:8000` (kubectl oidc-login) + Sidero
  Omni/Rancher URLs if/when those are added.
- **Group claims:** `k8s-admin`, `k8s-readonly`, `k8s-tenant-itar`,
  `k8s-tenant-rackrental`, etc. These map to Kubernetes RBAC `ClusterRoleBinding`.

---

## 6. Tenant Isolation (vcluster)

### 6.1 Why vcluster

[vcluster](https://www.vcluster.com/) runs a **virtual Kubernetes control
plane** (API server, scheduler, controller-manager, etcd) inside a namespace
of the host cluster. Tenant workloads run on the host's worker nodes but
are isolated by:

- Separate API server (tenant cannot see host cluster objects).
- Separate RBAC and admission control.
- Separate network policies (per-namespace).
- Separate resource quotas.

This aligns with the user's per-tenant plan from
[`../proxmox/K8S.md`](../proxmox/K8S.md) §1.

### 6.2 Tenant registry

| Tenant | Compliance | Workload example | vcluster name |
|--------|-----------|------------------|---------------|
| RackRental | None (internal R&D) | containerlab topology tests | `vc-rackrental` |
| Suborbital non-ITAR | EAR/ITAR-aware but unclassified | Payload telemetry processing | `vc-suborbital-open` |
| Suborbital ITAR | **ITAR-controlled** | Firmware build for USML items | `vc-suborbital-itar` |
| Starting Line Productions | Commercial | Customer media pipeline | `vc-slp` |

```mermaid
flowchart TB
    subgraph HOST["Talos host cluster"]
        CP[Host control plane<br/>3 cnodes · etcd · Keycloak RBAC]
        subgraph NS["Host cluster namespaces"]
            NS_RR[ns: vc-rackrental]
            NS_SO[ns: vc-suborbital-open]
            NS_SI[ns: vc-suborbital-itar]
            NS_SLP[ns: vc-slp]
        end
    end

    subgraph VRR["vcluster: vc-rackrental"]
        API_RR[k8s API + etcd]
    end
    subgraph VSO["vcluster: vc-suborbital-open"]
        API_SO[k8s API + etcd]
    end
    subgraph VSI["vcluster: vc-suborbital-itar"]
        API_SI[k8s API + etcd]
    end
    subgraph VSLP["vcluster: vc-slp"]
        API_SLP[k8s API + etcd]
    end

    CP --> NS_RR & NS_SO & NS_SI & NS_SLP
    NS_RR --> API_RR
    NS_SO --> API_SO
    NS_SI --> API_SI
    NS_SLP --> API_SLP

    classDef itar fill:#3a0000,stroke:#ff0000,color:#fff
    class NS_SI,API_SI itar
```

### 6.3 ITAR enforcement at host layer

For the ITAR tenant (`vc-suborbital-itar`), enforce additional host-layer
controls:

- **Node taint** `workload=itar:NoSchedule` on worker nodes dedicated to
  ITAR workloads (subset of wnodes, marked in node labels).
- **NetworkPolicy** default-deny egress for the `vc-suborbital-itar`
  namespace. Allow only explicit destinations (registry, NFS for ITAR
  data tier, Keycloak).
- **Storage isolation:** ITAR PVCs target a dedicated NFS export (e.g.,
  `D3-itar` on tsys5) that no other tenant can mount.
- **Audit:** Hubble flows + auditd on the host worker nodes capture all
  access to ITAR data.

Rancher (or Sidero Omni) sits above this, presenting each tenant's
vcluster as a separate "cluster" in its UI, with Keycloak SSO gating
access per tenant group claim.

---

## 7. Storage Integration

Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Three StorageClasses:

| StorageClass | Provisioner | Backing | Speed | Use |
|--------------|------------|---------|-------|-----|
| `local-fast` | local-path | wnode local disk (NVMe/SSD/HDD depending on host) | 100-3500 MB/s | Container runtime, scratch, ephemeral |
| `nfs-hdd` | nfs.csi.k8s.io | tsys4 D2/D5, tsys5 S1-S4 | 80-120 MB/s | Bulk data, weather/GIS datasets |
| `nfs-ssd` | nfs.csi.k8s.io | tsys5 D3, tsys5 T5-SSD | 200-400 MB/s | Latency-sensitive persistent data |

### 7.1 CSI driver notes

- **NFS CSI:** [`csi-driver-nfs`](https://github.com/kubernetes-csi/csi-driver-nfs)
  (CNCF sandbox). Deploys via Helm. Each StorageClass points at a specific
  NFS server + base export path.
- **local-path:** Rancher Local Path Provisioner. Single-binary, deploys
  with one manifest. Uses wnode's kubelet root dir.

### 7.2 ITAR data isolation

The ITAR tenant should target a dedicated NFS export, not shared
`nfs-hdd`. Recommended:

- Allocate `S4` on tsys5 (currently 99% empty, 435 GB free) as
  `nfs-itar` StorageClass. Mountable only from `vc-suborbital-itar`
  namespace via RBAC + NetworkPolicy.

---

## 8. Local Image Registry

### 8.1 Recommendation: Harbor on D3 SSD

D3 SSD (tsys5, post-Friday SAS relocation) is 445 GB and 99% empty. Use it
for a **Harbor** instance:

| Property | Value |
|----------|-------|
| **Storage** | D3 SSD on tsys5 (NFS export, fast tier) |
| **VM** | New VM `pfv-registry` on tsys5, local-nonprod boot, D3 data |
| **Function** | (a) Pull-through cache for Docker Hub / Quay / gcr.io<br/>(b) Host private images<br/>(c) Cosign image signing verification |
| **Exposure** | LAN-only. `192.168.3.50:443`. Not exposed to internet. |

### 8.2 Pull-through cache benefit

Cluster nodes have zero internet egress (per §3.3). Without a local cache,
image pulls fail. With Harbor as a pull-through cache:

```mermaid
sequenceDiagram
    WNODE->>HARBOR: docker pull nginx:1.25
    alt cache hit
        HARBOR-->>WNODE: layer bytes (LAN-speed)
    else cache miss
        HARBOR->>DOCKERHUB: pull nginx:1.25 (egress)
        DOCKERHUB-->>HARBOR: layer bytes
        HARBOR-->>WNODE: layer bytes (cached for next time)
    end
```

Cluster nodes pull from Harbor over LAN (gigabit). Harbor is the only
machine in the cluster with container-registry internet egress, and that
egress can be locked to specific upstreams (docker.io, quay.io, gcr.io,
ghcr.io).

### 8.3 Supply-chain integrity (future)

Harbor + Cosign lets you require that all images deployed to the ITAR
tenant are signed by a trusted key. This is a strong ITAR/CISA-attestation
control. Implementation deferred to a later session.

---

## 9. Bootstrap Procedure

### 9.1 One-time setup

```mermaid
sequenceDiagram
    autonumber
    participant OP as Operator
    participant GIT as Git repo
    participant ONEPW as 1Password
    participant PBS as PBS (tsys4)
    OP->>GIT: Clone PFVCluster repo
    OP->>ONEPW: Generate Talos secrets bundle (offline)
    ONEPW-->>OP: secrets.yaml
    OP->>PBS: Backup secrets.yaml (encrypted)
    OP->>GIT: Write machine configs (no secrets)
```

### 9.2 Provision first cnode (bootstrap)

```mermaid
sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C1 as cnode1
    participant ETCD as etcd (new)

    OP->>PX: qm create VM (Talos QCOW2 disk, local-lvm)
    OP->>PX: qm start VMID
    C1->>C1: Boots Talos (no config yet, "maintenance mode")
    OP->>C1: talosctl apply --patch cnode1.yaml (with secrets)
    C1->>C1: Applies config, restarts services
    OP->>C1: talosctl bootstrap
    C1->>ETCD: Initialize single-node raft
    ETCD-->>C1: ready
    OP->>C1: talosctl kubeconfig (fetch admin kubeconfig)
    OP->>C1: talosctl etcd snapshot (initial backup → PBS)
```

### 9.3 Add second and third cnodes

```mermaid
sequenceDiagram
    autonumber
    participant OP as Operator
    participant PX as Proxmox host
    participant C2 as cnode2
    participant C3 as cnode3
    participant C1 as cnode1 (existing)

    OP->>PX: qm create + start cnode2 VM
    C2->>C2: Boots Talos maintenance mode
    OP->>C2: talosctl apply --patch cnode2.yaml
    C2->>C1: Join etcd cluster
    OP->>PX: qm create + start cnode3 VM
    C3->>C3: Boots Talos maintenance mode
    OP->>C3: talosctl apply --patch cnode3.yaml
    C3->>C1: Join etcd cluster
    Note over C1,C3: etcd now has 3/3 members → HA quorum
```

### 9.4 Post-bootstrap cluster configuration

Once 3 cnodes are up and joined:

1. **Install Cilium** (CNI) via Helm. Enable node-to-node encryption.
2. **Install CSI drivers** — nfs-csi + local-path provisioner.
3. **Create StorageClasses** — `local-fast`, `nfs-hdd`, `nfs-ssd`.
4. **Deploy Harbor** on the `pfv-registry` VM, exposed at `192.168.3.50`.
5. **Configure Kubernetes API OIDC** — Keycloak client (§5.3).
6. **Apply default-deny NetworkPolicy** in all namespaces.
7. **Install vcluster CLI** + create 4 tenant vclusters (§6).
8. **First etcd snapshot** + automated daily snapshot cron → PBS.

### 9.5 Add workers

Workers are simpler (no etcd):

```mermaid
sequenceDiagram
    OP->>PX: qm create + start wnode-X VM (Talos QCOW2)
    WNODE->>WNODE: Boots maintenance mode
    OP->>WNODE: talosctl apply --patch worker-X.yaml
    WNODE->>C1: Kubelet registers with API server
    C1-->>WNODE: Approved (auto via bootstrap token)
    Note over WNODE: Joins cluster, becomes Ready
```

---

## 10. Disaster Recovery

### 10.1 Backup strategy

| Artifact | Frequency | Storage | Tool |
|----------|-----------|---------|------|
| **Talos secrets bundle** | Once (regen only on rotation) | 1Password + PBS (encrypted) | Manual |
| **Machine configs** | Continuous (Git) | Git remote + PBS | Git |
| **etcd snapshot** | Daily + before each change | PBS (tsys4 SMR target, 4.3 TB free) | `talosctl etcd snapshot` |
| **vcluster etcd** | Daily per vcluster | PBS | `kubectl exec ... etcdctl snapshot` |
| **Harbor metadata** | Daily | PBS | Harbor built-in backup |

### 10.2 Restore scenarios

**Lost 1 cnode (e.g., tsys9 disk failure):**

1. Provision new VM on tsys9 (or other host with local SSD).
2. Apply cnode2 machine config patch.
3. New cnode joins etcd, syncs state from survivors.
4. Quorum was never lost (2 of 3 alive throughout).

**Lost 2 cnodes simultaneously (quorum lost):**

1. Use surviving cnode's etcd snapshot.
2. Provision 3 new cnode VMs.
3. On first: `talosctl bootstrap --recover-from=snapshot.db`.
4. Join other 2 cnodes.
5. Workers reconnect automatically once API server is back.

**Total cluster loss (all 3 cnodes):**

1. Restore from latest PBS etcd snapshot.
2. Provision new cnode VMs.
3. `talosctl bootstrap --recover-from=snapshot.db`.
4. Re-join workers.
5. Verify tenant vclusters restored.

### 10.3 Recovery time objectives

| Scenario | RTO | RPO |
|----------|-----|-----|
| Single cnode failure | < 30 min | 0 (no data loss) |
| Quorum loss (2 cnodes) | < 2 hours | ≤ 24 hours (last snapshot) |
| Total cluster loss | < 4 hours | ≤ 24 hours |

---

## 11. Migration from Current State

### 11.1 Current state

- 5 cnode VMs exist (Debian stock + Tailscale).
- **No k3s deployed yet.** Cluster was never bootstrapped.
- 6 wnode VMs exist (some stopped).
- No workloads running in k8s.

### 11.2 Migration: clean cutover (not a migration)

Since there is no etcd data and no workloads to preserve, the path is a
**clean rebuild**:

| Phase | Action | Risk |
|-------|--------|------|
| **0. Prep** | Generate Talos secrets. Store in 1Password + PBS. Write machine configs to Git. | Low. |
| **1. Bootstrap 3 new cnodes** | Build 3 NEW Talos cnode VMs (not the existing 5). Use local-lvm boot disks (tsys9 × 2, tsys3 × 1 per §2.4 recommended spread). | Low. Existing Debian cnodes can keep running idle. |
| **2. Configure cluster** | Install Cilium, CSI, StorageClasses, OIDC, Harbor. | Low. |
| **3. Add workers** | Re-image existing wnode VMs as Talos, or build new ones. | Low. No workloads to drain. |
| **4. Decommission old Debian cnodes** | Once cluster is stable, shut down + delete the 5 old Debian cnode VMs. | Low. |
| **5. Tenant vclusters** | Stand up per-tenant vclusters. | Medium (policy tuning). |

### 11.3 Open dependency: Friday hardware work

Phases 1-2 require:

- **tsys3 local-lvm available.** Per
  [`../proxmox/PROJECT.md`](../proxmox/PROJECT.md) §3.3, tsys3 has 349 GB
  free NVMe local-lvm. Currently unused. **Ready.**
- **tsys9 local-lvm available.** 136 GB PNY CS900 SSD. **Ready.**
- **D3 SSD relocated to tsys5 SAS.** Currently USB on tsys4. Per
  [`../proxmox/TODO.md`](../proxmox/TODO.md) §2, scheduled for Friday.
  Harbor depends on D3 being available on tsys5.

Bootstrap of the cnodes does NOT depend on Friday hardware work. Only the
Harbor registry does.

---

## Appendix: Open questions for next session

1. **Confirm 3 vs 5 cnodes** (§2.1). Recommendation: 3.
2. **Confirm cnode host spread** (§2.4). Recommendation: 3-host spread
   (tsys9, tsys1, tsys3).
3. **Static IPs for cnodes** (§3.2). Recommendation: yes, `192.168.3.31-33`.
4. **Rancher vs Sidero Omni** for cluster management UI. Both viable.
   Defer until cluster is up.
5. **Subnet router ACL approval** on Tailscale admin console (§3). Needs
   approval of `192.168.3.0/24` route advertisement.
6. **ITAR worker node subset** (§6.3). Which wnodes are tainted for ITAR?
   Recommendation: tsys6 + tsys7 (heaviest hosts, NFS-only boot) as
   general capacity; tsys3 + tsys5 (local fast storage) reserved for
   non-ITAR HPC.