PFVCluster/k8s/docs/DISTRO-DECISION.md

# Distro Decision: Talos Linux vs k3s

> **Recommendation: Talos Linux.**
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
> entered scope. Once classified workloads are on the table, Talos's
> immutable, API-only, measured-boot-capable posture is materially easier
> to certify and defend.

**Last updated:** 2026-07-28

---

## 1. Decision context

| Factor | Constraint |
|--------|-----------|
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |

---

## 2. Head-to-head comparison

### 2.1 ITAR / classified posture

| Property | Talos Linux | k3s on Debian |
|----------|-------------|---------------|
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |

**Bottom line:** For classified workloads, an auditor's first question is
"how do you prevent unauthorized changes to a node?" Talos's answer is
"the OS is immutable and the only path is a signed API call." k3s's answer
is"SSH is locked down and we scan with STIG." The first is structurally
stronger; the second is operationally maintained.

### 2.2 Operational considerations

| Property | Talos | k3s |
|----------|-------|-----|
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |

### 2.3 Cost of choosing Talos over the existing k3s plan

The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:

- **No etcd data to migrate.** Clean cutover, not a migration.
- **No workloads to drain.** The cluster is empty.
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
  either way it's a `qm` script, not a stateful migration).
- **Tailscale config shifts** from "installed via apt" to "Talos system
  extension." (Or, per our recommendation in
  [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
  nodes entirely** and onto the existing `tailscale-router` bastion as a
  subnet router. Cluster nodes become LAN-only.)

**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
machine config YAML. The hardening investment already encoded in
`provisioning/Modules/Security/` is **not wasted** — it still applies to
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
the cnodes/wnodes move to Talos.

---

## 3. Tailscale compatibility (deep-dive)

Tailscale on Talos is well-supported but introduces a configuration dimension
worth being explicit about. Three patterns exist:

### Pattern A — Tailscale on every cluster node (what you have now, on Debian)

Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
internet egress (to Tailscale DERP servers and for coordinate).

- **Talos implementation:** add `siderolabs/tailscale` system extension to
  each machine config, configure `machine.network.interfaces`.
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
  finding (data exfiltration path).

### Pattern B — Tailscale on bastion only, SSH/API jump

Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
`tailscale-router` VM), then runs `talosctl` from the bastion.

- **Pros:** Zero internet egress from cluster nodes.
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
  action originates from the bastion (auditable but clunky).

### Pattern C — Tailscale subnet router on bastion (recommended)

The existing `tailscale-router` VM advertises the cluster LAN subnet
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
Tailscale client transparently routes cluster-bound traffic through the
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
"just works."

- **Pros:**
  - Cluster nodes have **zero internet egress** (strongest ITAR posture).
  - Operator UX is unchanged from direct LAN access.
  - All access is mediated by Tailscale's identity + ACLs (already integrated
    with your env).
  - Audit trail lives in Tailscale + bastion logs.
- **Cons:**
  - Bastion becomes a dependency for remote admin (LAN-local admin still
    works without it).
  - Must enable IP forwarding + subnet route approval in Tailscale ACLs.

**Recommendation: Pattern C.** Documented in
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.

---

## 4. Recommendation

**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.

### Justification

1. **Compliance posture is structural, not operational.** "Immutable,
   API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
   is a property of how Debian is operated. The first is dramatically
   easier to argue to an ITAR counsel or classified accreditation officer
   (DSS, DCSA) than the second.
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
   3 cnodes with Talos is a `qm` script invocation, not a stateful
   migration. The sunk cost of "we planned k3s" is **zero deployed state**.
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
   `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
   learn but more error-prone to operate at HA.

### Acknowledged tradeoffs

- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
  replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
  in. Mitigation: `talosctl logs/support` produces a support bundle
  equivalent to a sosreport.
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
  beyond virtio + common NIC drivers needs a system extension. On Proxmox
  VMs this is **not expected to be a problem** — virtio is the path.
- **Tailscale via system extension.** Adds one config dimension per node.
  Mitigated by Pattern C (above), which removes Tailscale from cluster
  nodes entirely.

### What we keep from the k3s mental model

- **Single binary on each node** semantics (Talos is conceptually similar).
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
  API. `kubectl`, `helm`, `kustomize` all work as-is.
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
  distro-independent.

---

## 5. What we are NOT deciding here

| Topic | Deferred to |
|--------|-------------|
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
| Solar-aware scale-out hosts | Future capacity planning session |
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |

---

## 6. Next step

Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
design, network topology, identity flow, and bootstrap procedure.