# Distro Decision: Talos Linux vs k3s > **Recommendation: Talos Linux.** > The k3s-on-Debian plan was sound before the ITAR/classified requirement > entered scope. Once classified workloads are on the table, Talos's > immutable, API-only, measured-boot-capable posture is materially easier > to certify and defend. **Last updated:** 2026-07-28 --- ## 1. Decision context | Factor | Constraint | |--------|-----------| | **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) | | **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) | | **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. | | **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). | | **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. | | **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. | --- ## 2. Head-to-head comparison ### 2.1 ITAR / classified posture | Property | Talos Linux | k3s on Debian | |----------|-------------|---------------| | **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. | | **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. | | **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. | | **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. | | **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. | | **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). | | **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. | | **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. | **Bottom line:** For classified workloads, an auditor's first question is "how do you prevent unauthorized changes to a node?" Talos's answer is "the OS is immutable and the only path is a signed API call." k3s's answer is"SSH is locked down and we scan with STIG." The first is structurally stronger; the second is operationally maintained. ### 2.2 Operational considerations | Property | Talos | k3s | |----------|-------|-----| | **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. | | **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. | | **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. | | **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). | | **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. | | **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. | | **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. | ### 2.3 Cost of choosing Talos over the existing k3s plan The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically, **no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore: - **No etcd data to migrate.** Clean cutover, not a migration. - **No workloads to drain.** The cluster is empty. - **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch — either way it's a `qm` script, not a stateful migration). - **Tailscale config shifts** from "installed via apt" to "Talos system extension." (Or, per our recommendation in [`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster nodes entirely** and onto the existing `tailscale-router` bastion as a subnet router. Cluster nodes become LAN-only.) **Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of machine config YAML. The hardening investment already encoded in `provisioning/Modules/Security/` is **not wasted** — it still applies to every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only the cnodes/wnodes move to Talos. --- ## 3. Tailscale compatibility (deep-dive) Tailscale on Talos is well-supported but introduces a configuration dimension worth being explicit about. Three patterns exist: ### Pattern A — Tailscale on every cluster node (what you have now, on Debian) Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have internet egress (to Tailscale DERP servers and for coordinate). - **Talos implementation:** add `siderolabs/tailscale` system extension to each machine config, configure `machine.network.interfaces`. - **Pros:** Operator can hit any node's Talos API from any Tailscale device. - **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a finding (data exfiltration path). ### Pattern B — Tailscale on bastion only, SSH/API jump Cluster nodes are LAN-only. Operator Tunnels to bastion (existing `tailscale-router` VM), then runs `talosctl` from the bastion. - **Pros:** Zero internet egress from cluster nodes. - **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator action originates from the bastion (auditable but clunky). ### Pattern C — Tailscale subnet router on bastion (recommended) The existing `tailscale-router` VM advertises the cluster LAN subnet (e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's Tailscale client transparently routes cluster-bound traffic through the bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x` "just works." - **Pros:** - Cluster nodes have **zero internet egress** (strongest ITAR posture). - Operator UX is unchanged from direct LAN access. - All access is mediated by Tailscale's identity + ACLs (already integrated with your env). - Audit trail lives in Tailscale + bastion logs. - **Cons:** - Bastion becomes a dependency for remote admin (LAN-local admin still works without it). - Must enable IP forwarding + subnet route approval in Tailscale ACLs. **Recommendation: Pattern C.** Documented in [`ARCHITECTURE.md`](ARCHITECTURE.md) §3. --- ## 4. Recommendation **Deploy Talos Linux** as the k8s distribution for `pfv-k8s`. ### Justification 1. **Compliance posture is structural, not operational.** "Immutable, API-only, measured-boot" is a property of Talos itself; "STIG-hardened" is a property of how Debian is operated. The first is dramatically easier to argue to an ITAR counsel or classified accreditation officer (DSS, DCSA) than the second. 2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging 3 cnodes with Talos is a `qm` script invocation, not a stateful migration. The sunk cost of "we planned k3s" is **zero deployed state**. 3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` + `apply-mode auto` reduce solo-founder ops burden. k3s is simpler to learn but more error-prone to operate at HA. ### Acknowledged tradeoffs - **Learning curve.** The Talos mental model (`machine config` + `talosctl`) replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops. - **No shell debugging.** When something breaks on a node, you cannot `ssh` in. Mitigation: `talosctl logs/support` produces a support bundle equivalent to a sosreport. - **Hardware/module surprises.** Talos ships a curated kernel. Anything beyond virtio + common NIC drivers needs a system extension. On Proxmox VMs this is **not expected to be a problem** — virtio is the path. - **Tailscale via system extension.** Adds one config dimension per node. Mitigated by Pattern C (above), which removes Tailscale from cluster nodes entirely. ### What we keep from the k3s mental model - **Single binary on each node** semantics (Talos is conceptually similar). - **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes API. `kubectl`, `helm`, `kustomize` all work as-is. - **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are distro-independent. --- ## 5. What we are NOT deciding here | Topic | Deferred to | |--------|-------------| | ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design | | HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy | | Per-tenant vcluster policy templates | Future session, post-bootstrap | | Solar-aware scale-out hosts | Future capacity planning session | | Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) | --- ## 6. Next step Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane design, network topology, identity flow, and bootstrap procedure.