PFVCluster/k8s/docs/DISTRO-DECISION.md

Distro Decision: Talos Linux vs k3s

Recommendation: Talos Linux. The k3s-on-Debian plan was sound before the ITAR/classified requirement entered scope. Once classified workloads are on the table, Talos's immutable, API-only, measured-boot-capable posture is materially easier to certify and defend.

Last updated: 2026-07-28

---

1. Decision context

Factor	Constraint
Workload class	R&D + RackRental (containerlab) + ITAR / classified suborbital workloads + commercial (Starting Line Productions)
Compliance drivers	ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253)
Hardware	7 standalone Proxmox hosts (no pvecm), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI.
Network	Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access).
Current cnode state	Stock Debian VMs joined to Tailscale. No k8s distribution has been deployed yet. Clean cutover possible.
Operations	Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead.

---

2. Head-to-head comparison

2.1 ITAR / classified posture

Property	Talos Linux	k3s on Debian
Node OS mutability	Immutable rootfs (squashfs, read-only). Reboot returns to known-good state.	Mutable. apt install, file edits persist.
Shell / SSH access	None. No SSH daemon, no shell, no kubectl debug node shell.	Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface.
Operational surface	Single gRPC API (mTLS, signed certs, audit log) on port 50000.	SSH + kubelet API + etcd API + package manager + cron + systemd + userland.
Measured boot	Supported. TPM attestation can prove the node booted the signed Talos image you pinned.	Possible but bolt-on; auditors will ask why you didn't disable the bootloader first.
Configuration provenance	Entire node state is a YAML machine config in Git. talosctl apply is the only mutation path.	Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply.
Supply chain	Every Talos release is a signed artifact (cosign). Pin by image digest.	Debian package provenance is good but the surface is enormous (~30K packages in a base install).
Forensic readiness	API log + kernel log + Talos event log = sufficient for "what ran, when, with what config."	Same possible but requires explicit configuration to be trustworthy.
STIG / CIS conformance	Intrinsically close. Talos publishes CIS benchmark results per release.	Requires running SCAP-STIG (already in this repo) and remediating findings continuously.

Bottom line: For classified workloads, an auditor's first question is "how do you prevent unauthorized changes to a node?" Talos's answer is "the OS is immutable and the only path is a signed API call." k3s's answer is"SSH is locked down and we scan with STIG." The first is structurally stronger; the second is operationally maintained.

2.2 Operational considerations

Property	Talos	k3s
Familiarity	New model (talosctl apply, no SSH). Learning curve.	Stock Debian + k3s binary. Familiar.
Debugging	talosctl logs, talosctl dmesg, talosctl dashboard. No shell.	ssh, journalctl, crictl. Full shell.
Tailscale integration	System extension (siderolabs/tailscale). Stable since Talos 1.3.	Native — apt install tailscale. Zero friction.
Backup / DR	talosctl etcd snapshot (one command). Cluster can be restored from snapshot + machine configs.	DIY (etcdctl snapshot + manual cert management).
Upgrades	talosctl upgrade — atomic, automated rollback on health-check failure.	Manual: drain, k3s package update, reboot, uncordon.
Proxmox compatibility	QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported.	Same.
Ecosystem maturity	Production-grade. Sidero (the company) offers Omni (managed control plane for Talos).	Production-grade. Rancher (SUSE) backs it.

2.3 Cost of choosing Talos over the existing k3s plan

The cnodes are currently stock Debian VMs joined to Tailscale. Critically, no k3s cluster has been deployed yet — k3s was only the plan. Therefore:

• No etcd data to migrate. Clean cutover, not a migration.
• No workloads to drain. The cluster is empty.
• Cnode VMs get re-imaged with Talos QCOW2 (or rebuilt from scratch — either way it's a qm script, not a stateful migration).
• Tailscale config shifts from "installed via apt" to "Talos system extension." (Or, per our recommendation in ARCHITECTURE.md §3, Tailscale moves off the cluster nodes entirely and onto the existing tailscale-router bastion as a subnet router. Cluster nodes become LAN-only.)

Net cost: rebuilding 3 cnode VMs as Talos + writing ~200 lines of machine config YAML. The hardening investment already encoded in provisioning/Modules/Security/ is not wasted — it still applies to every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only the cnodes/wnodes move to Talos.

---

3. Tailscale compatibility (deep-dive)

Tailscale on Talos is well-supported but introduces a configuration dimension worth being explicit about. Three patterns exist:

Pattern A — Tailscale on every cluster node (what you have now, on Debian)

Each cnode/wnode runs tailscaled and joins the tailnet. Cluster nodes have internet egress (to Tailscale DERP servers and for coordinate).

• Talos implementation: add siderolabs/tailscale system extension to each machine config, configure machine.network.interfaces.
• Pros: Operator can hit any node's Talos API from any Tailscale device.
• Cons: Cluster nodes have internet egress. For ITAR workloads, this is a finding (data exfiltration path).

Pattern B — Tailscale on bastion only, SSH/API jump

Cluster nodes are LAN-only. Operator Tunnels to bastion (existing tailscale-router VM), then runs talosctl from the bastion.

• Pros: Zero internet egress from cluster nodes.
• Cons: Two-step access. Bastion must run recent talosctl. Each operator action originates from the bastion (auditable but clunky).

Pattern C — Tailscale subnet router on bastion (recommended)

The existing tailscale-router VM advertises the cluster LAN subnet (e.g. 192.168.3.0/24) into the tailnet as a subnet route. Operator's Tailscale client transparently routes cluster-bound traffic through the bastion. From the operator's workstation, talosctl --nodes 192.168.3.x "just works."

• Pros:
  • Cluster nodes have zero internet egress (strongest ITAR posture).
  • Operator UX is unchanged from direct LAN access.
  • All access is mediated by Tailscale's identity + ACLs (already integrated with your env).
  • Audit trail lives in Tailscale + bastion logs.
• Cons:
  • Bastion becomes a dependency for remote admin (LAN-local admin still works without it).
  • Must enable IP forwarding + subnet route approval in Tailscale ACLs.

Recommendation: Pattern C. Documented in ARCHITECTURE.md §3.

---

4. Recommendation

Deploy Talos Linux as the k8s distribution for pfv-k8s.

Justification

1. Compliance posture is structural, not operational. "Immutable, API-only, measured-boot" is a property of Talos itself; "STIG-hardened" is a property of how Debian is operated. The first is dramatically easier to argue to an ITAR counsel or classified accreditation officer (DSS, DCSA) than the second.
2. Zero migration cost. The k3s cluster was never deployed. Reimaging 3 cnodes with Talos is a qm script invocation, not a stateful migration. The sunk cost of "we planned k3s" is zero deployed state.
3. Operational headroom. Talos's etcd snapshot + upgrade --stage + apply-mode auto reduce solo-founder ops burden. k3s is simpler to learn but more error-prone to operate at HA.

Acknowledged tradeoffs

• Learning curve. The Talos mental model (machine config + talosctl) replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
• No shell debugging. When something breaks on a node, you cannot ssh in. Mitigation: talosctl logs/support produces a support bundle equivalent to a sosreport.
• Hardware/module surprises. Talos ships a curated kernel. Anything beyond virtio + common NIC drivers needs a system extension. On Proxmox VMs this is not expected to be a problem — virtio is the path.
• Tailscale via system extension. Adds one config dimension per node. Mitigated by Pattern C (above), which removes Tailscale from cluster nodes entirely.

What we keep from the k3s mental model

• Single binary on each node semantics (Talos is conceptually similar).
• kubectl workflow unchanged. Talos exposes a standard Kubernetes API. kubectl, helm, kustomize all work as-is.
• Storage CSI choices (local-fast, nfs-hdd, nfs-ssd) are distro-independent.

---

5. What we are NOT deciding here

Topic	Deferred to
ETL tooling (GDAL/PostGIS/xarray/Dask)	Future session — affects StorageClass RWX/RWO design
HPC scheduler (Jobs/Argo/Volcano)	Future session — affects taint/label strategy
Per-tenant vcluster policy templates	Future session, post-bootstrap
Solar-aware scale-out hosts	Future capacity planning session
Container network plugin (CNI) details	Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants)

---

6. Next step

Proceed to ARCHITECTURE.md for the control-plane design, network topology, identity flow, and bootstrap procedure.