docs: migrate all documentation to Discourse wiki topics
All knowledge docs (architecture, runbooks, references, audits, policies) have been migrated to community.turnsys.com as wiki topics in the VP TechOps category. Discourse is now the canonical source of truth for documentation; git edit history no longer serves as the doc changelog. 37 .md files converted to short pointers linking to their Discourse topics. AGENTS.md updated with new documentation workflow policy. Code (scripts, configs, playbooks) remains authoritative in git.
This commit is contained in:
+9
-187
@@ -1,188 +1,10 @@
|
||||
# Distro Decision: Talos Linux vs k3s
|
||||
# k8s/docs/DISTRO-DECISION.md
|
||||
|
||||
> **Recommendation: Talos Linux.**
|
||||
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
|
||||
> entered scope. Once classified workloads are on the table, Talos's
|
||||
> immutable, API-only, measured-boot-capable posture is materially easier
|
||||
> to certify and defend.
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision context
|
||||
|
||||
| Factor | Constraint |
|
||||
|--------|-----------|
|
||||
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
|
||||
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
|
||||
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
|
||||
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
|
||||
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
|
||||
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Head-to-head comparison
|
||||
|
||||
### 2.1 ITAR / classified posture
|
||||
|
||||
| Property | Talos Linux | k3s on Debian |
|
||||
|----------|-------------|---------------|
|
||||
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
|
||||
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
|
||||
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
|
||||
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
|
||||
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
|
||||
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
|
||||
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
|
||||
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
|
||||
|
||||
**Bottom line:** For classified workloads, an auditor's first question is
|
||||
"how do you prevent unauthorized changes to a node?" Talos's answer is
|
||||
"the OS is immutable and the only path is a signed API call." k3s's answer
|
||||
is"SSH is locked down and we scan with STIG." The first is structurally
|
||||
stronger; the second is operationally maintained.
|
||||
|
||||
### 2.2 Operational considerations
|
||||
|
||||
| Property | Talos | k3s |
|
||||
|----------|-------|-----|
|
||||
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
|
||||
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
|
||||
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
|
||||
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
|
||||
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
|
||||
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
|
||||
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
|
||||
|
||||
### 2.3 Cost of choosing Talos over the existing k3s plan
|
||||
|
||||
The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
|
||||
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
|
||||
|
||||
- **No etcd data to migrate.** Clean cutover, not a migration.
|
||||
- **No workloads to drain.** The cluster is empty.
|
||||
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
|
||||
either way it's a `qm` script, not a stateful migration).
|
||||
- **Tailscale config shifts** from "installed via apt" to "Talos system
|
||||
extension." (Or, per our recommendation in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
|
||||
nodes entirely** and onto the existing `tailscale-router` bastion as a
|
||||
subnet router. Cluster nodes become LAN-only.)
|
||||
|
||||
**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
|
||||
machine config YAML. The hardening investment already encoded in
|
||||
`provisioning/Modules/Security/` is **not wasted** — it still applies to
|
||||
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
|
||||
the cnodes/wnodes move to Talos.
|
||||
|
||||
---
|
||||
|
||||
## 3. Tailscale compatibility (deep-dive)
|
||||
|
||||
Tailscale on Talos is well-supported but introduces a configuration dimension
|
||||
worth being explicit about. Three patterns exist:
|
||||
|
||||
### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
|
||||
|
||||
Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
|
||||
internet egress (to Tailscale DERP servers and for coordinate).
|
||||
|
||||
- **Talos implementation:** add `siderolabs/tailscale` system extension to
|
||||
each machine config, configure `machine.network.interfaces`.
|
||||
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
|
||||
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
|
||||
finding (data exfiltration path).
|
||||
|
||||
### Pattern B — Tailscale on bastion only, SSH/API jump
|
||||
|
||||
Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
|
||||
`tailscale-router` VM), then runs `talosctl` from the bastion.
|
||||
|
||||
- **Pros:** Zero internet egress from cluster nodes.
|
||||
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
|
||||
action originates from the bastion (auditable but clunky).
|
||||
|
||||
### Pattern C — Tailscale subnet router on bastion (recommended)
|
||||
|
||||
The existing `tailscale-router` VM advertises the cluster LAN subnet
|
||||
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
|
||||
Tailscale client transparently routes cluster-bound traffic through the
|
||||
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
|
||||
"just works."
|
||||
|
||||
- **Pros:**
|
||||
- Cluster nodes have **zero internet egress** (strongest ITAR posture).
|
||||
- Operator UX is unchanged from direct LAN access.
|
||||
- All access is mediated by Tailscale's identity + ACLs (already integrated
|
||||
with your env).
|
||||
- Audit trail lives in Tailscale + bastion logs.
|
||||
- **Cons:**
|
||||
- Bastion becomes a dependency for remote admin (LAN-local admin still
|
||||
works without it).
|
||||
- Must enable IP forwarding + subnet route approval in Tailscale ACLs.
|
||||
|
||||
**Recommendation: Pattern C.** Documented in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
|
||||
|
||||
### Justification
|
||||
|
||||
1. **Compliance posture is structural, not operational.** "Immutable,
|
||||
API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
|
||||
is a property of how Debian is operated. The first is dramatically
|
||||
easier to argue to an ITAR counsel or classified accreditation officer
|
||||
(DSS, DCSA) than the second.
|
||||
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
|
||||
3 cnodes with Talos is a `qm` script invocation, not a stateful
|
||||
migration. The sunk cost of "we planned k3s" is **zero deployed state**.
|
||||
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
|
||||
`apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
|
||||
learn but more error-prone to operate at HA.
|
||||
|
||||
### Acknowledged tradeoffs
|
||||
|
||||
- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
|
||||
replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
|
||||
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
|
||||
in. Mitigation: `talosctl logs/support` produces a support bundle
|
||||
equivalent to a sosreport.
|
||||
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
|
||||
beyond virtio + common NIC drivers needs a system extension. On Proxmox
|
||||
VMs this is **not expected to be a problem** — virtio is the path.
|
||||
- **Tailscale via system extension.** Adds one config dimension per node.
|
||||
Mitigated by Pattern C (above), which removes Tailscale from cluster
|
||||
nodes entirely.
|
||||
|
||||
### What we keep from the k3s mental model
|
||||
|
||||
- **Single binary on each node** semantics (Talos is conceptually similar).
|
||||
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
|
||||
API. `kubectl`, `helm`, `kustomize` all work as-is.
|
||||
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
|
||||
distro-independent.
|
||||
|
||||
---
|
||||
|
||||
## 5. What we are NOT deciding here
|
||||
|
||||
| Topic | Deferred to |
|
||||
|--------|-------------|
|
||||
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
|
||||
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
|
||||
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
|
||||
| Solar-aware scale-out hosts | Future capacity planning session |
|
||||
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Next step
|
||||
|
||||
Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
|
||||
design, network topology, identity flow, and bootstrap procedure.
|
||||
> **Documentation moved to Discourse — the canonical source of truth.**
|
||||
>
|
||||
> **Talos vs k3s distro analysis and decision**
|
||||
>
|
||||
> **Read it here:** https://community.turnsys.com/t/305
|
||||
>
|
||||
> *Migrated 2026-08-06. This file is kept as a pointer for git-browsing context.
|
||||
> Do not update content here — edit the Discourse wiki topic instead.*
|
||||
|
||||
Reference in New Issue
Block a user