refactor(docs): co-locate k8s/proxmox docs with their code + fix broken perf paths
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -0,0 +1,188 @@
|
||||
# Distro Decision: Talos Linux vs k3s
|
||||
|
||||
> **Recommendation: Talos Linux.**
|
||||
> The k3s-on-Debian plan was sound before the ITAR/classified requirement
|
||||
> entered scope. Once classified workloads are on the table, Talos's
|
||||
> immutable, API-only, measured-boot-capable posture is materially easier
|
||||
> to certify and defend.
|
||||
|
||||
**Last updated:** 2026-07-28
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision context
|
||||
|
||||
| Factor | Constraint |
|
||||
|--------|-----------|
|
||||
| **Workload class** | R&D + RackRental (containerlab) + **ITAR / classified** suborbital workloads + commercial (Starting Line Productions) |
|
||||
| **Compliance drivers** | ITAR (USML categories), possible classified handling (NIST 800-171, CNSSI 1253) |
|
||||
| **Hardware** | 7 standalone Proxmox hosts (no `pvecm`), managed via PDM. Live migration NOT available — disk moves via Proxmox "Storage Migrate" UI. |
|
||||
| **Network** | Gigabit symmetric fiber to residence. LAN-only cluster traffic desirable. Tailscale already in use (overlay for admin access). |
|
||||
| **Current cnode state** | Stock Debian VMs joined to Tailscale. **No k8s distribution has been deployed yet.** Clean cutover possible. |
|
||||
| **Operations** | Solo founder. Must be reproducible from Git, low-touch, low-debug-overhead. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Head-to-head comparison
|
||||
|
||||
### 2.1 ITAR / classified posture
|
||||
|
||||
| Property | Talos Linux | k3s on Debian |
|
||||
|----------|-------------|---------------|
|
||||
| **Node OS mutability** | Immutable rootfs (squashfs, read-only). Reboot returns to known-good state. | Mutable. `apt install`, file edits persist. |
|
||||
| **Shell / SSH access** | **None.** No SSH daemon, no shell, no `kubectl debug node` shell. | Full SSH + bash. STIG hardening reduces (does not eliminate) attack surface. |
|
||||
| **Operational surface** | Single gRPC API (mTLS, signed certs, audit log) on port 50000. | SSH + kubelet API + etcd API + package manager + cron + systemd + userland. |
|
||||
| **Measured boot** | Supported. TPM attestation can prove the node booted the signed Talos image you pinned. | Possible but bolt-on; auditors will ask why you didn't disable the bootloader first. |
|
||||
| **Configuration provenance** | Entire node state is a YAML machine config in Git. `talosctl apply` is the only mutation path. | Config drift via SSH edits, package updates, manual service restarts. STIG/CAT-IV findings multiply. |
|
||||
| **Supply chain** | Every Talos release is a signed artifact (cosign). Pin by image digest. | Debian package provenance is good but the surface is enormous (~30K packages in a base install). |
|
||||
| **Forensic readiness** | API log + kernel log + Talos event log = sufficient for "what ran, when, with what config." | Same possible but requires explicit configuration to be trustworthy. |
|
||||
| **STIG / CIS conformance** | Intrinsically close. Talos publishes CIS benchmark results per release. | Requires running SCAP-STIG (already in this repo) and remediating findings continuously. |
|
||||
|
||||
**Bottom line:** For classified workloads, an auditor's first question is
|
||||
"how do you prevent unauthorized changes to a node?" Talos's answer is
|
||||
"the OS is immutable and the only path is a signed API call." k3s's answer
|
||||
is"SSH is locked down and we scan with STIG." The first is structurally
|
||||
stronger; the second is operationally maintained.
|
||||
|
||||
### 2.2 Operational considerations
|
||||
|
||||
| Property | Talos | k3s |
|
||||
|----------|-------|-----|
|
||||
| **Familiarity** | New model (`talosctl apply`, no SSH). Learning curve. | Stock Debian + k3s binary. Familiar. |
|
||||
| **Debugging** | `talosctl logs`, `talosctl dmesg`, `talosctl dashboard`. No shell. | `ssh`, `journalctl`, `crictl`. Full shell. |
|
||||
| **Tailscale integration** | System extension (`siderolabs/tailscale`). Stable since Talos 1.3. | Native — `apt install tailscale`. Zero friction. |
|
||||
| **Backup / DR** | `talosctl etcd snapshot` (one command). Cluster can be restored from snapshot + machine configs. | DIY (`etcdctl snapshot` + manual cert management). |
|
||||
| **Upgrades** | `talosctl upgrade` — atomic, automated rollback on health-check failure. | Manual: drain, `k3s` package update, reboot, uncordon. |
|
||||
| **Proxmox compatibility** | QCOW2 image boots natively on KVM/QEMU. virtio-net, virtio-scsi, virtio-rng all supported. | Same. |
|
||||
| **Ecosystem maturity** | Production-grade. Sidero (the company) offers Omni (managed control plane for Talos). | Production-grade. Rancher (SUSE) backs it. |
|
||||
|
||||
### 2.3 Cost of choosing Talos over the existing k3s plan
|
||||
|
||||
The cnodes are currently **stock Debian VMs joined to Tailscale**. Critically,
|
||||
**no k3s cluster has been deployed yet** — k3s was only the *plan*. Therefore:
|
||||
|
||||
- **No etcd data to migrate.** Clean cutover, not a migration.
|
||||
- **No workloads to drain.** The cluster is empty.
|
||||
- **Cnode VMs get re-imaged** with Talos QCOW2 (or rebuilt from scratch —
|
||||
either way it's a `qm` script, not a stateful migration).
|
||||
- **Tailscale config shifts** from "installed via apt" to "Talos system
|
||||
extension." (Or, per our recommendation in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3, **Tailscale moves off the cluster
|
||||
nodes entirely** and onto the existing `tailscale-router` bastion as a
|
||||
subnet router. Cluster nodes become LAN-only.)
|
||||
|
||||
**Net cost:** rebuilding 3 cnode VMs as Talos + writing ~200 lines of
|
||||
machine config YAML. The hardening investment already encoded in
|
||||
`provisioning/Modules/Security/` is **not wasted** — it still applies to
|
||||
every non-cluster VM (netinfra, UCS, LibreNMS, SIEM, bastion, etc.). Only
|
||||
the cnodes/wnodes move to Talos.
|
||||
|
||||
---
|
||||
|
||||
## 3. Tailscale compatibility (deep-dive)
|
||||
|
||||
Tailscale on Talos is well-supported but introduces a configuration dimension
|
||||
worth being explicit about. Three patterns exist:
|
||||
|
||||
### Pattern A — Tailscale on every cluster node (what you have now, on Debian)
|
||||
|
||||
Each cnode/wnode runs `tailscaled` and joins the tailnet. Cluster nodes have
|
||||
internet egress (to Tailscale DERP servers and for coordinate).
|
||||
|
||||
- **Talos implementation:** add `siderolabs/tailscale` system extension to
|
||||
each machine config, configure `machine.network.interfaces`.
|
||||
- **Pros:** Operator can hit any node's Talos API from any Tailscale device.
|
||||
- **Cons:** Cluster nodes have internet egress. For ITAR workloads, this is a
|
||||
finding (data exfiltration path).
|
||||
|
||||
### Pattern B — Tailscale on bastion only, SSH/API jump
|
||||
|
||||
Cluster nodes are LAN-only. Operator Tunnels to bastion (existing
|
||||
`tailscale-router` VM), then runs `talosctl` from the bastion.
|
||||
|
||||
- **Pros:** Zero internet egress from cluster nodes.
|
||||
- **Cons:** Two-step access. Bastion must run recent `talosctl`. Each operator
|
||||
action originates from the bastion (auditable but clunky).
|
||||
|
||||
### Pattern C — Tailscale subnet router on bastion (recommended)
|
||||
|
||||
The existing `tailscale-router` VM advertises the cluster LAN subnet
|
||||
(e.g. `192.168.3.0/24`) into the tailnet as a **subnet route**. Operator's
|
||||
Tailscale client transparently routes cluster-bound traffic through the
|
||||
bastion. From the operator's workstation, `talosctl --nodes 192.168.3.x`
|
||||
"just works."
|
||||
|
||||
- **Pros:**
|
||||
- Cluster nodes have **zero internet egress** (strongest ITAR posture).
|
||||
- Operator UX is unchanged from direct LAN access.
|
||||
- All access is mediated by Tailscale's identity + ACLs (already integrated
|
||||
with your env).
|
||||
- Audit trail lives in Tailscale + bastion logs.
|
||||
- **Cons:**
|
||||
- Bastion becomes a dependency for remote admin (LAN-local admin still
|
||||
works without it).
|
||||
- Must enable IP forwarding + subnet route approval in Tailscale ACLs.
|
||||
|
||||
**Recommendation: Pattern C.** Documented in
|
||||
[`ARCHITECTURE.md`](ARCHITECTURE.md) §3.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation
|
||||
|
||||
**Deploy Talos Linux** as the k8s distribution for `pfv-k8s`.
|
||||
|
||||
### Justification
|
||||
|
||||
1. **Compliance posture is structural, not operational.** "Immutable,
|
||||
API-only, measured-boot" is a property of Talos itself; "STIG-hardened"
|
||||
is a property of how Debian is operated. The first is dramatically
|
||||
easier to argue to an ITAR counsel or classified accreditation officer
|
||||
(DSS, DCSA) than the second.
|
||||
2. **Zero migration cost.** The k3s cluster was never deployed. Reimaging
|
||||
3 cnodes with Talos is a `qm` script invocation, not a stateful
|
||||
migration. The sunk cost of "we planned k3s" is **zero deployed state**.
|
||||
3. **Operational headroom.** Talos's `etcd snapshot` + `upgrade --stage` +
|
||||
`apply-mode auto` reduce solo-founder ops burden. k3s is simpler to
|
||||
learn but more error-prone to operate at HA.
|
||||
|
||||
### Acknowledged tradeoffs
|
||||
|
||||
- **Learning curve.** The Talos mental model (`machine config` + `talosctl`)
|
||||
replaces SSH + systemd. Expect a one-week ramp for comfortable daily ops.
|
||||
- **No shell debugging.** When something breaks on a node, you cannot `ssh`
|
||||
in. Mitigation: `talosctl logs/support` produces a support bundle
|
||||
equivalent to a sosreport.
|
||||
- **Hardware/module surprises.** Talos ships a curated kernel. Anything
|
||||
beyond virtio + common NIC drivers needs a system extension. On Proxmox
|
||||
VMs this is **not expected to be a problem** — virtio is the path.
|
||||
- **Tailscale via system extension.** Adds one config dimension per node.
|
||||
Mitigated by Pattern C (above), which removes Tailscale from cluster
|
||||
nodes entirely.
|
||||
|
||||
### What we keep from the k3s mental model
|
||||
|
||||
- **Single binary on each node** semantics (Talos is conceptually similar).
|
||||
- **`kubectl` workflow unchanged.** Talos exposes a standard Kubernetes
|
||||
API. `kubectl`, `helm`, `kustomize` all work as-is.
|
||||
- **Storage CSI choices** (`local-fast`, `nfs-hdd`, `nfs-ssd`) are
|
||||
distro-independent.
|
||||
|
||||
---
|
||||
|
||||
## 5. What we are NOT deciding here
|
||||
|
||||
| Topic | Deferred to |
|
||||
|--------|-------------|
|
||||
| ETL tooling (GDAL/PostGIS/xarray/Dask) | Future session — affects StorageClass RWX/RWO design |
|
||||
| HPC scheduler (Jobs/Argo/Volcano) | Future session — affects taint/label strategy |
|
||||
| Per-tenant vcluster policy templates | Future session, post-bootstrap |
|
||||
| Solar-aware scale-out hosts | Future capacity planning session |
|
||||
| Container network plugin (CNI) details | Will be specified in ARCHITECTURE.md §4 — recommendation is Cilium (supports NetworkPolicy, BPF, and encrypted node-to-node traffic for ITAR tenants) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Next step
|
||||
|
||||
Proceed to [`ARCHITECTURE.md`](ARCHITECTURE.md) for the control-plane
|
||||
design, network topology, identity flow, and bootstrap procedure.
|
||||
Reference in New Issue
Block a user