mrcharles
174deea2a9
Author the docs/k8s/ directory capturing the pfv-k8s control-plane design:
- README.md: TL;DR of all decisions (distro, runtime, cnode count, admin
access, identity, tenancy, registry, storage)
- DISTRO-DECISION.md: Talos vs k3s analysis. Recommend Talos because the
ITAR/classified requirement makes its immutable, API-only, measured-boot
posture structurally easier to certify than SCAP-hardened Debian. k3s was
only ever a plan (no cluster deployed), so cutover cost is ~zero.
- ARCHITECTURE.md: target arch with mermaid diagrams covering control
plane, LAN-only network with Tailscale subnet-router admin, Cilium CNI,
OIDC to Keycloak, per-tenant vcluster isolation (incl. ITAR tenant),
Harbor pull-through cache on D3 SSD, bootstrap sequence, and DR.
Gardening: register docs/k8s/ in docmap.md, update STATUS.md with the new
k8s section and the three pending user decisions (cnode count, host spread,
Tailscale pattern).
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
Kubernetes Architecture & Build Plan
Status: Draft for review. Companion to
../proxmox/K8S.md(which captured the storage/host analysis from the Proxmox audit). This directory takes the next step: which distro, how to build it, how to operate it.
Last updated: 2026-07-28
Documents in this directory
| Document | Purpose |
|---|---|
DISTRO-DECISION.md |
Talos vs k3s analysis. Recommendation: Talos, with rationale grounded in the ITAR/classified requirement. |
ARCHITECTURE.md |
Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
TL;DR
| Decision | Recommendation | Why |
|---|---|---|
| Distro | Talos Linux (not k3s) | Immutable, API-only, measured-boot-capable. Intrinsically close to a STIG/CIS posture, where k3s-on-Debian requires constant re-certification. The k3s plan was reasonable before the ITAR/classified requirement was on the table. |
| Runtime | containerd (Talos default) | Only runtime Talos supports; also the secure default. Non-decision. |
| Cnode count | 3 (down from 5) | Standard HA. Tolerates 1 failure. Frees 2 VMs for tenant worker capacity. Re-evaluate if a regulator mandates 5. |
| Admin access | Tailscale subnet router on existing tailscale-router VM |
Cluster nodes stay LAN-only (no internet egress = stronger ITAR posture). Operator reaches Talos API from anywhere via Tailscale. |
| Cluster network | LAN mTLS only | All cnode/wnode traffic on Proxmox LAN. No Tailscale on cluster nodes themselves. |
| Identity | OIDC to Keycloak on Cloudron (production) | Already planned. Talos API + Kubernetes API both trust Keycloak tokens. |
| Multi-tenancy | vcluster (per tenant) inside Talos host cluster | Strong workload isolation. One vcluster per tenant: ITAR, non-ITAR, SLP, RackRental. |
| Local registry | Harbor on D3 SSD (tsys5, 445 GB free) | Avoids needless round-trips to Docker Hub / Cloudron. Adds image-signing (Cosign) capability for supply-chain integrity. |
| Storage classes | local-fast, nfs-hdd, nfs-ssd |
Per ../proxmox/K8S.md §6. Unchanged. |
What this directory does NOT cover (deferred)
- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
- vcluster per-tenant policy templates.
- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
These are tracked as future session work in ../../STATUS.md.
Open question for the user
- Cnode count: confirm 3 vs 5. Recommendation is 3 (rationale in
ARCHITECTURE.md§2). If your ITAR counsel requires 2-failure tolerance on the control plane, keep 5.