mrcharles
428e9ecb1d
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
45 lines
2.4 KiB
Markdown
45 lines
2.4 KiB
Markdown
# Kubernetes Architecture & Build Plan
|
|
|
|
> **Status:** Draft for review. Companion to [`../proxmox/K8S.md`](../proxmox/K8S.md)
|
|
> (which captured the storage/host analysis from the Proxmox audit).
|
|
> This directory takes the next step: **which distro, how to build it,
|
|
> how to operate it.**
|
|
|
|
**Last updated:** 2026-07-28
|
|
|
|
## Documents in this directory
|
|
|
|
| Document | Purpose |
|
|
|----------|---------|
|
|
| [`DISTRO-DECISION.md`](DISTRO-DECISION.md) | Talos vs k3s analysis. Recommendation: **Talos**, with rationale grounded in the ITAR/classified requirement. |
|
|
| [`ARCHITECTURE.md`](ARCHITECTURE.md) | Target architecture: control plane, network, identity, storage, tenant isolation. Mermaid diagrams included. |
|
|
|
|
## TL;DR
|
|
|
|
| Decision | Recommendation | Why |
|
|
|----------|----------------|-----|
|
|
| **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. |
|
|
| **Runtime** | **containerd** | Talos/k3s default. |
|
|
| **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. |
|
|
| **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. |
|
|
| **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. |
|
|
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. |
|
|
| **Multi-tenancy** | **vcluster** (per tenant) | Future work. |
|
|
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. |
|
|
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. |
|
|
|
|
## What this directory does NOT cover (deferred)
|
|
|
|
- ETL tooling choice (GDAL/PostGIS/xarray/Dask) — affects RWX vs RWO design.
|
|
- HPC job scheduler (Jobs / Argo Workflows / Volcano) — affects taint/label strategy.
|
|
- vcluster per-tenant policy templates.
|
|
- Solar-aware scale-out (PowerEdge 19xx/2950 hosts) — capacity planning only.
|
|
|
|
These are tracked as future session work in [`../../STATUS.md`](../../STATUS.md).
|
|
|
|
## Open question for the user
|
|
|
|
1. **Cnode count: confirm 3 vs 5.** Recommendation is 3 (rationale in
|
|
[`ARCHITECTURE.md`](ARCHITECTURE.md) §2). If your ITAR counsel requires
|
|
2-failure tolerance on the control plane, keep 5.
|