KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(k8s): deploy 3-node k3s HA control plane over Tailscale

Browse Source 
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.

Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
  CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check

All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.

Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-28 12:21:33 -05:00
parent 174deea2a9
commit 8f1642bf96
13 changed files with 693 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docmap.md
+1
View File
@@ -42,6 +42,7 @@ Server provisioning, security hardening, DNS/NTP configuration.
| Document | Description | Last Reviewed |
|----------|-------------|---------------|
| [`../k8s/README.md`](../k8s/README.md) | k3s cluster setup scripts: wipe, bootstrap, taint, verify (3-node HA over Tailscale) | 2026-07-28 |
| [`../dns-cluster-setup/README.md`](../dns-cluster-setup/README.md) | Technitium DNS cluster setup: export, deploy, cluster, verify | 2026-07-28 |
| [`../tests/README.md`](../tests/README.md) | Test suite documentation: unit, security, validation tests | 2026-07-28 |
| [`../netinfra/pfv-netinfra-setup.md`](../netinfra/pfv-netinfra-setup.md) | pfv-netinfra-01/02 initial setup guide | 2026-07-27 |
docs/k8s/README.md
+9 -9
View File
@@ -18,15 +18,15 @@
| Decision | Recommendation | Why |
|----------|----------------|-----|
| **Distro** | **Talos Linux** (not k3s) | Immutable, API-only, measured-boot-capable. Intrinsically close to a STIG/CIS posture, where k3s-on-Debian requires constant re-certification. The k3s plan was reasonable before the ITAR/classified requirement was on the table. |
| **Runtime** | **containerd** (Talos default) | Only runtime Talos supports; also the secure default. Non-decision. |
| **Cnode count** | **3** (down from 5) | Standard HA. Tolerates 1 failure. Frees 2 VMs for tenant worker capacity. Re-evaluate if a regulator mandates 5. |
| **Admin access** | **Tailscale subnet router** on existing `tailscale-router` VM | Cluster nodes stay LAN-only (no internet egress = stronger ITAR posture). Operator reaches Talos API from anywhere via Tailscale. |
| **Cluster network** | LAN mTLS only | All cnode/wnode traffic on Proxmox LAN. No Tailscale on cluster nodes themselves. |
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Already planned. Talos API + Kubernetes API both trust Keycloak tokens. |
| **Multi-tenancy** | **vcluster** (per tenant) inside Talos host cluster | Strong workload isolation. One vcluster per tenant: ITAR, non-ITAR, SLP, RackRental. |
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Avoids needless round-trips to Docker Hub / Cloudron. Adds image-signing (Cosign) capability for supply-chain integrity. |
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Unchanged. |
| **Distro** | **k3s** (deployed) / **Talos** (for future ITAR) | k3s chosen for the regular R&D cluster now live on cnode1/2/3. Talos is the recommendation for when the ITAR/classified cluster comes online. |
| **Runtime** | **containerd** | Talos/k3s default. |
| **Cnode count** | **3** (deployed) | Standard HA. Tolerates 1 failure. |
| **Admin access** | **Tailscale (all nodes joined)** | Currently all cnodes are on Tailscale directly. For ITAR cluster, move to subnet-router pattern. |
| **Cluster network** | **Tailscale-only IPs** | All node-ip, advertise-address, TLS-SANs are 100.x Tailscale IPs. Zero LAN IPs in cluster state. |
| **Identity** | **OIDC to Keycloak** on Cloudron (production) | Future work. |
| **Multi-tenancy** | **vcluster** (per tenant) | Future work. |
| **Local registry** | **Harbor on D3 SSD** (tsys5, 445 GB free) | Future work. |
| **Storage classes** | `local-fast`, `nfs-hdd`, `nfs-ssd` | Per [`../proxmox/K8S.md`](../proxmox/K8S.md) §6. Future work. |
## What this directory does NOT cover (deferred)