KNEL/PFVCluster
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(k8s): deploy 3-node k3s HA control plane over Tailscale

Browse Source 
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.

Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
  CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check

All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.

Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2
This commit is contained in:
mrcharles
2026-07-28 12:21:33 -05:00
parent 174deea2a9
commit 8f1642bf96
13 changed files with 693 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/README.md
+55
View File
@@ -0,0 +1,55 @@
# k8s/ — pfv-k8s Cluster Setup Scripts
Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
All cluster communication goes over Tailscale IPs — no LAN traffic.
## Current State
3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
| Node | Tailscale IP | Role | Tainted |
|------|-------------|------|---------|
| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
## Scripts
| Script | Purpose |
|--------|---------|
| [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
| [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). |
| [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
| [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). |
| [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. |
| [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
| [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. |
## Usage
```bash
# Full bootstrap from scratch:
bash k8s/wipe.sh
bash k8s/install-cp.sh
bash k8s/post-setup.sh
bash k8s/verify.sh
# Access the cluster:
export KUBECONFIG=~/.kube/config.pfv-k8s
kubectl get nodes
```
## Design Decisions
- **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified.
Talos architecture is documented in [`docs/k8s/`](../docs/k8s/) for when
that requirement comes online. k3s on stock Debian is simpler to operate.
- **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and
`--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node
status or certificate.
- **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of
Tailscale's WireGuard. Double-encrypted, but functional and reliable.
- **NoSchedule taint:** All 3 cnodes are tainted so no user workloads
schedule on the control plane. Only system components (CoreDNS,
metrics-server, flannel, kube-proxy) with built-in tolerations run here.
- **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure.