mrcharles
bad40feae6
- env.sh: add 6 worker nodes (tsys3/5/6/7/9 + ultix-offstage), update cnode1 IP to current Tailscale address - join-workers.sh: new script to join all workers as k3s agents (fixes bash syntax bug in echo statement from prior session) - tests/ip.sh: pass-through wrapper for ip command [#367] [#368]
k8s/ — pfv-k8s Cluster Setup Scripts
Redmine: #362 (initial build, closed) · #367 (rebuild, open) · #368 (worker join, open)
Scripts to bootstrap and manage the k3s control plane on cnode1/2/3. All cluster communication goes over Tailscale IPs — no LAN traffic.
Current State
3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
| Node | Tailscale IP | Role | Tainted |
|---|---|---|---|
| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
Scripts
| Script | Purpose |
|---|---|
env.sh |
Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
wipe.sh |
Remove existing k3s from all cnodes (clean slate). |
install-cp.sh |
Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
join-servers.sh |
Re-join cnode2/3 only (if cnode1 is already up). |
post-setup.sh |
Apply NoSchedule taints, fetch kubeconfig, verify. |
verify.sh |
Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
probe-nodes.sh |
Verify SSH + Tailscale reachability. |
Usage
# Full bootstrap from scratch:
bash k8s/wipe.sh
bash k8s/install-cp.sh
bash k8s/post-setup.sh
bash k8s/verify.sh
# Access the cluster:
export KUBECONFIG=~/.kube/config.pfv-k8s
kubectl get nodes
Design Decisions
- k3s (not Talos): This is a regular R&D cluster, not ITAR/classified.
Talos architecture is documented in
docs/for when that requirement comes online. k3s on stock Debian is simpler to operate. - Tailscale-only transport:
--node-ip,--advertise-address, and--tls-sanare all set to Tailscale IPs. No LAN IP appears in any node status or certificate. - VXLAN flannel: Pods communicate via flannel VXLAN overlay on top of Tailscale's WireGuard. Double-encrypted, but functional and reliable.
- NoSchedule taint: All 3 cnodes are tainted so no user workloads schedule on the control plane. Only system components (CoreDNS, metrics-server, flannel, kube-proxy) with built-in tolerations run here.
- Embedded etcd: 3-node HA etcd quorum. Tolerates 1 node failure.