mrcharles
6c58671d15
Each component README now links to its corresponding Redmine tickets (closed for completed work, open for pending items) for bidirectional traceability between code and system of record.
58 lines
2.5 KiB
Markdown
58 lines
2.5 KiB
Markdown
# k8s/ — pfv-k8s Cluster Setup Scripts
|
|
|
|
> **Redmine:** [#362](https://projects.knownelement.com/issues/362) (initial build, closed) · [#367](https://projects.knownelement.com/issues/367) (rebuild, open) · [#368](https://projects.knownelement.com/issues/368) (worker join, open)
|
|
|
|
Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
|
|
All cluster communication goes over Tailscale IPs — no LAN traffic.
|
|
|
|
## Current State
|
|
|
|
3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
|
|
|
|
| Node | Tailscale IP | Role | Tainted |
|
|
|------|-------------|------|---------|
|
|
| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
|
|
| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
|
|
| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
|
|
|
|
## Scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
|
|
| [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). |
|
|
| [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
|
|
| [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). |
|
|
| [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. |
|
|
| [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
|
|
| [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Full bootstrap from scratch:
|
|
bash k8s/wipe.sh
|
|
bash k8s/install-cp.sh
|
|
bash k8s/post-setup.sh
|
|
bash k8s/verify.sh
|
|
|
|
# Access the cluster:
|
|
export KUBECONFIG=~/.kube/config.pfv-k8s
|
|
kubectl get nodes
|
|
```
|
|
|
|
## Design Decisions
|
|
|
|
- **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified.
|
|
Talos architecture is documented in [`docs/`](docs/) for when
|
|
that requirement comes online. k3s on stock Debian is simpler to operate.
|
|
- **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and
|
|
`--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node
|
|
status or certificate.
|
|
- **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of
|
|
Tailscale's WireGuard. Double-encrypted, but functional and reliable.
|
|
- **NoSchedule taint:** All 3 cnodes are tainted so no user workloads
|
|
schedule on the control plane. Only system components (CoreDNS,
|
|
metrics-server, flannel, kube-proxy) with built-in tolerations run here.
|
|
- **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure.
|