mrcharles
428e9ecb1d
Docs moved from docs/k8s -> k8s/docs and docs/proxmox -> proxmox/docs to sit
beside their code. Updated all cross-references (docmap, STATUS, README, AGENTS,
k8s README). Fixed SCRIPT_DIR path resolution in 3 perf scripts broken by the
reorg (deploy-check/and-fix/finish-host pointed at root/scripts instead of
perf/scripts). Registered the new top-level proxmox/ dir in all listings.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
56 lines
2.2 KiB
Markdown
56 lines
2.2 KiB
Markdown
# k8s/ — pfv-k8s Cluster Setup Scripts
|
|
|
|
Scripts to bootstrap and manage the k3s control plane on cnode1/2/3.
|
|
All cluster communication goes over Tailscale IPs — no LAN traffic.
|
|
|
|
## Current State
|
|
|
|
3-node HA control plane (k3s v1.36.2+k3s1, embedded etcd):
|
|
|
|
| Node | Tailscale IP | Role | Tainted |
|
|
|------|-------------|------|---------|
|
|
| pfv-k8s-cnode1 | 100.97.178.106 | control-plane, etcd | NoSchedule |
|
|
| pfv-k8s-cnode2 | 100.109.34.72 | control-plane, etcd | NoSchedule |
|
|
| pfv-k8s-cnode3 | 100.106.222.18 | control-plane, etcd | NoSchedule |
|
|
|
|
## Scripts
|
|
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| [`env.sh`](env.sh) | Shared config: node IPs, SSH opts, k3s version. Sourced by all scripts. |
|
|
| [`wipe.sh`](wipe.sh) | Remove existing k3s from all cnodes (clean slate). |
|
|
| [`install-cp.sh`](install-cp.sh) | Full bootstrap: cnode1 (--cluster-init) then cnode2/3 join. |
|
|
| [`join-servers.sh`](join-servers.sh) | Re-join cnode2/3 only (if cnode1 is already up). |
|
|
| [`post-setup.sh`](post-setup.sh) | Apply NoSchedule taints, fetch kubeconfig, verify. |
|
|
| [`verify.sh`](verify.sh) | Health check: nodes Ready, Tailscale IPs, taints, etcd, CoreDNS. |
|
|
| [`probe-nodes.sh`](probe-nodes.sh) | Verify SSH + Tailscale reachability. |
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Full bootstrap from scratch:
|
|
bash k8s/wipe.sh
|
|
bash k8s/install-cp.sh
|
|
bash k8s/post-setup.sh
|
|
bash k8s/verify.sh
|
|
|
|
# Access the cluster:
|
|
export KUBECONFIG=~/.kube/config.pfv-k8s
|
|
kubectl get nodes
|
|
```
|
|
|
|
## Design Decisions
|
|
|
|
- **k3s (not Talos):** This is a regular R&D cluster, not ITAR/classified.
|
|
Talos architecture is documented in [`docs/`](docs/) for when
|
|
that requirement comes online. k3s on stock Debian is simpler to operate.
|
|
- **Tailscale-only transport:** `--node-ip`, `--advertise-address`, and
|
|
`--tls-san` are all set to Tailscale IPs. No LAN IP appears in any node
|
|
status or certificate.
|
|
- **VXLAN flannel:** Pods communicate via flannel VXLAN overlay on top of
|
|
Tailscale's WireGuard. Double-encrypted, but functional and reliable.
|
|
- **NoSchedule taint:** All 3 cnodes are tainted so no user workloads
|
|
schedule on the control plane. Only system components (CoreDNS,
|
|
metrics-server, flannel, kube-proxy) with built-in tolerations run here.
|
|
- **Embedded etcd:** 3-node HA etcd quorum. Tolerates 1 node failure.
|