feat(k8s): deploy 3-node k3s HA control plane over Tailscale
Bootstrap a regular (non-ITAR) k3s cluster on cnode1/2/3 with embedded
etcd. All cluster communication — node registration, API server, etcd
peering, flannel VXLAN — runs exclusively over Tailscale IPs. Zero LAN
addresses appear in node status or TLS certificates.
Scripts (k8s/):
- env.sh: shared config (Tailscale IPs, SSH opts, k3s version)
- wipe.sh: remove existing k3s from all cnodes
- install-cp.sh: full bootstrap (cnode1 --cluster-init, then cnode2/3 join)
- join-servers.sh: re-join cnode2/3 only (fixes broken join state)
- post-setup.sh: apply NoSchedule taints, fetch kubeconfig, verify
- verify.sh: 13-point health check (nodes, Tailscale IPs, taints, etcd,
CoreDNS, API server, workload isolation)
- probe-nodes.sh: SSH + Tailscale reachability check
All 3 cnodes are tainted control-plane:NoSchedule so no user workloads
can schedule on the control plane. 13/13 health checks pass.
Docs updated: k8s README TL;DR reflects k3s (not Talos) as the deployed
choice, with Talos preserved for the future ITAR cluster.
💘 Generated with Crush
Assisted-by: Crush:glm-5.2
This commit is contained in:
@@ -3,12 +3,12 @@
|
||||
> **Human read-only. Agents maintain this file automatically after each work
|
||||
> session.** Do not edit by hand — the next agent run will overwrite it.
|
||||
|
||||
> **Last updated:** 2026-07-28 by Crush (GLM-5)
|
||||
> **Last updated:** 2026-07-28 by Crush (GLM-5.2)
|
||||
|
||||
## Current State: STABLE
|
||||
|
||||
The merged PFVCluster repo is fully operational across provisioning, DNS
|
||||
infrastructure, and Proxmox cluster ops.
|
||||
infrastructure, Proxmox cluster ops, and k8s control plane.
|
||||
|
||||
## Completed Work
|
||||
|
||||
@@ -35,15 +35,17 @@ infrastructure, and Proxmox cluster ops.
|
||||
- [ ] tsys5: blocked on 2nd ethernet cable + NVMe install
|
||||
- [ ] tsys2: pending rebuild from Win10 to Proxmox
|
||||
|
||||
### Kubernetes Architecture (docs authored, pending implementation)
|
||||
- [x] `docs/k8s/` created: README, DISTRO-DECISION, ARCHITECTURE
|
||||
- [x] Distro decision: **Talos Linux** (over k3s), driven by ITAR/classified
|
||||
requirement. Immutable + API-only + measured-boot = structurally stronger
|
||||
compliance posture than SCAP-hardened Debian.
|
||||
- [x] Target arch documented: 3 cnodes (local-lvm boot), Cilium CNI, vcluster
|
||||
per tenant, Harbor pull-through cache on D3 SSD, OIDC→Keycloak (Cloudron),
|
||||
Tailscale subnet-router for admin (cluster nodes LAN-only, zero egress).
|
||||
- [ ] 3 decisions pending user confirm (see Pending below)
|
||||
### Kubernetes Control Plane (k3s HA — LIVE)
|
||||
- [x] 3-node k3s HA control plane deployed: cnode1/2/3 (v1.36.2+k3s1, embedded etcd)
|
||||
- [x] **All traffic over Tailscale IPs** — no LAN IPs in node status or certs
|
||||
- [x] All 3 cnodes tainted `control-plane:NoSchedule` (zero user workloads)
|
||||
- [x] 13/13 health checks pass (verify.sh): nodes Ready, etcd quorum,
|
||||
Tailscale IPs, CoreDNS, API server, workload isolation
|
||||
- [x] Scripts in `k8s/`: wipe, install-cp, join-servers, post-setup, verify
|
||||
- [x] Kubeconfig saved to `~/.kube/config.pfv-k8s` (gitignored, embedded certs)
|
||||
- [ ] Workers (wnodes) not yet joined to this cluster
|
||||
- [ ] Distro decision: **k3s chosen for regular R&D cluster**. Talos docs
|
||||
preserved in `docs/k8s/` for future ITAR/classified cluster.
|
||||
|
||||
### Repo Merge
|
||||
- [x] KNELServerBuild merged into PFVCluster (history preserved)
|
||||
@@ -60,15 +62,16 @@ infrastructure, and Proxmox cluster ops.
|
||||
|
||||
## Pending (next session priorities)
|
||||
|
||||
1. **k8s: confirm 3 decisions** before bootstrap:
|
||||
1. **k8s workers:** Join wnodes to the k3s cluster (agents, not servers)
|
||||
2. **k8s: confirm 3 decisions** for future Talos/ITAR cluster:
|
||||
- Cnode count: 3 (recommended) vs 5
|
||||
- Cnode host spread: tsys9 + tsys1 + tsys3 (recommended, 3-host, local-lvm)
|
||||
- Tailscale pattern: subnet router on `tailscale-router` (recommended)
|
||||
2. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
3. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
4. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
5. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
6. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
||||
3. Provisioning: add `tailscale up --accept-dns=false` after resolv.conf deploy
|
||||
4. DNS: verify Pi-hole forwards knel.net to local Technitium on both nodes
|
||||
5. Perf: complete tsys4/5 tuning after hardware install (Friday)
|
||||
6. tsys2: rebuild from Win10 to Proxmox (k8s-dedicated host)
|
||||
7. **k8s deferred topics** (tracked for future sessions): ETL tooling
|
||||
(GDAL/PostGIS/xarray/Dask → RWX vs RWO), HPC scheduler (Argo/Volcano),
|
||||
per-tenant vcluster policy templates, solar-aware scale-out
|
||||
|
||||
@@ -83,4 +86,5 @@ infrastructure, and Proxmox cluster ops.
|
||||
| NTP | pfv-netinfra-01/02 (redundant, LAN IPs, stratum 2/3) |
|
||||
| Sandbox VM | sectestbed-sandbox (VMID 6000 on pfv-tsys5) |
|
||||
| Backup | Proxmox Backup Server (PBS) |
|
||||
| **k8s control plane** | **3-node k3s HA (cnode1/2/3), all traffic over Tailscale** |
|
||||
| Production | Cloudron VPS, Reston VA (this cluster is R&D only) |
|
||||
|
||||
Reference in New Issue
Block a user