From d923b5f34b61da2d792157a926dc1a5e0b8e0be8 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Sat, 5 Sep 2026 10:06:35 -0500 Subject: [PATCH] [#800] ACME prod runbook: endpoint usage + http-01 root-cause debug log https://projects.knownelement.com/issues/800 --- docs/runbook-acme-prod.md | 44 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/runbook-acme-prod.md diff --git a/docs/runbook-acme-prod.md b/docs/runbook-acme-prod.md new file mode 100644 index 0000000..a24b4b9 --- /dev/null +++ b/docs/runbook-acme-prod.md @@ -0,0 +1,44 @@ +# Prod ACME endpoint runbook — step-ca on tsys-ca + +> Ticket [#800](https://projects.knownelement.com/issues/800) · parent +> [#697](https://projects.knownelement.com/issues/697) · lane index +> https://community.turnsys.com/t/297 + +## Live endpoint (2026-09-05) + +- Directory: `https://100.102.96.24:8443/acme/acme/directory` (tailnet only) +- Chain: ECDSA `KNEL Fleet ACME Intermediate` ← fleet RSA root + (`/root/ca-root`, stays on the host; key never leaves) +- Container: `ukrrs-oam-ca-stepca`, host networking, listen :8443 (ca.json) +- State: `/opt/stepca-prod/` (data/ + secrets/password, root-owned) +- Repo files: `docker/stepca-prod/{compose.yaml,init-stepca.sh}` +- Re-init: `bash init-stepca.sh --reinit` (wipes data/, new intermediate) + +## Client usage (HTTP-01) + + lego --server https://100.102.96.24:8443/acme/acme/directory \ + --email agent@knel.net --accept-tos \ + --domains --http run + # trust: install fleet root from http://tsys-ca.knel.net/ca/dist/ + +## DEBUG LOG — http-01 validation (2026-09-05) + +Root cause of the "could not connect to validation target" loop on the +workstation test target: **ultix-streaming (the workstation) is a joined +k8s worker node**. kube-proxy installs CNI portmap rules that DNAT **all +tcp/80** traffic to the node's IPs (incl. 127.0.0.1 and the tailnet IP) +into a cluster ingress pod — so lego's challenge listener never receives +the CA's fetch (it gets the ingress pod's Go-404 instead). strace proved +step-ca DID resolve and dial correctly; the fetch landed on the pod. + +Consequences / rules for ACME http-01 across the fleet: + +1. http-01 targets must have a FREE, un-intercepted port 80. Any host + running a k3s worker + ingress (the workstation, future k8s nodes) + CANNOT answer http-01 for its own names. +2. For such hosts use **DNS-01** (pending TSIG approval, #800 Q1) or a + k8s-native cert solution (cert-manager) — do NOT fight the port map. +3. Plain fleet VMs (no k8s) are fine for http-01. + +E2E proof plan: boot sectestbed VM (5104), run lego there with an IP +identifier (fresh VM, port 80 free), issue via the prod endpoint.