KNELCa/docs/runbook-acme-prod.md

Prod ACME endpoint runbook — step-ca on tsys-ca

Ticket #800 · parent #697 · lane index https://community.turnsys.com/t/297

Live endpoint (2026-09-05)

• Directory: https://100.102.96.24:8443/acme/acme/directory (tailnet only)
• Chain: ECDSA KNEL Fleet ACME Intermediate ← fleet RSA root (/root/ca-root, stays on the host; key never leaves)
• Container: ukrrs-oam-ca-stepca, host networking, listen :8443 (ca.json)
• State: /opt/stepca-prod/ (data/ + secrets/password, root-owned)
• Repo files: docker/stepca-prod/{compose.yaml,init-stepca.sh}
• Re-init: bash init-stepca.sh --reinit (wipes data/, new intermediate)

Client usage (HTTP-01)

lego --server https://100.102.96.24:8443/acme/acme/directory \
     --email agent@knel.net --accept-tos \
     --domains <fqdn> --http run
# trust: install fleet root from http://tsys-ca.knel.net/ca/dist/

DEBUG LOG — http-01 validation (2026-09-05)

Root cause of the "could not connect to validation target" loop on the workstation test target: ultix-streaming (the workstation) is a joined k8s worker node. kube-proxy installs CNI portmap rules that DNAT all tcp/80 traffic to the node's IPs (incl. 127.0.0.1 and the tailnet IP) into a cluster ingress pod — so lego's challenge listener never receives the CA's fetch (it gets the ingress pod's Go-404 instead). strace proved step-ca DID resolve and dial correctly; the fetch landed on the pod.

Consequences / rules for ACME http-01 across the fleet:

1. http-01 targets must have a FREE, un-intercepted port 80. Any host running a k3s worker + ingress (the workstation, future k8s nodes) CANNOT answer http-01 for its own names.
2. For such hosts use DNS-01 (pending TSIG approval, #800 Q1) or a k8s-native cert solution (cert-manager) — do NOT fight the port map.
3. Plain fleet VMs (no k8s) are fine for http-01.

E2E proof plan: boot sectestbed VM (5104), run lego there with an IP identifier (fresh VM, port 80 free), issue via the prod endpoint.