mrcharles
d2541647f2
ci / audit (push) Failing after 54s
http-01 validation, issuance, and chain all verified against the fleet root via the prod endpoint; 5104 shut back down after the test. https://projects.knownelement.com/issues/800
62 lines
2.8 KiB
Markdown
62 lines
2.8 KiB
Markdown
# Prod ACME endpoint runbook — step-ca on tsys-ca
|
|
|
|
> Ticket [#800](https://projects.knownelement.com/issues/800) · parent
|
|
> [#697](https://projects.knownelement.com/issues/697) · lane index
|
|
> https://community.turnsys.com/t/297
|
|
|
|
## Live endpoint (2026-09-05)
|
|
|
|
- Directory: `https://100.102.96.24:8443/acme/acme/directory` (tailnet only)
|
|
- Chain: ECDSA `KNEL Fleet ACME Intermediate` ← fleet RSA root
|
|
(`/root/ca-root`, stays on the host; key never leaves)
|
|
- Container: `ukrrs-oam-ca-stepca`, host networking, listen :8443 (ca.json)
|
|
- State: `/opt/stepca-prod/` (data/ + secrets/password, root-owned)
|
|
- Repo files: `docker/stepca-prod/{compose.yaml,init-stepca.sh}`
|
|
- Re-init: `bash init-stepca.sh --reinit` (wipes data/, new intermediate)
|
|
|
|
## Client usage (HTTP-01)
|
|
|
|
lego --server https://100.102.96.24:8443/acme/acme/directory \
|
|
--email agent@knel.net --accept-tos \
|
|
--domains <fqdn> --http run
|
|
# trust: install fleet root from http://tsys-ca.knel.net/ca/dist/
|
|
|
|
## DEBUG LOG — http-01 validation (2026-09-05)
|
|
|
|
Root cause of the "could not connect to validation target" loop on the
|
|
workstation test target: **ultix-streaming (the workstation) is a joined
|
|
k8s worker node**. kube-proxy installs CNI portmap rules that DNAT **all
|
|
tcp/80** traffic to the node's IPs (incl. 127.0.0.1 and the tailnet IP)
|
|
into a cluster ingress pod — so lego's challenge listener never receives
|
|
the CA's fetch (it gets the ingress pod's Go-404 instead). strace proved
|
|
step-ca DID resolve and dial correctly; the fetch landed on the pod.
|
|
|
|
Consequences / rules for ACME http-01 across the fleet:
|
|
|
|
1. http-01 targets must have a FREE, un-intercepted port 80. Any host
|
|
running a k3s worker + ingress (the workstation, future k8s nodes)
|
|
CANNOT answer http-01 for its own names.
|
|
2. For such hosts use **DNS-01** (pending TSIG approval, #800 Q1) or a
|
|
k8s-native cert solution (cert-manager) — do NOT fight the port map.
|
|
3. Plain fleet VMs (no k8s) are fine for http-01.
|
|
|
|
E2E proof plan: boot sectestbed VM (5104), run lego there with an IP
|
|
identifier (fresh VM, port 80 free), issue via the prod endpoint.
|
|
|
|
## E2E PROOF (2026-09-05) — http-01 issuance VERIFIED
|
|
|
|
sectestbed VM 5104 (dev PBS, tailnet 100.127.238.29, port 80 free):
|
|
|
|
lego --server https://100.102.96.24:8443/acme/acme/directory \
|
|
--email agent@knel.net --accept-tos \
|
|
--domains 100.127.238.29 --http run
|
|
→ Served key authentication → The server validated our request
|
|
→ Server responded with a certificate
|
|
Issuer: CN=KNEL Fleet ACME Intermediate (ECDSA, under fleet RSA root)
|
|
SAN: IP:100.127.238.29
|
|
|
|
IP identifiers work. DNS identifiers work when the name resolves from
|
|
tsys-ca (rooted FQDN — add /etc/hosts entry on tsys-ca for test names).
|
|
Verify chain: openssl verify -CAfile root.crt -untrusted issuer.crt leaf.crt
|
|
VM shut back down after the test (lane rule).
|