# Prod ACME endpoint runbook — step-ca on tsys-ca

> Ticket [#800](https://projects.knownelement.com/issues/800) · parent
> [#697](https://projects.knownelement.com/issues/697) · lane index
> https://community.turnsys.com/t/297

## Live endpoint (2026-09-05)

- Directory: `https://100.102.96.24:8443/acme/acme/directory` (tailnet only)
- Chain: ECDSA `KNEL Fleet ACME Intermediate` ← fleet RSA root
  (`/root/ca-root`, stays on the host; key never leaves)
- Container: `ukrrs-oam-ca-stepca`, host networking, listen :8443 (ca.json)
- State: `/opt/stepca-prod/` (data/ + secrets/password, root-owned)
- Repo files: `docker/stepca-prod/{compose.yaml,init-stepca.sh}`
- Re-init: `bash init-stepca.sh --reinit` (wipes data/, new intermediate)

## Client usage (HTTP-01)

    lego --server https://100.102.96.24:8443/acme/acme/directory \
         --email agent@knel.net --accept-tos \
         --domains <fqdn> --http run
    # trust: install fleet root from http://tsys-ca.knel.net/ca/dist/

## DEBUG LOG — http-01 validation (2026-09-05)

Root cause of the "could not connect to validation target" loop on the
workstation test target: **ultix-streaming (the workstation) is a joined
k8s worker node**. kube-proxy installs CNI portmap rules that DNAT **all
tcp/80** traffic to the node's IPs (incl. 127.0.0.1 and the tailnet IP)
into a cluster ingress pod — so lego's challenge listener never receives
the CA's fetch (it gets the ingress pod's Go-404 instead). strace proved
step-ca DID resolve and dial correctly; the fetch landed on the pod.

Consequences / rules for ACME http-01 across the fleet:

1. http-01 targets must have a FREE, un-intercepted port 80. Any host
   running a k3s worker + ingress (the workstation, future k8s nodes)
   CANNOT answer http-01 for its own names.
2. For such hosts use **DNS-01** (pending TSIG approval, #800 Q1) or a
   k8s-native cert solution (cert-manager) — do NOT fight the port map.
3. Plain fleet VMs (no k8s) are fine for http-01.

E2E proof plan: boot sectestbed VM (5104), run lego there with an IP
identifier (fresh VM, port 80 free), issue via the prod endpoint.

## E2E PROOF (2026-09-05) — http-01 issuance VERIFIED

sectestbed VM 5104 (dev PBS, tailnet 100.127.238.29, port 80 free):

    lego --server https://100.102.96.24:8443/acme/acme/directory \
         --email agent@knel.net --accept-tos \
         --domains 100.127.238.29 --http run
    → Served key authentication → The server validated our request
    → Server responded with a certificate
    Issuer: CN=KNEL Fleet ACME Intermediate (ECDSA, under fleet RSA root)
    SAN: IP:100.127.238.29

IP identifiers work. DNS identifiers work when the name resolves from
tsys-ca (rooted FQDN — add /etc/hosts entry on tsys-ca for test names).
Verify chain: openssl verify -CAfile root.crt -untrusted issuer.crt leaf.crt
VM shut back down after the test (lane rule).