docs: gardening after APISIX rewrite + production deploy

STATUS/README inventory updated (APISIX deployed at
apigw.knownelement.com); JOURNAL section 20 (rewrite + deploy).

Umbrella: https://projects.knownelement.com/issues/632

JOURNAL.md

@@ -1526,3 +1526,77 @@ CMD ["/app/start.sh"]
 
 ---
 
+
+## 20. APISIX: Package Rewrite + Production Deploy (2026-09-07)
+
+**Trigger**: founder directive — "APISIX needs to be on Cloudron at
+apigw.knownelement.com". The existing package (#2, 2026-09-01) turned out
+to be unbuildable as authored: it referenced a Cloudron **etcd addon that
+does not exist** (verified against the platform source on the host — no
+etcd anywhere in box/src), used unpinned `apache/apisix:latest`, had a
+broken config.yaml (quoted heredoc blocked env substitution, invalid
+deployment schema), and pointed the healthcheck at the auth-protected
+admin API (would always 401).
+
+**Final design (v1.0.2)** — official-image wrapper, rewritten:
+
+- `apache/apisix:3.18.0-debian` pinned by tag AND digest
+- **Embedded single-node etcd** (quay.io/coreos/etcd v3.5.33, digest-pinned,
+  binaries COPYed into the image) — state under `/app/data/etcd`; keeps the
+  Admin API + dynamic routes with zero platform addon dependencies
+- Admin API bound to **127.0.0.1:9180 only**, protected by a persisted
+  random 64-hex key in `/app/data/.admin_key`; operators use `cloudron exec`
+- `healthCheckPath "/"` — Cloudron treats 2xx/3xx/4xx as alive (verified in
+  box/src/apphealthmonitor.js); `/healthz` seeded into etcd once as a
+  serverless-pre-function direct response for external monitoring
+- Manifest: semver `1.0.2`, `localstorage: {}` (object form — boolean form
+  fails current schema validation), `type` field removed (rejected),
+  1 GiB memory limit, NO tcpPorts (all traffic via the platform proxy)
+
+**Lessons (all verified the hard way):**
+
+1. **Cloudron 8 app rootfs is READ-ONLY** (writable: /app/data, /tmp, /run).
+   APISIX must live its runtime life on /app/data: APISIX_PREFIX +
+   a path-patched copy of the CLI tree — `apisix_home` is HARDCODED
+   `/usr/local/apisix` in apisix/cli/apisix.lua, so the whole apisix/ +
+   deps/ trees (~62 MB) are copied to /app/data/apisix and sed-patched on
+   every start.
+2. **Cloudron does not chown /app/data to arbitrary image users** — the
+   apisix (uid 636) image user got EACCES. Apps run as root (platform
+   convention); then nginx's workers run as **nobody** (no `user`
+   directive in the generated nginx.conf), which forced config.yaml to 644.
+3. **Pre-existing DNS records block app install** ("DNS A record already
+   exists"): apigw pointed at a stale 2024 external host
+   (ruby.ontrixsolutions.com). `POST /api/v1/apps/:id/repair` re-runs the
+   task with `overwriteDns: true` (apps.js:2550) and upserts the record
+   through the platform's own DNS provider creds — the clean fix, no
+   provider API access needed.
+4. **Custom app install API**: `POST /api/v1/apps` with `appStoreId: ""`,
+   a full manifest, and `manifest.dockerImage` for the image; updates via
+   `POST /api/v1/apps/:id/update` with `{manifest, skipBackup, force}`.
+   Server source on the host (/home/yellowtent/box/src) is the
+   authoritative API spec — docs pages 404/JS-render.
+5. **grind-stack.sh fidelity upgrades** (kept for every future package):
+   - /app/data chowned to the image's uid (Cloudron localstorage contract)
+   - `--read-only --tmpfs /tmp --tmpfs /run` on the app container (would
+     have caught the rootfs issue BEFORE production)
+   - `sbom` subcommand wired into the dispatch table (it existed but was
+     unreachable)
+   - probe containers need `--entrypoint` when the image wraps everything
+     in start.sh
+
+**Delivery path**: built under ukrrs-batch.slice → pushed to the platform's
+own **cloudron-docker-registry** app (Basic auth through the Cloudron proxy
+validates platform user creds). Created svc account `svpengops-registry`
+added to the registry app's accessRestriction; the platform's docker
+registry entry (previously mrcharles, whose stored password 403'd) was
+updated to the svc account. Password vaulting pending (sm session expired).
+
+**Verification trail**: grind-stack green (healthz 200, admin 401/200 with
+key, restart persistence in etcd, all under --read-only) → push digest
+sha256:754731d8… → live update task 15710 → `https://apigw.knownelement.com/healthz`
+= 200 from production; container stable; etcd healthy via exec.
+
+**Status**: DEPLOYED — awaiting founder UAT. Redmine note deferred (vault
+session expired → mred key unreadable); dedicated ticket to be filed under
+the cloudron project on next vault login (gate ran under umbrella #632).