docs: gardening after APISIX rewrite + production deploy

STATUS/README inventory updated (APISIX deployed at
apigw.knownelement.com); JOURNAL section 20 (rewrite + deploy).

Umbrella: https://projects.knownelement.com/issues/632

JOURNAL.md

@@ -1526,3 +1526,77 @@ CMD ["/app/start.sh"]
 
 ---
 
+
+## 20. APISIX: Package Rewrite + Production Deploy (2026-09-07)
+
+**Trigger**: founder directive — "APISIX needs to be on Cloudron at
+apigw.knownelement.com". The existing package (#2, 2026-09-01) turned out
+to be unbuildable as authored: it referenced a Cloudron **etcd addon that
+does not exist** (verified against the platform source on the host — no
+etcd anywhere in box/src), used unpinned `apache/apisix:latest`, had a
+broken config.yaml (quoted heredoc blocked env substitution, invalid
+deployment schema), and pointed the healthcheck at the auth-protected
+admin API (would always 401).
+
+**Final design (v1.0.2)** — official-image wrapper, rewritten:
+
+- `apache/apisix:3.18.0-debian` pinned by tag AND digest
+- **Embedded single-node etcd** (quay.io/coreos/etcd v3.5.33, digest-pinned,
+  binaries COPYed into the image) — state under `/app/data/etcd`; keeps the
+  Admin API + dynamic routes with zero platform addon dependencies
+- Admin API bound to **127.0.0.1:9180 only**, protected by a persisted
+  random 64-hex key in `/app/data/.admin_key`; operators use `cloudron exec`
+- `healthCheckPath "/"` — Cloudron treats 2xx/3xx/4xx as alive (verified in
+  box/src/apphealthmonitor.js); `/healthz` seeded into etcd once as a
+  serverless-pre-function direct response for external monitoring
+- Manifest: semver `1.0.2`, `localstorage: {}` (object form — boolean form
+  fails current schema validation), `type` field removed (rejected),
+  1 GiB memory limit, NO tcpPorts (all traffic via the platform proxy)
+
+**Lessons (all verified the hard way):**
+
+1. **Cloudron 8 app rootfs is READ-ONLY** (writable: /app/data, /tmp, /run).
+   APISIX must live its runtime life on /app/data: APISIX_PREFIX +
+   a path-patched copy of the CLI tree — `apisix_home` is HARDCODED
+   `/usr/local/apisix` in apisix/cli/apisix.lua, so the whole apisix/ +
+   deps/ trees (~62 MB) are copied to /app/data/apisix and sed-patched on
+   every start.
+2. **Cloudron does not chown /app/data to arbitrary image users** — the
+   apisix (uid 636) image user got EACCES. Apps run as root (platform
+   convention); then nginx's workers run as **nobody** (no `user`
+   directive in the generated nginx.conf), which forced config.yaml to 644.
+3. **Pre-existing DNS records block app install** ("DNS A record already
+   exists"): apigw pointed at a stale 2024 external host
+   (ruby.ontrixsolutions.com). `POST /api/v1/apps/:id/repair` re-runs the
+   task with `overwriteDns: true` (apps.js:2550) and upserts the record
+   through the platform's own DNS provider creds — the clean fix, no
+   provider API access needed.
+4. **Custom app install API**: `POST /api/v1/apps` with `appStoreId: ""`,
+   a full manifest, and `manifest.dockerImage` for the image; updates via
+   `POST /api/v1/apps/:id/update` with `{manifest, skipBackup, force}`.
+   Server source on the host (/home/yellowtent/box/src) is the
+   authoritative API spec — docs pages 404/JS-render.
+5. **grind-stack.sh fidelity upgrades** (kept for every future package):
+   - /app/data chowned to the image's uid (Cloudron localstorage contract)
+   - `--read-only --tmpfs /tmp --tmpfs /run` on the app container (would
+     have caught the rootfs issue BEFORE production)
+   - `sbom` subcommand wired into the dispatch table (it existed but was
+     unreachable)
+   - probe containers need `--entrypoint` when the image wraps everything
+     in start.sh
+
+**Delivery path**: built under ukrrs-batch.slice → pushed to the platform's
+own **cloudron-docker-registry** app (Basic auth through the Cloudron proxy
+validates platform user creds). Created svc account `svpengops-registry`
+added to the registry app's accessRestriction; the platform's docker
+registry entry (previously mrcharles, whose stored password 403'd) was
+updated to the svc account. Password vaulting pending (sm session expired).
+
+**Verification trail**: grind-stack green (healthz 200, admin 401/200 with
+key, restart persistence in etcd, all under --read-only) → push digest
+sha256:754731d8… → live update task 15710 → `https://apigw.knownelement.com/healthz`
+= 200 from production; container stable; etcd healthy via exec.
+
+**Status**: DEPLOYED — awaiting founder UAT. Redmine note deferred (vault
+session expired → mred key unreadable); dedicated ticket to be filed under
+the cloudron project on next vault login (gate ran under umbrella #632).

README.md

@@ -23,7 +23,7 @@ The Cloudron component focuses on packaging upstream free/libre/open application
 | # | Application | Category | Size | Ports | Addons | Status |
 |---|------------|----------|-------|--------|---------|--------|
 | 1 | Webhook | API-Gateway | 775MB | 9000 | localstorage | ✅ Committed |
-| 2 | APISIX | API-Gateway | 143MB | 9080, 9180, 9443 | localstorage, etcd | ✅ Committed |
+| 2 | APISIX | API-Gateway | ~700MB | 9080 | localstorage (embedded etcd) | ✅ **DEPLOYED** apigw.knownelement.com |
 | 3 | Healthchecks | Monitoring | 105MB | 8000 | localstorage, postgresql | ✅ Committed |
 | 4 | Review Board | Development | 1.29GB | 8080 | localstorage, postgresql | ✅ Committed |
 | 5 | WireViz Web | Documentation-Tools | 378MB | 3005 | localstorage | ✅ Committed |
@@ -120,7 +120,7 @@ Applications are organized by function rather than programming language:
 | [FX](https://github.com/metrue/fx) | [GitHub](https://github.com/metrue/fx) | Function as a Service platform | DevOps-Tools | ✅ Packaged |
 | [PayrollEngine](https://github.com/Payroll-Engine/PayrollEngine) | [GitHub](https://github.com/Payroll-Engine/PayrollEngine) | Payroll processing engine | Business-Apps |
 | [Nautilus Trader](https://github.com/nautechsystems/nautilus_trader) | [GitHub](https://github.com/nautechsystems/nautilus_trader) | Algorithmic trading platform | Financial-Trading |
-| [APISIX](https://github.com/apache/apisix) | [GitHub](https://github.com/apache/apisix) | Cloud-native API gateway | API-Gateway | ✅ Packaged |
+| [APISIX](https://github.com/apache/apisix) | [GitHub](https://github.com/apache/apisix) | Cloud-native API gateway | API-Gateway | ✅ **DEPLOYED** ([apigw.knownelement.com](https://apigw.knownelement.com)) |
 | [Grist Core](https://github.com/gristlabs/grist-core) | [GitHub](https://github.com/gristlabs/grist-core) | Modern data spreadsheet and database | Collaboration | ⏭ Excluded — packaged upstream (Cloudron App Store) |
 | [Healthchecks](https://github.com/healthchecks/healthchecks) | [GitHub](https://github.com/healthchecks/healthchecks) | Cron job monitoring service | Monitoring | ✅ Packaged |
 | [EmonCMS](https://github.com/emoncms/emoncms) | [GitHub](https://github.com/emoncms/emoncms) | OpenEnergyMonitor energy monitoring platform | Monitoring | ❓ local-only auth — founder ruling (#907) |

STATUS.md

@@ -3,11 +3,13 @@
 > **Human read-only. Agents maintain this file automatically after each work
 > session.** Do not edit by hand — the next agent run will overwrite it.
 >
-> **Last updated:** 2026-09-06 by VpEngOps agent (ZCode/GLM-5.3) — rathole
-> REMOVED by founder ruling ("we will use netbird for all enterprise network
-> access"): package dir deleted, GitUrlList 46→36, completed count 19→18.
-> Landed tickets sit in **Feedback** (awaiting live-Cloudron UAT); builds ON
-> HOLD pending the founder's queue review. Standing policies: no-Node,
+> **Last updated:** 2026-09-07 by VpEngOps agent (ZCode/GLM-5.3) — **APISIX
+> REWRITTEN + DEPLOYED to production Cloudron at apigw.knownelement.com**
+> (package 1.0.2: the 09-01 version used a nonexistent "etcd addon" and
+> :latest; now digest-pinned 3.18.0 with embedded etcd, verified against
+> Cloudron's read-only rootfs contract). Landed tickets sit in **Feedback**
+> (awaiting live-Cloudron UAT); other builds remain ON HOLD pending the
+> founder's queue review. Standing policies: no-Node,
 > no-open-core, no-SSO-tax, netbird for network access, RustFS makes S3 a
 > non-blocker, ClickHouse + ActiveMQ on infra (CH/MQ reqs = not blockers;
 > verify protocol compat per app). Killed tickets Closed at founder direction.
@@ -35,7 +37,7 @@ ready for the sequential grind-driver pattern.
 | # | Application | Category | Pattern | Port(s) | Addons |
 |---|-------------|----------|---------|---------|--------|
 | 1 | Webhook | API-Gateway | Multi-stage (Go) | 9000 | localstorage |
-| 2 | APISIX | API-Gateway | Official-image wrapper | 9080, 9180, 9443 | localstorage, etcd |
+| 2 | APISIX | API-Gateway | Official-image wrapper (v1.0.2) | 9080 | localstorage (embedded etcd) — **DEPLOYED apigw.knownelement.com** |
 | 3 | Healthchecks | Monitoring | Django + PostgreSQL | 8000 | localstorage, postgresql |
 | 4 | Review Board | Development | Django + PostgreSQL | 8080 | localstorage, postgresql |
 | 5 | WireViz Web | Documentation-Tools | Python build | 3005 | localstorage |