feat: apply memory shrink + daemon.json — ground truth update [#731]

11 apps reslidered via API (53.4->29.0GB explicit limits); daemon.json
live-restore + nofile defaults applied (log keys banned: unit flag
conflict — see #731 incident note). perf/apply-memory-shrink.sh codifies
the change. All 159 apps verified running.
Meat: https://projects.knownelement.com/issues/731#note-3929

README.md

@@ -11,7 +11,7 @@ Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).
 
 | Path | What |
 |------|------|
-| `perf/` | [#685](https://projects.knownelement.com/issues/685) perf work: memory census, pg role-cap SQL, applied sysctl, planned docker daemon.json |
+| `perf/` | [#685](https://projects.knownelement.com/issues/685) perf work: memory census, pg role-cap SQL, applied sysctl, applied `daemon.json`, `apply-memory-shrink.sh` |
 | `monitoring/` | Uptime Kuma Cloudron app coverage — 55-app inventory + sync checker (moved from PFVCluster, #435/#685/#727) |
 | `ansible/` | Reserved — AWX/Ansible IaC convergence ([#454](https://projects.knownelement.com/issues/454)); playbooks land here as manual changes are codified |
 | `scripts/` | Rule engine + git hooks (enforcement layer, ADOPTING pattern) |
@@ -25,16 +25,22 @@ Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).
   idle-in-transaction timeout 15min. Victim `too many clients` FATALs → zero.
 - Host sysctl: `vm.swappiness=10` (`perf/sysctl/`). THP already `never`, virtio
   scheduler `none` — optimal, left alone.
-- **`/etc/docker/daemon.json` is still ABSENT on the host** (`live-restore=false`,
+- **`/etc/docker/daemon.json` APPLIED 2026-09-02** (#731): `live-restore: true`
-  containers default to nofile=1024 unless bridged). The postgres fd bridge
+  (future dockerd restarts are zero-downtime) + `default-ulimits nofile
-  (prlimit 262144) is EPHEMERAL — lost on any postgres container recreate.
+  262144` (every container start gets real fd limits — the postgres prlimit
-  Applying `perf/docker-daemon.json.planned` needs ONE planned dockerd bounce
+  bridge is no longer needed; `perf/docker-daemon.json` is the SoR copy).
-  (restarts all ~300 containers) in a human-chosen window.
+  NOTE: this host's dockerd runs `--log-driver=journald` as a unit flag —
-- Memory sliders: shrink table on t/316 (~28GB headroom). Do NOT shrink
+  daemon.json must NOT set log-driver/log-opts or dockerd refuses to start
-  `bi.knownelement.com` or platform services (Cloudron auto-sizes those).
+  (bit us once, see #731). Container log rotation is journald's job here.
+- **Memory shrink table APPLIED 2026-09-02** (#731) via the panel API
+  (`perf/apply-memory-shrink.sh`): 11 apps resliders, explicit limits
+  53.4 GB → 29.0 GB (~24 GB of the ~28 GB headroom reclaimed). Skips:
+  photos (manifest floor 3.5 GB), `bi.knownelement.com` + platform services
+  (do NOT shrink). All 159 apps verified running post-change.
 
 ## Access
 
-SSH `root@my.knownelement.com` is the only access path — routed through the
+SSH `root@my.knownelement.com` — routed through the KNEL/PFVCluster
-KNEL/PFVCluster `tests/remote.sh` chokepoint (see AGENTS.md). Cloudron API
+`tests/remote.sh` chokepoint (see AGENTS.md). Panel API token issued
-token pending; once issued, refs live in `~/.creds/cloudron.env`, never here.
+2026-09-02 (`~/.creds/cloudron.env`, refs only in this repo) — prefer the
+API over SSH for reads/limits; base image rebuilds (#710) still need SSH.

STATUS.md

@@ -7,8 +7,9 @@ Last updated: 2026-09-02
 
 | ticket | state | item |
 |--------|-------|------|
-| #727   | 🔄    | repo bootstrap + perf script capture |
+| #727   | ✅    | repo bootstrap + perf script capture + API token received (suggest close) |
-| #685   | ⏳    | founder soak — evening storm window is the real test; caps stay until tymeslot ships a POOL_SIZE env |
+| #731   | ✅    | memory shrink applied (53.4→29.0GB) + daemon.json live-restore bounce (suggest close after soak) |
+| #685   | ⏳    | tonight 20:00-04:00 CDT storm window is the soak test; caps stay till tymeslot POOL_SIZE |
 | #626   | ⏳    | Cloudron proxying (Grocy multi-user + HA general access) |
 | #710   | ⏳    | custom Ubuntu 24.04 base image for Cloudron hosts |
 | #723   | ⏳    | linkwarden save-to function |
@@ -16,6 +17,7 @@ Last updated: 2026-09-02
 
 ## Inbox (founder action items)
 
-- Cloudron API token (asks in #685/#727): create in the panel → material value
+- axiosheartstudios.com returns 500 — pre-existing app breakage (founder
-  to `~/.creds/cloudron.env` — unblocks API-first access + future AWX modules.
+  confirmed 2026-09-02), unrelated to its memory reslider. Needs an app-level
-- docker daemon.json bounce window (see README) — ticket it when scheduled.
+  look when scheduled.
+- Tymeslot upstream: ask for a POOL_SIZE env so the role caps can be removed.

perf/apply-memory-shrink.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# perf/apply-memory-shrink.sh — apply the t/316 memory shrink table via the
+# Cloudron API. Applied live 2026-09-02 (Redmine #731). Idempotent: re-running
+# sets the same values; 202 = accepted, app restarts.
+#
+# Requires: curl, jq. Creds: ~/.creds/cloudron.env (CLOUDRON_URL, CLOUDRON_API_TOKEN).
+#
+# Gotchas learned on this host:
+# - Route is POST /api/v1/apps/<id>/configure/memory_limit (NOT /memory_limit).
+# - Values <= manifest.memoryLimit (package floor) are rejected 400 "too small"
+#   (e.g. photos/immich floor 3.5GB — left at 4096).
+# - This host's dockerd runs --log-driver=journald as a systemd-unit flag;
+#   daemon.json must NOT set log-driver/log-opts (dockerd refuses to start).
+
+set -euo pipefail
+
+# shellcheck disable=SC1090  # creds path is env-overridable by design
+source "${CLOUDRON_CREDS:-$HOME/.creds/cloudron.env}"
+
+# fqdn|target MB (t/316 table; photos skipped — manifest floor 3.5GB)
+TABLE='
+axiosheartstudios.com|512
+community.turnsys.com|4096
+nextcloud.knownelement.com|2048
+bookmarks.knownelement.com|2048
+share.knownelement.com|512
+hfnfc.net|512
+notes.knownelement.com|1024
+digital-assets-manager.knownelement.com|512
+finance.turnsys.com|512
+finance.reachableceo.com|512
+'
+
+APPS=$(curl -s --max-time 20 -H "Authorization: Bearer $CLOUDRON_API_TOKEN" "$CLOUDRON_URL/api/v1/apps")
+
+echo "$TABLE" | while IFS='|' read -r fqdn mb; do
+  [ -z "$fqdn" ] && continue
+  app_id=$(jq -r --arg f "$fqdn" '.apps[] | select(.fqdn==$f) | .id' <<<"$APPS")
+  [ -z "$app_id" ] && { echo "SKIP (not found): $fqdn"; continue; }
+  code=$(curl -s --max-time 20 -X POST \
+    -H "Authorization: Bearer $CLOUDRON_API_TOKEN" -H "Content-Type: application/json" \
+    -d "{\"memoryLimit\": $((mb * 1048576))}" \
+    "$CLOUDRON_URL/api/v1/apps/$app_id/configure/memory_limit" -o /dev/null -w '%{http_code}')
+  echo "$code  $fqdn -> ${mb}MB"
+  sleep 10
+done

perf/docker-daemon.json

@@ -1,6 +1,4 @@
 {
-  "log-driver": "json-file",
-  "log-opts": { "max-size": "10m", "max-file": "3" },
   "default-ulimits": {
     "nofile": { "Name": "nofile", "Soft": 262144, "Hard": 262144 }
   },

questions-v1.md

@@ -26,5 +26,9 @@ bounces (~300). live-restore is currently false so there is no zero-downtime
 path. Until then the postgres nofile bridge is ephemeral (lost on postgres
 container recreate).
 
-- [ ] Pick a window (suggest: pre-storm weekday morning) — agent will ticket
+## 2. docker daemon.json bounce window (see README "Ground truth")
-      the runbook + execute on your go
+
+- [x] Pick a window — founder chose 2026-09-02 ~19:30 CDT (pre-storm).
+      APPLIED: live-restore + default-ulimits nofile 262144 now active; first
+      attempt failed (log-driver flag conflict — unit passes journald), fixed
+      by dropping log keys; see #731 for the incident record.