mrcharles
1267e60e8e
11 apps reslidered via API (53.4->29.0GB explicit limits); daemon.json live-restore + nofile defaults applied (log keys banned: unit flag conflict — see #731 incident note). perf/apply-memory-shrink.sh codifies the change. All 159 apps verified running. Meat: https://projects.knownelement.com/issues/731#note-3929
2.9 KiB
2.9 KiB
Knowledge docs live on Discourse — perf write-up: t/316 · VP TechOps category · Work tracking: Redmine — Cloudron project · Infra audit log: t/298
KNEL/cloudron — tsys-cloudron fleet-machine repo
Production Cloudron host my.knownelement.com (tailnet name
tsys-cloudron.knel.net). ~55 apps behind the Cloudron panel, platform
Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).
| Path | What |
|---|---|
perf/ |
#685 perf work: memory census, pg role-cap SQL, applied sysctl, applied daemon.json, apply-memory-shrink.sh |
monitoring/ |
Uptime Kuma Cloudron app coverage — 55-app inventory + sync checker (moved from PFVCluster, #435/#685/#727) |
ansible/ |
Reserved — AWX/Ansible IaC convergence (#454); playbooks land here as manual changes are codified |
scripts/ |
Rule engine + git hooks (enforcement layer, ADOPTING pattern) |
tests/ |
shellcheck wrapper (Docker — host stays clean) |
Ground truth (2026-09-02, #685 — narrative + measurements on t/316)
- Platform Postgres slot exhaustion root-caused to 7 booking instances + Penpot
pinning ~61 idle connections each. Role caps applied (booking 10, Penpot 15,
every other app role 20),
max_connections500→700,shared_buffers2GB, idle-in-transaction timeout 15min. Victimtoo many clientsFATALs → zero. - Host sysctl:
vm.swappiness=10(perf/sysctl/). THP alreadynever, virtio schedulernone— optimal, left alone. /etc/docker/daemon.jsonAPPLIED 2026-09-02 (#731):live-restore: true(future dockerd restarts are zero-downtime) +default-ulimits nofile 262144(every container start gets real fd limits — the postgres prlimit bridge is no longer needed;perf/docker-daemon.jsonis the SoR copy). NOTE: this host's dockerd runs--log-driver=journaldas a unit flag — daemon.json must NOT set log-driver/log-opts or dockerd refuses to start (bit us once, see #731). Container log rotation is journald's job here.- Memory shrink table APPLIED 2026-09-02 (#731) via the panel API
(
perf/apply-memory-shrink.sh): 11 apps resliders, explicit limits 53.4 GB → 29.0 GB (~24 GB of the ~28 GB headroom reclaimed). Skips: photos (manifest floor 3.5 GB),bi.knownelement.com+ platform services (do NOT shrink). All 159 apps verified running post-change.
Access
SSH root@my.knownelement.com — routed through the KNEL/PFVCluster
tests/remote.sh chokepoint (see AGENTS.md). Panel API token issued
2026-09-02 (~/.creds/cloudron.env, refs only in this repo) — prefer the
API over SSH for reads/limits; base image rebuilds (#710) still need SSH.