> Knowledge docs live on Discourse — perf write-up: [t/316](https://community.turnsys.com/t/316) ·
> [VP TechOps category](https://community.turnsys.com/c/vp-techops) ·
> Work tracking: [Redmine — Cloudron project](https://projects.knownelement.com/projects/88) ·
> Infra audit log: [t/298](https://community.turnsys.com/t/298)

# KNEL/cloudron — tsys-cloudron fleet-machine repo

Production Cloudron host **my.knownelement.com** (tailnet name
`tsys-cloudron.knel.net`). ~55 apps behind the Cloudron panel, platform
Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).

| Path | What |
|------|------|
| `perf/` | [#685](https://projects.knownelement.com/issues/685) perf work: memory census, pg role-cap SQL, applied sysctl, applied `daemon.json`, `apply-memory-shrink.sh` |
| `monitoring/` | Uptime Kuma Cloudron app coverage — 55-app inventory + sync checker (moved from PFVCluster, #435/#685/#727) |
| `ansible/` | Reserved — AWX/Ansible IaC convergence ([#454](https://projects.knownelement.com/issues/454)); playbooks land here as manual changes are codified |
| `scripts/` | Rule engine + git hooks (enforcement layer, ADOPTING pattern) |
| `tests/` | shellcheck wrapper (Docker — host stays clean) |

## Ground truth (2026-09-02, #685 — narrative + measurements on [t/316](https://community.turnsys.com/t/316))

- Platform Postgres slot exhaustion root-caused to 7 booking instances + Penpot
  pinning ~61 idle connections each. Role caps applied (booking 10, Penpot 15,
  every other app role 20), `max_connections` 500→700, `shared_buffers` 2GB,
  idle-in-transaction timeout 15min. Victim `too many clients` FATALs → zero.
- Host sysctl: `vm.swappiness=10` (`perf/sysctl/`). THP already `never`, virtio
  scheduler `none` — optimal, left alone.
- **`/etc/docker/daemon.json` APPLIED 2026-09-02** (#731): `live-restore: true`
  (future dockerd restarts are zero-downtime) + `default-ulimits nofile
  262144` (every container start gets real fd limits — the postgres prlimit
  bridge is no longer needed; `perf/docker-daemon.json` is the SoR copy).
  NOTE: this host's dockerd runs `--log-driver=journald` as a unit flag —
  daemon.json must NOT set log-driver/log-opts or dockerd refuses to start
  (bit us once, see #731). Container log rotation is journald's job here.
- **Memory shrink table APPLIED 2026-09-02** (#731) via the panel API
  (`perf/apply-memory-shrink.sh`): 11 apps resliders, explicit limits
  53.4 GB → 29.0 GB (~24 GB of the ~28 GB headroom reclaimed). Skips:
  photos (manifest floor 3.5 GB), `bi.knownelement.com` + platform services
  (do NOT shrink). All 159 apps verified running post-change. **2026-09-03 addendum:** OOM-driven bumps (thefnf 512M→1G, grafana 512M→1G, projects 600M→1.5G) + inotify-exhaustion dockerd/containerd incident — see #731 + [t/298](https://community.turnsys.com/t/298).

## Access

SSH `root@my.knownelement.com` — routed through the KNEL/PFVCluster
`tests/remote.sh` chokepoint (see AGENTS.md). Panel API token issued
2026-09-02 (`~/.creds/cloudron.env`, refs only in this repo) — prefer the
API over SSH for reads/limits; base image rebuilds (#710) still need SSH.