47 lines
3.1 KiB
Markdown
47 lines
3.1 KiB
Markdown
> Knowledge docs live on Discourse — perf write-up: [t/316](https://community.turnsys.com/t/316) ·
|
|
> [VP TechOps category](https://community.turnsys.com/c/vp-techops) ·
|
|
> Work tracking: [Redmine — Cloudron project](https://projects.knownelement.com/projects/88) ·
|
|
> Infra audit log: [t/298](https://community.turnsys.com/t/298)
|
|
|
|
# KNEL/cloudron — tsys-cloudron fleet-machine repo
|
|
|
|
Production Cloudron host **my.knownelement.com** (tailnet name
|
|
`tsys-cloudron.knel.net`). ~55 apps behind the Cloudron panel, platform
|
|
Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).
|
|
|
|
| Path | What |
|
|
|------|------|
|
|
| `perf/` | [#685](https://projects.knownelement.com/issues/685) perf work: memory census, pg role-cap SQL, applied sysctl, applied `daemon.json`, `apply-memory-shrink.sh` |
|
|
| `monitoring/` | Uptime Kuma Cloudron app coverage — 55-app inventory + sync checker (moved from PFVCluster, #435/#685/#727) |
|
|
| `ansible/` | Reserved — AWX/Ansible IaC convergence ([#454](https://projects.knownelement.com/issues/454)); playbooks land here as manual changes are codified |
|
|
| `scripts/` | Rule engine + git hooks (enforcement layer, ADOPTING pattern) |
|
|
| `tests/` | shellcheck wrapper (Docker — host stays clean) |
|
|
|
|
## Ground truth (2026-09-02, #685 — narrative + measurements on [t/316](https://community.turnsys.com/t/316))
|
|
|
|
- Platform Postgres slot exhaustion root-caused to 7 booking instances + Penpot
|
|
pinning ~61 idle connections each. Role caps applied (booking 10, Penpot 15,
|
|
every other app role 20), `max_connections` 500→700, `shared_buffers` 2GB,
|
|
idle-in-transaction timeout 15min. Victim `too many clients` FATALs → zero.
|
|
- Host sysctl: `vm.swappiness=10` (`perf/sysctl/`). THP already `never`, virtio
|
|
scheduler `none` — optimal, left alone.
|
|
- **`/etc/docker/daemon.json` APPLIED 2026-09-02** (#731): `live-restore: true`
|
|
(future dockerd restarts are zero-downtime) + `default-ulimits nofile
|
|
262144` (every container start gets real fd limits — the postgres prlimit
|
|
bridge is no longer needed; `perf/docker-daemon.json` is the SoR copy).
|
|
NOTE: this host's dockerd runs `--log-driver=journald` as a unit flag —
|
|
daemon.json must NOT set log-driver/log-opts or dockerd refuses to start
|
|
(bit us once, see #731). Container log rotation is journald's job here.
|
|
- **Memory shrink table APPLIED 2026-09-02** (#731) via the panel API
|
|
(`perf/apply-memory-shrink.sh`): 11 apps resliders, explicit limits
|
|
53.4 GB → 29.0 GB (~24 GB of the ~28 GB headroom reclaimed). Skips:
|
|
photos (manifest floor 3.5 GB), `bi.knownelement.com` + platform services
|
|
(do NOT shrink). All 159 apps verified running post-change. **2026-09-03 addendum:** OOM-driven bumps (thefnf 512M→1G, grafana 512M→1G, projects 600M→1.5G) + inotify-exhaustion dockerd/containerd incident — see #731 + [t/298](https://community.turnsys.com/t/298).
|
|
|
|
## Access
|
|
|
|
SSH `root@my.knownelement.com` — routed through the KNEL/PFVCluster
|
|
`tests/remote.sh` chokepoint (see AGENTS.md). Panel API token issued
|
|
2026-09-02 (`~/.creds/cloudron.env`, refs only in this repo) — prefer the
|
|
API over SSH for reads/limits; base image rebuilds (#710) still need SSH.
|