mrcharles
0e67f17112
Enforcement (check-rules/hooks/shellcheck) per ADOPTING pattern; perf/ SoR from the #685 session (memcensus, pg role-caps SQL, applied sysctl, planned daemon.json); questions-v1 (API token + dockerd window asks). Meat + verification: https://projects.knownelement.com/issues/727#note-3913
40 lines
2.4 KiB
Markdown
40 lines
2.4 KiB
Markdown
> Knowledge docs live on Discourse — perf write-up: [t/316](https://community.turnsys.com/t/316) ·
|
|
> [VP TechOps category](https://community.turnsys.com/c/vp-techops) ·
|
|
> Work tracking: [Redmine — Cloudron project](https://projects.knownelement.com/projects/88) ·
|
|
> Infra audit log: [t/298](https://community.turnsys.com/t/298)
|
|
|
|
# KNEL/cloudron — tsys-cloudron fleet-machine repo
|
|
|
|
Production Cloudron host **my.knownelement.com** (tailnet name
|
|
`tsys-cloudron.knel.net`). ~55 apps behind the Cloudron panel, platform
|
|
Postgres/MySQL/mail containers, monitored by Uptime Kuma (Cloudron group).
|
|
|
|
| Path | What |
|
|
|------|------|
|
|
| `perf/` | [#685](https://projects.knownelement.com/issues/685) perf work: memory census, pg role-cap SQL, applied sysctl, planned docker daemon.json |
|
|
| `ansible/` | Reserved — AWX/Ansible IaC convergence ([#454](https://projects.knownelement.com/issues/454)); playbooks land here as manual changes are codified |
|
|
| `scripts/` | Rule engine + git hooks (enforcement layer, ADOPTING pattern) |
|
|
| `tests/` | shellcheck wrapper (Docker — host stays clean) |
|
|
|
|
## Ground truth (2026-09-02, #685 — narrative + measurements on [t/316](https://community.turnsys.com/t/316))
|
|
|
|
- Platform Postgres slot exhaustion root-caused to 7 booking instances + Penpot
|
|
pinning ~61 idle connections each. Role caps applied (booking 10, Penpot 15,
|
|
every other app role 20), `max_connections` 500→700, `shared_buffers` 2GB,
|
|
idle-in-transaction timeout 15min. Victim `too many clients` FATALs → zero.
|
|
- Host sysctl: `vm.swappiness=10` (`perf/sysctl/`). THP already `never`, virtio
|
|
scheduler `none` — optimal, left alone.
|
|
- **`/etc/docker/daemon.json` is still ABSENT on the host** (`live-restore=false`,
|
|
containers default to nofile=1024 unless bridged). The postgres fd bridge
|
|
(prlimit 262144) is EPHEMERAL — lost on any postgres container recreate.
|
|
Applying `perf/docker-daemon.json.planned` needs ONE planned dockerd bounce
|
|
(restarts all ~300 containers) in a human-chosen window.
|
|
- Memory sliders: shrink table on t/316 (~28GB headroom). Do NOT shrink
|
|
`bi.knownelement.com` or platform services (Cloudron auto-sizes those).
|
|
|
|
## Access
|
|
|
|
SSH `root@my.knownelement.com` is the only access path — routed through the
|
|
KNEL/PFVCluster `tests/remote.sh` chokepoint (see AGENTS.md). Cloudron API
|
|
token pending; once issued, refs live in `~/.creds/cloudron.env`, never here.
|