fix(sysctl): inotify headroom 1024 instances after containerd fd death [#755]

https://projects.knownelement.com/issues/755
2026-09-03 10:18:10 -05:00
parent 01b12493a3
commit 87606da528
2 changed files with 8 additions and 0 deletions
@@ -44,3 +44,4 @@ SSH `root@my.knownelement.com` — routed through the KNEL/PFVCluster
 `tests/remote.sh` chokepoint (see AGENTS.md). Panel API token issued
 2026-09-02 (`~/.creds/cloudron.env`, refs only in this repo) — prefer the
 API over SSH for reads/limits; base image rebuilds (#710) still need SSH.
+- **Inotify headroom APPLIED 2026-09-03** (#755, founder-approved): `fs.inotify.max_user_instances` 128→1024 + `max_user_watches` →1048576 in the same sysctl file, after containerd/dockerd died of inotify exhaustion (full fleet resync 09:13–09:51). Runtime-applied, no daemon restarts; error stream dry from 10:08. OOM fail set complete: thefnf/grafana/projects bumped (1G/1G/1.5G), zero OOMs post-recovery.
@@ -3,3 +3,10 @@
 # SoR copy for IaC; deploy = copy to /etc/sysctl.d/ + sysctl --system
 # (ansible/ playbook target).
 vm.swappiness=10
+
+# 2026-09-03 [#755, founder-approved in chat]: inotify headroom — containerd
+# died of inotify-instance exhaustion at the stock 128 (~160 cgroup-watched
+# containers + storm memcg churn). Applied live (no daemon restarts needed);
+# runtime verified 1024/1048576; error stream dry from 10:08 CT onward.
+fs.inotify.max_user_instances=1024
+fs.inotify.max_user_watches=1048576