skills: agent-provisioning - codified identity stand-up workflow

Policy rulings (#942): unique passwords, Cloudron-only SSO accounts,
no reachableceo credentials, TOTP seeds MUST be captured to the vault.
Cloudron admin API wrapper, cookie-jar OIDC login, RFC-6238-tested
TOTP helper, per-system SSO/API references, incident log.

skills/agent-provisioning/references/incident-log.md

@@ -0,0 +1,52 @@
+# Incident log — what broke and what it taught (append-only)
+
+## 2026-09-07 ~20:00 CT — shared-box degradation (RESOLVED 09-08)
+
+One box (152.53.37.179, mail.knownelement.com) hosts gitea, redmine,
+discourse, GLPI, pwvault, and fronts my.knownelement.com. During the
+incident:
+
+- Gitea `git-receive-pack` → HTTP 500 on ANY ref (pushes impossible);
+  `POST /api/v1/repos` → plain-text 404. Reads + `POST /api/v1/markdown`
+  worked fine — partial failure, not an outage.
+- my.knownelement.com → HTTP 500 leaking
+  `"Access denied for user 'root'@'localhost'"` (a MySQL root credential
+  breakage in whatever backs that vhost).
+- pwvault: unauthenticated endpoints answered in 0.2s, but AUTHENTICATED
+  `/api/sync` hung to timeout — a lock/db-layer signature. All vault item
+  reads fleet-wide were down; nothing credential-driven could proceed.
+- tsys-cloudron (tailscale 100.107.35.78, self-signed cert — use `curl -k`)
+  404'd its hostname; with `Host: my.knownelement.com` it served the same
+  JSON as the public vhost (proxy topology). devbox/preprod/sectestbed
+  cloudron peers were offline for 1–2 days.
+
+Lessons: (1) when several "unrelated" systems on that box fail at once,
+treat it as one box incident — file one ticket, verify recovery per system;
+(2) HTTP-status-only checks lie — the vault was "up" (200s on static paths)
+while 100% of useful reads hung; test the authenticated path you actually
+need; (3) correlate carefully — the MySQL leak was the public box's vhost,
+not the cloudron control box; wrong attribution sent diagnosis down a
+rabbit hole (see #939 correction).
+
+## 2026-09-07 — identity/SSO archaeology (the reason this skill exists)
+
+- The August "10 identities enrolled + TOTP" batch captured NO seeds. coo's
+  TOTP is therefore enforced-but-unrecoverable; no admin reset exists in
+  this Cloudron version. Every future enrollment MUST store the seed.
+- `creds/sec-*` / `creds/sre-*` (Sep 07 batch, by VpEngOps) used ONE shared
+  password across cloudron+gitea+redmine+discourse, and the stored password
+  already fails Cloudron auth (401 tested). Shared passwords are banned
+  (#942); treat those items as stale until their owning lane remediates.
+- Several "admin" vault candidates resolved to the founder's personal
+  reachableceo accounts (gitea id 3, redmine user 5) and none were admin
+  (gitea: 403 "user should be the site admin"; redmine: 403 on /users.json).
+  They now carry RESTRICTED notes. The compliant Cloudron admin lever is the
+  `api-token` user's token in `creds/cloudron`.
+- Debug order that works for auth weirdness: (1) what identity does this
+  credential ACTUALLY resolve to (call /user or /users/current)?
+  (2) does the route exist at all (SPA-bundle grep beats guessing)?
+  (3) is the failure at nginx (HTML status pages) or the app (JSON errors)?
+
+---
+
+**Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)