KNELAgentIdentityProvisioning/skills/agent-provisioning/references/incident-log.md

Incident log — what broke and what it taught (append-only)

2026-09-07 ~20:00 CT — shared-box degradation (RESOLVED 09-08)

One box (152.53.37.179, mail.knownelement.com) hosts gitea, redmine, discourse, GLPI, pwvault, and fronts my.knownelement.com. During the incident:

• Gitea git-receive-pack → HTTP 500 on ANY ref (pushes impossible); POST /api/v1/repos → plain-text 404. Reads + POST /api/v1/markdown worked fine — partial failure, not an outage.
• my.knownelement.com → HTTP 500 leaking "Access denied for user 'root'@'localhost'" (a MySQL root credential breakage in whatever backs that vhost).
• pwvault: unauthenticated endpoints answered in 0.2s, but AUTHENTICATED /api/sync hung to timeout — a lock/db-layer signature. All vault item reads fleet-wide were down; nothing credential-driven could proceed.
• tsys-cloudron (tailscale 100.107.35.78, self-signed cert — use curl -k) 404'd its hostname; with Host: my.knownelement.com it served the same JSON as the public vhost (proxy topology). devbox/preprod/sectestbed cloudron peers were offline for 1–2 days.

Lessons: (1) when several "unrelated" systems on that box fail at once, treat it as one box incident — file one ticket, verify recovery per system; (2) HTTP-status-only checks lie — the vault was "up" (200s on static paths) while 100% of useful reads hung; test the authenticated path you actually need; (3) correlate carefully — the MySQL leak was the public box's vhost, not the cloudron control box; wrong attribution sent diagnosis down a rabbit hole (see #939 correction).

2026-09-07 — identity/SSO archaeology (the reason this skill exists)

• The August "10 identities enrolled + TOTP" batch captured NO seeds. coo's TOTP is therefore enforced-but-unrecoverable; no admin reset exists in this Cloudron version. Every future enrollment MUST store the seed.
• creds/sec-* / creds/sre-* (Sep 07 batch, by VpEngOps) used ONE shared password across cloudron+gitea+redmine+discourse, and the stored password already fails Cloudron auth (401 tested). Shared passwords are banned (#942); treat those items as stale until their owning lane remediates.
• Several "admin" vault candidates resolved to reachableceo's personal reachableceo accounts (gitea id 3, redmine user 5) and none were admin (gitea: 403 "user should be the site admin"; redmine: 403 on /users.json). They now carry RESTRICTED notes. The compliant Cloudron admin lever is the api-token user's token in creds/cloudron.
• Debug order that works for auth weirdness: (1) what identity does this credential ACTUALLY resolve to (call /user or /users/current)? (2) does the route exist at all (SPA-bundle grep beats guessing)? (3) is the failure at nginx (HTML status pages) or the app (JSON errors)?

---

Records: Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)

2026-09-09 — pwvault web-vault crash: EncString InvalidTypeSymm

• Symptom: web vault (all browsers, incl. fresh private windows) spins forever after successful password + authenticator 2FA login; zero items. iPad native app + extension worked throughout; CLI (API key) worked.
• Root cause: the account's accountKeys.publicKeyEncryptionKeyPair.publicKey stored a RAW base64 public key ("MI…", not an EncString). The 2026.7.0 web client's strict SDK parser rejects it → InvalidTypeSymm { enc_type: "0", parts: 1 } → unhandled Angular error aborts vault render. Upstream: vaultwarden discussions #7334, #7464; issue #7469.
• Repair: UPDATE users SET public_key = NULL (MySQL db for the app; coo@ = user uuid 98954ef9-…), then client login regenerates the pair. Cloudron app backup taken first via the backup API (backup_sites → task poll). Verified end-to-end same day.
• Diagnostic technique that found it: Firefox F12 → Console tab captured the unhandled Angular exception; server logs + API probes showed everything healthy; the crash was client-render only.
• Related: CLI-style logins (client_id=cli) are offered only the recovery code 2FA provider — authenticator TOTP codes from a vault seed are rejected for CLI logins even with the authenticator correctly enrolled. Automation uses API keys (bypass 2FA), not passwords+TOTP.