mrcharles
f7d54a30ae
Full symptom/root-cause/repair record for the InvalidTypeSymm web-vault crash so it is never re-derived. https://projects.knownelement.com/issues/942
75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
# Incident log — what broke and what it taught (append-only)
|
||
|
||
## 2026-09-07 ~20:00 CT — shared-box degradation (RESOLVED 09-08)
|
||
|
||
One box (152.53.37.179, mail.knownelement.com) hosts gitea, redmine,
|
||
discourse, GLPI, pwvault, and fronts my.knownelement.com. During the
|
||
incident:
|
||
|
||
- Gitea `git-receive-pack` → HTTP 500 on ANY ref (pushes impossible);
|
||
`POST /api/v1/repos` → plain-text 404. Reads + `POST /api/v1/markdown`
|
||
worked fine — partial failure, not an outage.
|
||
- my.knownelement.com → HTTP 500 leaking
|
||
`"Access denied for user 'root'@'localhost'"` (a MySQL root credential
|
||
breakage in whatever backs that vhost).
|
||
- pwvault: unauthenticated endpoints answered in 0.2s, but AUTHENTICATED
|
||
`/api/sync` hung to timeout — a lock/db-layer signature. All vault item
|
||
reads fleet-wide were down; nothing credential-driven could proceed.
|
||
- tsys-cloudron (tailscale 100.107.35.78, self-signed cert — use `curl -k`)
|
||
404'd its hostname; with `Host: my.knownelement.com` it served the same
|
||
JSON as the public vhost (proxy topology). devbox/preprod/sectestbed
|
||
cloudron peers were offline for 1–2 days.
|
||
|
||
Lessons: (1) when several "unrelated" systems on that box fail at once,
|
||
treat it as one box incident — file one ticket, verify recovery per system;
|
||
(2) HTTP-status-only checks lie — the vault was "up" (200s on static paths)
|
||
while 100% of useful reads hung; test the authenticated path you actually
|
||
need; (3) correlate carefully — the MySQL leak was the public box's vhost,
|
||
not the cloudron control box; wrong attribution sent diagnosis down a
|
||
rabbit hole (see #939 correction).
|
||
|
||
## 2026-09-07 — identity/SSO archaeology (the reason this skill exists)
|
||
|
||
- The August "10 identities enrolled + TOTP" batch captured NO seeds. coo's
|
||
TOTP is therefore enforced-but-unrecoverable; no admin reset exists in
|
||
this Cloudron version. Every future enrollment MUST store the seed.
|
||
- `creds/sec-*` / `creds/sre-*` (Sep 07 batch, by VpEngOps) used ONE shared
|
||
password across cloudron+gitea+redmine+discourse, and the stored password
|
||
already fails Cloudron auth (401 tested). Shared passwords are banned
|
||
(#942); treat those items as stale until their owning lane remediates.
|
||
- Several "admin" vault candidates resolved to reachableceo's personal
|
||
reachableceo accounts (gitea id 3, redmine user 5) and none were admin
|
||
(gitea: 403 "user should be the site admin"; redmine: 403 on /users.json).
|
||
They now carry RESTRICTED notes. The compliant Cloudron admin lever is the
|
||
`api-token` user's token in `creds/cloudron`.
|
||
- Debug order that works for auth weirdness: (1) what identity does this
|
||
credential ACTUALLY resolve to (call /user or /users/current)?
|
||
(2) does the route exist at all (SPA-bundle grep beats guessing)?
|
||
(3) is the failure at nginx (HTML status pages) or the app (JSON errors)?
|
||
|
||
---
|
||
|
||
**Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)
|
||
|
||
## 2026-09-09 — pwvault web-vault crash: EncString InvalidTypeSymm
|
||
|
||
- Symptom: web vault (all browsers, incl. fresh private windows) spins
|
||
forever after successful password + authenticator 2FA login; zero items.
|
||
iPad native app + extension worked throughout; CLI (API key) worked.
|
||
- Root cause: the account's `accountKeys.publicKeyEncryptionKeyPair.publicKey`
|
||
stored a RAW base64 public key ("MI…", not an EncString). The 2026.7.0
|
||
web client's strict SDK parser rejects it → `InvalidTypeSymm { enc_type:
|
||
"0", parts: 1 }` → unhandled Angular error aborts vault render.
|
||
Upstream: vaultwarden discussions #7334, #7464; issue #7469.
|
||
- Repair: `UPDATE users SET public_key = NULL` (MySQL db for the app;
|
||
coo@ = user uuid 98954ef9-…), then client login regenerates the pair.
|
||
Cloudron app backup taken first via the backup API (backup_sites →
|
||
task poll). Verified end-to-end same day.
|
||
- Diagnostic technique that found it: Firefox F12 → Console tab captured
|
||
the unhandled Angular exception; server logs + API probes showed
|
||
everything healthy; the crash was client-render only.
|
||
- Related: CLI-style logins (client_id=cli) are offered only the recovery
|
||
code 2FA provider — authenticator TOTP codes from a vault seed are
|
||
rejected for CLI logins even with the authenticator correctly enrolled.
|
||
Automation uses API keys (bypass 2FA), not passwords+TOTP.
|