From f7d54a30ae5b5c06c61583ad10a00af9a821cd52 Mon Sep 17 00:00:00 2001
From: reachableceo <charles@turnsys.com>
Date: Wed, 9 Sep 2026 18:59:11 -0500
Subject: [PATCH] incident-log: pwvault web-vault EncString crash case
 (2026-09-09)

Full symptom/root-cause/repair record for the InvalidTypeSymm
web-vault crash so it is never re-derived.
https://projects.knownelement.com/issues/942
---
 skills/agent-provisioning/SKILL.md            | 41 +++++++++++++++++++
 .../references/incident-log.md                | 22 ++++++++++
 2 files changed, 63 insertions(+)

diff --git a/skills/agent-provisioning/SKILL.md b/skills/agent-provisioning/SKILL.md
index 0c6d30f..1ba9d50 100644
--- a/skills/agent-provisioning/SKILL.md
+++ b/skills/agent-provisioning/SKILL.md
@@ -267,3 +267,44 @@ the workspace STATUS/memory.
   first login.
 - `scripts/totp.sh <base32-seed>` — current 6-digit TOTP code (pure bash,
   RFC-7636-vector tested).
+
+## Vaultwarden app (pwvault.turnsys.com) — known issue, quirks, and repair (2026-09-09)
+
+The fleet vault is a Cloudron **Vaultwarden** app (`com.github.bitwardenrs`,
+container id 48757d8a-… on tsys-cloudron-new; DB = **MySQL** database from
+the app container env `CLOUDRON_MYSQL_DATABASE`, on the shared `mysql`
+container — NOT sqlite; a stray `db.sqlite3` in /app/data is junk).
+
+**Known web-vault crash (upstream bug, Vaultwarden discussions #7334/#7464):**
+the 2026.7.0 web client + strict WASM SDK crash with
+`Unhandled error in angular: EncString(InvalidTypeSymm { enc_type: "0", parts: 1 })`
+during vault load → empty vault / endless spinner, on every browser, while
+native apps (iPad/desktop) and API-key clients work fine. Triggered by an
+account key-pair record the client's strict parser rejects
+(`accountKeys.publicKeyEncryptionKeyPair.publicKey` stored as a raw base64
+public key instead of an EncString). Repair that worked:
+`UPDATE users SET public_key = NULL WHERE uuid = '<uid>';` in the MySQL DB
+(take a Cloudron app backup first: `POST /api/v1/apps/<id>/backup` with
+`{"backupSiteId":"<id from GET /api/v1/backup_sites>"}`, poll
+`/api/v1/tasks/<taskId>`); the client regenerates a proper key pair on next
+login. NEVER write to the DB without a fresh backup + explicit founder
+approval for the pwvault container.
+
+**CLI 2FA quirk:** CLI-style logins (`client_id=cli`) are offered ONLY the
+recovery-code provider — an authenticator TOTP seed from the vault will be
+rejected at the 2FA prompt even when the authenticator is correctly
+enrolled (browser sessions get the authenticator prompt; CLI does not).
+Automated flows must use the **API key** (client_credentials grant), which
+bypasses 2FA entirely — `creds/cloudron`-style API keys are the automation
+path, passwords+TOTP are for humans in browsers.
+
+**Cloudron backup API pattern** (for pre-surgery backups): GET
+`/api/v1/backup_sites` → site id; POST `/api/v1/apps/<id>/backup` body
+`{"backupSiteId":"<id>"}`; poll `GET /api/v1/tasks/<taskId>` until
+`success:true`. Response field is `taskId` (not `id`).
+
+**TOTP seeds**: the account's authenticator seed for coo@turnsys.com is
+`BW_TOTP_SECRET` in `/data2/TSGCOO/.config/bw/env` — keep it in sync with
+what is actually enrolled server-side; a stale seed there was the source of
+a multi-hour misdiagnosis (the "wrong password"/"stale seed" theories).
+Clock skew was ruled out (workstation and server within 1s).
diff --git a/skills/agent-provisioning/references/incident-log.md b/skills/agent-provisioning/references/incident-log.md
index 010dd5e..45c7541 100644
--- a/skills/agent-provisioning/references/incident-log.md
+++ b/skills/agent-provisioning/references/incident-log.md
@@ -50,3 +50,25 @@ rabbit hole (see #939 correction).
 ---
 
 **Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)
+
+## 2026-09-09 — pwvault web-vault crash: EncString InvalidTypeSymm
+
+- Symptom: web vault (all browsers, incl. fresh private windows) spins
+  forever after successful password + authenticator 2FA login; zero items.
+  iPad native app + extension worked throughout; CLI (API key) worked.
+- Root cause: the account's `accountKeys.publicKeyEncryptionKeyPair.publicKey`
+  stored a RAW base64 public key ("MI…", not an EncString). The 2026.7.0
+  web client's strict SDK parser rejects it → `InvalidTypeSymm { enc_type:
+  "0", parts: 1 }` → unhandled Angular error aborts vault render.
+  Upstream: vaultwarden discussions #7334, #7464; issue #7469.
+- Repair: `UPDATE users SET public_key = NULL` (MySQL db for the app;
+  coo@ = user uuid 98954ef9-…), then client login regenerates the pair.
+  Cloudron app backup taken first via the backup API (backup_sites →
+  task poll). Verified end-to-end same day.
+- Diagnostic technique that found it: Firefox F12 → Console tab captured
+  the unhandled Angular exception; server logs + API probes showed
+  everything healthy; the crash was client-render only.
+- Related: CLI-style logins (client_id=cli) are offered only the recovery
+  code 2FA provider — authenticator TOTP codes from a vault seed are
+  rejected for CLI logins even with the authenticator correctly enrolled.
+  Automation uses API keys (bypass 2FA), not passwords+TOTP.