skills: agent-provisioning - codified identity stand-up workflow

Policy rulings (#942): unique passwords, Cloudron-only SSO accounts,
no reachableceo credentials, TOTP seeds MUST be captured to the vault.
Cloudron admin API wrapper, cookie-jar OIDC login, RFC-6238-tested
TOTP helper, per-system SSO/API references, incident log.

skills/agent-provisioning/references/cloudron.md

@@ -0,0 +1,62 @@
+# Cloudron admin API + OIDC (verified 2026-09-08)
+
+## Instance
+
+- Dashboard: `https://my.knownelement.com` (admin surface; this host also
+  fronts the org's public vhosts — it is NOT the tailscale-only control box).
+- API token: vault item `creds/cloudron`, password field, 64 chars, service
+  user `api-token`. Verified: reads full user directory.
+
+## Routes (this is a newer Cloudron — older route knowledge may not apply)
+
+- Users are addressed by `uid-<uuid>`, never username.
+  `GET /api/v1/users?limit=100` → list; `GET /api/v1/users/<uid>` → object
+  (`twoFactorAuthenticationEnabled` is null/absent on objects even when
+  enforced — the OIDC interaction is the source of truth).
+- There is NO `POST /api/v1/login`. Dashboard auth is its own OIDC browser
+  flow; the SPA bundle (as of 2026-09-08) contains no login route.
+- There is NO admin route to reset another user's 2FA (PUT/PATCH/DELETE on
+  `/users/<uid>` 2FA variants all 404; the dashboard UI only offers profile
+  self-service enrollment). Consequence: TOTP seeds not captured at
+  enrollment are unrecoverable without the founder.
+- Other useful routes seen in the SPA bundle: `/api/v1/tokens`,
+  `/api/v1/oidc/clients`, `/api/v1/groups`, `/api/v1/eventlog`,
+  `/api/v1/app_passwords`. Verify behavior at runtime before relying on a
+  specific payload shape.
+
+## OIDC interaction login (how identity SSO works)
+
+The GLPI/Gitea/Discourse/Redmine SSO flows all redirect here:
+
+1. `GET <app SSO start point>` → 302 to
+   `https://my.knownelement.com/openid/interaction/<uid>` (Cloudron runs a
+   node-oidc-provider at `/openid`). Cookies matter — keep a cookie jar.
+2. `POST /openid/interaction/<uid>/login` with JSON
+   `{"username": "<login>", "password": "<unique password>"}`.
+   - Field name is **`username`** — `login` returns
+     `"A username must be non-empty string"` (400).
+   - Success → `{"redirectTo": "..."}` (or
+     `{"twoFactorRequired":true,"totpRequired":true}` if 2FA is enabled).
+   - Wrong credentials → 401 `{"status":"Unauthorized","message":"Username
+     and password does not match"}`.
+3. `GET` the `redirectTo` (same cookie jar) → completes at the app.
+
+Email-style usernames work (`tsgstaff-coo@turnsys.com`); bare usernames also
+exist (`coo`). Identity emails follow `tsgstaff-<login>@turnsys.com`.
+
+## 2FA
+
+- TOTP enforced per user at the interaction step
+  (`twoFactorRequired`/`totpRequired` in the login response).
+- Enrollment is profile self-service ONLY. Capture the base32 seed into the
+  vault (`sm setfield "<login> Cloudron" totp_seed <seed>`) at enrollment —
+  there is no admin reset and no recovery path.
+- Policy RULED (founder, 2026-09-08): TOTP seeds MUST be stored in Bitwarden.
+  Enroll 2FA only when the seed is being captured to the vault item in the
+  same step. Legacy identities whose seeds were never captured (the whole
+  August batch) are reset-on-sight candidates: founder clears 2FA or hands
+  over the seed, then re-enroll with capture.
+
+---
+
+**Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)

skills/agent-provisioning/references/incident-log.md

@@ -0,0 +1,52 @@
+# Incident log — what broke and what it taught (append-only)
+
+## 2026-09-07 ~20:00 CT — shared-box degradation (RESOLVED 09-08)
+
+One box (152.53.37.179, mail.knownelement.com) hosts gitea, redmine,
+discourse, GLPI, pwvault, and fronts my.knownelement.com. During the
+incident:
+
+- Gitea `git-receive-pack` → HTTP 500 on ANY ref (pushes impossible);
+  `POST /api/v1/repos` → plain-text 404. Reads + `POST /api/v1/markdown`
+  worked fine — partial failure, not an outage.
+- my.knownelement.com → HTTP 500 leaking
+  `"Access denied for user 'root'@'localhost'"` (a MySQL root credential
+  breakage in whatever backs that vhost).
+- pwvault: unauthenticated endpoints answered in 0.2s, but AUTHENTICATED
+  `/api/sync` hung to timeout — a lock/db-layer signature. All vault item
+  reads fleet-wide were down; nothing credential-driven could proceed.
+- tsys-cloudron (tailscale 100.107.35.78, self-signed cert — use `curl -k`)
+  404'd its hostname; with `Host: my.knownelement.com` it served the same
+  JSON as the public vhost (proxy topology). devbox/preprod/sectestbed
+  cloudron peers were offline for 1–2 days.
+
+Lessons: (1) when several "unrelated" systems on that box fail at once,
+treat it as one box incident — file one ticket, verify recovery per system;
+(2) HTTP-status-only checks lie — the vault was "up" (200s on static paths)
+while 100% of useful reads hung; test the authenticated path you actually
+need; (3) correlate carefully — the MySQL leak was the public box's vhost,
+not the cloudron control box; wrong attribution sent diagnosis down a
+rabbit hole (see #939 correction).
+
+## 2026-09-07 — identity/SSO archaeology (the reason this skill exists)
+
+- The August "10 identities enrolled + TOTP" batch captured NO seeds. coo's
+  TOTP is therefore enforced-but-unrecoverable; no admin reset exists in
+  this Cloudron version. Every future enrollment MUST store the seed.
+- `creds/sec-*` / `creds/sre-*` (Sep 07 batch, by VpEngOps) used ONE shared
+  password across cloudron+gitea+redmine+discourse, and the stored password
+  already fails Cloudron auth (401 tested). Shared passwords are banned
+  (#942); treat those items as stale until their owning lane remediates.
+- Several "admin" vault candidates resolved to the founder's personal
+  reachableceo accounts (gitea id 3, redmine user 5) and none were admin
+  (gitea: 403 "user should be the site admin"; redmine: 403 on /users.json).
+  They now carry RESTRICTED notes. The compliant Cloudron admin lever is the
+  `api-token` user's token in `creds/cloudron`.
+- Debug order that works for auth weirdness: (1) what identity does this
+  credential ACTUALLY resolve to (call /user or /users/current)?
+  (2) does the route exist at all (SPA-bundle grep beats guessing)?
+  (3) is the failure at nginx (HTML status pages) or the app (JSON errors)?
+
+---
+
+**Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)

skills/agent-provisioning/references/systems.md

@@ -0,0 +1,73 @@
+# Per-system SSO + API key details (verified 2026-09-07/08 unless noted)
+
+## GLPI (cmdb.knownelement.com) — SSO VERIFIED WORKING
+
+- SSO: "Login with KNEL Cloud" = plugin route
+  `/plugins/singlesignon/front/callback.php/provider/1?remember=1` → Cloudron
+  OIDC → auto-provision.
+- The login page defaults to whichever tab was last toggled ("Use GLPI login
+  form" / "Use Single Sign-on"); a hidden `noAUTO=1` input forces the local
+  form — irrelevant under Cloudron-only policy, useful to recognize.
+- API: GLPI 11 is OAuth/token-based. The legacy REST at
+  `/apirest.php/initSession` exists: password login is DISABLED (400);
+  user_token auth exists but the tokens in `creds/glpi` / `creds/mglpi` are
+  STALE (401 ERROR_GLPI_LOGIN_USER_TOKEN). The app token in `creds/glpi`
+  password field IS accepted (used as `App-Token` header).
+- Per-user API token: My account (self-service) after SSO provisioning. Pair
+  with the app token: `GET /apirest.php/initSession` with headers
+  `App-Token: <app>` + `Authorization: user_token <user>`.
+- Local web login as `glpi` super-admin hits a 6-digit TOTP at
+  `/MFA/Prompt` whose secret is not stored anywhere — do not rely on the
+  admin UI; the SSO path makes it unnecessary.
+
+## Gitea (git.knownelement.com)
+
+- SSO: `/user/oauth2/cloudron` (button "Sign in with …" on the login page).
+- Standalone install behind nginx on the shared box — local accounts exist
+  from the August batch; Cloudron-only policy means new identities use SSO
+  and legacy ones migrate.
+- API: tokens via `POST /api/v1/users/<login>/tokens` REQUIRE basic auth AS
+  that user (admin tokens cannot mint for others; there is no
+  `/admin/users/<name>/tokens` route; `Sudo` header needs a real site admin).
+- Token auth against `/api/v1/user`: the JSON has `login`; admin-ness shows
+  in `site_admin` (absent = not admin; do not misread null as false).
+- KNOWN BROKEN (2026-09-07/08): receive-pack 500s during the box incident —
+  RESOLVED after the 09-08 repair (pushes verified). If pushes 500 again,
+  check the shared box first (see incident-log.md).
+
+## Discourse (community.turnsys.com)
+
+- SSO: `/auth/openid_connect` — a bare GET returns 403 (CSRF-gated); that is
+  NORMAL and means the route exists. Complete the flow from the login page
+  button or with a session + CSRF token.
+- The login page is an Ember SPA: server-rendered HTML shows no SSO markers,
+  which fooled one audit (2026-09-07) into "no SSO" — wrong. Check routes,
+  not page source.
+- API: user API keys (User-Api-Key header) minted via the user-approval flow;
+  admin Api-Key + Api-Username impersonation needs an admin key. The vault's
+  `creds/discourse` NOTE says its keys are NOT admin (they are also not hex —
+  do not assume key format). `creds/vptechops-discourse` holds a WORKING
+  user key (user `vptechopsvptechops`) — #933's "dead key" note is wrong.
+
+## Redmine (projects.knownelement.com)
+
+- SSO: "Continue with KNEL Cloud" button POSTs/GETs `/oauth`.
+  KNOWN BROKEN: `/oauth` 302-bounces straight back to /login with no
+  provider redirect — verified 2026-09-07 AND 2026-09-08 (IdP healthy both
+  times), so it is a Redmine plugin configuration issue, NOT the IdP.
+  Retest before relying on it; until fixed, Redmine accounts cannot be
+  created via SSO.
+- API keys: `GET /users/current.json` (session or basic auth) returns the
+  user's `api_key`; X-Redmine-API-Key header for calls. Vault convention:
+  the API key is stored in the item's password field.
+- Admin surface: `/users.json` lists users (admin only — 403 otherwise).
+  Every admin candidate tested in the vault so far is NON-admin (they are
+  all the founder's personal accounts anyway — RESTRICTED).
+
+## Shared infrastructure hazard
+
+Gitea, Redmine, Discourse, GLPI, the pwvault, and the my. vhost all run on
+ONE box (152.53.37.179 = mail.knownelement.com). Symptoms there cluster:
+when several systems misbehave at once, suspect the box (see
+incident-log.md), file/update one ticket, and do not burn hours debugging
+four clients.

skills/agent-provisioning/references/vault.md

@@ -0,0 +1,48 @@
+# TSGCOO vault (`sm` CLI) — entry points and gotchas
+
+The KNELSecretsManager vault (Vaultwarden at pwvault.turnsys.com, account
+coo@turnsys.com) is the ONLY credential store. Full doctrine:
+OAM workspace memory `tsgcoo-account-ops` + `creds-migration`; tool
+reference: KNEL workspace memory `knelsecretsmanager-reference`.
+
+## Invocation
+
+```bash
+# from the reachableceo account (this workstation):
+sudo -u TSGCOO /data2/TSGCOO/.local/bin/sm <cmd>   # TSGCOO crossover
+# from inside the TSGCOO account:
+/data2/TSGCOO/.local/bin/sm <cmd>
+# raw container form:
+docker exec -i ukrrs-secretsmgr-cli smcli <cmd>
+```
+
+Commands: `status`, `list [pattern]`, `get <name> [--field KEY]`,
+`env <name>`, `set`, `setfield <name> <key> <value>`, `rm`, `folders`.
+Item naming: `creds/<name>` for services; `<login> <System>` for identity
+login items; `creds/<login>` for the new per-identity schema.
+
+## Gotchas that have actually bitten
+
+1. `sm env` / `sm get` in scripts: append `</dev/null` — docker exec -i eats
+   stdin and hangs the call inside loops/pipelines.
+2. `--field URI` NEVER matches — URIs are login URIs, not custom fields. Use
+   `sm env` (synthetic URI/USERNAME/PASSWORD) or `--field password`.
+3. `sm env` PASSWORD is the login password, which for token-style items IS
+   the token (e.g. `creds/cloudron`). Always confirm what a field holds
+   before assuming password-vs-token semantics.
+4. zsh does not word-split `$VAR` — `SM="sudo -u ... sm"; $SM get` fails with
+   "no such file". Use a function: `smget() { sudo -u TSGCOO /data2/TSGCOO/.local/bin/sm "$@" </dev/null; }`.
+5. Mask secrets in ALL visible output: pipe through
+   `sed -E 's/((password|token|secret|key|seed|totp)[^:=]*[=:]).*/\1 [MASKED]/Ig'`
+   and jq projections of field NAMES/lengths, never values.
+6. `smcli` self-heals expired sessions (v13: refresh-token rotation +
+   fallback relogin using injected env). If `smcli status` is fine but
+   fetches hang (`/api/sync` context deadline), the vault server is sick —
+   stop retrying, check the shared box (incident-log.md), escalate.
+7. Never print a filled password form's DOM snapshot (browser work) — the
+   ARIA tree includes field values; one coo password hit a transcript this
+   way (2026-09-07).
+
+---
+
+**Records:** Redmine #942 — https://projects.knownelement.com/issues/942 · Discourse docs home: https://community.turnsys.com (skill topic pending admin-key provisioning, tracked on #942)