--- name: agent-provisioning description: > Provision TSYS Group agent identities end-to-end: Cloudron account (unique password, TOTP with seed captured to the TSGCOO vault), SSO first-login into Redmine / Gitea / Discourse / GLPI, per-system API key minting, vault item schema, and verification. Use whenever asked to "stand up" an identity or agent, create/fix Cloudron users, wire SSO, debug identity logins or 2FA, or store identity credentials — even if the word "provisioning" is not used. Policy-critical: encodes founder rulings #942 (unique passwords, Cloudron-only accounts, no reachableceo credentials). --- # Agent identity provisioning (TSYS Group) Stand up an agent identity so it can operate its own accounts in Cloudron and the four SSO-wired systems, with every credential landing in the TSGCOO Bitwarden vault. The Cloudron is the ONLY identity source: never create local accounts in Redmine, Gitea, Discourse, or GLPI — the system account comes into existence when the identity completes its first SSO login there. ## Non-negotiable policy (founder rulings — violating these invalidates the work) 1. **Unique password per identity.** One identity, one password, used only for its Cloudron account. Shared passwords are banned (Redmine #942). 2. **Cloudron-only accounts.** System access = SSO first-login. No local accounts, no admin-created users in the downstream systems. 3. **Never use reachableceo credentials.** Items like `creds/gitea-admin`, `creds/gitea`, `creds/redmine` are the founder's personal accounts and carry RESTRICTED notes. If an auth test resolves to user `reachableceo`, stop — you are holding the founder's identity. 4. **TOTP seeds MUST be stored in Bitwarden** (founder ruling 2026-09-08). Enroll TOTP and write the base32 seed to the vault item in the same breath — a seed that is not in the vault is treated as lost, because there is no admin 2FA reset on this Cloudron and no other record exists. 5. **Secrets live only in the vault** (KNELSecretsManager `sm` CLI) — never in repos, command output, logs, or transcripts. Mask credentials in every command's visible output. 6. **Bookkeeping in the same turn:** Redmine ticket on the provisioning program (#942 or its successor), cross-linked; vault item created;STATUS updated. ## Scope check Identity stand-up is the COO C2 lane's assigned program (founder directive, 2026-09-08). If your chat runs in another lane (SRE, security, OAM), confirm the founder assigned the stand-up to you before executing, or route a ticket to the COO C2 lane instead. ## Admin lever: Cloudron API token `creds/cloudron` holds a working admin API token (service user `api-token`, 64 chars). Use it as `Authorization: Bearer $PASSWORD`. Do NOT log into the dashboard interactively for admin work — the token is the compliant path. ```bash scripts/cloudron-api.sh GET /users?limit=100 # list users (note uid- UUIDs) scripts/cloudron-api.sh GET /users/ # fetch one user object ``` Route notes (this Cloudron, 2026-09): users are addressed by `uid-…` UUIDs, not usernames; there is NO `/api/v1/login` (dashboard auth is its own OIDC flow, client `cid-webadmin`, PKCE S256); admin CAN clear a user's TOTP via `POST /api/v1/users//totp_disable` (empty JSON body) — use it when a seed was lost, then re-enroll with capture. Admin password set: `POST /api/v1/users//password {"password":"…"}`. ## Procedure ### 0. Prerequisites - Vault reachable: `sm get creds/cloudron` works (see references/vault.md). - For a NEW Cloudron user: the founder generates the invite, or you create the user via the admin token if the founder has authorized self-service. - A Redmine ticket to record the work (create one if none exists). ### 1. Cloudron account (unique password) Verify the account exists and matches the identity email pattern (`tsgstaff-@turnsys.com` for COO-line identities). Generate the unique password (32 chars), set it via the admin API, and store it in the vault item login password field: ```bash scripts/cloudron-api.sh POST /users//password '{"password":"<32-char>"}' sm setfield " Cloudron" password "<32-char>" ``` ### 2. TOTP 2FA — enroll and CAPTURE THE SEED Two cases: - **Seed lost / never captured** (TOTP enforced, no `totp_seed` in vault): clear first — `scripts/cloudron-api.sh POST /users//totp_disable` — then proceed to enroll fresh. - **No 2FA yet**: enroll directly. Enrollment runs in the identity's own dashboard session (the totp routes are self-scoped — an admin token gets 404s there): 1. `scripts/cloudron-oidc-login.sh ` — full OIDC login (PKCE + interaction + consent) yielding the identity's Bearer at `/tmp/.oidc-bearer-`. 2. `scripts/cloudron-totp-enroll.sh ` — reads the pending secret, enables with a computed code, stores the seed via `sm setfield`, and verifies end-to-end. If the API confirm path is rejected, fall back to the dashboard UI in the browser: the forced "Enable TOTP" dialog shows the seed; read it, compute with `scripts/totp.sh`, fill the Token field and submit. CRITICAL: the pending secret REGENERATES every time the dialog re-renders — always read the seed and submit the code within the same window, and verify enrollment by a real TOTP login (a totpToken in a login is silently ignored for users without 2FA — "success" there proves nothing). If the identity already has 2FA enforced and no seed exists anywhere: STOP and ask the founder. There is no programmatic recovery (verified 2026-09-08). Compute codes offline when needed: `scripts/totp.sh `. ### 3. TOTP enrollment (in the identity's dashboard session) ```bash scripts/cloudron-oidc-login.sh # OIDC login -> Bearer scripts/cloudron-totp-enroll.sh # enroll + seed + verify ``` If the API enable path is rejected, fall back to the dashboard UI dialog (see step 2's note). The verify pass inside the scripts performs a real TOTP login and is the ONLY trustworthy success signal. ### 4. SSO first-login into each system The interactive login is fully scriptable per app — use the per-app scripts; each runs the whole flow (entry → Cloudron interaction → TOTP from the vault seed → resume → consent → callback) with a cookie jar and verifies a live session: ```bash scripts/gitea-sso-login.sh [vault-item] # /user/oauth2/cloudron scripts/redmine-sso-login.sh [vault-item] # /oauth?oauth_provider=1 scripts/discourse-sso-login.sh [vault-item] # /auth/oidc (+ CSRF + confirm form) ``` | System | SSO start point | Gotchas | |---|---|---| | GLPI | `https://cmdb.knownelement.com/plugins/singlesignon/front/callback.php/provider/1?remember=1` | account auto-provisioned; no scripted flow yet | | Gitea | `https://git.knownelement.com/user/oauth2/cloudron` | redirects to `client_id=-oidc`; gitea may NORMALIZE the username (e.g. `tsgstaff-coo-vpperf` → `vpperf`) — verify which login got created | | Discourse | `https://community.turnsys.com/auth/oidc` | hostname is community.**turnsys.com**; provider name is `oidc`, NOT openid_connect; needs `/session/csrf` token then a confirm-page authenticity_token POST | | Redmine | `https://projects.knownelement.com/oauth?oauth_provider=1` | the "bounce to /login" mystery = the plugin's GET form needs `oauth_provider=1` | A failed session-verify can still mean the ACCOUNT was created (common) — check via the admin API before retrying. App-level access control can deny the whole flow with `access_denied: User has no access to this app`: the Cloudron app's `accessRestriction` (per-user/group allow-list) must include the identity. Set it with `POST /api/v1/apps//configure/access_restriction` body `{"accessRestriction":{"users":[uids…],"groups":[gids…]}}` — the `groups` array is REQUIRED even when empty. Gitea (1c91ff45-…) and Discourse (b506a0c1-…) both carry allow-lists. Key gotcha (cost a debugging session — do not rediscover): the OIDC interaction login is `POST /openid/interaction//login` with JSON body `{"username": "...", "password": "...", "totpToken": "..."}` — the field is **`username`**, not `login`, and the response is `{"redirectTo": ...}` on success (absolute URL — never concatenate a base onto it). ### 5. Mint per-system API keys vptechops holds ADMIN in all four systems (2026-09-09), so keys are minted centrally — no per-identity session needed: - **Gitea** (admin basic auth; the token route REQUIRES basic auth — a token works as the basic password): `POST /api/v1/users//tokens -u "vptechops:$admin_token"` with `{"name":"agent-","scopes":["all"]}` → `.sha1`. Minting is scoped to the authenticating token's scopes, so use the full-scope admin token (`creds/vptechops-gitea` `admin_token`, generated via `gitea admin user generate-access-token` in the container with `--config /run/gitea/app.ini`). - **Redmine**: api_key is not in `GET /users/:id.json` for others — dump via rails runner in the app container (`User.where(status:1).each { |u| puts "#{u.login}|#{u.api_key}" }`; needs `HOME=/tmp`, `SECRET_KEY_BASE` from `/app/data/secrets.yml`, rbenv shims in PATH, `bundle exec rails runner`). - **Discourse**: rails runner `ApiKey.create!(user_id:, description:, created_by_id:)` per user — `.key` readable only at creation. Note: Discourse usernames can differ from Cloudron usernames (legacy `coo1coo`, prefix-stripped `vpengops`…) — resolve via `/admin/users/list/active.json` first. Admin Api-Key minting uses the same model; admin key for vptechops is in `creds/vptechops-discourse` (`admin_api_key`). - **GLPI**: DEFERRED on #947 — REST `user_token` auth 401s even for a super-admin with api_token/personal_token set; do not burn time re-deriving that failure. Store each key in the vault item immediately (schema below), and always verify the write (`sm get … --field …` readback) — `sm setfield` 400s ("Data missing") on some ciphers; fall back to a full item recreate that PRESERVES every existing field (username, password, uri, totp_seed, tokens). ### 6. Vault item schema — `creds/` One item per identity (NOT per system), created as a LOGIN item (never a note — `sm set` makes notes; `sm convert ` converts, but REGENERATE the password via the admin API afterwards because convert drops field values): - login username = Cloudron username; password = the unique Cloudron password - URI = `https://my.knownelement.com` - fields: `email`, `organization`, `role`, `reports_to`, `created`, `services`, plus per-system: `redmine_api_key`, `gitea_token`, `discourse_user_api_key`, `glpi_user_token`, and `totp_seed` Historical items that predate this schema (e.g. `sec-*`, `sre-*` shared- password items) are remediation targets under #942 — upgrade them when the owner lane assigns the work, do not replicate their shape. ### 7. Update the org chart (MANDATORY — an uncharted identity is an incomplete task) After provisioning changes the identity estate: 1. Update the org table in `TSYSGroupCorporate/TSYSGroupAIOS` `agents/global/AGENTS.md` (branch + PR per the house-rules flow; the symlink serves it live once merged). 2. Update `reachableceo/org-buildout` `docs/org-chart.md` (the rendered chart) in the same turn. 3. Note the change on the provisioning ticket. Details: `references/org-chart-update.md`. ### 8. Verify the trail Prove each system knows the identity (expected: 200 + the identity's own name; a 200 from the wrong identity is a FAIL): ```bash curl -sf -H "X-Redmine-API-Key: $KEY" https://projects.knownelement.com/users/current.json | jq .user.login curl -sf -H "Authorization: token $TOKEN" https://git.knownelement.com/api/v1/user | jq .login ``` Then update the Redmine ticket (full URLs, `[agent] ` note) and the workspace STATUS/memory. ## References — read before touching the corresponding system - `references/cloudron.md` — admin API routes, OIDC interaction flow, 2FA. - `references/systems.md` — per-system SSO/API details and known breakage. - `references/vault.md` — sm CLI entry points, gotchas, fallbacks. - `references/org-chart-update.md` — the mandatory post-provisioning chart sync. - `references/incident-log.md` — what went wrong on 2026-09-07/08 and why, so the same failures are not re-derived. ## Scripts - `scripts/cloudron-api.sh METHOD PATH [JSON]` — authenticated admin call. - `scripts/cloudron-oidc-login.sh [vault-item]` — full OIDC dashboard login (PKCE + interaction + consent); Bearer at `/tmp/.oidc-bearer-`. - `scripts/cloudron-totp-enroll.sh [vault-item]` — enroll TOTP (Cloudron 10 profile routes: POST `/api/v1/profile/totp_secret` + `/api/v1/profile/totp_enable`), capture the seed to the vault (setfield → verify → full-recreate fallback), verify end-to-end. - `scripts/gitea-sso-login.sh [vault-item]` — Gitea SSO first login. - `scripts/redmine-sso-login.sh [vault-item]` — Redmine SSO first login. - `scripts/discourse-sso-login.sh [vault-item]` — Discourse SSO first login. - `scripts/totp.sh ` — current 6-digit TOTP code (pure bash, RFC-7636-vector tested).