KNELAgentIdentityProvisioning/skills/agent-provisioning/SKILL.md

name, description

name

description

agent-provisioning

Provision TSYS Group agent identities end-to-end: Cloudron account (unique password, TOTP with seed captured to the TSGCOO vault), SSO first-login into Redmine / Gitea / Discourse / GLPI, per-system API key minting, vault item schema, and verification. Use whenever asked to "stand up" an identity or agent, create/fix Cloudron users, wire SSO, debug identity logins or 2FA, or store identity credentials — even if the word "provisioning" is not used. Policy-critical: encodes founder rulings #942 (unique passwords, Cloudron-only accounts, no reachableceo credentials).

Agent identity provisioning (TSYS Group)

Stand up an agent identity so it can operate its own accounts in Cloudron and the four SSO-wired systems, with every credential landing in the TSGCOO Bitwarden vault. The Cloudron is the ONLY identity source: never create local accounts in Redmine, Gitea, Discourse, or GLPI — the system account comes into existence when the identity completes its first SSO login there.

Non-negotiable policy (founder rulings — violating these invalidates the work)

1. Unique password per identity. One identity, one password, used only for its Cloudron account. Shared passwords are banned (Redmine #942).
2. Cloudron-only accounts. System access = SSO first-login. No local accounts, no admin-created users in the downstream systems.
3. Never use reachableceo credentials. Items like creds/gitea-admin, creds/gitea, creds/redmine are the founder's personal accounts and carry RESTRICTED notes. If an auth test resolves to user reachableceo, stop — you are holding the founder's identity.
4. TOTP seeds MUST be stored in Bitwarden (founder ruling 2026-09-08). Enroll TOTP and write the base32 seed to the vault item in the same breath — a seed that is not in the vault is treated as lost, because there is no admin 2FA reset on this Cloudron and no other record exists.
5. Secrets live only in the vault (KNELSecretsManager sm CLI) — never in repos, command output, logs, or transcripts. Mask credentials in every command's visible output.
6. Bookkeeping in the same turn: Redmine ticket on the provisioning program (#942 or its successor), cross-linked; vault item created;STATUS updated.

Scope check

Identity stand-up is the COO C2 lane's assigned program (founder directive, 2026-09-08). If your chat runs in another lane (SRE, security, OAM), confirm the founder assigned the stand-up to you before executing, or route a ticket to the COO C2 lane instead.

Admin lever: Cloudron API token

creds/cloudron holds a working admin API token (service user api-token, 64 chars). Use it as Authorization: Bearer $PASSWORD. Do NOT log into the dashboard interactively for admin work — the token is the compliant path.

scripts/cloudron-api.sh GET /users?limit=100     # list users (note uid- UUIDs)
scripts/cloudron-api.sh GET  /users/<uid>        # fetch one user object

Route notes (this Cloudron, 2026-09): users are addressed by uid-… UUIDs, not usernames; there is NO /api/v1/login (dashboard auth is its own OIDC flow, client cid-webadmin, PKCE S256); admin CAN clear a user's TOTP via POST /api/v1/users/<uid>/totp_disable (empty JSON body) — use it when a seed was lost, then re-enroll with capture. Admin password set: POST /api/v1/users/<uid>/password {"password":"…"}.

Procedure

0. Prerequisites

• Vault reachable: sm get creds/cloudron works (see references/vault.md).
• For a NEW Cloudron user: the founder generates the invite, or you create the user via the admin token if the founder has authorized self-service.
• A Redmine ticket to record the work (create one if none exists).

1. Cloudron account (unique password)

Verify the account exists and matches the identity email pattern (tsgstaff-<login>@turnsys.com for COO-line identities). Generate the unique password (32 chars), set it via the admin API, and store it in the vault item login password field:

scripts/cloudron-api.sh POST /users/<uid>/password '{"password":"<32-char>"}'
sm setfield "<login> Cloudron" password "<32-char>"

2. TOTP 2FA — enroll and CAPTURE THE SEED

Two cases:

• Seed lost / never captured (TOTP enforced, no totp_seed in vault): clear first — scripts/cloudron-api.sh POST /users/<uid>/totp_disable — then proceed to enroll fresh.
• No 2FA yet: enroll directly.

Enrollment runs in the identity's own dashboard session (the totp routes are self-scoped — an admin token gets 404s there):

1. scripts/cloudron-oidc-login.sh <username> <vault-item> — full OIDC login (PKCE + interaction + consent) yielding the identity's Bearer at /tmp/.oidc-bearer-<username>.
2. scripts/cloudron-totp-enroll.sh <username> <vault-item> — reads the pending secret, enables with a computed code, stores the seed via sm setfield, and verifies end-to-end.

If the API confirm path is rejected, fall back to the dashboard UI in the browser: the forced "Enable TOTP" dialog shows the seed; read it, compute with scripts/totp.sh, fill the Token field and submit. CRITICAL: the pending secret REGENERATES every time the dialog re-renders — always read the seed and submit the code within the same window, and verify enrollment by a real TOTP login (a totpToken in a login is silently ignored for users without 2FA — "success" there proves nothing).

If the identity already has 2FA enforced and no seed exists anywhere: STOP and ask the founder. There is no programmatic recovery (verified 2026-09-08). Compute codes offline when needed: scripts/totp.sh <seed>.

3. TOTP enrollment (in the identity's dashboard session)

scripts/cloudron-oidc-login.sh <username> <vault-item>   # OIDC login -> Bearer
scripts/cloudron-totp-enroll.sh <username> <vault-item>  # enroll + seed + verify

If the API enable path is rejected, fall back to the dashboard UI dialog (see step 2's note). The verify pass inside the scripts performs a real TOTP login and is the ONLY trustworthy success signal.

4. SSO first-login into each system

The interactive login is fully scriptable per app — use the per-app scripts; each runs the whole flow (entry → Cloudron interaction → TOTP from the vault seed → resume → consent → callback) with a cookie jar and verifies a live session:

scripts/gitea-sso-login.sh     <username> [vault-item]   # /user/oauth2/cloudron
scripts/redmine-sso-login.sh   <username> [vault-item]   # /oauth?oauth_provider=1
scripts/discourse-sso-login.sh <username> [vault-item]   # /auth/oidc (+ CSRF + confirm form)

System	SSO start point	Gotchas
GLPI	https://cmdb.knownelement.com/plugins/singlesignon/front/callback.php/provider/1?remember=1	script exists (glpi-sso-login.sh) but the plugin callback is BROKEN on GLPI 11 (include-path + session/CSRF Access denied) — provision accounts via REST instead (recipe below)
Gitea	https://git.knownelement.com/user/oauth2/cloudron	redirects to client_id=<app-uuid>-oidc; gitea may NORMALIZE the username (e.g. tsgstaff-coo-vpperf → vpperf) — verify which login got created
Discourse	https://community.turnsys.com/auth/oidc	hostname is community.turnsys.com; provider name is oidc, NOT openid_connect; needs /session/csrf token then a confirm-page authenticity_token POST
Redmine	https://projects.knownelement.com/oauth?oauth_provider=1	the "bounce to /login" mystery = the plugin's GET form needs oauth_provider=1

A failed session-verify can still mean the ACCOUNT was created (common) — check via the admin API before retrying. App-level access control can deny the whole flow with access_denied: User has no access to this app: the Cloudron app's accessRestriction (per-user/group allow-list) must include the identity. Set it with POST /api/v1/apps/<app-id>/configure/access_restriction body {"accessRestriction":{"users":[uids…],"groups":[gids…]}} — the groups array is REQUIRED even when empty. Gitea (1c91ff45-…) and Discourse (b506a0c1-…) both carry allow-lists.

Key gotcha (cost a debugging session — do not rediscover): the OIDC interaction login is POST /openid/interaction/<uid>/login with JSON body {"username": "...", "password": "...", "totpToken": "..."} — the field is username, not login, and the response is {"redirectTo": ...} on success (absolute URL — never concatenate a base onto it).

5. Mint per-system API keys

vptechops holds ADMIN in all four systems (2026-09-09), so keys are minted centrally — no per-identity session needed:

• Gitea (admin basic auth; the token route REQUIRES basic auth — a token works as the basic password): POST /api/v1/users/<login>/tokens -u "vptechops:$admin_token" with {"name":"agent-<date>","scopes":["all"]} → .sha1. Minting is scoped to the authenticating token's scopes, so use the full-scope admin token (creds/vptechops-gitea admin_token, generated via gitea admin user generate-access-token in the container with --config /run/gitea/app.ini).
• Redmine: api_key is not in GET /users/:id.json for others — dump via rails runner in the app container (User.where(status:1).each { |u| puts "#{u.login}|#{u.api_key}" }; needs HOME=/tmp, SECRET_KEY_BASE from /app/data/secrets.yml, rbenv shims in PATH, bundle exec rails runner).
• Discourse: rails runner ApiKey.create!(user_id:, description:, created_by_id:) per user — .key readable only at creation. Note: Discourse usernames can differ from Cloudron usernames (legacy coo1coo, prefix-stripped vpengops…) — resolve via /admin/users/list/active.json first. Admin Api-Key minting uses the same model; admin key for vptechops is in creds/vptechops-discourse (admin_api_key).
• GLPI (WORKING as of 2026-09-09, #947 root-caused): REST needs BOTH user_token AND app_token: GET /apirest.php/initSession?user_token=<plaintext user api_token>&app_token=<plaintext app_token> → session_token; then Session-Token: + App-Token: headers. THE TRAP: GLPI 11 ENCRYPTS tokens at rest (are_apiclients_tokens_encrypted / are_users_tokens_encrypted, sodium XChaCha20, key /app/data/config/glpicrypt.key) and DECRYPTS before comparing — the ciphertext you see in glpi_users.api_token / glpi_apiclients.app_token NEVER authenticates. Plaintext values: app_token = creds/glpi password; glpi super-admin api_token = creds/glpi username; per-user tokens are vaulted in each identity's master item (glpi_api_token). To mint: openssl rand -hex 20, encrypt with GLPIKey('/app/data/config')->encrypt() inside the container, write back via SQL (plaintext legacy rows never authenticate). Create accounts via REST as the glpi super-admin (POST /User with name/email matching the Cloudron identity) — the SSO plugin's auto-register is broken on GLPI 11 (#947).

Store each key in the vault item immediately (schema below), and always verify the write (sm get … --field … readback) — sm setfield 400s ("Data missing") on some ciphers; fall back to a full item recreate that PRESERVES every existing field (username, password, uri, totp_seed, tokens).

6. Vault item schema — creds/<login>

One item per identity (NOT per system), created as a LOGIN item (never a note — sm set makes notes; sm convert <name> converts, but REGENERATE the password via the admin API afterwards because convert drops field values):

• login username = Cloudron username; password = the unique Cloudron password
• URI = https://my.knownelement.com
• fields: email, organization, role, reports_to, created, services, plus per-system: redmine_api_key, gitea_token, discourse_user_api_key, glpi_user_token, and totp_seed

Historical items that predate this schema (e.g. sec-*, sre-* shared- password items) are remediation targets under #942 — upgrade them when the owner lane assigns the work, do not replicate their shape.

7. Update the org chart (MANDATORY — an uncharted identity is an incomplete task)

After provisioning changes the identity estate:

1. Update the org table in TSYSGroupCorporate/TSYSGroupAIOS agents/global/AGENTS.md (branch + PR per the house-rules flow; the symlink serves it live once merged).
2. Update reachableceo/org-buildout docs/org-chart.md (the rendered chart) in the same turn.
3. Note the change on the provisioning ticket.

Details: references/org-chart-update.md.

8. Verify the trail

Prove each system knows the identity (expected: 200 + the identity's own name; a 200 from the wrong identity is a FAIL):

curl -sf -H "X-Redmine-API-Key: $KEY" https://projects.knownelement.com/users/current.json | jq .user.login
curl -sf -H "Authorization: token $TOKEN" https://git.knownelement.com/api/v1/user | jq .login

Then update the Redmine ticket (full URLs, [agent] <your-identity> note) and the workspace STATUS/memory.

References — read before touching the corresponding system

• references/cloudron.md — admin API routes, OIDC interaction flow, 2FA.
• references/systems.md — per-system SSO/API details and known breakage.
• references/vault.md — sm CLI entry points, gotchas, fallbacks.
• references/org-chart-update.md — the mandatory post-provisioning chart sync.
• references/incident-log.md — what went wrong on 2026-09-07/08 and why, so the same failures are not re-derived.

Scripts

• scripts/cloudron-api.sh METHOD PATH [JSON] — authenticated admin call.
• scripts/cloudron-oidc-login.sh <username> [vault-item] — full OIDC dashboard login (PKCE + interaction + consent); Bearer at /tmp/.oidc-bearer-<username>.
• scripts/cloudron-totp-enroll.sh <username> [vault-item] — enroll TOTP (Cloudron 10 profile routes: POST /api/v1/profile/totp_secret + /api/v1/profile/totp_enable), capture the seed to the vault (setfield → verify → full-recreate fallback), verify end-to-end.
• scripts/gitea-sso-login.sh <username> [vault-item] — Gitea SSO first login.
• scripts/redmine-sso-login.sh <username> [vault-item] — Redmine SSO first login.
• scripts/discourse-sso-login.sh <username> [vault-item] — Discourse SSO first login.
• scripts/totp.sh <base32-seed> — current 6-digit TOTP code (pure bash, RFC-7636-vector tested).

Vaultwarden app (pwvault.turnsys.com) — known issue, quirks, and repair (2026-09-09)

The fleet vault is a Cloudron Vaultwarden app (com.github.bitwardenrs, container id 48757d8a-… on tsys-cloudron-new; DB = MySQL database from the app container env CLOUDRON_MYSQL_DATABASE, on the shared mysql container — NOT sqlite; a stray db.sqlite3 in /app/data is junk).

Known web-vault crash (upstream bug, Vaultwarden discussions #7334/#7464): the 2026.7.0 web client + strict WASM SDK crash with Unhandled error in angular: EncString(InvalidTypeSymm { enc_type: "0", parts: 1 }) during vault load → empty vault / endless spinner, on every browser, while native apps (iPad/desktop) and API-key clients work fine. Triggered by an account key-pair record the client's strict parser rejects (accountKeys.publicKeyEncryptionKeyPair.publicKey stored as a raw base64 public key instead of an EncString). Repair that worked: UPDATE users SET public_key = NULL WHERE uuid = '<uid>'; in the MySQL DB (take a Cloudron app backup first: POST /api/v1/apps/<id>/backup with {"backupSiteId":"<id from GET /api/v1/backup_sites>"}, poll /api/v1/tasks/<taskId>); the client regenerates a proper key pair on next login. NEVER write to the DB without a fresh backup + explicit founder approval for the pwvault container.

CLI 2FA quirk: CLI-style logins (client_id=cli) are offered ONLY the recovery-code provider — an authenticator TOTP seed from the vault will be rejected at the 2FA prompt even when the authenticator is correctly enrolled (browser sessions get the authenticator prompt; CLI does not). Automated flows must use the API key (client_credentials grant), which bypasses 2FA entirely — creds/cloudron-style API keys are the automation path, passwords+TOTP are for humans in browsers.

Cloudron backup API pattern (for pre-surgery backups): GET /api/v1/backup_sites → site id; POST /api/v1/apps/<id>/backup body {"backupSiteId":"<id>"}; poll GET /api/v1/tasks/<taskId> until success:true. Response field is taskId (not id).

TOTP seeds: the account's authenticator seed for coo@turnsys.com is BW_TOTP_SECRET in /data2/TSGCOO/.config/bw/env — keep it in sync with what is actually enrolled server-side; a stale seed there was the source of a multi-hour misdiagnosis (the "wrong password"/"stale seed" theories). Clock skew was ruled out (workstation and server within 1s).