KNELAgentIdentityProvisioning/skills/agent-provisioning/SKILL.md

---
name: agent-provisioning
description: >
  Provision TSYS Group agent identities end-to-end: Cloudron account (unique
  password, TOTP with seed captured to the TSGCOO vault), SSO first-login into
  Redmine / Gitea / Discourse / GLPI, per-system API key minting, vault item
  schema, and verification. Use whenever asked to "stand up" an identity or
  agent, create/fix Cloudron users, wire SSO, debug identity logins or 2FA,
  or store identity credentials — even if the word "provisioning" is not used.
  Policy-critical: encodes founder rulings #942 (unique passwords, Cloudron-only
  accounts, no reachableceo credentials).
---

# Agent identity provisioning (TSYS Group)

Stand up an agent identity so it can operate its own accounts in Cloudron and
the four SSO-wired systems, with every credential landing in the TSGCOO
Bitwarden vault. The Cloudron is the ONLY identity source: never create local
accounts in Redmine, Gitea, Discourse, or GLPI — the system account comes into
existence when the identity completes its first SSO login there.

## Non-negotiable policy (founder rulings — violating these invalidates the work)

1. **Unique password per identity.** One identity, one password, used only for
   its Cloudron account. Shared passwords are banned (Redmine #942).
2. **Cloudron-only accounts.** System access = SSO first-login. No local
   accounts, no admin-created users in the downstream systems.
3. **Never use reachableceo credentials.** Items like `creds/gitea-admin`,
   `creds/gitea`, `creds/redmine` are the founder's personal accounts and
   carry RESTRICTED notes. If an auth test resolves to user `reachableceo`,
   stop — you are holding the founder's identity.
4. **TOTP seeds MUST be stored in Bitwarden** (founder ruling 2026-09-08).
   Enroll TOTP and write the base32 seed to the vault item in the same
   breath — a seed that is not in the vault is treated as lost, because
   there is no admin 2FA reset on this Cloudron and no other record exists.
5. **Secrets live only in the vault** (KNELSecretsManager `sm` CLI) — never in
   repos, command output, logs, or transcripts. Mask credentials in every
   command's visible output.
6. **Bookkeeping in the same turn:** Redmine ticket on the provisioning
   program (#942 or its successor), cross-linked; vault item created;STATUS
   updated.

## Scope check

Identity stand-up is the COO C2 lane's assigned program (founder directive,
2026-09-08). If your chat runs in another lane (SRE, security, OAM), confirm
the founder assigned the stand-up to you before executing, or route a ticket
to the COO C2 lane instead.

## Admin lever: Cloudron API token

`creds/cloudron` holds a working admin API token (service user `api-token`,
64 chars). Use it as `Authorization: Bearer $PASSWORD`. Do NOT log into the
dashboard interactively for admin work — the token is the compliant path.

```bash
scripts/cloudron-api.sh GET /users?limit=100     # list users (note uid- UUIDs)
scripts/cloudron-api.sh GET  /users/<uid>        # fetch one user object
```

Route notes (this Cloudron, 2026-09): users are addressed by `uid-…` UUIDs,
not usernames; there is NO `/api/v1/login` (dashboard auth is its own OIDC
flow, client `cid-webadmin`, PKCE S256); admin CAN clear a user's TOTP via
`POST /api/v1/users/<uid>/totp_disable` (empty JSON body) — use it when a
seed was lost, then re-enroll with capture. Admin password set:
`POST /api/v1/users/<uid>/password {"password":"…"}`.

## Procedure

### 0. Prerequisites

- Vault reachable: `sm get creds/cloudron` works (see references/vault.md).
- For a NEW Cloudron user: the founder generates the invite, or you create the
  user via the admin token if the founder has authorized self-service.
- A Redmine ticket to record the work (create one if none exists).

### 1. Cloudron account (unique password)

Verify the account exists and matches the identity email pattern
(`tsgstaff-<login>@turnsys.com` for COO-line identities). Generate the unique
password (32 chars), set it via the admin API, and store it in the vault item
login password field:

```bash
scripts/cloudron-api.sh POST /users/<uid>/password '{"password":"<32-char>"}'
sm setfield "<login> Cloudron" password "<32-char>"
```

### 2. TOTP 2FA — enroll and CAPTURE THE SEED

Two cases:

- **Seed lost / never captured** (TOTP enforced, no `totp_seed` in vault):
  clear first — `scripts/cloudron-api.sh POST /users/<uid>/totp_disable` —
  then proceed to enroll fresh.
- **No 2FA yet**: enroll directly.

Enrollment runs in the identity's own dashboard session (the totp routes are
self-scoped — an admin token gets 404s there):

1. `scripts/cloudron-oidc-login.sh <username> <vault-item>` — full OIDC
   login (PKCE + interaction + consent) yielding the identity's Bearer at
   `/tmp/.oidc-bearer-<username>`.
2. `scripts/cloudron-totp-enroll.sh <username> <vault-item>` — reads the
   pending secret, enables with a computed code, stores the seed via
   `sm setfield`, and verifies end-to-end.

If the API confirm path is rejected, fall back to the dashboard UI in the
browser: the forced "Enable TOTP" dialog shows the seed; read it, compute
with `scripts/totp.sh`, fill the Token field and submit. CRITICAL: the
pending secret REGENERATES every time the dialog re-renders — always read
the seed and submit the code within the same window, and verify enrollment
by a real TOTP login (a totpToken in a login is silently ignored for users
without 2FA — "success" there proves nothing).

If the identity already has 2FA enforced and no seed exists anywhere: STOP and
ask the founder. There is no programmatic recovery (verified 2026-09-08).
Compute codes offline when needed: `scripts/totp.sh <seed>`.

### 3. TOTP enrollment (in the identity's dashboard session)

```bash
scripts/cloudron-oidc-login.sh <username> <vault-item>   # OIDC login -> Bearer
scripts/cloudron-totp-enroll.sh <username> <vault-item>  # enroll + seed + verify
```

If the API enable path is rejected, fall back to the dashboard UI dialog (see
step 2's note). The verify pass inside the scripts performs a real TOTP login
and is the ONLY trustworthy success signal.

### 4. SSO first-login into each system

The interactive login is fully scriptable per app — use the per-app scripts;
each runs the whole flow (entry → Cloudron interaction → TOTP from the vault
seed → resume → consent → callback) with a cookie jar and verifies a live
session:

```bash
scripts/gitea-sso-login.sh     <username> [vault-item]   # /user/oauth2/cloudron
scripts/redmine-sso-login.sh   <username> [vault-item]   # /oauth?oauth_provider=1
scripts/discourse-sso-login.sh <username> [vault-item]   # /auth/oidc (+ CSRF + confirm form)
```

| System | SSO start point | Gotchas |
|---|---|---|
| GLPI | `https://cmdb.knownelement.com/plugins/singlesignon/front/callback.php/provider/1?remember=1` | account auto-provisioned; no scripted flow yet |
| Gitea | `https://git.knownelement.com/user/oauth2/cloudron` | redirects to `client_id=<app-uuid>-oidc`; gitea may NORMALIZE the username (e.g. `tsgstaff-coo-vpperf` → `vpperf`) — verify which login got created |
| Discourse | `https://community.turnsys.com/auth/oidc` | hostname is community.**turnsys.com**; provider name is `oidc`, NOT openid_connect; needs `/session/csrf` token then a confirm-page authenticity_token POST |
| Redmine | `https://projects.knownelement.com/oauth?oauth_provider=1` | the "bounce to /login" mystery = the plugin's GET form needs `oauth_provider=1` |

A failed session-verify can still mean the ACCOUNT was created (common) —
check via the admin API before retrying. App-level access control can deny
the whole flow with `access_denied: User has no access to this app`: the
Cloudron app's `accessRestriction` (per-user/group allow-list) must include
the identity. Set it with
`POST /api/v1/apps/<app-id>/configure/access_restriction` body
`{"accessRestriction":{"users":[uids…],"groups":[gids…]}}` — the `groups`
array is REQUIRED even when empty. Gitea (1c91ff45-…) and Discourse
(b506a0c1-…) both carry allow-lists.

Key gotcha (cost a debugging session — do not rediscover): the OIDC
interaction login is `POST /openid/interaction/<uid>/login` with JSON body
`{"username": "...", "password": "...", "totpToken": "..."}` — the field is
**`username`**, not `login`, and the response is `{"redirectTo": ...}` on
success (absolute URL — never concatenate a base onto it).

### 5. Mint per-system API keys

vptechops holds ADMIN in all four systems (2026-09-09), so keys are minted
centrally — no per-identity session needed:

- **Gitea** (admin basic auth; the token route REQUIRES basic auth — a token
  works as the basic password):
  `POST /api/v1/users/<login>/tokens -u "vptechops:$admin_token"` with
  `{"name":"agent-<date>","scopes":["all"]}` → `.sha1`. Minting is scoped to
  the authenticating token's scopes, so use the full-scope admin token
  (`creds/vptechops-gitea` `admin_token`, generated via
  `gitea admin user generate-access-token` in the container with
  `--config /run/gitea/app.ini`).
- **Redmine**: api_key is not in `GET /users/:id.json` for others — dump via
  rails runner in the app container (`User.where(status:1).each { |u| puts
  "#{u.login}|#{u.api_key}" }`; needs `HOME=/tmp`, `SECRET_KEY_BASE` from
  `/app/data/secrets.yml`, rbenv shims in PATH, `bundle exec rails runner`).
- **Discourse**: rails runner `ApiKey.create!(user_id:, description:,
  created_by_id:)` per user — `.key` readable only at creation. Note:
  Discourse usernames can differ from Cloudron usernames (legacy
  `coo1coo`, prefix-stripped `vpengops`…) — resolve via
  `/admin/users/list/active.json` first. Admin Api-Key minting uses the same
  model; admin key for vptechops is in `creds/vptechops-discourse`
  (`admin_api_key`).
- **GLPI**: DEFERRED on #947 — REST `user_token` auth 401s even for a
  super-admin with api_token/personal_token set; do not burn time re-deriving
  that failure.

Store each key in the vault item immediately (schema below), and always
verify the write (`sm get … --field …` readback) — `sm setfield` 400s
("Data missing") on some ciphers; fall back to a full item recreate that
PRESERVES every existing field (username, password, uri, totp_seed, tokens).

### 6. Vault item schema — `creds/<login>`

One item per identity (NOT per system), created as a LOGIN item (never a
note — `sm set` makes notes; `sm convert <name>` converts, but REGENERATE
the password via the admin API afterwards because convert drops field
values):

- login username = Cloudron username; password = the unique Cloudron password
- URI = `https://my.knownelement.com`
- fields: `email`, `organization`, `role`, `reports_to`, `created`,
  `services`, plus per-system: `redmine_api_key`, `gitea_token`,
  `discourse_user_api_key`, `glpi_user_token`, and `totp_seed`

Historical items that predate this schema (e.g. `sec-*`, `sre-*` shared-
password items) are remediation targets under #942 — upgrade them when the
owner lane assigns the work, do not replicate their shape.

### 7. Update the org chart (MANDATORY — an uncharted identity is an incomplete task)

After provisioning changes the identity estate:

1. Update the org table in `TSYSGroupCorporate/TSYSGroupAIOS`
   `agents/global/AGENTS.md` (branch + PR per the house-rules flow; the
   symlink serves it live once merged).
2. Update `reachableceo/org-buildout` `docs/org-chart.md` (the rendered
   chart) in the same turn.
3. Note the change on the provisioning ticket.

Details: `references/org-chart-update.md`.

### 8. Verify the trail

Prove each system knows the identity (expected: 200 + the identity's own
name; a 200 from the wrong identity is a FAIL):

```bash
curl -sf -H "X-Redmine-API-Key: $KEY" https://projects.knownelement.com/users/current.json | jq .user.login
curl -sf -H "Authorization: token $TOKEN" https://git.knownelement.com/api/v1/user | jq .login
```

Then update the Redmine ticket (full URLs, `[agent] <your-identity>` note) and
the workspace STATUS/memory.

## References — read before touching the corresponding system

- `references/cloudron.md` — admin API routes, OIDC interaction flow, 2FA.
- `references/systems.md` — per-system SSO/API details and known breakage.
- `references/vault.md` — sm CLI entry points, gotchas, fallbacks.
- `references/org-chart-update.md` — the mandatory post-provisioning chart sync.
- `references/incident-log.md` — what went wrong on 2026-09-07/08 and why, so
  the same failures are not re-derived.

## Scripts

- `scripts/cloudron-api.sh METHOD PATH [JSON]` — authenticated admin call.
- `scripts/cloudron-oidc-login.sh <username> [vault-item]` — full OIDC
  dashboard login (PKCE + interaction + consent); Bearer at
  `/tmp/.oidc-bearer-<username>`.
- `scripts/cloudron-totp-enroll.sh <username> [vault-item]` — enroll TOTP
  (Cloudron 10 profile routes: POST `/api/v1/profile/totp_secret` +
  `/api/v1/profile/totp_enable`), capture the seed to the vault
  (setfield → verify → full-recreate fallback), verify end-to-end.
- `scripts/gitea-sso-login.sh <username> [vault-item]` — Gitea SSO first
  login.
- `scripts/redmine-sso-login.sh <username> [vault-item]` — Redmine SSO first
  login.
- `scripts/discourse-sso-login.sh <username> [vault-item]` — Discourse SSO
  first login.
- `scripts/totp.sh <base32-seed>` — current 6-digit TOTP code (pure bash,
  RFC-7636-vector tested).

## Vaultwarden app (pwvault.turnsys.com) — known issue, quirks, and repair (2026-09-09)

The fleet vault is a Cloudron **Vaultwarden** app (`com.github.bitwardenrs`,
container id 48757d8a-… on tsys-cloudron-new; DB = **MySQL** database from
the app container env `CLOUDRON_MYSQL_DATABASE`, on the shared `mysql`
container — NOT sqlite; a stray `db.sqlite3` in /app/data is junk).

**Known web-vault crash (upstream bug, Vaultwarden discussions #7334/#7464):**
the 2026.7.0 web client + strict WASM SDK crash with
`Unhandled error in angular: EncString(InvalidTypeSymm { enc_type: "0", parts: 1 })`
during vault load → empty vault / endless spinner, on every browser, while
native apps (iPad/desktop) and API-key clients work fine. Triggered by an
account key-pair record the client's strict parser rejects
(`accountKeys.publicKeyEncryptionKeyPair.publicKey` stored as a raw base64
public key instead of an EncString). Repair that worked:
`UPDATE users SET public_key = NULL WHERE uuid = '<uid>';` in the MySQL DB
(take a Cloudron app backup first: `POST /api/v1/apps/<id>/backup` with
`{"backupSiteId":"<id from GET /api/v1/backup_sites>"}`, poll
`/api/v1/tasks/<taskId>`); the client regenerates a proper key pair on next
login. NEVER write to the DB without a fresh backup + explicit founder
approval for the pwvault container.

**CLI 2FA quirk:** CLI-style logins (`client_id=cli`) are offered ONLY the
recovery-code provider — an authenticator TOTP seed from the vault will be
rejected at the 2FA prompt even when the authenticator is correctly
enrolled (browser sessions get the authenticator prompt; CLI does not).
Automated flows must use the **API key** (client_credentials grant), which
bypasses 2FA entirely — `creds/cloudron`-style API keys are the automation
path, passwords+TOTP are for humans in browsers.

**Cloudron backup API pattern** (for pre-surgery backups): GET
`/api/v1/backup_sites` → site id; POST `/api/v1/apps/<id>/backup` body
`{"backupSiteId":"<id>"}`; poll `GET /api/v1/tasks/<taskId>` until
`success:true`. Response field is `taskId` (not `id`).

**TOTP seeds**: the account's authenticator seed for coo@turnsys.com is
`BW_TOTP_SECRET` in `/data2/TSGCOO/.config/bw/env` — keep it in sync with
what is actually enrolled server-side; a stale seed there was the source of
a multi-hour misdiagnosis (the "wrong password"/"stale seed" theories).
Clock skew was ruled out (workstation and server within 1s).