# REPORT: LiteLLM proxy deployed + RCEO pilot wired

Date: 2026-08-28 (turn TASK-20260828-1716-litellm)
Operator: RCEO-Work
Status: COMPLETE (one deviation, pre-approved — see "DB deviation")

## Containers

| name | image | status | port |
|---|---|---|---|
| litellm | ghcr.io/berriai/litellm-database:main-stable | Up, restart unless-stopped | **192.168.3.78:4001** -> 4000 |
| litellm-db | postgres:alpine | Up, restart unless-stopped | none (docker-internal only) |

Port note: host 4000 is taken (kneldevstack-supportstack-demo-homepage) -> used
4001 per task instruction. No existing container was stopped/restarted/touched.

## DB deviation (pre-approved fallback)

v2.2 specified "embedded SQLite". Upstream fact: LiteLLM has NEVER supported
SQLite for virtual keys/spend — Prisma schema is postgresql-pinned in every
release tag I checked (v1.53.7 Apr 2025, v1.63.14, v1.67.0, main); current
main-stable fails fast: "database features ... require PostgreSQL". No sqlite
code path exists in the installed package. Deployed the fallback PMO
pre-approved in STATE-SNAPSHOT-20260828-1720.md: "dedicated litellm-db postgres
container (standard minimal deploy, not overbuild)". Everything else per spec.
Full recipe was validated with throwaway containers first, then deployed.

## Virtual keys (8) — NAMES only, values are secrets

Values: /home/_crossfeed/tooling/credentials/litellm-virtual-keys.md (mode 600,
gitignored). Budgets are USD per 30d window, enforced at the proxy.

| key_alias | budget | scope |
|---|---|---|
| rceo | $50 / 30d | onstage pilot (this host, crush) — ACTIVE |
| bod | $30 / 30d | onstage BOD |
| cco | $30 / 30d | onstage CCO |
| cto | $30 / 30d | onstage CTO |
| coo | $30 / 30d | onstage COO |
| offstage-1 | $15 / 30d | offstage (capped lower per v2.2) |
| offstage-2 | $15 / 30d | offstage |
| offstage-3 | $15 / 30d | offstage |

All keys allow all 12 z.ai models (glm-5.3, 5.2, 5.1, 5-turbo, 5, 4.7,
4.7-flash, 4.6, 4.6v, 4.5, 4.5-air, 4.5v). Global hard cap: 3 concurrent
(config.yaml max_parallel_requests). Offstage hosts reach the proxy at
http://192.168.3.78:4001/v1 (OpenAI-compatible).

## Config / secrets layout

- /home/_crossfeed/tooling/litellm/config.yaml — 12 z.ai models via
  https://api.z.ai/api/coding/paas/v4 (OpenAI-compatible), master_key from env,
  per-token USD pricing mirrored from the crush catalog, drop_params on.
- /home/_crossfeed/tooling/credentials/litellm-secrets.env (600, gitignored) —
  LITELLM_MASTER_KEY, ZAI_API_KEY (real key; only place outside crush state),
  DATABASE_URL. Passed via --env-file, never in config or logs.
- /home/_crossfeed/tooling/credentials/litellm-db.env (600, gitignored) —
  postgres creds for litellm-db.
- /home/_crossfeed/tooling/litellm/data/pg — postgres data (persists across
  container recreation).

## RCEO pilot wiring

- Backups: /home/reachableceo/.local/share/crush/providers.json.pre-litellm.bak
  (task-required) and crush.json.pre-litellm.bak (companion; holds the direct
  z.ai key override).
- Actual wiring lives in /home/reachableceo/.config/crush/crush.json (600):
  custom provider "litellm" -> base_url http://192.168.3.78:4001/v1 with the
  rceo virtual key; models large=glm-5.2, small=glm-5-turbo.
- WHY not providers.json: crush auto-regenerates providers.json from its
  bundled catalog on startup (verified — my endpoint edit there was silently
  reverted, which caused a transient 401 during wiring). ~/.local/share/crush/
  is machine state; the durable override point is the config file.

## Verification (all redacted, no key values)

1. Headless crush via proxy: `crush run --quiet -m litellm/glm-5.2 "Reply with
   exactly: PROXY-OK..."` -> **PROXY-OK**; flagless default run -> "READY";
   proxy access log shows the corresponding `POST /v1/chat/completions 200`.
2. Usage/key-info API sample (GET /key/info, master auth):
   `{"key_alias":"rceo","spend_usd":2.035e-05,"max_budget_usd":50.0,
   "budget_duration":"30d","budget_reset_at":"2026-09-01T00:00:00+00:00"}`
   GET /global/spend/logs: `[{"date":"2026-08-28","spend":2.035e-05}]`
   -> per-key attribution + dollar spend recording confirmed end-to-end.

## Emergency fallback (restore direct z.ai)

1. cp -p ~/.local/share/crush/providers.json.pre-litellm.bak ~/.local/share/crush/providers.json
2. rm ~/.config/crush/crush.json   (removes proxy wiring; model reverts to zai)
   (crush.json.pre-litellm.bak also restorable if state was touched)
3. Optional teardown: docker rm -f litellm litellm-db
Direct z.ai key remains intact in crush state the whole time.

## Notes / caveats

- glm-5.2 is priced 0/0 in the crush catalog (coding-plan included), so its
  proxy spend tracks $0 by design; other models track real USD.
- Test artifacts from validation (throwaway containers/network/envs) fully
  removed; no residue. Images kept locally for fast redeploys.
- credentials/.gitignore extended with the three litellm secret filenames.