mopac-pmo/archive/REPORT-litellm.md

REPORT: LiteLLM proxy deployed + RCEO pilot wired

Date: 2026-08-28 (turn TASK-20260828-1716-litellm) Operator: RCEO-Work Status: COMPLETE (one deviation, pre-approved — see "DB deviation")

Containers

name	image	status	port
litellm	ghcr.io/berriai/litellm-database:main-stable	Up, restart unless-stopped	192.168.3.78:4001 -> 4000
litellm-db	postgres:alpine	Up, restart unless-stopped	none (docker-internal only)

Port note: host 4000 is taken (kneldevstack-supportstack-demo-homepage) -> used 4001 per task instruction. No existing container was stopped/restarted/touched.

DB deviation (pre-approved fallback)

v2.2 specified "embedded SQLite". Upstream fact: LiteLLM has NEVER supported SQLite for virtual keys/spend — Prisma schema is postgresql-pinned in every release tag I checked (v1.53.7 Apr 2025, v1.63.14, v1.67.0, main); current main-stable fails fast: "database features ... require PostgreSQL". No sqlite code path exists in the installed package. Deployed the fallback PMO pre-approved in STATE-SNAPSHOT-20260828-1720.md: "dedicated litellm-db postgres container (standard minimal deploy, not overbuild)". Everything else per spec. Full recipe was validated with throwaway containers first, then deployed.

Virtual keys (8) — NAMES only, values are secrets

Values: /home/_crossfeed/tooling/credentials/litellm-virtual-keys.md (mode 600, gitignored). Budgets are USD per 30d window, enforced at the proxy.

key_alias	budget	scope
rceo	$50 / 30d	onstage pilot (this host, crush) — ACTIVE
bod	$30 / 30d	onstage BOD
cco	$30 / 30d	onstage CCO
cto	$30 / 30d	onstage CTO
coo	$30 / 30d	onstage COO
offstage-1	$15 / 30d	offstage (capped lower per v2.2)
offstage-2	$15 / 30d	offstage
offstage-3	$15 / 30d	offstage

All keys allow all 12 z.ai models (glm-5.3, 5.2, 5.1, 5-turbo, 5, 4.7, 4.7-flash, 4.6, 4.6v, 4.5, 4.5-air, 4.5v). Global hard cap: 3 concurrent (config.yaml max_parallel_requests). Offstage hosts reach the proxy at http://192.168.3.78:4001/v1 (OpenAI-compatible).

Config / secrets layout

• /home/_crossfeed/tooling/litellm/config.yaml — 12 z.ai models via https://api.z.ai/api/coding/paas/v4 (OpenAI-compatible), master_key from env, per-token USD pricing mirrored from the crush catalog, drop_params on.
• /home/_crossfeed/tooling/credentials/litellm-secrets.env (600, gitignored) — LITELLM_MASTER_KEY, ZAI_API_KEY (real key; only place outside crush state), DATABASE_URL. Passed via --env-file, never in config or logs.
• /home/_crossfeed/tooling/credentials/litellm-db.env (600, gitignored) — postgres creds for litellm-db.
• /home/_crossfeed/tooling/litellm/data/pg — postgres data (persists across container recreation).

RCEO pilot wiring

• Backups: /home/reachableceo/.local/share/crush/providers.json.pre-litellm.bak (task-required) and crush.json.pre-litellm.bak (companion; holds the direct z.ai key override).
• Actual wiring lives in /home/reachableceo/.config/crush/crush.json (600): custom provider "litellm" -> base_url http://192.168.3.78:4001/v1 with the rceo virtual key; models large=glm-5.2, small=glm-5-turbo.
• WHY not providers.json: crush auto-regenerates providers.json from its bundled catalog on startup (verified — my endpoint edit there was silently reverted, which caused a transient 401 during wiring). ~/.local/share/crush/ is machine state; the durable override point is the config file.

Verification (all redacted, no key values)

1. Headless crush via proxy: crush run --quiet -m litellm/glm-5.2 "Reply with exactly: PROXY-OK..." -> PROXY-OK; flagless default run -> "READY"; proxy access log shows the corresponding POST /v1/chat/completions 200.
2. Usage/key-info API sample (GET /key/info, master auth): {"key_alias":"rceo","spend_usd":2.035e-05,"max_budget_usd":50.0, "budget_duration":"30d","budget_reset_at":"2026-09-01T00:00:00+00:00"} GET /global/spend/logs: [{"date":"2026-08-28","spend":2.035e-05}] -> per-key attribution + dollar spend recording confirmed end-to-end.

Emergency fallback (restore direct z.ai)

1. cp -p ~/.local/share/crush/providers.json.pre-litellm.bak ~/.local/share/crush/providers.json
2. rm ~/.config/crush/crush.json (removes proxy wiring; model reverts to zai) (crush.json.pre-litellm.bak also restorable if state was touched)
3. Optional teardown: docker rm -f litellm litellm-db Direct z.ai key remains intact in crush state the whole time.

Notes / caveats

• glm-5.2 is priced 0/0 in the crush catalog (coding-plan included), so its proxy spend tracks $0 by design; other models track real USD.
• Test artifacts from validation (throwaway containers/network/envs) fully removed; no residue. Images kept locally for fast redeploys.
• credentials/.gitignore extended with the three litellm secret filenames.