mopac-pmo/archive/REPORT-20260829-0500-quota.md

# REPORT — Quota monitoring + back-pressure + usage mgmt + resource gate (Redmine 490+491)

- **When**: 2026-08-29 ~05:00-05:50 CST
- **Repo**: `projects/meta/MOPAC/harness` @ `fc518c4` (pushed to origin main)
- **Spec**: `docs/SPEC-20260829-charles-brief.md` — "It's a marathon" + Roadmap 1
- **Gate**: `./dev.sh check` (build + vet + test) clean, all packages `ok`
  (config, quota, loop, serve, events, intake, llm, models, tools, writeback)

## 1. Decision: Redis container (not LiteLLM-native)

**Chosen: one Redis container, spoken to by a stdlib RESP2 mini-client.**

Rationale:

1. **LiteLLM tracks dollars, z.ai meters credits.** The thing we must
   share is the coding-plan 5-hour + weekly CREDIT buckets with peak
   multipliers (input x6.9 + cached x1.7 + output x24 per 10k tokens;
   flash 2.3/0.56/8; off-peak 50% off). LiteLLM's budget system is
   per-virtual-key $ spend against budgets it enforces itself — a lossy
   double-accounting that breaks on every plan change and cannot express
   "two harness accounts share one z.ai key".
2. **We own the write path anyway.** With no z.ai usage endpoint (see
   §2), consumption is OUR estimate; a store only we write to is the
   natural fit. Doing it "in LiteLLM" would mean SQL against another
   project's schema — an upgrade hazard, not an integration.
3. **9 instances, 2 hosts, one account each**: TTL'd rolling-window
   counters (`INCRBYFLOAT` + `EXPIRE`), snapshot publish/read, atomic
   cross-instance — this is exactly Redis. Postgres via LiteLLM gives
   us none of those primitives for free.
4. **Charles's constraints honored**: docker container only
   (`redis:7-alpine`, runbook in README; bind to 192.168.3.78:6390),
   config in `harness.toml` (`redis_url`), zero host packages, zero Go
   dependencies (the repo remains stdlib-only; the RESP2 client is
   ~150 lines and fake-server tested).
5. **Fail-soft**: redis down = this instance's local estimate, loop
   keeps running. LiteLLM-native would couple the loop's health to the
   proxy's DB.

Keys: `mopac:quota:<account>:{snapshot, est:5h:<window>, est:weekly:<week>}`.
LiteLLM $-spend budget keys remain a separate, still-open item (README
status table says so honestly).

## 2. z.ai usage endpoint findings (LIVE VERIFICATION: OPEN)

Probed 2026-08-29 ~05:05 with the real plan key (from the litellm
container env; never logged, never written to disk):

- `GET /api/coding/paas/v4/{usage,limits,info,quota,credit/usage,...}`
  -> **404** (JSON 404 from the gateway, so the route does not exist)
- `/api/coding/paas/v4` (the live chat endpoint LiteLLM uses) is real
  and serving — the 404s are route-level, not auth or reachability.
- `/api/coding/{user/info,usage}` -> 200 with `{"code":500,"msg":"404
  NOT_FOUND"}` — gateway routes, inner services absent.
- Docs sweep (`docs.z.ai` llms.txt + devpack pages): the plan DOCS the
  buckets in detail (5h + weekly credits per tier, multipliers,
  off-peak discount, reset rules) and points humans at the billing web
  page for consumption — **no public usage REST route is documented.**

Per the task's fallback: the parser targets the documented bucket model
(strict decode; unknown shapes error so an HTML error page can never
read as "quota fine"; alias-tolerant field names), is table-driven
tested against a fake server (bearer header asserted, key-leak
asserted), and `usage_url` in `[quota]` flips it on the day z.ai ships
one. Until then the gate runs on locally estimated consumption from the
documented credit formula — which is exactly the number the loop
controls anyway.

## 3. Peak window — Charles's guess verified

z.ai docs: **peak = Mon-Fri 14:00-18:00 Singapore (UTC+8), off-peak
50% off.** 14:00 SGT = 01:00 CST (winter, UTC-6) / 00:00 CDT (summer,
UTC-5). So "0100 to 0500 CST" is correct for winter and one hour early
during US DST — config carries the window + `timezone` + `peak_weekdays_only`
(defaults `01:00`-`05:00` America/Chicago, weekdays only), and the
report notes the March/November drift. Schedule logic is table-tested
across the edges: window boundaries, weekend exclusion, overnight-wrap
windows (weekday check on the window's start day), and zone-vs-instant
semantics.

## 4. Config surface (see `harness.toml.example` + README table)

- `[quota]`: `enabled`, `account`, `plan_5h_credits` (28000),
  `plan_weekly_credits` (140000), `usage_url`+`key_ref`+`poll_interval_secs`,
  `defer_at_pct` (85) / `block_at_pct` (95), `peak_start`/`peak_end`/
  `timezone`/`peak_weekdays_only`, `peak_classes` (study, read),
  `redis_url`.
- `[resources]`: `enabled`, `max_load_avg` (6), `min_mem_available_mb`
  (2048), `min_disk_free_mb` (5120), `max_io_delay_pct` (90), test seams
  `proc_root`/`sys_root`.
- Both **off by default** (existing configs unchanged — test-asserted).
- CLI: `harness quota status | probe | gate`.

## 5. How the loop behaves at quota exhaustion (the 19:00 wall, replayed)

`TestLoopQuotaWallDefersAndRecovers` (fake Redmine + fake LLM + fake
usage endpoint at 97% weekly):

1. Scan sees the issue -> gate decides DEFER -> **zero LLM calls**, one
   stdout line + one `"type":"defer"` JSONL event with the reason
   ("quota: weekly bucket at 97% (>= block 95%): all classes deferred
   until reset").
2. The task is **not consumed** (no dispatch marker, no note) — defer is
   a throttle, so quota recovery needs no human touch.
3. Endpoint flips healthy -> next scan dispatches normally, notes the
   REPORT, status transitions. The 19:00-class failure no longer
   exists: pre-wall, the loop runs LLM-lite classes only past
   `defer_at`; at `block_at` everything pauses, surfaced, until reset.

Decision order: resources -> block wall -> peak-window class
restriction -> soft-quota heavy deferral. Never a hard fail; unknown
quota state is permissive.

## 6. Usage accounting (feeds the Discourse reports)

Every dispatched turn appends `class`, `prompt/completion/total_tokens`,
and estimated `credits` to its `report` event in `loop.jsonl`;
`loopState` rebuilds per-class totals from the JSONL at startup, and
`harness quota status` renders the table (turns/tokens/credits per
class + TOTAL). Also verified live in-container: real `/proc` reads
(load 13.43, PSI io 0.3%, disk 44.9GB) and correct TZ evaluation.

## 7. Test results (TDD, table-driven, fake clock/servers)

- `internal/quota`: parser (canonical + aliases + hostile bodies),
  schedule edges (TZ, wrap, weekdays), credit math (peak/off-peak,
  flash/flagship, unknown-model conservatism), decision levels (healthy
  / defer / block / peak interactions), estimate snapshots (redis +
  local), fake-redis RESP2 server round-trips, dead/nil-state fail-soft,
  resource fixtures (PSI present/absent, unreadable).
- `internal/loop`: wall defer+recover, peak heavy-defer + flash-run +
  off-peak release, resource-busy defer, usage accounting JSONL +
  table, gate-off-by-default regression.
- `internal/config`: [quota]/[resources] load + 7 validation-error
  cases; TOML float support.
- Docker dev discipline throughout (`./dev.sh` only); host untouched.

## 8. Resource gate (491) + cgroup runbook

Read-only monitor (loadavg, MemAvailable, statfs disk free,
`/proc/pressure/io` some-avg60 — skipped when PSI absent, read errors
never defer). cgroup enforcement documented as deploy-time in the README
runbook (`--memory/--cpus/--pids-limit` / systemd slice), per Charles's
"docker + cgroups for all work" rule.

## 9. Open items

- **LIVE VERIFICATION open**: z.ai usage endpoint (flip `usage_url`
  when it ships; parser + probe command ready).
- Weekly bucket reset is approximated to Monday 00:00 plan-TZ until
  `reset_at` arrives from a real endpoint.
- Turn `cached_tokens` not yet captured from LiteLLM responses (LLM
  client returns prompt/completion only) — credits currently
  conservative (cache discount not credited).
- Redis container not yet deployed to 192.168.3.78 (runbook written;
  `enabled = false` defaults mean nothing regresses until then).
- LiteLLM $-spend budget keys (separate from credit buckets) still
  phase 3.

— PMO worker, MOPAC harness self-host loop