REPORT — Quota monitoring + back-pressure + usage mgmt + resource gate (Redmine 490+491)

When: 2026-08-29 ~05:00-05:50 CST
Repo: projects/meta/MOPAC/harness @ fc518c4 (pushed to origin main)
Spec: docs/SPEC-20260829-charles-brief.md — "It's a marathon" + Roadmap 1
Gate: ./dev.sh check (build + vet + test) clean, all packages ok (config, quota, loop, serve, events, intake, llm, models, tools, writeback)

1. Decision: Redis container (not LiteLLM-native)

Chosen: one Redis container, spoken to by a stdlib RESP2 mini-client.

Rationale:

LiteLLM tracks dollars, z.ai meters credits. The thing we must share is the coding-plan 5-hour + weekly CREDIT buckets with peak multipliers (input x6.9 + cached x1.7 + output x24 per 10k tokens; flash 2.3/0.56/8; off-peak 50% off). LiteLLM's budget system is per-virtual-key $ spend against budgets it enforces itself — a lossy double-accounting that breaks on every plan change and cannot express "two harness accounts share one z.ai key".
We own the write path anyway. With no z.ai usage endpoint (see §2), consumption is OUR estimate; a store only we write to is the natural fit. Doing it "in LiteLLM" would mean SQL against another project's schema — an upgrade hazard, not an integration.
9 instances, 2 hosts, one account each: TTL'd rolling-window counters (INCRBYFLOAT + EXPIRE), snapshot publish/read, atomic cross-instance — this is exactly Redis. Postgres via LiteLLM gives us none of those primitives for free.
Charles's constraints honored: docker container only (redis:7-alpine, runbook in README; bind to 192.168.3.78:6390), config in harness.toml (redis_url), zero host packages, zero Go dependencies (the repo remains stdlib-only; the RESP2 client is ~150 lines and fake-server tested).
Fail-soft: redis down = this instance's local estimate, loop keeps running. LiteLLM-native would couple the loop's health to the proxy's DB.

Keys: mopac:quota:<account>:{snapshot, est:5h:<window>, est:weekly:<week>}. LiteLLM $-spend budget keys remain a separate, still-open item (README status table says so honestly).

2. z.ai usage endpoint findings (LIVE VERIFICATION: OPEN)

Probed 2026-08-29 ~05:05 with the real plan key (from the litellm container env; never logged, never written to disk):

GET /api/coding/paas/v4/{usage,limits,info,quota,credit/usage,...} -> 404 (JSON 404 from the gateway, so the route does not exist)
/api/coding/paas/v4 (the live chat endpoint LiteLLM uses) is real and serving — the 404s are route-level, not auth or reachability.
/api/coding/{user/info,usage} -> 200 with {"code":500,"msg":"404 NOT_FOUND"} — gateway routes, inner services absent.
Docs sweep (docs.z.ai llms.txt + devpack pages): the plan DOCS the buckets in detail (5h + weekly credits per tier, multipliers, off-peak discount, reset rules) and points humans at the billing web page for consumption — no public usage REST route is documented.

Per the task's fallback: the parser targets the documented bucket model (strict decode; unknown shapes error so an HTML error page can never read as "quota fine"; alias-tolerant field names), is table-driven tested against a fake server (bearer header asserted, key-leak asserted), and usage_url in [quota] flips it on the day z.ai ships one. Until then the gate runs on locally estimated consumption from the documented credit formula — which is exactly the number the loop controls anyway.

3. Peak window — Charles's guess verified

z.ai docs: peak = Mon-Fri 14:00-18:00 Singapore (UTC+8), off-peak 50% off. 14:00 SGT = 01:00 CST (winter, UTC-6) / 00:00 CDT (summer, UTC-5). So "0100 to 0500 CST" is correct for winter and one hour early during US DST — config carries the window + timezone + peak_weekdays_only (defaults 01:00-05:00 America/Chicago, weekdays only), and the report notes the March/November drift. Schedule logic is table-tested across the edges: window boundaries, weekend exclusion, overnight-wrap windows (weekday check on the window's start day), and zone-vs-instant semantics.

4. Config surface (see `harness.toml.example` + README table)

[quota]: enabled, account, plan_5h_credits (28000), plan_weekly_credits (140000), usage_url+key_ref+poll_interval_secs, defer_at_pct (85) / block_at_pct (95), peak_start/peak_end/ timezone/peak_weekdays_only, peak_classes (study, read), redis_url.
[resources]: enabled, max_load_avg (6), min_mem_available_mb (2048), min_disk_free_mb (5120), max_io_delay_pct (90), test seams proc_root/sys_root.
Both off by default (existing configs unchanged — test-asserted).
CLI: harness quota status | probe | gate.

5. How the loop behaves at quota exhaustion (the 19:00 wall, replayed)

TestLoopQuotaWallDefersAndRecovers (fake Redmine + fake LLM + fake usage endpoint at 97% weekly):

Scan sees the issue -> gate decides DEFER -> zero LLM calls, one stdout line + one "type":"defer" JSONL event with the reason ("quota: weekly bucket at 97% (>= block 95%): all classes deferred until reset").
The task is not consumed (no dispatch marker, no note) — defer is a throttle, so quota recovery needs no human touch.
Endpoint flips healthy -> next scan dispatches normally, notes the REPORT, status transitions. The 19:00-class failure no longer exists: pre-wall, the loop runs LLM-lite classes only past defer_at; at block_at everything pauses, surfaced, until reset.

Decision order: resources -> block wall -> peak-window class restriction -> soft-quota heavy deferral. Never a hard fail; unknown quota state is permissive.

6. Usage accounting (feeds the Discourse reports)

Every dispatched turn appends class, prompt/completion/total_tokens, and estimated credits to its report event in loop.jsonl; loopState rebuilds per-class totals from the JSONL at startup, and harness quota status renders the table (turns/tokens/credits per class + TOTAL). Also verified live in-container: real /proc reads (load 13.43, PSI io 0.3%, disk 44.9GB) and correct TZ evaluation.

7. Test results (TDD, table-driven, fake clock/servers)

internal/quota: parser (canonical + aliases + hostile bodies), schedule edges (TZ, wrap, weekdays), credit math (peak/off-peak, flash/flagship, unknown-model conservatism), decision levels (healthy / defer / block / peak interactions), estimate snapshots (redis + local), fake-redis RESP2 server round-trips, dead/nil-state fail-soft, resource fixtures (PSI present/absent, unreadable).
internal/loop: wall defer+recover, peak heavy-defer + flash-run + off-peak release, resource-busy defer, usage accounting JSONL + table, gate-off-by-default regression.
internal/config: [quota]/[resources] load + 7 validation-error cases; TOML float support.
Docker dev discipline throughout (./dev.sh only); host untouched.

8. Resource gate (491) + cgroup runbook

Read-only monitor (loadavg, MemAvailable, statfs disk free, /proc/pressure/io some-avg60 — skipped when PSI absent, read errors never defer). cgroup enforcement documented as deploy-time in the README runbook (--memory/--cpus/--pids-limit / systemd slice), per Charles's "docker + cgroups for all work" rule.

9. Open items

LIVE VERIFICATION open: z.ai usage endpoint (flip usage_url when it ships; parser + probe command ready).
Weekly bucket reset is approximated to Monday 00:00 plan-TZ until reset_at arrives from a real endpoint.
Turn cached_tokens not yet captured from LiteLLM responses (LLM client returns prompt/completion only) — credits currently conservative (cache discount not credited).
Redis container not yet deployed to 192.168.3.78 (runbook written; enabled = false defaults mean nothing regresses until then).
LiteLLM $-spend budget keys (separate from credit buckets) still phase 3.

— PMO worker, MOPAC harness self-host loop

7.9 KiB Raw Blame History