55 lines
3.2 KiB
Markdown
55 lines
3.2 KiB
Markdown
# TASK: Quota monitoring + back-pressure + usage mgmt + resource gate (Redmine 490+491)
|
|
|
|
## Context
|
|
Charles's spec of record: `docs/SPEC-20260829-charles-brief.md` (READ IT
|
|
FIRST, esp. "It's a marathon" + Roadmap 1). Last night the z.ai quota wall
|
|
killed two dispatched turns at 19:00 — this system prevents that class of
|
|
failure and maximizes the flat-rate plan 24x7 (legacy plan ends 9/22).
|
|
Redmine tickets: 490 (quota) + 491 (resource monitor).
|
|
|
|
## Scope — `internal/quota` + `harness quota` + loop integration
|
|
1. **z.ai usage polling**: research the z.ai coding-plan usage/limits
|
|
endpoint (it exists per Charles; find the API surface from public docs
|
|
+ the z.ai provider behavior we already see). Poll on interval; parse
|
|
buckets/limits/reset times; expose `QuotaSnapshot` (struct: per-bucket
|
|
used/limit/window-reset).
|
|
- Auth via key_ref (never logged, same discipline as keyproxy).
|
|
- If the endpoint proves unreachable/undocumented: build against a
|
|
documented interface + fake server, flag LIVE VERIFICATION as open.
|
|
2. **Central state decision** (Charles leaves it to us): Redis container
|
|
vs LiteLLM-native (postgres spend tracking already live at
|
|
192.168.3.78:4001). Decide with rationale in the REPORT; implement the
|
|
chosen one. Requirements: multiple harness instances (9 accounts, 2
|
|
hosts) share quota state; redis (if chosen) runs as a docker container
|
|
on this host, config in harness.toml, no host packages.
|
|
3. **Back-pressure in the loop** (internal/loop): before dispatching a
|
|
turn, consult quota snapshot + schedule:
|
|
- Peak window (config, default 0100-0500 CST — VERIFY empirically if
|
|
possible; Charles sleeps 2300-0500): restrict to flash-tier classes;
|
|
defer heavy classes with a logged reason; prefer LLM-lite work.
|
|
- Quota nearly exhausted: defer + surface status; never hard-fail the
|
|
loop.
|
|
- Schedule + thresholds all in `[quota]` harness.toml section (TZ-aware,
|
|
CST default).
|
|
4. **Resource gate (ticket 491)**: read-only system monitor (load avg,
|
|
mem available, disk free, IO delay from /proc + /sys) with
|
|
configurable busy thresholds; loop defers dispatch when busy. cgroup
|
|
enforcement is deploy-time — document the runbook section only.
|
|
5. **Usage accounting**: per vertical/class token+model accounting into
|
|
the JSONL state (extend loop state) — feeds the eventual
|
|
per-instance Discourse usage reports.
|
|
6. TDD red/green: quota parser, schedule logic (peak/offpeak across TZ
|
|
edges), back-pressure decisions, resource thresholds — table-driven +
|
|
fake clock. Docker dev discipline. Docs standard (README + config
|
|
table). Push to origin main.
|
|
|
|
## Deliverable
|
|
`REPORT-20260829-0500-quota.md` in `~/.coordinate/inbox-pmo/`: the Redis
|
|
vs LiteLLM decision + rationale, endpoint findings, config surface, test
|
|
results, and how the loop now behaves at quota exhaustion (the 19:00-wall
|
|
scenario replayed as a test).
|
|
|
|
## ADDENDUM (redispatch, ~05:10 CST)
|
|
- Previous attempt died on a sourcegraph.com timeout — do NOT use sourcegraph; research the z.ai usage endpoint via direct fetch of z.ai docs / the API itself.
|
|
- BE QUOTA-LEAN (weekly bucket at 93%): minimal exploration, write code in large correct chunks, one test pass, no gold-plating, finish and report.
|