ops(archive): TASK prompts and REPORT verifications, studies through V1

2026-08-29 05:22:43 -05:00
parent bd0cea1757
commit b2c4b74d76
37 changed files with 2049 additions and 0 deletions
@@ -0,0 +1,54 @@
+# TASK: Quota monitoring + back-pressure + usage mgmt + resource gate (Redmine 490+491)
+
+## Context
+Charles's spec of record: `docs/SPEC-20260829-charles-brief.md` (READ IT
+FIRST, esp. "It's a marathon" + Roadmap 1). Last night the z.ai quota wall
+killed two dispatched turns at 19:00 — this system prevents that class of
+failure and maximizes the flat-rate plan 24x7 (legacy plan ends 9/22).
+Redmine tickets: 490 (quota) + 491 (resource monitor).
+
+## Scope — `internal/quota` + `harness quota` + loop integration
+1. **z.ai usage polling**: research the z.ai coding-plan usage/limits
+   endpoint (it exists per Charles; find the API surface from public docs
+   + the z.ai provider behavior we already see). Poll on interval; parse
+   buckets/limits/reset times; expose `QuotaSnapshot` (struct: per-bucket
+   used/limit/window-reset).
+   - Auth via key_ref (never logged, same discipline as keyproxy).
+   - If the endpoint proves unreachable/undocumented: build against a
+     documented interface + fake server, flag LIVE VERIFICATION as open.
+2. **Central state decision** (Charles leaves it to us): Redis container
+   vs LiteLLM-native (postgres spend tracking already live at
+   192.168.3.78:4001). Decide with rationale in the REPORT; implement the
+   chosen one. Requirements: multiple harness instances (9 accounts, 2
+   hosts) share quota state; redis (if chosen) runs as a docker container
+   on this host, config in harness.toml, no host packages.
+3. **Back-pressure in the loop** (internal/loop): before dispatching a
+   turn, consult quota snapshot + schedule:
+   - Peak window (config, default 0100-0500 CST — VERIFY empirically if
+     possible; Charles sleeps 2300-0500): restrict to flash-tier classes;
+     defer heavy classes with a logged reason; prefer LLM-lite work.
+   - Quota nearly exhausted: defer + surface status; never hard-fail the
+     loop.
+   - Schedule + thresholds all in `[quota]` harness.toml section (TZ-aware,
+     CST default).
+4. **Resource gate (ticket 491)**: read-only system monitor (load avg,
+     mem available, disk free, IO delay from /proc + /sys) with
+     configurable busy thresholds; loop defers dispatch when busy. cgroup
+     enforcement is deploy-time — document the runbook section only.
+5. **Usage accounting**: per vertical/class token+model accounting into
+     the JSONL state (extend loop state) — feeds the eventual
+     per-instance Discourse usage reports.
+6. TDD red/green: quota parser, schedule logic (peak/offpeak across TZ
+   edges), back-pressure decisions, resource thresholds — table-driven +
+   fake clock. Docker dev discipline. Docs standard (README + config
+   table). Push to origin main.
+
+## Deliverable
+`REPORT-20260829-0500-quota.md` in `~/.coordinate/inbox-pmo/`: the Redis
+vs LiteLLM decision + rationale, endpoint findings, config surface, test
+results, and how the loop now behaves at quota exhaustion (the 19:00-wall
+scenario replayed as a test).
+
+## ADDENDUM (redispatch, ~05:10 CST)
+- Previous attempt died on a sourcegraph.com timeout — do NOT use sourcegraph; research the z.ai usage endpoint via direct fetch of z.ai docs / the API itself.
+- BE QUOTA-LEAN (weekly bucket at 93%): minimal exploration, write code in large correct chunks, one test pass, no gold-plating, finish and report.