ops(archive): quota, deploy, pdf TASK/REPORT pairs verified+archived

archive/REPORT-20260829-0500-quota.md

@@ -0,0 +1,159 @@
+# REPORT — Quota monitoring + back-pressure + usage mgmt + resource gate (Redmine 490+491)
+
+- **When**: 2026-08-29 ~05:00-05:50 CST
+- **Repo**: `projects/meta/MOPAC/harness` @ `fc518c4` (pushed to origin main)
+- **Spec**: `docs/SPEC-20260829-charles-brief.md` — "It's a marathon" + Roadmap 1
+- **Gate**: `./dev.sh check` (build + vet + test) clean, all packages `ok`
+  (config, quota, loop, serve, events, intake, llm, models, tools, writeback)
+
+## 1. Decision: Redis container (not LiteLLM-native)
+
+**Chosen: one Redis container, spoken to by a stdlib RESP2 mini-client.**
+
+Rationale:
+
+1. **LiteLLM tracks dollars, z.ai meters credits.** The thing we must
+   share is the coding-plan 5-hour + weekly CREDIT buckets with peak
+   multipliers (input x6.9 + cached x1.7 + output x24 per 10k tokens;
+   flash 2.3/0.56/8; off-peak 50% off). LiteLLM's budget system is
+   per-virtual-key $ spend against budgets it enforces itself — a lossy
+   double-accounting that breaks on every plan change and cannot express
+   "two harness accounts share one z.ai key".
+2. **We own the write path anyway.** With no z.ai usage endpoint (see
+   §2), consumption is OUR estimate; a store only we write to is the
+   natural fit. Doing it "in LiteLLM" would mean SQL against another
+   project's schema — an upgrade hazard, not an integration.
+3. **9 instances, 2 hosts, one account each**: TTL'd rolling-window
+   counters (`INCRBYFLOAT` + `EXPIRE`), snapshot publish/read, atomic
+   cross-instance — this is exactly Redis. Postgres via LiteLLM gives
+   us none of those primitives for free.
+4. **Charles's constraints honored**: docker container only
+   (`redis:7-alpine`, runbook in README; bind to 192.168.3.78:6390),
+   config in `harness.toml` (`redis_url`), zero host packages, zero Go
+   dependencies (the repo remains stdlib-only; the RESP2 client is
+   ~150 lines and fake-server tested).
+5. **Fail-soft**: redis down = this instance's local estimate, loop
+   keeps running. LiteLLM-native would couple the loop's health to the
+   proxy's DB.
+
+Keys: `mopac:quota:<account>:{snapshot, est:5h:<window>, est:weekly:<week>}`.
+LiteLLM $-spend budget keys remain a separate, still-open item (README
+status table says so honestly).
+
+## 2. z.ai usage endpoint findings (LIVE VERIFICATION: OPEN)
+
+Probed 2026-08-29 ~05:05 with the real plan key (from the litellm
+container env; never logged, never written to disk):
+
+- `GET /api/coding/paas/v4/{usage,limits,info,quota,credit/usage,...}`
+  -> **404** (JSON 404 from the gateway, so the route does not exist)
+- `/api/coding/paas/v4` (the live chat endpoint LiteLLM uses) is real
+  and serving — the 404s are route-level, not auth or reachability.
+- `/api/coding/{user/info,usage}` -> 200 with `{"code":500,"msg":"404
+  NOT_FOUND"}` — gateway routes, inner services absent.
+- Docs sweep (`docs.z.ai` llms.txt + devpack pages): the plan DOCS the
+  buckets in detail (5h + weekly credits per tier, multipliers,
+  off-peak discount, reset rules) and points humans at the billing web
+  page for consumption — **no public usage REST route is documented.**
+
+Per the task's fallback: the parser targets the documented bucket model
+(strict decode; unknown shapes error so an HTML error page can never
+read as "quota fine"; alias-tolerant field names), is table-driven
+tested against a fake server (bearer header asserted, key-leak
+asserted), and `usage_url` in `[quota]` flips it on the day z.ai ships
+one. Until then the gate runs on locally estimated consumption from the
+documented credit formula — which is exactly the number the loop
+controls anyway.
+
+## 3. Peak window — Charles's guess verified
+
+z.ai docs: **peak = Mon-Fri 14:00-18:00 Singapore (UTC+8), off-peak
+50% off.** 14:00 SGT = 01:00 CST (winter, UTC-6) / 00:00 CDT (summer,
+UTC-5). So "0100 to 0500 CST" is correct for winter and one hour early
+during US DST — config carries the window + `timezone` + `peak_weekdays_only`
+(defaults `01:00`-`05:00` America/Chicago, weekdays only), and the
+report notes the March/November drift. Schedule logic is table-tested
+across the edges: window boundaries, weekend exclusion, overnight-wrap
+windows (weekday check on the window's start day), and zone-vs-instant
+semantics.
+
+## 4. Config surface (see `harness.toml.example` + README table)
+
+- `[quota]`: `enabled`, `account`, `plan_5h_credits` (28000),
+  `plan_weekly_credits` (140000), `usage_url`+`key_ref`+`poll_interval_secs`,
+  `defer_at_pct` (85) / `block_at_pct` (95), `peak_start`/`peak_end`/
+  `timezone`/`peak_weekdays_only`, `peak_classes` (study, read),
+  `redis_url`.
+- `[resources]`: `enabled`, `max_load_avg` (6), `min_mem_available_mb`
+  (2048), `min_disk_free_mb` (5120), `max_io_delay_pct` (90), test seams
+  `proc_root`/`sys_root`.
+- Both **off by default** (existing configs unchanged — test-asserted).
+- CLI: `harness quota status | probe | gate`.
+
+## 5. How the loop behaves at quota exhaustion (the 19:00 wall, replayed)
+
+`TestLoopQuotaWallDefersAndRecovers` (fake Redmine + fake LLM + fake
+usage endpoint at 97% weekly):
+
+1. Scan sees the issue -> gate decides DEFER -> **zero LLM calls**, one
+   stdout line + one `"type":"defer"` JSONL event with the reason
+   ("quota: weekly bucket at 97% (>= block 95%): all classes deferred
+   until reset").
+2. The task is **not consumed** (no dispatch marker, no note) — defer is
+   a throttle, so quota recovery needs no human touch.
+3. Endpoint flips healthy -> next scan dispatches normally, notes the
+   REPORT, status transitions. The 19:00-class failure no longer
+   exists: pre-wall, the loop runs LLM-lite classes only past
+   `defer_at`; at `block_at` everything pauses, surfaced, until reset.
+
+Decision order: resources -> block wall -> peak-window class
+restriction -> soft-quota heavy deferral. Never a hard fail; unknown
+quota state is permissive.
+
+## 6. Usage accounting (feeds the Discourse reports)
+
+Every dispatched turn appends `class`, `prompt/completion/total_tokens`,
+and estimated `credits` to its `report` event in `loop.jsonl`;
+`loopState` rebuilds per-class totals from the JSONL at startup, and
+`harness quota status` renders the table (turns/tokens/credits per
+class + TOTAL). Also verified live in-container: real `/proc` reads
+(load 13.43, PSI io 0.3%, disk 44.9GB) and correct TZ evaluation.
+
+## 7. Test results (TDD, table-driven, fake clock/servers)
+
+- `internal/quota`: parser (canonical + aliases + hostile bodies),
+  schedule edges (TZ, wrap, weekdays), credit math (peak/off-peak,
+  flash/flagship, unknown-model conservatism), decision levels (healthy
+  / defer / block / peak interactions), estimate snapshots (redis +
+  local), fake-redis RESP2 server round-trips, dead/nil-state fail-soft,
+  resource fixtures (PSI present/absent, unreadable).
+- `internal/loop`: wall defer+recover, peak heavy-defer + flash-run +
+  off-peak release, resource-busy defer, usage accounting JSONL +
+  table, gate-off-by-default regression.
+- `internal/config`: [quota]/[resources] load + 7 validation-error
+  cases; TOML float support.
+- Docker dev discipline throughout (`./dev.sh` only); host untouched.
+
+## 8. Resource gate (491) + cgroup runbook
+
+Read-only monitor (loadavg, MemAvailable, statfs disk free,
+`/proc/pressure/io` some-avg60 — skipped when PSI absent, read errors
+never defer). cgroup enforcement documented as deploy-time in the README
+runbook (`--memory/--cpus/--pids-limit` / systemd slice), per Charles's
+"docker + cgroups for all work" rule.
+
+## 9. Open items
+
+- **LIVE VERIFICATION open**: z.ai usage endpoint (flip `usage_url`
+  when it ships; parser + probe command ready).
+- Weekly bucket reset is approximated to Monday 00:00 plan-TZ until
+  `reset_at` arrives from a real endpoint.
+- Turn `cached_tokens` not yet captured from LiteLLM responses (LLM
+  client returns prompt/completion only) — credits currently
+  conservative (cache discount not credited).
+- Redis container not yet deployed to 192.168.3.78 (runbook written;
+  `enabled = false` defaults mean nothing regresses until then).
+- LiteLLM $-spend budget keys (separate from credit buckets) still
+  phase 3.
+
+— PMO worker, MOPAC harness self-host loop