quota: z.ai credit-bucket back-pressure + resource gate + usage accounting (Redmine 490+491)

The loop now consults quota and host state before every dispatch and
DEFERS gated work with a logged reason instead of letting turns die at
the provider (the 2026-08-28 19:00 quota-wall failure mode, replayed as
a test). Adds internal/quota: 5h/weekly credit buckets (provider poll
when z.ai ships an endpoint - fake-server tested - else locally
estimated from the documented credit formula), TZ-aware peak window
(default 01:00-05:00 America/Chicago weekdays, matching the documented
z.ai peak Mon-Fri 14:00-18:00 Singapore), block/defer thresholds, a
read-only load/mem/disk/IO-PSI monitor, an optional redis shared-state
hop (stdlib RESP2 mini-client) so all instances of an account
coordinate, per-class token+credit accounting in loop.jsonl, and
`harness quota status|probe|gate`. Config: [quota] + [resources]
sections; README runbook covers the redis container and deploy-time
cgroup enforcement.

💘 Generated with Crush

Assisted-by: Crush:glm-5.2

README.md

@@ -9,12 +9,15 @@ other agents interact with a stack only through Redmine (SoR), Discourse
 (docs) and Gitea (code), never by attaching to the loop.
 
 Status: 2026-08-29 — skeleton + event receiver + self-host loop + **OWUI
-front door live**: `harness serve` answers OpenAI-compatible
-`/v1/models` + `/v1/chat/completions` on LAN port 8090 (bearer vkey,
-stateless bounded turns, live-proven through LiteLLM to glm-5.3 and
-glm-4.7-flash). Also live: `harness loop` (Redmine SoR self-hosting, fake-
-Redmine e2e test-asserted), the MVP demo path, and `harness events` on port
-4100.
+front door live** + **quota/resource gates** (Redmine 490+491): `harness
+loop` consults the z.ai credit buckets (5h + weekly, polled or locally
+estimated), a TZ-aware peak window (default 01:00-05:00 CST weekdays) and
+host load/mem/disk/IO before every dispatch — gated work DEFERS with a
+logged reason and is reconsidered next scan, never hard-failed; per-class
+token+credit accounting lands in the loop JSONL (`harness quota status`).
+`harness serve` (OWUI front door, LAN 8090), `harness loop` (fake-Redmine
+e2e test-asserted), the MVP demo path and `harness events` (port 4100)
+all live.
 
 ## Quickstart
 
@@ -226,6 +229,57 @@ Live on the LAN 2026-08-29: catalog of 9 models, 401/400/200 paths, and real
 turns through LiteLLM (`mopac-primary` -> glm-5.3, `mopac-study` ->
 glm-4.7-flash) with usage accounting, driven by python urllib.
 
+### Quota + resource gates (`[quota]` / `[resources]`, Redmine 490+491)
+
+The 2026-08-28 ~19:00 quota wall killed dispatched turns mid-flight; the
+gates turn that failure mode into a logged throttle. Before EVERY dispatch
+the loop consults, in order:
+
+1. **Host resources** (`[resources]`, read-only `/proc` + statfs): loadavg,
+   mem available, work-root disk free, IO pressure (`/proc/pressure/io`,
+   skipped when PSI is absent). Any violation defers with all reasons
+   surfaced.
+2. **Quota buckets** (`[quota]`): the z.ai coding plan's 5-hour and weekly
+   credit windows. `>= block_at_pct` (default 95%) defers EVERYTHING with
+   the bucket/ratio in the reason — the wall, caught early.
+3. **Peak window**: z.ai peak hours (documented Mon-Fri 14:00-18:00
+   Singapore == 01:00-05:00 CST in winter) charge full rate; inside the
+   window only `peak_classes` (the flash/LLM-lite tier) dispatch, heavy
+   classes defer to off-peak (50% credit cost).
+4. **Soft quota**: `>= defer_at_pct` (default 85%) defers heavy classes
+   while LLM-lite continues.
+
+A deferred task is NOT consumed: no turn, no note, no dedup marker — the
+next scan reconsiders it (the 19:00-wall scenario is replayed as a test:
+wall up -> defer + loop clean -> quota recovers -> dispatch). Defer events
+land in `loop.jsonl` (`"type":"defer"` + reason), deduped per task+reason.
+
+**Quota state** comes from two sources: the provider endpoint (`usage_url`,
+bearer `key_ref`, parsed into buckets with reset times) and, when that is
+unconfigured/unreachable, locally ESTIMATED consumption — per-turn credits
+computed from the documented z.ai formula (input x 6.9 + cached x 1.7 +
+output x 24, per 10k tokens; flash 2.3/0.56/8; off-peak 50% off) against
+the configured plan limits. z.ai documents the buckets but publishes no
+usage REST route today (probed 2026-08-29 — see the REPORT); the parser
+targets the documented shape and is fake-server-tested, so flipping
+`usage_url` on when z.ai ships it is a config edit. LIVE VERIFICATION open.
+
+**Shared state**: with `redis_url` set, the latest snapshot and the credit
+estimates live in one redis container so all harness instances of an
+account (9 accounts across 2 hosts) coordinate — see the runbook below.
+Redis down = this instance's local estimate; the loop never stops for it.
+
+**Usage accounting**: every dispatched turn appends class + tokens +
+estimated credits to its `report` event in `loop.jsonl`; `harness quota
+status` renders the per-class table (the feed for the per-instance
+Discourse usage reports).
+
+```sh
+./bin/harness quota status   # snapshot + peak window + resources + usage table
+./bin/harness quota gate     # the allow/defer verdict per class, right now
+./bin/harness quota probe    # one usage_url poll; parsed buckets or the error
+```
+
 ### Help
 
 ```sh
@@ -317,9 +371,10 @@ Subcommands (from `harness help`):
 |---|---|
 | `harness help` | print usage (also `-h`, `--help`) |
 | `harness once` | run ONE conductor iteration, then exit |
-| `harness loop` | run the self-host daemon until SIGINT (poll -> turn -> note/status writeback) |
+| `harness loop` | run the self-host daemon until SIGINT (poll -> gate -> turn -> note/status writeback) |
 | `harness events` | run the webhook receiver until SIGINT/SIGTERM |
 | `harness serve` | run the OpenAI-compatible front door until SIGINT/SIGTERM (the OWUI connection) |
+| `harness quota` | gate surface: `status` (snapshot + usage accounting), `gate` (per-class verdicts), `probe` (one usage poll) |
 
 Flags for `once`:
 
@@ -353,6 +408,12 @@ Flags for `serve`:
 | `-config PATH` | config file (default `$HARNESS_CONFIG`, then `./harness.toml`) |
 | `-listen ADDR` | bind address (overrides `[serve]` listen) |
 
+Flags for `quota` (`harness quota <status|probe|gate> [flags]`):
+
+| Flag | Meaning |
+|---|---|
+| `-config PATH` | config file (default `$HARNESS_CONFIG`, then `./harness.toml`) |
+
 Exit codes:
 
 | Code | Meaning |
@@ -458,9 +519,12 @@ Sourced from [REPORT.md](REPORT.md) — keep both in sync.
 | Status transitions | Works | `[redmine.status_map]` names → ids via `/issue_statuses.json`; refresh advances the dedup marker past its own writes |
 | `mpk:` key refs | Works | `[keyproxy]` hop (POST `/v1/resolve`, bearer, cached); local refs unaffected |
 | Gitea REPORT commit | Works (off) | `[gitea] commit_reports`: contents-API create-or-update right after the REPORT lands |
+| Quota gate (490) | Works (off) | `[quota]`: 5h/weekly credit buckets (poll or estimate), block/defer/peak back-pressure, defer-not-fail; usage accounting per class in loop.jsonl; z.ai usage endpoint LIVE VERIFICATION open |
+| Shared quota state | Works (off) | `redis_url`: one redis container, all instances of an account share snapshot + estimates; stdlib RESP2 mini-client, fail-soft |
+| Resource gate (491) | Works (off) | `[resources]`: loadavg/mem/disk/IO-PSI thresholds, read-only, defer-not-fail |
 | Event → turn dispatch | Stubbed | conductor `DispatchEvent` prints what it would do; wiring is phase 3 |
 | Tests | Works | table-driven, stdlib only; build/vet/test clean on go1.26 |
-| Budget/semaphore gate | Stubbed | tokens in REPORT, no cost/spend enforcement yet |
+| Budget/semaphore gate | Partial | credit-bucket back-pressure is live (rows above); LiteLLM-$-spend budget keys remain open |
 | `bw:` key refs | Stubbed | error until the bitwarden wrapper (phase 3) |
 | Loop concurrency | Stubbed | v0 = one turn at a time; in-process concurrency knob later |
 | Streaming + turn resume | Stubbed | retry is request-level today; serve door is non-streaming by design v0 |
@@ -470,6 +534,37 @@ Sourced from [REPORT.md](REPORT.md) — keep both in sync.
 | Local inbox intake | Stubbed | not started |
 | Next (phase 3) | Next | bitwarden wrapper, full permission layer, event → turn wiring, budget gate via LiteLLM spend APIs, serve streaming + serve tools, loop concurrency knob |
 
+## Runbook: shared quota state + resource control (deploy-time)
+
+**Redis container** (the shared-state hop for `[quota] redis_url`; one per
+host-pair, container only, no host packages):
+
+```sh
+docker run -d --name mopac-quota-redis --restart unless-stopped \
+  -p 192.168.3.78:6390:6379 \
+  -v /srv/mopac-quota-redis:/data \
+  redis:7-alpine --appendonly yes
+```
+
+Every harness instance of the same z.ai account then sets the same
+`account` + `redis_url` in `[quota]`; keys are namespaced
+`mopac:quota:<account>:{snapshot,est:*}`. Redis unreachable = local
+estimates only (fail-soft, logged). The harness speaks RESP2 directly —
+no client library, no host redis-cli needed.
+
+**cgroup enforcement** (ticket 491, deploy-time): the in-harness gate is
+read-only and advisory — it defers dispatch when the HOST is busy. To keep
+builds/turns from making the host busy in the first place, run each loop
+container under cgroup limits at deploy:
+
+```sh
+docker run ... --memory 4g --cpus 2 --pids-limit 512 \
+  --io-max bandwidth=/data:100mb ... # device-specific; see docker run(1)
+```
+
+or a systemd slice for non-container deploys (`CPUQuota=200%`,
+`MemoryMax=4G`, `IOWeight`). The gate catches what the limits don't.
+
 ## Docs and links
 
 - [DESIGN.md](DESIGN.md) — design spec (hard rules, build order, org model)