Files
mopac-pmo/STATE-SNAPSHOT-20260829-0530.md
T

4.1 KiB

STATE SNAPSHOT #6 — 2026-08-29 ~05:30 CST (pre-wrap, crush update pending)

READ THIS FIRST after relaunch. Then: ~/.coordinate/log.md tail, ~/.coordinate/logs/queue.status tail, screen -ls.

Who/where

PMO agent reachableceo-PMO, host ultix-streaming, screen RCEO-PMO (relaunch PMO under EXACTLY this name — heartbeat stuffs into it). Charles = reachableceo (never "founder"; never track his time/availability). PMO = traffic cop ONLY: meaty TASK prompts -> headless workers, verify REPORTs, archive to ~/.coordinate/archive/, mirror Redmine. One item per turn, HUD table at end, medium verbosity.

Mission status (spec of record: harness/docs/SPEC-20260829-charles-brief.md)

  • MVP bar MET (8/28 19:29) and V1 FEATURE-COMPLETE: harness loop (self-hosting PROVEN live, issue 487), events (:4100 webhooks), serve (:8090 OWUI front door), keyproxy v0 (mpk_ refs, file/env backends, BW/Vault stubs), mopac-bitwarden-go v0 (fake-server tested).
  • Timeline: beta 8/31 0600 CST, production + first 3-COS briefing 9/1 0630 CST.
  • Redmine backlog: tickets 490-499 (490 quota, 491 resource, 492 phase routing, 493 queue selection, 494 multi-account deploy, 495 briefing, 496 review chain, 497 keyproxy BW live, 498 MCP fleet, 499 PDF).

QUOTA CRISIS — careful-weekend mode (Charles at 93% weekly)

  • Worker turns default glm-4.7-flash (dispatch-turn.sh, TURN_MODEL override for escalation to glm-5.2). AFTER CRUSH UPDATE: check catalog for glm-5.3-flash; if present switch the default in dispatch-turn.sh.
  • queue-after.sh has QUOTA WALL GUARD: any q-*.log with "Usage limit reached" halts the gated chain + pings RCEO-PMO.
  • Do NOT freeze the weekend — careful sustained building (Charles's order).
  • 1900 tonight Charles connects: 1. Bitwarden creds (ticket 497), 2. Signal/Discord, 3. Google, 4. OpenRouter key (separate quota pool).

Chain running (survives crush update; gates use fresh crush runs)

  • work-q9 RUNNING (quota/back-pressure + resource gate, tickets 490+491, on glm-5.2 — sunk cost, let finish) -> work-q11 deploy (494, flash) -> work-q12 briefing (495, flash) -> work-q10 dispatcher (492+493, escalate to 5.2 if flash output garbage).
  • TASK files: ~/.coordinate/inbox-work/TASK-20260829-{0500-quota, 0700-dispatcher, 0900-deploy, 1100-briefing}.md
  • pmo-heartbeat screen: 1h interval, wakes PMO with chain-check prompt.
  • If q9 was mid-flight during the binary update: running proc unaffected (old inode); gates launch NEW crush runs post-update.

Ops facts (hard-won)

  • LiteLLM http://192.168.3.78:4001, 13 z.ai models; /config/update hot-reload LIES — docker restart litellm (~60s).
  • Harness vkey: ~/.coordinate/secrets/mopac-harness-vkey.env (0600, alias mopac-harness-demo-v1). Redmine: source ~/.creds/redmine.env with set -a, var REDMINE_API_KEY.
  • dispatch rc: 3=deferred(cap), 99=gate unavailable, 143=self-pkill (happened once — TASKs now say never broad pkill), z.ai flake "Message content shorter than read bytes" = redispatch.
  • dispatch-turn doorbell targets ${USER}-PMO (dead, cosmetic); real wake = pmo-heartbeat.
  • PMO cannot ssh/sudo: ticket 494 deploy runs via Charles window (~10 min).
  • keyproxy repo: default branch fixed to main (was master).
  • Absolute paths for dispatch TASK args (relative path = empty prompt bug).
  • Gitea API pattern: tea token from ~/.config/tea/config.yml, never echo.

First moves for fresh PMO

  1. Confirm chain state (queue.status, screens). Verify q9 REPORT when it lands -> archive -> Redmine 490+491 Done.
  2. Post-update: check crush.json/catalog for glm-5.3-flash; switch dispatch default if present.
  3. Continue verify->archive->mirror->redispatch loop per heartbeat.
  4. At 1900: walk Charles's connection list (BW > Signal/Discord > Google > OpenRouter); dispatch ticket 497 turn after BW creds land.

AMENDMENT ~06:30

Charles upgraded to MAX PLAN — all quota buckets ZERO. Crisis over; quality-first restored (dispatch default glm-5.2; flash via TURN_MODEL for grind). q9 must verify its polling against MAX-plan bucket shapes. PDF pipeline (499) dispatched parallel (work-q13, new repo, no collision).