MOPAC/README.md

MOPAC harness

The MOPAC harness is a headless agent conductor written in Go: it pulls a task from Redmine (or a demo issue), routes it to the right model through LiteLLM, runs ONE bounded turn with an allow-listed bash tool, and writes the result as a REPORT file — with harness loop it also drives ITSELF: Redmine is the SoR, the loop is the worker, no bash middle layer. There is no TUI — humans and other agents interact with a stack only through Redmine (SoR), Discourse (docs) and Gitea (code), never by attaching to the loop.

Status: 2026-08-29 — skeleton + event receiver + self-host loop + OWUI front door live + quota/resource gates (Redmine 490+491): harness loop consults the z.ai credit buckets (5h + weekly, polled or locally estimated), a TZ-aware peak window (default 01:00-05:00 CST weekdays) and host load/mem/disk/IO before every dispatch — gated work DEFERS with a logged reason and is reconsidered next scan, never hard-failed; per-class token+credit accounting lands in the loop JSONL (harness quota status). harness serve (OWUI front door, LAN 8090), harness loop (fake-Redmine e2e test-asserted), the MVP demo path and harness events (port 4100) all live. Multi-account deploy packaging (Redmine 494): make release + deploy/ — 9 accounts / 2 hosts, per-account ports + state, no root.

Quickstart

All dev work happens inside a Docker builder (host stays toolchain-free); docker pull of the builder is pre-authorized. Commands below were verified on 2026-08-28 from a fresh clone.

Build and test

dev.sh routes every compile/vet/test path through the digest-pinned builder container (the host stays toolchain-free):

./dev.sh check   # = go build + go vet + go test, all inside the builder

The equivalent raw command:

docker run --rm -v "$PWD:/h" -w /h \
  -u "$(id -u):$(id -g)" -e HOME=/tmp \
  golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 \
  sh -c 'go build -o bin/harness ./cmd/harness && go vet ./... && go test ./...'

(that digest = golang:1.26-bookworm; alpine lacks bash, which the exec tool's tests need.)

make release builds the deployable static binary (bin/harness-linux-amd64, CGO off) through the same pinned builder; make deploy-test tests the multi-account packaging — see "Deploy: 9 accounts / 2 hosts" below.

Expected output (tail):

ok  	ukrrs.com/mopac/harness/internal/config
ok  	ukrrs.com/mopac/harness/internal/events
ok  	ukrrs.com/mopac/harness/internal/intake
ok  	ukrrs.com/mopac/harness/internal/llm
ok  	ukrrs.com/mopac/harness/internal/loop
ok  	ukrrs.com/mopac/harness/internal/models
ok  	ukrrs.com/mopac/harness/internal/serve
ok  	ukrrs.com/mopac/harness/internal/tools
ok  	ukrrs.com/mopac/harness/internal/writeback

(The module path is ukrrs.com/mopac/harness; the repo lives at git.knownelement.com/ukrrs/MOPAC.)

Configure

cp harness.toml.example harness.toml

harness.toml is gitignored. It holds no secrets — only key refs (env:NAME, file:PATH, literal:VALUE; bw: is reserved for the bitwarden wrapper, build phase 3) that are resolved at runtime and redacted from every error path.

Dry-run (no LLM call, no REPORT)

./bin/harness once --dry-run --demo

Expected output:

harness: intake: demo issue from harness.toml [demo]
harness: task demo-1 (demo): "MVP demo: GLM self-description" class=primary -> mopac-primary -> glm-5.3
PLAN (dry-run)
  vertical: demo
  task: [demo-1] MVP demo: GLM self-description (source demo)
  routing: class "primary" -> tier mopac-primary -> model glm-5.3
  tools: bash (allow=18 deny=7 default=deny timeout=60s)
  bound: max 8 rounds
harness: dry-run complete, no LLM call made

Live demo turn (the MVP bar)

HARNESS_LITELLM_KEY=<vertical virtual key> ./bin/harness once --demo

The model's reply lands verbatim in reports/REPORT-latest.md with model / tier / class / tokens / rounds telemetry. Without the key the run fails fast and clean (verified):

harness: litellm key: environment variable HARNESS_LITELLM_KEY is not set

exit code 1 (config error). The with-key path was proven live on 2026-08-28: the [demo] prompt "tell me about yourself" routed to glm-5.3 via tier mopac-primary and the GLM self-description landed as the REPORT.

Self-host loop (harness loop)

The loop makes the harness its own worker: Redmine is the SoR, the loop is the conductor. It polls the /issues.json intake every [loop] poll_interval_secs (default 120; -interval overrides, --once = single scan for cron-style setups) and, for every issue not yet processed at its current updated_on:

1. runs ONE bounded turn (the same conductor once uses; sequential, v0 — one turn at a time, no concurrency knob yet);
2. writes the REPORT file as always;
3. POSTs the REPORT body back onto the issue as a Redmine journal note (PUT /issues/{id}.json);
4. transitions status per [redmine.status_map] (e.g. "In Progress" = "Done"; names resolved via /issue_statuses.json, cached);
5. optionally commits the REPORT to Gitea ([gitea] commit_reports, off by default);
6. refreshes its dedup marker to the issue's post-writeback updated_on so its own note never re-triggers it.

State is an append-only state/loop/loop.jsonl ([loop] state_dir): one JSON line per action (dispatch / report / note / status / commit / refresh / error), rebuilt into the dedup index at startup — restarts keep exactly-once-reaction semantics. A failed turn is recorded and NOT retried (update the issue to re-release it), so a down proxy cannot hot-loop the poll. Every action logs one line to stdout and one to the JSONL.

./bin/harness loop                # daemon: poll, dispatch, write back; SIGINT stops
./bin/harness loop --dry-run      # scan + print what would dispatch; no turns, no state
./bin/harness loop --once         # single scan (cron-able)

Secrets may come from the keyproxy hop: key_ref = "mpk:mpk-redmine" in harness.toml plus a [keyproxy] section resolves through mopac-keyproxy's POST /v1/resolve (short in-memory cache). env: / file: / literal: refs keep working untouched, so the loop runs with or without keyproxy up.

Webhook receiver (harness events)

Redmine / Discourse / Gitea webhooks punch the harness through the event receiver. Configure [events] in harness.toml (secrets as env refs — see harness.toml.example), then:

HARNESS_REDMINE_WEBHOOK_SECRET=... HARNESS_DISCOURSE_WEBHOOK_SECRET=... \
HARNESS_GITEA_WEBHOOK_SECRET=... ./dev.sh events

publishes port 4100 on the host LAN (override with -listen). Routes:

Route	Verification	Maps to
POST /hooks/redmine	shared-secret header (default X-Redmine-Webhook-Secret)	dispatch_turn on issue update/note
POST /hooks/discourse	shared-secret header (default X-Discourse-Webhook-Secret)	respond_turn on post reply
POST /hooks/gitea	HMAC-SHA256 X-Gitea-Signature over the raw body	pipeline_step on PR approved/merged
GET /healthz	none	liveness

Unsigned or unverified deliveries get 401; verified ones are normalized to one internal event record, appended to state/events/events.jsonl (0600, deduped by provider event id: X-Gitea-Delivery / X-Discourse-Event-Id, payload-digest fallback), and handed to the conductor (turn dispatch is a stub until phase 3 wiring). End-to-end smoke, including the 401/200 paths and the JSONL record, driven with python urllib (curl is banned on host):

./dev.sh smoke

OpenAI-compatible front door (harness serve)

OpenWebUI (Cloudron + SSO) is the interactive surface; MOPAC is just another OpenAI connection to it. Configure [serve] in harness.toml, then:

HARNESS_SERVE_VKEY=<vkey> HARNESS_LITELLM_KEY=<vertical key> ./dev.sh serve

publishes port 8090 on the host LAN (override with -listen). OWUI connection settings: base URL http://<host>:8090/v1, API key = the vkey. Routes:

Route	Auth	Behavior
POST /v1/chat/completions	Bearer vkey	one bounded stateless conductor turn over the sent history; reply = single assistant message + usage
GET /v1/models	Bearer vkey	the servable catalog: one model per [models.classes] class, named mopac-<class>
GET /healthz	none	liveness

Semantics (v0):

• Stateless: OWUI sends the full conversation history each call; the harness runs ONE bounded turn over it (same runTurn machinery once uses) and returns the final text. No session storage.
• Model routing: request.model (e.g. mopac-primary) maps through the existing class -> tier -> concrete-model tables; unknown model = 400 naming the valid ones. [serve] enabled_models optionally narrows the catalog.
• History assembly: the client's leading system message is preserved verbatim (OWUI personas win); only when none is sent does the harness prepend a minimal identity prompt for the vertical.
• Auth: the Bearer vkey is compared as SHA-256 digests in constant time; missing/wrong keys get one byte-identical generic 401. The vkey is a ref (vkey_ref), resolved at startup, never logged.
• Non-streaming: stream:true gets an explicit 400 (OWUI tolerates non-streaming providers); streaming is Next.
• Tools OFF: serve turns are pure chat — any tool call a model produces anyway is refused as a tool result, never executed. Tool use inside serve turns is Next.
• temperature / max_tokens are forwarded upstream; usage is summed over rounds and returned with the reply; upstream failures surface as a terse 502 (upstream bodies never reach the client).

Live on the LAN 2026-08-29: catalog of 9 models, 401/400/200 paths, and real turns through LiteLLM (mopac-primary -> glm-5.3, mopac-study -> glm-4.7-flash) with usage accounting, driven by python urllib.

Quota + resource gates ([quota] / [resources], Redmine 490+491)

The 2026-08-28 ~19:00 quota wall killed dispatched turns mid-flight; the gates turn that failure mode into a logged throttle. Before EVERY dispatch the loop consults, in order:

1. Host resources ([resources], read-only /proc + statfs): loadavg, mem available, work-root disk free, IO pressure (/proc/pressure/io, skipped when PSI is absent). Any violation defers with all reasons surfaced.
2. Quota buckets ([quota]): the z.ai coding plan's 5-hour and weekly credit windows. >= block_at_pct (default 95%) defers EVERYTHING with the bucket/ratio in the reason — the wall, caught early.
3. Peak window: z.ai peak hours (documented Mon-Fri 14:00-18:00 Singapore == 01:00-05:00 CST in winter) charge full rate; inside the window only peak_classes (the flash/LLM-lite tier) dispatch, heavy classes defer to off-peak (50% credit cost).
4. Soft quota: >= defer_at_pct (default 85%) defers heavy classes while LLM-lite continues.

A deferred task is NOT consumed: no turn, no note, no dedup marker — the next scan reconsiders it (the 19:00-wall scenario is replayed as a test: wall up -> defer + loop clean -> quota recovers -> dispatch). Defer events land in loop.jsonl ("type":"defer" + reason), deduped per task+reason.

Quota state comes from two sources: the provider endpoint (usage_url, bearer key_ref, parsed into buckets with reset times) and, when that is unconfigured/unreachable, locally ESTIMATED consumption — per-turn credits computed from the documented z.ai formula (input x 6.9 + cached x 1.7 + output x 24, per 10k tokens; flash 2.3/0.56/8; off-peak 50% off) against the configured plan limits. z.ai documents the buckets but publishes no usage REST route today (probed 2026-08-29 — see the REPORT); the parser targets the documented shape and is fake-server-tested, so flipping usage_url on when z.ai ships it is a config edit. LIVE VERIFICATION open.

Shared state: with redis_url set, the latest snapshot and the credit estimates live in one redis container so all harness instances of an account (9 accounts across 2 hosts) coordinate — see the runbook below. Redis down = this instance's local estimate; the loop never stops for it.

Usage accounting: every dispatched turn appends class + tokens + estimated credits to its report event in loop.jsonl; harness quota status renders the per-class table (the feed for the per-instance Discourse usage reports).

./bin/harness quota status   # snapshot + peak window + resources + usage table
./bin/harness quota gate     # the allow/defer verdict per class, right now
./bin/harness quota probe    # one usage_url poll; parsed buckets or the error

Help

./bin/harness help

Prints the full usage block reproduced under CLI reference.

Architecture

flowchart LR
    P(["harness loop<br/>poll every poll_interval_secs"]) --> I["INTAKE<br/>/issues.json scope query"]
    I -->|"id + updated_on not in loop.jsonl"| T["BOUNDED TURN<br/>per issue, sequential (v0)"]
    T --> R["REPORT<br/>reports/ file (atomic)"]
    R --> N["REDMINE WRITEBACK<br/>journal note (PUT /issues/{id}.json)<br/>+ status per [redmine.status_map]"]
    G["GITEA COMMIT<br/>optional, [gitea] commit_reports"] -.-> N
    R -.-> G
    N --> S[("state/loop/loop.jsonl<br/>append-only, dedup index<br/>refreshed to post-writeback updated_on")]
    S -->|"next scan skips processed"| I

The loop replaces the crossfeed bash stack (semaphore slot files, queue scripts, screen doorbells): concurrency is simply "one turn at a time" for v0, wake discipline is the poll interval, and the only cross-process state is the append-only JSONL.

One conductor iteration per process (harness once) still works exactly as before for manual/chained runs:

flowchart TD
    S(["harness once"]) --> I["INTAKE<br/>Redmine /issues.json scope query<br/>or the [demo] issue"]
    I -->|"task + class"| RT["ROUTING<br/>class -&gt; tier -&gt; concrete model<br/>[models] + [models.classes]"]
    RT --> T["BOUNDED TURN<br/>OpenAI-compatible chat via LiteLLM<br/>gated bash tool, max_rounds cap"]
    T --> W["REPORT<br/>reports/REPORT-&lt;vertical&gt;-&lt;task&gt;-&lt;ts&gt;.md<br/>+ REPORT-latest.md, atomic write"]
    W --> E(["exit 0"])
    E -.->|"re-invoke to chain the next turn"| S

One conductor iteration per process: INTAKE resolves the released scope (a Redmine query, or the [demo] issue with --demo) into a task with a class; ROUTING maps the class through the config-only tier table to a concrete proxy model; the bounded TURN drives an OpenAI-compatible chat loop through LiteLLM (retry/backoff on 429/5xx/transport, usage accounting) with a tool-calling loop capped at max_rounds, where the only tool today is an allow-listed bash exec whose gate denials feed back to the model as tool results instead of failing the turn; WRITEBACK persists the final assistant reply as a REPORT file plus REPORT-latest.md (atomic tmp+rename). On a mid-turn LLM failure after content exists, a partial REPORT is still written with the error as the stop reason. Chaining is by re-invocation; the loop is the daemon form of the same iteration, run per new/updated issue with the SoR writebacks on top.

The event path is a second door into the same loop:

flowchart LR
    RM["Redmine webhook"] --> V["VERIFY<br/>shared-secret / HMAC headers"]
    DC["Discourse webhook"] --> V
    GT["Gitea webhook"] --> V
    V -->|401 unverified| X(["rejected"])
    V --> N["NORMALIZE<br/>one Event record: source, kind, actor,<br/>subject id, digest, received-at"]
    N --> S["STORE<br/>state/events/events.jsonl, append-only,<br/>dedup by provider event id"]
    S -->|"dispatch_turn / respond_turn / pipeline_step"| C(["conductor (dispatch stub)"])

Verification rejects unsigned and unverified deliveries with 401 (no detail beyond "unverified"); secret material and header values never reach logs or the event log — only normalized fields and a payload digest do.

OWUI chat is a second door into the same loop (the bounded-turn core is shared, tools off):

flowchart LR
    O(["OpenWebUI (Cloudron + SSO)"]) -->|"Bearer vkey"| A["AUTH<br/>constant-time vkey compare<br/>401 generic on miss"]
    A -->|"POST /v1/chat/completions"| C["CATALOG ROUTING<br/>model mopac-<class> -> tier -> concrete model<br/>"]
    C --> T["BOUNDED TURN (stateless)<br/>full client history per call<br/>tools OFF v0"]
    T -->|"assistant message + usage"| O
    A -->|"GET /v1/models"| M(["catalog: one model per class"])

CLI reference

Subcommands (from harness help):

Subcommand	Purpose
harness help	print usage (also -h, --help)
harness once	run ONE conductor iteration, then exit
harness loop	run the self-host daemon until SIGINT (poll -> gate -> turn -> note/status writeback)
harness events	run the webhook receiver until SIGINT/SIGTERM
harness serve	run the OpenAI-compatible front door until SIGINT/SIGTERM (the OWUI connection)
harness quota	gate surface: status (snapshot + usage accounting), gate (per-class verdicts), probe (one usage poll)

Flags for once:

Flag	Meaning
-config PATH	config file (default $HARNESS_CONFIG, then ./harness.toml)
-dry-run	intake + plan only; no LLM call, no REPORT
-demo	run the [demo] issue instead of Redmine intake
-task-id ID	run only the task/issue with this id

Flags for loop:

Flag	Meaning
-config PATH	config file (default $HARNESS_CONFIG, then ./harness.toml)
-interval DUR	poll interval (overrides [loop] poll_interval_secs)
-once	single scan then exit (cron-able)
-dry-run	scan + print what would dispatch; no turns, no state writes

Flags for events:

Flag	Meaning
-config PATH	config file (default $HARNESS_CONFIG, then ./harness.toml)
-listen ADDR	bind address (overrides [events] listen)

Flags for serve:

Flag	Meaning
-config PATH	config file (default $HARNESS_CONFIG, then ./harness.toml)
-listen ADDR	bind address (overrides [serve] listen)

Flags for quota (harness quota <status|probe|gate> [flags]):

Flag	Meaning
-config PATH	config file (default $HARNESS_CONFIG, then ./harness.toml)

Exit codes:

Code	Meaning
0	ok (including "no tasks in scope"; loop: clean SIGINT stop)
1	usage / config / routing / writeback error
2	intake error
4	llm / turn error

Configuration

Every harness.toml section (see harness.toml.example, load-tested so it cannot rot):

Section	Keys	Meaning
top level	vertical	vertical/stack identity for this harness instance
	work_root	where the bash tool executes (default .)
	report_dir	where REPORT files land (default reports)
[loop]	max_rounds	max LLM round trips per turn, tool calls included (default 8)
	poll_interval_secs	harness loop scan interval (default 120)
	state_dir	loop state dir (default state/loop); loop.jsonl created 0600, one JSON line per action
[redmine]	url, key_ref	SoR issues endpoint + auth key ref
	scope_query / scope_query_id	released-scope filter: raw /issues.json params, or a saved query id (scope_query wins when both are set)
	class_field, default_class	custom field carrying the task class; fallback for issues without it (default Class / primary)
	limit	max issues fetched per intake (default 50)
[redmine.status_map]	"In Progress" = "Done" style entries	loop: after a REPORT is noted, an issue whose current status matches a key moves to the value (names resolved via /issue_statuses.json); empty map = no transitions. Quoted keys allow spaces
[litellm]	base_url, key_ref	OpenAI-compatible proxy endpoint + key ref
	timeout_secs, max_retries	per-request timeout (default 120) and retry count (default 2)
[keyproxy]	url, token_ref	optional mopac-keyproxy resolve hop for mpk: refs (bearer token is itself a LOCAL ref); url + token_ref are all-or-nothing
	cache_ttl_secs	in-memory mpk: resolution cache TTL (default 60)
[gitea]	url, key_ref, owner, repo	optional REPORT-commit step; all required when commit_reports = true
	branch, commit_reports	target branch (empty = repo default); the whole step is off by default
[models]	tier keys, default_tier	tier alias → concrete proxy model (see below)
[models.classes]	class keys	task class → tier alias (see below)
[tools.bash]	enabled, timeout_secs, max_output_bytes	bash gate on/off, per-command timeout (default 60s), output truncation (default 100000)
	default	verdict for commands matching no rule: allow or deny (org preset: deny)
	allow, deny	rule lists; deny beats allow; cmd * word-boundary, pfx* raw prefix, pfx/** path prefix, * universal; compound commands checked segment by segment; command substitution and subshells always denied
[demo]	id, subject, prompt, class	the issue --demo runs instead of Redmine intake
[events]	listen	receiver bind address (default :4100; publish via docker -p)
	state_dir	event log dir (default state/events); events.jsonl created 0600
[events.redmine]	secret_ref, secret_header	shared-secret ref + header name (default X-Redmine-Webhook-Secret)
[events.discourse]	secret_ref, secret_header	shared-secret ref + header name (default X-Discourse-Webhook-Secret)
[events.gitea]	secret_ref	HMAC secret ref; verification is always X-Gitea-Signature (hex HMAC-SHA256 of the raw body)
[serve]	listen	harness serve bind address (default :8090; own port, coexists with [events])
	vkey_ref	bearer vkey ref for the OpenAI-compatible front door (the key the OWUI connection presents)
	enabled_models	optional subset of the catalog (mopac-<class> names); empty = every [models.classes] class

Key refs accepted anywhere a *_ref appears: env:NAME, file:PATH, literal:VALUE (last resort), mpk:PLACEHOLDER (resolved through the optional [keyproxy] hop — see mopac-keyproxy; mpk:redmine and mpk:mpk-redmine are equivalent), and bw:REF (reserved; errors until the bitwarden wrapper lands in build phase 3).

Model routing

Static, config-only, no heuristics: a task's class maps to a tier alias, the tier alias maps to the concrete model actually sent to the proxy. Unknown class = hard error naming the config section.

Tiers ([models]):

Tier	Concrete model	Role
mopac-study	glm-4.7-flash	flash tier
mopac-code	glm-5.2	flagship
mopac-review	glm-5-turbo	mid
mopac-primary	glm-5.3	default / flagship+
mopac-vision	glm-4.6v	vision when needed

Classes ([models.classes]):

Task class	Tier	Concrete model
study	mopac-study	glm-4.7-flash
read	mopac-study	glm-4.7-flash
code	mopac-code	glm-5.2
architecture	mopac-code	glm-5.2
review	mopac-review	glm-5-turbo
summarize	mopac-review	glm-5-turbo
writeback	mopac-review	glm-5-turbo
vision	mopac-vision	glm-4.6v
primary	mopac-primary	glm-5.3
(no class)	default_tier = mopac-primary	glm-5.3

Status

Sourced from REPORT.md — keep both in sync.

Area	State	Detail
Config	Works	TOML-subset parser, defaults + validation, secret refs only, redacted errors
Model routing v0	Works	tier + class maps; concrete model resolved before the request leaves
Conductor single-shot	Works	harness once chainable; --dry-run makes zero LLM calls (test-asserted)
Redmine intake	Works	/issues.json scope query, class custom field, --demo fallback
Bounded turn	Works	LiteLLM chat, retry/backoff, tool loop capped at max_rounds, partial REPORT on mid-turn failure
Exec tool	Works	allow-listed bash, segment-wise compound checks, timeout + truncation
REPORT writeback	Works	timestamped file + REPORT-latest.md, atomic, full telemetry
Event receiver	Works	harness events: verify (HMAC/shared-secret) → normalize → append-only JSONL with provider-id dedup → action mapping; smoke-proven on LAN port 4100
OpenAI front door	Works	harness serve: /v1/models catalog (one model per class, mopac-<class>), stateless bounded /v1/chat/completions turns, constant-time vkey auth, tier-map routing, usage accounting; live-proven on LAN port 8090 via LiteLLM (glm-5.3 + glm-4.7-flash)
Self-host loop	Works	harness loop: poll intake → sequential bounded turns → REPORT note writeback → status map → dedup by id + updated_on (append-only loop.jsonl); --once, --dry-run, -interval; fake-Redmine e2e test-asserted
Redmine note writeback	Works	loop path POSTs the REPORT body as a journal note (PUT /issues/{id}.json); once stays file-only
Status transitions	Works	[redmine.status_map] names → ids via /issue_statuses.json; refresh advances the dedup marker past its own writes
mpk: key refs	Works	[keyproxy] hop (POST /v1/resolve, bearer, cached); local refs unaffected
Gitea REPORT commit	Works (off)	[gitea] commit_reports: contents-API create-or-update right after the REPORT lands
Quota gate (490)	Works (off)	[quota]: 5h/weekly credit buckets (poll or estimate), block/defer/peak back-pressure, defer-not-fail; usage accounting per class in loop.jsonl; z.ai usage endpoint LIVE VERIFICATION open
Shared quota state	Works (off)	redis_url: one redis container, all instances of an account share snapshot + estimates; stdlib RESP2 mini-client, fail-soft
Resource gate (491)	Works (off)	[resources]: loadavg/mem/disk/IO-PSI thresholds, read-only, defer-not-fail
Event → turn dispatch	Stubbed	conductor DispatchEvent prints what it would do; wiring is phase 3
Tests	Works	table-driven, stdlib only; build/vet/test clean on go1.26
Budget/semaphore gate	Partial	credit-bucket back-pressure is live (rows above); LiteLLM-$-spend budget keys remain open
bw: key refs	Stubbed	error until the bitwarden wrapper (phase 3)
Loop concurrency	Stubbed	v0 = one turn at a time; in-process concurrency knob later
Streaming + turn resume	Stubbed	retry is request-level today; serve door is non-streaming by design v0
Serve tool use	Stubbed	serve turns are pure chat (tools off); wiring the gated tool palette into serve turns is Next
Session persistence/repair	Stubbed	not started
Write confinement	Stubbed	declared roots + symlink checks land with the permission layer
Local inbox intake	Stubbed	not started
Next (phase 3)	Next	bitwarden wrapper, full permission layer, event → turn wiring, budget gate via LiteLLM spend APIs, serve streaming + serve tools, loop concurrency knob

Runbook: shared quota state + resource control (deploy-time)

Redis container (the shared-state hop for [quota] redis_url; one per host-pair, container only, no host packages):

docker run -d --name mopac-quota-redis --restart unless-stopped \
  -p 192.168.3.78:6390:6379 \
  -v /srv/mopac-quota-redis:/data \
  redis:7-alpine --appendonly yes

Every harness instance of the same z.ai account then sets the same account + redis_url in [quota]; keys are namespaced mopac:quota:<account>:{snapshot,est:*}. Redis unreachable = local estimates only (fail-soft, logged). The harness speaks RESP2 directly — no client library, no host redis-cli needed.

cgroup enforcement (ticket 491, deploy-time): the in-harness gate is read-only and advisory — it defers dispatch when the HOST is busy. To keep builds/turns from making the host busy in the first place, run each loop container under cgroup limits at deploy:

docker run ... --memory 4g --cpus 2 --pids-limit 512 \
  --io-max bandwidth=/data:100mb ... # device-specific; see docker run(1)

or a systemd slice for non-container deploys (CPUQuota=200%, MemoryMax=4G, IOWeight). The gate catches what the limits don't.

Deploy: 9 accounts / 2 hosts (Redmine 494)

The full multi-account packaging lives in deploy/ — one harness instance per Linux account (5 on ultix-streaming, 4 on ultix-offstage), each entirely under that account's ~/.mopac/ (binary, config, 0600 env secrets, state, reports). No root, no systemd, no host packages: sudo is only used to switch identity when installing into the other accounts.

make release      # static linux/amd64 binary: bin/harness-linux-amd64
                  # (docker builder, CGO_ENABLED=0 — runs anywhere)
make deploy-test  # packaging tests: port scheme, template substitution,
                  # installer idempotence in a fake HOME

• deploy/accounts.tsv — the fleet authority: account, host, vertical, Redmine project, quota group, and the port scheme events = 4100 + index / serve = 8090 + index (index 0-8, so the daemons of any two accounts on one host never collide; loop state is per-account under ~/.mopac/state/).
• deploy/install-account.sh <account> — idempotent installer, run AS the target user: creates ~/.mopac/{bin,state,reports,work}, installs the static binary, renders harness.toml from deploy/harness.toml.in
  • the TSV row, writes the 0600 ~/.mopac/env secrets template and mopac-start/mopac-stop helpers. Existing harness.toml/env are NEVER overwritten (re-running = the upgrade path).
• deploy/runbook.md — the exact Charles sequence: build once, stage per host, install per account, secrets bootstrap, verify (once --dry-run --demo, healthz, first loop --once --dry-run), start (nohup helpers + optional cron @reboot), rollback (stop + rm -rf ~/.mopac), per-host time estimates.

Generated configs keep [quota]/[resources] commented with values pre-filled per account — flipping them on is a per-account one-liner once the shared redis container (above) is up.

Docs and links

• DESIGN.md — design spec (hard rules, build order, org model)
• REPORT.md — current build status
• docs/SPEC-20260829-charles-brief.md — Charles's end-to-end design brief (spec of record)
• docs/PORTING-NOTES-crush.md — sessions/MCP/provider study of the crush agent
• docs/PORTING-NOTES-maki.md — permission parsing and token-reduction study of maki
• docs/PORTING-NOTES-secrets.md — Bitwarden secrets study; feeds the bitwarden-go tool
• Repository: https://git.knownelement.com/ukrrs/MOPAC
• Sibling tool repos (spec seeds): mopac-keyproxy, mopac-bitwarden-go
• SoR: Redmine project MOPAC; docs: Discourse

License

AGPLv3 — see LICENSE.