docs: events receiver in README, phase 2b status in REPORT

README: dev.sh quickstart (digest-pinned builder), webhook receiver
section (routes, verification, config, smoke), event-path mermaid, CLI
flag table, [events] config rows, status rows. REPORT: phase 2b works/
stubbed list and phase 3 next chunk (event-to-turn dispatch wiring).
This commit is contained in:
2026-08-28 21:38:34 -05:00
parent 88e7b7b12e
commit f89b6945e3
2 changed files with 137 additions and 5 deletions
+79 -3
View File
@@ -8,8 +8,9 @@ re-invoking `harness once`. It is the execution core for the org's vertical
stacks: humans and other agents interact with a stack only through Redmine stacks: humans and other agents interact with a stack only through Redmine
(SoR), Discourse (docs) and Gitea (code), never by attaching to the loop. (SoR), Discourse (docs) and Gitea (code), never by attaching to the loop.
Status: 2026-08-28 — skeleton; MVP demo path live (single GLM turn through Status: 2026-08-28 — skeleton + event receiver; MVP demo path live (single
LiteLLM to REPORT, proven live 2026-08-28). GLM turn through LiteLLM to REPORT, proven live 2026-08-28); `harness
events` webhook receiver smoke-proven on LAN port 4100 the same day.
## Quickstart ## Quickstart
@@ -19,13 +20,25 @@ on 2026-08-28 from a fresh clone.
### Build and test ### Build and test
`dev.sh` routes every compile/vet/test path through the digest-pinned
builder container (the host stays toolchain-free):
```sh
./dev.sh check # = go build + go vet + go test, all inside the builder
```
The equivalent raw command:
```sh ```sh
docker run --rm -v "$PWD:/h" -w /h \ docker run --rm -v "$PWD:/h" -w /h \
-u "$(id -u):$(id -g)" -e HOME=/tmp \ -u "$(id -u):$(id -g)" -e HOME=/tmp \
golang:1.26-bookworm \ golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 \
sh -c 'go build -o bin/harness ./cmd/harness && go vet ./... && go test ./...' sh -c 'go build -o bin/harness ./cmd/harness && go vet ./... && go test ./...'
``` ```
(that digest = `golang:1.26-bookworm`; alpine lacks bash, which the exec
tool's tests need.)
Expected output (tail): Expected output (tail):
```text ```text
@@ -90,6 +103,37 @@ exit code 1 (config error). The with-key path was proven live on 2026-08-28:
the `[demo]` prompt "tell me about yourself" routed to `glm-5.3` via tier the `[demo]` prompt "tell me about yourself" routed to `glm-5.3` via tier
`mopac-primary` and the GLM self-description landed as the REPORT. `mopac-primary` and the GLM self-description landed as the REPORT.
### Webhook receiver (`harness events`)
Redmine / Discourse / Gitea webhooks punch the harness through the event
receiver. Configure `[events]` in `harness.toml` (secrets as env refs — see
`harness.toml.example`), then:
```sh
HARNESS_REDMINE_WEBHOOK_SECRET=... HARNESS_DISCOURSE_WEBHOOK_SECRET=... \
HARNESS_GITEA_WEBHOOK_SECRET=... ./dev.sh events
```
publishes port 4100 on the host LAN (override with `-listen`). Routes:
| Route | Verification | Maps to |
|---|---|---|
| `POST /hooks/redmine` | shared-secret header (default `X-Redmine-Webhook-Secret`) | `dispatch_turn` on issue update/note |
| `POST /hooks/discourse` | shared-secret header (default `X-Discourse-Webhook-Secret`) | `respond_turn` on post reply |
| `POST /hooks/gitea` | HMAC-SHA256 `X-Gitea-Signature` over the raw body | `pipeline_step` on PR approved/merged |
| `GET /healthz` | none | liveness |
Unsigned or unverified deliveries get 401; verified ones are normalized to
one internal event record, appended to `state/events/events.jsonl` (0600,
deduped by provider event id: `X-Gitea-Delivery` / `X-Discourse-Event-Id`,
payload-digest fallback), and handed to the conductor (turn dispatch is a
stub until phase 3 wiring). End-to-end smoke, including the 401/200 paths
and the JSONL record, driven with python urllib (curl is banned on host):
```sh
./dev.sh smoke
```
### Help ### Help
```sh ```sh
@@ -122,6 +166,23 @@ file plus `REPORT-latest.md` (atomic tmp+rename). On a mid-turn LLM failure
after content exists, a partial REPORT is still written with the error as the after content exists, a partial REPORT is still written with the error as the
stop reason. Chaining is by re-invocation — there is no daemon. stop reason. Chaining is by re-invocation — there is no daemon.
The event path is a second door into the same loop:
```mermaid
flowchart LR
RM["Redmine webhook"] --> V["VERIFY<br/>shared-secret / HMAC headers"]
DC["Discourse webhook"] --> V
GT["Gitea webhook"] --> V
V -->|401 unverified| X(["rejected"])
V --> N["NORMALIZE<br/>one Event record: source, kind, actor,<br/>subject id, digest, received-at"]
N --> S["STORE<br/>state/events/events.jsonl, append-only,<br/>dedup by provider event id"]
S -->|"dispatch_turn / respond_turn / pipeline_step"| C(["conductor (dispatch stub)"])
```
Verification rejects unsigned and unverified deliveries with 401 (no detail
beyond "unverified"); secret material and header values never reach logs or
the event log — only normalized fields and a payload digest do.
## CLI reference ## CLI reference
Subcommands (from `harness help`): Subcommands (from `harness help`):
@@ -130,6 +191,7 @@ Subcommands (from `harness help`):
|---|---| |---|---|
| `harness help` | print usage (also `-h`, `--help`) | | `harness help` | print usage (also `-h`, `--help`) |
| `harness once` | run ONE conductor iteration, then exit | | `harness once` | run ONE conductor iteration, then exit |
| `harness events` | run the webhook receiver until SIGINT/SIGTERM |
Flags for `once`: Flags for `once`:
@@ -140,6 +202,13 @@ Flags for `once`:
| `-demo` | run the `[demo]` issue instead of Redmine intake | | `-demo` | run the `[demo]` issue instead of Redmine intake |
| `-task-id ID` | run only the task/issue with this id | | `-task-id ID` | run only the task/issue with this id |
Flags for `events`:
| Flag | Meaning |
|---|---|
| `-config PATH` | config file (default `$HARNESS_CONFIG`, then `./harness.toml`) |
| `-listen ADDR` | bind address (overrides `[events]` listen) |
Exit codes: Exit codes:
| Code | Meaning | | Code | Meaning |
@@ -172,6 +241,11 @@ cannot rot):
| | `default` | verdict for commands matching no rule: `allow` or `deny` (org preset: deny) | | | `default` | verdict for commands matching no rule: `allow` or `deny` (org preset: deny) |
| | `allow`, `deny` | rule lists; deny beats allow; `cmd *` word-boundary, `pfx*` raw prefix, `pfx/**` path prefix, `*` universal; compound commands checked segment by segment; command substitution and subshells always denied | | | `allow`, `deny` | rule lists; deny beats allow; `cmd *` word-boundary, `pfx*` raw prefix, `pfx/**` path prefix, `*` universal; compound commands checked segment by segment; command substitution and subshells always denied |
| `[demo]` | `id`, `subject`, `prompt`, `class` | the issue `--demo` runs instead of Redmine intake | | `[demo]` | `id`, `subject`, `prompt`, `class` | the issue `--demo` runs instead of Redmine intake |
| `[events]` | `listen` | receiver bind address (default `:4100`; publish via `docker -p`) |
| | `state_dir` | event log dir (default `state/events`); `events.jsonl` created 0600 |
| `[events.redmine]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Redmine-Webhook-Secret`) |
| `[events.discourse]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Discourse-Webhook-Secret`) |
| `[events.gitea]` | `secret_ref` | HMAC secret ref; verification is always `X-Gitea-Signature` (hex HMAC-SHA256 of the raw body) |
Key refs accepted anywhere a `*_ref` appears: `env:NAME`, `file:PATH`, Key refs accepted anywhere a `*_ref` appears: `env:NAME`, `file:PATH`,
`literal:VALUE` (last resort), and `bw:REF` (reserved; errors until the `literal:VALUE` (last resort), and `bw:REF` (reserved; errors until the
@@ -221,6 +295,8 @@ Sourced from [REPORT.md](REPORT.md) — keep both in sync.
| Bounded turn | Works | LiteLLM chat, retry/backoff, tool loop capped at `max_rounds`, partial REPORT on mid-turn failure | | Bounded turn | Works | LiteLLM chat, retry/backoff, tool loop capped at `max_rounds`, partial REPORT on mid-turn failure |
| Exec tool | Works | allow-listed bash, segment-wise compound checks, timeout + truncation | | Exec tool | Works | allow-listed bash, segment-wise compound checks, timeout + truncation |
| REPORT writeback | Works | timestamped file + `REPORT-latest.md`, atomic, full telemetry | | REPORT writeback | Works | timestamped file + `REPORT-latest.md`, atomic, full telemetry |
| Event receiver | Works | `harness events`: verify (HMAC/shared-secret) → normalize → append-only JSONL with provider-id dedup → action mapping; smoke-proven on LAN port 4100 |
| Event → turn dispatch | Stubbed | conductor `DispatchEvent` prints what it would do; wiring is phase 3 |
| Tests | Works | table-driven, stdlib only; build/vet/test clean on go1.26 | | Tests | Works | table-driven, stdlib only; build/vet/test clean on go1.26 |
| Redmine note writeback | Stubbed | REPORT is file-only today | | Redmine note writeback | Stubbed | REPORT is file-only today |
| Budget/semaphore gate | Stubbed | tokens in REPORT, no cost/spend enforcement yet | | Budget/semaphore gate | Stubbed | tokens in REPORT, no cost/spend enforcement yet |
+58 -2
View File
@@ -61,6 +61,8 @@ without a key, so the live run needs exactly one thing:
## Stubbed / known gaps ## Stubbed / known gaps
- Event → turn dispatch: `harness events` stores + maps events and hands
them to `Conductor.DispatchEvent`, which is a printing stub (phase 3).
- Redmine issue-note writeback (SoR note after the REPORT) — file only. - Redmine issue-note writeback (SoR note after the REPORT) — file only.
- Budget/semaphore GATE (LiteLLM spend APIs, class-aware slots) and cost - Budget/semaphore GATE (LiteLLM spend APIs, class-aware slots) and cost
in REPORT (tokens only today). in REPORT (tokens only today).
@@ -73,11 +75,65 @@ without a key, so the live run needs exactly one thing:
transitions, so chained `once` re-picks the same issue (use --task-id). transitions, so chained `once` re-picks the same issue (use --task-id).
- Local inbox intake (DESIGN core-loop step 1, second half) — not started. - Local inbox intake (DESIGN core-loop step 1, second half) — not started.
## Build phase 2b — events receiver (`mopac events`) — 2026-08-28
Works:
- **`harness events` CLI**: stdlib net/http receiver (no frameworks), runs
until SIGINT/SIGTERM, graceful shutdown, `-config` / `-listen` flags.
Routes `POST /hooks/{redmine,discourse,gitea}` + `GET /healthz`; GET on a
hook = 405, unknown path = 404, oversized body (> 1 MiB) = 413.
- **Verification, deny-first**: gitea = hex HMAC-SHA256 of the raw body in
`X-Gitea-Signature` (constant-time `hmac.Equal`); redmine/discourse =
shared-secret header (constant-time compare; header names configurable,
defaults `X-Redmine-Webhook-Secret` / `X-Discourse-Webhook-Secret`).
Unsigned/unverified = 401 with a single generic error body; rejection
logs name the failure class only, never header values or secrets.
- **Normalization**: one internal Event (source, kind, actor, canonical
subject id `redmine:issue:42` / `discourse:topic:7` /
`gitea:pr:ukrrs/MOPAC#5`, title, repo, provider event id, payload
sha256 digest, received-at) — tolerant extraction across known payload
variants (redmine_webhooks + flat shapes, Discourse headers + payload,
Gitea action/pull_request/issue shapes incl. closed+merged → pr_merged).
- **Action mapping (DESIGN)**: redmine issue update/note/journal →
`dispatch_turn`; discourse post reply → `respond_turn`; gitea PR
approved/merged → `pipeline_step`; everything else stored-but-ignored.
- **Persistence**: append-only `state/events/events.jsonl` (0600, dir
0700), dedup by provider event id (`X-Gitea-Delivery`,
`X-Discourse-Event-Id`, `X-Redmine-Delivery` when present; payload
digest fallback). Dedup index rebuilt from the file at startup (torn
tail line skipped), so replays across restarts still dedup —
at-least-once delivery, exactly-once reaction.
- **Config**: `[events]` listen/state_dir + per-source secret refs
(`env:`/`file:`/`literal:`, resolved at startup; server refuses to start
with zero secrets). Full validation, ref values never echoed.
- **dev.sh**: every build/vet/test/run path routes through the
digest-pinned builder `golang@sha256:e8c859f...` (= golang:1.26-bookworm;
alpine has no bash for the exec tool's tests).
- **Tests**: table-driven — signature verification (7 HMAC + 5 shared
secret cases), normalization (13 payload/shape cases + action map),
store dedup/restart/torn-tail/permissions, end-to-end httptest (401/400/
200 paths, replay dedup, ignore-not-dispatched, oversized body, log/JSONL
secret-leak assertions). Docker `build/vet/test` clean.
- **Smoke (live, LAN port 4100)**: containerized receiver driven by host
python3 urllib (curl banned on host): unsigned → 401, bad HMAC → 401,
wrong shared secret → 401, valid → 200 `stored` + JSONL line, replay →
200 `duplicate`, per-provider actions correct, stub dispatch fired once
per stored actionable event. `./dev.sh smoke` reproduces it end-to-end.
Stubbed:
- Event → turn dispatch (`Conductor.DispatchEvent` prints the would-be
action; real wiring lands in phase 3 per plan).
- Cross-host/event forwarding (T1 replication later; JSONL is local-only).
## Next chunk (phase 3) ## Next chunk (phase 3)
1. bitwarden-go wrapper + `bw:` refs (unblocks secret posture). 1. bitwarden-go wrapper + `bw:` refs (unblocks secret posture).
2. Full permission layer: tree-sitter bash scopes, write roots, symlink 2. Full permission layer: tree-sitter bash scopes, write roots, symlink
checks, per-vertical allow/deny presets. checks, per-vertical allow/deny presets.
3. Redmine note writeback + status transition so chaining advances scope. 3. Redmine note writeback + status transition so chaining advances scope.
4. Budget gate via LiteLLM spend APIs; cost line in REPORT. 4. Event → turn dispatch wiring: stored actionable events actually chain
5. Streaming with truncation retry + turn resume. conductor iterations for the affected stack.
5. Budget gate via LiteLLM spend APIs; cost line in REPORT.
6. Streaming with truncation retry + turn resume.