From f89b6945e3a872deba81d4598315587dd4f26f88 Mon Sep 17 00:00:00 2001 From: reachableceo Date: Fri, 28 Aug 2026 21:38:34 -0500 Subject: [PATCH] docs: events receiver in README, phase 2b status in REPORT README: dev.sh quickstart (digest-pinned builder), webhook receiver section (routes, verification, config, smoke), event-path mermaid, CLI flag table, [events] config rows, status rows. REPORT: phase 2b works/ stubbed list and phase 3 next chunk (event-to-turn dispatch wiring). --- README.md | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- REPORT.md | 60 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 137 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 20722d0..36ea0d0 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,9 @@ re-invoking `harness once`. It is the execution core for the org's vertical stacks: humans and other agents interact with a stack only through Redmine (SoR), Discourse (docs) and Gitea (code), never by attaching to the loop. -Status: 2026-08-28 — skeleton; MVP demo path live (single GLM turn through -LiteLLM to REPORT, proven live 2026-08-28). +Status: 2026-08-28 — skeleton + event receiver; MVP demo path live (single +GLM turn through LiteLLM to REPORT, proven live 2026-08-28); `harness +events` webhook receiver smoke-proven on LAN port 4100 the same day. ## Quickstart @@ -19,13 +20,25 @@ on 2026-08-28 from a fresh clone. ### Build and test +`dev.sh` routes every compile/vet/test path through the digest-pinned +builder container (the host stays toolchain-free): + +```sh +./dev.sh check # = go build + go vet + go test, all inside the builder +``` + +The equivalent raw command: + ```sh docker run --rm -v "$PWD:/h" -w /h \ -u "$(id -u):$(id -g)" -e HOME=/tmp \ - golang:1.26-bookworm \ + golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 \ sh -c 'go build -o bin/harness ./cmd/harness && go vet ./... && go test ./...' ``` +(that digest = `golang:1.26-bookworm`; alpine lacks bash, which the exec +tool's tests need.) + Expected output (tail): ```text @@ -90,6 +103,37 @@ exit code 1 (config error). The with-key path was proven live on 2026-08-28: the `[demo]` prompt "tell me about yourself" routed to `glm-5.3` via tier `mopac-primary` and the GLM self-description landed as the REPORT. +### Webhook receiver (`harness events`) + +Redmine / Discourse / Gitea webhooks punch the harness through the event +receiver. Configure `[events]` in `harness.toml` (secrets as env refs — see +`harness.toml.example`), then: + +```sh +HARNESS_REDMINE_WEBHOOK_SECRET=... HARNESS_DISCOURSE_WEBHOOK_SECRET=... \ +HARNESS_GITEA_WEBHOOK_SECRET=... ./dev.sh events +``` + +publishes port 4100 on the host LAN (override with `-listen`). Routes: + +| Route | Verification | Maps to | +|---|---|---| +| `POST /hooks/redmine` | shared-secret header (default `X-Redmine-Webhook-Secret`) | `dispatch_turn` on issue update/note | +| `POST /hooks/discourse` | shared-secret header (default `X-Discourse-Webhook-Secret`) | `respond_turn` on post reply | +| `POST /hooks/gitea` | HMAC-SHA256 `X-Gitea-Signature` over the raw body | `pipeline_step` on PR approved/merged | +| `GET /healthz` | none | liveness | + +Unsigned or unverified deliveries get 401; verified ones are normalized to +one internal event record, appended to `state/events/events.jsonl` (0600, +deduped by provider event id: `X-Gitea-Delivery` / `X-Discourse-Event-Id`, +payload-digest fallback), and handed to the conductor (turn dispatch is a +stub until phase 3 wiring). End-to-end smoke, including the 401/200 paths +and the JSONL record, driven with python urllib (curl is banned on host): + +```sh +./dev.sh smoke +``` + ### Help ```sh @@ -122,6 +166,23 @@ file plus `REPORT-latest.md` (atomic tmp+rename). On a mid-turn LLM failure after content exists, a partial REPORT is still written with the error as the stop reason. Chaining is by re-invocation — there is no daemon. +The event path is a second door into the same loop: + +```mermaid +flowchart LR + RM["Redmine webhook"] --> V["VERIFY
shared-secret / HMAC headers"] + DC["Discourse webhook"] --> V + GT["Gitea webhook"] --> V + V -->|401 unverified| X(["rejected"]) + V --> N["NORMALIZE
one Event record: source, kind, actor,
subject id, digest, received-at"] + N --> S["STORE
state/events/events.jsonl, append-only,
dedup by provider event id"] + S -->|"dispatch_turn / respond_turn / pipeline_step"| C(["conductor (dispatch stub)"]) +``` + +Verification rejects unsigned and unverified deliveries with 401 (no detail +beyond "unverified"); secret material and header values never reach logs or +the event log — only normalized fields and a payload digest do. + ## CLI reference Subcommands (from `harness help`): @@ -130,6 +191,7 @@ Subcommands (from `harness help`): |---|---| | `harness help` | print usage (also `-h`, `--help`) | | `harness once` | run ONE conductor iteration, then exit | +| `harness events` | run the webhook receiver until SIGINT/SIGTERM | Flags for `once`: @@ -140,6 +202,13 @@ Flags for `once`: | `-demo` | run the `[demo]` issue instead of Redmine intake | | `-task-id ID` | run only the task/issue with this id | +Flags for `events`: + +| Flag | Meaning | +|---|---| +| `-config PATH` | config file (default `$HARNESS_CONFIG`, then `./harness.toml`) | +| `-listen ADDR` | bind address (overrides `[events]` listen) | + Exit codes: | Code | Meaning | @@ -172,6 +241,11 @@ cannot rot): | | `default` | verdict for commands matching no rule: `allow` or `deny` (org preset: deny) | | | `allow`, `deny` | rule lists; deny beats allow; `cmd *` word-boundary, `pfx*` raw prefix, `pfx/**` path prefix, `*` universal; compound commands checked segment by segment; command substitution and subshells always denied | | `[demo]` | `id`, `subject`, `prompt`, `class` | the issue `--demo` runs instead of Redmine intake | +| `[events]` | `listen` | receiver bind address (default `:4100`; publish via `docker -p`) | +| | `state_dir` | event log dir (default `state/events`); `events.jsonl` created 0600 | +| `[events.redmine]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Redmine-Webhook-Secret`) | +| `[events.discourse]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Discourse-Webhook-Secret`) | +| `[events.gitea]` | `secret_ref` | HMAC secret ref; verification is always `X-Gitea-Signature` (hex HMAC-SHA256 of the raw body) | Key refs accepted anywhere a `*_ref` appears: `env:NAME`, `file:PATH`, `literal:VALUE` (last resort), and `bw:REF` (reserved; errors until the @@ -221,6 +295,8 @@ Sourced from [REPORT.md](REPORT.md) — keep both in sync. | Bounded turn | Works | LiteLLM chat, retry/backoff, tool loop capped at `max_rounds`, partial REPORT on mid-turn failure | | Exec tool | Works | allow-listed bash, segment-wise compound checks, timeout + truncation | | REPORT writeback | Works | timestamped file + `REPORT-latest.md`, atomic, full telemetry | +| Event receiver | Works | `harness events`: verify (HMAC/shared-secret) → normalize → append-only JSONL with provider-id dedup → action mapping; smoke-proven on LAN port 4100 | +| Event → turn dispatch | Stubbed | conductor `DispatchEvent` prints what it would do; wiring is phase 3 | | Tests | Works | table-driven, stdlib only; build/vet/test clean on go1.26 | | Redmine note writeback | Stubbed | REPORT is file-only today | | Budget/semaphore gate | Stubbed | tokens in REPORT, no cost/spend enforcement yet | diff --git a/REPORT.md b/REPORT.md index 54ef22a..d5260c7 100644 --- a/REPORT.md +++ b/REPORT.md @@ -61,6 +61,8 @@ without a key, so the live run needs exactly one thing: ## Stubbed / known gaps +- Event → turn dispatch: `harness events` stores + maps events and hands + them to `Conductor.DispatchEvent`, which is a printing stub (phase 3). - Redmine issue-note writeback (SoR note after the REPORT) — file only. - Budget/semaphore GATE (LiteLLM spend APIs, class-aware slots) and cost in REPORT (tokens only today). @@ -73,11 +75,65 @@ without a key, so the live run needs exactly one thing: transitions, so chained `once` re-picks the same issue (use --task-id). - Local inbox intake (DESIGN core-loop step 1, second half) — not started. +## Build phase 2b — events receiver (`mopac events`) — 2026-08-28 + +Works: + +- **`harness events` CLI**: stdlib net/http receiver (no frameworks), runs + until SIGINT/SIGTERM, graceful shutdown, `-config` / `-listen` flags. + Routes `POST /hooks/{redmine,discourse,gitea}` + `GET /healthz`; GET on a + hook = 405, unknown path = 404, oversized body (> 1 MiB) = 413. +- **Verification, deny-first**: gitea = hex HMAC-SHA256 of the raw body in + `X-Gitea-Signature` (constant-time `hmac.Equal`); redmine/discourse = + shared-secret header (constant-time compare; header names configurable, + defaults `X-Redmine-Webhook-Secret` / `X-Discourse-Webhook-Secret`). + Unsigned/unverified = 401 with a single generic error body; rejection + logs name the failure class only, never header values or secrets. +- **Normalization**: one internal Event (source, kind, actor, canonical + subject id `redmine:issue:42` / `discourse:topic:7` / + `gitea:pr:ukrrs/MOPAC#5`, title, repo, provider event id, payload + sha256 digest, received-at) — tolerant extraction across known payload + variants (redmine_webhooks + flat shapes, Discourse headers + payload, + Gitea action/pull_request/issue shapes incl. closed+merged → pr_merged). +- **Action mapping (DESIGN)**: redmine issue update/note/journal → + `dispatch_turn`; discourse post reply → `respond_turn`; gitea PR + approved/merged → `pipeline_step`; everything else stored-but-ignored. +- **Persistence**: append-only `state/events/events.jsonl` (0600, dir + 0700), dedup by provider event id (`X-Gitea-Delivery`, + `X-Discourse-Event-Id`, `X-Redmine-Delivery` when present; payload + digest fallback). Dedup index rebuilt from the file at startup (torn + tail line skipped), so replays across restarts still dedup — + at-least-once delivery, exactly-once reaction. +- **Config**: `[events]` listen/state_dir + per-source secret refs + (`env:`/`file:`/`literal:`, resolved at startup; server refuses to start + with zero secrets). Full validation, ref values never echoed. +- **dev.sh**: every build/vet/test/run path routes through the + digest-pinned builder `golang@sha256:e8c859f...` (= golang:1.26-bookworm; + alpine has no bash for the exec tool's tests). +- **Tests**: table-driven — signature verification (7 HMAC + 5 shared + secret cases), normalization (13 payload/shape cases + action map), + store dedup/restart/torn-tail/permissions, end-to-end httptest (401/400/ + 200 paths, replay dedup, ignore-not-dispatched, oversized body, log/JSONL + secret-leak assertions). Docker `build/vet/test` clean. +- **Smoke (live, LAN port 4100)**: containerized receiver driven by host + python3 urllib (curl banned on host): unsigned → 401, bad HMAC → 401, + wrong shared secret → 401, valid → 200 `stored` + JSONL line, replay → + 200 `duplicate`, per-provider actions correct, stub dispatch fired once + per stored actionable event. `./dev.sh smoke` reproduces it end-to-end. + +Stubbed: + +- Event → turn dispatch (`Conductor.DispatchEvent` prints the would-be + action; real wiring lands in phase 3 per plan). +- Cross-host/event forwarding (T1 replication later; JSONL is local-only). + ## Next chunk (phase 3) 1. bitwarden-go wrapper + `bw:` refs (unblocks secret posture). 2. Full permission layer: tree-sitter bash scopes, write roots, symlink checks, per-vertical allow/deny presets. 3. Redmine note writeback + status transition so chaining advances scope. -4. Budget gate via LiteLLM spend APIs; cost line in REPORT. -5. Streaming with truncation retry + turn resume. +4. Event → turn dispatch wiring: stored actionable events actually chain + conductor iterations for the affected stack. +5. Budget gate via LiteLLM spend APIs; cost line in REPORT. +6. Streaming with truncation retry + turn resume.