docs: events receiver in README, phase 2b status in REPORT
README: dev.sh quickstart (digest-pinned builder), webhook receiver section (routes, verification, config, smoke), event-path mermaid, CLI flag table, [events] config rows, status rows. REPORT: phase 2b works/ stubbed list and phase 3 next chunk (event-to-turn dispatch wiring).
This commit is contained in:
@@ -8,8 +8,9 @@ re-invoking `harness once`. It is the execution core for the org's vertical
|
||||
stacks: humans and other agents interact with a stack only through Redmine
|
||||
(SoR), Discourse (docs) and Gitea (code), never by attaching to the loop.
|
||||
|
||||
Status: 2026-08-28 — skeleton; MVP demo path live (single GLM turn through
|
||||
LiteLLM to REPORT, proven live 2026-08-28).
|
||||
Status: 2026-08-28 — skeleton + event receiver; MVP demo path live (single
|
||||
GLM turn through LiteLLM to REPORT, proven live 2026-08-28); `harness
|
||||
events` webhook receiver smoke-proven on LAN port 4100 the same day.
|
||||
|
||||
## Quickstart
|
||||
|
||||
@@ -19,13 +20,25 @@ on 2026-08-28 from a fresh clone.
|
||||
|
||||
### Build and test
|
||||
|
||||
`dev.sh` routes every compile/vet/test path through the digest-pinned
|
||||
builder container (the host stays toolchain-free):
|
||||
|
||||
```sh
|
||||
./dev.sh check # = go build + go vet + go test, all inside the builder
|
||||
```
|
||||
|
||||
The equivalent raw command:
|
||||
|
||||
```sh
|
||||
docker run --rm -v "$PWD:/h" -w /h \
|
||||
-u "$(id -u):$(id -g)" -e HOME=/tmp \
|
||||
golang:1.26-bookworm \
|
||||
golang@sha256:e8c859f5632dcfde7b32d2012b4351728f6437930887c2f6a91ea242459e5514 \
|
||||
sh -c 'go build -o bin/harness ./cmd/harness && go vet ./... && go test ./...'
|
||||
```
|
||||
|
||||
(that digest = `golang:1.26-bookworm`; alpine lacks bash, which the exec
|
||||
tool's tests need.)
|
||||
|
||||
Expected output (tail):
|
||||
|
||||
```text
|
||||
@@ -90,6 +103,37 @@ exit code 1 (config error). The with-key path was proven live on 2026-08-28:
|
||||
the `[demo]` prompt "tell me about yourself" routed to `glm-5.3` via tier
|
||||
`mopac-primary` and the GLM self-description landed as the REPORT.
|
||||
|
||||
### Webhook receiver (`harness events`)
|
||||
|
||||
Redmine / Discourse / Gitea webhooks punch the harness through the event
|
||||
receiver. Configure `[events]` in `harness.toml` (secrets as env refs — see
|
||||
`harness.toml.example`), then:
|
||||
|
||||
```sh
|
||||
HARNESS_REDMINE_WEBHOOK_SECRET=... HARNESS_DISCOURSE_WEBHOOK_SECRET=... \
|
||||
HARNESS_GITEA_WEBHOOK_SECRET=... ./dev.sh events
|
||||
```
|
||||
|
||||
publishes port 4100 on the host LAN (override with `-listen`). Routes:
|
||||
|
||||
| Route | Verification | Maps to |
|
||||
|---|---|---|
|
||||
| `POST /hooks/redmine` | shared-secret header (default `X-Redmine-Webhook-Secret`) | `dispatch_turn` on issue update/note |
|
||||
| `POST /hooks/discourse` | shared-secret header (default `X-Discourse-Webhook-Secret`) | `respond_turn` on post reply |
|
||||
| `POST /hooks/gitea` | HMAC-SHA256 `X-Gitea-Signature` over the raw body | `pipeline_step` on PR approved/merged |
|
||||
| `GET /healthz` | none | liveness |
|
||||
|
||||
Unsigned or unverified deliveries get 401; verified ones are normalized to
|
||||
one internal event record, appended to `state/events/events.jsonl` (0600,
|
||||
deduped by provider event id: `X-Gitea-Delivery` / `X-Discourse-Event-Id`,
|
||||
payload-digest fallback), and handed to the conductor (turn dispatch is a
|
||||
stub until phase 3 wiring). End-to-end smoke, including the 401/200 paths
|
||||
and the JSONL record, driven with python urllib (curl is banned on host):
|
||||
|
||||
```sh
|
||||
./dev.sh smoke
|
||||
```
|
||||
|
||||
### Help
|
||||
|
||||
```sh
|
||||
@@ -122,6 +166,23 @@ file plus `REPORT-latest.md` (atomic tmp+rename). On a mid-turn LLM failure
|
||||
after content exists, a partial REPORT is still written with the error as the
|
||||
stop reason. Chaining is by re-invocation — there is no daemon.
|
||||
|
||||
The event path is a second door into the same loop:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
RM["Redmine webhook"] --> V["VERIFY<br/>shared-secret / HMAC headers"]
|
||||
DC["Discourse webhook"] --> V
|
||||
GT["Gitea webhook"] --> V
|
||||
V -->|401 unverified| X(["rejected"])
|
||||
V --> N["NORMALIZE<br/>one Event record: source, kind, actor,<br/>subject id, digest, received-at"]
|
||||
N --> S["STORE<br/>state/events/events.jsonl, append-only,<br/>dedup by provider event id"]
|
||||
S -->|"dispatch_turn / respond_turn / pipeline_step"| C(["conductor (dispatch stub)"])
|
||||
```
|
||||
|
||||
Verification rejects unsigned and unverified deliveries with 401 (no detail
|
||||
beyond "unverified"); secret material and header values never reach logs or
|
||||
the event log — only normalized fields and a payload digest do.
|
||||
|
||||
## CLI reference
|
||||
|
||||
Subcommands (from `harness help`):
|
||||
@@ -130,6 +191,7 @@ Subcommands (from `harness help`):
|
||||
|---|---|
|
||||
| `harness help` | print usage (also `-h`, `--help`) |
|
||||
| `harness once` | run ONE conductor iteration, then exit |
|
||||
| `harness events` | run the webhook receiver until SIGINT/SIGTERM |
|
||||
|
||||
Flags for `once`:
|
||||
|
||||
@@ -140,6 +202,13 @@ Flags for `once`:
|
||||
| `-demo` | run the `[demo]` issue instead of Redmine intake |
|
||||
| `-task-id ID` | run only the task/issue with this id |
|
||||
|
||||
Flags for `events`:
|
||||
|
||||
| Flag | Meaning |
|
||||
|---|---|
|
||||
| `-config PATH` | config file (default `$HARNESS_CONFIG`, then `./harness.toml`) |
|
||||
| `-listen ADDR` | bind address (overrides `[events]` listen) |
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
@@ -172,6 +241,11 @@ cannot rot):
|
||||
| | `default` | verdict for commands matching no rule: `allow` or `deny` (org preset: deny) |
|
||||
| | `allow`, `deny` | rule lists; deny beats allow; `cmd *` word-boundary, `pfx*` raw prefix, `pfx/**` path prefix, `*` universal; compound commands checked segment by segment; command substitution and subshells always denied |
|
||||
| `[demo]` | `id`, `subject`, `prompt`, `class` | the issue `--demo` runs instead of Redmine intake |
|
||||
| `[events]` | `listen` | receiver bind address (default `:4100`; publish via `docker -p`) |
|
||||
| | `state_dir` | event log dir (default `state/events`); `events.jsonl` created 0600 |
|
||||
| `[events.redmine]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Redmine-Webhook-Secret`) |
|
||||
| `[events.discourse]` | `secret_ref`, `secret_header` | shared-secret ref + header name (default `X-Discourse-Webhook-Secret`) |
|
||||
| `[events.gitea]` | `secret_ref` | HMAC secret ref; verification is always `X-Gitea-Signature` (hex HMAC-SHA256 of the raw body) |
|
||||
|
||||
Key refs accepted anywhere a `*_ref` appears: `env:NAME`, `file:PATH`,
|
||||
`literal:VALUE` (last resort), and `bw:REF` (reserved; errors until the
|
||||
@@ -221,6 +295,8 @@ Sourced from [REPORT.md](REPORT.md) — keep both in sync.
|
||||
| Bounded turn | Works | LiteLLM chat, retry/backoff, tool loop capped at `max_rounds`, partial REPORT on mid-turn failure |
|
||||
| Exec tool | Works | allow-listed bash, segment-wise compound checks, timeout + truncation |
|
||||
| REPORT writeback | Works | timestamped file + `REPORT-latest.md`, atomic, full telemetry |
|
||||
| Event receiver | Works | `harness events`: verify (HMAC/shared-secret) → normalize → append-only JSONL with provider-id dedup → action mapping; smoke-proven on LAN port 4100 |
|
||||
| Event → turn dispatch | Stubbed | conductor `DispatchEvent` prints what it would do; wiring is phase 3 |
|
||||
| Tests | Works | table-driven, stdlib only; build/vet/test clean on go1.26 |
|
||||
| Redmine note writeback | Stubbed | REPORT is file-only today |
|
||||
| Budget/semaphore gate | Stubbed | tokens in REPORT, no cost/spend enforcement yet |
|
||||
|
||||
@@ -61,6 +61,8 @@ without a key, so the live run needs exactly one thing:
|
||||
|
||||
## Stubbed / known gaps
|
||||
|
||||
- Event → turn dispatch: `harness events` stores + maps events and hands
|
||||
them to `Conductor.DispatchEvent`, which is a printing stub (phase 3).
|
||||
- Redmine issue-note writeback (SoR note after the REPORT) — file only.
|
||||
- Budget/semaphore GATE (LiteLLM spend APIs, class-aware slots) and cost
|
||||
in REPORT (tokens only today).
|
||||
@@ -73,11 +75,65 @@ without a key, so the live run needs exactly one thing:
|
||||
transitions, so chained `once` re-picks the same issue (use --task-id).
|
||||
- Local inbox intake (DESIGN core-loop step 1, second half) — not started.
|
||||
|
||||
## Build phase 2b — events receiver (`mopac events`) — 2026-08-28
|
||||
|
||||
Works:
|
||||
|
||||
- **`harness events` CLI**: stdlib net/http receiver (no frameworks), runs
|
||||
until SIGINT/SIGTERM, graceful shutdown, `-config` / `-listen` flags.
|
||||
Routes `POST /hooks/{redmine,discourse,gitea}` + `GET /healthz`; GET on a
|
||||
hook = 405, unknown path = 404, oversized body (> 1 MiB) = 413.
|
||||
- **Verification, deny-first**: gitea = hex HMAC-SHA256 of the raw body in
|
||||
`X-Gitea-Signature` (constant-time `hmac.Equal`); redmine/discourse =
|
||||
shared-secret header (constant-time compare; header names configurable,
|
||||
defaults `X-Redmine-Webhook-Secret` / `X-Discourse-Webhook-Secret`).
|
||||
Unsigned/unverified = 401 with a single generic error body; rejection
|
||||
logs name the failure class only, never header values or secrets.
|
||||
- **Normalization**: one internal Event (source, kind, actor, canonical
|
||||
subject id `redmine:issue:42` / `discourse:topic:7` /
|
||||
`gitea:pr:ukrrs/MOPAC#5`, title, repo, provider event id, payload
|
||||
sha256 digest, received-at) — tolerant extraction across known payload
|
||||
variants (redmine_webhooks + flat shapes, Discourse headers + payload,
|
||||
Gitea action/pull_request/issue shapes incl. closed+merged → pr_merged).
|
||||
- **Action mapping (DESIGN)**: redmine issue update/note/journal →
|
||||
`dispatch_turn`; discourse post reply → `respond_turn`; gitea PR
|
||||
approved/merged → `pipeline_step`; everything else stored-but-ignored.
|
||||
- **Persistence**: append-only `state/events/events.jsonl` (0600, dir
|
||||
0700), dedup by provider event id (`X-Gitea-Delivery`,
|
||||
`X-Discourse-Event-Id`, `X-Redmine-Delivery` when present; payload
|
||||
digest fallback). Dedup index rebuilt from the file at startup (torn
|
||||
tail line skipped), so replays across restarts still dedup —
|
||||
at-least-once delivery, exactly-once reaction.
|
||||
- **Config**: `[events]` listen/state_dir + per-source secret refs
|
||||
(`env:`/`file:`/`literal:`, resolved at startup; server refuses to start
|
||||
with zero secrets). Full validation, ref values never echoed.
|
||||
- **dev.sh**: every build/vet/test/run path routes through the
|
||||
digest-pinned builder `golang@sha256:e8c859f...` (= golang:1.26-bookworm;
|
||||
alpine has no bash for the exec tool's tests).
|
||||
- **Tests**: table-driven — signature verification (7 HMAC + 5 shared
|
||||
secret cases), normalization (13 payload/shape cases + action map),
|
||||
store dedup/restart/torn-tail/permissions, end-to-end httptest (401/400/
|
||||
200 paths, replay dedup, ignore-not-dispatched, oversized body, log/JSONL
|
||||
secret-leak assertions). Docker `build/vet/test` clean.
|
||||
- **Smoke (live, LAN port 4100)**: containerized receiver driven by host
|
||||
python3 urllib (curl banned on host): unsigned → 401, bad HMAC → 401,
|
||||
wrong shared secret → 401, valid → 200 `stored` + JSONL line, replay →
|
||||
200 `duplicate`, per-provider actions correct, stub dispatch fired once
|
||||
per stored actionable event. `./dev.sh smoke` reproduces it end-to-end.
|
||||
|
||||
Stubbed:
|
||||
|
||||
- Event → turn dispatch (`Conductor.DispatchEvent` prints the would-be
|
||||
action; real wiring lands in phase 3 per plan).
|
||||
- Cross-host/event forwarding (T1 replication later; JSONL is local-only).
|
||||
|
||||
## Next chunk (phase 3)
|
||||
|
||||
1. bitwarden-go wrapper + `bw:` refs (unblocks secret posture).
|
||||
2. Full permission layer: tree-sitter bash scopes, write roots, symlink
|
||||
checks, per-vertical allow/deny presets.
|
||||
3. Redmine note writeback + status transition so chaining advances scope.
|
||||
4. Budget gate via LiteLLM spend APIs; cost line in REPORT.
|
||||
5. Streaming with truncation retry + turn resume.
|
||||
4. Event → turn dispatch wiring: stored actionable events actually chain
|
||||
conductor iterations for the affected stack.
|
||||
5. Budget gate via LiteLLM spend APIs; cost line in REPORT.
|
||||
6. Streaming with truncation retry + turn resume.
|
||||
|
||||
Reference in New Issue
Block a user