120 lines
6.0 KiB
Markdown
120 lines
6.0 KiB
Markdown
# REPORT — fleet unlocks: turn pool (529) + Redmine-claim dispatch (530)
|
||
|
||
- date: 2026-08-29T23:00:00Z (lane: dedicated fleet-unlocks lane, off loop rotation)
|
||
- repo: ukrrs/MOPAC harness @ main, pushed through `dca9b06`
|
||
- baseline: `a3d312c` → 5 commits, 17 files, +1550/−61
|
||
- verification: `./dev.sh check` green (build + vet + test, 12 packages ok) on the
|
||
final tree; pushed HEAD builds standalone
|
||
- Redmine writes during this lane: **NONE** (all testing against in-process fakes)
|
||
|
||
## Commits (in order, all pushed to origin/main)
|
||
|
||
| commit | what |
|
||
|---|---|
|
||
| `cfdfaf7` | 529 config knobs: `[loop] max_concurrent_turns` + `[loop.repo_map]` (+2 tests) |
|
||
| `8b27127` | 529 turn pool: semaphore + per-repo mutex, scan integration, heartbeat active/pending, fixes a REPORT tmp-file race the pool exposed (+7 tests) |
|
||
| `d71bc91` | 530 journals read primitive: `IssueJournals` via `?include=journals` (+1 test) |
|
||
| `82cfb01` | 530 claim dispatch: claim-before-turn, epoch race resolution, stale sweep, failure/park release (+9 tests) |
|
||
| `dca9b06` | fixup: pool.go reserve/launch refactor that `82cfb01` calls (see Incident note) |
|
||
|
||
### Incident note (transparency)
|
||
`82cfb01` was pushed with `daemon.go` staged but the matching `pool.go`
|
||
refactor left unstaged — pushed HEAD did not compile for ~4 minutes until
|
||
`dca9b06` landed (verified broken, then verified building after). No other
|
||
tree was affected; the final HEAD is build+check green.
|
||
|
||
## Part 1 — 529 turn pool
|
||
|
||
- `[loop] max_concurrent_turns` (default 1 = sequential v0; fleet 2-3).
|
||
Bounded worker pool: channel semaphore caps concurrent turns per scan;
|
||
tasks that don't fit stay **pending** and are reconsidered next scan
|
||
(never dropped, never marked processed).
|
||
- Per-repo mutex: one turn per work repo at a time, different repos in
|
||
parallel. Repo derived from `[loop.repo_map]` by task category (Class
|
||
field) or task-id prefix; unmapped tasks share the harness root
|
||
(work_root). Slot + repo lock are reserved *before* claiming/dispatch so
|
||
a claimed issue always gets its turn immediately.
|
||
- Scan-end barrier: a scan returns only after every dispatched turn
|
||
settles (dedup consistency, clean `--once` exits).
|
||
- status.json additions: `active_turns` (live), `pending` (per scan),
|
||
`max_concurrent_turns`.
|
||
- Bonus fix the pool exposed: concurrent REPORT writes into one directory
|
||
raced on a shared `.tmp` name (one rename stole another turn's file);
|
||
tmp names are now unique per write.
|
||
|
||
## Part 2 — 530 Redmine-claim dispatch
|
||
|
||
- `[redmine] claim_enabled` (default off = single-loop v0), `claim_status`
|
||
(default "In Progress"), `claimable_status` (default "New").
|
||
- On dispatch, BEFORE the turn: PUT status → In Progress + journal note
|
||
`claimed by <vertical>@<host> at <RFC3339>`.
|
||
- Race resolution (no CAS in the Redmine API): the **first claimant of the
|
||
current epoch** wins. Epoch = journal tail since the last boundary note
|
||
(a `claim released…` or `# REPORT -…` note). Losers see a foreign first
|
||
claim, clear their own junk claim with a concede note, and skip to the
|
||
next candidate. Boundaries keep history from blocking future claims.
|
||
- Success → existing `[redmine.status_map]` keyed on the claim status
|
||
("In Progress" → Resolved/Done). Failure/timeout/park → claim released
|
||
back to New + explanatory note (next scan retries; a release that can't
|
||
land because Redmine is down is backstopped by the stale sweep).
|
||
- Stale-claim release: claim older than **2× `turn_timeout_secs`** with no
|
||
REPORT after it → status back to New + release note, swept every scan.
|
||
Issues with no loop-claim note (humans moving cards) are never touched.
|
||
Turn timeout 0 disables the sweep.
|
||
- New JSONL events: `claim`, `claim_lost`, `release`.
|
||
|
||
## Tests (18 new, full suite green)
|
||
|
||
- pool (5): bound respected (peak concurrency == 2, backlog drains), per-repo
|
||
serialization (peak == 1 with pool 3), cross-repo parallelism
|
||
(deterministic rendezvous), repo_map mapping table, heartbeat counters.
|
||
- claim (9): two-loop race exactly-one-wins (both claims interleave via a
|
||
response-hold in the fake; asserts 1 turn, 1 REPORT, 1 concede), stale
|
||
release + redispatch, fresh-claim untouched, human-In-Progress untouched,
|
||
claim-write failure skips candidate (next candidate still dispatched),
|
||
lost-race skip + concede, turn-failure release + retry, epoch parsing table.
|
||
- writeback (2): journals decode (object-shaped user, note-less entries
|
||
skipped), concurrent REPORT writes in a shared dir.
|
||
- config (2): pool knobs parse/defaults/validation, claim knobs
|
||
parse/defaults/validation.
|
||
- Fixed one latent test race surfaced by mid-scan heartbeat beats
|
||
(TestLoopStatusFileHeartbeat now waits for a completed scan).
|
||
|
||
## Smoke evidence — live daemon, max_concurrent_turns=2, fake Redmine+LLM
|
||
|
||
Real `RunLoop` (`--once` scans), 3 tasks across 3 repos, pool 2:
|
||
|
||
```
|
||
scan 1: 3 task(s) in scope, 3 new/updated
|
||
dispatch 803 ... dispatch 801 <- two turns interleaved
|
||
REPORT 801 / REPORT 803 <- both land while scan holds
|
||
pool: 2 dispatched, 1 pending (max 2 turn(s) at once; pending reconsidered next scan)
|
||
scan 2: 1 new/updated -> dispatch 802 -> noted -> Done <- backlog drained
|
||
peak concurrent LLM requests: 2 (pool bound 2)
|
||
status.json: {"active_turns":0,"pending":0,"max_concurrent_turns":2,
|
||
"dispatched":1,"reports":1,"running":false,...}
|
||
```
|
||
|
||
Two-loop race test output evidence (from the suite): both verticals claim,
|
||
exactly one turn runs, loser logs `claim lost #700 ... skipping to next
|
||
candidate`, winner completes `In Progress -> Done`.
|
||
|
||
## Config knobs (fleet rollout)
|
||
|
||
```toml
|
||
[loop]
|
||
max_concurrent_turns = 2 # or 3
|
||
[loop.repo_map]
|
||
code = "/srv/work/harness"
|
||
study = "/srv/work/notes"
|
||
|
||
[redmine]
|
||
claim_enabled = true
|
||
claim_status = "In Progress" # default
|
||
claimable_status = "New" # default
|
||
```
|
||
|
||
Fleet loops: set both blocks; keep `[redmine.status_map] "In Progress" =
|
||
"Resolved"` (it now keys off the claim status). Single-loop hosts need no
|
||
config change (both features default off/1).
|