pmo: beta core verified+closed; repo production cleanup; loop owns queue

2026-08-29 16:07:57 -05:00
parent 2a65d7c585
commit 7b6d4454d5
3 changed files with 179 additions and 0 deletions
@@ -0,0 +1,151 @@
 # REPORT — Beta finish: 525 / 522 / 495 (2026-08-29 15:30 CST)
 Repo: `~/projects/meta/MOPAC/harness` (main). All work in the digest-pinned
 Docker builder via `./dev.sh`. `./dev.sh check` (build + vet + test ./...)
 ran GREEN immediately before EVERY commit below. Branch pushed:
 `19d61a0..f9fb729` on `origin/main`. Working tree clean at handoff.
 Note: no Redmine tickets were written from this session (PMO owns tickets),
 per instructions.
 ## Commit map
 | Commit | Part | Subject |
 |---|---|---|
 | `e8f50d3` | env chore | deploy: add host docker group to the loop container (uncommitted WIP found in tree; prerequisite for the Docker builders) |
 | `3927b38` | **525** | loop: retry Redmine writebacks with backoff, verify, and park on failure |
 | `5784698` | **522** | notify: native SMS notifier for loop events via stdlib SMTP |
 | `f9fb729` | **495** | brief: wire the 0630 pipeline - journal note, sections, daemon schedule |
 Net: 24 files changed, +2732 / -112.
 ## 525 — retry/backoff on status/note writes
 **Compile fix (first):** the loop turn's WIP did not build —
 `retry.go:74` field/method collision (`Backoff`), `redmine.go:186-210`
 `Attempts` used as a value, and the `[redmine] writeback_*` config fields
 did not exist yet. All fixed (method renamed `BackoffFor`; config knobs
 added to `RedmineConfig` + TOML parsing).
 **Finished behavior**
 - Bounded retries with exponential backoff + ±25% jitter on the note and
  status writes: `[redmine] writeback_retries = 3`, `writeback_backoff_secs
  = 2`, `writeback_max_backoff_secs = 30`. Retryable: 5xx, 429, transport
  (dial/reset/EOF). Never retried: 4xx, caller context cancel. Retried
  PUTs replay an identical body; every retry logs one audit line to the
  loop stdout.
 - **Verification re-read** after the writes: one GET confirms the status
  transition actually landed and advances the dedup marker in the same
  round trip (`IssueState`; supersedes the old refresh-only read).
 - **Park on final failure:** when the outage outlives the bound the task
  PARKS — issue status left exactly as-is (New stays New), one `park`
  event in loop.jsonl pins the dedup marker (no hot-loop), heartbeat
  `status.json` gains `last_writeback_error` + a `parked` counter
  (cleared by the next verified writeback), and the next PMO update of
  the issue re-releases the task. A paid-for REPORT is never silently
  stranded.
 **Tests (all green, fake-server based)**
 - `internal/writeback` — 14 tests total incl. new: 502-then-success
  (replayed body + audit line asserted), bounded give-up after exactly N
  attempts, 401 one-shot, verify re-read shape, RetryPolicyFrom
  defaults/overrides, BackoffFor jitter band, Retryable classification.
 - `internal/loop` — 502-flap recovery at the daemon level, park path
  (no note, no status move, no hot-loop, PMO update re-dispatches),
  `last_writeback_error`/`parked` surfaced in status.json.
 ## 522 — native SMS notifier
 **`internal/notify` (rewritten from the WIP sketch) + `[notify]` config**
 - Stdlib-only send (`net/smtp` via `NewClient` + `DialTimeout`, deadline-
  bounded so a black-holed relay cannot stall the sequential loop;
  optional PLAIN AUTH from refs). Recipients = carrier email-to-SMS
  gateway addresses, resolved from one ref.
 - Events wired in the daemon: turn complete, turn failed (incl. parked
  writebacks), quota/resource deferral, scan error, daemon start/stop.
 - Per-kind rate limit with **collapsed repeats**: repeats inside the
  window are counted and ride along on the next send
  ("+N suppressed repeat(s)"); no phone spam during a flapping scan.
 - **Offline queue on disk** (one JSON file per event, FIFO by name,
  bounded with oldest-dropped, corrupt-entry tolerant) drained after
  every scan and at daemon stop.
 - **Redaction rules** applied BEFORE queueing (so secrets are never at
  rest either): credential-shaped assignments (`token=`, `password:`,
  bearer/api-key/secret/auth) and key refs (`env:`/`file:`/`literal:`/
  `mpk:`/`bw:` values).
 - Disabled `[notify]` yields a nil notifier; all methods are no-ops, so
  call sites never gate. `once` never notifies (daemon-only surface).
 **Tests** — 7 in `internal/notify` against a local fake SMTP server
 (real SMTP dialogue: EHLO/AUTH/MAIL/RCPT/DATA/QUIT): send + headers +
 recipients, rate-limit collapse + follow-up count, offline queue drain,
 bounded queue oldest-drop, redaction (no secret bytes on the wire), nil
 no-op, AUTH path. Plus a loop-level wiring test (turn_complete +
 daemon_start fire and queue while SMTP is dead).
 ## 495 — 0630 briefing pipeline (surface existed at 064aed0; now wired)
 - **Render** grows the 0630 sections: one-line **executive summary**
  (shipped/moved/incoming/open · review chain · decisions needed ·
  quota pulse), per-vertical **open-now backlog table** (window-
  independent, capped at 15 rows) alongside the existing shipped/moved/
  incoming window tables, **decisions-needed** (open issues in Feedback/
  Needs-Decision statuses), **review-chain line** (open PR count + oldest
  age), **quota line from quota state** (unchanged section + exec pulse),
  and **system health** (loop heartbeat status.json: running/scans/
  dispatched/reports/parked/last_scan_error + host load/mem/disk).
 - **Delivery:** journal note on `[brief] redmine_issue` (the SoR copy,
  e.g. 495 — benefits from 525's retry on the note write) + Discourse
  topic when creds are present; missing creds = graceful logged skip
  (never a failed briefing when the SoR note landed). Legacy
  discourse-with-fallback path preserved when no redmine_issue is set.
  Marker records `redmine` / `discourse` / `redmine+discourse`.
 - **Daemon schedule hook:** the loop fires the brief daily at
  `[brief] schedule` (default **06:00 America/Chicago** — 06:00 render
  for the 0630 read; configurable HH:MM + timezone; "" = off; catch-up
  on late start). Injected via `loop.DailyHook` from the CLI to avoid an
  import cycle (brief already reads loop state). Idempotent per calendar
  day via the existing brief.jsonl marker; `harness brief` unchanged as
  the one-shot/cron surface.
 **Tests** — 16 in `internal/brief` (9 new): render section coverage
 (exec line / open-now / decisions / review chain / quota / health),
 classify open-backlog + decisions, health-file reading, Redmine-primary
 delivery (note content + graceful discourse skip), redmine+discourse
 both-land marker, redmine-fails-discourse-saves, full render against the
 fake Redmine; plus loop-side daily-hook tests (timezone math table +
 fires-exactly-once-per-day under a live daemon loop).
 ## Smoke evidence
 - `./dev.sh check` green before each commit (build + go vet + go test
  ./... — 12 packages ok, no failures; final run at commit `f9fb729`).
 - Live read-only smoke of the 495 render (dry-run, **no writes to
  Redmine**, no marker): authenticated GET of the real scope at
  projects.knownelement.com (fixed_version_id=35, status New — genuinely
  empty; the live loop heartbeat's "29 scans / 0 dispatched" corroborates),
  live system-health section reading this host's running daemon
  status.json. Transcript: `SMOKE-20260829-brief-dryrun.txt` beside this
  report.
 - 525/522 behaviors are exercised in-test against fake Redmine/SMTP
  servers (see test lists above); no live 502-injection or SMS send was
  performed against production systems.
 ## Deferred / open
 - **Live Discourse send** (495): vendored client is wired and fake-tested,
  but no live topic has been created — awaiting the 1900 CST creds
  session Charles offered (discourse key ref + category). Config slot is
  ready (`[brief] discourse_url` / `discourse_key_ref` / `category_slug`).
 - **Live SMS send** (522): fake-SMTP-tested; a real relay + carrier
  gateway addresses are needed to enable `[notify]` in production.
 - **Quota pulse depth** (495): quota line currently reads "gate off"
  until `[quota]` is enabled per instance (existing 490 surface, opt-in).
 - gofmt drift exists across the repo (pre-existing, incl. at 19d61a0);
  the check gate is build+vet+test, unchanged.
 ## Handoff
 - `git status` clean; `origin/main` at `f9fb729`.
 - New config surfaces documented in `harness.toml.example` ([redmine]
  writeback knobs, [notify], [brief]) and README tables/usage.
@@ -0,0 +1,27 @@
 # TASK: harness Beta finish — retry writebacks (525) + SMS notifier (522) + briefing pipeline (495)
 ## Context
 Redmine 525, 522, 495 — all in this repo (~/projects/meta/MOPAC/harness, main, last commit 19d61a0). Loop turns left UNCOMMITTED WIP that DOES NOT COMPILE. You are the consolidated finish turn. Three tickets, three surfaces, serial in one session.
 ## Current WIP state (verify with git status)
 - `internal/writeback/retry.go` (untracked) + `internal/writeback/redmine.go` (modified) — 525
 - `internal/notify/` (untracked) — 522
 - briefing surface EXISTS and is committed: 064aed0 + vendored discourse client — 495
 ## Part 1 — 525 retry/backoff (FIX COMPILE FIRST)
 Exact current errors: retry.go:74 field and method with the same name Backoff (other decl L35); redmine.go:186-210 attempts is func() int vs int (mismatched operations). Fix types, then finish per ticket: bounded retries w/ backoff on status/note writes, verification re-read after write, on final failure park task (leave New + local flag) and surface last_writeback_error in heartbeat status.json. Tests (fake server, incl. a 502-then-success case).
 ## Part 2 — 522 native notifier
 Finish internal/notify: [notify] config section, stdlib net/smtp send, rate-limit, collapsed repeats, offline queue (disk), redaction rules, events: turn complete/failed, quota deferral, scan error, daemon start/stop. Tests with a local fake SMTP server.
 ## Part 3 — 495 briefing pipeline
 Wire: render 0630 CST brief (executive line, per-vertical table from open issues, quota line from quota state, review-chain line, decisions-needed, system health) -> deliver as Redmine journal note on 495 + Discourse topic if creds present (graceful skip + log otherwise). Schedule hook in daemon (configurable time, default 06:00 CST render for 06:30 delivery). Test render against fake Redmine.
 ## Discipline (hard requirements)
 - Docker builders WORK in your environment (docker group fixed): `./dev.sh check` MUST be green before EACH commit.
 - One atomic conventional commit per part (3+ commits), then `git push origin main`.
 - No broad pkill. No new deps beyond stdlib (+ existing vendored).
 - Exit 0 only if all three parts are committed, pushed, check-green.
 ## Deliverable
 `REPORT-20260829-1530-beta-finish.md` in ~/.coordinate/inbox-pmo/: per-ticket commit hashes, test counts, smoke evidence, anything deferred (e.g. live Discourse send awaiting 1900 creds). Do NOT write to Redmine — PMO handles tickets.
@@ -104,3 +104,4 @@
 2026-08-29T15:05:15-05:00 | PMO | hb 15:02 ROOT CAUSE DAY: loop turns could NEVER build/test — docker.sock mounted but permission denied (root:988 vs uid 1001); 3 more false Resolves (525/522/495, 30/30 rounds, 4.3M prompt tokens, zero commits, retry.go still broken). FIXED: group_add DOCKER_GID to compose + recreated w/ full env (first recreate dropped HARNESS_UID=1001 -> crash loop, re-upped correct); docker server 29.7.2 reachable from loop container, healthy. Consolidated finish turn work-q18 dispatched on 5.2 (all 3 tickets, check-green-before-commit, push). Token-burn evidence -> 516. Loop idle (scope 0) until PMO refeeds post-q18
 2026-08-29T15:38:43-05:00 | PMO | CHARLES: full prod by Mon 8/31 0500 (versions: Beta due 8/29, Prod due 8/30 internal = buffer). 5 parallel lanes launched on distinct repos: 508 gitea-go, 502 linkwarden-go, 511 rest-driver, 512 cloudron-go, 497 keyproxy (fake-tested, live creds 1900). q18: 525+522 committed+pushed, 495 in progress. Harness-core queue (515/516/517/496/492/493) serial after q18. Wave 2 at 1900 wiring: 504/505/501/503. Wave 3 overnight: 500/509/510/513/514/507
 2026-08-29T16:02:19-05:00 | PMO | OPS-MANUAL.md written+committed (ukrrs/mopac-pmo), filed as Redmine 527 @Production w/ full text — Charles's post-teardown user guide: labeling semantics, Class->model routing, recipes (add/requeue/park/feedback/escalate/kill-switch), verification duty, secrets posture. Copy into harness/docs queued after q18
 2026-08-29T16:07:56-05:00 | PMO | BETA CORE DONE: q18 verified (3927b38 retry, 5784698 notify, f9fb729 brief pipeline; +2732/-112, check-green per commit) -> 525/522/495 Resolved+archived. Beta open: only 497 (loop building). mopac-pmo repo production-cleaned (56 logs/33 handoff/session cruft untracked; tracked = archive 40 + log + PROTOCOL + scripts; secrets never tracked). OPS-MANUAL -> harness/docs/OPERATIONS.md (0cb4742) per Charles. Loop scope now project-wide New (sed & trap fixed); 529 turn-pool RUNNING on loop (Go-native concurrency replaces bash/semaphore orchestration); 508/511 lanes finishing