diff --git a/docs/PORTING-NOTES-maki.md b/docs/PORTING-NOTES-maki.md new file mode 100644 index 0000000..581fb00 --- /dev/null +++ b/docs/PORTING-NOTES-maki.md @@ -0,0 +1,180 @@ +# Porting notes: maki (reference/maki @ plugins/ + maki-agent/) + +Scope for MOPAC harness: tree-sitter bash permission parsing, token-reduction +(index / MCP tool_search), subagent tiers, headless loop. Paths relative to +`reference/maki/`; line refs from study 2026-08-28, drift possible. Rust core ++ Lua tool plugins; Lua API mirrors tree-sitter via maki-lua. + +## 1. Tree-sitter bash permission parsing + +Two halves: Lua plugin extracts "scopes" from command text; Rust +PermissionManager matches them against allow/deny rules. + +**Scope extraction** — `plugins/bash/init.lua:275-296` (`permission_scopes`): +- Parse `command` with tree-sitter-bash. Parse error (`root:has_error()`) + OR any node (recursive) of type `command_substitution | + process_substitution | subshell | arithmetic_expansion` (is_complex, + :176-193) -> `{scopes=[whole command], force_prompt=true}`. Never + decompose `$(...)`; force prompt instead. Else `collect_commands` + (:209-235): `program`/`list` -> recurse (flattens `&&`/`||`/`;`); + `pipeline` -> one scope per named child; leaf types `command, + redirected_statement, negated_command, subshell, compound_statement, + if/while/for/case, function_definition, c_style_for_statement` -> + trimmed node text. `git diff && rm -rf /` -> ["git diff","rm -rf /"]; + `a | b` -> [a, b]. `cd X && cmd` -> workdir+cmd hint (:37-49); the + `workdir` input param is preferred (description enforces). + +**Rule engine** — `maki-agent/src/permissions.rs`: +- `check_inner` (:308-417): per scope, iterate rules session -> config -> + builtin -> plugin. ANY matching deny => whole call Denied immediately. + Matching allow => covered. Uncovered => NeedsPrompt (or per-tool/global + default Deny|Allow|Prompt). `force_prompt` => prompt all scopes despite + allows. Yolo => Allowed but only AFTER deny scan (:356): deny beats yolo. +- `scope_matches` (:680-706), order critical: `pfx/**` => normalized path + prefix (absolutize + `incremental_canonicalize`, symlink-aware); + `cmd *` => equals `cmd` or starts `"cmd "` (word boundary: `pwd *` + matches `pwd`/`pwd -L`, not `pwdx`); `pfx*` => raw prefix; else exact. + Universal `*`,`/**`,`/*` short-circuit every scope. +- Builtin rules (:32-48): file-write tools allow `cwd/**`; `task` `*`; + nothing for bash. Answers (:113-183): allow once/session/always_local/ + always_global, deny(+guidance)/always. `apply_decision` (:495-545) + GENERALIZES before persisting: bash -> first token + `" *"` + (`cargo test -p x` -> `cargo *`, :737-740); file writes -> parent + `/**`; MCP -> `*`. Deny persists EXACT scope (tight deny, broad + allow — deliberate). +- `enforce` (:548-633): NeedsPrompt => PermissionRequest event + await on + response channel; NO channel (headless) => deny (user_abort). Headless + needs rules/yolo covering everything it will do. + +**Go port**: `github.com/tree-sitter/tree-sitter-bash`; node kinds are +identical — port COMPLEX_TYPES/LEAF_COMMAND_TYPES verbatim. Rules as +`{tool, scopePattern, effect}` + the 4-way suffix matcher. Keep +scopes=segments: it's what makes `allow bash cargo *` safe under +chaining. Skip: plugin rule store, OTel, plan-path auto-allow. + +## 2. Token reduction + +Rationale: `site/docs/content/token-economy/_index.md` — every result is +re-paid each turn; attack result size AND round-trip count. + +**index tool** — `plugins/index/{init,indexer}.lua`, 30+ extractors in +`plugins/index/lang/*.lua`: sections are `imports: [lo-hi]` with paths as +a trie tree; module/type names as wrapped CSV; signatures one per line +suffixed ` [start-end]`; test lines separate; module doc range. ~70-90% +smaller than full read. Contract: index first, then `read offset/limit` +the ranges (hints init.lua:144-152 + description). 2MB cap. PORT: yes, +high value, pure tree-sitter. Start: go, rust, python, ts, markdown, +json/yaml fallback. Keep the `text [lo-hi]` suffix format. + +**MCP tool_search** — `maki-agent/src/mcp/mod.rs`: +- Defer when server's non-`always_load` tools > `defer_tools` (dflt 10; + below that defs cost less than a catalog). Hidden tools become ONE + synthetic `tool_search` whose description embeds a name-only catalog + per server (:1199-1241). 117-tool server -> 1 def. `search_tools` + (:400-485): name hit=2, description/schema-prop hit=1; exact name + always wins; top 5; model-origin search inserts matches into session + `loaded` set -> full defs on NEXT request; overflow <=20 names. + Calling a deferred tool by name also loads it (:506). +- Session-private `loaded` (:306-344); resume seeds from history tool_use + names containing `__`. `extend_tools` recomputed EVERY request + (`agent/run.rs:287-299`) — caching freezes the catalog. PORT: yes, with + MCP client (crush notes §1); per-turn recompute, per-session loads, + resume-seeding are all cheap and load-bearing. + +**code_execution** — `plugins/code_execution/init.lua` + `maki-interpreter` +(pydantic `monty` embedded Rust Python): tools as async fns, chained/ +filtered calls stay in sandbox, only prints enter context; custom +`gather()` keeps sibling results when one call fails (:34-50); 30s budget +(tool-call await excluded), 50MB, no fs/network, fresh sandbox; audience- +gated so read-only subs can't reach edit/write through it (:171-184). +SKIP: violates 100%-Go HARD RULE. Substitute now = `batch` tool +(`plugins/batch/init.lua`: N independent calls, one turn/request); Go +filter DSL later; keep the "only compact output enters context" rule. + +**Also portable cheaply**: global truncation (max lines/bytes per tool; +overlong grep lines skipped); partial-output preservation on Esc/deadline +(bash keeps streamed lines, task keeps half transcript) — for us: on turn +timeout return what the tool produced so far, tagged partial. +## 3. Subagent model -> TASK/dispatch + +`plugins/task/init.lua` + audiences `maki-agent/src/tools/registry.rs:21-48`:- Types: `research` (default, read-only) vs `general` (full). Enforcement + = audience bitflags (MAIN, RESEARCH_SUB, GENERAL_SUB, INTERPRETER, + WORKFLOW). edit/write declare `{main, general_sub, interpreter}` + (edit:254+, write:33) so research lacks them; `research.md` prompt + doubles down. bash/read/grep default ALL — research CAN bash (git log) + but shares parent PermissionManager (Arc clone, api/agent.rs:589), so + write-ish bash still denies/prompts. Advertised == callable via one + shared filter fn. Tiers: weak|medium|strong (Haiku/Sonnet/Opus, strong + ~5x medium). `resolve_model_from_ctx` (api/agent.rs:44-63) CLAMPS + `requested.min(ctx.tier)` — subagents only go cheaper, never escalate. + Tier->model: registry overrides -> discovered -> positional + (Strong=models[0], Medium=[1], Weak=[2]) (`maki-providers/src/ + model_registry.rs:171-238`); pricing per provider in providers/*. + Concurrency: process-wide semaphore dflt 8; pcall so error can't leak + permit (:178-239). +- Structured output: optional JSON Schema -> session-local + `structured_output` tool; invalid input = inline error model fixes + same-run; <=2 nudges, <=3 schema errors. Isolation: fresh session, + prompt inlined, single result string (or validated JSON) back; + description demands "concise summaries with file:line refs". + +**Map to MOPAC**: grinder dispatch = weak read-only research; P1/feature += medium general; founder gate stays human. structured_output schema == +REPORT file contract. Semaphore == class-aware LLM slot GATE. Skip the +Lua layer: task tool = Go fn spawning a nested bounded agent run with +filtered toolset + clamped model alias. + +## 4. Headless / --print + session parallelism +- `src/print.rs:140-362`: prompt arg or stdin; output text|json|stream- + json. JSON = Claude-Code-style events: `system/init` (cwd, session_id, + tools, model), `assistant` per turn (content, usage, model, + parent_tool_use_id for subagent turns), `user` tool results, + `system/api_retry`, final `result` {subtype, duration_ms, num_turns, + result, stop_reason, session_id, total_cost_usd, usage}. Cost summed as + turns land (rates move mid-run).- `maki-agent/src/headless.rs`: `spawn` (:164-282) one-shot agent on smol + task: own PermissionManager, fresh FileReadTracker, event channel, + session persisted per turn. `question` excluded (:182). NO response + channel => uncovered scopes deny (§1). `spawn_interactive` (:318+) + adds input/answer/cancel/model channels (ACP/SDK, `src/sdk_mode.rs`). + SessionMailbox drained before each user turn (run.rs:266). Each spawn + self-contained => N concurrent sessions trivial (SDK mode does it); + subagent semaphore is the only global cap. Resumable by id from state dir. + +**Port**: this IS our core loop — bounded headless run, JSON events to +log, session file per turn for resume/audit. Copy the result envelope +(cost/usage/stop_reason) so drip-chaining can budget-gate. +## 5. Re-read during build (top 10) + +1. `plugins/bash/init.lua` — scopes, exec, cd-hint, partials. +2. `maki-agent/src/permissions.rs` — engine+matcher+generalization; tests :776+ are the spec. +3. `maki-agent/src/mcp/mod.rs:358-510,1199+` — tool_search catalog. +4. `plugins/index/indexer.lua` + `lang/go.lua` — skeleton format. +5. `plugins/task/init.lua` — subagent lifecycle, nudges, semaphore. +6. `maki-lua/src/api/agent.rs` — model clamp, session, tools fn. +7. `maki-agent/src/headless.rs` — spawn/spawn_interactive wiring. +8. `src/print.rs` — event wire format + result envelope. +9. `maki-agent/src/tools/registry.rs` — audiences + definitions filter. +10. `maki-agent/src/agent/run.rs` — loop, per-turn request_tools, mailbox. + +## 6. Gotchas (top 5) + +1. Headless prompt==deny: parse-error/complex bash forces prompt, and + headless has no channel => deny. Rules must be prefix scopes (`cargo + *`, `git *`) or commands stay trivial; test compound commands against + the decomposer, not just single words. +2. Matcher order: try `/**` and `" *"` suffixes BEFORE bare `*` or a + plain prefix swallows them; `*`/`/**`/`/*` are universal (deny `*` + blocks every tool). `pwd *` must not match `pwdx`. +3. Never cache tool definitions: MCP catalog + loads recomputed every + request; advertise==callable must hold for audiences too, or a + read-only sub gets edit schemas it can't call — or calls them. +4. Generalization asymmetry: allow-always broadens (bash first word, + writes to parent dir, MCP `*`), deny stays exact. Copy deliberately — + broad denies brick sessions, exact allows spam prompts. +5. Path scopes need symlink-aware canonicalization incrementally + left-to-right BEFORE `..` applies (`physical_boundary_check`, + permissions.rs:720-735); lexical prefix matching is smugglable via + `symlink/..`. Same care for "writes confined to declared roots". +Bonus: tier clamp silently downgrades (medium parent + strong -> medium) +— that's the cost control, don't "fix" it. tree-sitter-bash node names +shift across grammar majors (e.g. `command_name`); pin your test version.