Dark-Alex-17/coyote
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Dark-Alex-17/coyote:5fe3198df71cda1c2d85c0fca40984ea83ca1dfc
Dark-Alex-17/coyote:main Dark-Alex-17/coyote:feat/rag-phase5-qdrant-write Dark-Alex-17/coyote:feat/rag-driver-abstraction-v3 Dark-Alex-17/coyote:graph-based-rag Dark-Alex-17/coyote:skills Dark-Alex-17/coyote:develop Dark-Alex-17/coyote:graph-agents Dark-Alex-17/coyote:config-refactor Dark-Alex-17/coyote:restful-api-docs Dark-Alex-17/coyote:langchain
Dark-Alex-17/coyote:v0.7.3 Dark-Alex-17/coyote:v0.7.2 Dark-Alex-17/coyote:v0.7.1 Dark-Alex-17/coyote:v0.7.0 Dark-Alex-17/coyote:v0.6.0 Dark-Alex-17/coyote:v0.5.0 Dark-Alex-17/coyote:v0.4.0 Dark-Alex-17/coyote:v0.3.0 Dark-Alex-17/coyote:v0.2.0 Dark-Alex-17/coyote:v0.1.3 Dark-Alex-17/coyote:v0.1.2 Dark-Alex-17/coyote:v0.1.1
..
compare: Dark-Alex-17/coyote:cfb83a629e5f086a001b83f97e2034bb66a32d00
Dark-Alex-17/coyote:main Dark-Alex-17/coyote:feat/rag-phase5-qdrant-write Dark-Alex-17/coyote:feat/rag-driver-abstraction-v3 Dark-Alex-17/coyote:graph-based-rag Dark-Alex-17/coyote:skills Dark-Alex-17/coyote:develop Dark-Alex-17/coyote:graph-agents Dark-Alex-17/coyote:config-refactor Dark-Alex-17/coyote:restful-api-docs Dark-Alex-17/coyote:langchain
Dark-Alex-17/coyote:v0.7.3 Dark-Alex-17/coyote:v0.7.2 Dark-Alex-17/coyote:v0.7.1 Dark-Alex-17/coyote:v0.7.0 Dark-Alex-17/coyote:v0.6.0 Dark-Alex-17/coyote:v0.5.0 Dark-Alex-17/coyote:v0.4.0 Dark-Alex-17/coyote:v0.3.0 Dark-Alex-17/coyote:v0.2.0 Dark-Alex-17/coyote:v0.1.3 Dark-Alex-17/coyote:v0.1.2 Dark-Alex-17/coyote:v0.1.1
183 Commits
5fe3198df7 .. cfb83a629e
Author SHA1 Message Date
Dark-Alex-17 cfb83a629e feat: create a config alias for default_reasoning_level (to match some wording and make the config flexible)
CI / All (ubuntu-latest) (push) Failing after 30s
Details
CI / All (macos-latest) (push) Canceled after 0s
Details
CI / All (windows-latest) (push) Canceled after 0s
Details
2026-09-14 17:47:01 -06:00
Dark-Alex-17 8377f8287b feat: support custom resource definitions for openai-compatible Oauth customization 2026-09-14 17:46:44 -06:00
Dark-Alex-17 ffb75cb168 feat: Support specifying the compression model to use when compressing a session for greater cost controls 2026-09-14 15:49:44 -06:00
Dark-Alex-17 58b0dbd812 feat: Added full cost tracking to Coyote and added it to the default right prompt 2026-09-14 15:24:48 -06:00
Dark-Alex-17 bd5c0a7cdd chore: updated the models.yaml to incorporate cached read/write prices pulled from models.dev and openrouter 2026-09-14 14:47:22 -06:00
Dark-Alex-17 d84dea21b6 feat: added support for input/output token cache cost tracking 2026-09-14 14:47:01 -06:00
Dark-Alex-17 755b25f9b2 fix: empty tool results can't cause errors in bedrock, claude, openai, or vertexai clients anymore 2026-09-14 14:46:46 -06:00
Dark-Alex-17 23f941b09f fix: proper support for reasoning/thinking in bedrock models 2026-09-14 14:46:15 -06:00
Dark-Alex-17 77da48a40f feat: Support persisting paused todo lists to sessions so users can exit and resume paused todo sessions; Also fixed a bug so todo list pausing is only supported when being used in a session since it typically requires multi-turn communication 2026-09-14 13:57:08 -06:00
Dark-Alex-17 5d59adadbc feat: Support token caching to reduce LLM costs in Anthropic models (not typically handled automatically by APIs) 2026-09-14 13:56:13 -06:00
Alex Clarke 9a6ffcb8ca Merge pull request #28 from Dark-Alex-17/feat/PLAN-review-retry 
PLAN-review-retry: self-healing review gate (fault-only lane retries) + untimed approvals
 2026-09-14 11:46:15 -06:00
Alex Clarke 62e6fafaef Merge pull request #26 from Dark-Alex-17/feat/PLAN-agent-node-resilience 
PLAN-agent-node-resilience: graph agent-node max_attempts + fallback
 2026-09-14 11:35:24 -06:00
Dark-Alex-17 beca133082 style: removed unneccessary path prefixes and organized imports 2026-09-14 11:13:34 -06:00
Dark-Alex-17 c7ec4ed124 docs: restored formatting to agent example config file 2026-09-14 11:13:34 -06:00
Dark-Alex-17 cee5cf0f36 test(rag): make TempDir names unique across parallel tests 
SystemTime nanos alone collide on coarse clocks, so concurrent tests shared
a scratch dir and read each other's generated sbx mixins (flaky
generated_sbx_mixin_* failures in CI).
 2026-09-14 11:13:34 -06:00
Dark-Alex-17 85cb7b0fbf test: normalize CRLF in the gauntlet README contract pin (Windows CI) 2026-09-14 11:13:34 -06:00
Dark-Alex-17 d89592fee6 test: normalize CRLF in the gauntlet script/README source pins (Windows CI) 2026-09-14 11:13:34 -06:00
Dark-Alex-17 342f39f30e ci: run pull_request checks for PRs based on any branch 
'*' does not match '/' in GitHub branch filters, so a PR whose base is a
feature branch (e.g. a stacked PR) never triggered CI.
 2026-09-14 11:13:34 -06:00
Dark-Alex-17 4f0c7e4431 test(TASK-030): pin fallback-note placement; detect concrete task ids (with illustrative-example allowlist) instead of run-local literals 2026-09-14 11:13:34 -06:00
Dark-Alex-17 fcf67b07a7 test(TASK-030): pin placement, ladder clause, findings-first rule, README sync; reconcile gauntlet README caller guidance 2026-09-14 11:13:34 -06:00
Dark-Alex-17 78e326039d feat(TASK-030): callers escalate on GAUNTLET_REVIEW_INCOMPLETE instead of findings-routing 2026-09-14 11:13:34 -06:00
Dark-Alex-17 f18c436e2c fix(TASK-032): enforce incomplete⇒BLOCKED structurally; anchor adversary degraded detection 2026-09-14 11:13:34 -06:00
Dark-Alex-17 786e5542d4 feat(TASK-032): review-gauntlet incomplete-review machine line + ×N attempt details 
verdict_gate.py now names every incomplete review on a machine line the
done template renders immediately after the verdict:
`GAUNTLET_REVIEW_INCOMPLETE: <name>[, <name>]` (sorted). Names are the
lanes retry_gate gave up on (`review_incomplete`) plus the `pipeline`
pseudo-lane whenever `signals_error` carries a `PIPELINE-FAULT:` —
parse death, builder / retry-gate / default-lanes crash guards — and the
gate's own crash guard emits the `pipeline` line too. The report gains a
`## Review incomplete` section (after the signals_error blockquote,
immediately before `## Full lane reports`) and the three PIPELINE-FAULT
lane rows carry ` ×N` when the lane was re-run (N = 1 texts unchanged).

retry_gate.py stands in a synthetic `PIPELINE-FAULT: <lane> lane produced
no report` marker for a lane that ran but collected nothing, so it is
stashed, classified FAULTED and never renders SKIPPED downstream.
MAX_RETRY_ELAPSED_SECS trimmed 45600 → 45000 for margin slack.

graph.yaml: `gauntlet_incomplete_line` seeded in initial_state and
spliced into the done output. README + mermaid document retry_gate, the
dotted _next back-edge, the 3-attempt / wall-clock budget, that retry
never narrows selection, the crash-guard append semantics, and the
machine-line contract (present whenever any lane or the pipeline was
incomplete; it can accompany real findings).

Tests: ×N on all three branches (N>1, N=1 byte-identical, no leak onto
non-fault rows), exact incomplete line / section placement, pipeline
pseudo-lane alone and alongside real lanes (via default_lanes.py's real
crash guard), mixed finding+fault, crash-guard chaining from
build_items / retry_gate / parse_fault / verdict_gate, synthetic
no-report marker, done-template wiring, README pins (positive + stale
negative).
 2026-09-14 11:13:34 -06:00
Dark-Alex-17 18f5ada585 feat(TASK-029): review-gauntlet fault-only lane re-run loop 
Give the review gauntlet a deterministic retry loop for lanes that FAULT
(PIPELINE-FAULT marker, missing verdict sentinel, or a nested graph's own
degraded-run wording) without ever re-running a lane that reached a real
verdict.

- graph.yaml: the four lane maps route to a new `retry_gate` script node
  (static `next: verdict_gate`; emits `_next: build_items` to retry).
  initial_state gains lane_attempts / kept_results / retry_lanes /
  review_incomplete / gauntlet_started_at. settings.timeout 24000 -> 69600
  (3 passes x 2 attempts x 11400 + 1200 margin).
- scripts/retry_gate.py (new): stashes every lane report that ran
  (completed and faulted alike) and restores reports a re-run pass
  overwrote with []; classifies with verdict_gate.py's sentinel regexes
  verbatim; counts attempts; retries while attempts < 3 and inside a
  45600 s wall-clock budget; otherwise records the lane in
  review_incomplete and falls through. Crash guard appends to
  signals_error and never retries.
- scripts/signals.py: stamps gauntlet_started_at before the git work so a
  failed diff still starts the clock.
- scripts/build_items.py: retry pass emits items only for retry_lanes
  (with attempt numbers), adds a `retried:` note to lanes_summary, resets
  retry_lanes; falsy-guarded stamp fallback; crash guard appends to
  signals_error and preserves lanes_summary.
- Tests: hermetic python3-guarded script tests for retry_gate /
  build_items / signals, graph wiring + initial_state pins, sentinel and
  LANE_KEYS cross-pins, retry-budget envelope vs settings.timeout, a
  first-pass golden payload, and an engine-level executor test proving a
  script `_next` back-edge re-fires a static fan-out and an empty map
  re-run clears its collect_into.

Plan: PLAN-review-retry section A items 1-3 and 5. Verdict/report surface
(section A items 4 and 6) is TASK-032.
 2026-09-14 11:13:34 -06:00
Dark-Alex-17 b90e04f227 fix(TASK-028): pin example-doc wording; make not-found escalation hint match queue semantics 2026-09-14 11:13:34 -06:00
Dark-Alex-17 3339920377 fix(TASK-028): pin global-template exclusion, tighten timeout boundary test, reword stale hint 2026-09-14 11:13:34 -06:00
Dark-Alex-17 9a505f951a feat(TASK-028): escalation_timeout 0 waits indefinitely and is the default 
Child escalations (coder plan approvals, deviation gates, user__* calls
from subagents) previously expired after the escalating agent's
escalation_timeout, defaulting to 300 s, and 0 timed out instantly. In
autonomous runs the orchestrator often could not reply within that
window, so approvals silently fell through to the best-judgment
fallback.

- Extract await_escalation_reply as the wait seam: 0 awaits the oneshot
  with no deadline; nonzero keeps tokio::time::timeout with the same
  timeout message. A dropped sender still resolves to the existing
  cancelled reply either way.
- DEFAULT_ESCALATION_TIMEOUT_SECS and default_escalation_timeout() -> 0,
  so graph agents (which inherit the default via from_graph) are covered.
- Document the new semantics in config.agent.example.yaml and drop the
  5-minute-timeout claim from the default spawn instructions.
- Tests: paused-clock unlimited wait (3600 s), nonzero expiry message,
  dropped-sender completion, default pin, and a grep pin that the stale
  timeout wording is gone from src/, assets/, and the example config.

Plan: PLAN-review-retry §C (D3).
 2026-09-14 11:13:34 -06:00
Dark-Alex-17 b87367bb1b style: Refactored some imports and LLM hints 2026-09-14 10:50:34 -06:00
Dark-Alex-17 ad3a5c6d49 test: normalize CRLF in the run_checks docstring pin (Windows CI) 2026-09-14 10:50:34 -06:00
Dark-Alex-17 68e893d684 fix(adversary): red verification runs block CONFORMS; re-derive timeout envelope; robust runner 
verdict.py: a recorded verification run with exit≠0 (including the -1
TIMEOUT sentinel) or a skipped record now forces DIVERGES with a numbered
complaint naming the command — previously a [FAIL] line sat in the report
appendix while the sentinel read CONFORMS, and callers route on the
sentinel alone. Fail-closed by the same rule: an ENVIRONMENT runner-error
marker when commands WERE declared is one red complaint (the declared
commands never ran, so the run is unproven); the "none declared" marker
and an empty declaration stay non-red. Headers name the red-run count.

run_checks.py: decode command output with encoding=utf-8/errors=replace
(a single non-UTF-8 byte used to raise inside communicate() and collapse
the whole evidence record to an ENVIRONMENT marker); bound the post-kill
pipe drain by DRAIN_TIMEOUT_SECS so a session-escaped grandchild holding
the pipe cannot turn a TIMEOUT record into a node kill.

graph.yaml: settings.timeout 5400 → 9000 = 3600 (run_checks node timeout)
+ 5400 (the previous whole-graph budget, kept for every other stage); the
old value predated run_checks and a graph-level timeout kills from
outside, losing the DIVERGES sentinel. Descriptions and README doctrine
updated (CONFORMS = all MET + zero holistic complaints + zero pipeline
faults + zero red runs).

Tests: envelope pin (settings >= run_checks + 5400), gauntlet lane
timeouts > child graph timeouts and gauntlet >= max_attempts × lane +
margin over every agent lane, non-UTF-8 output, bounded drain (unix),
red/skipped/timeout/none-declared/ENVIRONMENT×declaration matrix,
numbering continuity, header punctuation, wording pins on graph.yaml,
run_checks.py and both READMEs. The deadline test's hung command is now
python3-based instead of GNU sleep (cmd.exe on the Windows CI leg).
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 156f3193a9 fix(review-suite): detect nested finding-verifier faults; harden render/fault markers 
code-reviewer verdict.py: finding-verifier now completes "successfully"
with fault markers INSIDE its FINDING_VERIFIER_RESULTS payload
(parse_fault's `pipeline-fault` sentinel id, verify_fault / verdict_gate
crash-guard notes prefixed PIPELINE-FAULT:), which the old
"Agent node failed:" prefix check never saw — a dead parse degraded
silently to "unverified" with no fault recorded. Detect them, anchored on
the sentinel id / note prefix only (never a substring over the report),
counted over the raw entry list so duplicate crash-guard ids are not
collapsed. Faults lead the attention list uncapped; the display cap of 5
applies to non-fault entries; the "always-human trigger(s) fired" reason
fires only when non-fault triggers exist.

code-reviewer render.py: CONTRACT comments at the verdict and summary
lines review-gauntlet's gate parses; every LLM-influenced value rendered
after the summary line (dropped_titles, rigor, bar_provenance,
resolved_surfaces, reason) is newline-sanitised so none can forge a
second summary line.

finding-verifier verdict_gate.py: the outer crash-guard note carries the
PIPELINE-FAULT: prefix (the after-retry UNVERIFIABLE path stays
unprefixed — model misbehaviour, not infrastructure); README names all
three emitters.

deep-research: vet_fault's UNVETTED note tells the critic not to request
revision solely because the assessment is unavailable (it matched
critique's PREPRINT/UNVERIFIED REVISE trigger verbatim, burning the
revision budget on an infra fault); vet_fault and critique_fault dedupe
repeated faults and preserve accumulated faults on crash via
safe_faults(); question_fault keeps its plan-pinned prefix and appends the
lane's question identity so distinct dead lanes no longer collapse in
combine_findings' dedupe.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 4555af8299 fix(review-gauntlet): summary-line critical count, nested-fault hook, fail-closed lane fallback 
verdict_gate.py: the code-review critical count is read from render.py's
deterministic summary line (full-shape regex, LAST match wins — a 🔴
quoted inside a finding body or Changes-table row is not a finding); the
raw 🔴 count remains the fail-closed fallback when the line is absent. A
code-review lane whose own rendered verdict line records a pipeline
fault (degraded nested run) now BLOCKS instead of passing as GREEN
(attention), and fault + reds append both blockers. A non-string lane
result item is BLOCKED as malformed, never routed to GREEN.

default_lanes.py: the crash handler now emits the WIDEST safe selection
(code-review ∪ adversary ∪ security) and records a PIPELINE-FAULT in
signals_error so the gate blocks — previously it repaired malformed state
into a NARROWER passable selection with no fault. Probe is added only
when consumer_surface AND a probe_context (local-run recipe) are present,
mirroring build_items' guard, so degraded selection cannot spawn an
INCONCLUSIVE-by-construction probe.

graph.yaml: re-derive the timeout envelope. Agent-node retries each get a
fresh per-attempt budget (agent.rs retry_transient/bounded_attempt) and
the child's "Graph … timed out" matches the transient anchor, so lanes
exceed their child graph's settings.timeout (code-review 11400 > 10800,
adversary 9600 > 9000) and the gauntlet's own timeout covers
max_attempts × lane timeout + margin (2 × 11400 + 1200 = 24000). The
child graphs' internal retry envelopes are deliberately not derived here
(tracked as a follow-up). README/descriptions updated and pinned.

Tests: summary-count/quoted-🔴/fallback/last-match/nested-fault/both-
blockers/malformed-lane, crash-widens-and-records-fault, probe guard,
ALIASES parity with build_items, and e2e render.py → verdict_gate runs.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 ea76331c49 fix(graph): typed ApiStatusError transient classification; warn on bare output_schema 
is_transient_error: a typed crate::client::ApiStatusError anywhere in the
chain is now authoritative (positive and negative) — catch_error renders
429s as "… (type: rate_limit_error)" / "… (status: 429)", which none of
the old string anchors matched, so provider rate limits were never
retried, while a 400 whose body mentioned "rate limit" was. Transient
statuses: 429, 500, 502, 503, 504, 529; message text only classifies
untyped errors, and those anchors are now case-insensitive.

validator: warn for an llm/agent node whose output_schema lacks a
non-empty object `properties` map (auto-merge writes nothing since
402d25a) unless the node lifts keys via a non-empty state_updates; the
same absent-or-empty predicate now also gates validate_fallback_capture.
Bundled and graph.example.yaml warning baselines are unchanged.

Tests build the 429 fixture through real catch_error output and a
.context() chain, cover the status table, typed-overrides-text, and all
validator branches.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 84b5a72eac fix(review-suite): mark example verification commands as placeholders; pin variable description and prompt note 
Adversary re-review follow-ups to 665fee9/402d25a:

- architect/sisyphus spawn templates: the pinned cargo commands are only an
  EXAMPLE in a language-agnostic orchestrator — a literal follower in a
  non-Rust repo would record 'cargo: command not found' and yield a false
  DIVERGES. Each --variables line now carries a substitution cue (pinned).
- run_checks.py: a non-list prior pipeline_faults value is no longer iterated
  character by character (test case added).
- Pins: both graphs' verification_commands description wording; the
  run_adversary prompt states where the commands come from; the replacement
  sentence sits on one physical line.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 27101909e0 fix(graph): auto-merge only schema-declared keys from llm output 
state_updates::apply merged EVERY top-level key of an llm/agent node's JSON
output into graph state whenever the node had an output_schema, and
structured::extract never validated the output against that schema. Any
prompt text able to steer the model — including plan/diff text pasted from a
repo under review — could therefore plant an undeclared key (e.g.
verification_commands) into state and overwrite a caller-declared value
before a downstream script consumed it.

- apply() now takes the node's output_schema and merges only keys present in
  schema.properties; undeclared keys are dropped. No schema, or a schema
  without an object properties map, merges nothing.
- Nodes without output_schema are unchanged (None == the old false path);
  explicit state_updates are unchanged and still see the full output via
  {{output.<key>}}, which remains the author-controlled way to lift a key.
- Callers: llm.rs, agent.rs (x2), rag.rs (None).
- Sweep test bundled_graph_output_schemas_declare_every_key_relied_on: for
  every bundled graph, every consumed state key has a declared writer
  (initial_state, variable, state_updates target, map key, script-emitted key,
  or an output_schema property), and no schema node's field list names a key
  its schema omits — so this restriction cannot silently break a shipped
  graph. It passes with no graph.yaml changes.
- graph.example.yaml documents the declared-keys-only merge; the residual
  disclosed in 665fee9's README/docstring notes is now recorded as closed.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 300c7aeff0 fix(review-suite): pass verification_commands as a declared variable, never LLM-extracted from prose 
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.

- adversary + review-gauntlet graphs declare verification_commands as a graph
  variable (default '[]'); the key leaves initial_state (or_insert seeding
  would otherwise shadow it) and leaves both parse nodes' instructions,
  output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
  passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
  arrive as strings); '[]'/[]/missing = none declared; any other shape fails
  closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
  DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
  --variables / --agent-variable form and the trust boundary. Known residual
  (llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
  prompt-prose injection proof, verdict wiring, asset pins for both graphs and
  orchestrator configs, dispatch-inputs > variable-override > default
  precedence.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 00c3b60c2f feat(graph): stream llm-node completions so node timeout governs long generations 
Graph llm nodes (and the structured-output extractor they share with agent
nodes) now obtain the model reply through a quiet streaming transport,
`call_chat_completions_streaming_quiet`, instead of the non-streaming call.
Because chunks arrive as they are generated, reqwest's client-level
`read_timeout` (default 300s) bounds only a genuine stall between chunks and
the node's own `timeout` becomes the effective bound on total generation.
Previously zero bytes arrived until the whole reply finished, so a 300s
read_timeout silently capped every llm node regardless of its `timeout`
(seen live: code-reviewer's `synthesize` node dying on a 61-file review).

The quiet transport renders nothing (silent SseHandler, no spinner), keeps
tool-call handling identical to the non-streaming path (call-loop detection
off), falls back to the non-streaming request for catalog `no_stream` models
(raced against the abort), and never returns partial output as success.

Supporting changes:
- `sse_stream`/`json_stream` keep the typed `reqwest::Error` in the anyhow
  chain so `is_transient_error` still classifies read_timeout stalls as
  transient and `max_attempts` retries them (previously stringified).
- `structured::extract` takes the graph/session AbortSignal and refuses to
  issue an extraction request for an already-aborted graph.
- `run_chat_loop` gains an injectable CompletionRunner seam (owning client
  construction + the before-hook) so the timeout/retry/abort behaviour is
  unit-testable over a paused clock; a fake `impl Client` drives the real
  trait-default streaming for 600s of virtual time; real-socket tests pin
  that read_timeout fires on a stall but not on a slow continuous stream.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 22421a4e74 test(adversary): pin run_checks process-group kill and deadline-vs-timeout ordering 
run_checks.py SIGKILLs the timed-out command's whole process group so a
backgrounded grandchild cannot outlive the record; nothing proved it.
Add a unix-only test that backgrounds `sleep 30`, records its pid, and
probes `kill -0` after the 1s deadline fires.

TOTAL_DEADLINE_SECS (run_checks.py) and the run_checks node's `timeout:`
(graph.yaml) live in different files and could drift until the node
executor kills the script before its in-script degradation runs. Parse
the script's default and assert it stays strictly below the node
timeout, without hardcoding either value.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 fa5ee5f4fc test(graphs): lock retry counts, fallback wiring, and check_criterion's read-only tools 
Sweep every agent node in the six bundled graphs and assert max_attempts
is exactly 2 (step-runner/implement stays 1) against a pinned node list,
so a dropped `max_attempts:` line cannot silently revert a lane to
one-shot (the serde default is 1). Sweep every llm/agent node for a
declared, existing fallback and pin the specific fault routes in
review-gauntlet, code-reviewer, and adversary.

Pin check_criterion's tool list to fs_read/fs_cat/fs_grep/ast_grep and
its 'judged ONLY against the recorded verification runs' instruction,
so the criterion branch can never grow an execution tool unnoticed.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 ba3a904baf test(graph): prove per-attempt timeout budget and cover extraction-fault text 
agent.rs: the retry-after-timeout test ran its second attempt with no
timeout bound, so it passed whether or not the retry owned a fresh
budget. Attempt 2 now runs under the same 1s bounded_attempt and sleeps
900ms under the paused clock; it only succeeds if the deadline is
per-attempt rather than shared with the attempt that already used it.

llm.rs: the `LLM node structured-extraction failed: {e:#}` branch had no
test because `run` cannot succeed without a model. Split the post-run
half of execute into `finish(node, state, ctx, result)` (byte-identical
body, mirrors agent.rs's outcome_from seam) and add tests feeding
Ok("not json") through it, with and without a fallback, asserting the
captured text prefix and the extractor's full context chain.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 8e42a59b98 docs(adversary): fix mermaid node label and criterion_fault join edge 
Node B is the graph's `facts` node (script diff_facts.py), not a node
named diff_facts. criterion_fault has no `next`; it reaches holistic
through the map join, so draw it dashed like the other branch-terminal
edges instead of as a solid successor edge.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 081fa2c9d4 test(suite): replace plan-label references with self-contained wording 
Rust test comments/assert messages and deep-research script docstrings
cited numbered requirement labels (R3/R5) and task labels (TASK-016,
TASK-021) that are defined only in an external plan document. Reword
each to say what it means on its own: fail-safe crash guards, the
validator's fallback-capture warning, task-queue anchors.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 48753eaa46 fix(agents): scrub GRAPH_STATE* from adversary verification commands; raise review-suite max_agent_depth 
FIX A: run_checks.py passed no env= to Popen, so verification commands
inherited the live GRAPH_STATE/GRAPH_STATE_FILE. Every suite script's
load_state() prefers GRAPH_STATE_FILE, so when the adversary reviews a repo
whose own tests exec these scripts (this repo's R5 tests) the leaked file
silently overrode the test state -> false failures. Popen now gets an env
with all GRAPH_STATE* keys removed. The five sibling script-test helpers
(review-gauntlet, code-reviewer, finding-verifier, step-runner,
deep-research) gain .env_remove("GRAPH_STATE_FILE") for hermeticity. New
R5 test loads state via GRAPH_STATE_FILE and asserts a shell-neutral
python3 -c probe sees both vars UNSET.

FIX B: the depth check compares the child's absolute depth against the
SPAWNER's own max_agent_depth (root = 0), so under
architect(0)->sisyphus(1)->review-gauntlet(2)->code-reviewer(3) the
code-review lane faulted "Max agent depth exceeded (4/3)". Limits are now
child depth + 1 headroom along the documented chains, with the chain in a
comment next to each key: code-reviewer 3->5, domain-reviewer 2->6,
architecture-reviewer 2->4, step-runner default->4 (explicit),
review-gauntlet stays 4. domain-reviewer's file-reviewer hop is documented
but inert (can_spawn_agents unset) - left unwired by decision; noted in
config and pinned via a DOCUMENTED_BUT_UNWIRED exemption in the new
bundled-config test, which pins every value and asserts per-hop headroom +
reachability along four chains.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 62ecb5b415 test(adversary): pin exact died-evidence prefix on the criterion_fault crash path 
The crash-path assertion checked only "PIPELINE-FAULT:"; verdict.py's died() keys on "PIPELINE-FAULT: criterion ", so a regression there would have passed the test yet stopped the crashed marker from rendering as DIED.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 7ada3fb47c fix(adversary): narrow died-marker predicate, degraded banner for died criteria, hermetic script tests 2026-09-14 10:50:34 -06:00
Dark-Alex-17 1cf42fffa2 fix(adversary): fail closed on PIPELINE-FAULT evidence regardless of status; pin bail-chain and validator shape 2026-09-14 10:50:34 -06:00
Dark-Alex-17 66b30bb06a test(code-reviewer): pin synthesize max_attempts and fail-closed wiring 
A live code-reviewer run degraded through synthesize's fallback; the
graph already retries once (max_attempts: 2) and falls back to verdict
with synth_failure captured, but nothing pinned that wiring. Add the
wiring test so a future edit cannot silently drop the retry or route.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 3f49db86b2 fix(adversary): fail closed per criterion when check_criterion dies 
A live run died at map_criteria: one heavy criterion exhausted
max_iterations (10) with no fallback and sank the whole 17-criterion
review. check_criterion now has a 30-iteration budget, a tool-budget
instruction, and falls back to criterion_fault, which records a
schema-shaped UNMET verdict with a PIPELINE-FAULT evidence marker for
that one criterion; verdict.py renders it as DIED rather than judged.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 7dac4620a8 fix(engine): carry full error chain in llm node fault text 
A live code-reviewer run reported "LLM node failed: Failed to call
chat-completions api" with the root cause invisible, because the
state_updates fault text used `{e}` (top context only). Format with
`{e:#}` so fault scripts and warn! logs see the whole anyhow chain.
 2026-09-14 10:50:34 -06:00
Dark-Alex-17 5b4462135f style: cargo fmt after doc-comment cleanup 2026-09-14 10:50:34 -06:00
Dark-Alex-17 fdae98adff docs: sync agent READMEs + diagrams with resilience hardening 2026-09-14 10:50:34 -06:00
Dark-Alex-17 f230c28bf4 docs: removed references to individual task files in the code documentation 2026-09-14 10:50:33 -06:00
Dark-Alex-17 13b8eb87a9 fix(TASK-020): skip POSIX bash script tests on Windows 2026-09-14 10:50:33 -06:00
Dark-Alex-17 0ce2654173 feat(TASK-023): deep-research hardening — fail-safe scripts, fault fallbacks, pipeline notes 
R3 guards: parse_request (crash → ask_topic), bootstrap_research,
combine_findings (crash → fault-text findings; also lifts branch-local
PIPELINE-FAULT findings into pipeline_faults, deduped), reflexion_gate
(crash fails forward to synthesize), incorporate_feedback (crash still
loops with a generic feedback note), verify_sources (top-level guard +
pipeline_notes fold).

Fallbacks per the plan's naming pins: plan → plan_fault → end_fault
(fail-closed; gains state_updates plan_failure per the TASK-016 rule);
research_one_question → branch-local question_fault (map collects the
fault finding); vet_sources/critique → degrade-visibly markers that
continue to the stage each node's own next pointed to; synthesize →
synth_fault → end_fault. end_fault renders
'DEEP_RESEARCH FAILED — PIPELINE-FAULT: <fault text>'.

Accepted-path visibility: verify_sources folds pipeline_faults into
pipeline_notes ('' when fault-free), appended to end_accepted's output
and the approval question — happy-path output stays byte-identical.
initial_state seeds questions: [] so a dying plan inside the static
fan-out leaves the sibling knowledge_lookup lane mapping over zero
items while end_fault renders.

R5: 19 python3-guarded regression tests + GraphParser wiring test;
bundled warning baseline unchanged.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 c98b6bc9cd test(TASK-021): pin task-queue guidance text in shipped artifacts 2026-09-14 10:50:33 -06:00
Dark-Alex-17 f2d90ed7a6 docs(TASK-021): fix task-queue example IDs + document dispatch/collect/failure semantics 
- prompts.rs DEFAULT_SPAWN_INSTRUCTIONS: example IDs task_1/task_2 -> "1"/"2"
  (real numeric-string IDs; old format silently produced dep_error + unblocked
  task), plus task-queue rules: root tasks with agent are never auto-dispatched,
  task_complete is what dispatches dependents, auto-dispatched agents must be
  collected, failed tasks permanently block dependents.
- architect config: optional task-queue mirroring guidance for parallel mode
  (disk task files stay the durable store; queue is in-memory).
- sisyphus config: short note on encoding dependency-shaped bursts as
  agent__task_create chains; todos remain the tracking source of truth.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 22976a79f9 fix(TASK-020): surface step-runner LLM-fault captures where the fallback lands 
The three fallback-capture keys (orient_failure, sweep_failure,
handoff_failure) were write-only dead state: captured on LLM-node failure
but interpolated nowhere, so end_failure rendered an empty STEP_FAILED
shell and write_handoff carried no trace a dead sweep.

- new crash-proof script node note_llm_fault distills orient/handoff
  'LLM node...' failure text into fault_note (prefix-anchored; empty when
  neither key holds a fault); orient and write_handoff fallbacks now route
  through it before end_failure
- end_failure renders 'Pipeline fault (if any): {{fault_note}}' (sentinel
  and all existing fields preserved; fault_note defaulted in initial_state)
- write_handoff prompt interpolates {{sweep_failure}} with a
  PIPELINE-FAULT VISIBILITY instruction mirroring the review_report
  treatment (fault -> Deviations entry + step_summary; success/empty
  ignored)
- tests: note_llm_fault fault/clean/crash/truncation paths via the
  GRAPH_STATE harness; wiring test pins the retargeted fallbacks,
  note_llm_fault routing, {{fault_note}} in end_failure, and the
  sweep-fault prompt/instruction anchors; warning baseline unchanged
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 4263d684e9 feat(TASK-020): finding-verifier + step-runner fail-closed hardening 
finding-verifier: parse gains max_attempts 2 + parse_failure capture +
fallback to a new parse_fault marker that rewrites verdicts to a single
UNVERIFIABLE PIPELINE-FAULT entry and continues to done (sentinel
preserved — the caller's verify lane degrades instead of dying);
verify_one gains a branch-local verify_fault marker normalizing the
engine failure text into a schema-conformant UNVERIFIABLE verdict.

step-runner: implement gains fallback end_failure (no retry — reruns
against a mutated tree are unsafe; STEP_FAILED renders the failure via
coder_result); independent_review gains fallback write_handoff with the
handoff instructions and route_review treating an "Agent node failed:"
review_report as review-did-not-run (fail-visible, guard ahead of the
🔴 grep); orient/edge_case_sweep/write_handoff capture their failure
text via state_updates, shrinking the bundled fallback-capture warning
baseline back to zero.

R5 tests: fault-marker normalization + crash paths (raw non-JSON
GRAPH_STATE), route_review.sh fault/critical/clean routing via bash+jq,
and structural wiring tests pinning fallbacks, sentinels, and
end_failure's initial_state-backed template keys.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 a8496aa50e fix(TASK-019): carry PIPELINE-FAULT prefix through verdict.py's crash guard 
render.py's empty-diff stub bypass is anchored on the 'PIPELINE-FAULT:'
prefix in reason/attention, but verdict.py's crash guard emitted
'verdict computation error: …' / 'verdict script error: …' without it —
on an empty diff a crashed verdict gate collapsed into the
'No changes to review.' stub and the fail-closed NEEDS-HUMAN verdict
never reached the final report.

- prefix the crash-guard reason and attention entry with PIPELINE-FAULT:
- strengthen the existing crash test to assert the prefix on both fields
- add R5 chained test: empty diff + crash-inducing state piped
  verdict.py -> render.py must render the fault, never the stub
- add crash-guard symmetry tests for parse_fault.py / aux_fault.py
  (raw non-JSON GRAPH_STATE keeps the fault marker) and a test that a
  fault survives the attention[:5] cap alongside 6 user flags
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 4e72c6f5e6 feat(TASK-019): code-reviewer fail-closed review pipeline 
Every llm/agent stage that can die persistently now falls back to a
machine-readable PIPELINE-FAULT sentinel instead of killing the graph:

- review_domain -> branch-local domain_fault marker (banner-prefixed
  domain_report the map collects; completeness_gate precedent)
- verify -> verdict (verifier fault renders findings unverified + a
  PIPELINE-FAULT attention line)
- parse -> parse_fault -> render (complete NEEDS-HUMAN verdict_out;
  render's no-changes stub is bypassed for fault verdicts)
- refine_groups -> cover_gate (deterministic-proposal fallback absorbs
  the failure; degradation noted in groups_note)
- aux_lanes -> aux_fault -> synthesize (aux_note fault marker; visible,
  never blocking - aux is enrichment)
- synthesize -> verdict (synth_failure marker; keyed off the failure
  prefix + non-empty diff, never findings==[], so clean zero-finding
  reviews stay MERGE-READY)

verdict.py gains unified fault accounting (R2): any PIPELINE-FAULT in
domain_reports / verifier_output / state fault keys forces NEEDS-HUMAN,
prefix-anchored so a report merely quoting the marker mid-text never
trips it; old 'PIPELINE FAULT' spelling migrated. Faults are prepended
to attention so the [:5] cap can never hide one. Dedicated *_failure
state keys capture engine failure text on every fallback+output_schema
node, keeping the bundled warning baseline at zero.

15 python3-guarded R5 tests: fault paths, quoted-marker precedence,
happy-path byte-compat regression, marker normalization, crash-path
fail-closed (verdict/cover_gate/domain_fault), verifier-fault-on-empty-
diff render, and the no-changes stub regression.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 ebb9e9af62 fix(TASK-018): harden verdict_gate fault routing and widen default_lanes fallback 
Adversary finding 1: verdict_gate treated any lane report CONTAINING
'PIPELINE-FAULT:' as a lane fault, so a real review quoting the marker
silently bypassed red-finding counting and sentinel parsing. Lane-fault
detection is now precise — only a report that IS the lane_fault.py
marker (starts with it) routes to the fault rule. Adds the missing
regression tests: reds block despite MERGE-READY, NEEDS-HUMAN without
reds passes with attention, probe INCONCLUSIVE blocks with the
environment note, a gate crash fails closed, and a quoting report is
routed to the red rule, not the fault rule.

Adversary finding 2: default_lanes overwrote caller-forced lanes and
dropped the security lane the deterministic signals guaranteed. The
fallback now unions its defaults with caller-forced lanes (canonicalized
via the same aliases build_items uses) and forces security on
auth/deps/exec signals or hardened posture — selection degrades wider,
never narrower, and stays deterministic via a fixed lane order. Tests
cover forced-lane survival, alias canonicalization, unknown-name
tolerance, each security signal, and determinism.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 80c20afcdd test(TASK-018): cover parse_fault.py fault normalization 2026-09-14 10:50:33 -06:00
Dark-Alex-17 8407729d5a feat(TASK-018): review-gauntlet lane degradation + verification passthrough 
- lane_fault.py + fallback on all four run_* lane nodes: an agent-node
  failure is normalized into a PIPELINE-FAULT lane result instead of
  killing the graph
- verdict_gate.py: PIPELINE-FAULT lane results block distinctly from
  SKIPPED and missing-sentinel; a PIPELINE-FAULT in signals_error blocks
  at the gauntlet level
- build_items.py: restructured into main() + fail-closed guard (crash
  emits empty lanes + PIPELINE-FAULT) and folds lanes_degraded into
  lanes_summary
- parse: state_updates/fallback -> parse_fault.py, which records the
  fault in signals_error and jumps to the verdict gate
- select_lanes: state_updates/fallback -> default_lanes.py, a
  deterministic wider-never-narrower lane fallback
- verification_commands threaded through parse into run_adversary's
  prompt (declared-only, never invented)
- run_probe prompt gains the aborted-attempt reconcile line
- 8 gauntlet_* R5 tests in src/config/request_context.rs (script harness
  + graph interpolation)
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 1e46743420 fix(TASK-017): bound run_checks runtime and fail-close its node-level death 
Adversary conformance-review fixes for the TASK-017 exec-evidence stage:

- run_checks.py: total-runtime deadline (3300s = node timeout minus margin);
  each command gets min(900s, remaining budget), commands past the deadline
  are recorded as {cmd, skipped: deadline} entries so in-script handling
  stays authoritative instead of the node timeout killing the script from
  outside. POSIX hardening: per-command process group (start_new_session)
  + killpg on timeout so hung grandchildren cannot linger; non-POSIX
  limitation documented in the docstring.
- graph.yaml: run_checks gains fallback: pipeline_fault; check_criterion
  treats SKIPPED records as at-best PARTIAL (unproven, never failed).
- pipeline_fault.py: run_checks-stage attribution grounded in real engine
  behavior (script-node fallback writes nothing into state, executor.rs
  script arm) — nonempty diff_text proves facts completed, so the fault is
  the verification runner; emits 'PIPELINE-FAULT: verification runner
  killed'. Empty diff_text still attributes to diff resolution.
- verdict.py: renders skipped records as [SKIPPED] instead of a bogus
  [FAIL] exit-? line; non-degraded verdict semantics untouched.
- tests: outer-except ENVIRONMENT degradation, deadline timeout+skip (env
  seam ADVERSARY_RUN_CHECKS_DEADLINE_SECS), pipeline_fault stage
  attribution (parse / facts / run_checks), check_criterion prompt cites
  {{exec_results}}, verdict SKIPPED rendering; run_adversary_script_env
  helper for env-carrying invocations.
- cargo fmt violations in 3d7dea9 test code fixed (fmt --check exit 0).
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 d50059a56f feat(TASK-017): adversary execution evidence + fail-closed routing 
- parse extracts caller-declared verification_commands (schema + initial_state);
  never invents commands. parse gains fallback + state_updates capturing the
  failure string (TASK-016 fallback-capture warning stays silent).
- new run_checks script node (facts -> run_checks -> map_criteria, timeout
  3600): runs each declared command in the project dir (900s per-command
  timeout), records exec_results {cmd, exit, duration_s, tail~50}; 'none
  declared' / ENVIRONMENT markers instead of guessing or failing the node.
- check_criterion judges execution criteria against {{exec_results}} only:
  recorded green run = MET citing it, recorded failure = UNMET, markers =
  PARTIAL with an ENVIRONMENT note.
- fail-closed routing (R1/R2): parse/facts fallback -> pipeline_fault marker
  script -> verdict; holistic fallback -> verdict with holistic_failure
  captured. verdict.py: any PIPELINE-FAULT forces DIVERGES with the fault as
  complaint #1, per-criterion results still reported; report gains a
  'Verification runs:' section. Non-degraded semantics unchanged.
- R5 rust tests (python3-guarded, local cmd_available): verdict fault path,
  holistic-failure fault, CONFORMS regression, run_checks markers + recorded
  runs, crit_gate contract regression.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 3dcb9541b1 test(TASK-016): assert fallback warn carries the error root cause 2026-09-14 10:50:33 -06:00
Dark-Alex-17 7dc950a092 test(TASK-016): assert the fallback warn names the node id and fallback target 
The adversary conformance review flagged that the fallback warn! at
outcome_from (src/graph/agent.rs:71) was never asserted — only the retry
warn had captured-text coverage. Adds one test installing the shared warn
collector, filtered by the unique node id 'fallback_warn_capture' for
parallel safety, asserting the warn names both the node and the fallback
target and carries the error chain.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 80ace77e44 feat(TASK-016): engine + test fixes from the Phase-1 re-review 
- warn-per-retry test via shared test-only WarnCollector (src/testing.rs)
- extraction-placement test: extraction failures are never retried
- is_transient_error: typed reqwest/Elapsed pre-check + anchored 429 matchers
- outcome_from docstring corrected; validator warning for
  fallback+output_schema without state_updates (llm + agent)
- validator error for max_attempts == 0 (llm + agent); runtime backstop kept
- nits: generalized branch-escape error text, Continue payload comment,
  BFS test assignments assert, node_id threaded into AgentNodeExecutor warns
- bundled-graph warning baseline records 3 true-positive step-runner nodes
  (orient, edge_case_sweep, write_handoff) pending state_updates there
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 18875a4cc9 feat(TASK-011): traverse agent fallback edges in map peer-discovery BFS 2026-09-14 10:50:33 -06:00
Dark-Alex-17 75cc2a4b9c feat(TASK-010): add max_attempts to read-only agent lanes in bundled graphs 2026-09-14 10:50:33 -06:00
Dark-Alex-17 be0c5462ed feat(TASK-009): add fallback routing to agent nodes 
Agent nodes gain an optional fallback target with llm-node parity: on any
failure (retries exhausted, hard error, or structured-extraction failure)
the failure chain is written into state as 'Agent node failed: <chain>'
and the chain routes to the fallback node. Without a fallback the error
propagates unchanged. Validator collects agent fallback edges for
missing-target checks, reachability, and map-branch locality.
 2026-09-14 10:50:33 -06:00
Dark-Alex-17 1c68331738 feat(TASK-008): add shared transient matcher and retry support to agent nodes 
Hoist the llm node's transient-error matcher to graph::is_transient_error,
extend it with broken-pipe / connection-error / error-sending-request
signatures, and give AgentNode an opt-in max_attempts retry loop with
llm-parity semantics: contexts applied before the transient check, fresh
wall-clock budget per attempt, peer identity (assignment + registry)
re-armed before each retry, retirement exactly once after the final
attempt, and structured extraction outside the loop.
 2026-09-14 10:50:33 -06:00
github-actions[bot] 677bce18b8 chore: refresh generated model catalog [skip ci] 2026-09-14 12:52:58 +00:00
github-actions[bot] 32625cdc76 chore: refresh generated model catalog [skip ci] 2026-09-14 00:55:14 +00:00
github-actions[bot] 50d1e78f6d chore: refresh generated model catalog [skip ci] 2026-09-13 12:30:29 +00:00
github-actions[bot] 2d43745b57 chore: refresh generated model catalog [skip ci] 2026-09-13 01:18:08 +00:00
github-actions[bot] b8466e4284 chore: refresh generated model catalog [skip ci] 2026-09-12 12:29:42 +00:00
Dark-Alex-17 d5d6dd9df4 test: fixed flaky windows timing test for graph agents 2026-09-12 02:00:25 -06:00
Alex Clarke fca086ba09 Merge pull request #25 from Dark-Alex-17/feat/spawn-agent-variables 
feat(agents): allow passing agent variables to agent__spawn and agent__task_create
 2026-09-12 01:44:08 -06:00
Dark-Alex-17 3254fe9f3c feat: Fixed graph agents with new agent variable passing functionality 2026-09-12 01:37:20 -06:00
Dark-Alex-17 a7528d6e1a feat(graph): warn when a declared variable is shadowed by an initial_state key 2026-09-12 01:37:20 -06:00
Dark-Alex-17 442953ef7d style: applied different styling to agent functions 2026-09-12 01:37:20 -06:00
Dark-Alex-17 1ec7916816 feat(agents): advertise declared agent variables in agent__list_available 2026-09-12 01:37:20 -06:00
Dark-Alex-17 eb7d1421da fix(tests): collision-proof parser test temp files to deflake macOS CI 2026-09-12 01:37:20 -06:00
Dark-Alex-17 f97a83f165 feat(agents): allow passing agent variables to agent__spawn and agent__task_create 
- agent__spawn accepts an optional variables object (string->string);
  values overlay the parent-inherited agent variables per key, with
  inheritance preserved as the fallback layer
- unknown variable names, non-string values, and non-object shapes
  return self-correcting status:error JSON listing the agent's
  declared variables
- spawned children can no longer trigger the interactive variable
  prompt: a new interactive gate in Agent::init_agent_variables routes
  missing required variables to the existing bail path while leaving
  CLI/REPL prompting and --info skip semantics unchanged
- agent__task_create persists variables on the task, agent__task_list
  echoes them, and task auto-dispatch threads them into the spawn
- tool schemas and built-in spawn docs updated; unit tests extended
  in agents.rs and taskqueue.rs
 2026-09-12 01:37:20 -06:00
Dark-Alex-17 3599b0a861 Merge branch 'main' of github.com:Dark-Alex-17/coyote 2026-09-11 22:42:20 -06:00
Dark-Alex-17 2e7cd6581e feat: seed variables into initial graph state 2026-09-11 22:42:07 -06:00
github-actions[bot] 406a850bdb chore: refresh generated model catalog [skip ci] 2026-09-12 00:48:47 +00:00
Dark-Alex-17 99e592565a feat: Significant code review suite rewrite with improved graph agent features 2026-09-11 15:33:21 -06:00
Dark-Alex-17 838335f84d feat: let install commands remove files that have been removed from bundles, both internal and external 2026-09-11 15:28:42 -06:00
Dark-Alex-17 b50ec42868 fix: Don't infinitely compress a session once it passes a threshold when the threshold is close to the max token count 2026-09-11 14:02:28 -06:00
Dark-Alex-17 e744da8870 feat: update the web_search_coyote tool to attempt to use the current model first then fallback to the configured model for web searches 2026-09-11 13:47:44 -06:00
Dark-Alex-17 eb78a0e32b feat: expose the current coyote model to tools 2026-09-11 13:47:20 -06:00
Alex Clarke e8237bd924 Merge pull request #24 from Dark-Alex-17/feat/map-subgraph-branches 
feat(graph): map branches as per-item subgraph chains + templated max_concurrency
 2026-09-11 13:12:09 -06:00
Dark-Alex-17 2f85462e2a fix: fix spinner ctrl-c early exit not clearing message and restoring cursor 2026-09-11 13:01:27 -06:00
Dark-Alex-17 3e91813cff feat: removed temperature specification in deep-research and finding-verifier agents to make them more univerally usable 2026-09-11 11:22:06 -06:00
Dark-Alex-17 99fb6e24f1 fmt: applied formatting 2026-09-11 10:56:28 -06:00
Dark-Alex-17 24b5504e21 fix(graph): build peer retirement guards before spawning their tasks 
A task aborted before its first poll never runs its body, so a guard
constructed inside the future did not exist yet and the pre-provisioned
teammate identity stayed messageable. Built outside and moved in, the
guard drops with the unpolled future on both the frontier and map items.

Also rewords the two timeout: 0 validator warnings: the graph-level one
no longer claims max_loop_iterations always applies (it may be 0 too),
and the node-level one names the enclosing bounds that still end the
node instead of saying it runs until it returns.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 02e2227aa0 fix(graph): reject agents that ship both config.yaml and graph.yaml at load 
Agent::init refuses that layout, so any agent node pointing at such a
directory failed at run time while validation passed it as graph-capable.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 778aa6449c fix(graph): bound concurrency caps by the semaphore permit limit 
Semaphore::new panics above tokio's MAX_PERMITS, so a fixed or templated
map cap (or settings.max_concurrency) that large crashed the run instead
of erroring. The validator rejects both at load, resolve_max_concurrency
rejects every arm at run time, and the executor checks the graph setting
before building its own semaphore, so validation mode makes no difference.

Also corrects run_child_agent_with_graph_inputs' contract comment: inputs
for a config-only agent are rejected, not ignored.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 a9f89f0414 test(graph): wait for the dispatcher before aborting the retirement test 
On slow runners bash start-up alone can exceed the fixed 300 ms delay, so
the abort landed before the worker tasks (and their retirement guards)
existed and the observer saw nothing. The dispatcher now leaves a marker
and the trigger waits for it.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 f072aa1b2c docs(graph): drop the output-null gotcha from the example graph 2026-09-11 10:56:28 -06:00
Dark-Alex-17 32f4adb71f fix(graph): make turn-cap and template checks consistent across validation modes 2026-09-11 10:56:28 -06:00
Dark-Alex-17 b4c01e0f28 chore(agents): drop the inert output reducers from librarian and finding-verifier 2026-09-11 10:56:28 -06:00
Dark-Alex-17 1a5824ce4f fix(graph): leave output absent after state_updates when it was absent before 2026-09-11 10:56:28 -06:00
Dark-Alex-17 324dd3397a docs(graph): document abort semantics and map binding rules in the example graph 
Comment-only: settings.timeout notes that Ctrl-C ends a run promptly from
anywhere; the map node's as:/output_key: comments state that the two must
differ, and max_concurrency describes the template form as exactly one
{{key}} with nothing around it.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 727cab11ff fix(graph): validate the whole max_concurrency template string 
A string max_concurrency passed the load-time check whenever it merely
contained a {{...}} pair, so forms like "n={{k}}" or "{{ key }}" that
can never resolve at runtime were only caught after the map's inputs had
been computed. The check now requires the trimmed string to be exactly
one template reference, matching what interpolate_raw resolves as a pure
reference; padded forms like " {{budget}} " stay valid.

Also: the bash tool build-failure context names the source file instead
of the destination path, and the map-subgraph validator no longer
shadows its member id list with the resolved member nodes.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 ccec38c5b9 fix(graph): reject as == output_key on map nodes 
A map node whose item binding shares its name with output_key would have
the binding collected as the chain's result whenever the chain never
wrote it, while the missing-writer warning stayed silent. The collision
is now a validator error attributed to the map node and, with validation
off, a runtime error in run_map before any item forks.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 cce8381962 feat(graph): thread the graph abort into llm nodes and tool execution 
LlmNodeExecutor::execute now takes the graph's abort signal and threads
it through run -> run_with_retries -> run_chat_loop, replacing the
loop-private signal, so an in-flight model call is interrupted and every
turn starts with an abort check that bails 'llm node aborted' before any
client is created.

run_llm_function gains an optional abort signal: the synchronous tool
wait loop kills the child and returns a 'Tool call ... aborted' error
within one poll interval when it is set. The tool dispatcher passes the
session abort; the agent _instructions runner passes None.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 1ea755e864 feat(graph): abort a graph run on Ctrl-C from anywhere 
A graph run now observes user interrupts wherever it is: GraphExecutor::run
spawns a run-scoped bridge that maps SIGINT or the enclosing session abort
onto the graph's own abort flag (only when a session signal exists; a
pre-set session trips a synchronous check). The frontier join selects on
the abort and, like the timeout path, cancels every in-flight branch task,
failing the run with 'Graph ... aborted during super-step'.

Branch and map item tasks are cancellation-owned via TaskCancelGuard, so
dropping a map task's future no longer detaches its item chains. Peer
retirement is a Drop guard (PeerRetireGuard) constructed before the
semaphore acquire and dropped explicitly right after step(), so the
success-path ordering is unchanged while task abort, panic, and the
pre-step abort return all still retire the identity.
 2026-09-11 10:56:28 -06:00
Dark-Alex-17 6aad82c6fd fix(graph): inherit the session abort in agent-node children 
A child agent spawned from a graph node now carries its parent's session
abort signal, so a user interrupt is observable from every sync wait in
the agent tree. Background agent__spawn children are unchanged.

run_child_agent_with_graph_inputs also guards the child's own supervisor
with a disarmable CancelOnDrop: if the future is dropped mid-flight or
the child fails before its normal cleanup, its spawned subagents are
cancelled instead of running on invisibly.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 aa1287b902 docs(graph): document llm max_iterations: 0 in the example graph 2026-09-11 10:56:27 -06:00
Dark-Alex-17 d7a16363b8 feat(graph): warn when an llm node's turn cap is disabled 2026-09-11 10:56:27 -06:00
Dark-Alex-17 36601d8668 feat(graph): treat llm max_iterations: 0 as no turn cap 2026-09-11 10:56:27 -06:00
Dark-Alex-17 8e9a1f1ee8 docs: removed unnecessary comment on GraphState.history field 2026-09-11 10:56:27 -06:00
Dark-Alex-17 8f9284cb92 refactor(graph): keep the ordered visit log test-only 
GraphState.history was pushed to on every node visit and deep-cloned on
every super-step snapshot and map fork, but its only reader is the
cfg(test) current_node accessor. Gate the field, its initializer, and
the push behind cfg(test) so production builds no longer carry an
unbounded Vec<String> that nothing observes.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 69a3d928ee docs(graph): document max_loop_iterations: 0 in the example graph 2026-09-11 10:56:27 -06:00
Dark-Alex-17 4c5aa8d895 feat(graph): warn when the visit cap is disabled 
validate_timeouts now also flags settings.max_loop_iterations: 0, ahead
of the settings.timeout: 0 warning so both opt-outs read top-to-bottom
as they appear in settings. (design §3.14 / C22)
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 3aa35e1259 feat(graph): treat max_loop_iterations: 0 as no visit cap 
Both enforcement sites (frontier loop and map chain step) skip the
comparison when the cap is 0; visits are still counted and logged.
Adds an executor test that loops past the default cap with 0, a test
pinning the frontier cap message for a non-zero value, and a map-chain
twin at 0. (design §3.14 / C22)
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 4dee1a6849 docs: Removed unnecessary unicode arrow from example graph config 2026-09-11 10:56:27 -06:00
Dark-Alex-17 f342dd7dd7 docs(graph): document timeout: 0 in the example graph 
Comment-only: each of the five timeout fields in graph.example.yaml now
says that 0 disables the bound. Parsed content is unchanged.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 cd0318f384 feat(graph): warn when a timeout is disabled 
A timeout: 0 on settings.timeout or on an agent/script/llm/rag node is
the author's explicit opt-out of the wall-clock bound. It stays legal;
the validator now emits one warning per zero so the choice is visible in
--validate output. Non-zero and unset values do not warn (unset already
means no bound for settings.timeout and llm nodes, matching the
explicit-zero-only convention of the orchestration-limit warnings).

Design §3.13 / C21.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 d630c7e30b feat(graph): treat timeout: 0 as no wall-clock bound 
A zero on any of the five timeout fields (settings.timeout, agent, script,
llm, rag) now disables the bound instead of firing an immediate
"timed out after 0s" or graph bail. Unset values keep their defaults
(agent 300, script 30, rag 120, llm/settings none); non-zero values are
unchanged. One resolution helper, wall_clock(secs) -> Option<Duration>,
feeds all five call sites; the C17 peer-retirement mark and every error
message stay where they were.

The one existing test that relied on timeout: 0 firing
(run_item_chain_marks_identity_finished_when_agent_step_times_out) moves
to timeout: 1 with its 5s hold unchanged.

Design §3.13 / C21.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 b666b7e494 docs(graph): illustrate the inputs map on the example agent node 
Comment-only block on deep_dive; the example's parsed YAML is unchanged.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 e26543f21a test(graph): cover agent node inputs end to end and at the runtime guards 
Two offline chain tests materialize an echo-inputs graph agent whose
script reports the width it sees: a parent with inputs.width gets the
parent's value back, a parent without inputs gets the child's YAML
default. Two agents.rs tests pin the exact runtime guard texts for a
config-only target and a reserved initial_prompt key.

Task T5 of plans/map-subgraph-branches-design.md §3.12.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 10b8fe5cae feat(graph): validate agent node inputs at load time 
Inside validate_agents: an agent node with inputs whose target has a
config.yaml but no graph.yaml is an error, and an inputs key named
initial_prompt is rejected. primary_templated_fields now includes every
inputs value for agent nodes, so validate_parallel_reads treats them as
cross-branch reads with no new pass.

Design: plans/map-subgraph-branches-design.md §3.12 V10a–V10c, task T5.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 3393123baf feat(graph): seed a child graph agent's state from an agent node's inputs map 
Agent nodes gain an optional `inputs:` map of templates resolved against
the parent state and overlaid on the child graph agent's initial_state
before its executor starts. A lone {{key}} keeps the state value's JSON
type; mixed text renders to a string; initial_prompt is still owned by the
dispatcher. run_child_agent and run_active_agent_graph keep their
signatures as wrappers over the new *_with_inputs variants, and
run_agent_for_graph rejects inputs aimed at a config-only agent or
containing initial_prompt.

Design: plans/map-subgraph-branches-design.md §3.12 (C18), task T5.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 0bef0f2c39 fix(graph): retire the teammate identity before structured extraction 
PR #24 follow-up 10. The identity was only retired by the frontier and
chain-exit safety nets, so a peer stayed messageable during
structured::extract after its agent had already stopped. Mark it
finished the moment the agent future resolves, on success and on both
error paths.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 874ef7f6ec fix(graph): reject a literal max_concurrency of 0 instead of clamping it 
Copilot review on PR #24. The validator already rejects 0 and the
template path errors on it, but a literal 0 loaded with
validate_before_run off was silently clamped to 1. Fail with the same
error shape as the template path; only the absent-cap default clamps.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 e746ca1f87 fix(graph): clear peer identity fields after every map chain step 
Copilot review on PR #24. The fields were only ever cleared by
run_agent_for_graph taking them, so a failure between arming and that
take (e.g. prompt interpolation) left them set on the item's ctx.
Reset both unconditionally once the step returns.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 dc5e3201da refactor(graph): move the cfg(test) OUTPUT_KEY import into llm's test module 
PR #24 follow-up 7.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 cdbff88b4f fix(graph): pick the map teammate identity name by BFS from the branch entry 
PR #24 follow-up 6. The identity used to fall back to YAML declaration
order after the entry, so the name depended on how the author ordered
the nodes map. Walk next/fallback edges from the entry instead.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 62122fc698 refactor(graph): drop the duplicated locator prefix from map chain step errors 
PR #24 follow-ups 3 and 4. run_chain_steps already wraps every step
error with the map/item/node/step locator, so the inner messages
repeated it. Make them prefix-free and turn the unreachable End arm
into an honest internal-invariant message.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 31c33b5a23 fix(graph): name the offending node type when a map branch reaches a disallowed node 
PR #24 follow-up 2. With validate_before_run off, a user-authored
approval/input/end/map node inside a branch reached a message that
called itself an internal error. Name the node type and point at the
validator setting instead.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 2f839b332d fix(function): build bundled bash tools without writing back to the source tree 
PR #24 follow-up 1. The bundled-tools test called the writing
generate_bash_declarations on the repo's assets/functions/tools/*.sh,
so cargo test could rewrite tracked files. Split the build+export step
into a pure build_bash_tool so the test never touches the source tree.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 b65aa76d2d docs: updated the formatting of a few comments 2026-09-11 10:56:27 -06:00
Dark-Alex-17 d8f1778bc7 docs: describe map branches as per-item subgraph chains in graph.example.yaml 
Comment-only rewrite of the map/branch and teammates comments: a branch is
the entry of a per-item chain; chain nodes may use next, state_updates,
output_schema, fallback and script _next inside the chain; output_key is
removed from the fork and must be written by the chain; max_concurrency
accepts an integer or a template; a teammate identity spans the item's
whole chain.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 c10965cd71 test(graph): pin the per-item retry chain and bundled validator warning parity 
Add the motivating map-chain integration test: draft -> check, where check
re-routes to draft via script `_next` until an item-local attempts counter
reaches 2, run with validate_before_run on. A sibling test drives the same
map node against a parent state to pin that only collect_into lands on the
parent and that no chain visit touches the parent's loop counts.

Extend bundled_graph_agents_parse_and_validate: the sorted (node, warning)
set of every bundled graph must equal the set the pre-subgraph validator
produced, and graph.example.yaml must load through GraphParser, validate
without errors, and declare a writer for its map's output_key.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 b4a73aae7f test(graph): pin teammate retirement on agent timeout and frontier completion 2026-09-11 10:56:27 -06:00
Dark-Alex-17 846bc39263 test(graph): run a validated two-step map chain end to end 
Same two-script chain as `map_chain_two_script_steps_route_via_next`,
with `validate_before_run: true`, proving the validator admits the
chains the runtime executes.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 824edf587b feat(graph): validate map branch subgraphs instead of single branch nodes 
`validate_map_branches` checked only the entry node and forbade `next`,
non-`output_key` `state_updates` and `output_schema` on it. The runtime
now executes the whole subgraph reachable from `branch`, so the validator
follows suit as `validate_map_subgraphs`, at the same position in
`validate()`:

- approval/input/end/map nodes are rejected anywhere in the subgraph,
  attributed to the map node; when the offending node is a `fallback`
  target the message says why a previously accepted `fallback` now fails
- a fan-out `next` inside the subgraph is rejected
- a subgraph that shares nodes with the main flow (reachable from
  `start` without entering any map branch, via the new
  `main_flow_reachable`) is rejected, naming the shared nodes
- a new warning fires when no subgraph node declares a write to
  `output_key` via `state_updates` or `output_schema`; any script in the
  subgraph suppresses it, since scripts may write any key
- the `next`, `state_updates` and `output_schema` restrictions are gone:
  intra-branch writes are fork-local scratch

Two maps may share a subgraph. `find_reachable_nodes` and
`compute_parallel_groups` are unchanged.

Tests: the three rules that inverted are renamed to say what they now
assert (`map_branch_with_in_subgraph_next_passes`,
`map_branch_state_updates_scratch_key_passes`,
`map_branch_with_output_schema_passes`); the four
`map_branch_cannot_be_*` tests pass unmodified. Thirteen new tests cover
each rule, the fallback hint, the missing-writer warning and its three
suppressions, a three-node chain, and `main_flow_reachable`.
 2026-09-11 10:56:27 -06:00
Dark-Alex-17 0083cf0c42 test(graph): cover map subgraph chains, output contract and teammate lifetime 2026-09-11 10:56:26 -06:00
Dark-Alex-17 f1cf9a6ae7 feat(graph): run map branches as per-item subgraph chains 
A map item no longer runs a single branch node; it walks the subgraph
reachable from `branch` over `next`/`fallback` edges (branch_subgraph),
one chain per item, holding one semaphore permit for the whole chain.
Chain steps go through the shared `step()` so llm retries, fallbacks,
script `_next` and agent timeouts keep their existing semantics. A node
without `next` ends the chain (StepContext.branch_mode), a route outside
the subgraph, a fan-out, an end node or a loop past
max_loop_iterations fails the item, and every item error carries
`map node 'X': sub-branch [i] failed at node 'Y' (step n)`.

`output_key` is removed from each item's fork before the chain runs, so
the collected value must be written by the chain rather than inherited
from the parent.

Teammate identities are now per item for the whole chain: provisioned
when any flagged agent is in the subgraph, armed only before
`teammates: true` agent steps, and retired at chain exit. mark_finished
moves out of run_agent_for_graph to the two provisioning sites (frontier
task and chain exit), so timed-out and early-failing agents are also
marked finished.
 2026-09-11 10:56:26 -06:00
Dark-Alex-17 2049316364 feat(graph): accept a {{template}} for a map node's max_concurrency 
`MapNode.max_concurrency` becomes `Option<ConcurrencyCap>`, where
`ConcurrencyCap` is `Fixed(usize) | Template(String)`. Integers parse
exactly as before (`4` -> `Fixed(4)`, `0` -> `Fixed(0)`, absent -> `None`);
a string parses as a template that the map executor resolves against the
parent state right before the fan-out via `StateManager::interpolate_raw`,
mirroring `over:`. A template must resolve to an integer >= 1 or a numeric
string (scripts often emit numbers as strings); anything else, including 0,
is an error naming the map node, the template, and the resolved value/type.
The fixed path keeps its `.max(1)` clamp for byte-compatibility.

`Deserialize` is hand-written so a malformed value (`-1`, `4.5`, `true`,
a list, a map) reports the offending value and the accepted forms rather
than serde's opaque untagged-enum "did not match any variant".

Validator: a string cap that contains no `{{ ... }}` pair is a load-time
error (catches `"4"` typed for `4`); a templated cap joins the map node's
read set so `validate_parallel_reads` flags a read of a sibling branch's
write. The existing `Fixed(0)` rule and every other rule keep their order.

Design: plans/map-subgraph-branches-design.md §3.5, §3.11 (C3, I3).
 2026-09-11 10:56:26 -06:00
Dark-Alex-17 a7744e7f89 fix(agents): name the map-branch output key in deep-research and finding-verifier 
An llm node never persists its completion under `output` (the binding
is transient), so a map branch without state_updates collected the
stale inherited value: deep-research produced `None` findings and
finding-verifier errored on its first item because `output` was absent.

Both maps now declare an explicit output_key and their branch nodes
write it via state_updates. graph.example.yaml's research_subject gets
the same named-key pattern and its branch comment describes it.

Design: plans/map-subgraph-branches-design.md §2.6, §3.7 (C9).
 2026-09-11 10:56:26 -06:00
Dark-Alex-17 24088a48a1 fix(graph): honor an explicit output key in state_updates 
Hoist the three copies of apply_state_updates (llm, agent, rag) into
graph::state_updates::apply(state, output, has_schema, updates). The
helper computes every state_updates value first, restores the transient
`output` binding, and only then writes the computed updates, so a
`state_updates: { output: "{{output}}" }` entry is no longer clobbered
by the restore step.

The per-file wrappers stay as one-line calls so their existing tests are
unchanged; new direct helper tests live in state_updates.rs, including
the regression test for the clobber.

Design: plans/map-subgraph-branches-design.md §2.6, §3.7 (C5, D3).
 2026-09-11 10:56:26 -06:00
github-actions[bot] 1df82b50b6 chore: refresh generated model catalog [skip ci] 2026-09-11 12:33:25 +00:00
github-actions[bot] 452d9e1b60 chore: refresh generated model catalog [skip ci] 2026-09-11 00:49:42 +00:00
github-actions[bot] 0f0f5448ef chore: refresh generated model catalog [skip ci] 2026-09-10 12:33:02 +00:00
github-actions[bot] 6e1683393f chore: refresh generated model catalog [skip ci] 2026-09-10 00:49:13 +00:00
Dark-Alex-17 d71cf75209 fix: Fixed the bad formatting in the finding-verifier agent 2026-09-09 17:39:56 -06:00
Dark-Alex-17 ddce9e330e docs: Added whetstone to the architect and sisyphus READMEs 2026-09-09 17:39:42 -06:00
Dark-Alex-17 2928125b0e feat: created the finding-verifier and whetstone agents for continuous code quality improvements 2026-09-09 17:22:26 -06:00
Dark-Alex-17 71c3283b87 test: updated macro test 2026-09-09 16:44:23 -06:00
Dark-Alex-17 47edaa35ff feat: teammates flag enabling peer messaging between concurrent graph agent nodes 
Agent nodes marked teammates: true whose sub-agents run concurrently (map fan-out branches or multiple flagged agent nodes in one super-step) now form a team: each member gets a pre-provisioned inbox in a messaging-only PeerRegistry before any branch starts, a teammate roster injected into its prompt, peer resolution in agent__send_message, and a peers section in agent__list_running. Registries are scoped to a single fan-out and carry no lifecycle semantics: outputs still flow exclusively through collect_into/state reducers, finished peers reject new messages, and nothing registers in a supervisor. Also hardens the agent-recursion depth guard to work without a supervisor and adds validator warnings for zero-valued orchestration limits.
 2026-09-09 16:44:23 -06:00
Dark-Alex-17 7e84804ed1 feat: support explicit can_spawn_agents, max_concurrent_agents, and max_agent_depth keys in graph.yaml 
Graph agents previously derived can_spawn_agents from the presence of an agent node, so a pure-LLM-node graph could never spawn dynamic sub-agents. The three optional graph-level keys now override the derivation/defaults in AgentConfig::from_graph, enabling orchestrator graphs (can_spawn_agents: true with only llm nodes). Explicit false on a graph with agent nodes still runs those nodes; only dynamic spawning from llm nodes is disabled.
 2026-09-09 16:44:23 -06:00
Dark-Alex-17 7f5b75dbdf fix: properly anchored the role frontmatter regex so it doesn't catch in-instruction separators 2026-09-09 13:23:33 -06:00
Dark-Alex-17 3865750f5a Merge branch 'main' of github.com:Dark-Alex-17/coyote 2026-09-09 12:33:04 -06:00
Dark-Alex-17 dbec2f77d1 fix: support non-object tool result outputs for bedrock 2026-09-09 12:32:54 -06:00
github-actions[bot] 6a4352d8b4 chore: refresh generated model catalog [skip ci] 2026-09-09 12:32:09 +00:00
github-actions[bot] 3e8ecdbeab chore: refresh generated model catalog [skip ci] 2026-09-09 00:52:07 +00:00
Dark-Alex-17 13ebe04472 feat: support setting the tool_timeout in the main config file instead of only via an environment variable 2026-09-08 17:22:21 -06:00
Dark-Alex-17 695153f88e feat: Added new environment variable to force showing tool call logging when not in a tty 2026-09-08 16:47:40 -06:00
Dark-Alex-17 6104a48e56 Merge branch 'main' of github.com:Dark-Alex-17/coyote 2026-09-08 11:25:38 -06:00
Dark-Alex-17 a5984eca80 fix: exclude null bytes from fs_grep output on binaries 2026-09-08 11:24:59 -06:00
Dark-Alex-17 4651658b34 fix: suppress subagent error and warning output to stderr/stdout 2026-09-08 11:24:43 -06:00
github-actions[bot] fc8114433b chore: refresh generated model catalog [skip ci] 2026-09-08 12:35:17 +00:00
github-actions[bot] 7c09593c84 chore: refresh generated model catalog [skip ci] 2026-09-08 00:52:16 +00:00
github-actions[bot] e0845ff8e9 chore: refresh generated model catalog [skip ci] 2026-09-07 12:34:18 +00:00
github-actions[bot] ed41844d6f chore: refresh generated model catalog [skip ci] 2026-09-07 00:55:46 +00:00
github-actions[bot] 34f60ead6c chore: refresh generated model catalog [skip ci] 2026-09-06 12:30:32 +00:00
github-actions[bot] 106813263a chore: refresh generated model catalog [skip ci] 2026-09-06 00:56:42 +00:00
github-actions[bot] ff6289c89a chore: refresh generated model catalog [skip ci] 2026-09-05 12:30:54 +00:00
github-actions[bot] cec3453da8 chore: refresh generated model catalog [skip ci] 2026-09-05 00:49:21 +00:00
github-actions[bot] 66ced35943 chore: refresh generated model catalog [skip ci] 2026-09-04 12:56:04 +00:00
github-actions[bot] d6dc6c96d6 chore: bump Cargo.toml and sandbox image to 0.10.2 2026-09-04 04:15:00 +00:00
github-actions[bot] cb0c3a9653 bump: version 0.10.1 → 0.10.2 [skip ci] 2026-09-04 04:14:49 +00:00
Dark-Alex-17 a0dc4550b1 feat: added a clear line separator for the end of LLM turns to make it more clear where they are 2026-09-03 22:01:54 -06:00
Dark-Alex-17 f66036e9d8 feat: sort models by release date 2026-09-03 21:48:10 -06:00
github-actions[bot] 61c3350ad3 chore: refresh generated model catalog [skip ci] 2026-09-04 03:19:55 +00:00
Dark-Alex-17 9f389ba82b ci: fix the update-models workflow 2026-09-03 20:45:45 -06:00
Dark-Alex-17 2ee4e58933 Merge branch 'main' of github.com:Dark-Alex-17/coyote 2026-09-03 19:55:14 -06:00
Dark-Alex-17 eaab57375e fix: --generate-models parsed before vault initialization 2026-09-03 19:55:03 -06:00
145 changed files with 34001 additions and 4394 deletions
Expand all files Collapse all files

No files matched your search

.github/workflows/ci.yaml
+1 -1
View File
@@ -3,7 +3,7 @@ name: CI
on:
pull_request:
branches:
- '*'
- '**'
push:
branches:
- main
.github/workflows/update-models.yaml
+52 -57
View File
@@ -6,7 +6,7 @@ on:
workflow_dispatch:
permissions:
contents: write
contents: read
concurrency:
group: update-models
@@ -26,70 +26,65 @@ jobs:
RUSTFLAGS: --deny warnings
steps:
- uses: actions/checkout@v4
with:
persist-credentials: true
- uses: actions/checkout@v4
with:
ssh-key: ${{ secrets.RELEASE_BOT_SSH_KEY }}
persist-credentials: true
- name: Install Rust Toolchain Components
uses: dtolnay/rust-toolchain@stable
- name: Install Rust Toolchain Components
uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: Swatinem/rust-cache@v2
- name: Cache DuckDB Extensions
id: duckdb-extensions
uses: actions/cache@v4
with:
path: ~/.duckdb/extensions
key: duckdb-ext-ubuntu-latest-${{ hashFiles('Cargo.lock') }}
- name: Cache DuckDB Extensions
id: duckdb-extensions
uses: actions/cache@v4
with:
path: ~/.duckdb/extensions
key: duckdb-ext-ubuntu-latest-${{ hashFiles('Cargo.lock') }}
- name: Install DuckDB Extensions
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
run: cargo test --all duckdb
- name: Install DuckDB Extensions
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
run: cargo test --all duckdb
# Coyote loads its config before handling --generate-models/--list-models, and a
# bare non-TTY runner has no config file (and no way to answer the first-run
# prompt), so point COYOTE_CONFIG_DIR at a throwaway dir with a minimal stub.
- name: Write stub config
run: |
COYOTE_CONFIG_DIR="$(mktemp -d)"
printf 'clients:\n - type: openai\n - type: claude\n' > "$COYOTE_CONFIG_DIR/config.yaml"
echo "COYOTE_CONFIG_DIR=$COYOTE_CONFIG_DIR" >> "$GITHUB_ENV"
- name: Write stub config
run: |
COYOTE_CONFIG_DIR="$(mktemp -d)"
printf 'clients:\n - type: openai\n - type: claude\n' > "$COYOTE_CONFIG_DIR/config.yaml"
echo "COYOTE_CONFIG_DIR=$COYOTE_CONFIG_DIR" >> "$GITHUB_ENV"
- name: Regenerate model catalog
run: cargo run --release -- --generate-models
- name: Regenerate model catalog
run: cargo run --release -- --generate-models
- name: Detect catalog changes
id: diff
run: echo "changed=$(git diff --quiet models.yaml && echo false || echo true)" >> "$GITHUB_OUTPUT"
- name: Detect catalog changes
id: diff
run: echo "changed=$(git diff --quiet models.yaml && echo false || echo true)" >> "$GITHUB_OUTPUT"
# The binary built by the regeneration step embedded the OLD models.yaml (the file
# was rewritten after that build), so rebuild before smoke-testing: cargo tracks
# include_str! inputs, making this a cheap incremental rebuild of the final crate.
- name: Rebuild with the refreshed catalog
if: steps.diff.outputs.changed == 'true'
run: cargo build --release
# The binary built by the regeneration step embedded the OLD models.yaml (the file
# was rewritten after that build), so rebuild before smoke-testing: cargo tracks
# include_str! inputs, making this a cheap incremental rebuild of the final crate.
- name: Rebuild with the refreshed catalog
if: steps.diff.outputs.changed == 'true'
run: cargo build --release
- name: Test
if: steps.diff.outputs.changed == 'true'
run: cargo test --all
- name: Test
if: steps.diff.outputs.changed == 'true'
run: cargo test --all
- name: Smoke test the refreshed catalog
if: steps.diff.outputs.changed == 'true'
run: |
COUNT=$(./target/release/coyote --list-models | wc -l)
echo "Listed $COUNT chat models"
test "$COUNT" -ge 40
- name: Smoke test the refreshed catalog
if: steps.diff.outputs.changed == 'true'
run: |
COUNT=$(./target/release/coyote --list-models | wc -l)
echo "Listed $COUNT chat models"
test "$COUNT" -ge 40
# Pushes made with the default GITHUB_TOKEN do not trigger other workflows,
# so this push cannot start a workflow loop. The pull --rebase covers a human
# push landing mid-run; if the race is lost anyway, the next cron run retries.
- name: Commit and push
if: steps.diff.outputs.changed == 'true'
run: |
git add models.yaml
git checkout -- .
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -m 'chore: refresh generated model catalog'
git pull --rebase origin main
git push origin main
- name: Commit and push
if: steps.diff.outputs.changed == 'true'
run: |
git add models.yaml
git checkout -- .
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -m 'chore: refresh generated model catalog [skip ci]'
git pull --rebase origin main
git push origin main
CHANGELOG.md
+39
View File
@@ -1,3 +1,42 @@
## v0.10.2 (2026-09-04)
### Feat
- added a clear line separator for the end of LLM turns to make it more clear where they are
- sort models by release date
- Improved the architect, explore, and sisyphus agents to have a team mode so they can all work in parallel as a team and updated the parallel-research skill to make explicit mention of this functionality
- support auto continuation in subagents and prevent them from calling todo__pause
- apply provider quirks to models created from a name
- refresh the model catalog from live API data at runtime
- generate the model catalog from live API data
- add model catalog source adapters and merge engine
- add hand-owned model quirks overlay
- apply provider quirks to models created from a name
- refresh the model catalog from live API data at runtime
- generate the model catalog from live API data
- add model catalog source adapters and merge engine
- add hand-owned model quirks overlay
- Improved the architect, explore, and sisyphus agents to have a team mode so they can all work in parallel as a team and updated the parallel-research skill to make explicit mention of this functionality
- added tab completions to --update-bundle flag
- added a dedicated config_dir output to coyote --info/.info
### Fix
- children subagents couldn't properly send messages to parent inboxes
- --generate-models parsed before vault initialization
- drive tool rounds iteratively instead of recursively to prevent stack overflow on long agentic turns to fix latent stack overflow recursion bug
- session replays now include mid-turn messages so the full session gets replayed
- resolve duplicate model ids by requested type
- let ctrl-c interrupt blocking job and agent collects
- skip terminal color-scheme queries in test builds
- memoize quirk glob patterns and isolate concurrent override writes
- memoize quirk glob patterns and isolate concurrent override writes
- children subagents couldn't properly send messages to parent inboxes
### Perf
- probe models-override staleness in the background refresh task
## v0.10.1 (2026-08-31)
### Fix
Cargo.lock
Generated
+52 -42
View File
@@ -411,9 +411,9 @@ dependencies = [
[[package]]
name = "async-compression"
version = "0.4.43"
version = "0.4.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3976abdc8fe7d1133d43d304afd42abdf5bc3e1319d263d223bde07b5efc4be8"
checksum = "515a1f282e33d55983c499d7e9e87082e81cbc32974825bf9032f928392d5844"
dependencies = [
"compression-codecs",
"compression-core",
@@ -520,11 +520,11 @@ dependencies = [
[[package]]
name = "aws-lc-rs"
version = "1.18.0"
version = "1.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e"
checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
dependencies = [
"aws-lc-sys 0.44.0",
"aws-lc-sys 0.45.0",
"zeroize",
]
@@ -543,9 +543,9 @@ dependencies = [
[[package]]
name = "aws-lc-sys"
version = "0.44.0"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483"
checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
dependencies = [
"cc",
"cmake",
@@ -1513,9 +1513,9 @@ dependencies = [
[[package]]
name = "compression-codecs"
version = "0.4.38"
version = "0.4.39"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf"
checksum = "2fe67f2944eef52fc7b106b8c9450d243a88701a0c065f7f57235e76abaed7df"
dependencies = [
"compression-core",
"flate2",
@@ -1524,9 +1524,9 @@ dependencies = [
[[package]]
name = "compression-core"
version = "0.4.32"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789"
checksum = "6e8ccc4ea9f6acc32d102c0f6d471d11d913ad15f20c04de743374861fa1d414"
[[package]]
name = "confy"
@@ -1663,7 +1663,7 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "coyote-ai"
version = "0.10.1"
version = "0.10.2"
dependencies = [
"ansi_colours",
"anyhow",
@@ -1845,7 +1845,7 @@ dependencies = [
"parking_lot",
"rustix 1.1.4",
"serde",
"signal-hook",
"signal-hook 0.3.18",
"signal-hook-mio",
"winapi",
]
@@ -2297,9 +2297,9 @@ dependencies = [
[[package]]
name = "duct"
version = "1.1.1"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e66e9c0c03d094e1a0ba1be130b849034aa80c3a2ab8ee94316bc809f3fa684"
checksum = "61b9e2a29ff01e8bb526a571ad06c10ed72aae80d5999ed204f7971f99f19974"
dependencies = [
"libc",
"os_pipe",
@@ -3838,9 +3838,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
[[package]]
name = "libredox"
version = "0.1.21"
version = "0.1.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7955dfc218a8afb29dfeffd540e3a6e96baeb94fe7138228dd7cc6937fbbf96"
checksum = "8d8f1ea3f21fd3405dcaf6c9b5c1630af9afc422d9073ea39c5f6d6c772e08ed"
dependencies = [
"libc",
]
@@ -4061,9 +4061,9 @@ dependencies = [
[[package]]
name = "mio"
version = "1.2.2"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427"
checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8"
dependencies = [
"libc",
"log",
@@ -4480,9 +4480,9 @@ checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
[[package]]
name = "open"
version = "5.4.2"
version = "5.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ade3be4664bc1ef537ce133015f04c176b737815c2ba9fd60edf212d6e90dd55"
checksum = "7c603ab8300cf18bc3b14146b19fe3dfcc4843ae5a400cd0e7a30b95aa366634"
dependencies = [
"is-wsl",
"libc",
@@ -5371,9 +5371,9 @@ dependencies = [
[[package]]
name = "rmcp"
version = "3.1.4"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a15bc53261a9dc37e105df006e4656c598379a8f9581f8950debb130f27a7cf"
checksum = "42b6914fac0be956fe704a38239c3f44a9f841d1b06a5713d2f638065593f5b5"
dependencies = [
"base64 0.23.1",
"bytes",
@@ -5400,9 +5400,9 @@ dependencies = [
[[package]]
name = "rmcp-macros"
version = "3.1.4"
version = "3.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a85d45508e9b4ba024fe996c2638799635d75b6dd0ba8f32ccf08f8026f0c780"
checksum = "cdf1c49bd4d52014b94db0877410db273c2008f01628b0252a2e9460ad9b7fda"
dependencies = [
"darling 0.24.1",
"proc-macro2",
@@ -6034,20 +6034,20 @@ dependencies = [
[[package]]
name = "shared_child"
version = "1.1.1"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e362d9935bc50f019969e2f9ecd66786612daae13e8f277be7bfb66e8bed3f7"
checksum = "607549934f6cc26b89cfecfdc46fa90f1e5d1536a68349b0c3a4f9d1c0d37959"
dependencies = [
"libc",
"sigchld",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
name = "shared_thread"
version = "0.2.0"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52b86057fcb5423f5018e331ac04623e32d6b5ce85e33300f92c79a1973928b0"
checksum = "de1c6cdf07f3a4b1900680728ac1a12f72aa6424f37138f9253116461576e00f"
[[package]]
name = "shell-words"
@@ -6069,13 +6069,13 @@ checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "sigchld"
version = "0.2.4"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47106eded3c154e70176fc83df9737335c94ce22f821c32d17ed1db1f83badb1"
checksum = "24f2b37f04360cd465089b87a9c3869c08220a2f3458463f0adf8badf5e77f2c"
dependencies = [
"libc",
"os_pipe",
"signal-hook",
"signal-hook 0.4.4",
]
[[package]]
@@ -6088,6 +6088,16 @@ dependencies = [
"signal-hook-registry",
]
[[package]]
name = "signal-hook"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2a0c28ca5908dbdbcd52e6fdaa00358ab88637f8ab33e1f188dd510eb44b53d"
dependencies = [
"libc",
"signal-hook-registry",
]
[[package]]
name = "signal-hook-mio"
version = "0.2.5"
@@ -6096,7 +6106,7 @@ checksum = "b75a19a7a740b25bc7944bdee6172368f988763b744e3d4dfe753f6b4ece40cc"
dependencies = [
"libc",
"mio",
"signal-hook",
"signal-hook 0.3.18",
]
[[package]]
@@ -6167,9 +6177,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "smallvec"
version = "1.15.2"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"
checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f"
[[package]]
name = "smawk"
@@ -6220,9 +6230,9 @@ dependencies = [
[[package]]
name = "sse-stream"
version = "0.2.5"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c123f296ade4ec4b8b0f6162116e6629f5146922ca5ab40ca9d3c2e73ab4761e"
checksum = "c25ac7aff0abd1dbc474536e40416e1102c7dd9bfba0b9861c6d357f835dcfb4"
dependencies = [
"bytes",
"futures-util",
@@ -6621,9 +6631,9 @@ dependencies = [
[[package]]
name = "tinyvec"
version = "1.12.0"
version = "1.13.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb4ebadaa0af04fab11ae01eb5f9fdb5f9c5b875506e210e71c07873528baa7f"
checksum = "4cf0ded5c4e56918d8f8a339e1bb67d038d3bc6d144ac407904015ba2e4cde9b"
dependencies = [
"tinyvec_macros",
]
@@ -6719,9 +6729,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.1.4+spec-1.1.0"
version = "1.1.5+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3aace63f4bbcdfc2c965b059de67119c89c4017a70d633be6c104910f67056f5"
checksum = "12c0ba9680044b4ce98d391a62094047eada0d64860b80166c39f4a6b5640785"
dependencies = [
"serde_core",
"serde_spanned",
Cargo.toml
+2 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "coyote-ai"
version = "0.10.1"
version = "0.10.2"
edition = "2024"
authors = ["Alex Clarke <alex.j.tusa@gmail.com>"]
description = "The batteries-included runtime for LLMs"
@@ -146,6 +146,7 @@ ctor = "1.0.13"
pretty_assertions = "1.4.0"
rmcp = { version = "3.1.2", features = ["server"] }
serial_test = "3"
tokio = { version = "1.34.0", features = ["test-util"] }
[[bin]]
name = "coyote"
assets/agents/adversary/README.md
+74 -9
View File
@@ -1,5 +1,16 @@
# Adversary
> [!IMPORTANT]
> **v2: now a GRAPH agent.** The plan's acceptance criteria are extracted into a list and the graph
> fans out ONE fresh-context verification branch per criterion — partial coverage ("checked 8 of 12")
> is structurally impossible. Each verdict passes a deterministic contract gate (MET requires
> evidence including a proving test; no test = at best PARTIAL), a holistic ADDITIVE-only pass hunts
> absence/scope-drift/substitution/gamed-tests, and the CONFORMS/DIVERGES sentinel is computed by a
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is structurally
> enforced: CONFORMS requires ALL MET + zero holistic complaints + zero pipeline faults +
> zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
asks *"is this code good?"*, `adversary` asks a different, harder question:
@@ -31,6 +42,9 @@ ADVERSARIAL_REVIEW: CONFORMS
Criteria: N/N met (all with tests).
```
CONFORMS is deterministic: ALL MET + zero holistic complaints + zero pipeline faults +
zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
```
ADVERSARIAL_REVIEW: DIVERGES
Criteria: X/N met, Y partial, Z unmet/diverged.
@@ -39,6 +53,12 @@ Complaints:
2. ...
```
Every verdict — CONFORMS or DIVERGES — ends with a `Verification runs:` section: the recorded
PASS/FAIL results (command, exit code, duration, failure tail) of the caller-declared verification
commands, or an explicit marker when none were declared or the run never reached that stage. The
one exception is the verdict script's own crash guard, which emits a bare DIVERGES with a
"verdict computation error" line — and no section — when verdict computation itself dies.
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
@@ -48,18 +68,43 @@ violation) and cites `file:line`. Vague complaints are not emitted.
## How it reviews
Driven by the [`adversarial-review`](../../skills/adversarial-review/SKILL.md) skill:
```mermaid
flowchart TD
A["parse (llm): extract acceptance criteria"] --> B["facts (diff_facts.py): resolve + bound the diff"]
V["verification_commands (declared graph variable — never parsed from the prompt)"] --> R
B --> R["run_checks (script): run the declared verification commands ONCE, record results"]
R --> C["map over criteria: one fresh-context branch each (parallel)"]
C --> D["check_criterion (llm + fs/ast tools)"]
D --> E["crit_gate (script): MET needs change+test evidence"]
E -. "reject-retry once; exhausted = PARTIAL" .-> D
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults + zero red runs; no criteria = DIVERGES"]
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
B -. "fallback" .-> P
R -. "fallback" .-> P
D -. "fallback" .-> CF["criterion_fault (script): UNMET + PIPELINE-FAULT evidence for that ONE criterion"]
CF -. "map join" .-> F
F -. "fallback" .-> G
P --> G
```
1. Map **every** acceptance criterion to specific evidence in the diff → ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. No test proving the behavior ⇒ at best ⚠️ Partial.
The conformance doctrine is baked into the graph's own nodes:
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
3. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL. Independently of the criterion judgments, the verdict script blocks CONFORMS on any red run (exit≠0/timeout/skipped) and on an ENVIRONMENT runner-error marker when commands were declared — the declared commands never ran, so the run is unproven. The commands arrive through the declared `verification_commands` graph variable (see [Declaring verification commands](#declaring-verification-commands)) — `parse` never extracts them from the prompt, so the prompt is no longer an extraction source for the commands.
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT: criterion ` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. That exact marker prefix (the one `criterion_fault` emits) forces the criterion into unmet regardless of the status it carries, so a verdict that echoes it with `MET` can never count toward CONFORMS; `check_criterion` is told never to begin its own evidence with `PIPELINE-FAULT:`.
It is **read-only** — it produces a verdict, never a fix.
## Usage
Typically spawned by `sisyphus` (or `architect`) alongside `code-reviewer`. The spawn prompt IS its
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria:
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria.
Verification commands travel separately, as a declared variable:
```sh
agent__spawn --agent adversary --prompt "
@@ -71,24 +116,44 @@ Run get_diff (or --base main), or: <paste diff>
## PLAN — acceptance criteria to check against
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
"
" --variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
```
Direct invocation for ad-hoc use:
```sh
coyote -a adversary --agent-variable project_dir /path/to/repo \
--agent-variable verification_commands '["cargo test --all"]' \
"Review staged changes against these criteria: <paste criteria>"
```
### Declaring verification commands
`verification_commands` is a declared graph variable (default `'[]'`): a JSON array of shell
commands, passed as a string because agent variables always arrive as strings. `run_checks` runs
exactly those commands with a shell — so the declaration is a **trust boundary**. The prompt is the
wrong side of it: it also carries plan and diff text pasted from the repo under review, and an LLM
`parse` node extracting commands from that prose would let anything in the repo's text run under
the reviewer's shell. Hence the commands are never in the prompt and never in `parse`'s output
schema; `review-gauntlet` forwards its own `verification_commands` variable to the adversary lane
as a structured `inputs:` passthrough, not as prompt text.
- `'[]'` (or omitted) ⇒ the "none declared" marker; execution criteria are at best PARTIAL.
- Anything that is not a JSON array of strings (unparsable JSON, an object, non-string items) is
an invalid declaration: `run_checks` executes **nothing** and records a
`PIPELINE-FAULT: verification_commands declaration invalid — …` marker, which forces DIVERGES:
it fails closed as a PIPELINE-FAULT (complaint #1) — it never degrades to an ENVIRONMENT marker.
- The engine merges only the keys declared under `parse`'s `output_schema.properties` into state,
so an injected instruction that makes the parse LLM emit an extra `verification_commands` key
cannot reach state: the undeclared key is dropped and the declared variable stands.
### Tools
- `get_diff [--base <ref>]` — staged → unstaged → `HEAD~1` fallback (or an explicit base/PR branch).
- `get_changed_files [--base <ref>]` — quick changed-file map.
- Plus read-only `fs_*` and `ast_grep` for ground-truth checks.
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
- The `run_checks` script executes the commands declared via the `verification_commands` variable and records the results — criterion branches cite the record and never run anything themselves.
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
## Related
- [`adversarial-review`](../../skills/adversarial-review/SKILL.md) — the conformance methodology it runs on.
- [`code-reviewer`](../code-reviewer/README.md) — the quality reviewer it runs alongside.
- [`plan-review`](../../skills/plan-review/SKILL.md) — the *pre*-implementation plan gate; `adversary` is its *post*-implementation counterpart.
assets/agents/adversary/config.yaml
-118
View File
@@ -1,118 +0,0 @@
name: adversary
description: Adversarial plan-conformance reviewer - judges whether an implementation matches the task/plan it was supposed to satisfy (not code quality). Returns a blocking CONFORMS/DIVERGES verdict. Complements code-reviewer. Designed to be delegated to by sisyphus.
version: 1.0.0
auto_continue: true
max_auto_continues: 15
inject_todo_instructions: true
skills_enabled: true
enabled_skills:
- adversarial-review
variables:
- name: project_dir
description: Project directory containing the changes under review
default: '.'
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools:
- ast_grep.sh
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
- fs_glob.sh
- fs_ls.sh
- execute_command.sh
instructions: |
You are an adversarial plan-conformance reviewer. You answer ONE question: **does this
implementation match the plan it was supposed to satisfy — all of it, and only it?** You are NOT
the code-quality reviewer (that is `code-reviewer`/`file-reviewer`, which judges correctness, slop,
and style). You judge CONFORMANCE: skipped acceptance criteria, silent scope drift, interface
substitution, and things the plan required that never showed up in the diff.
Your value is independence and suspicion. Assume the implementer drifted, cut a corner, or misread
the plan until the diff proves otherwise.
## Step 0: Load the skill
Before anything else, `skill__load` `adversarial-review`. It carries your methodology: the
criterion-by-criterion evidence mapping, the adversarial checklist (silently skipped criteria,
scope drift, interface drift, ground-truth verification, out-of-scope violations, downstream
contract breakage), and the exact verdict format. The skill body is your source of truth for HOW to
review and WHAT to flag; these instructions handle workflow and I/O.
## Input (the spawn prompt IS your entire context)
You are given:
1. **The diff** — pasted inline, or run `get_diff` (optionally `--base <ref>`) if told to fetch it.
2. **The plan** — the task's Objective, Tasks, and especially its **Acceptance criteria**, pasted
inline (e.g. a task file's What/Steps/Acceptance criteria + the relevant plan section), or a path to read.
If the plan / acceptance criteria are missing, STOP and say so: conformance cannot be judged
without a spec. Do not invent criteria or guess intent.
## Workflow
1. Load `adversarial-review`.
2. Get the diff (inline or via `get_diff`) and identify the changed files.
3. For EACH acceptance criterion: find the specific evidence in the diff that satisfies it and
classify it ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. A criterion with no test proving its
behavior is at best ⚠️ Partial.
4. Ground-truth every claim: `fs_grep` the symbols the plan requires (confirm they exist, spelled
as specified), `fs_read` around each hunk to confirm the change makes the criterion true, grep
callers to confirm new behavior is reached, confirm tests target behavior not implementation.
Use `ast_grep` for structural checks (e.g. "was this function signature actually changed?").
5. Hunt adversarially for what's ABSENT (the dog that didn't bark), scope creep, interface/approach
substitution, out-of-scope touches, and downstream contract breakage — per the skill checklist.
6. Emit the verdict in the skill's exact format.
## Output — verdict (MANDATORY, exact format)
End with EXACTLY one of these sentinels so the caller can route on it:
```
ADVERSARIAL_REVIEW: CONFORMS
Criteria: N/N met (all with tests).
<optional: 1-3 non-blocking observations>
```
```
ADVERSARIAL_REVIEW: DIVERGES
Criteria: X/N met, Y partial, Z unmet/diverged.
Complaints:
1. Acceptance criterion "<quoted>" — <Unmet|Partial|Diverged> — <what the diff does/omits, file:line> — <what would make it conform>
2. Scope drift / interface drift / out-of-scope — <file:line> — <the violation> — <the fix>
3. ...
```
Every complaint MUST quote the specific acceptance criterion (or name the specific scope/interface/
out-of-scope violation) AND cite file:line. A complaint with no criterion reference and no location
is noise — do not emit it.
## Rules
1. **You are read-only.** Never modify files. You produce a verdict; the implementer owns the fix.
2. **Conformance, not quality.** Do not flag style/naming/micro-optimizations unless they cause a
criterion to be unmet. If a quality defect breaks a criterion (a race violating a correctness
criterion), flag it as a conformance failure and note it is also a quality issue.
3. **No test ⇒ not met.** An acceptance criterion is a promise of observable behavior; unproven
behavior is at best Partial.
4. **Absence is a finding.** Review what SHOULD be in the diff per the plan, not only what IS.
5. **Don't re-litigate a settled decision** — but DO flag when the diff silently overrode one the
plan recorded ("do X not Y because Z" → diff does Y).
6. **The plan can be the culprit.** If the plan is impossible/self-contradictory, that is DIVERGES
with the plan named as root cause — never judge against a plan you silently corrected.
7. Be terse and decisive. Three real divergences beat fifteen weak ones. If everything is a nitpick,
it CONFORMS — say so.
## Context
- Project: {{project_dir}}
- CWD: {{__cwd__}}
- Shell: {{__shell__}}
## Available Tools
{{__tools__}}
assets/agents/adversary/graph.yaml
+370
View File
@@ -0,0 +1,370 @@
name: adversary
description: |
Graph-based adversarial plan-conformance reviewer. The plan's acceptance
criteria are extracted into a list and the graph mechanically fans out ONE
fresh-context verification branch per criterion — "checked 8 of 12 and waved
the rest through" becomes structurally impossible. Each criterion verdict
(MET / PARTIAL / UNMET / DIVERGED) passes a deterministic contract gate
(MET requires evidence including a test proving the behavior — no test means
at best PARTIAL). A holistic pass then hunts what per-criterion checks cannot
see: absence (the dog that didn't bark), scope drift, interface substitution,
silently-overridden recorded decisions, out-of-scope touches, and gamed
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
requires every criterion MET, zero holistic complaints, zero pipeline faults,
and zero red verification runs (exit≠0/timeout/skipped, or a runner error
when commands were declared). Missing plan = DIVERGES (fail-closed) naming
the missing spec. Caller-declared verification
commands are executed ONCE by a deterministic stage and recorded — criterion
branches cite the record and never run anything themselves. Any pipeline
fault (parse, diff resolution, verification runner, or holistic failure)
lands a PIPELINE-FAULT marker and fails closed into DIVERGES: degraded is
never passing.
version: '1.0'
global_tools:
- ast_grep.sh
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
variables:
- name: project_dir
description: Project directory containing the changes under review
default: '.'
- name: verification_commands
description: |
JSON array of shell commands the CALLER declares for the deterministic
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
prompt; empty array = none declared.
default: '[]'
settings:
max_loop_iterations: 10
log_state_snapshots: false
# 3600 (run_checks node timeout, worst case) + 5400 (the previous
# whole-graph budget, kept as the envelope for every stage other than
# run_checks: parse/facts before it, criteria map/holistic/verdict after)
# = 9000. A graph timeout kills from outside — no fallback fires and the
# sentinel is lost.
timeout: 9000
initial_state:
project_dir_in: ''
criteria: []
diff_spec: auto
plan_notes: ''
scope_notes: ''
extra_context: ''
changed_files: []
diff_text: ''
diff_note: ''
exec_results: ''
crit_verdicts: []
extra_complaints: []
observations: ''
adv_report: ''
gate_feedback: ''
gate_attempts: 0
parse_failure: ''
holistic_failure: ''
pipeline_faults: []
start: parse
nodes:
parse:
id: parse
type: llm
description: Extract the acceptance criteria and plan facts from the spawn prompt. Extraction only — no judging, no criteria invented.
instructions: |
You extract a plan-conformance review request into structured fields.
EXTRACTION ONLY — never invent criteria, never guess intent.
- `criteria`: every acceptance criterion from the plan/task, one entry each,
with a short id (c1, c2, …) and SELF-CONTAINED text (what observable
behavior is promised, including any test expectations the plan states).
Split compound criteria ("A and B") into separate entries when they are
independently verifiable. When the caller supplied NO plan/criteria at
all, emit an empty array — downstream fails closed.
- `diff_spec`: the git diff basis the caller named — a range, a ref/SHA,
"staged", "worktree", or "auto" when unnamed (a caller phrase like
"get_diff --base X" means diff_spec = "X").
- `project_dir_in`: the directory/worktree path the caller named as
holding the code under review; empty string when the caller named none.
- `plan_notes`: recorded decisions the plan states ("do X not Y because Z"),
verbatim — the holistic pass checks the diff didn't silently override them.
- `scope_notes`: what the plan says NOT to touch / explicit boundaries.
- `extra_context`: anything else conformance-relevant, condensed.
prompt: |
Extract the conformance-review facts from this request:
{{initial_prompt}}
tools: []
output_schema:
type: object
properties:
criteria:
type: array
items:
type: object
properties:
id:
type: string
text:
type: string
required: [id, text]
diff_spec:
type: string
plan_notes:
type: string
scope_notes:
type: string
extra_context:
type: string
project_dir_in:
type: string
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in]
max_iterations: 3
max_attempts: 2
state_updates:
parse_failure: '{{output}}'
fallback: pipeline_fault
next: facts
facts:
id: facts
type: script
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
script: scripts/diff_facts.py
timeout: 120
fallback: pipeline_fault
next: run_checks
run_checks:
id: run_checks
type: script
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker (criterion branches judge it PARTIAL/unproven; the verdict additionally blocks CONFORMS when commands were declared); a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
script: scripts/run_checks.py
timeout: 3600
fallback: pipeline_fault
next: map_criteria
map_criteria:
id: map_criteria
type: map
over: '{{criteria}}'
as: criterion
branch: check_criterion
output_key: crit_verdict
collect_into: crit_verdicts
max_concurrency: 4
next: holistic
check_criterion:
id: check_criterion
type: llm
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree (ENVIRONMENT/SKIPPED runs judge PARTIAL here; the deterministic verdict blocks CONFORMS when declared commands did not run).
instructions: |
You verify ONE acceptance criterion against a code change. Fresh context,
one criterion, full suspicion: assume the implementer drifted, cut a
corner, or misread the plan until the diff proves otherwise.
Classification (exactly one):
- `MET` — the diff satisfies the criterion AND a test proves the promised
behavior. Evidence MUST cite both: the satisfying change (file:line) and
the test targeting the behavior (file:line). No test ⇒ NOT MET — at best
PARTIAL (a criterion is a promise of observable behavior; unproven
behavior is unverified behavior).
- `PARTIAL` — partially satisfied, or satisfied but unproven (missing/weak
test), or you could not establish it either way (say why — doubt lands
here, never in MET).
- `UNMET` — the plan requires it and the diff does not do it. Absence is a
finding: review what SHOULD be there, not only what IS.
- `DIVERGED` — the diff does something ELSE instead (interface substituted,
approach swapped, criterion reinterpreted).
Ground-truth discipline: fs_grep the symbols the criterion names (confirm
they exist spelled as specified), fs_read around the relevant hunks to
confirm the change makes the criterion true, confirm the test targets
BEHAVIOR (not implementation echo), use ast_grep for structural claims.
Watch for gamed evidence: a test weakened/deleted so it passes, hardcoded
values shaped to the criterion's example, TODO/stub presented as done —
those are PARTIAL or DIVERGED with the gaming named in the complaint.
Execution criteria ("tests pass", "lint clean", "build succeeds") are
judged ONLY against the recorded verification runs in your prompt — you
have no execution tools BY DESIGN and must never claim to have run
anything yourself:
- a recorded green run (exit 0) of a command covering the criterion is
the proving evidence — MET, citing that run (the command + exit 0).
- a recorded failing run covering the criterion ⇒ UNMET, citing the
recorded exit code and output tail.
- "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
ENVIRONMENT note in the complaint naming the exact command the caller
should declare via the `verification_commands` variable (structured
input, e.g. "cargo test --all") — commands are never declared in the
prompt.
- a SKIPPED record (the runner's total deadline expired before that
command ran) ⇒ at best PARTIAL for the criteria it covers, same
treatment as an ENVIRONMENT marker: unproven at the criterion level.
The deterministic verdict additionally blocks CONFORMS when declared
commands did not run — that is its job, not yours.
Output format — respond with ONLY this JSON object, no prose around it:
{"id": "<the criterion's id, echoed>",
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
"evidence": "<for MET: the satisfying change file:line AND the proving test
file:line, quoted decisively; otherwise the closest relevant citations or empty>",
"complaint": "<required for non-MET: what the diff does/omits (file:line) and
what would make it conform; empty string for MET>"}
Your response is machine-validated: a malformed response is rejected back
to you exactly once with the reason, then recorded as PARTIAL (unproven).
The id is stamped from the criterion itself — your echo is informative only.
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
whole-file reads, and conclude as soon as the evidence is decisive.
Never begin `evidence` with the literal `PIPELINE-FAULT:` — that prefix is
reserved for the graph's own fault markers.
prompt: |
Repository: {{project_dir}}
Changed files: {{changed_files}}
Criterion to verify (JSON):
{{criterion}}
Plan-recorded decisions (context): {{plan_notes}}
{{extra_context}}
The diff{{diff_note}}:
{{diff_text}}
Verification runs (recorded by the deterministic run_checks stage):
{{exec_results}}
{{gate_feedback}}
tools:
- fs_read
- fs_cat
- fs_grep
- ast_grep
max_iterations: 30
max_attempts: 2
state_updates:
crit_verdict: '{{output}}'
fallback: criterion_fault
next: crit_gate
crit_gate:
id: crit_gate
type: script
description: Deterministic contract gate per criterion verdict — status enum, MET requires evidence, non-MET requires a complaint; reject-and-retry once, then record PARTIAL (unproven). Never sinks the map.
script: scripts/crit_gate.py
timeout: 30
criterion_fault:
id: criterion_fault
type: script
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met.
script: scripts/criterion_fault.py
timeout: 30
holistic:
id: holistic
type: llm
description: Cross-criterion adversarial pass — hunts what per-criterion checks cannot see. ADDITIVE-only; cannot flip criterion verdicts.
instructions: |
You run the holistic adversarial pass over a whole change AFTER each
acceptance criterion was independently verified. You hunt what
per-criterion checks cannot see. You are ADDITIVE-ONLY: you may add
complaints; you may NOT flip, soften, or re-judge any criterion verdict.
The checklist (each hit becomes one complaint with kind, file:line
location, the violation, and the fix):
- **Absence** — something the PLAN requires that maps to no criterion and
never showed up in the diff (the dog that didn't bark).
- **Scope drift / out-of-scope** — the diff touches what the plan's scope
notes say not to touch, or quietly does extra unplanned work.
- **Interface/approach substitution** — the plan named an interface,
library, or approach; the diff substituted another without the plan
recording that decision.
- **Silently overridden recorded decision** — plan says "do X not Y
because Z"; diff does Y.
- **Downstream contract breakage** — the diff changes a contract the plan
promised to preserve.
- **Gamed tests** — tests weakened, deleted, or shaped (hardcoded
expectations, skipped assertions) so the criteria APPEAR met.
- **The plan itself is the culprit** — impossible or self-contradictory
requirements: that is a complaint naming the PLAN as root cause; never
silently judge against a corrected version of it.
Be terse and decisive: three real complaints beat fifteen weak ones. A
complaint with no location and no criterion/scope reference is noise — do
not emit it. When everything is a nitpick, emit zero complaints and put
1-3 non-blocking notes in `observations`.
prompt: |
Repository: {{project_dir}}
Changed files: {{changed_files}}
Acceptance criteria: {{criteria}}
Per-criterion verdicts (already final): {{crit_verdicts}}
Plan-recorded decisions: {{plan_notes}}
Scope boundaries: {{scope_notes}}
{{extra_context}}
The diff{{diff_note}}:
{{diff_text}}
tools:
- fs_read
- fs_cat
- fs_grep
- ast_grep
output_schema:
type: object
properties:
extra_complaints:
type: array
items:
type: object
properties:
kind:
type: string
location:
type: string
violation:
type: string
fix:
type: string
required: [kind, location, violation, fix]
observations:
type: string
required: [extra_complaints, observations]
max_iterations: 20
max_attempts: 2
state_updates:
holistic_failure: '{{output}}'
fallback: verdict
next: verdict
pipeline_fault:
id: pipeline_fault
type: script
description: Fail-closed fault marker (reached only via fallback routes from parse/facts/run_checks) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
script: scripts/pipeline_fault.py
timeout: 30
next: verdict
verdict:
id: verdict
type: script
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, zero pipeline faults, and zero recorded red verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error marker when commands were declared); no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
script: scripts/verdict.py
timeout: 30
next: done
done:
id: done
type: end
output: |
{{adv_report}}
assets/agents/adversary/scripts/crit_gate.py
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""Contract gate for the adversary's per-criterion verify chain.
Validates each criterion verdict against the contract:
- a single JSON object with status in {MET, PARTIAL, UNMET, DIVERGED}
- MET requires non-empty evidence (the change AND the proving test)
- non-MET requires a non-empty complaint
The authoritative id is stamped from the criterion item — the model's echo is
never trusted. Malformed output is rejected back to check_criterion exactly
once with the reason; a second failure records PARTIAL (unproven) so a
malformed item can never sink the map or masquerade as MET.
"""
import json
import os
import re
STATUSES = {"MET", "PARTIAL", "UNMET", "DIVERGED"}
MAX_RETRIES = 1
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def criterion_fields(state):
c = state.get("criterion")
if isinstance(c, dict):
cid = c.get("id")
text = c.get("text")
return (
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
text.strip() if isinstance(text, str) else "",
)
return "unknown", ""
def extract_json(text):
if not isinstance(text, str) or not text.strip():
raise ValueError("criterion-check output is empty or not text")
text = re.sub(r"```[a-zA-Z]*", "", text).replace("```", "")
start, end = text.find("{"), text.rfind("}")
if start == -1 or end <= start:
raise ValueError("no JSON object found in criterion-check output")
return json.loads(text[start : end + 1])
def validate(obj):
if not isinstance(obj, dict):
raise ValueError("criterion-check output is not a JSON object")
status = obj.get("status")
if status not in STATUSES:
raise ValueError(f"status must be one of {sorted(STATUSES)}, got {status!r}")
evidence = obj.get("evidence") if isinstance(obj.get("evidence"), str) else ""
complaint = obj.get("complaint") if isinstance(obj.get("complaint"), str) else ""
if status == "MET" and not evidence.strip():
raise ValueError(
"MET requires non-empty evidence citing the satisfying change AND the "
"test proving the behavior (file:line)"
)
if status != "MET" and not complaint.strip():
raise ValueError(
f"{status} requires a non-empty complaint (what the diff does/omits at "
"file:line, and what would make it conform)"
)
return status, evidence, complaint
def main():
state = load_state()
cid, ctext = criterion_fields(state)
attempts = state.get("gate_attempts")
if not isinstance(attempts, int):
attempts = 0
try:
status, evidence, complaint = validate(extract_json(state.get("crit_verdict")))
print(
json.dumps(
{
"crit_verdict": {
"id": cid,
"text": ctext,
"status": status,
"evidence": evidence,
"complaint": complaint,
}
}
)
)
return
except Exception as e: # noqa: BLE001
reason = str(e)
if attempts < MAX_RETRIES:
print(
json.dumps(
{
"_next": "check_criterion",
"gate_attempts": attempts + 1,
"gate_feedback": (
"PREVIOUS RESPONSE REJECTED by the verdict gate: " + reason +
". Respond with ONLY the JSON object described in your instructions."
),
}
)
)
return
print(
json.dumps(
{
"crit_verdict": {
"id": cid,
"text": ctext,
"status": "PARTIAL",
"evidence": "",
"complaint": (
"criterion check output failed machine validation after retry: "
+ reason
+ " — treat this criterion as unproven"
),
}
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the gate must never crash the chain
print(
json.dumps(
{
"crit_verdict": {
"id": "unknown",
"text": "",
"status": "PARTIAL",
"evidence": "",
"complaint": f"criterion gate error: {e} — treat as unproven",
}
}
)
)
assets/agents/adversary/scripts/criterion_fault.py
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""Fail-closed per-criterion fault marker for the adversary graph.
Reached ONLY via check_criterion's `fallback:` route, after the engine has
written its "LLM node …failed: <chain>" text into `crit_verdict` through
state_updates. Converts that text into a schema-shaped UNMET verdict whose
evidence carries a "PIPELINE-FAULT:" marker, so one criterion check that dies
(max_iterations exhausted, API failure) is recorded as DIED rather than
judged, the other criteria still get verdicts, and the run is DIVERGES. The
map collects it like any other verdict — the branch never sinks the map. The
engine applies state_updates before routing to the fallback (src/graph/llm.rs:
`apply_state_updates_with_output`, then `outcome_from`), so the "no failure
text captured" branch is belt-and-braces rather than an expected path.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def criterion_fields(state):
c = state.get("criterion")
if isinstance(c, dict):
cid = c.get("id")
text = c.get("text")
return (
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
text.strip() if isinstance(text, str) else "",
)
return "unknown", ""
def failure_detail(state):
captured = state.get("crit_verdict")
detail = captured.strip() if isinstance(captured, str) else ""
if not detail.startswith("LLM node"):
return "no failure text captured"
detail = detail.replace("\n", " ")
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
return detail
def main():
state = load_state()
cid, ctext = criterion_fields(state)
detail = failure_detail(state)
print(
json.dumps(
{
"crit_verdict": {
"id": cid,
"text": ctext,
"status": "UNMET",
"evidence": f"PIPELINE-FAULT: criterion check failed — {detail}",
"complaint": (
f"criterion check DIED (pipeline fault) — {detail}; the "
"criterion was NOT verified and is treated as unmet "
"(fail-closed)"
),
}
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"crit_verdict": {
"id": "unknown",
"text": "",
"status": "UNMET",
"evidence": f"PIPELINE-FAULT: criterion fault-marker script error: {e}",
"complaint": (
f"criterion check DIED (pipeline fault) — fault-marker script "
f"error: {e}; the criterion was NOT verified and is treated as "
"unmet (fail-closed)"
),
}
}
)
)
assets/agents/adversary/scripts/diff_facts.py
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Deterministic diff resolution for the adversary graph.
Resolves the diff spec (get_diff semantics: staged → unstaged → HEAD~1 for
"auto"), lists changed files, and captures the bounded diff text every
criterion branch verifies against. Failures degrade into diff_note — the
review proceeds fail-closed rather than crashing.
"""
import json
import os
import subprocess
MAX_DIFF_CHARS = 60000
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
state = load_state()
proj = os.path.expanduser(
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
)
spec = (state.get("diff_spec") or "auto").strip()
def git(*args):
r = subprocess.run(["git", "-C", proj, *args], capture_output=True, text=True, timeout=90)
if r.returncode != 0:
raise RuntimeError((r.stderr or r.stdout).strip()[:300])
return r.stdout
out = {"diff_note": "", "project_dir": proj}
try:
if spec in ("auto", ""):
text = git("diff", "--cached")
if not text.strip():
text = git("diff")
if not text.strip():
text = git("diff", "HEAD~1")
elif spec == "staged":
text = git("diff", "--cached")
elif spec == "worktree":
text = git("diff", "HEAD")
else:
text = git("diff", spec)
files = sorted({
line[6:].strip()
for line in text.splitlines()
if line.startswith("+++ b/")
})
out["changed_files"] = files[:300]
if len(text) > MAX_DIFF_CHARS:
out["diff_note"] = (
f" (truncated to {MAX_DIFF_CHARS} chars of {len(text)} — use the fs tools "
"to inspect files beyond the cut)"
)
text = text[:MAX_DIFF_CHARS]
out["diff_text"] = text if text.strip() else "(empty diff — nothing changed)"
except Exception as e: # noqa: BLE001
out["changed_files"] = []
out["diff_text"] = "(diff unavailable)"
out["diff_note"] = f" (diff resolution failed for spec {spec!r}: {e})"
print(json.dumps(out))
assets/agents/adversary/scripts/pipeline_fault.py
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""Fail-closed fault marker for the adversary graph.
Reached ONLY via `fallback:` routes (parse, facts, run_checks). Appends a
machine-readable "PIPELINE-FAULT:" marker to `pipeline_faults` and continues
to `verdict`, which turns any recorded fault into DIVERGES — the graph never
dies without its sentinel, and a degraded run can never read as CONFORMS.
Stage attribution — grounded in engine behavior: a SCRIPT node's fallback
route writes NOTHING into state (src/graph/executor.rs script arm routes to
the fallback without recording the error), so the two script stages are
discriminated by which stage outputs are present:
- parse (llm): its `state_updates` wrote an "LLM node …failed: <chain>"
string into `parse_failure` — attribute to parse.
- run_checks vs facts (scripts): diff_facts.py unconditionally writes a
NONEMPTY `diff_text` (even its internal-failure path writes
"(diff unavailable)"), while the graph's initial value is ''. A nonempty
`diff_text` proves facts completed, so the fault is the verification
runner (run_checks killed at the node level, e.g. by the node timeout);
an empty `diff_text` means facts itself died.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
parse_failure = state.get("parse_failure")
detail = parse_failure.strip() if isinstance(parse_failure, str) else ""
diff_text = state.get("diff_text")
facts_completed = isinstance(diff_text, str) and bool(diff_text.strip())
if detail.startswith("LLM node"):
detail = detail.replace("\n", " ")
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
faults.append(f"PIPELINE-FAULT: parse failed — cannot review: {detail}")
elif facts_completed:
faults.append(
"PIPELINE-FAULT: verification runner killed — the run_checks stage "
"died at the node level before recording exec_results; execution "
"criteria are unproven"
)
else:
faults.append(
"PIPELINE-FAULT: diff resolution failed — changed files and diff text "
"could not be resolved; criterion verification never ran"
)
print(json.dumps({"pipeline_faults": faults}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"pipeline_faults": [
f"PIPELINE-FAULT: fault-marker script error: {e} — treat the "
"run as degraded"
]
}
)
)
assets/agents/adversary/scripts/run_checks.py
+207
View File
@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""Deterministic execution-evidence recorder for the adversary graph.
Runs ONLY the caller-declared verification commands (verification_commands)
in the project dir and records the outcomes as `exec_results` — a list of
{cmd, exit, duration_s, tail} records. Criterion branches cite this record;
they never execute anything themselves.
NO auto-detection: undeclared commands mean a "none declared" marker
(fail-visible, not fail-guessed). Any runner error degrades into an
ENVIRONMENT marker — this script never fails the node.
Trust boundary: `verification_commands` is a DECLARED graph variable (or the
review-gauntlet's structured `inputs:` passthrough), never a field an LLM
extracts from the spawn prompt — that prose also carries plan/diff text pasted
from the repo under review, and whatever is declared here runs with
shell=True. Variables land in state as strings, so a JSON-encoded list is
accepted alongside a real list. A declaration that is not a JSON array of
strings is a caller contract violation, not a runner hiccup: it executes
NOTHING and fails closed as a PIPELINE-FAULT (complaint #1) — it never
degrades to an ENVIRONMENT marker. The engine merges only the keys an
llm node's `output_schema.properties` declares, so a parse LLM coaxed into
emitting an extra `verification_commands` key cannot overwrite the declared
value: the undeclared key is dropped before it reaches state.
Budget: commands run sequentially, so TOTAL runtime is bounded by a deadline
(TOTAL_DEADLINE_SECS = 3300s, the run_checks node's 3600s timeout minus
margin). Without it, a handful of hanging commands (4 x 900s) would blow the
NODE timeout and the executor would kill this script from OUTSIDE — the
in-script degradation would never fire. Each command gets
min(PER_COMMAND_TIMEOUT_SECS, remaining budget); commands the deadline
leaves no budget for are recorded as {"cmd", "skipped": "deadline"} entries,
so the in-script handling stays authoritative. The env var
ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam).
Process hygiene: commands run through the shell. On POSIX each command gets
its own session (start_new_session) and a timeout SIGKILLs the whole process
group, so grandchildren of a hung command cannot linger. The post-kill
reap/drain is itself bounded by DRAIN_TIMEOUT_SECS: a grandchild that escaped
the session (its own start_new_session) but still holds the stdout pipe
would otherwise block the drain until it exits. On non-POSIX platforms only
the direct shell child is killed — grandchildren of a timed-out command may
survive (documented limitation).
Verification commands run with GRAPH_STATE* (GRAPH_STATE, GRAPH_STATE_FILE)
scrubbed from their environment: the reviewed repo's own tests may exec these very
scripts with their own GRAPH_STATE, and load_state() prefers
GRAPH_STATE_FILE, so an inherited live state file would silently win.
"""
import json
import os
import signal
import subprocess
import time
PER_COMMAND_TIMEOUT_SECS = 900
TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
DRAIN_TIMEOUT_SECS = 10
TAIL_LINES = 50
MAX_TAIL_CHARS = 8000
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
class InvalidDeclaration(ValueError):
pass
def declared_commands(declared):
"""Normalize the verification_commands declaration to a list of commands.
Accepts a JSON list or a JSON-encoded string of one; missing/None means
none declared. Anything else raises InvalidDeclaration.
"""
if declared is None:
return []
if isinstance(declared, str):
if not declared.strip():
raise InvalidDeclaration("empty string (declare '[]' for none)")
try:
declared = json.loads(declared)
except json.JSONDecodeError as e:
raise InvalidDeclaration(f"not valid JSON: {e}") from e
if not isinstance(declared, list):
raise InvalidDeclaration(
f"expected a JSON array of strings, got {type(declared).__name__}"
)
bad = [c for c in declared if not isinstance(c, str)]
if bad:
raise InvalidDeclaration(f"non-string item(s): {json.dumps(bad)}")
return [c.strip() for c in declared if c.strip()]
def tail_of(text):
tail = "\n".join(text.splitlines()[-TAIL_LINES:])
if len(tail) > MAX_TAIL_CHARS:
tail = "…" + tail[-MAX_TAIL_CHARS:]
return tail
def run_one(cmd, proj, budget):
"""Run one command, hard-bounded by `budget` seconds; return its record."""
start = time.monotonic()
popen_kwargs = {"start_new_session": True} if os.name == "posix" else {}
env = {k: v for k, v in os.environ.items() if not k.startswith("GRAPH_STATE")}
p = subprocess.Popen(
cmd,
shell=True,
cwd=proj,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
env=env,
**popen_kwargs,
)
try:
out_s, err_s = p.communicate(timeout=budget)
except subprocess.TimeoutExpired:
if os.name == "posix":
try:
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
except (ProcessLookupError, PermissionError):
pass
else:
p.kill()
try:
p.communicate(timeout=DRAIN_TIMEOUT_SECS)
except subprocess.TimeoutExpired:
pass
return {
"cmd": cmd,
"exit": -1,
"duration_s": round(time.monotonic() - start, 1),
"tail": f"TIMEOUT: no exit within {budget:.0f}s — recorded as a failing run",
}
combined = (out_s or "") + (("\n" + err_s) if err_s else "")
return {
"cmd": cmd,
"exit": p.returncode,
"duration_s": round(time.monotonic() - start, 1),
"tail": tail_of(combined),
}
def main():
state = load_state()
try:
cmds = declared_commands(state.get("verification_commands"))
except InvalidDeclaration as e:
msg = f"PIPELINE-FAULT: verification_commands declaration invalid — {e}"
prior = state.get("pipeline_faults")
faults = [f for f in prior if isinstance(f, str)] if isinstance(prior, list) else []
faults.append(msg)
print(json.dumps({"exec_results": msg, "pipeline_faults": faults}))
return
if not cmds:
print(
json.dumps(
{
"exec_results": (
"none declared — the caller supplied no verification_commands; "
"execution criteria (build/test/lint) cannot be independently "
"confirmed. Declare the exact commands via the "
"verification_commands variable to get recorded runs."
)
}
)
)
return
proj = os.path.expanduser(
(state.get("project_dir") or "").strip() or "."
)
deadline = time.monotonic() + TOTAL_DEADLINE_SECS
results = []
for cmd in cmds:
remaining = deadline - time.monotonic()
if remaining <= 0:
results.append({"cmd": cmd, "skipped": "deadline"})
continue
results.append(run_one(cmd, proj, min(PER_COMMAND_TIMEOUT_SECS, remaining)))
print(json.dumps({"exec_results": results}))
try:
main()
except Exception as e: # noqa: BLE001 — never fail the node; degrade visibly
print(
json.dumps(
{
"exec_results": (
f"ENVIRONMENT: verification runner error: {e} — the declared "
"commands could not be executed; execution criteria are unproven "
"(PARTIAL at the criterion level; the verdict blocks CONFORMS when "
"commands were declared)"
)
}
)
)
assets/agents/adversary/scripts/verdict.py
+293
View File
@@ -0,0 +1,293 @@
#!/usr/bin/env python3
"""Deterministic conformance verdict for the adversary graph.
CONFORMS requires: at least one criterion, every criterion MET, zero
holistic complaints, zero pipeline faults, and zero recorded red
verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error
marker when commands were declared — the declared commands never ran, so the
run is unproven). Everything else — including a missing plan or a degraded
(faulted) pipeline — is DIVERGES (fail-closed).
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
llm-node failure captured in `holistic_failure`) becomes complaint #1;
per-criterion results are still reported. A criterion whose check DIED
(criterion_fault's UNMET verdict, evidence prefixed DIED_EVIDENCE_PREFIX) is
rendered as died rather than judged and counted as unmet regardless of the
status it carries. Assembles the exact sentinel format the callers route on.
Never crashes into a silent verdict.
"""
import json
import os
MAX_FAULT_DETAIL_CHARS = 300
MAX_MARKER_CHARS = 200
DIED_MARKER = "criterion check DIED (pipeline fault)"
DIED_EVIDENCE_PREFIX = "PIPELINE-FAULT: criterion "
def died(v):
evidence = v.get("evidence")
return isinstance(evidence, str) and evidence.startswith(DIED_EVIDENCE_PREFIX)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def collect_pipeline_faults(state):
"""Recorded fault markers, plus a holistic llm-node failure if one occurred.
`holistic_failure` holds the "LLM node …failed: <chain>" string when the
holistic pass fell back to verdict; on a successful pass it holds the
rendered schema output (never that prefix).
"""
faults = [
f.strip()
for f in (state.get("pipeline_faults") or [])
if isinstance(f, str) and f.strip()
]
hf = state.get("holistic_failure")
if isinstance(hf, str) and hf.strip().startswith("LLM node"):
detail = hf.strip().replace("\n", " ")
if len(detail) > MAX_FAULT_DETAIL_CHARS:
detail = detail[: MAX_FAULT_DETAIL_CHARS - 1] + "…"
faults.append(
"PIPELINE-FAULT: holistic pass failed — criterion verdicts stand "
f"but cross-cutting hunt did not run ({detail})"
)
return faults
def append_criterion_complaints(lines, verdicts, extra, start):
"""Numbered complaints for non-MET verdicts and holistic extras."""
i = start
for v in verdicts:
if v.get("status") == "MET" and not died(v):
continue
i += 1
text = (v.get("text") or "").strip().replace("\n", " ")
if len(text) > 140:
text = text[:137] + "…"
complaint = (v.get("complaint") or "").strip()
if died(v):
if not complaint.startswith(DIED_MARKER):
complaint = f"{DIED_MARKER} — {complaint}" if complaint else DIED_MARKER
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
else:
lines.append(
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
f"{complaint}"
)
for c in extra:
i += 1
lines.append(
f"{i}. {c.get('kind', 'violation')} — {c.get('location', '?')} — "
f"{c.get('violation', '')} — {c.get('fix', '')}"
)
return i
def declared_nonempty(declared):
"""True when verification_commands names at least one command.
Accepts a list or a JSON-encoded string of one. None, unparseable, or
empty declarations count as not declared — run_checks already records
a PIPELINE-FAULT for the invalid ones.
"""
if isinstance(declared, str):
try:
declared = json.loads(declared)
except json.JSONDecodeError:
return False
if not isinstance(declared, list):
return False
return any(isinstance(c, str) and c.strip() for c in declared)
def exec_run_complaints(exec_results, declared):
"""One complaint per red verification run recorded by run_checks.py.
A red run is a record that was skipped (deadline) or exited nonzero
(including the -1 TIMEOUT sentinel). An "ENVIRONMENT: …" runner-error
marker with a non-empty declaration is one complaint: the declared
commands never ran. Other string markers ("none declared",
"PIPELINE-FAULT…") are not runs and yield nothing.
"""
if isinstance(exec_results, str):
marker = exec_results.strip().replace("\n", " ")
if not marker.startswith("ENVIRONMENT:") or not declared_nonempty(declared):
return []
if len(marker) > MAX_MARKER_CHARS:
marker = marker[: MAX_MARKER_CHARS - 1] + "…"
return [
f"Verification runner error — the declared command(s) never ran ({marker}); "
"unproven runs block CONFORMS"
]
if not isinstance(exec_results, list):
return []
out = []
for r in exec_results:
if not isinstance(r, dict):
continue
cmd = r.get("cmd", "?")
if r.get("skipped"):
out.append(f"Verification run `{cmd}` — never ran: deadline expired (unproven)")
continue
code = r.get("exit")
if code == 0:
continue
outcome = "TIMEOUT (exit -1)" if code == -1 else f"exit {code}"
out.append(
f"Verification run `{cmd}` — {outcome} (recorded FAIL; a red declared "
"command blocks CONFORMS regardless of criterion judgments)"
)
return out
def render_exec_results(exec_results):
"""Lines for the "Verification runs:" report section.
exec_results is a list of {cmd, exit, duration_s, tail} records (or
{cmd, skipped} for commands the runner's total deadline left no budget
for) from run_checks.py, or a string marker ("none declared —…" /
"ENVIRONMENT: …"), or empty when the run never reached the stage.
"""
if isinstance(exec_results, list):
out = []
for r in exec_results:
if not isinstance(r, dict):
continue
if r.get("skipped"):
out.append(
f"- [SKIPPED] `{r.get('cmd', '?')}` — never ran: the runner's "
"total deadline expired first; covered criteria are unproven"
)
continue
status = "PASS" if r.get("exit") == 0 else "FAIL"
out.append(
f"- [{status}] `{r.get('cmd', '?')}` — exit {r.get('exit', '?')}, "
f"{r.get('duration_s', '?')}s"
)
if status == "FAIL":
tail = (r.get("tail") or "").strip()
out.extend(" " + t for t in tail.splitlines()[-15:])
return out or ["- (no verification commands were run)"]
if isinstance(exec_results, str) and exec_results.strip():
return [f"- {exec_results.strip()}"]
return ["- none recorded (the run did not reach the verification stage)"]
def main():
state = load_state()
verdicts = [v for v in (state.get("crit_verdicts") or []) if isinstance(v, dict)]
extra = [c for c in (state.get("extra_complaints") or []) if isinstance(c, dict)]
observations = (state.get("observations") or "").strip()
faults = collect_pipeline_faults(state)
reds = exec_run_complaints(
state.get("exec_results"), state.get("verification_commands")
)
n = len(verdicts)
met = [v for v in verdicts if v.get("status") == "MET" and not died(v)]
partial = [v for v in verdicts if v.get("status") == "PARTIAL" and not died(v)]
bad = [
v for v in verdicts if died(v) or v.get("status") in ("UNMET", "DIVERGED")
]
lines = []
if faults:
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
if n:
lines.append(
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, "
f"{len(bad)} unmet/diverged — degraded run: pipeline fault recorded "
"(fail-closed)."
)
else:
lines.append(
"Criteria: none verified — degraded run: pipeline fault recorded "
"(fail-closed)."
)
lines.append("Complaints:")
i = 0
for f in faults:
i += 1
lines.append(f"{i}. {f}")
i = append_criterion_complaints(lines, verdicts, extra, i)
for c in reds:
i += 1
lines.append(f"{i}. {c}")
if observations:
lines.append("")
lines.append("Non-blocking observations:")
lines.append(observations)
elif n == 0:
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
lines.append("Criteria: none provided.")
lines.append("Complaints:")
lines.append(
"1. No acceptance criteria were provided or extractable — conformance cannot "
"be judged without a spec (fail-closed). Supply the plan's Objective/Tasks/"
"Acceptance criteria and re-run."
)
for i, c in enumerate(reds, start=2):
lines.append(f"{i}. {c}")
elif not partial and not bad and not extra and not reds:
lines.append("ADVERSARIAL_REVIEW: CONFORMS")
lines.append(f"Criteria: {n}/{n} met (all with tests).")
if observations:
lines.append("")
lines.append("Non-blocking observations:")
lines.append(observations)
else:
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
header = f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged"
notes = []
if died_count := sum(1 for v in verdicts if died(v)):
notes.append(f"degraded run: {died_count} criterion check(s) died (fail-closed)")
if reds:
notes.append(f"{len(reds)} red verification run(s) (fail-closed)")
lines.append(header + "".join(f" — {note}" for note in notes) + ".")
lines.append("Complaints:")
i = append_criterion_complaints(lines, verdicts, extra, 0)
for c in reds:
i += 1
lines.append(f"{i}. {c}")
if observations:
lines.append("")
lines.append("Non-blocking observations:")
lines.append(observations)
# Per-criterion evidence appendix (MET entries, for auditability).
if met:
lines.append("")
lines.append("Met criteria (evidence):")
for v in met:
text = (v.get("text") or "").strip().replace("\n", " ")
if len(text) > 100:
text = text[:97] + "…"
lines.append(f'- "{text}" — {(v.get("evidence") or "").strip()}')
lines.append("")
lines.append("Verification runs:")
lines.extend(render_exec_results(state.get("exec_results")))
print(json.dumps({"adv_report": "\n".join(lines)}))
try:
main()
except Exception as e: # noqa: BLE001 — never a silent verdict
print(
json.dumps(
{
"adv_report": (
"ADVERSARIAL_REVIEW: DIVERGES\n"
f"Criteria: verdict computation error: {e} — treat as failed, not as conforming."
)
}
)
)
assets/agents/adversary/tools.sh
-78
View File
@@ -1,78 +0,0 @@
#!/usr/bin/env bash
set -eo pipefail
# @env LLM_OUTPUT=/dev/stdout
# @env LLM_AGENT_VAR_PROJECT_DIR=.
# @describe Adversarial plan-conformance reviewer tools
_project_dir() {
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
}
# @cmd Get the git diff to review for plan conformance. Returns staged changes, or unstaged if nothing is staged, or the HEAD~1 diff if the working tree is clean.
# @option --base Optional base ref to diff against (e.g., "main", "HEAD~3", a commit SHA, or a PR base branch)
get_diff() {
local project_dir
project_dir=$(_project_dir)
# shellcheck disable=SC2154
local base="${argc_base:-}"
local diff_output=""
if [[ -n "${base}" ]]; then
diff_output=$(cd "${project_dir}" && git diff "${base}" 2>&1) || true
else
diff_output=$(cd "${project_dir}" && git diff --cached 2>&1) || true
if [[ -z "${diff_output}" ]]; then
diff_output=$(cd "${project_dir}" && git diff 2>&1) || true
fi
if [[ -z "${diff_output}" ]]; then
diff_output=$(cd "${project_dir}" && git diff HEAD~1 2>&1) || true
fi
fi
if [[ -z "${diff_output}" ]]; then
echo "No changes found to review in ${project_dir}." >> "$LLM_OUTPUT"
return 0
fi
local file_count
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
{
echo "Diff contains changes to ${file_count} file(s):"
echo ""
echo "${diff_output}"
} >> "$LLM_OUTPUT"
}
# @cmd Get the list of changed files with stats (a quick map of what to check against the plan).
# @option --base Optional base ref to diff against
get_changed_files() {
local project_dir
project_dir=$(_project_dir)
local base="${argc_base:-}"
local stat_output=""
if [[ -n "${base}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat "${base}" 2>&1) || true
else
stat_output=$(cd "${project_dir}" && git diff --cached --stat 2>&1) || true
if [[ -z "${stat_output}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat 2>&1) || true
fi
if [[ -z "${stat_output}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat HEAD~1 2>&1) || true
fi
fi
if [[ -z "${stat_output}" ]]; then
echo "No changes found in ${project_dir}." >> "$LLM_OUTPUT"
return 0
fi
{
echo "Changed files:"
echo ""
echo "${stat_output}"
} >> "$LLM_OUTPUT"
}
assets/agents/architect/README.md
+6 -1
View File
@@ -137,6 +137,7 @@ coyote -a architect \
| `design_doc` | *(empty)* | Path to the design doc; asked for if unset. |
| `base_branch` | `main` | Branch the run branch forks from and the PR targets. |
| `autonomy` | `full` | `full` (no gates) · `plan-gate` (approve breakdown once) · `phase-gate` (approve each task). |
| `escalation_policy` | `interactive` | `interactive` (pause and ask) · `autonomous` (self-resolve decisions covered by a standing ruling in `~/.config/coyote/share/standing-rulings.md` or a clearly recommended option; everything self-resolved lands in a “Decisions taken autonomously” digest in the Phase F report and PR body; secrets/money/destructive actions and novel conflicts still escalate). |
| `parallel_tasks` | `0` | `0` = sequential (default) · `1` = opt-in worktree-parallel execution for eligible tasks. |
| `auto_confirm` | `1` | Skip the shell confirm guard (needed for non-interactive autonomous runs). |
@@ -159,6 +160,9 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
rebase and re-verify.
- Optionally, the `agent__task_*` queue may mirror the on-disk `blocked_by` graph so completing a
task auto-dispatches its unblocked dependents. The queue is in-memory convenience only — disk task
files remain the durable source of truth.
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
CI-check watch) is unchanged in both modes.
@@ -172,6 +176,8 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
| [`oracle`](../oracle/README.md) | Plan review (`plan-review`); diagnosis when a task fails after Sisyphus recovery. |
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
| [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. |
| [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; a `GAUNTLET_REVIEW_INCOMPLETE:` line is an infrastructure fault escalated to the user (retry / accept NEEDS-HUMAN / abort), never a divergence; direct adversary/probe spawns remain the fallback. |
## Related skills
@@ -179,4 +185,3 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
- [`task-tracking`](../../skills/task-tracking/SKILL.md) — the task-file schema, lifecycle, and consistency checks.
- [`plan-gatekeeping`](../../skills/plan-gatekeeping/SKILL.md) — the gatekeeper's self-containedness manifest.
- [`plan-authoring`](../../skills/plan-authoring/SKILL.md) / [`plan-review`](../../skills/plan-review/SKILL.md) — plan schema + oracle's executability review.
- [`adversarial-review`](../../skills/adversarial-review/SKILL.md) — the adversary's conformance methodology.
assets/agents/architect/config.yaml
+116 -2
View File
@@ -6,7 +6,7 @@ description: |
plan-conformance check (plus a black-box usage-pattern probe for consumer-facing surface), and
finishes with ONE draft PR (CI checks watched to green) plus tracked follow-up tasks. Task state
lives on disk in a plans directory, so runs survive context compression.
version: 2.2.0
version: 2.5.0
agent_session: temp
auto_continue: true
max_auto_continues: 100
@@ -20,7 +20,9 @@ spawnable_agents:
- librarian
- adversary
- probe
- review-gauntlet
- gatekeeper
- whetstone
max_concurrent_agents: 10
max_agent_depth: 10
inject_spawn_instructions: true
@@ -52,6 +54,9 @@ variables:
- name: autonomy
description: 'How autonomous the run is: full (no gates), plan-gate (approve breakdown once, then autonomous), phase-gate (approve each task)'
default: full
- name: escalation_policy
description: "How escalation-worthy decisions (yours and relayed child escalations) are handled: 'interactive' pauses and asks the user; 'autonomous' self-resolves decisions covered by a standing ruling or a clearly recommended option, reports them in the Phase F digest, and only pings on hard triggers or novel conflicts"
default: 'interactive'
- name: auto_confirm
description: Auto-confirm command execution (1 = skip the shell guard_operation TTY prompt, needed for non-interactive autonomous runs)
default: '1'
@@ -113,6 +118,37 @@ instructions: |
AFTER its merge landed and integration tests passed is routine documented cleanup, not a
destructive action.
## Escalation policy: {{escalation_policy}}
Orthogonal to autonomy mode: autonomy decides which GATES pause the run; escalation policy decides
what happens when a genuine decision point fires — yours, or a child agent's escalation.
- **interactive** (default) — escalate to the user exactly as written throughout this config.
- **autonomous** — the user pre-delegated decision-making. Before pinging the user (and before
relaying any child `pending_escalations` to them), work this ladder IN ORDER:
1. **Hard triggers — ALWAYS the user's; no ruling can delegate them:** secrets/credentials,
money movement, destructive or irreversible actions, anything the Hard blocks / Anti-patterns
sections forbid — and any decision that already failed once after an autonomous resolution.
2. **Standing rulings** — read `$HOME/.config/coyote/share/standing-rulings.md` (skip silently
if absent). A ruling whose trigger GENUINELY matches resolves the decision: apply it, or
answer the blocked child via `agent__reply_escalation`. Match on substance, not resemblance;
unsure means no match.
3. **Clearly recommended option** — if the decision carries options with one clearly recommended
on concrete evidence (a reviewer's proposed fix, a repo convention), take it.
4. **Otherwise — escalate** as interactive mode would. Autonomy never converts "I don't know"
into a guess.
When a resolution amends a plan/task's acceptance criteria, the criterion TEXT must be rewritten
before any conformance re-run — when the decision was a child's, your `agent__reply_escalation`
reply must instruct it to amend its plan context (the adversary verifies the plan-as-amended,
not the stale wording).
Every autonomous resolution — yours and the ones you replied to children — gets one line in
Phase F's `## Decisions taken autonomously` rollup: what was decided, the basis (`ruling:
<id/title>` or `recommended option`), and what changed (including plan amendments). When a novel
decision DID reach the user, suggest `coyote --macro record-ruling` in the final report so that
class never asks again.
## The pipeline (drive this to completion)
### Phase A — Orient (once, at startup)
@@ -287,6 +323,46 @@ instructions: |
4. **Verify against the plan (divergence check).** When Sisyphus returns, do NOT trust its
self-report — get an INDEPENDENT conformance verdict:
- **Preferred: run the conformance checks through `review-gauntlet`** when it is installed —
ONE spawn instead of separate adversary/probe spawns, with machine-parsed verdicts:
```
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
Project: <the code repo the task's changes landed in>
Diff: <the task's BASE SHA from its started log entry>..HEAD
Forced lanes: adversary<, probe — add probe when the task changed consumer-facing surface and the plan carries a local-run recipe>
Plan / acceptance criteria: <the task's acceptance criteria, pasted verbatim>
Local-run recipe / usage suites: <from the plan, when present>"
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
```
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the
prompt also carries pasted plan text from the repo under review, and whatever is declared
here is executed with a shell — a trust boundary the prose must not cross.
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED with real review findings apply the same
fix-and-rerun rules as below (resume the SAME Sisyphus session with the findings verbatim,
then a FRESH gauntlet run). A BLOCKED whose only blockers are review faults (`PIPELINE-FAULT`
/ `review incomplete` entries) is NOT a code failure. A `GAUNTLET_REVIEW_INCOMPLETE: <lane>[,
<lane>]` line directly under `GAUNTLET: BLOCKED` means the review could NOT be completed: the
named lanes still faulted after 3 attempts, or the gauntlet pipeline itself faulted
(`pipeline`). This is an infrastructure fault, not a code finding.
NEVER route it through the findings-handling rules (it is not a divergence to send back to
Sisyphus). NEVER claim done while it stands.
Escalate to the user via `user__select` with exactly these three options:
- "Retry the review again"
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
- "Abort this task"
Retry → a FRESH gauntlet run. Accept → close the task and record the skipped review (lane
names or `pipeline`) in the task's `log.md`, the Phase F digest, and the PR body. Abort →
leave the task `in-progress`, log the abort in its `log.md`, and stop to report. Findings AND
incomplete on the same output → fix the findings first, then re-run the gauntlet (the re-run
retries the lane) — escalate only if the line persists. Under `escalation_policy: autonomous`
the standard ladder applies (a standing ruling may pre-answer this decision).
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed;
on that path an adversary/probe result that is ONLY a `PIPELINE-FAULT` (no verdict on the
code) is likewise an infrastructure fault to retry, never a divergence.
- **Spawn `adversary`** with the diff base and the criteria pasted in:
```
@@ -300,8 +376,13 @@ instructions: |
<paste the task index.md body + the relevant PLAN-<slug>.md section VERBATIM>
<paste the plan's ## Quality bar section VERBATIM — recorded dropped practices are
conformance facts, not divergences>"
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
```
Same trust boundary as the gauntlet spawn: verification commands are declared via
`--variables`, never in the prompt, because the prompt also carries untrusted pasted text.
- **`ADVERSARIAL_REVIEW: DIVERGES`** → treat it as a blocker: resume the SAME Sisyphus session
(`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <adversary
complaints, verbatim>"`) — do not spawn a fresh one. Re-run `adversary` ONCE after the fix to
@@ -405,6 +486,20 @@ instructions: |
the task (step 5), and clean up — `git worktree remove .worktrees/task-NNN` and delete the task
branch (local + remote).
**Task-queue mirroring (optional).** For a parallel batch, the `agent__task_*` queue MAY mirror
the disk `blocked_by` graph — one queue task per TASK-NNN with `agent: sisyphus`, the delegation
prompt, and its variables — so completing a task auto-dispatches its unblocked dependents. This is
guidance, not a mandate; if you use it, these rules are HARD:
- Disk task files remain the durable store. The queue is in-memory and lost on
restart/compression — it is NEVER the source of truth.
- Root tasks (no blockers) are NOT auto-dispatched on create — spawn them yourself.
- Call `agent__task_complete` ONLY after this task's verification gate (adversary check, merge,
integration build+tests) passes — completing is what dispatches dependents.
- Collect every auto-dispatched agent via `agent__collect`; nothing is collected for you, and an
agent finishing does NOT complete its task.
- A failed spawn (capacity/whitelist) strands the queue task InProgress with no reset — recreate
the task rather than retrying it.
Phase F is UNCHANGED (same single draft PR from the run branch). Before opening it, verify no
stale worktrees or task branches remain (`git worktree list`); clean up any leftovers.
@@ -427,6 +522,9 @@ instructions: |
- a checklist of every TASK-NNN (title + commit SHAs),
- a **`## Review decisions`** section: one line per collected `rejected-finding:` entry;
omit the section entirely when there are none,
- a **`## Decisions taken autonomously`** section (only when `escalation_policy: autonomous`
resolved anything): one line per self-resolved decision with its basis and what changed —
rolled up from your own resolutions and each Sisyphus report's section; omit otherwise,
- a **`## Follow-up / manual actions`** section: one checkbox line per follow-up (WHAT, WHERE,
WHY, WHEN — pre-merge items FIRST and clearly marked), or "None." if there are none. This
section is the reviewer's contract for what the code does NOT do by itself.
@@ -452,7 +550,16 @@ instructions: |
the run branch (`chore(plan): ...`) — they become part of the PR.
6. Run the `task-tracking` consistency checks; fix anything you introduced.
7. Report: the PLAN, every TASK-NNN with its commits, the single draft PR URL with checks green,
the follow-up TASKs created (with their WHEN), and anything deferred/escalated. STOP.
the follow-up TASKs created (with their WHEN), and anything deferred/escalated — plus the
`## Decisions taken autonomously` rollup when the escalation policy resolved anything. STOP.
Continuous improvement: if any part of this run addressed PR review feedback that did not come
from the run's own reviewer lanes — humans, a teammate's review agent, another team's bot — spawn
`whetstone` ONCE before the final report: pass the PR ref, the human feedback verbatim, and
what the run's own review passes flagged. Collect it; its approval questions escalate to the
user directly. Never block the run's completion on it; skip silently when all feedback came from
the run's own lanes. (Per-task feedback handled inside a Sisyphus session is covered by Sisyphus's
own whetstone hook — don't double-spawn for the same feedback.)
## Durable state (survive context compression)
@@ -475,6 +582,7 @@ instructions: |
| `gatekeeper` | Plan self-containedness gate (Phase C Gate 1): audits the PLAN for the "docker container" standard, returns SEALED/LEAKY with the missing implementer questions. |
| `adversary` | Post-implementation plan-conformance verdict per task (CONFORMS/DIVERGES). |
| `probe` | Black-box behavioral verdict on a task's consumer-facing surface: boots the code locally from clean state, runs existing usage suites + spec-first tests. Returns USAGE_PROBE PASS/FAIL/INCONCLUSIVE. |
| `whetstone` | Post-run continuous improvement: classifies human review feedback vs what the suite flagged, generalizes misses, proposes suite edits / ledger entries (user-approved). Never a reviewer lane. |
## Escalation handling
@@ -482,6 +590,11 @@ instructions: |
Answer from context if you can, else prompt the user, then `agent__reply_escalation` to unblock the
child. Do not leave a child hanging.
In `autonomous` escalation policy, work the Escalation-policy ladder BEFORE prompting the user: a
matching standing ruling or a clearly recommended option answers the child directly (hard triggers
always go to the user), plan-amending replies instruct the child to rewrite the criterion text
first, and every autonomous reply lands in the Phase F digest.
## Anti-patterns (BLOCKING)
- Opening a PER-TASK PR → the design doc gets exactly ONE PR, opened in Phase F.
@@ -532,6 +645,7 @@ instructions: |
- Design doc: {{design_doc}}
- Base branch: {{base_branch}}
- Autonomy: {{autonomy}}
- Escalation policy: {{escalation_policy}}
- Parallel tasks: {{parallel_tasks}} (0 = sequential, 1 = worktree-parallel)
- OS: {{__os__}} Shell: {{__shell__}} CWD: {{__cwd__}} Now: {{__now__}}
assets/agents/architecture-reviewer/config.yaml
+5 -1
View File
@@ -12,7 +12,11 @@ spawnable_agents:
- explore
- oracle
max_concurrent_agents: 4
max_agent_depth: 2
# The limit is consulted by the SPAWNER against the child's absolute depth
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
# architecture-reviewer(2) → {explore | oracle}(3). This agent spawns depth-3
# children; 4 = one level of headroom.
max_agent_depth: 4
inject_spawn_instructions: true
skills_enabled: true
assets/agents/code-reviewer/README.md
+59
View File
@@ -1,5 +1,58 @@
# Code Reviewer
> [!IMPORTANT]
> **v3: now a GRAPH agent.** The orchestration shell is deterministic — a script resolves the
> diff/quality bar/ledger/history, domain linters run unconditionally, changed files are grouped
> into DOMAIN slices (deterministic proposal → LLM refinement → exact-cover gate) reviewed in
> parallel by [`domain-reviewer`](../domain-reviewer/README.md) leaders (peer messaging between
> leaders; per-file depth via `file-reviewer`), every leader report passes a completeness gate,
> findings are verified by `finding-verifier`, and the rigor folding + `MERGE-READY`/`NEEDS-HUMAN`
> verdict are computed BY A SCRIPT. Judgment stays in the agents; discipline lives in the graph.
> The review semantics below (severity mapping, folding rules, verdict rules, report format) are
> unchanged — they are now structurally enforced instead of prompt-enforced.
```mermaid
flowchart TD
A["parse (llm)"] --> B["facts (script): diff, quality bar, ledger, history, trigger signals"]
A -. "fallback" .-> PF["parse_fault (script): PIPELINE-FAULT verdict"]
PF --> L
B --> C["linters (script): tflint / hadolint / actionlint / buf breaking"]
C --> D1["group_propose (script): deterministic domain proposal"]
D1 --> D2["refine_groups (llm)"]
D2 --> D3["cover_gate (script: exact cover)"]
D2 -. "fallback" .-> D3
D3 --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on, max_attempts: 2"]
D3 --> F["aux_lanes (llm): org-context + prior-art spawns"]
D3 --> G["downstream_sweep (script)"]
E --> H["completeness_gate (script): required sections even when clean"]
E -. "fallback" .-> DF["domain_fault (script): ⚠️ PIPELINE-FAULT slice banner"]
H -. "reject-retry once" .-> E
H --> I["synthesize (llm: dedup + prose, NO verdict authority)"]
F --> I
F -. "fallback" .-> AF["aux_fault (script): PIPELINE-FAULT lane-skip note"]
AF --> I
G --> I
I --> J["verify: finding-verifier (agent node, unskippable, max_attempts: 2)"]
I -. "fallback" .-> K
J --> K["verdict (script): FALSE-drops, rigor folding, fault accounting, MERGE-READY / NEEDS-HUMAN arithmetic"]
J -. "fallback" .-> K
K --> L["render (script): the standard report"]
```
**Failure fails closed.** Every LLM/agent node retries once (`max_attempts: 2` — the
domain-reviewer lanes and the verifier included), then routes to a `fallback:` instead of killing
the graph, and every fallback that loses review coverage leaves a `PIPELINE-FAULT` marker: a dead
`parse` emits a complete fault verdict and jumps straight to render; a dead domain lane becomes a
`> ⚠️ PIPELINE-FAULT` banner where its slice report would be; a dead `refine_groups` falls
through to the cover gate, which keeps the deterministic proposal; a dead synthesis or verifier
falls through to the verdict script. The verdict script counts every fault and forces
`NEEDS-HUMAN` — a degraded run never reads `MERGE-READY`. The aux context lanes are the one
exception: they are enrichment, so their fault (`aux_fault`) is surfaced as a lane-skip note but
never blocks. Fault detection is prefix-anchored (a report that merely *quotes* a marker doesn't
trip it), and the report is always emitted: the verdict script's own crash guard emits a
`PIPELINE-FAULT` NEEDS-HUMAN, and render bypasses its "No changes to review." stub for fault
verdicts — a fault renders as a full report, never silence.
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
report.
@@ -13,6 +66,12 @@ agents while handling coordination and final reporting.
- 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels.
- ⚡ **Parallel Execution**: Runs reviews concurrently for maximum speed.
- 🚨 **Operational History (optional)**: Checks the change against past production incidents via the [`incident-prior-art`](../../skills/incident-prior-art/SKILL.md) skill.
- 📦 **Context pack**: Every file-reviewer gets the same assembled inputs — change intent, repo convention docs, and matching entries from the review-miss ledger — so reviews judge against the house rules, not generic taste.
- 🏛️ **Org context (optional)**: When the diff touches decision-laden surface (public API shapes, money/auth logic), the agent named by the `org_context_agent` variable is spawned to find recorded decisions the diff must honor — contradicting one is a 🔴 finding. Disabled by default (`org_context_agent: ''`).
- 📚 **Review-miss ledger**: Loads the [`review-misses`](../../skills/review-misses/SKILL.md) skill — defect classes that previously ESCAPED review run as first-class checks; recurrences default to 🔴.
- ✅ **Finding Verifier**: Before the report posts, every 🔴/🟡 is re-validated against the working tree with quoted evidence; provably-false findings are dropped (and tallied) — false positives kill trust faster than missed bugs. Verification is delegated to the [`finding-verifier`](../finding-verifier/README.md) graph agent (one map branch per finding — structurally unskippable, fresh-context independent), with an inline fallback when it isn't installed.
- 🌐 **Downstream consumers (optional)**: When the diff changes consumer-facing contract surface (routes, protos, exported API, CLI flags), sibling checkouts under `consumers_root` and a `github_org` code search are grepped for consumers of the changed elements — a consumer matching a removed/changed contract is a 🔴 finding naming repo + file:line. Public-library consumers are unenumerable; the check degrades to semver/changelog discipline. Both probes disabled by default (`consumers_root: ''`, `github_org: ''`).
- 🚦 **Review Verdict**: The report opens with `MERGE-READY` or `NEEDS-HUMAN` (+ the 1-3 items a human must judge). Always-human triggers (irreversible migrations, authn/authz, money movement, secrets/crypto, incident-linked guard deletion) force `NEEDS-HUMAN` regardless of findings. Routing signal only — merging stays a human act.
## Operational History Lane
assets/agents/code-reviewer/config.yaml
-243
View File
@@ -1,243 +0,0 @@
name: code-reviewer
description: CodeRabbit-style code reviewer - spawns per-file reviewers, synthesizes findings
version: 2.4.0
auto_continue: true
max_auto_continues: 20
inject_todo_instructions: true
can_spawn_agents: true
max_concurrent_agents: 10
max_agent_depth: 2
skills_enabled: true
enabled_skills:
- delegation-protocol
- parallel-research
- incident-prior-art
variables:
- name: project_dir
description: Project directory to review
default: '.'
- name: prior_art_agent
description: Optional agent that can search the incident record (Slack/Jira/postmortems) for operational prior art. Empty disables the delegation lane; git archaeology still runs.
default: ''
- name: rigor
description: Quality bar governing finding folding (poc | prototype | production)
default: 'production'
- name: surfaces
description: Declared surfaces as a CSV (e.g. 'rest-api,db-migration'); empty = auto-detect from the diff
default: ''
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools:
- ast_grep.sh
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
- fs_glob.sh
- execute_command.sh
instructions: |
You are a code review orchestrator, similar to CodeRabbit. You coordinate per-file reviews and produce a unified report.
## Step 0: Load orchestration skills
Before doing anything else, call `skill__load` for `delegation-protocol` and `parallel-research`. They carry the methodology you need:
- **`delegation-protocol`** — how to write delegation prompts that give the sub-agent its full context (TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT). Apply this format when spawning each file-reviewer.
- **`parallel-research`** — the spawn-and-wait protocol, the anti-duplication rule (don't redo work you delegated), and the rule about ending your response and letting the system notify you on agent completion.
Both skills are always-on for this agent's workflow. Skill bodies are your source of truth for HOW to delegate and HOW to coordinate parallel work; this agent's instructions handle the CodeRabbit-specific shape.
## Workflow
1. **Get the diff:** Run `get_diff` to get the git diff (defaults to staged changes, falls back to unstaged)
2. **Resolve quality bar:** Determine the rigor and surfaces governing this review, in strict precedence order:
- **Caller-passed wins.** If the caller passed a quality bar (`surfaces` is non-empty, or `rigor` was explicitly set by the spawner — current values: rigor='{{rigor}}', surfaces='{{surfaces}}'), use those values verbatim. Provenance: `passed`.
- **Else plan frontmatter.** If the repo's plans directory contains a `PLAN-*.md` whose frontmatter says `status: active`, read `rigor` and `surfaces` from that frontmatter. Provenance: `plan`.
- **Else detect from the diff.** Infer surfaces (route/handler files → rest-api; argparse/clap/cobra parser definitions → cli; `*.tf`/Helm charts/Dockerfiles → iac; migration dirs → db-migration; queue-consumer registration → worker; lib manifest + exported-API changes → library; workflow files → ci-cd) and keep rigor=production. Provenance: `detected-default`.
Record the resolved bar and its provenance (`passed | plan | detected-default`) — both appear in the final report footer.
3. **Parse changed files:** Extract the list of files from the diff
4. **Create todos:** One todo per phase (get diff, resolve quality bar, domain linter pass, spawn reviewers, operational-history lane, collect results, synthesize report)
5. **Domain linter pass:** For each resolved surface with a mechanized checker configured in the repo — `tflint`/`checkov` for iac (Terraform), `hadolint` for Dockerfiles, `actionlint` for CI workflow files, `kubeconform` for Kubernetes manifests — run the checker ONCE via `execute_command`, as a read-only invocation scoped to this repo. Route its output: findings relevant to a specific changed file are pasted into that file-reviewer's CONTEXT section; repo-level residue that maps to no single changed file folds into the synthesis under the owning surface. If a resolved surface has no linter configured in the repo, skip it with a one-line note in the synthesis. Never install linters and never write files in this pass.
6. **Spawn file-reviewers:** One `file-reviewer` agent per changed file, in parallel. Apply the `delegation-protocol` structured prompt format.
7. **Broadcast sibling roster:** Send each file-reviewer a message with all sibling IDs and their file assignments
8. **Operational-history lane (conditional):** Load `incident-prior-art` and follow it. If the diff touches operationally-relevant surface (per the skill's trigger list), run its git-archaeology pass yourself, and — if `prior_art_agent` is set (currently: '{{prior_art_agent}}') — spawn that agent in REVIEW MODE alongside the file-reviewers using the skill's prompt template. If the surface is not operationally relevant, skip with a one-line note.
9. **Collect all results:** Per `parallel-research`, do not poll. End your response after spawns + roster; the system will notify you when agents complete.
10. **Synthesize:** Combine all findings into a CodeRabbit-style report, applying the rigor folding rules below before assembling it. Prior-art findings go under an "Operational history" section using the skill's severity folding (reintroduction of a past incident's failure mode = CRITICAL).
## Spawning File Reviewers
Apply the `delegation-protocol` structured prompt format. Each spawn gets the full TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT sections — the file-reviewer hasn't seen the codebase or the broader PR; the spawn prompt IS its entire context.
```
agent__spawn --agent file-reviewer --prompt "
## TASK
Review the git diff for <file_path>. Produce structured findings per your output format.
## EXPECTED OUTCOME
A REVIEW_COMPLETE-terminated report following your standard format:
- ## File: <file_path>
- ### Summary (1-2 sentences)
- ### Findings (each with severity, lines, description, suggestion)
- ### Cross-File Concerns (or 'None')
## MUST DO
- Load `code-review` and `ai-slop-remover` skills before reading any code
- Load `transactional-integrity` as well if this file's diff touches state-changing code (DB writes, transactions, queue/webhook/job handlers, retries, external side effects)
- Load `logging-discipline` as well if this file's diff touches boundaries, error paths, background jobs, or state transitions
- Load the surface skill(s) routed to this file from the table below. Load rule: load a row's skill when the file matches that surface's trigger AND the surface is in the resolved surfaces list; when the surfaces were detected from the diff rather than declared (provenance `detected-default`), a trigger match alone suffices.
| declared surface | skill loaded |
|---|---|
| `rest-api` | `rest-api-review` |
| `grpc` (alias) | `rest-api-review` (gRPC section) |
| `graphql` (alias) | `rest-api-review` (GraphQL section) |
| `cli` | `cli-review` |
| `library` | `library-review` |
| `worker` | `worker-review` |
| `iac` | `iac-review` |
| `db-migration` | `migration-review` |
| `ci-cd` | `cicd-review` |
| `frontend` | no file-reviewer skill in v1 — note the declared surface in the synthesis; generic review + aspect skills still apply |
- Apply all loaded skill checklists to the diff
- Use targeted fs_read with offset/limit; max 5 file reads
- End with REVIEW_COMPLETE
## MUST NOT DO
- Do not modify files (you are read-only)
- Do not review unchanged code unrelated to the diff
- Do not omit findings to keep the report short
## CONTEXT
Project: {{project_dir}}
File under review: <file_path>
Rigor: <resolved rigor>
Surfaces: <resolved surfaces list — note when detected rather than declared>
Linter output for this file (from the domain linter pass; omit when none):
<linter findings relevant to this file>
Diff:
<diff content for this file>
"
```
Paste the actual diff hunk(s) inline — the reviewer can't see your context. If you have prior knowledge of the change's intent (PR description, ticket), include it in CONTEXT.
### Surface triggers (for routing and detection)
A file "matches a surface's trigger" when its diff touches that surface's territory, mirroring each surface skill's own load trigger:
- `rest-api` (and the `grpc`/`graphql` aliases): HTTP route or handler definitions, request/response types, OpenAPI/Swagger specs, gRPC `.proto` files or service implementations, GraphQL schemas or resolvers
- `cli`: argument-parser definitions (flag/option/subcommand declarations), a binary's main/entrypoint, subcommand modules
- `library`: the public API of a lib crate/package — exported symbols, `pub` items, `__init__`/index exports, re-export lists — or its manifest version
- `worker`: queue/stream consumer registration, job/worker handler wiring, cron or schedule definitions, or the transport configuration behind them (retry counts, prefetch, visibility timeouts, shutdown hooks)
- `iac`: `*.tf` files or Terraform modules, Helm charts or values files, Kubernetes manifests, Dockerfiles, compose files
- `db-migration`: migration directories or files, schema definition files, ORM model changes that generate schema changes
- `ci-cd`: workflow/pipeline files — `.github/workflows/*`, GitLab CI config, or equivalent pipeline definitions
## Sibling Roster Broadcast
After spawning ALL file-reviewers (collecting their IDs), send each one a message with the roster:
```
agent__send_message --id <agent_id> --message "SIBLING_ROSTER:
- <agent_id_1>: reviewing <file_1>
- <agent_id_2>: reviewing <file_2>
...
Send cross-cutting alerts to relevant siblings if your changes affect their files."
```
## Diff Parsing
Split the diff by file. Each file's diff starts with `diff --git a/<path> b/<path>`. Extract:
- The file path (from the `+++ b/<path>` line)
- All hunks for that file (from `@@` markers to the next `diff --git` or end)
Skip binary files and files with only whitespace changes.
## Rigor Folding (synthesis)
Before assembling the final report, fold findings by the resolved quality bar. Folding operates on severity plus the optional `[convention]`/`[correctness]` marker file-reviewers emit in finding titles:
- **🔴 CRITICAL never folds** — at any rigor, regardless of marker.
- **`production`**: nothing folds; report every finding as-is.
- **`prototype`**: 🟡 `[convention]` findings and all 🟢 findings move to `## Deferred by quality bar`.
- **`poc`**: 🟡 `[convention]` findings move to `## Deferred by quality bar`; 🟢 and 💡 `[convention]` findings are dropped from the report entirely.
Rules:
- Folding moves findings between sections; it never rewrites their severity tags.
- The `## Deferred by quality bar` section is excluded from the blocking counts (the footer's tallies) and from any block/no-block verdict.
- Rigor never suppresses 🔴/🟡 visibility — below-threshold 🟡s are deferred, not deleted. poc's 🟢/💡 `[convention]` drop is the one deliberate visibility exception.
- **Dedup:** an identical finding reported by two skills (e.g. a surface skill and an aspect skill like `transactional-integrity` or `logging-discipline`) → keep the aspect skill's copy and drop the duplicate.
- Repo-level residue from the domain linter pass lands under the owning surface in Detailed Findings (or Cross-File Concerns when it spans files) and folds by the same rules.
## Final Report Format
After collecting all file-reviewer results, synthesize into:
```
# Code Review Summary
## Walkthrough
<2-3 sentence overview of what the changes do as a whole>
## Changes
| File | Changes | Findings |
|------|---------|----------|
| `path/to/file1.rs` | <brief description> | 🔴 1 🟡 2 🟢 1 |
| `path/to/file2.rs` | <brief description> | 🟢 2 💡 1 |
## Detailed Findings
### `path/to/file1.rs`
<paste file-reviewer's findings here, cleaned up>
### `path/to/file2.rs`
<paste file-reviewer's findings here, cleaned up>
## Cross-File Concerns
<any cross-cutting issues identified by the teammate pattern>
## Operational history
<only when the lane ran: archaeology + prior-art findings with incident/commit references, or "no relevant incident history found">
## Deferred by quality bar
<findings folded out of the blocking sections by the resolved rigor, severity tags preserved — excluded from the counts below; omit this section at production or when nothing folded>
---
*Reviewed N files, found X critical, Y warnings, Z suggestions, W nitpicks (D deferred by quality bar)*
*Quality bar: <resolved rigor> — provenance: <passed | plan | detected-default>; surfaces: <resolved surfaces>*
```
## Edge Cases
- **Single file changed:** Still spawn one file-reviewer (for consistency), skip roster broadcast
- **Too many files (>10):** Group small files (< 20 lines changed) and review them together
- **No changes found:** Report "No changes to review" and exit
- **Binary files:** Skip with a note in the summary
## Rules
1. **Always use `get_diff` first:** Don't assume what changed
2. **Spawn in parallel:** All file-reviewers should be spawned before collecting any
3. **Don't review code yourself:** Delegate ALL review work to file-reviewers
4. **Preserve severity tags:** Don't downgrade or remove severity from file-reviewer findings — rigor folding relocates or (at poc) drops findings per its rules, but never rewrites a severity
5. **Include ALL findings:** Don't summarize away specific issues
6. **File reads:** If you do read a file directly (e.g. to verify a finding before synthesis), `fs_read` returns a TRUNCATED view with line numbers (default 2000 lines, long lines cut at 2000 chars). Use `fs_cat` only when you need the FULL untruncated contents of a file.
## Context
- Project: {{project_dir}}
- CWD: {{__cwd__}}
- Shell: {{__shell__}}
## Available Tools:
{{__tools__}}
assets/agents/code-reviewer/graph.yaml
+547
View File
@@ -0,0 +1,547 @@
name: code-reviewer
description: |
Graph-based review orchestrator — the deterministic shell around LLM review
judgment. A script resolves the diff, quality bar, conventions, review-miss
ledger patterns, and repo-history signals; domain linters run unconditionally;
changed files are grouped into DOMAIN slices (deterministic proposal, LLM
refinement, exact-cover gate) and each slice is reviewed by a domain-reviewer
leader — in parallel, with peer messaging between leaders — which delegates
per-file depth to file-reviewer. Every leader's report passes a completeness
gate (mandatory sections present even when clean, reject-and-retry once).
Findings are re-validated by the finding-verifier graph, folded by rigor and
scored into the MERGE-READY / NEEDS-HUMAN verdict BY A SCRIPT — the model
that wrote the findings never grades its own homework. Judgment stays in the
agents (per-file review, synthesis, dedup); discipline lives in the graph
(unskippable steps, identical context to every reviewer, machine-parsed
verdict math).
version: '1.0'
can_spawn_agents: true
# The limit is consulted by the SPAWNER against the child's absolute depth
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
# review-gauntlet | step-runner(2) → code-reviewer(3) →
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
# This agent spawns depth-4 children; 5 = one level of headroom.
max_agent_depth: 5
max_concurrent_agents: 16
variables:
- name: project_dir
description: Project directory containing the changes to review
default: '.'
- name: rigor
description: Quality bar rigor (poc|prototype|production); empty = resolve from plan frontmatter or default to production
default: ''
- name: surfaces
description: Comma-separated surface list governing skill routing; empty = detect from the diff
default: ''
- name: prior_art_agent
description: Agent for the operational-history lane (incident prior art); empty = skip the agent lane
default: ''
- name: org_context_agent
description: Agent for the org-context lane (recorded decisions, PRDs); empty = skip the lane
default: ''
- name: consumers_root
description: Directory of sibling consumer checkouts for the downstream sweep; empty = skip the local probe
default: ''
- name: github_org
description: GitHub org for the downstream code-search probe; empty = skip it
default: ''
settings:
max_loop_iterations: 12
log_state_snapshots: false
timeout: 10800
initial_state:
project_dir_in: ''
diff_spec: auto
intent: ''
rigor_in: ''
surfaces_in: ''
extra_context: ''
resolved_rigor: production
resolved_surfaces: ''
bar_provenance: detected-default
resolved_diff_spec: ''
changed_files: []
file_count: 0
facts_note: ''
conventions: ''
ledger_matches: ''
history_notes: ''
touches_auth: false
destructive_migration: false
consumer_surface: false
linter_notes: ''
proposed_groups: []
groups_note: ''
group_items: []
domain_reports: []
org_context_notes: ''
prior_art_notes: ''
aux_note: ''
downstream_notes: ''
walkthrough: ''
changes_rows: []
findings: []
cross_file: ''
operational_history: ''
org_context_section: ''
attention_flags: []
placement_notes: ''
verifier_output: ''
verdict_out: {}
final_report: ''
gate_feedback: ''
gate_attempts: 0
parse_failure: ''
refine_failure: ''
aux_failure: ''
synth_failure: ''
start: parse
nodes:
parse:
id: parse
type: llm
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging.
instructions: |
You extract the facts of a code-review request. EXTRACTION ONLY.
- `diff_spec`: the git diff basis the caller named — a range ("main...HEAD"),
a ref/SHA, "staged", "worktree" (uncommitted vs HEAD), or "auto" when the
caller did not name one (auto = staged, falling back to unstaged, falling
back to HEAD~1 — the classic get_diff semantics; a caller phrase like
"run get_diff --base X" means diff_spec = "X").
- `intent`: the PR description / ticket / task intent text, condensed but
specific. Empty string when the caller gave none.
- `rigor_in`: ONLY when the caller explicitly set a rigor (poc|prototype|production),
e.g. via a "Quality bar:" line; else empty string.
- `surfaces_in`: ONLY when the caller explicitly declared surfaces; else empty.
- `project_dir_in`: the directory/worktree path the caller named as
holding the code under review (e.g. "in /tmp/pr-review-x", "the repo
at ~/code/foo"); empty string when the caller named none.
- `extra_context`: any remaining reviewer-relevant caller context, condensed.
prompt: |
Extract the review-request facts from this request:
{{initial_prompt}}
tools: []
output_schema:
type: object
properties:
diff_spec:
type: string
intent:
type: string
rigor_in:
type: string
surfaces_in:
type: string
project_dir_in:
type: string
extra_context:
type: string
required: [diff_spec, intent, rigor_in, surfaces_in, project_dir_in, extra_context]
max_iterations: 3
max_attempts: 2
state_updates:
parse_failure: '{{output}}'
fallback: parse_fault
next: facts
parse_fault:
id: parse_fault
type: script
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — emits a complete PIPELINE-FAULT verdict_out and jumps straight to render; a review that could not parse its request never reads as MERGE-READY.
script: scripts/parse_fault.py
timeout: 30
next: render
facts:
id: facts
type: script
description: Deterministic facts — diff resolution, changed files, quality-bar resolution (passed → plan frontmatter → detected), conventions pointer, review-miss ledger matches, repo-history sweep, verdict-trigger signals. Pure git/gh/filesystem; no LLM.
script: scripts/facts.py
timeout: 240
next: linters
linters:
id: linters
type: script
description: Domain linter pass — tflint/hadolint/actionlint on matching changed files, buf breaking when buf-configured protos changed. Only tools already on PATH; never installs; degrades silently per tool.
script: scripts/linters.py
timeout: 420
next: group_propose
group_propose:
id: group_propose
type: script
description: Deterministic domain grouping of changed files by surface-trigger path patterns.
script: scripts/group_domains.py
timeout: 30
next: refine_groups
refine_groups:
id: refine_groups
type: llm
description: Judgment refinement of the proposed domain groups — may rebalance, never drop or invent files (the cover gate enforces it).
instructions: |
You refine a deterministic domain grouping of changed files. You may:
- move a file whose CONTENT belongs to a different domain than its path suggests;
- split a group larger than ~8 files into coherent subgroups (name them
"<domain>-1", "<domain>-2" or by subsystem);
- merge groups that are really one change (a migration and its models);
- rename a domain for clarity.
You MUST include every changed file in exactly one group — no drops, no
inventions, no duplicates. A downstream gate validates the cover and falls
back to the deterministic proposal if you violate it.
prompt: |
Proposed groups (deterministic):
{{proposed_groups}}
All changed files (every one must appear in exactly one group):
{{changed_files}}
Change intent: {{intent}}
tools: []
output_schema:
type: object
properties:
groups:
type: array
items:
type: object
properties:
domain:
type: string
files:
type: array
items:
type: string
rationale:
type: string
required: [domain, files]
required: [groups]
max_iterations: 3
max_attempts: 2
state_updates:
refine_failure: '{{output}}'
fallback: cover_gate
next: cover_gate
cover_gate:
id: cover_gate
type: script
description: Exact-cover validation of the refined groups — every changed file in exactly one group, no unknown files. Violations fall back to the deterministic proposal (noted). Emits the map items.
script: scripts/cover_gate.py
timeout: 30
next: [map_domains, aux_lanes, downstream_sweep]
map_domains:
id: map_domains
type: map
over: '{{group_items}}'
as: domain_group
branch: review_domain
output_key: domain_report
collect_into: domain_reports
max_concurrency: 4
next: synthesize
review_domain:
id: review_domain
type: agent
description: One domain-reviewer leader per domain slice — reviews the slice as a unit, delegates per-file depth to file-reviewer, messages the other leaders.
agent: domain-reviewer
teammates: true
timeout: 7200
prompt: |
## TASK
Review the changes in your assigned domain slice as ONE coherent unit and
produce a DOMAIN_REVIEW_COMPLETE-terminated report per your output format.
## ASSIGNMENT (JSON)
{{domain_group}}
## MUST DO
- Fetch your slice's diff yourself: `get_diff --files <your files, comma-separated>`
(diff spec: {{resolved_diff_spec}} — pass it as `--base` when it is a ref or range;
omit `--base` for staged/unstaged specs)
- Apply your domain charter: per-file review depth per your sizing rules, PLUS the
domain-level pass (guard symmetry across the slice, literal/enum drift,
schema↔usage mismatch, slice-level blast radius / test adequacy / removed guards)
- Report the mandatory checks EVEN WHEN CLEAN
- Message the other domain leaders when your slice changes something their domain consumes
{{gate_feedback}}
## MUST NOT DO
- No file modification; no findings omitted; never rewrite severities
## CONTEXT
Project: {{project_dir}}
Rigor: {{resolved_rigor}} (provenance: {{bar_provenance}})
Surfaces: {{resolved_surfaces}}
Intent: {{intent}}
Conventions: {{conventions}}
Review-miss patterns (run as FIRST-CLASS checks; cite MISS-NNN in the finding title when one fires):
{{ledger_matches}}
Repo-history signals:
{{history_notes}}
Linter output (extract the lines for your files):
{{linter_notes}}
{{extra_context}}
max_attempts: 2
state_updates:
domain_report: '{{output}}'
fallback: domain_fault
next: completeness_gate
completeness_gate:
id: completeness_gate
type: script
description: Deterministic report-contract gate — required sections present (even when clean) and the completion sentinel. Rejects back to the leader exactly once with the missing sections, then keeps the report banner-flagged. Never sinks the map.
script: scripts/completeness_gate.py
timeout: 30
domain_fault:
id: domain_fault
type: script
description: Fail-closed domain lane fault marker (reached only via review_domain's fallback route inside the map branch). Normalizes the engine's "Agent node failed" text in domain_report into a > ⚠️ PIPELINE-FAULT banner the map collects; verdict.py counts any faulted lane and forces NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
script: scripts/domain_fault.py
timeout: 30
aux_lanes:
id: aux_lanes
type: llm
description: Conditional context lanes — org-context and operational-history/prior-art agents, spawned only when configured AND triggered. Additive context; never blocks.
instructions: |
You run the review's conditional context lanes by spawning agents (you have the
agent__spawn / agent__collect tools). Evaluate each lane's trigger against the
change facts; spawn the applicable ones IN ONE BURST, collect, and distill.
**Org-context lane** — only if `{{org_context_agent}}` is a non-empty agent name AND
the diff touches surface where a recorded decision plausibly exists (public API
shapes, money/entitlement logic, auth flows, data retention/deletion, domain state
machines and lifecycle semantics) OR the diff's scope visibly mismatches the
ticket/intent it cites. Spawn it with ONE question: "Any recorded decisions, PRDs,
or incident-driven constraints about <changed area>? Return quoted decisions with
dates/sources." (For a ticket mismatch: "which recorded ticket/PRD does THIS change
actually implement?") Distill the answer into `org_context_notes` — quoted decisions
with dates, and whether the diff appears to honor them.
**Operational-history lane** — only if `{{prior_art_agent}}` is a non-empty agent
name AND the diff touches operationally relevant surface (workers/queues,
migrations, infra, auth, retry/timeout logic, external calls, feature flags).
Spawn it in REVIEW MODE: "REVIEW MODE: incident prior art for <the changed
paths/surfaces>. Return incidents, postmortems, alert history, and prior-PR
lessons tied to these paths, with dates and links — especially any GUARD present
in the past that this diff deletes or weakens." Distill into `prior_art_notes`;
quote anything tying a DELETED guard to a past incident verbatim (the verdict
treats reintroduced failure modes as blocking).
A lane whose trigger does not match, whose agent name is empty, or whose agent
fails: skip it and say why in `aux_note` (one line each). Never invent lane
content; empty results are valid results.
prompt: |
Change facts:
- files: {{changed_files}}
- intent: {{intent}}
- auth surface: {{touches_auth}}; migrations: {{destructive_migration}}; consumer surface: {{consumer_surface}}
- resolved surfaces: {{resolved_surfaces}}
- extra: {{extra_context}}
Configured lane agents: org-context='{{org_context_agent}}', prior-art='{{prior_art_agent}}'.
Project: {{project_dir}}
output_schema:
type: object
properties:
org_context_notes:
type: string
prior_art_notes:
type: string
aux_note:
type: string
required: [org_context_notes, prior_art_notes, aux_note]
max_iterations: 10
max_attempts: 2
timeout: 3600
state_updates:
aux_failure: '{{output}}'
fallback: aux_fault
next: synthesize
aux_fault:
id: aux_fault
type: script
description: Fail-closed marker for dead aux context lanes (reached only via aux_lanes' fallback) — records the failure as a PIPELINE-FAULT in aux_note, which synthesize surfaces as a lane skip. Aux lanes are enrichment; the degradation is visible but never blocks.
script: scripts/aux_fault.py
timeout: 30
next: synthesize
downstream_sweep:
id: downstream_sweep
type: script
description: Downstream-consumers sweep — deterministic contract-token extraction from the diff, grep of sibling checkouts, bounded org code search. Runs only when consumer-facing surface changed; degrades silently.
script: scripts/downstream_sweep.py
timeout: 240
state_updates:
downstream_notes: '{{downstream_notes}}'
next: synthesize
synthesize:
id: synthesize
type: llm
description: The judgment core — dedup, cross-file/cross-domain reasoning, prose. Emits STRUCTURED findings for the deterministic verdict; has NO verdict authority and cannot drop findings.
instructions: |
You synthesize domain review reports into the structured inputs of a code-review
report. You have NO verdict authority: the verdict, rigor folding, and severity
math happen downstream in a script, and finding verification happens in a separate
fresh-context agent. Your job is judgment, not grading.
Rules (non-negotiable):
- **Include ALL findings** from every domain report — never summarize one away.
Preserve each severity tag and any `[convention]`/`[correctness]` title marker
VERBATIM (marker goes in the `marker` field: "", "convention", or "correctness").
- **Dedup**: an identical finding reported twice (e.g. a surface skill and an
aspect skill like transactional-integrity) → keep the aspect skill's copy, drop
the duplicate. Dedup is the ONLY permitted removal.
- **Lane-derived findings you MUST add** when present in the lane notes:
- a diff that contradicts a recorded org decision (org_context_notes) → a
🔴 CRITICAL finding quoting the decision;
- a downstream consumer hit on a changed/removed contract element
(downstream_notes) → a 🔴 CRITICAL finding naming the consuming repo + file:line;
- repo-level linter residue that maps to no single file → a finding under the
owning surface (severity per the linter's own level; buf breaking = 🔴).
- **Every finding needs**: id (f1, f2, …), severity (icon + word verbatim, e.g.
"🔴 CRITICAL"), marker, file, lines, title, and `block` = the finding's full
markdown block exactly as it will appear in the report (#### title + Severity /
Lines / Description / Suggestion bullets).
- **attention_flags** (ADDITIVE-ONLY human-attention triggers; scripts add the
deterministic ones — migrations, auth — separately): money movement or
billing-amount computation changes; secrets/cryptography/key handling; a deleted
guard the operational-history lane tied to a past incident; a reintroduced
incident failure mode; a contradicted recorded decision. One short line each.
- **The placement question**: when the diff adds tolerance for a state its own
comments/errors call impossible ("should never happen", "defensive"), name the
PRODUCER of that state at path:line and put the question in `placement_notes` —
it is design intent, never auto-resolved.
- Prose sections: `walkthrough` (2-3 sentences on the change as a whole),
`changes_rows` (one {file, desc} per changed file, brief), `cross_file`
(cross-domain concerns from the leaders' reports + teammate alerts, or "None"),
`operational_history` (from prior_art_notes; "" when the lane didn't run),
`org_context_section` (from org_context_notes; "" when the lane didn't run).
prompt: |
Domain reports ({{file_count}} changed files, grouped into the domain slices below):
{{domain_reports}}
Lane notes:
- org context: {{org_context_notes}}
- operational history / prior art: {{prior_art_notes}}
- downstream consumers: {{downstream_notes}}
- lane skips: {{aux_note}}
- linter summary: {{linter_notes}}
- facts notes: {{facts_note}} {{groups_note}}
Intent: {{intent}}
Rigor: {{resolved_rigor}}; surfaces: {{resolved_surfaces}}
output_schema:
type: object
properties:
walkthrough:
type: string
changes_rows:
type: array
items:
type: object
properties:
file:
type: string
desc:
type: string
required: [file, desc]
findings:
type: array
items:
type: object
properties:
id:
type: string
severity:
type: string
marker:
type: string
file:
type: string
lines:
type: string
title:
type: string
block:
type: string
required: [id, severity, marker, file, lines, title, block]
cross_file:
type: string
operational_history:
type: string
org_context_section:
type: string
attention_flags:
type: array
items:
type: string
placement_notes:
type: string
required: [walkthrough, changes_rows, findings, cross_file, operational_history, org_context_section, attention_flags, placement_notes]
max_iterations: 6
max_attempts: 2
state_updates:
synth_failure: '{{output}}'
fallback: verdict
next: verify
verify:
id: verify
type: agent
description: Independent fresh-context verification of every 🔴/🟡 finding — structurally on the path to the report; cannot be skipped.
agent: finding-verifier
timeout: 3600
prompt: |
Verify these code-review findings against the working tree.
Project: {{project_dir}}
Findings (JSON; verify every 🔴 and 🟡 entry — id, severity, path, lines, claim
are the fields; `title` + `block` together are the claim):
{{findings}}
max_attempts: 2
state_updates:
verifier_output: '{{output}}'
fallback: verdict
next: verdict
verdict:
id: verdict
type: script
description: Deterministic verdict math — applies verifier drops (FALSE, tallied), attaches evidence, folds by rigor, computes MERGE-READY/NEEDS-HUMAN from severity counts + always-human triggers. The model never grades its own homework.
script: scripts/verdict.py
timeout: 60
next: render
render:
id: render
type: script
description: Deterministic report assembly in the standard format from the structured findings and prose sections.
script: scripts/render.py
timeout: 30
next: done
done:
id: done
type: end
output: |
{{final_report}}
assets/agents/code-reviewer/scripts/aux_fault.py
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""Fail-closed marker for dead aux context lanes of the code-reviewer graph.
Reached ONLY via aux_lanes' `fallback:` route. aux_lanes' state_updates wrote
an "LLM node …failed: <chain>" string into `aux_failure` before routing here;
this node records the failure as a PIPELINE-FAULT in `aux_note`, which flows
into synthesize's "lane skips" prompt line. Aux lanes are ENRICHMENT, not
review coverage: the degradation is made VISIBLE in the report's inputs but
never blocks — verdict.py deliberately does NOT scan aux_note for faults.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
raw = state.get("aux_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(
json.dumps(
{"aux_note": f"PIPELINE-FAULT: aux context lanes failed after retries — {detail}"}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"aux_note": "PIPELINE-FAULT: aux context lanes failed after retries — "
f"fault-marker script error: {e}"
}
)
)
assets/agents/code-reviewer/scripts/completeness_gate.py
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Report-contract gate for each domain-reviewer's output.
"Silence reads as not done": the mandatory sections must be PRESENT even when
clean. A report missing required sections is rejected back to the leader
exactly once with the missing list; a second failure keeps the report,
banner-flagged, so the synthesis (and the human) can see the gap — a
malformed report never sinks the whole review and never silently passes as
complete.
"""
import json
import os
import re
REQUIRED = [
("## Domain:", "the `## Domain:` header"),
("### Findings", "the `### Findings` section"),
("### Mandatory checks", "the `### Mandatory checks` section (report blast radius / test adequacy / guard symmetry even when clean)"),
("DOMAIN_REVIEW_COMPLETE", "the `DOMAIN_REVIEW_COMPLETE` sentinel"),
]
MAX_RETRIES = 1
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
report = state.get("domain_report")
report = report if isinstance(report, str) else ""
attempts = state.get("gate_attempts")
if not isinstance(attempts, int):
attempts = 0
missing = [desc for marker, desc in REQUIRED if marker not in report]
if not missing:
print(json.dumps({}))
return
if attempts < MAX_RETRIES:
print(
json.dumps(
{
"_next": "review_domain",
"gate_attempts": attempts + 1,
"gate_feedback": (
"- COMPLETENESS GATE REJECTED your previous report — missing: "
+ "; ".join(missing)
+ ". Re-emit the FULL report with every required section "
"(mandatory checks are reported even when clean) and end with "
"DOMAIN_REVIEW_COMPLETE."
),
}
)
)
return
group = state.get("domain_group")
domain = group.get("domain", "?") if isinstance(group, dict) else "?"
banner = (
f"> ⚠️ COMPLETENESS GATE: this domain report ({domain}) is missing required "
f"sections after one retry: {'; '.join(missing)}. Treat the missing checks as "
"NOT DONE, not as clean.\n\n"
)
print(json.dumps({"domain_report": banner + report}))
try:
main()
except Exception as e: # noqa: BLE001 — a gate crash must never sink the map
print(json.dumps({"domain_report": f"> ⚠️ COMPLETENESS GATE ERROR: {e}\n\n(report unavailable)"}))
assets/agents/code-reviewer/scripts/cover_gate.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Exact-cover gate for the refined domain groups.
The LLM refinement may rebalance the deterministic proposal, but it may not
lose, invent, or duplicate files. This gate verifies the refined groups are an
EXACT COVER of the changed-file set; any violation falls back to the
deterministic proposal with a note. The LLM can shape the grouping, never
shrink it. Emits `group_items` — the map node's item list.
Crash-safe: this node has a `next:` edge, so an unguarded crash would
tolerant-fail ONWARD with an empty `group_items` — an empty review that could
masquerade as MERGE-READY. On any internal error it therefore emits one
catch-all group so the review still runs (and verdict.py's pipeline tripwire
catches the empty-report case independently).
"""
import json
import os
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
changed = state.get("changed_files") or []
proposed = state.get("proposed_groups") or []
refined = state.get("groups") or []
note = ""
chosen = None
if isinstance(refined, list) and refined:
seen = []
valid = True
for g in refined:
if not isinstance(g, dict) or not isinstance(g.get("files"), list) or not g.get("domain"):
valid = False
note = "refined groups malformed — fell back to the deterministic grouping"
break
seen.extend(g["files"])
if valid:
if sorted(seen) != sorted(changed):
missing = sorted(set(changed) - set(seen))
extra = sorted(set(seen) - set(changed))
dupes = sorted({f for f in seen if seen.count(f) > 1})
note = (
"refined groups failed the exact-cover check "
f"(missing={missing[:5]} extra={extra[:5]} duplicated={dupes[:5]}) — "
"fell back to the deterministic grouping"
)
else:
chosen = [{"domain": g["domain"], "files": g["files"]} for g in refined]
if chosen is None:
# refine_groups' fallback lands here with `groups` never set; make the
# degradation visible in the synthesis prompt via groups_note.
refine_failure = state.get("refine_failure")
if not refined and isinstance(refine_failure, str) and refine_failure.startswith("LLM node"):
note = "refinement lane failed after retries — fell back to the deterministic grouping"
chosen = [{"domain": g["domain"], "files": g["files"]} for g in proposed if isinstance(g, dict)]
print(json.dumps({"group_items": chosen, "groups_note": note}))
try:
main()
except Exception as e: # noqa: BLE001 — a crash must never tolerant-fail into an empty review
try:
files = load_state().get("changed_files") or []
except Exception: # noqa: BLE001
files = []
print(
json.dumps(
{
"group_items": [{"domain": "all-changes", "files": files}],
"groups_note": f"cover gate error: {e} — fell back to a single catch-all group",
}
)
)
assets/agents/code-reviewer/scripts/domain_fault.py
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Fail-closed domain lane fault marker for the code-reviewer graph.
Reached ONLY via review_domain's `fallback:` route inside the map branch.
Grounded in engine behavior: on agent-node failure the engine writes
"Agent node failed: {e:#}" into `domain_report` via the node's state_updates
BEFORE routing to the fallback, so the raw failure text is already in state.
This node normalizes it into a "> ⚠️ PIPELINE-FAULT: domain review lane
failed after retries — …" banner the map collects as the lane's report;
verdict.py counts any banner-prefixed report as a fault and forces
NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
Slice identity comes from `domain_group` (the map item cover_gate emitted,
which carries `domain` and `files`); a missing/malformed item degrades to
the failure detail alone.
"""
import json
import os
MAX_DETAIL_CHARS = 500
FAULT_PREFIX = "> ⚠️ PIPELINE-FAULT: domain review lane failed after retries — "
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
slice_desc = ""
group = state.get("domain_group")
if isinstance(group, dict):
domain = group.get("domain")
files = [f for f in (group.get("files") or []) if isinstance(f, str)]
if isinstance(domain, str) and domain.strip():
slice_desc = f"domain '{domain.strip()}' (files: {', '.join(files)}): "
raw = state.get("domain_report")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(json.dumps({"domain_report": f"{FAULT_PREFIX}{slice_desc}{detail}"}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"domain_report": "> ⚠️ PIPELINE-FAULT: domain review lane failed "
f"after retries — fault-marker script error: {e}"
}
)
)
assets/agents/code-reviewer/scripts/downstream_sweep.py
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""Downstream-consumers sweep for the code-reviewer graph.
Runs only when consumer-facing contract surface changed. Extracts the most
specific changed contract tokens from the diff deterministically (route path
literals, proto message/service/rpc names, exported symbols), greps sibling
checkouts under `consumers_root`, and runs a bounded org code search via gh.
Hits are reported for the synthesis to raise as 🔴 findings naming the
consuming repo + file:line. Degrades silently when unconfigured/offline;
public-library caveats are noted, not guessed at.
"""
import json
import os
import re
import shutil
import subprocess
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
state = load_state()
proj = os.path.abspath(state.get("project_dir") or ".")
spec = state.get("resolved_diff_spec") or "worktree"
consumers_root = os.path.expanduser((state.get("consumers_root") or "").strip())
github_org = (state.get("github_org") or "").strip()
if not state.get("consumer_surface"):
print(json.dumps({"downstream_notes": "(skipped — no consumer-facing contract surface in the diff)"}))
raise SystemExit(0)
def diff_text():
args = {"staged": ["--cached"], "worktree": ["HEAD"], "HEAD~1": ["HEAD~1"]}.get(spec, [spec])
r = subprocess.run(["git", "-C", proj, "diff", *args], capture_output=True, text=True, timeout=90)
return r.stdout if r.returncode == 0 else ""
try:
changed = [
l[1:] for l in diff_text().splitlines()
if (l.startswith("+") or l.startswith("-")) and not l.startswith(("+++", "---"))
]
text = "\n".join(changed)
tokens = set()
tokens.update(re.findall(r'["\'](/[a-z0-9][a-z0-9/_{}.:-]{5,})["\']', text)) # route paths
tokens.update(re.findall(r"^\s*(?:message|service|rpc)\s+([A-Z][A-Za-z0-9_]{5,})", text, re.M)) # proto names
tokens.update(re.findall(r"^\s*(?:pub\s+fn|pub\s+struct|func|export\s+(?:function|const|class))\s+([A-Z][A-Za-z0-9_]{5,})", text, re.M))
tokens = sorted(tokens, key=len, reverse=True)[:5]
notes = []
if not tokens:
notes.append("(no specific contract tokens extractable from the diff — sweep skipped)")
else:
notes.append(f"contract tokens probed: {tokens}")
if consumers_root and os.path.isdir(consumers_root):
self_name = os.path.basename(proj)
for tok in tokens:
r = subprocess.run(
["grep", "-rIl", "--exclude-dir=.git", "--exclude-dir=node_modules",
"--exclude-dir=target", "--exclude-dir=vendor", tok, consumers_root],
capture_output=True, text=True, timeout=60,
)
hits = [
h for h in r.stdout.splitlines()
if h.strip() and f"/{self_name}/" not in h and not h.startswith(proj)
][:4]
for h in hits:
notes.append(f"CONSUMER HIT (sibling checkout): {h} references '{tok}' — "
"a changed/removed contract element with a live consumer is a 🔴 finding")
elif consumers_root:
notes.append(f"(consumers_root '{consumers_root}' not found — local probe skipped)")
else:
notes.append("(consumers_root unset — local probe skipped)")
if github_org and shutil.which("gh"):
env = dict(os.environ)
env.pop("CLICOLOR_FORCE", None)
env.pop("FORCE_COLOR", None)
env["GH_NO_UPDATE_NOTIFIER"] = "1"
for tok in tokens:
r = subprocess.run(
["gh", "api", "search/code", "-f", f'q=org:{github_org} "{tok}"',
"--jq", r'.items[:4][] | "\(.repository.full_name): \(.path)"'],
capture_output=True, text=True, timeout=30, env=env,
)
if r.returncode == 0 and r.stdout.strip():
for line in r.stdout.strip().splitlines():
notes.append(f"CONSUMER HIT (org search): {line} references '{tok}'")
elif github_org:
notes.append("(gh unavailable — org code search skipped)")
else:
notes.append("(github_org unset — org code search skipped)")
notes.append("Public-library caveat: consumers of a PUBLIC library are unenumerable — "
"for exported-API changes the check degrades to semver/changelog discipline.")
print(json.dumps({"downstream_notes": "\n".join(notes)}))
except Exception as e: # noqa: BLE001
print(json.dumps({"downstream_notes": f"(sweep degraded: {e})"}))
assets/agents/code-reviewer/scripts/facts.py
+283
View File
@@ -0,0 +1,283 @@
#!/usr/bin/env python3
"""Deterministic review facts for the code-reviewer graph.
Resolves the diff, the changed-file list, the quality bar (caller-passed →
plan frontmatter → detected default), the conventions pointer, review-miss
ledger matches, the repo-history sweep (fix/revert clusters, bounded gh
PR-comment pull), and the deterministic verdict-trigger signals. Pure
git/gh/filesystem — no LLM. Failures degrade into `facts_note`; they never
sink the review.
"""
import glob
import json
import os
import re
import subprocess
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
state = load_state()
proj = os.path.expanduser(
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
)
spec = (state.get("diff_spec") or "auto").strip()
notes = []
out = {"project_dir": proj}
def git(*args, timeout=60):
r = subprocess.run(["git", "-C", proj, *args], capture_output=True, text=True, timeout=timeout)
if r.returncode != 0:
raise RuntimeError((r.stderr or r.stdout).strip()[:300])
return r.stdout
def scrubbed_env():
env = dict(os.environ)
env.pop("CLICOLOR_FORCE", None)
env.pop("FORCE_COLOR", None)
env["GH_NO_UPDATE_NOTIFIER"] = "1"
env["NO_COLOR"] = "1"
return env
# --- diff resolution (get_diff semantics) -----------------------------------
resolved = spec
files = []
try:
if spec in ("auto", ""):
names = git("diff", "--cached", "--numstat")
resolved = "staged"
if not names.strip():
names = git("diff", "--numstat")
resolved = "worktree"
if not names.strip():
names = git("diff", "HEAD~1", "--numstat")
resolved = "HEAD~1"
elif spec == "staged":
names = git("diff", "--cached", "--numstat")
elif spec == "worktree":
names = git("diff", "HEAD", "--numstat")
else:
names = git("diff", spec, "--numstat")
for line in names.splitlines():
parts = line.split("\t")
if len(parts) == 3:
added, deleted, path = parts
if added == "-" and deleted == "-":
notes.append(f"binary file skipped: {path}")
continue
files.append(path.strip())
except Exception as e: # noqa: BLE001
notes.append(f"diff resolution failed for spec {spec!r}: {e}")
out["resolved_diff_spec"] = resolved
out["changed_files"] = files[:300]
out["file_count"] = len(files)
def diff_text():
try:
if resolved == "staged":
return git("diff", "--cached", timeout=90)
if resolved == "worktree":
return git("diff", "HEAD", timeout=90) if spec == "worktree" else git("diff", timeout=90)
if resolved == "HEAD~1":
return git("diff", "HEAD~1", timeout=90)
return git("diff", resolved, timeout=90)
except Exception: # noqa: BLE001
return ""
dtext = diff_text()
added_lines = "\n".join(
l for l in dtext.splitlines() if l.startswith("+") and not l.startswith("+++")
)
# --- quality bar resolution --------------------------------------------------
rigor_in = (state.get("rigor_in") or state.get("rigor") or "").strip()
surfaces_in = (state.get("surfaces_in") or state.get("surfaces") or "").strip()
resolved_rigor, resolved_surfaces, provenance = "production", "", "detected-default"
if rigor_in or surfaces_in:
resolved_rigor = rigor_in or "production"
resolved_surfaces = surfaces_in
provenance = "passed"
else:
for plan in sorted(glob.glob(os.path.join(proj, "plans", "PLAN-*.md"))):
try:
with open(plan) as f:
head = f.read(4000)
m = re.match(r"\A---\n(.*?)\n---", head, re.S)
if m and re.search(r"^status:\s*active\s*$", m.group(1), re.M):
rig = re.search(r"^rigor:\s*(\S+)", m.group(1), re.M)
sur = re.search(r"^surfaces:\s*(.+)$", m.group(1), re.M)
if rig:
resolved_rigor = rig.group(1).strip()
if sur:
resolved_surfaces = sur.group(1).strip("[] \n")
provenance = "plan"
break
except OSError:
continue
SURFACE_PATTERNS = [
("db-migration", r"migrations?/|\.sql$|schema\.(rb|prisma|sql)"),
("iac", r"\.tf$|helm|charts?/|Dockerfile|docker-compose|k8s/|manifests?/"),
("ci-cd", r"\.github/workflows/|\.gitlab-ci|Jenkinsfile|\.circleci/"),
("worker", r"workers?/|jobs?/|consumers?/|cron|celery|sidekiq|river"),
("rest-api", r"routes?|handlers?|controllers?|endpoints?|\bapi\b|\.proto$|openapi|swagger|graphql|resolvers?"),
("cli", r"\bcmd/|\bcli\b"),
("library", r"go\.mod$|package\.json$|Cargo\.toml$|pyproject\.toml$|setup\.py$|__init__\.py$"),
]
if provenance == "detected-default" and files:
detected = []
for surface, pat in SURFACE_PATTERNS:
if any(re.search(pat, f, re.I) for f in files):
detected.append(surface)
resolved_surfaces = ", ".join(detected)
out["resolved_rigor"] = resolved_rigor
out["resolved_surfaces"] = resolved_surfaces
out["bar_provenance"] = provenance
# --- conventions pointer ------------------------------------------------------
conv = []
for pat in ("CLAUDE.md", "COYOTE.md", "CONTRIBUTING.md", ".golangci*", ".eslintrc*",
"rustfmt.toml", ".rubocop.yml", "ruff.toml", ".editorconfig", "Makefile"):
conv.extend(sorted(glob.glob(os.path.join(proj, pat)))[:2])
out["conventions"] = ", ".join(os.path.relpath(p, proj) for p in conv) or "none found"
# --- review-miss ledger -------------------------------------------------------
ledger_text = ""
for cand in (
os.path.join(os.path.expanduser("~"), ".config", "coyote", "share", "review-misses.md"),
os.path.join(os.path.expanduser("~"), ".config", "coyote", "review-misses.md"),
):
if os.path.isfile(cand):
try:
with open(cand) as f:
ledger_text = f.read()
except OSError:
pass
break
matches = []
if ledger_text:
try:
repo_name = os.path.basename(git("rev-parse", "--show-toplevel").strip())
except Exception: # noqa: BLE001
repo_name = os.path.basename(os.path.abspath(proj))
keywords = {repo_name.lower()}
keywords.update(s.strip().lower() for s in resolved_surfaces.split(",") if s.strip())
for f in files:
keywords.update(p.lower() for p in re.split(r"[/_.-]", f) if len(p) >= 4)
entries = re.split(r"(?=^## MISS-)", ledger_text, flags=re.M)
for entry in entries:
if not entry.startswith("## MISS-"):
continue
body = entry.lower()
if any(k in body for k in keywords):
lines = entry.strip().splitlines()
matches.append("\n".join(lines[:8]))
if len(matches) >= 8:
break
out["ledger_matches"] = "\n\n".join(matches) if matches else "(no matching ledger entries)"
# --- repo-history sweep -------------------------------------------------------
history = []
fragile = []
try:
if files:
log = git("log", "--oneline", "-15", "--", *files[:100])
history.append(log.strip()[:2000])
per_file = {}
for f in files[:50]:
try:
fl = git("log", "--oneline", "-10", "--", f)
except Exception: # noqa: BLE001
continue
hits = [l for l in fl.splitlines() if re.search(r"\b(fix|revert|hotfix)\b", l, re.I)]
if len(hits) >= 2:
fragile.append(f)
per_file[f] = hits[:2]
for f, hits in per_file.items():
history.append(f"FRAGILE: {f} — {len(hits)}+ fix/revert commits recently — raise scrutiny")
except Exception as e: # noqa: BLE001
notes.append(f"history sweep failed: {e}")
# gh one-level-deeper: PR review comments for fragile files (bounded, silent skip)
try:
if fragile:
remote = git("remote", "get-url", "origin").strip()
m = re.search(r"github\.com[:/]([^/]+/[^/.]+)", remote)
if m:
repo_slug = m.group(1)
env = scrubbed_env()
shas = []
for f in fragile[:2]:
lg = git("log", "--format=%H", "-2", "--", f)
shas.extend(lg.split())
pr_nums = []
for sha in shas[:3]:
r = subprocess.run(
["gh", "api", f"repos/{repo_slug}/commits/{sha}/pulls", "--jq", ".[].number"],
capture_output=True, text=True, timeout=30, env=env,
)
if r.returncode == 0:
pr_nums.extend(n for n in r.stdout.split() if n.isdigit())
seen = []
for n in pr_nums:
if n not in seen:
seen.append(n)
comments = []
for n in seen[:3]:
r = subprocess.run(
["gh", "api", f"repos/{repo_slug}/pulls/{n}/comments",
"--jq", r'.[] | "\(.path): \(.body)"'],
capture_output=True, text=True, timeout=30, env=env,
)
if r.returncode == 0:
for line in r.stdout.splitlines():
path = line.split(":", 1)[0]
if any(path in f or f in path for f in files):
comments.append(line[:200])
if len(comments) >= 6:
break
if comments:
history.append("Prior PR review comments on fragile paths:\n" + "\n".join(comments[:6]))
except Exception: # noqa: BLE001
pass # offline / not GitHub / no gh — silent by design
out["history_notes"] = "\n".join(h for h in history if h) or "(no notable history)"
# --- deterministic verdict-trigger signals -----------------------------------
def any_path(pattern):
return any(re.search(pattern, f, re.I) for f in files)
migration_files = [f for f in files if re.search(r"migrations?/|\.sql$", f, re.I)]
mig_added = "\n".join(
l for l in dtext.splitlines()
if l.startswith("+") and not l.startswith("+++")
) if migration_files else ""
out["destructive_migration"] = bool(
migration_files
and re.search(r"(?i)drop\s+(table|column)|rename\s+(table|column|to)|alter\s+\S+\s+drop", mig_added)
)
out["touches_auth"] = bool(
any_path(r"auth(?!or)|login|logout|token|session|permission|rbac|\bacl\b|credential")
or re.search(r"auth(?!or)|password|jwt|session|credential", added_lines, re.I)
)
out["consumer_surface"] = any_path(
r"routes?|handlers?|\bapi\b|endpoints?|\bcmd/|\bcli\b|\.proto$|openapi|swagger|controllers?|graphql"
)
if notes:
out["facts_note"] = "; ".join(notes)[:800]
print(json.dumps(out))
assets/agents/code-reviewer/scripts/group_domains.py
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Deterministic domain grouping of changed files for the code-reviewer graph.
Each file lands in the FIRST matching domain (patterns ordered most-specific
first); unmatched files land in `general`. When the grouping is very wide,
single-file groups merge into `general` so the fan-out stays sane. The LLM
refinement node may rebalance this proposal; the cover gate guarantees no
file is lost either way. Crash-safe: an error emits one catch-all proposal
rather than tolerant-failing into an empty grouping.
"""
import json
import os
import re
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
DOMAINS = [
("db", r"migrations?/|\.sql$|schema\.(rb|prisma|sql)|models?/"),
("iac", r"\.tf$|helm|charts?/|(^|/)Dockerfile|docker-compose|k8s/|manifests?/"),
("ci-cd", r"\.github/workflows/|\.gitlab-ci|Jenkinsfile|\.circleci/"),
("workers", r"workers?/|jobs?/|consumers?/|cron|tasks?/"),
("api", r"routes?|handlers?|controllers?|endpoints?|\bapi\b|\.proto$|openapi|swagger|graphql|resolvers?"),
("cli", r"\bcmd/|\bcli\b"),
("deps", r"go\.(mod|sum)$|package(-lock)?\.json$|yarn\.lock$|Cargo\.(toml|lock)$|requirements[^/]*\.txt$|pyproject\.toml$|Gemfile|pom\.xml$|composer\.json$"),
("tests", r"(^|/)tests?/|_test\.|\.test\.|spec\."),
("docs", r"\.(md|rst|adoc|txt)$"),
]
def main():
state = load_state()
files = state.get("changed_files") or []
groups = {}
for f in files:
domain = "general"
for name, pat in DOMAINS:
if re.search(pat, f, re.I):
domain = name
break
groups.setdefault(domain, []).append(f)
# Wide fan-out: merge single-file groups into general.
if len(groups) > 6:
for name in list(groups):
if name != "general" and len(groups[name]) == 1:
groups.setdefault("general", []).extend(groups.pop(name))
proposed = [{"domain": k, "files": v} for k, v in sorted(groups.items())]
print(json.dumps({"proposed_groups": proposed}))
try:
main()
except Exception as e: # noqa: BLE001 — never tolerant-fail into an empty grouping
try:
files = load_state().get("changed_files") or []
except Exception: # noqa: BLE001
files = []
print(json.dumps({"proposed_groups": [{"domain": "all-changes", "files": files}]}))
assets/agents/code-reviewer/scripts/linters.py
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Domain linter pass for the code-reviewer graph.
Runs whichever mechanized checkers exist on PATH against the matching changed
files: tflint (Terraform), hadolint (Dockerfiles), actionlint (workflow
files), and `buf breaking` when buf-configured protos changed. Read-only,
never installs anything, and each tool degrades silently to a skip note.
Output is routed per-file where possible so the domain leaders can extract
their slice's lines.
"""
import json
import os
import re
import shutil
import subprocess
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
state = load_state()
proj = state.get("project_dir") or "."
files = state.get("changed_files") or []
spec = state.get("resolved_diff_spec") or "worktree"
notes = []
def run(cmd, timeout=60):
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, cwd=proj)
return (r.stdout + r.stderr).strip()
def capped(text, n=40):
lines = text.splitlines()
return "\n".join(lines[:n]) + (f"\n… ({len(lines) - n} more lines)" if len(lines) > n else "")
try:
tf = [f for f in files if f.endswith(".tf")]
if tf and shutil.which("tflint"):
notes.append("[tflint]\n" + capped(run(["tflint", "--no-color"], 120) or "clean"))
elif tf:
notes.append("[tflint] skipped — not on PATH")
dockers = [f for f in files if re.search(r"(^|/)Dockerfile", f)]
if dockers and shutil.which("hadolint"):
notes.append("[hadolint]\n" + capped(run(["hadolint", "--no-color", *dockers[:10]], 60) or "clean"))
elif dockers:
notes.append("[hadolint] skipped — not on PATH")
workflows = [f for f in files if ".github/workflows/" in f]
if workflows and shutil.which("actionlint"):
notes.append("[actionlint]\n" + capped(run(["actionlint", "-no-color", *workflows[:10]], 60) or "clean"))
elif workflows:
notes.append("[actionlint] skipped — not on PATH")
protos = [f for f in files if f.endswith(".proto")]
if protos and os.path.isfile(os.path.join(proj, "buf.yaml")) and shutil.which("buf"):
base = None
if "..." in spec:
base = spec.split("...")[0]
elif ".." in spec:
base = spec.split("..")[0]
elif spec not in ("staged", "worktree", "HEAD~1"):
base = spec
else:
try:
r = subprocess.run(["git", "-C", proj, "rev-parse", "HEAD"],
capture_output=True, text=True, timeout=15)
base = r.stdout.strip() if r.returncode == 0 else None
except Exception: # noqa: BLE001
base = None
if base:
out = run(["buf", "breaking", "--against", f".git#ref={base}"], 120)
notes.append("[buf breaking] (wire-compat breaks are 🔴 by default)\n" + capped(out or "clean"))
else:
notes.append("[buf breaking] skipped — no base ref resolvable")
elif protos:
notes.append("[buf breaking] skipped — no buf.yaml or buf not on PATH")
except Exception as e: # noqa: BLE001
notes.append(f"[linters] pass degraded: {e}")
print(json.dumps({"linter_notes": "\n\n".join(notes) if notes else "(no applicable linters)"}))
assets/agents/code-reviewer/scripts/parse_fault.py
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead parse stage of the code-reviewer graph.
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
this node emits a complete PIPELINE-FAULT `verdict_out` and the graph jumps
straight to render — a review that could not even parse its request never
reads as MERGE-READY. render.py renders a PIPELINE-FAULT verdict despite the
empty changed-file list (its "No changes to review." early return skips
fault-reason verdicts).
"""
import json
import os
MAX_DETAIL_CHARS = 500
ATTENTION_LINE = (
"PIPELINE-FAULT: parse failed — the review request could not be parsed; "
"the review did not run"
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def fault_verdict(detail):
return {
"verdict": "NEEDS-HUMAN",
"reason": f"PIPELINE-FAULT: parse failed — {detail}",
"counts": {},
"deferred_count": 0,
"dropped_count": 0,
"dropped_titles": [],
"attention": [ATTENTION_LINE],
"findings_final": [],
}
def main():
state = load_state()
raw = state.get("parse_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the parse stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(json.dumps({"verdict_out": fault_verdict(detail)}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(json.dumps({"verdict_out": fault_verdict(f"fault-marker script error: {e}")}))
assets/agents/code-reviewer/scripts/render.py
+125
View File
@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""Deterministic report assembly for the code-reviewer graph.
Builds the standard report format from the structured findings and prose
sections. Pure templating — no judgment, no re-summarization, no finding
ever added or removed here.
"""
import json
import os
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
# LLM-influenced values rendered after the summary line: a newline could
# forge a second summary line that review-gauntlet/verdict_gate.py parses.
def one_line(x):
return str(x).replace("\n", " ")
def main():
state = load_state()
v = state.get("verdict_out") or {}
findings = v.get("findings_final") or []
counts = v.get("counts") or {}
changed = state.get("changed_files") or []
rigor = one_line(state.get("resolved_rigor") or "production")
# A fault can leave changed_files empty (a parse fault dies before the
# diff is resolved; a verifier fault fires even on an empty diff) — a
# fault-degraded verdict must still render as a full report. parse_fault
# puts the marker in the reason; verdict.py prepends it to attention.
faulted = str(v.get("reason") or "").startswith("PIPELINE-FAULT:") or any(
isinstance(a, str) and a.startswith("PIPELINE-FAULT:") for a in v.get("attention") or []
)
if not changed and not faulted:
print(json.dumps({"final_report": "# Code Review Summary\n\nNo changes to review."}))
return
verdict = v.get("verdict", "NEEDS-HUMAN")
lines = [
"# Code Review Summary",
"",
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
f"**Verdict: {verdict}** — {one_line(v.get('reason', ''))}",
"",
"## Walkthrough",
state.get("walkthrough") or "(no walkthrough provided)",
"",
"## Changes",
"",
"| File | Changes | Findings |",
"|------|---------|----------|",
]
desc_by_file = {}
for row in state.get("changes_rows") or []:
if isinstance(row, dict) and row.get("file"):
desc_by_file[row["file"]] = row.get("desc", "")
blocking = [f for f in findings if f.get("section") == "blocking"]
deferred = [f for f in findings if f.get("section") == "deferred"]
for f in changed:
per = [x for x in blocking if x.get("file") == f]
tally = " ".join(
f"{icon} {n}" for icon, n in
((i, sum(1 for x in per if x.get("icon") == i)) for i in ("🔴", "🟡", "🟢", "💡"))
if n
) or "—"
lines.append(f"| `{f}` | {desc_by_file.get(f, '')} | {tally} |")
lines += ["", "## Detailed Findings", ""]
by_file = {}
for x in blocking:
by_file.setdefault(x.get("file", "(unattributed)"), []).append(x)
if by_file:
for f in sorted(by_file):
lines += [f"### `{f}`", ""]
for x in by_file[f]:
lines += [x.get("block", ""), ""]
else:
lines += ["No blocking findings.", ""]
lines += ["## Cross-File Concerns", state.get("cross_file") or "None", ""]
if (state.get("operational_history") or "").strip():
lines += ["## Operational history", state["operational_history"], ""]
if (state.get("org_context_section") or "").strip():
lines += ["## Org context", state["org_context_section"], ""]
if deferred and rigor != "production":
lines += ["## Deferred by quality bar", ""]
for x in deferred:
lines += [x.get("block", ""), ""]
attention = v.get("attention") or []
if verdict == "NEEDS-HUMAN" and attention:
lines += ["## Human attention required", ""]
lines += [f"- {a}" for a in attention]
lines += [""]
miss_ids = sorted({
m for x in findings for m in
__import__("re").findall(r"MISS-\d+", (x.get("title") or "") + (x.get("block") or ""))
})
dropped = ", ".join(one_line(t) for t in v.get("dropped_titles", [])[:5])
lines += [
"---",
# CONTRACT: review-gauntlet/verdict_gate.py parses this line — keep the format in sync
f"*Reviewed {len(changed)} files, found {counts.get('🔴', 0)} critical, "
f"{counts.get('🟡', 0)} warnings, {counts.get('🟢', 0)} suggestions, "
f"{counts.get('💡', 0)} nitpicks ({v.get('deferred_count', 0)} deferred by quality bar)*",
f"*Quality bar: {rigor} — provenance: {one_line(state.get('bar_provenance', '?'))}; "
f"surfaces: {one_line(state.get('resolved_surfaces') or 'none')}*",
f"*Verdict: {verdict}; verifier: dropped {v.get('dropped_count', 0)} provably-false"
+ (f" ({dropped})" if v.get("dropped_count") else "")
+ f"; review-miss patterns applied: {', '.join(miss_ids) if miss_ids else 'none relevant'}*",
]
print(json.dumps({"final_report": "\n".join(lines)}))
try:
main()
except Exception as e: # noqa: BLE001
print(json.dumps({"final_report": f"# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — report rendering error: {e}"}))
assets/agents/code-reviewer/scripts/verdict.py
+245
View File
@@ -0,0 +1,245 @@
#!/usr/bin/env python3
"""Deterministic verdict math for the code-reviewer graph.
The model that wrote the findings never grades its own homework:
- verifier verdicts are applied mechanically (FALSE → drop, tallied;
VERIFIED → evidence attached; UNVERIFIABLE → kept, marked);
- rigor folding is arithmetic over severity + [convention]/[correctness]
markers, exactly per the published rules;
- MERGE-READY/NEEDS-HUMAN is computed from counts + always-human triggers
(deterministic signals ∪ the synthesis's ADDITIVE-only flags).
Fail-closed: any PIPELINE-FAULT recorded upstream (dead domain lane, dead
verifier, verifier verdicts carrying a PIPELINE-FAULT note or the
`pipeline-fault` sentinel id, dead synthesis, missing domain reports)
forces NEEDS-HUMAN — a degraded run can never read MERGE-READY. Faults
always lead the attention list; the display cap applies only to the
non-fault entries so a degraded run can never hide a fault.
The gate never crashes into a silent verdict: any internal error emits
NEEDS-HUMAN naming the error.
"""
import json
import os
import re
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def sev_icon(severity):
for icon in ("🔴", "🟡", "🟢", "💡"):
if icon in (severity or ""):
return icon
return "🟢"
def parse_verifier_entries(text):
"""Extract the raw list of {id, verdict, evidence, note} dicts from
finding-verifier output. Order and duplicates are preserved so fault
accounting can count every entry (crash-guard entries all carry the
same "unknown" id)."""
if not isinstance(text, str) or not text.strip():
return []
start, end = text.find("["), text.rfind("]")
if start != -1 and end > start:
try:
return [
v for v in json.loads(text[start : end + 1]) if isinstance(v, dict) and v.get("id")
]
except (ValueError, TypeError):
pass
entries = []
for m in re.finditer(r'\{[^{}]*"id"[^{}]*\}', text):
try:
v = json.loads(m.group(0))
if v.get("id"):
entries.append(v)
except (ValueError, TypeError):
continue
return entries
def parse_verifier(entries):
"""Index verifier entries by finding id; the last entry per id wins."""
return {str(v["id"]): v for v in entries}
def main():
state = load_state()
findings = [f for f in (state.get("findings") or []) if isinstance(f, dict)]
rigor = (state.get("resolved_rigor") or "production").lower()
verifier_entries = parse_verifier_entries(state.get("verifier_output"))
verdicts = parse_verifier(verifier_entries)
kept, dropped_titles = [], []
for f in findings:
icon = sev_icon(f.get("severity"))
v = verdicts.get(str(f.get("id")))
block = f.get("block") or f"#### {f.get('title', '(untitled)')}"
if v and icon in ("🔴", "🟡"):
if v.get("verdict") == "FALSE":
dropped_titles.append(f.get("title", f.get("id", "?")))
continue
if v.get("verdict") == "VERIFIED" and v.get("evidence"):
block += f"\n- **Evidence** (verifier): {v['evidence']}"
elif v.get("verdict") == "UNVERIFIABLE":
block += "\n- *(unverified: " + (v.get("note") or "could not be established") + ")*"
elif icon in ("🔴", "🟡") and not v:
block += "\n- *(unverified: no verifier verdict returned for this finding)*"
kept.append({**f, "icon": icon, "block": block})
# --- rigor folding (arithmetic, never rewrites severities) --------------
for f in kept:
marker = (f.get("marker") or "").lower()
icon = f["icon"]
section = "blocking"
if icon == "🔴":
section = "blocking"
elif rigor == "prototype" and ((icon == "🟡" and marker == "convention") or icon == "🟢"):
section = "deferred"
elif rigor == "poc":
if icon == "🟡" and marker == "convention":
section = "deferred"
elif icon in ("🟢", "💡") and marker == "convention":
section = "dropped-by-poc"
f["section"] = section
final = [f for f in kept if f["section"] != "dropped-by-poc"]
blocking = [f for f in final if f["section"] == "blocking"]
deferred = [f for f in final if f["section"] == "deferred"]
counts = {i: sum(1 for f in blocking if f["icon"] == i) for i in ("🔴", "🟡", "🟢", "💡")}
yellow_correctness = sum(
1 for f in blocking if f["icon"] == "🟡" and (f.get("marker") or "").lower() == "correctness"
)
# --- always-human triggers ----------------------------------------------
attention = []
if state.get("destructive_migration"):
attention.append("Destructive/irreversible migration in the diff (DROP/RENAME) — needs a human signature.")
if state.get("touches_auth"):
attention.append("Authentication/authorization surface changed — needs a human signature.")
for flag in state.get("attention_flags") or []:
if isinstance(flag, str) and flag.strip():
attention.append(flag.strip())
placement = (state.get("placement_notes") or "").strip()
if placement:
attention.append("Placement question (design intent, never auto-resolved): " + placement)
# dedup, preserve order
seen = set()
attention = [a for a in attention if not (a.lower() in seen or seen.add(a.lower()))]
# --- pipeline-fault accounting (fail closed) -----------------------------
# PREFIX-anchored: a report that merely QUOTES a marker mid-text must not
# trip it. Any fault forces NEEDS-HUMAN.
faults = []
changed = state.get("changed_files") or []
reports = state.get("domain_reports") or []
faulted = sum(
1 for r in reports if isinstance(r, str) and r.lstrip().startswith("> ⚠️ PIPELINE-FAULT:")
)
if faulted:
faults.append(
f"PIPELINE-FAULT: {faulted} of {len(reports)} domain review lane(s) "
"failed after retries — their slices were not reviewed"
)
verifier_raw = state.get("verifier_output")
if isinstance(verifier_raw, str) and verifier_raw.lstrip().startswith("Agent node failed:"):
faults.append(
"PIPELINE-FAULT: finding verification failed — findings render as unverified"
)
# finding-verifier completes "successfully" with faults INSIDE the payload:
# parse_fault's `pipeline-fault` sentinel entry, or a per-finding
# PIPELINE-FAULT note from verify_fault or verdict_gate's crash guard.
# Anchored on the sentinel id / note prefix only. Counted over the raw
# entries, not the id-indexed dict, so N crash-guard entries sharing the
# "unknown" id count as N.
fv_faults = [
v
for v in verifier_entries
if v.get("id") == "pipeline-fault"
or (isinstance(v.get("note"), str) and v["note"].lstrip().startswith("PIPELINE-FAULT:"))
]
if fv_faults:
first = fv_faults[0].get("note")
detail = first.strip().replace("\n", " ") if isinstance(first, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
faults.append(
f"PIPELINE-FAULT: finding verification degraded — {len(fv_faults)} verifier "
f"verdict(s) carry a pipeline fault ({detail})"
)
# Keyed off synth_failure + a non-empty diff, NEVER off findings == [] —
# a clean review legitimately has zero findings and stays MERGE-READY.
synth_failure = state.get("synth_failure")
if isinstance(synth_failure, str) and synth_failure.startswith("LLM node") and changed:
detail = synth_failure.strip().replace("\n", " ")
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
faults.append(f"PIPELINE-FAULT: synthesis failed — findings unavailable; {detail}")
if changed and not reports:
faults.append(
"PIPELINE-FAULT: the diff is non-empty but no domain reports were produced — "
"the review did not actually run; do not trust this verdict."
)
# faults go FIRST and uncapped so the display cap can never hide one;
# the trigger reason keys off the human-facing triggers alone
has_triggers = bool(attention)
attention = faults + attention[:5]
reasons = []
if counts["🔴"]:
reasons.append(f"{counts['🔴']} 🔴 CRITICAL finding(s)")
if yellow_correctness:
reasons.append(f"{yellow_correctness} 🟡 [correctness] finding(s) outside the deferred section")
if faults:
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
reasons.append("pipeline fault(s) recorded — degraded run")
if has_triggers:
reasons.append("always-human trigger(s) fired")
verdict = "NEEDS-HUMAN" if reasons else "MERGE-READY"
reason_line = "; ".join(reasons) if reasons else "no blocking findings and no always-human triggers"
print(
json.dumps(
{
"verdict_out": {
"verdict": verdict,
"reason": reason_line,
"counts": counts,
"deferred_count": len(deferred),
"dropped_count": len(dropped_titles),
"dropped_titles": dropped_titles,
"attention": attention,
"findings_final": final,
}
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — never crash into a silent verdict
print(
json.dumps(
{
"verdict_out": {
"verdict": "NEEDS-HUMAN",
# PIPELINE-FAULT prefix is load-bearing: render.py's
# empty-diff stub bypass anchors on it — without it a
# crashed gate on an empty diff collapses into the
# "No changes to review." stub.
"reason": f"PIPELINE-FAULT: verdict computation error: {e} — human review required",
"counts": {}, "deferred_count": 0, "dropped_count": 0,
"dropped_titles": [], "attention": [f"PIPELINE-FAULT: verdict script error: {e}"],
"findings_final": [],
}
}
)
)
assets/agents/deep-research/README.md
+67 -6
View File
@@ -22,11 +22,13 @@ agent, this is the file to read alongside the
## Workflow
17 nodes. Solid arrows are static `next` / `routes` edges declared in
23 nodes. Solid arrows are static `next` / `routes` edges declared in
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
`research_each_question` node spawns one `research_one_question` branch per
sub-question and joins them before continuing.
sub-question and joins them before continuing — and `fallback` edges
(labeled "fallback"), taken when a node fails outright (see *Fault paths*
below).
```mermaid
flowchart TD
@@ -38,18 +40,27 @@ flowchart TD
bootstrap_research --> plan
bootstrap_research --> knowledge_lookup
plan["plan<br/>llm + output_schema"] --> research_each_question
plan -. "fallback" .-> plan_fault{"plan_fault<br/>script"}
plan_fault --> end_fault
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
research_each_question[\research_each_question<br/>map/]
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
research_one_question -. "fallback" .-> question_fault{"question_fault<br/>script"}
research_each_question --> combine_findings
combine_findings{"combine_findings<br/>script"} --> vet_sources
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
vet_sources -. "fallback" .-> vet_fault{"vet_fault<br/>script"}
vet_fault --> critique
critique["critique<br/>llm"] --> reflexion_gate
critique -. "fallback" .-> critique_fault{"critique_fault<br/>script"}
critique_fault --> reflexion_gate
reflexion_gate{"reflexion_gate<br/>script"}
reflexion_gate -->|"PASS"| synthesize
reflexion_gate -->|"REVISE (budget left)"| research_each_question
reflexion_gate -->|"REVISE (budget spent)"| synthesize
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
synthesize -. "fallback" .-> synth_fault{"synth_fault<br/>script"}
synth_fault --> end_fault
verify_sources{"verify_sources<br/>script"} --> approve
approve{{"approve<br/>approval"}}
approve -->|"accept"| end_accepted
@@ -59,13 +70,14 @@ flowchart TD
end_accepted(["end_accepted<br/>report"])
end_rejected(["end_rejected"])
end_fault(["end_fault<br/>DEEP_RESEARCH FAILED"])
```
### Node-type breakdown
| Type | Nodes |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback` |
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback`, plus the fault markers `plan_fault`, `question_fault`, `vet_fault`, `critique_fault`, `synth_fault` |
| `llm` (tools: `[]`) | `plan`, `critique` |
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
| `rag` | `knowledge_lookup` — local corpus retrieval |
@@ -73,7 +85,7 @@ flowchart TD
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
| `input` | `ask_topic` |
| `approval` | `approve` |
| `end` | `end_accepted`, `end_rejected` |
| `end` | `end_accepted`, `end_rejected`, `end_fault` |
## Parallel execution
@@ -217,6 +229,46 @@ folds that text into `research_feedback` and loops back to
`settings.max_loop_iterations` (40) is the engine's infinite-loop
backstop: it caps the total visits to any single node.
## Fault paths
Every stage that can die outright has a `fallback:` edge to a small
fault-marker script — the dashed edges in the diagram. A fallback fires
after a node's retries are exhausted (`research_one_question` and
`synthesize` each get two attempts via `max_attempts: 2`); the engine
writes its failure text into the node's state key before routing, and
the marker normalizes that text into a `PIPELINE-FAULT: ...` entry in
`pipeline_faults`. The five routes:
- `plan -> plan_fault -> end_fault` — fail-closed: no plan means
nothing to research.
- `research_one_question -> question_fault` — branch-local, inside the
map branch: one dead research lane becomes a fault finding instead of
sinking the whole map.
- `vet_sources -> vet_fault -> critique` — degrade-visibly: vetting is
a quality lane, not the product; `source_assessment` becomes a
neutral "unvetted" note.
- `critique -> critique_fault -> reflexion_gate` — degrade-visibly: the
rewritten critique deliberately has no `VERDICT:` line, so the gate's
malformed-critique PASS default proceeds to synthesis.
- `synthesize -> synth_fault -> end_fault` — fail-closed: no report
means no product.
On the fail-closed routes, `end_fault` renders the
`DEEP_RESEARCH FAILED — PIPELINE-FAULT: ...` sentinel. On the
degrade-visibly routes, `verify_sources` folds the accumulated faults
into a `## Pipeline notes` section shown at the approval step and
appended to the accepted report (`{{report}}{{pipeline_notes}}`;
empty when there were no faults). Degraded output is never a silent
pass — every fault is visible to the reviewer or the caller — and some
end node always renders its sentinel, so the graph degrades instead of
dying.
All 11 scripts also carry a top-level crash guard: a script that
throws records a fault and fails forward where the flow allows it (a
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
`verify_sources` reports that sources were NOT checked) instead of
killing the run.
## Running
```sh
@@ -241,6 +293,9 @@ coyote -a deep-research # no prompt -> triggers ask_topic
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
request and reports which are unreachable, so the human reviewer
sees broken citations before approving.
- Fault paths degrade visibly, never silently: a `PIPELINE-FAULT`
can only ever surface as a failure sentinel or a `## Pipeline
notes` section, never as a clean report.
## Customizing
@@ -270,7 +325,7 @@ coyote -a deep-research # no prompt -> triggers ask_topic
```
assets/agents/deep-research/
graph.yaml - agent config + 17-node workflow
graph.yaml - agent config + 23-node workflow
tools.sh - classify_source custom tool
README.md - this file
knowledge/
@@ -279,9 +334,15 @@ assets/agents/deep-research/
scripts/
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
plan_fault.py - fail-closed fault marker: dead plan -> end_fault
question_fault.py - branch-local fault marker for a dead research lane
combine_findings.py - joins map output (question_findings) into findings
vet_fault.py - degrade-visibly fault marker: unvetted sources
critique_fault.py - degrade-visibly fault marker: skipped critique
reflexion_gate.py - _next: research_each_question (revise) or synthesize
verify_sources.py - HTTP HEAD on cited URLs / DOIs
synth_fault.py - fail-closed fault marker: no report -> end_fault
verify_sources.py - HTTP HEAD on cited URLs / DOIs; folds pipeline_faults
into the report's "## Pipeline notes" section
incorporate_feedback.py - _next: research_each_question, with user feedback
```
assets/agents/deep-research/graph.yaml
+94 -26
View File
@@ -13,7 +13,7 @@ description: |
node type (script, llm, rag, map, agent, input, approval, end) and
both static fan-out and dynamic map fan-out.
version: "1.0"
version: '1.0'
global_tools:
- web_search_coyote.sh
@@ -24,8 +24,8 @@ mcp_servers:
- ddg-search
conversation_starters:
- "How does HTTP/3 differ from HTTP/2?"
- "Summarize recent advances in solid-state battery chemistry"
- 'How does HTTP/3 differ from HTTP/2?'
- 'Summarize recent advances in solid-state battery chemistry'
settings:
max_loop_iterations: 40
@@ -34,15 +34,22 @@ settings:
max_concurrency: 4
initial_state:
research_feedback: ""
research_feedback: ''
research_attempts: 0
local_context: ""
local_sources: ""
local_context: ''
local_sources: ''
# Fault-path defaults. `questions: []` also keeps the
# knowledge_lookup lane alive for one super-step when `plan` dies inside the
# static fan-out: research_each_question maps over zero items while the
# plan_fault → end_fault route renders the failure sentinel.
pipeline_faults: []
pipeline_notes: ''
fault_text: ''
questions: []
start: parse_request
nodes:
parse_request:
id: parse_request
type: script
@@ -52,10 +59,10 @@ nodes:
ask_topic:
id: ask_topic
type: input
question: "What would you like me to research?"
validation: "len(input) > 0"
question: 'What would you like me to research?'
validation: 'len(input) > 0'
state_updates:
topic: "{{input}}"
topic: '{{input}}'
next: bootstrap_research
bootstrap_research:
@@ -77,7 +84,7 @@ nodes:
are precise, self-contained queries (each one is sent on its own
to a separate research worker, so they must be answerable
without each other's context).
prompt: "Research topic: {{topic}}"
prompt: 'Research topic: {{topic}}'
tools: []
output_schema:
type: object
@@ -92,29 +99,44 @@ nodes:
maxItems: 6
description: 3-5 specific, self-contained sub-questions.
required: [research_plan, questions]
# On success plan_failure holds the node's structured JSON output; on
# failure the engine writes "LLM node ...failed: <chain>" into it BEFORE
# routing to the fallback (schema properties are only merged on success).
state_updates:
plan_failure: '{{output}}'
fallback: plan_fault
next: research_each_question
plan_fault:
id: plan_fault
type: script
description: Fail-closed marker for a dead planning stage (reached only via plan's fallback) — normalizes the failure text captured in plan_failure into a PIPELINE-FAULT and continues to end_fault; no plan means nothing to research, so the DEEP_RESEARCH FAILED sentinel always renders. The sibling knowledge_lookup lane survives one more super-step by mapping over the empty initial_state `questions` list.
script: scripts/plan_fault.py
fallback: end_fault
next: end_fault
knowledge_lookup:
id: knowledge_lookup
type: rag
documents:
- ./knowledge/
query: "{{topic}}"
query: '{{topic}}'
top_k: 6
chunk_size: 1000
chunk_overlap: 100
state_updates:
local_context: "{{output.context}}"
local_sources: "{{output.sources}}"
local_context: '{{output.context}}'
local_sources: '{{output.sources}}'
next: research_each_question
research_each_question:
id: research_each_question
type: map
over: "{{questions}}"
over: '{{questions}}'
as: question
branch: research_one_question
collect_into: question_findings
output_key: finding
max_concurrency: 3
next: combine_findings
@@ -151,7 +173,18 @@ nodes:
- mcp:ddg-search
max_iterations: 10
max_attempts: 2
temperature: 0.1
state_updates:
finding: '{{output}}'
# On failure the engine writes "LLM node ...failed: <chain>" into `finding`
# via the state_updates above BEFORE routing to the fallback. The marker
# is branch-local: a map branch's fallback must stay inside the branch.
fallback: question_fault
question_fault:
id: question_fault
type: script
description: Fail-closed marker for a dead research lane (reached only via research_one_question's fallback route inside the map branch). Normalizes the engine's failure text in `finding` into a PIPELINE-FAULT finding the map collects — one dead lane never sinks the whole map. combine_findings later lifts PIPELINE-FAULT findings into pipeline_faults for the report's Pipeline notes.
script: scripts/question_fault.py
combine_findings:
id: combine_findings
@@ -177,7 +210,16 @@ nodes:
- classify_source
max_iterations: 15
state_updates:
source_assessment: "{{output}}"
source_assessment: '{{output}}'
fallback: vet_fault
next: critique
vet_fault:
id: vet_fault
type: script
description: Degrade-visibly marker for a dead vetting stage (reached only via vet_sources' fallback) — vetting is a quality lane, not the product. Records a PIPELINE-FAULT, rewrites source_assessment to a neutral "unvetted" note, and continues to critique, the stage vet_sources' own next pointed to.
script: scripts/vet_fault.py
fallback: critique
next: critique
critique:
@@ -217,7 +259,16 @@ nodes:
{{source_assessment}}
tools: []
state_updates:
critique: "{{output}}"
critique: '{{output}}'
fallback: critique_fault
next: reflexion_gate
critique_fault:
id: critique_fault
type: script
description: Degrade-visibly marker for a dead critique stage (reached only via critique's fallback) — critique is a quality lane, not the product. Records a PIPELINE-FAULT and rewrites `critique` to a neutral note with deliberately NO "VERDICT:" line, so reflexion_gate's malformed-critique PASS default proceeds to synthesis; then continues to reflexion_gate, the stage critique's own next pointed to.
script: scripts/critique_fault.py
fallback: reflexion_gate
next: reflexion_gate
reflexion_gate:
@@ -241,10 +292,22 @@ nodes:
Produce the final report following your instructions.
timeout: 300
max_attempts: 2
state_updates:
report: "{{output}}"
report: '{{output}}'
# On failure the engine writes "Agent node failed: <chain>" into `report`
# via the state_updates above BEFORE routing to the fallback.
fallback: synth_fault
next: verify_sources
synth_fault:
id: synth_fault
type: script
description: Fail-closed marker for a dead synthesis stage (reached only via synthesize's fallback) — no report means no product. Normalizes the "Agent node failed:" text captured in `report` into a PIPELINE-FAULT and continues to end_fault, which renders the DEEP_RESEARCH FAILED sentinel.
script: scripts/synth_fault.py
fallback: end_fault
next: end_fault
verify_sources:
id: verify_sources
type: script
@@ -260,20 +323,20 @@ nodes:
{{report}}
----
{{source_check}}
{{source_check}}{{pipeline_notes}}
----
Accept this report? Pick "accept" or "reject", or type specific
feedback to send the research back for another pass.
options:
- "accept"
- "reject"
- 'accept'
- 'reject'
routes:
"accept": end_accepted
"reject": end_rejected
'accept': end_accepted
'reject': end_rejected
on_other: incorporate_feedback
state_updates:
decision: "{{choice}}"
decision: '{{choice}}'
incorporate_feedback:
id: incorporate_feedback
@@ -283,9 +346,14 @@ nodes:
end_accepted:
id: end_accepted
type: end
output: "{{report}}"
output: '{{report}}{{pipeline_notes}}'
end_rejected:
id: end_rejected
type: end
output: "Research on '{{topic}}' was rejected and discarded."
end_fault:
id: end_fault
type: end
output: 'DEEP_RESEARCH FAILED — PIPELINE-FAULT: {{fault_text}}'
assets/agents/deep-research/scripts/bootstrap_research.py
+9 -1
View File
@@ -6,6 +6,12 @@ list on this node fans out into two parallel branches (the LLM planner and
the RAG knowledge lookup) as a single super-step. The validator requires
declared parallel-branch script outputs, so we emit an empty JSON object
explicitly here.
Fail-safe: the node reads no state and computes nothing, so its
happy-path output IS the sane degraded output — the guard emits the same
empty object on any crash. No PIPELINE-FAULT is recorded: appending one
would require reading state (the very thing that crashed) or clobbering
faults recorded upstream by parse_request, and a crash here loses nothing.
"""
import json
@@ -14,5 +20,7 @@ def main():
print(json.dumps({}))
if __name__ == "__main__":
try:
main()
except Exception: # noqa: BLE001 — the fan-out source must never crash the graph
print(json.dumps({}))
assets/agents/deep-research/scripts/combine_findings.py
+33 -2
View File
@@ -6,6 +6,18 @@ one entry per sub-question, in input order). Downstream nodes
(`vet_sources`, `critique`, `synthesize`) read `{{findings}}` as a
single block, so this script renders the array as a Markdown document
with one section per question.
Fault visibility: a research lane that died lands here as a finding
starting with "PIPELINE-FAULT:" (written by question_fault inside the map
branch, where `pipeline_faults` itself is out of reach — map branches only
surface their collected output_key). This script lifts such findings into
`pipeline_faults` (deduplicated, since a reflexion or feedback loop can
re-run the map) so dead lanes surface in the report's Pipeline notes.
Fail-safe: on a crash (e.g. malformed GRAPH_STATE), `findings`
becomes the fault text itself and a PIPELINE-FAULT entry is recorded —
the pipeline proceeds visibly degraded, and the human approval gate is
the backstop. Existing faults are preserved best-effort.
"""
import json
import os
@@ -19,21 +31,40 @@ def load_state():
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def main():
state = load_state()
questions = state.get("questions") or []
per_question = state.get("question_findings") or []
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
sections = []
for idx, q in enumerate(questions):
body = per_question[idx] if idx < len(per_question) else ""
if isinstance(body, dict) or isinstance(body, list):
body = json.dumps(body, indent=2)
if isinstance(body, str) and body.startswith("PIPELINE-FAULT:") and body not in faults:
faults.append(body)
sections.append(f"## {q}\n\n{body}")
findings = "\n\n".join(sections) if sections else "No findings gathered."
print(json.dumps({"findings": findings}))
print(json.dumps({"findings": findings, "pipeline_faults": faults}))
if __name__ == "__main__":
try:
main()
except Exception as e: # noqa: BLE001 — a crashed join must degrade, not die
faults = safe_faults()
fault = (
f"PIPELINE-FAULT: combining findings crashed — {e}; per-question "
"findings may exist in state but could not be joined"
)
faults.append(fault)
print(json.dumps({"findings": fault, "pipeline_faults": faults}))
assets/agents/deep-research/scripts/critique_fault.py
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Degrade-visibly marker for a dead critique stage of deep-research.
Reached ONLY via critique's `fallback:` route. Grounded in engine behavior:
on llm-node failure the engine writes "LLM node …failed: <chain>" into
`critique` via the node's state_updates BEFORE routing to the fallback.
Critique is a quality lane, not the product, so this path degrades instead
of failing closed: it records a PIPELINE-FAULT (surfaced later in the
report's Pipeline notes) and rewrites `critique` to a neutral note, then
continues to `reflexion_gate` — the stage critique's own `next` pointed to.
The rewritten note deliberately contains NO "VERDICT:" line:
reflexion_gate's parse_verdict defaults to PASS when no verdict line is
found (its documented malformed-critique behavior), so a dead critique
proceeds to synthesis instead of burning the revision budget on a fault.
"""
import json
import os
MAX_DETAIL_CHARS = 500
CRITIQUE_NOTE = (
"Critique unavailable — the critique stage failed (pipeline fault); "
"proceeding to synthesis without an automated review pass."
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def main():
state = load_state()
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
raw = state.get("critique")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the critique stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
fault = f"PIPELINE-FAULT: critique failed — {detail}"
if fault not in faults:
faults.append(fault)
print(json.dumps({"pipeline_faults": faults, "critique": CRITIQUE_NOTE}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
faults = safe_faults()
faults.append(f"PIPELINE-FAULT: critique failed — fault-marker script error: {e}")
print(
json.dumps(
{
"pipeline_faults": faults,
"critique": CRITIQUE_NOTE,
}
)
)
assets/agents/deep-research/scripts/incorporate_feedback.py
+37 -1
View File
@@ -10,6 +10,13 @@ reflexion counter is reset so the user-driven pass gets a fresh revision
budget.
Routing (`_next`): always research_each_question.
Fail-safe: a crash here means the user's feedback text cannot be
recovered, but their INTENT — another research pass — is unambiguous
(they chose neither accept nor reject). The degraded route therefore
still loops to `research_each_question`, with a generic feedback note
naming the loss and a PIPELINE-FAULT entry (best-effort preserving
existing faults) that surfaces in the next approval's Pipeline notes.
"""
import json
import os
@@ -23,6 +30,14 @@ def load_state():
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def main():
state = load_state()
feedback = (state.get("decision") or "").strip()
@@ -37,5 +52,26 @@ def main():
print(json.dumps(output))
if __name__ == "__main__":
try:
main()
except Exception as e: # noqa: BLE001 — a crashed feedback fold must degrade, not die
faults = safe_faults()
faults.append(
f"PIPELINE-FAULT: feedback incorporation crashed — {e}; the reviewer's "
"feedback text could not be recovered"
)
print(
json.dumps(
{
"_next": "research_each_question",
"research_attempts": 0,
"research_feedback": (
"The user reviewed the report and asked for changes, but "
"the feedback text could not be recovered (pipeline "
"fault). Re-run the research pass with extra care and "
"note this loss."
),
"pipeline_faults": faults,
}
)
)
assets/agents/deep-research/scripts/parse_request.py
+20 -1
View File
@@ -9,6 +9,13 @@ supply one interactively.
Routing (`_next`):
- prompt present -> (no _next; static next: plan)
- prompt empty -> ask_topic
Fail-safe: a crash here (e.g. malformed GRAPH_STATE) means the
caller's prompt cannot be read at all, so proceeding silently is
meaningless — the degraded route is `ask_topic`, which asks the user for
the topic directly, plus a PIPELINE-FAULT entry so the loss is visible in
the final report's Pipeline notes. Clobbering `pipeline_faults` is safe
here: this is the start node, so the list can only hold its initial [].
"""
import json
import os
@@ -31,5 +38,17 @@ def main():
print(json.dumps({"_next": "ask_topic"}))
if __name__ == "__main__":
try:
main()
except Exception as e: # noqa: BLE001 — a crashed entry router must degrade, not die
print(
json.dumps(
{
"_next": "ask_topic",
"pipeline_faults": [
f"PIPELINE-FAULT: request parsing crashed — {e}; asked the "
"user for the research topic instead"
],
}
)
)
assets/agents/deep-research/scripts/plan_fault.py
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead planning stage of the deep-research graph.
Reached ONLY via plan's `fallback:` route. Grounded in engine behavior: on
llm-node failure the engine writes "LLM node …failed: <chain>" into
`plan_failure` via the node's state_updates BEFORE routing to the fallback
(on success the same key holds the node's structured JSON output, which
never starts with the "LLM node" prefix). No plan means no sub-questions —
there is nothing to research — so this path is fail-closed: record the
PIPELINE-FAULT and continue to `end_fault`, which renders the
DEEP_RESEARCH FAILED sentinel.
The planner runs inside the static fan-out [plan, knowledge_lookup]. While
this marker routes to end_fault, the sibling branch proceeds one more
super-step into `research_each_question`, whose `{{questions}}` resolves to
the initial_state [] — the doomed lane maps over zero items and the graph
returns end_fault's output at the next join.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
raw = state.get("plan_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the planning stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
fault_text = f"planning failed — nothing to research: {detail}"
faults.append(f"PIPELINE-FAULT: {fault_text}")
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
fault_text = f"planning failed — fault-marker script error: {e}"
print(
json.dumps(
{
"fault_text": fault_text,
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
}
)
)
assets/agents/deep-research/scripts/question_fault.py
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead research lane of the deep-research graph.
Reached ONLY via research_one_question's `fallback:` route inside the
`research_each_question` map branch (branch-local — a map branch's fallback
must stay inside the branch subgraph). Grounded in engine behavior: on
llm-node failure the engine writes "LLM node …failed: <chain>" into
`finding` via the node's state_updates BEFORE routing to the fallback (on
success the same key holds the researched findings, which never start with
the "LLM node" prefix). This node normalizes the failure text into a
PIPELINE-FAULT finding and the chain ends here — the map collects the fault
as the lane's `finding`, so one dead research lane never sinks the whole
map. combine_findings later lifts PIPELINE-FAULT findings into
`pipeline_faults` so the fault also surfaces in the report's Pipeline notes.
The question text is part of the fault so distinct dead lanes with identical
engine failure text do not collapse into one entry there.
"""
import json
import os
MAX_DETAIL_CHARS = 500
MAX_QUESTION_CHARS = 200
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def fault_finding(question, detail):
return {
"finding": (
"PIPELINE-FAULT: question research failed — finding unavailable — "
f'{detail} (question: "{question}")'
)
}
def question_label(state):
raw = state.get("question")
question = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not question:
return "(unknown question)"
if len(question) > MAX_QUESTION_CHARS:
question = question[: MAX_QUESTION_CHARS - 1] + "…"
return question
def main():
state = load_state()
raw = state.get("finding")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the research lane died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(json.dumps(fault_finding(question_label(state), detail)))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
fault_finding("(unknown question)", f"fault-marker script error: {e}")
)
)
assets/agents/deep-research/scripts/reflexion_gate.py
+22 -1
View File
@@ -15,6 +15,12 @@ Routing (`_next`):
Reflexion is a best-effort quality booster, not a hard gate: once the
budget is spent the workflow proceeds anyway, and the human approval
step is the final backstop.
Fail-safe: a crash here fails FORWARD to `synthesize` — the same
direction as the existing malformed-critique PASS default — with a
PIPELINE-FAULT entry recorded (best-effort preserving existing faults),
so a broken gate costs the run its automated critique routing, never the
run itself.
"""
import json
import os
@@ -33,6 +39,14 @@ def load_state():
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def as_int(value, default=0):
try:
return int(value)
@@ -72,5 +86,12 @@ def main():
print(json.dumps(output))
if __name__ == "__main__":
try:
main()
except Exception as e: # noqa: BLE001 — a crashed gate fails forward, never dies
faults = safe_faults()
faults.append(
f"PIPELINE-FAULT: reflexion gate crashed — {e}; proceeded to synthesis "
"without automated critique routing"
)
print(json.dumps({"_next": "synthesize", "pipeline_faults": faults}))
assets/agents/deep-research/scripts/synth_fault.py
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead synthesis stage of the deep-research graph.
Reached ONLY via synthesize's `fallback:` route. Grounded in engine
behavior: on agent-node failure the engine writes "Agent node failed:
{e:#}" into `report` via the node's state_updates BEFORE routing to the
fallback (on success the same key holds the report-writer's report, which
never starts with that prefix). No report means no product, so this path is
fail-closed: normalize the failure text, record the PIPELINE-FAULT, and
continue to `end_fault`, which renders the DEEP_RESEARCH FAILED sentinel.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
raw = state.get("report")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("Agent node failed:"):
detail = "the synthesis stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
fault_text = f"report synthesis failed — no report was produced: {detail}"
faults.append(f"PIPELINE-FAULT: {fault_text}")
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
fault_text = f"report synthesis failed — fault-marker script error: {e}"
print(
json.dumps(
{
"fault_text": fault_text,
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
}
)
)
assets/agents/deep-research/scripts/verify_sources.py
+49 -2
View File
@@ -6,6 +6,19 @@ request, and writes a `source_check` summary into state so the human
reviewer sees broken citations at the approval step.
Times out per request so a slow source cannot stall the graph.
Pipeline-notes fold: end-node templates render unconditionally, so the
conditional "## Pipeline notes" section of the accepted report is built
HERE — the last stop before the approval gate on every accepted path,
including reflexion and feedback re-loops. `pipeline_notes` is the
rendered section when `pipeline_faults` is non-empty and the empty
string otherwise, keeping the happy-path `{{report}}{{pipeline_notes}}`
output byte-identical to the pre-hardening `{{report}}`.
Fail-safe: per-URL probes were already guarded; the top level now
is too. On a crash the summary says sources were NOT checked, a
PIPELINE-FAULT is recorded (best-effort preserving existing faults), and
pipeline_notes still renders — the approval gate remains the backstop.
"""
import json
import os
@@ -25,6 +38,21 @@ def load_state():
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def render_notes(faults):
entries = [f for f in (faults or []) if isinstance(f, str) and f.strip()]
if not entries:
return ""
return "\n\n## Pipeline notes\n\n" + "\n".join(f"- {e}" for e in entries)
def reachable(url, timeout=5.0):
req = urllib.request.Request(url, method="HEAD")
try:
@@ -62,8 +90,27 @@ def main():
+ "\n".join(results)
)
print(json.dumps({"source_check": summary}))
notes = render_notes(state.get("pipeline_faults"))
print(json.dumps({"source_check": summary, "pipeline_notes": notes}))
if __name__ == "__main__":
try:
main()
except Exception as e: # noqa: BLE001 — a crashed source check must degrade, not die
faults = safe_faults()
faults.append(
f"PIPELINE-FAULT: source verification crashed — {e}; cited sources "
"were not checked for reachability"
)
print(
json.dumps(
{
"source_check": (
"Source verification crashed (pipeline fault) — cited "
"sources were NOT checked for reachability."
),
"pipeline_notes": render_notes(faults),
"pipeline_faults": faults,
}
)
)
assets/agents/deep-research/scripts/vet_fault.py
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Degrade-visibly marker for a dead source-vetting stage of deep-research.
Reached ONLY via vet_sources' `fallback:` route. Grounded in engine
behavior: on llm-node failure the engine writes "LLM node …failed: <chain>"
into `source_assessment` via the node's state_updates BEFORE routing to the
fallback. Vetting is a quality lane, not the product, so this path degrades
instead of failing closed: it records a PIPELINE-FAULT (surfaced later in
the report's Pipeline notes) and rewrites `source_assessment` to a neutral
"unvetted" note so downstream prompts (critique, synthesize) never carry
raw engine error text, then continues to `critique` — the stage
vet_sources' own `next` pointed to.
"""
import json
import os
MAX_DETAIL_CHARS = 500
UNVETTED_NOTE = (
"Source credibility assessment unavailable — the vetting stage failed "
"(pipeline fault); treat every cited source as unvetted and needing "
"corroboration. Do not request revision solely because this assessment "
"is unavailable — re-research cannot restore it; judge the findings on "
"their own evidence."
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def safe_faults():
try:
state = load_state()
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
except Exception: # noqa: BLE001 — best-effort preservation only
return []
def main():
state = load_state()
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
raw = state.get("source_assessment")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the vetting stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
fault = f"PIPELINE-FAULT: source vetting failed — {detail}"
if fault not in faults:
faults.append(fault)
print(json.dumps({"pipeline_faults": faults, "source_assessment": UNVETTED_NOTE}))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
faults = safe_faults()
faults.append(f"PIPELINE-FAULT: source vetting failed — fault-marker script error: {e}")
print(
json.dumps(
{
"pipeline_faults": faults,
"source_assessment": UNVETTED_NOTE,
}
)
)
assets/agents/domain-reviewer/README.md
+43
View File
@@ -0,0 +1,43 @@
# Domain Reviewer
A **domain-slice review leader**. Where `file-reviewer` reviews one file's diff
in isolation, domain-reviewer reviews a group of *related* changed files — the
DB migration plus the queries that use it, the routes plus their handlers — as
ONE coherent unit. It runs as a map branch of the `code-reviewer` graph: one
leader per domain, in parallel, with peer messaging between leaders.
## Why it exists
The strongest recurring review misses are **cross-file-within-domain** defects:
seed data drifting from canonical enums, a guard fixed in one of N twin paths,
a migration renaming a column the query layer still uses. Per-file reviewers
each hold one fragment of those bugs and only connect them if sibling messaging
happens to fire. A domain leader holds the whole slice in one context — the
connection is the default, not a lucky message.
## How it works
1. Fetches its slice's diff itself (`get_diff --files <its files>`).
2. **Sizing**: a small slice (≤3 files, <400 changed lines) is reviewed
directly with the full `code-review` skill stack; a larger one fans out one
`file-reviewer` per file (classic delegation-protocol prompts + sibling
roster) and the leader synthesizes.
3. **The domain pass** (always, its unique value): guard symmetry across the
slice, literal/constant drift between files, schema↔usage mismatch, and the
mandatory blast-radius / test-adequacy / removed-guards checks at slice
level — reported even when clean.
4. **Leader-to-leader messaging**: the code-reviewer graph runs leaders with
`teammates: true`, so the DB leader can tell the API leader "the migration
renames `status` → `state`, check your serializers" mid-review.
5. Emits a machine-checkable report (`## Domain:` / `### Findings` with a
`File` field per finding / `### Mandatory checks` / `### Cross-Domain
Concerns` / `DOMAIN_REVIEW_COMPLETE`) — the graph's completeness gate
rejects a report missing required sections exactly once, then flags it.
## Notes
- Read-only; severities per the standard 🔴/🟡/🟢/💡 mapping; never rewrites a
file-reviewer's severity; dedups identical findings (aspect skill's copy wins).
- Normally spawned by the `code-reviewer` graph, not directly — but a direct
spawn works: give it a domain name, a file list, a diff spec, and the quality
bar in the prompt.
assets/agents/domain-reviewer/config.yaml
+166
View File
@@ -0,0 +1,166 @@
name: domain-reviewer
description: Domain-slice review leader - reviews a group of related changed files as ONE coherent unit (cross-file-within-domain checks are first-class), delegating per-file depth to file-reviewer when the slice is large. Runs as a map branch of the code-reviewer graph with teammate messaging to the other domain leaders.
version: 1.0.0
auto_continue: true
max_auto_continues: 20
inject_todo_instructions: false
skills_enabled: true
enabled_skills:
- code-review
- ai-slop-remover
- transactional-integrity
- logging-discipline
- rest-api-review
- cli-review
- library-review
- worker-review
- iac-review
- migration-review
- cicd-review
spawnable_agents:
- file-reviewer
max_concurrent_agents: 8
# The limit is consulted by the SPAWNER against the child's absolute depth
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
# review-gauntlet | step-runner(2) → code-reviewer(3) → domain-reviewer(4) →
# file-reviewer(5). This agent spawns depth-5 children; 6 = one level of headroom.
# NOTE: can_spawn_agents is not yet set, so the file-reviewer hop is documented
# but inert; 6 keeps the limit forward-compatible for when it is wired.
max_agent_depth: 6
variables:
- name: project_dir
description: Project directory containing the changes under review
default: '.'
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools:
- fs_read.sh
- fs_grep.sh
- fs_glob.sh
- fs_cat.sh
- fs_ls.sh
instructions: |
You are a DOMAIN REVIEW LEADER. You review one domain's slice of a code change — a group of
related files (the DB migration plus the queries that use it; the routes plus their handlers) —
as ONE coherent unit. Your unique value over per-file review is the space BETWEEN the files:
cross-file-within-domain defects that no single-file reviewer can see. You either review the
slice directly or lead a team of `file-reviewer` agents through it, and you always add the
domain-level pass on top.
Your spawn prompt IS your entire context: an ASSIGNMENT (JSON with `domain` and `files`), a diff
spec, the quality bar (rigor/surfaces + provenance), and the shared context pack (intent,
conventions, review-miss patterns, repo-history signals, linter output).
## Step 0: Load review skills
Before reading any code, `skill__load` `code-review` and `ai-slop-remover`. Additionally load
`transactional-integrity` when the slice touches state-changing code (DB writes, transactions,
queue/webhook/job handlers, retries, external side effects) and `logging-discipline` when it
touches boundaries, error paths, background jobs, or state transitions. Load the surface skill(s)
for your domain per this table (load a row's skill when your files match that surface's trigger
AND the surface is in the resolved surfaces list; when provenance is `detected-default`, a
trigger match alone suffices):
| declared surface | skill loaded |
|---|---|
| `rest-api` (and `grpc`/`graphql` aliases) | `rest-api-review` |
| `cli` | `cli-review` |
| `library` | `library-review` |
| `worker` | `worker-review` |
| `iac` | `iac-review` |
| `db-migration` | `migration-review` |
| `ci-cd` | `cicd-review` |
## Workflow
1. **Fetch your slice's diff**: `get_diff --files <your files, comma-separated>` (add
`--base <ref-or-range>` when the diff spec names one; omit it for staged/unstaged review).
2. **Choose depth**:
- **Direct** (slice is ≤3 files AND <400 changed lines): review each file's diff yourself,
applying the loaded skill checklists exactly as a file-reviewer would.
- **Delegate** (bigger): spawn one `file-reviewer` per file IN PARALLEL, each with the full
TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT structure — pass through the
context pack fields from YOUR prompt verbatim (intent, conventions, review-miss patterns,
rigor/surfaces + provenance, this file's linter output) and paste each file's diff hunks
inline (the reviewer cannot see your context). Group tiny files (<20 changed lines) into a
shared reviewer when the slice exceeds ~8 files. After spawning ALL of them, send each the
sibling roster (`SIBLING_ROSTER:` with ids + file assignments) so they can exchange
cross-cutting alerts. Then collect results — do not poll.
3. **The domain pass** (ALWAYS yours, direct or delegated — this is why you exist). Check the
SLICE AS A WHOLE for:
- **Guard symmetry**: a guard/validation/limit changed in one file — grep the slice (and the
repo) for its siblings (same interface, same job kind, same naming family); a fix applied
to one of N twins is N-1 latent bugs.
- **Literal/constant drift**: values that must agree across the slice's files (enum literals
vs seed data, config keys vs readers, error codes vs handlers) — quote both sides when they
disagree.
- **Schema ↔ usage mismatch**: migrations vs the queries/models that use the schema; proto
definitions vs marshaling code; API shapes vs client calls within the slice.
- **Blast radius, test adequacy, and removed guards at slice level** (the mandatory checks
from `code-review`) — REPORT them under `### Mandatory checks` even when clean; silence
reads as not done.
4. **Talk to the other domain leaders** (you run alongside them with peer messaging armed):
check your inbox before finalizing, and send a leader a 1-2 sentence alert when your slice
changes something their domain consumes (a renamed column, a changed route shape, a moved
constant). Messages move facts sideways; they never replace your report.
5. **Synthesize the domain report** in the exact output format below.
## Output Format
Structure your final response EXACTLY as:
```
## Domain: <domain name>
### Summary
<2-3 sentences: what this slice changes as a unit>
### Findings
#### <finding title (optional [convention]/[correctness] marker per the loaded skills)>
- **Severity**: 🔴 CRITICAL | 🟡 WARNING | 🟢 SUGGESTION | 💡 NITPICK
- **File**: <path>
- **Lines**: <start>-<end>
- **Description**: <clear explanation of the issue>
- **Suggestion**: <how to fix it>
#### <next finding>
...
### Mandatory checks
- Blast radius: <result, or "clean — callers of the changed symbols checked: <list>">
- Test adequacy: <result, or "clean — behavior changes covered by: <tests>">
- Guard symmetry: <result, or "clean — no guard changes in this slice" / "twins checked: <list>">
### Cross-Domain Concerns
<alerts you sent or received and concerns affecting other domains, or "None">
DOMAIN_REVIEW_COMPLETE
```
## Rules
1. **You are read-only.** Never modify files.
2. **Every finding carries a File field** — your report covers multiple files and the synthesis
is machine-assisted; a finding without a file is unroutable.
3. **Severity tags per the standard mapping** (🔴 correctness/security/data-loss; 🟡 logic
errors, races, missing error handling; 🟢 clarity/coupling/missing tests; 💡 style/nitpicks).
Preserve your file-reviewers' severities verbatim; dedup identical findings (keep the aspect
skill's copy).
4. **Include ALL findings** — yours and your file-reviewers' — never summarize any away.
5. **Mandatory checks report even when clean.**
6. **End with DOMAIN_REVIEW_COMPLETE** on its own line.
## Context
- Project: {{project_dir}}
- CWD: {{__cwd__}}
## Available Tools
{{__tools__}}
assets/agents/domain-reviewer/tools.sh
Executable
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
set -eo pipefail
# @env LLM_OUTPUT=/dev/stdout
# @env LLM_AGENT_VAR_PROJECT_DIR=.
# @describe Domain-slice reviewer tools
_project_dir() {
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
}
# @cmd Get the git diff for this domain's file slice. Defaults to staged changes, falls back to unstaged, then HEAD~1. Pass --base for a ref or range (e.g. "main...HEAD").
# @option --base Optional base ref or range to diff against (e.g. "main...HEAD", "HEAD~3", a SHA)
# @option --files Comma-separated list of file paths to limit the diff to (your assignment's files)
get_diff() {
local project_dir
project_dir=$(_project_dir)
# shellcheck disable=SC2154
local base="${argc_base:-}"
# shellcheck disable=SC2154
local files_csv="${argc_files:-}"
local -a path_args=()
if [[ -n "${files_csv}" ]]; then
local IFS=','
local f
for f in ${files_csv}; do
f="$(echo "${f}" | sed -e 's/^ *//' -e 's/ *$//')"
[[ -n "${f}" ]] && path_args+=("${f}")
done
fi
_diff() {
if ((${#path_args[@]})); then
(cd "${project_dir}" && git diff "$@" -- "${path_args[@]}" 2>&1) || true
else
(cd "${project_dir}" && git diff "$@" 2>&1) || true
fi
}
local diff_output=""
if [[ -n "${base}" ]]; then
diff_output=$(_diff "${base}")
else
diff_output=$(_diff --cached)
[[ -z "${diff_output}" ]] && diff_output=$(_diff)
[[ -z "${diff_output}" ]] && diff_output=$(_diff HEAD~1)
fi
if [[ -z "${diff_output}" ]]; then
echo "No changes found for the requested files in ${project_dir}." >> "$LLM_OUTPUT"
return 0
fi
local file_count
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
{
echo "Diff contains changes to ${file_count} file(s):"
echo ""
echo "${diff_output}"
} >> "$LLM_OUTPUT"
}
assets/agents/file-reviewer/config.yaml
+4 -2
View File
@@ -1,6 +1,6 @@
name: file-reviewer
description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns
version: 2.3.0
version: 2.4.0
skills_enabled: true
enabled_skills:
@@ -85,6 +85,7 @@ instructions: |
The `code-review` skill teaches the investigation workflow. Apply these per-review caps on top:
- **Max 5 fs_read calls per review.** Be deliberate about which files you read.
- **`fs_grep`/`fs_glob` are NOT capped** — the skill's blast-radius caller sweep and test sweep run on greps; spend them freely and reserve reads for suspicious callers and enclosing scopes.
- **`fs_read` returns a TRUNCATED view** with line numbers (long lines cut at 2000 chars, output capped at 2000 lines by default). Use `--offset` and `--limit` (default 50 lines of context) to target specific sections. Never read entire large files.
- **Use `fs_cat` only when you genuinely need the full untruncated file** — for a diff review this should be rare; `fs_grep` + targeted `fs_read` usually answers the question with less context.
- **Focus on the diff.** Read surrounding code only when needed to evaluate the change; do not audit unrelated code in the same file.
@@ -134,7 +135,8 @@ instructions: |
1. **Be specific.** Reference exact line numbers and code.
2. **Be actionable.** Every finding must have a suggestion.
3. **Never modify files.** You are read-only.
4. **Always end with REVIEW_COMPLETE.**
4. **Mandatory checks report even when clean.** The `code-review` skill's blast-radius sweep, test-adequacy mapping, and removed-guard accounting each produce a one-line result in your Summary or Findings (`Blast radius: N callers checked, compatible` / `Tests: <behavior> covered by <test>` / `Removed guards: none` or the accounting). Silence on any of the three reads as "not done".
5. **Always end with REVIEW_COMPLETE.**
## Context
- Project: {{project_dir}}
assets/agents/finding-verifier/README.md
+73
View File
@@ -0,0 +1,73 @@
# finding-verifier
A **graph** agent that independently verifies code-review findings against the working
tree. It exists because false positives kill hands-off trust faster than missed bugs do —
and because an orchestrator verifying its own workers' findings inline can skip, fudge,
or confirmation-bias the check. A graph can't: the structure IS the discipline.
## How it works
```mermaid
flowchart TD
A["parse (llm, max_attempts 2)"] --> B["verify_each (map, max_concurrency 4): one branch per finding"]
A -. "fallback" .-> PF["parse_fault (script): verdicts = one UNVERIFIABLE PIPELINE-FAULT entry"]
PF --> D
B -. "spawns × N" .-> C["verify_one (llm + fs tools, max_attempts 2)"]
C --> G["verdict_gate (script): machine-validates the JSON contract"]
G -. "reject-retry once" .-> C
C -. "fallback" .-> VF["verify_fault (script): branch-local UNVERIFIABLE marker"]
B --> D(["done (end): FINDING_VERIFIER_RESULTS"])
```
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
Gets two attempts (`max_attempts: 2`); if both fail, its `fallback` routes to
`parse_fault`, which rewrites `verdicts` to a single UNVERIFIABLE entry naming the fault
and jumps straight to `done`.
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
finding is structurally impossible; results collect in input order.
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
and answers ONE narrow question: does the cited code exist, and does it behave as the
claim says? Verdicts:
- `VERIFIED` — with the decisive line(s) quoted verbatim (`path:line`).
- `FALSE` — the code doesn't exist anywhere or provably doesn't do what the claim says.
Requires PROOF, not disagreement — doubt never lands here.
- `UNVERIFIABLE` — couldn't establish either way (runtime behavior, external systems).
Each branch is a chain: `verify_one → verdict_gate`. The gate machine-validates the JSON
contract (verdict enum; evidence required for VERIFIED/FALSE; note for UNVERIFIABLE),
stamps the authoritative id FROM the finding item (the model's echo is never trusted),
rejects back to `verify_one` exactly once with the reason, and on a second failure records
UNVERIFIABLE — a malformed branch never sinks the map. If `verify_one` itself dies (both
`max_attempts: 2` attempts fail), its `fallback` routes to `verify_fault`, which
normalizes the engine's failure text into a branch-local UNVERIFIABLE verdict the map
still collects.
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
marked). No final LLM step that could editorialize.
The dashed edges are fault paths. All three fault emitters (`parse_fault`, `verify_fault`,
and `verdict_gate`'s own crash guard) surface as `PIPELINE-FAULT:` text in a verdict's
`note`, and a fault can only ever produce UNVERIFIABLE — degraded output is never a
passing verdict. Every path ends at `done`, so the `FINDING_VERIFIER_RESULTS`
sentinel is always emitted and the caller's verify lane degrades instead of dying.
## What it is NOT
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
"matters" — that authority stays with the reviewers and the orchestrator's rigor rules.
- **Not a reviewer.** It reads only what findings cite (plus greps to locate moved code).
## Caller
Spawned by [`code-reviewer`](../code-reviewer/README.md) during its Finding Verifier
pass, ONCE per review, with the full 🔴/🟡 list and `project_dir`. If this agent isn't
installed, code-reviewer runs the same contract inline as a fallback.
## Files
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
per node).
- `scripts/verdict_gate.py` — the deterministic contract gate.
- `scripts/parse_fault.py`, `scripts/verify_fault.py` — the fault markers behind the
dashed fallback edges.
assets/agents/finding-verifier/graph.yaml
+207
View File
@@ -0,0 +1,207 @@
name: finding-verifier
description: |
Graph-based independent verifier for code-review findings. Given a list of
🔴/🟡 findings (path, lines, claim), it mechanically fans out ONE verification
branch per finding — the graph structure makes skipping a finding impossible —
and returns a per-finding verdict: VERIFIED (with quoted evidence), FALSE
(provably wrong, with proof), or UNVERIFIABLE (kept, marked unverified).
Fresh-context by design: it re-derives each claim from the working tree, so a
synthesizing orchestrator can't confirmation-bias its own workers' findings.
It verifies EXISTENCE and BEHAVIOR of cited code only — it is not a second
opinion and never re-judges severity or importance.
version: '1.0'
global_tools:
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
variables:
- name: project_dir
description: Repository the findings cite; relative paths in findings resolve against it.
default: '.'
settings:
max_loop_iterations: 8
log_state_snapshots: true
timeout: 900
initial_state:
findings_list: []
verdicts: []
gate_feedback: ''
gate_attempts: 0
parse_failure: ''
start: parse
nodes:
parse:
id: parse
type: llm
description: Extract the 🔴/🟡 findings from the spawn prompt into a structured list. Extraction only — no judging.
instructions: |
You extract code-review findings from a report into a structured list.
EXTRACTION ONLY — do not judge, filter, merge, or rephrase-away any
finding. Rules:
- Include every 🔴 CRITICAL and 🟡 WARNING finding. Ignore 🟢/💡
findings (they post unverified by design).
- Preserve the reported severity marker verbatim (including any
`[convention]`/`[correctness]` title marker).
- `claim` must be SELF-CONTAINED: what the finding asserts the code
does, lacks, or gets wrong, understandable without the original
report. Include the suggested-fix context when it clarifies the claim.
- `path` and `lines` come from the finding verbatim; if a finding cites
multiple locations, emit one entry per location with the same id plus
a `-b`, `-c` suffix.
- `project_dir`: the repository directory named in the request (a
"Repository:" or "Project:" line). When none is named, echo the
default shown in the prompt.
prompt: |
Extract the findings from this verification request (default repository
directory when the request names none: {{project_dir}}):
{{initial_prompt}}
tools: []
output_schema:
type: object
properties:
findings_list:
type: array
items:
type: object
properties:
id:
type: string
description: Short stable slug for mapping the verdict back (e.g. "f1", "f2-b").
severity:
type: string
description: The severity marker verbatim (e.g. "🔴 CRITICAL", "🟡 WARNING [correctness]").
path:
type: string
description: File path the finding cites, as reported.
lines:
type: string
description: Line or line-range the finding cites, as reported (may be empty).
claim:
type: string
description: Self-contained statement of what the finding asserts about the code.
required: [id, severity, path, lines, claim]
project_dir:
type: string
description: Repository directory named in the request (echo the prompt default when none).
required: [findings_list, project_dir]
max_attempts: 2
state_updates:
parse_failure: '{{output}}'
fallback: parse_fault
next: verify_each
parse_fault:
id: parse_fault
type: script
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — rewrites `verdicts` to a single UNVERIFIABLE PIPELINE-FAULT entry and jumps straight to done, so the FINDING_VERIFIER_RESULTS sentinel still renders and the caller's verify lane degrades instead of dying.
script: scripts/parse_fault.py
timeout: 30
next: done
verify_each:
id: verify_each
type: map
over: '{{findings_list}}'
as: finding
branch: verify_one
collect_into: verdicts
output_key: verdict
max_concurrency: 4
next: done
verify_one:
id: verify_one
type: llm
description: Verify ONE finding against the working tree — existence and behavior of the cited code, nothing else.
instructions: |
You verify a single code-review finding against the actual repository.
Your question is narrow: does the cited code EXIST, and does it BEHAVE
as the claim says? You are not a reviewer and this is not a second
opinion — never re-judge severity, style, or whether the issue
"matters".
Method:
1. `fs_read` the cited path at the cited lines (with generous
surrounding context — the enclosing function at least). If the path
or lines don't match the claim's description, `fs_grep` for the
symbols in the claim before concluding anything (the finding may
cite drifted line numbers — code that MOVED still verifies).
2. Compare what the code actually does against the claim.
Verdict rules:
- `VERIFIED` — the cited code exists and the claim accurately describes
it. `evidence` MUST be the decisive line(s) quoted verbatim with
path:line.
- `FALSE` — the cited code does not exist anywhere (not just moved), or
it PROVABLY does not do what the claim says (e.g. the claim says a
nil-check is missing but it is right there). `evidence` MUST quote
the proof. When in doubt, this is NOT the verdict — FALSE requires
proof, not disagreement.
- `UNVERIFIABLE` — you could not establish it either way (claim depends
on runtime behavior, external systems, or code you cannot locate with
confidence). Explain why in `note`. Doubt lands here, never in FALSE.
Output format — respond with ONLY this JSON object, no prose before or
after it:
{"id": "<the finding's id, echoed back verbatim>",
"verdict": "VERIFIED" | "FALSE" | "UNVERIFIABLE",
"evidence": "<verbatim quoted line(s) with path:line — required for
VERIFIED and FALSE, empty string for UNVERIFIABLE>",
"note": "<one line — for FALSE the disproof in words, for UNVERIFIABLE
why it could not be established; empty string otherwise>"}
Your response is machine-validated by a deterministic gate: a malformed
response (wrong shape, unknown verdict value, missing required
evidence/note) is rejected back to you exactly once with the reason,
then recorded as UNVERIFIABLE. The `id` you echo is informative only —
the gate stamps the authoritative id from the finding itself.
prompt: |
Repository: {{project_dir}}
Verify this single finding:
{{finding}}
{{gate_feedback}}
tools:
- fs_read
- fs_cat
- fs_grep
max_iterations: 8
max_attempts: 2
state_updates:
verdict: '{{output}}'
fallback: verify_fault
next: verdict_gate
verdict_gate:
id: verdict_gate
type: script
description: Deterministic contract gate for each verdict — validates shape, verdict enum, and evidence/note rules; rejects back to verify_one exactly once with the reason, then records UNVERIFIABLE. No LLM judgment.
script: scripts/verdict_gate.py
timeout: 30
verify_fault:
id: verify_fault
type: script
description: Fail-closed marker for a dead verify branch (reached only via verify_one's fallback; branch-local). Normalizes the engine's failure text in `verdict` into a schema-conformant UNVERIFIABLE verdict the map collects — a dead verifier lane never sinks the whole map or masquerades as verified.
script: scripts/verify_fault.py
timeout: 30
done:
id: done
type: end
output: |
FINDING_VERIFIER_RESULTS
(verdict semantics for the caller: VERIFIED = keep the finding and paste its evidence
quote into the Description; FALSE = drop the finding and tally it in the footer;
UNVERIFIABLE = keep the finding, marked unverified. Severities are never rewritten.)
{{verdicts}}
assets/agents/finding-verifier/scripts/parse_fault.py
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead parse stage of the finding-verifier graph.
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
this node rewrites `verdicts` to a single schema-conformant UNVERIFIABLE
entry naming the fault and the graph jumps straight to `done` — the
FINDING_VERIFIER_RESULTS sentinel still renders, so the caller (e.g.
code-reviewer's verify lane) degrades instead of dying. Findings that were
never extracted cannot be verified; the caller's "UNVERIFIABLE = keep the
finding, marked unverified" semantics do the right thing with the entry.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def fault_verdicts(detail):
return {
"verdicts": [
{
"id": "pipeline-fault",
"verdict": "UNVERIFIABLE",
"evidence": "",
"note": (
"PIPELINE-FAULT: parse failed — findings could not be extracted "
f"from the verification request; no finding was verified: {detail}"
),
}
]
}
def main():
state = load_state()
raw = state.get("parse_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the parse stage died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(json.dumps(fault_verdicts(detail)))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(json.dumps(fault_verdicts(f"fault-marker script error: {e}")))
assets/agents/finding-verifier/scripts/verdict_gate.py
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""Contract gate for finding-verifier's per-finding verify chain.
Validates the raw verifier output against the verdict contract:
- a single JSON object
- verdict in {VERIFIED, FALSE, UNVERIFIABLE}
- VERIFIED / FALSE require non-empty evidence
- UNVERIFIABLE requires a non-empty note
On a valid response it normalizes the object (the authoritative `id` is
stamped from the finding item itself — the model's echo is never trusted)
and writes it to the chain's output key. On an invalid response it routes
back to `verify_one` exactly once with the rejection reason; if the retry
is also invalid, it records an UNVERIFIABLE verdict naming the failure so
a malformed item can never sink the whole map or masquerade as a pass.
Deterministic by design: no LLM judgment happens here, and the script
always exits 0 with a valid JSON object on stdout.
"""
import json
import os
import re
VERDICTS = {"VERIFIED", "FALSE", "UNVERIFIABLE"}
MAX_RETRIES = 1
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def finding_id(state):
finding = state.get("finding")
if isinstance(finding, dict):
fid = finding.get("id")
if isinstance(fid, str) and fid.strip():
return fid.strip()
return "unknown"
def extract_json(text):
if not isinstance(text, str) or not text.strip():
raise ValueError("verifier output is empty or not text")
text = re.sub(r"```[a-zA-Z]*", "", text).replace("```", "")
start, end = text.find("{"), text.rfind("}")
if start == -1 or end <= start:
raise ValueError("no JSON object found in verifier output")
return json.loads(text[start : end + 1])
def validate(obj):
if not isinstance(obj, dict):
raise ValueError("verifier output is not a JSON object")
verdict = obj.get("verdict")
if verdict not in VERDICTS:
raise ValueError(f"verdict must be one of {sorted(VERDICTS)}, got {verdict!r}")
evidence = obj.get("evidence") if isinstance(obj.get("evidence"), str) else ""
note = obj.get("note") if isinstance(obj.get("note"), str) else ""
if verdict in ("VERIFIED", "FALSE") and not evidence.strip():
raise ValueError(
f"{verdict} requires non-empty evidence quoting the decisive line(s) with path:line"
)
if verdict == "UNVERIFIABLE" and not note.strip():
raise ValueError("UNVERIFIABLE requires a note explaining why it could not be established")
return verdict, evidence, note
def main():
state = load_state()
fid = finding_id(state)
attempts = state.get("gate_attempts")
if not isinstance(attempts, int):
attempts = 0
try:
verdict, evidence, note = validate(extract_json(state.get("verdict")))
print(
json.dumps(
{"verdict": {"id": fid, "verdict": verdict, "evidence": evidence, "note": note}}
)
)
return
except Exception as e: # noqa: BLE001 — every failure is a rejection reason
reason = str(e)
if attempts < MAX_RETRIES:
print(
json.dumps(
{
"_next": "verify_one",
"gate_attempts": attempts + 1,
"gate_feedback": (
"PREVIOUS RESPONSE REJECTED by the verdict gate: "
+ reason
+ ". Respond with ONLY the JSON object described under"
" 'Output format' — no prose, no code fences."
),
}
)
)
return
# Deliberately not PIPELINE-FAULT-prefixed: model misbehavior, not infrastructure.
print(
json.dumps(
{
"verdict": {
"id": fid,
"verdict": "UNVERIFIABLE",
"evidence": "",
"note": f"verifier output failed machine validation after retry: {reason}",
}
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the gate must never crash the chain
print(
json.dumps(
{
"verdict": {
"id": "unknown",
"verdict": "UNVERIFIABLE",
"evidence": "",
"note": f"PIPELINE-FAULT: verdict gate error: {e}",
}
}
)
)
assets/agents/finding-verifier/scripts/verify_fault.py
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead verify branch of the finding-verifier graph.
Reached ONLY via verify_one's `fallback:` route inside the map branch.
Grounded in engine behavior: on llm-node failure the engine writes
"LLM node failed: <chain>" into `verdict` via the node's state_updates
BEFORE routing to the fallback, so the raw failure text is already in
state. This node normalizes it into a schema-conformant UNVERIFIABLE
verdict (authoritative id stamped from the finding item, mirroring
verdict_gate.py) and the chain ends there — the map collects a valid
verdict, so a dead verifier lane never sinks the whole map or masquerades
as verified.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def finding_id(state):
finding = state.get("finding")
if isinstance(finding, dict):
fid = finding.get("id")
if isinstance(fid, str) and fid.strip():
return fid.strip()
return "unknown"
def fault_verdict(fid, detail):
return {
"verdict": {
"id": fid,
"verdict": "UNVERIFIABLE",
"evidence": "",
"note": f"PIPELINE-FAULT: verifier lane failed after retries — {detail}",
}
}
def main():
state = load_state()
fid = finding_id(state)
raw = state.get("verdict")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if not detail.startswith("LLM node"):
detail = "the verifier lane died without recording a failure detail"
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(json.dumps(fault_verdict(fid, detail)))
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(json.dumps(fault_verdict("unknown", f"fault-marker script error: {e}")))
assets/agents/librarian/graph.yaml
-3
View File
@@ -32,9 +32,6 @@ settings:
log_state_snapshots: true
timeout: 600
reducers:
output: overwrite
initial_state:
language_ecosystem: 'general'
doc_domain_hints: ''
assets/agents/review-gauntlet/README.md
+219
View File
@@ -0,0 +1,219 @@
# Review Gauntlet
A graph-based post-implementation review gate — **"the done gate"**. An
orchestrator that must not claim *done* without green reviews spawns this agent
once instead of judging trigger rules and parsing verdict sentinels itself. The
graph structure makes skipping a lane or misreading a verdict impossible:
trigger signals are computed by a script, lanes run as isolated parallel
sub-agents, and the final verdict is a deterministic regex gate — an LLM never
gets the chance to rationalize past it.
```mermaid
flowchart TD
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
parse -. "fallback" .-> pfault["parse_fault<br/>(script: PIPELINE-FAULT marker)"]
pfault --> gate
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes override exactly)"]
select -. "fallback" .-> dlanes["default_lanes<br/>(script: deterministic degraded<br/>selection — wider, never narrower)"]
dlanes --> build
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer<br/>(max_attempts: 2)"]]
madv --> radv[["adversary<br/>(max_attempts: 2)"]]
msec --> rsec[["security-reviewer<br/>(max_attempts: 2)"]]
mpb --> rpb[["probe<br/>(max_attempts: 2)"]]
rcr -. "fallback" .-> lfault["lane_fault<br/>(script: PIPELINE-FAULT lane result,<br/>collected by the map)"]
radv -. "fallback" .-> lfault
rsec -. "fallback" .-> lfault
rpb -. "fallback" .-> lfault
mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
rgate -. "_next: retry faulted lanes" .-> build
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
```
## Why a graph
The four independent review lanes existed before this agent — as **prompt
discipline** inside the calling orchestrator: "spawn code-reviewer when the
change is broad", "🔴 always blocks", "INCONCLUSIVE is never PASS". Prompt
discipline degrades under context pressure: a lane gets skipped, a verdict
line gets paraphrased, a missing report reads as "nothing to report". Here
those rules are *structure*:
- **Lane selection floor is deterministic** (`build_items.py`): code-review
always runs; adversary runs when a spec/plan is present; security runs on
attack-surface signals (auth/deps/exec paths or diff content) or hardened
posture; probe runs when consumer-facing surface changed *and* a local-run
recipe exists. The `select_lanes` LLM can only **widen** the set — a false
flag never removes a rule-selected lane. `forced_lanes` from the caller
override everything exactly.
- **Conditional lanes without dynamic routing**: each lane is a `map` node
over a 0-or-1-item list — an empty list means the map runs zero branches
(map-over-nothing), which *is* the skip mechanism. All four maps join at
the retry gate on equal-length paths; it falls through to the verdict
gate, or routes back to `build_items` for faulted lanes only.
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
sentinel is a lane FAILURE, never a pass; the critical count from the
code-review report's summary line (raw 🔴 count if absent) blocks
regardless of its verdict line; a code-review lane whose own verdict line
records a pipeline fault blocks (a degraded review is never a pass); the
adversary's own degraded-run header (the `Criteria: …` line directly under
its sentinel recording a pipeline fault, died criterion checks, or its
verdict script's crash stub) blocks as a pipeline fault,
not as a DIVERGES finding;
`NEEDS-HUMAN` without 🔴 passes but
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
with an environment note — it is never treated as PASS or FAIL.
- **Verdict independence is structural**: lanes run as isolated sub-agents
with no `teammates:` flag — no cross-lane messaging, by construction.
- **Failure fails closed, structurally** (`fallback:` routes + `PIPELINE-FAULT`
markers): every LLM node and lane retries once (`max_attempts: 2`), then a
fallback fires instead of killing the graph — and every fallback degrades
toward BLOCKED, never toward a pass. A dead lane becomes a `PIPELINE-FAULT:`
lane result (`lane_fault.py`) that the gate reports as BLOCKED *naming the
lane* — distinct from SKIPPED (not selected). A dead `parse` records a fault
and jumps straight to the gate (`parse_fault.py`). A dead `select_lanes`
degrades to a deterministic selection (`default_lanes.py`): code-review +
adversary, plus probe on consumer surface *with a local-run recipe*, plus
security on auth/deps/exec
signals or hardened posture, unioned with caller-forced lanes — wider, never
narrower. A crashed `build_items` records a fault so an all-SKIPPED gate
still blocks. Fault detection is prefix-anchored — a real review that merely
*quotes* "PIPELINE-FAULT:" flows to the normal sentinel rules. And the
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
`GAUNTLET: BLOCKED`, never silence.
## Fault-only re-runs
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
only lanes that **faulted** — a missing report (stood in for by a synthetic
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
report, or a nested graph's own degraded-run wording (the code-review verdict
line / the adversary's header line directly under its sentinel). A real verdict
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
- **Stash + restore**: every lane report that ran this pass is stashed in
`kept_results`; a re-run pass overwrites the untouched lanes' collected
results with `[]`, so their stashed reports are restored before the gate.
The verdict gate always sees each lane's *last* report.
- **Retry never narrows selection**: lane selection is not re-run. A re-run
pass rebuilds items only for the retried lanes, and the stashed results
stand in for the rest.
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
attempts or the budget lands in `review_incomplete`, and its fault flows to
the verdict gate as-is.
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
guards append to an existing `signals_error` rather than overwrite it, and
`build_items.py` preserves the prior `lanes_summary`.
A non-empty `GAUNTLET_REVIEW_INCOMPLETE:` line structurally implies
`GAUNTLET: BLOCKED`: the verdict gate forces every lane named in
`review_incomplete` to a BLOCKED row (`review incomplete after N attempt(s)`)
and a matching blocker, so a restored earlier report can never turn an
incomplete review into a pass — and malformed `review_incomplete` bookkeeping
is itself recorded as a `pipeline` fault, never dropped.
Whenever at least one lane (or the pipeline) could not complete, the first
line reads `GAUNTLET: BLOCKED` followed immediately by
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
builder, the retry gate, the default-lanes fallback, or the verdict gate
itself (the default-lanes fallback's crash still runs the widened lanes, so
`pipeline` appears alongside real lane rows). The report gains a
`## Review incomplete` section naming each exhausted lane with its
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
re-run (N > 1). The *named* entries are faults to re-run, never code
findings; the line can accompany real findings from the other lanes, which
still appear under `## Blockers` and must be fixed as usual.
## Sentinels parsed
| Lane | Sentinel | Blocks on |
|------|----------|-----------|
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; own degraded-run header (pipeline fault, died criterion checks, or its verdict script's crash stub); missing sentinel |
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
## Spawning it
```
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
Project: /abs/path/to/repo
Diff: main...HEAD (a git range, a ref, or 'worktree' for uncommitted changes)
Rigor: production Security posture: standard
Plan / acceptance criteria:
<paste the criteria the implementation must satisfy — enables the adversary lane>
Local-run recipe / usage suites:
<how to boot the service locally + where existing suites live — enables the probe lane>
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)" \
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
```
Verification commands are NOT part of the prompt. They are the declared
`verification_commands` graph variable (default `'[]'`; a JSON array of shell
commands as a string, since agent variables arrive as strings), also settable
from the CLI:
```sh
coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
```
The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
table, the lane-selection reasons, blockers, attention items, and each lane's
full report in a collapsible section. On BLOCKED with a
`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — this is an
infrastructure fault, never a code finding. Fix any non-fault blockers listed
under `## Blockers` first (the re-run retries the lane); if the line persists,
escalate to the user with three options — retry the review again (a **fresh**
gauntlet run), accept NEEDS-HUMAN and proceed without that lane (recorded in the
report / PR body), or abort — never route it through your findings-handling
rules. On BLOCKED without it: fix the findings per your own findings-handling
rules, then spawn a **fresh** gauntlet run. On PASS with *Human attention
required*: complete the work but carry those items into your final report
verbatim.
## Notes
- The caller decides *whether* review is warranted at all (the gauntlet is for
non-trivial work — that's why code-review is unconditionally on); the
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
- Lane sub-agents get generous finite per-attempt timeouts (see the
`timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
is derived from that envelope in graph.yaml's comment. A lane that still
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
- The `verification_commands` variable is forwarded to the adversary lane as a
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
lands the raw value in the child's state) — never through the lane prompt
and never extracted by `parse`. The adversary's `run_checks` stage executes
those commands with a shell, which makes the declaration a trust boundary:
the prompt also carries plan text pasted from the repo under review, so
commands must never be inferred from it, and the engine merges only the keys
`parse`'s `output_schema.properties` declares, so an extra
`verification_commands` key emitted by the LLM is dropped before it reaches
state.
Undeclared stays undeclared
(`'[]'` ⇒ "none declared"); a declaration that is not a JSON array of
strings executes nothing and records a `PIPELINE-FAULT`, which the
adversary lane reports as DIVERGES and the gate as BLOCKED.
- Signals failures degrade gracefully: if git can't compute the diff, lane
selection falls back to caller context + forced lanes, and the report notes
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
auth/authn/authz/oauth but not "author") — the safe direction, since signals
only ever ADD the security lane.
- Requires the `code-reviewer`, `adversary`, `security-reviewer`, and `probe`
agents to be installed (they ship with Coyote).
assets/agents/review-gauntlet/graph.yaml
+444
View File
@@ -0,0 +1,444 @@
name: review-gauntlet
description: |
Graph-based post-implementation review gate — "the done gate". Given a diff
spec and the work's context (plan/acceptance criteria, rigor, security
posture), it deterministically computes diff signals, selects the applicable
independent review lanes — code-reviewer (always), adversary (when a
spec/plan exists), security-reviewer (attack-surface signals), probe
(consumer-facing surface with a local-run recipe) — runs the selected lanes
IN PARALLEL as isolated sub-agents, and machine-parses their verdict
sentinels into one deterministic verdict: GAUNTLET: PASS or
GAUNTLET: BLOCKED. An orchestrator that must not claim "done" without green
reviews spawns this once instead of judging trigger rules and parsing
verdicts itself — the graph structure makes skipping a lane or misreading a
verdict impossible. Invariants: lane selection can only be WIDENED by LLM
judgment, never narrowed (deterministic rules are the floor); a missing
verdict sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks
with an environment note (it is never PASS or FAIL); reviewer lanes run
verdict-independent — no cross-lane messaging.
version: '1.0'
# The limit is consulted by the SPAWNER against the child's absolute depth
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
# review-gauntlet(2) → code-reviewer(3) →
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
# This agent spawns depth-3 children; 4 = one level of headroom.
max_agent_depth: 4
max_concurrent_agents: 12
variables:
- name: project_dir
description: Repository the change lives in; signals and every review lane run against it.
default: '.'
- name: verification_commands
description: |
JSON array of shell commands the CALLER declares for the deterministic
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
prompt; empty array = none declared.
default: '[]'
settings:
max_loop_iterations: 10
log_state_snapshots: true
# A faulted lane can be re-run by retry_gate for up to 3 passes, and each
# pass's agent node retries up to max_attempts = 2 with a fresh per-attempt
# budget (src/graph/agent.rs retry_transient/bounded_attempt):
# 3 passes × 2 attempts × 11400 (max lane timeout) = 68400, + 1200 margin
# for the script stages (signals/select/build/retry gate/verdict gate)
# = 69600; the parse and select_lanes llm nodes carry no `timeout:` and
# are bounded by max_iterations.
timeout: 69600
initial_state:
diff_spec: worktree
plan_context: ''
rigor: production
security_posture: standard
probe_context: ''
extra_context: ''
forced_lanes: []
changed_files: []
file_count: 0
signals_summary: ''
signals_error: ''
touches_auth: false
touches_deps: false
touches_exec: false
touches_proto: false
touches_migrations: false
consumer_surface: false
docs_only: false
add_code_review: false
add_adversary: false
add_security: false
add_probe: false
lane_reasons: ''
parse_failure: ''
select_lanes_failure: ''
lanes_degraded: ''
code_review_items: []
adversary_items: []
security_items: []
probe_items: []
code_review_results: []
adversary_results: []
security_results: []
probe_results: []
lane_attempts: {}
kept_results: {}
retry_lanes: []
review_incomplete: []
gauntlet_started_at: 0
lanes_summary: ''
gauntlet_verdict: ''
gauntlet_report: ''
gauntlet_incomplete_line: ''
start: parse
nodes:
parse:
id: parse
type: llm
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging, no lane selection.
instructions: |
You extract the facts of a review request into structured fields.
EXTRACTION ONLY — do not decide which review lanes should run.
- `diff_spec`: the git diff specification the caller named — a ref
range ("main...HEAD", "HEAD~3..HEAD"), a single ref, or the literal
"worktree" for uncommitted changes against HEAD. When the caller
did not name one, use "worktree".
- `project_dir_in`: the repository directory the caller named as holding
the change (absolute path preferred); empty string when the caller
named none.
- `plan_context`: the plan / spec / acceptance-criteria text the
implementation must satisfy, verbatim (or a file path the caller
pointed at). Empty string when the caller supplied none.
- `rigor`: poc | prototype | production. Default "production" when
not stated.
- `security_posture`: prototype | standard | hardened. Default
"standard" when not stated.
- `probe_context`: the local-run recipe and/or existing usage-suite
pointers for black-box probing (how to boot the service, where the
suites live). Empty string when the caller supplied none.
- `forced_lanes`: lanes the caller explicitly demanded (verbatim
names). Empty array = automatic selection.
- `extra_context`: any remaining caller context reviewers should see,
condensed.
prompt: |
Extract the review-request facts from this request:
{{initial_prompt}}
tools: []
output_schema:
type: object
properties:
diff_spec:
type: string
plan_context:
type: string
rigor:
type: string
security_posture:
type: string
probe_context:
type: string
forced_lanes:
type: array
items:
type: string
extra_context:
type: string
project_dir_in:
type: string
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in]
max_iterations: 3
max_attempts: 2
state_updates:
parse_failure: '{{output}}'
fallback: parse_fault
next: signals
parse_fault:
id: parse_fault
type: script
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — records a PIPELINE-FAULT in signals_error and jumps straight to the verdict gate, which blocks on it. Lane item lists stay their initial [] so every lane records SKIPPED.
script: scripts/parse_fault.py
timeout: 30
next: verdict_gate
signals:
id: signals
type: script
description: Deterministic diff signals — file list, attack-surface and consumer-surface indicators. Pure git; no LLM judgment.
script: scripts/signals.py
timeout: 120
next: select_lanes
select_lanes:
id: select_lanes
type: llm
description: Additive-only lane judgment on top of the deterministic rules. It can WIDEN the gauntlet, never narrow it.
instructions: |
You look at a change's deterministic diff signals and decide whether any
review lane the hard rules might have missed should ALSO run. You can
only ADD lanes — the deterministic floor (enforced downstream, not by
you) is: code-review always runs; adversary runs when a plan/spec is
present; security runs on attack-surface signals or hardened posture;
probe runs when consumer-facing surface changed and a local-run recipe
exists. Setting a flag to false NEVER removes a lane the rules selected.
Reasons to add:
- `add_security`: the diff handles external/untrusted input, touches
crypto/secrets handling, or changes trust boundaries in ways the
path-based signals missed.
- `add_adversary`: the request references behavior it must preserve or
criteria scattered in prose that amount to a spec, even without a
formal plan.
- `add_probe`: the change alters observable API/CLI behavior even
though the path signals missed it (only useful when a local-run
recipe exists — the builder skips it otherwise, noting why).
- `add_code_review`: always selected by the rules; flag exists only
for forced-lane configurations.
Put one line per added lane into `lane_reasons`. When nothing needs
adding, all flags false and empty reasons is the correct answer.
prompt: |
Change signals (deterministic):
- diff spec: {{diff_spec}}
- files changed: {{file_count}}
- changed files (first 200): {{changed_files}}
- touches auth paths/content: {{touches_auth}}
- touches dependency manifests: {{touches_deps}}
- adds exec/shell/subprocess calls: {{touches_exec}}
- touches protobuf/contracts: {{touches_proto}}
- touches migrations: {{touches_migrations}}
- consumer-facing surface paths: {{consumer_surface}}
- docs-only diff: {{docs_only}}
- {{signals_summary}}
{{signals_error}}
Caller context:
- rigor: {{rigor}}; security posture: {{security_posture}}
- plan/spec present: {{plan_context}}
- probe recipe present: {{probe_context}}
- extra: {{extra_context}}
tools: []
output_schema:
type: object
properties:
add_code_review:
type: boolean
add_adversary:
type: boolean
add_security:
type: boolean
add_probe:
type: boolean
lane_reasons:
type: string
required: [add_code_review, add_adversary, add_security, add_probe, lane_reasons]
max_iterations: 3
max_attempts: 2
state_updates:
select_lanes_failure: '{{output}}'
fallback: default_lanes
next: build_items
default_lanes:
id: default_lanes
type: script
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface AND a probe_context (local-run recipe) is present, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
script: scripts/default_lanes.py
timeout: 30
next: build_items
build_items:
id: build_items
type: script
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map. On a re-run pass (retry_lanes non-empty) it skips selection and emits items only for those lanes, each carrying its attempt number.
script: scripts/build_items.py
timeout: 30
next: [map_code_review, map_adversary, map_security, map_probe]
map_code_review:
id: map_code_review
type: map
over: '{{code_review_items}}'
as: lane_ctx
branch: run_code_review
output_key: lane_out
collect_into: code_review_results
max_concurrency: 1
next: retry_gate
map_adversary:
id: map_adversary
type: map
over: '{{adversary_items}}'
as: lane_ctx
branch: run_adversary
output_key: lane_out
collect_into: adversary_results
max_concurrency: 1
next: retry_gate
map_security:
id: map_security
type: map
over: '{{security_items}}'
as: lane_ctx
branch: run_security
output_key: lane_out
collect_into: security_results
max_concurrency: 1
next: retry_gate
map_probe:
id: map_probe
type: map
over: '{{probe_items}}'
as: lane_ctx
branch: run_probe
output_key: lane_out
collect_into: probe_results
max_concurrency: 1
next: retry_gate
run_code_review:
id: run_code_review
type: agent
description: Independent code-quality review lane.
agent: code-reviewer
# Exceeds the code-reviewer graph's own settings.timeout (10800) by 600 so
# the child's graph timeout, not this lane timeout, is the binding bound.
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
# timeouts) is not derived here — deliberately out of scope for this
# change; tracked as a follow-up.
timeout: 11400
prompt: |
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
independent post-implementation review.
Quality bar: rigor={{rigor}}.
Plan / acceptance-criteria context (may be empty):
{{plan_context}}
{{extra_context}}
Close your report with your Review Verdict line.
max_attempts: 2
state_updates:
lane_out: '{{output}}'
fallback: lane_fault
run_adversary:
id: run_adversary
type: agent
description: Plan-conformance review lane.
agent: adversary
# Exceeds the adversary graph's own settings.timeout (9000) by 600 so the
# child's graph timeout, not this lane timeout, is the binding bound.
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
# timeouts) is not derived here — deliberately out of scope for this
# change; tracked as a follow-up.
timeout: 9600
inputs:
verification_commands: '{{verification_commands}}'
prompt: |
Adversarial plan-conformance review in {{project_dir}}.
The implementation to check: diff {{diff_spec}}.
The spec / plan / acceptance criteria it must satisfy:
{{plan_context}}
Verification commands arrive as a structured input (the verification_commands variable), not in this prompt.
{{extra_context}}
Return your ADVERSARIAL_REVIEW verdict.
max_attempts: 2
state_updates:
lane_out: '{{output}}'
fallback: lane_fault
run_security:
id: run_security
type: agent
description: Security review lane.
agent: security-reviewer
timeout: 7200
prompt: |
Security review of the code changes in {{project_dir}} (diff: {{diff_spec}}).
Security posture: {{security_posture}}.
{{extra_context}}
Return your SECURITY_REVIEW verdict.
max_attempts: 2
state_updates:
lane_out: '{{output}}'
fallback: lane_fault
run_probe:
id: run_probe
type: agent
description: Black-box usage-pattern verification lane.
agent: probe
timeout: 7200
prompt: |
Usage-pattern verification (black-box probe) of the changes in
{{project_dir}} (diff: {{diff_spec}}).
Local-run recipe / existing usage suites:
{{probe_context}}
Plan / acceptance-criteria context:
{{plan_context}}
The workspace may contain test files from an aborted earlier attempt of
this lane; reconcile rather than duplicate.
Derive expected behaviors from the plan/contract BEFORE reading
implementation code, run existing suites first, then author spec-first
tests for the uncovered delta. Return your USAGE_PROBE verdict.
max_attempts: 2
state_updates:
lane_out: '{{output}}'
fallback: lane_fault
lane_fault:
id: lane_fault
type: script
description: Fail-closed lane fault marker (reached only via the run_* lanes' fallback routes; shared across the four map branches). Normalizes the engine's "Agent node failed" text in lane_out into a PIPELINE-FAULT lane result the map collects; verdict_gate turns it into BLOCKED — a degraded lane is never a pass.
script: scripts/lane_fault.py
timeout: 30
retry_gate:
id: retry_gate
type: script
description: Deterministic fault-only re-run gate. Stashes every lane report that ran this pass, restores reports a re-run pass overwrote with [] (a lane not being re-run maps over nothing), classifies each lane that ran as completed / faulted / skipped, counts attempts, and routes back into build_items for faulted lanes that still have attempts left inside the wall-clock budget; otherwise it records them in review_incomplete and falls through to verdict_gate. A real verdict is never re-run, and a gate crash never retries — it records a PIPELINE-FAULT and falls through.
script: scripts/retry_gate.py
timeout: 30
next: verdict_gate
verdict_gate:
id: verdict_gate
type: script
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
script: scripts/verdict_gate.py
timeout: 60
next: done
done:
id: done
type: end
output: |
GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
{{gauntlet_report}}
assets/agents/review-gauntlet/scripts/build_items.py
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""Deterministic lane builder for the review gauntlet.
Selection = hard rules (the floor) ∪ LLM additions; forced lanes override
everything exactly. Emits one 0-or-1-item list per lane: an empty list makes
that lane's map node run zero branches (map-over-nothing), which IS the
skip mechanism — no dynamic routing anywhere.
The LLM's add_* flags can only ADD lanes. A flag that is false never
removes a lane the hard rules selected.
A re-run pass (retry_gate emitted `retry_lanes`) skips selection: only the
named lanes get an item, stamped with their attempt number, and every other
lane maps over nothing — retry_gate restores their earlier reports.
Fail-closed: a builder crash emits all four lists empty plus a
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
never kills the graph.
"""
import json
import os
import time
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
# canonicalize the same way on the degraded path as it does here.
ALIASES = {
"code-review": "code-review",
"code_review": "code-review",
"code-reviewer": "code-review",
"adversary": "adversary",
"adversarial": "adversary",
"adversarial-review": "adversary",
"security": "security",
"security-review": "security",
"security-reviewer": "security",
"probe": "probe",
"usage": "probe",
"usage-pattern": "probe",
"usage-pattern-testing": "probe",
}
# Keep in sync with retry_gate.LANES
LANE_KEYS = {
"code-review": "code_review_items",
"adversary": "adversary_items",
"security": "security_items",
"probe": "probe_items",
}
# Filled by main() so the crash guard can append to an upstream fault
# instead of overwriting it, and keep the first pass's lanes_summary.
crash_context = {"signals_error": "", "lanes_summary": ""}
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
prior = state.get("signals_error")
crash_context["signals_error"] = prior if isinstance(prior, str) else ""
# retry_gate's wall-clock budget is measured from this stamp; signals.py
# is the normal stamper, this is the defensive fallback.
stamp = {} if state.get("gauntlet_started_at") else {"gauntlet_started_at": time.time()}
def text(key):
v = state.get(key)
return v.strip() if isinstance(v, str) else ""
crash_context["lanes_summary"] = text("lanes_summary")
retry_lanes = [lane for lane in (state.get("retry_lanes") or []) if lane in LANE_KEYS]
if retry_lanes:
attempts = state.get("lane_attempts") or {}
retried = [(lane, int(attempts.get(lane) or 0) + 1) for lane in retry_lanes]
note = "- retried: [" + ", ".join(f"{lane} (attempt {n})" for lane, n in retried) + "]"
summary = text("lanes_summary")
print(
json.dumps(
{
**{key: [] for key in LANE_KEYS.values()},
**{
LANE_KEYS[lane]: [{"lane": lane, "attempt": n}] for lane, n in retried
},
"lanes_summary": f"{summary}\n{note}" if summary else note,
"retry_lanes": [],
**stamp,
}
)
)
return
forced_raw = state.get("forced_lanes") or []
forced, unknown = [], []
for lane in forced_raw:
canon = ALIASES.get(str(lane).strip().lower())
(forced.append(canon) if canon else unknown.append(str(lane)))
reasons = []
if forced:
lanes = set(forced)
reasons.append(f"forced lanes honored exactly: {sorted(lanes)}")
else:
lanes = {"code-review"}
reasons.append(
"code-review: always on — the gauntlet is only spawned for non-trivial work"
)
if text("plan_context"):
lanes.add("adversary")
reasons.append("adversary: a spec/plan/acceptance-criteria context is present")
if (
state.get("touches_auth")
or state.get("touches_deps")
or state.get("touches_exec")
or state.get("security_posture") == "hardened"
):
lanes.add("security")
reasons.append(
"security: attack-surface signals (auth/deps/exec) or hardened posture"
)
if state.get("consumer_surface") and text("probe_context"):
lanes.add("probe")
reasons.append(
"probe: consumer-facing surface changed and a local-run recipe/suite pointer exists"
)
# LLM judgment is additive-only.
llm_why = text("lane_reasons")[:300]
for flag, lane in (
("add_code_review", "code-review"),
("add_adversary", "adversary"),
("add_security", "security"),
("add_probe", "probe"),
):
if state.get(flag) and lane not in lanes:
if lane == "probe" and not text("probe_context"):
reasons.append(
"probe: requested by lane judgment but skipped — no local-run "
"recipe/suite pointer (a probe without one is guaranteed INCONCLUSIVE)"
)
continue
if lane == "adversary" and not text("plan_context"):
reasons.append(
"adversary: requested by lane judgment but skipped — no spec/plan "
"to check conformance against"
)
continue
lanes.add(lane)
reasons.append(f"{lane}: added by lane-selection judgment — {llm_why}")
if unknown:
reasons.append(f"ignored unknown forced lane name(s): {unknown}")
# A degraded lane selection (default_lanes fallback) must survive into the
# final report; lanes_summary is the only channel this script doesn't
# overwrite silently.
degraded = text("lanes_degraded")
if degraded:
reasons.insert(0, degraded)
print(
json.dumps(
{
"code_review_items": [{"lane": "code-review"}] if "code-review" in lanes else [],
"adversary_items": [{"lane": "adversary"}] if "adversary" in lanes else [],
"security_items": [{"lane": "security"}] if "security" in lanes else [],
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
**stamp,
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
fault = f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run"
if crash_context["signals_error"]:
fault = f"{crash_context['signals_error']}; {fault}"
print(
json.dumps(
{
"code_review_items": [],
"adversary_items": [],
"security_items": [],
"probe_items": [],
"lanes_summary": crash_context["lanes_summary"],
"signals_error": fault,
}
)
)
assets/agents/review-gauntlet/scripts/default_lanes.py
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env python3
"""Deterministic lane-selection fallback for the review gauntlet.
Reached ONLY via select_lanes' `fallback:` route, when the additive LLM
judgment dies after retries. Emits `forced_lanes` — code-review + adversary,
plus probe when the deterministic signals flagged a consumer-facing surface
AND a local-run recipe exists (build_items gates probe the same way — a
probe without a recipe is INCONCLUSIVE by construction),
plus security when the attack-surface signals (auth/deps/exec) or a hardened
posture demand it, unioned with any lanes the caller forced (canonicalized
with the same aliases build_items uses) — which build_items honors exactly,
so selection degrades WIDER, never narrower. The degradation note goes into
`lanes_degraded`; build_items folds it into lanes_summary so the final
report names the degradation.
Fail-closed: if the fallback itself crashes it emits the widest safe
selection (defaults ∪ security) plus a PIPELINE-FAULT in signals_error so
verdict_gate blocks — a crashed fallback never narrows selection silently.
"""
import json
import os
MAX_DETAIL_CHARS = 500
# Keep in sync with build_items.ALIASES — a caller-forced lane must
# canonicalize the same way here as it would on the non-degraded path.
ALIASES = {
"code-review": "code-review",
"code_review": "code-review",
"code-reviewer": "code-review",
"adversary": "adversary",
"adversarial": "adversary",
"adversarial-review": "adversary",
"security": "security",
"security-review": "security",
"security-reviewer": "security",
"probe": "probe",
"usage": "probe",
"usage-pattern": "probe",
"usage-pattern-testing": "probe",
}
# Stable emission order keeps the fallback deterministic.
LANE_ORDER = ["code-review", "adversary", "security", "probe"]
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
lanes = {"code-review", "adversary"}
probe_context = state.get("probe_context")
if (
state.get("consumer_surface")
and isinstance(probe_context, str)
and probe_context.strip()
):
lanes.add("probe")
# The deterministic security signals are already in state when this
# fallback runs; a degraded selection must never drop the security lane
# build_items' hard rules would have selected.
if (
state.get("touches_auth")
or state.get("touches_deps")
or state.get("touches_exec")
or state.get("security_posture") == "hardened"
):
lanes.add("security")
# Caller-forced lanes survive degradation — union, never overwrite.
for lane in state.get("forced_lanes") or []:
if canon := ALIASES.get(str(lane).strip().lower()):
lanes.add(canon)
forced = [lane for lane in LANE_ORDER if lane in lanes]
raw = state.get("select_lanes_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
note = "lane selection degraded to deterministic defaults"
if detail:
note += f" — {detail}"
print(json.dumps({"forced_lanes": forced, "lanes_degraded": note}))
try:
main()
except Exception as e: # noqa: BLE001 — the fallback itself must never crash
print(
json.dumps(
{
"forced_lanes": ["code-review", "adversary", "security"],
"lanes_degraded": "lane selection degraded to deterministic "
f"defaults (fallback script error: {e})",
"signals_error": (
f"PIPELINE-FAULT: lane-selection fallback crashed — {e}; "
"lanes widened to defaults ∪ security"
),
}
)
)
assets/agents/review-gauntlet/scripts/lane_fault.py
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Fail-closed lane fault marker for the review gauntlet.
Reached ONLY via the run_* lane nodes' `fallback:` routes inside their map
branches. Grounded in engine behavior: on agent-node failure the engine
writes "Agent node failed: {e:#}" into `lane_out` via the node's
state_updates BEFORE routing to the fallback, so the raw failure text is
already in state. This node normalizes it into a
"PIPELINE-FAULT: <lane> lane failed after retries — …" marker that the map
collects as the lane result; verdict_gate turns any PIPELINE-FAULT lane
result into BLOCKED — a degraded lane can never read as a pass.
Lane identity comes from `lane_ctx` (the map item build_items emitted, which
carries a `lane` field); a missing/malformed item degrades to "unknown".
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
lane = "unknown"
lane_ctx = state.get("lane_ctx")
if isinstance(lane_ctx, dict):
name = lane_ctx.get("lane")
if isinstance(name, str) and name.strip():
lane = name.strip()
raw = state.get("lane_out")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(
json.dumps(
{"lane_out": f"PIPELINE-FAULT: {lane} lane failed after retries — {detail}"}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"lane_out": "PIPELINE-FAULT: lane failed after retries — "
f"fault-marker script error: {e}"
}
)
)
assets/agents/review-gauntlet/scripts/parse_fault.py
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Fail-closed marker for a dead parse stage of the review gauntlet.
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
this node records the failure as a PIPELINE-FAULT in `signals_error` and the
graph jumps straight to the verdict gate. Every lane item list is still its
initial [] so every lane records SKIPPED, and verdict_gate's signals_error
rule makes the verdict BLOCKED — a gauntlet that could not even parse its
request never reads as a pass.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def main():
state = load_state()
raw = state.get("parse_failure")
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
print(
json.dumps(
{
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
f"gauntlet: {detail}"
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
f"gauntlet: fault-marker script error: {e}"
}
)
)
assets/agents/review-gauntlet/scripts/retry_gate.py
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""Deterministic fault-only re-run gate for the review gauntlet.
Sits between the four lane maps and the verdict gate. Every lane map runs
on every pass — a lane that is not being re-run maps over [] — so a re-run
pass overwrites the untouched lanes' collected results with []. The gate's
hard rules:
- every lane that ran this pass is stashed in kept_results, completed
and faulted alike, and a lane whose map ran nothing this pass gets
its stashed report restored — the verdict gate always sees each
lane's LAST report;
- a lane is FAULTED only when its report is missing (the map ran but
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
the verdict gate never reads it as SKIPPED), non-string, the
lane_fault.py marker, sentinel-less, or carries the nested graph's own
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
- a faulted lane is re-run only while it has attempts left AND the
gauntlet is inside its wall-clock budget; otherwise it is recorded in
review_incomplete and its fault flows to the verdict gate as-is;
- malformed review_incomplete bookkeeping (not a list, or non-str
entries) keeps its str entries and records a PIPELINE-FAULT in
signals_error — never silently dropped, never split into lane names;
- the gate itself must never retry by accident: any internal error
records a PIPELINE-FAULT in signals_error and falls through to the
verdict gate, which blocks on it.
"""
import json
import os
import re
import time
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
# = 45600; 600 s of slack below that.
MAX_RETRY_ELAPSED_SECS = 45000
MAX_ATTEMPTS = 3
# lane name -> (items key, results key)
LANES = {
"code-review": ("code_review_items", "code_review_results"),
"adversary": ("adversary_items", "adversary_results"),
"security": ("security_items", "security_results"),
"probe": ("probe_items", "probe_results"),
}
# Must stay identical to verdict_gate.py's sentinel regexes: a lane this
# gate calls completed must be one the verdict gate can parse, or a drift
# would re-run lanes the verdict gate accepts (or accept lanes it cannot
# read). A cross-pin test compares the literals in both files.
SENTINELS = {
"code-review": r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)",
"adversary": r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)",
"security": r"SECURITY_REVIEW:\s*(PASS|FAIL)",
"probe": r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)",
}
# Filled by main() so the crash guard can append to an upstream fault
# instead of overwriting it, and report the lanes it had already declined.
crash_context = {"signals_error": "", "review_incomplete": []}
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
ADVERSARY_DEGRADED_HEADER = re.compile(
r"Criteria: .*("
r"degraded run: pipeline fault recorded"
r"|degraded run: \d+ criterion check\(s\) died"
r"|verdict computation error:"
r")"
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def adversary_degraded(report):
# The adversary's verdict.py emits its degraded-run header on the line
# directly under the FIRST sentinel; anchoring there keeps a report that
# merely quotes the wording later (observations, complaints) a real verdict.
lines = report.splitlines()
for i, ln in enumerate(lines):
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
nxt = lines[i + 1] if i + 1 < len(lines) else ""
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
return False
def nested_degraded(lane, report):
# The nested graphs complete with a sentinel even when a stage inside
# them died; their own degraded-run wording marks a partial review,
# which is not a completed one. Same anchoring as verdict_gate.py.
if lane == "code-review":
verdict_line = re.search(r"^\*\*Verdict: .*$", report, re.MULTILINE)
return bool(
verdict_line
and re.search(
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
verdict_line.group(0),
)
)
if lane == "adversary":
return adversary_degraded(report)
return False
def is_faulted(lane, report):
if not isinstance(report, str):
return True
if report.strip().startswith("PIPELINE-FAULT:"):
return True
if not re.search(SENTINELS[lane], report):
return True
return nested_degraded(lane, report)
def attempt_count(attempts, lane):
try:
return int(attempts.get(lane) or 0)
except (TypeError, ValueError):
return 0
def main():
state = load_state()
prior = state.get("signals_error")
signals_error = prior if isinstance(prior, str) else ""
crash_context["signals_error"] = signals_error
out = {}
declared = state.get("review_incomplete")
if isinstance(declared, list):
incomplete = [name for name in declared if isinstance(name, str)]
malformed = len(incomplete) != len(declared)
else:
incomplete = []
malformed = declared is not None
if malformed:
fault = (
"PIPELINE-FAULT: malformed review_incomplete bookkeeping "
f"({repr(declared)[:80]})"
)
signals_error = f"{signals_error}; {fault}" if signals_error else fault
crash_context["signals_error"] = signals_error
out["signals_error"] = signals_error
crash_context["review_incomplete"] = incomplete
raw_attempts = state.get("lane_attempts")
attempts = dict(raw_attempts) if isinstance(raw_attempts, dict) else {}
kept = dict(state.get("kept_results") or {})
faulted = []
for lane, (items_key, results_key) in LANES.items():
results = state.get(results_key) or []
if not state.get(items_key):
if not results and lane in kept:
out[results_key] = [kept[lane]]
continue
report = results[0] if results else None
if report is None:
report = f"PIPELINE-FAULT: {lane} lane produced no report"
out[results_key] = [report]
kept[lane] = report
if is_faulted(lane, report):
faulted.append(lane)
attempts[lane] = attempt_count(attempts, lane) + 1
# A missing/zero stamp reads as an ancient start and declines every
# retry — the safe direction when the clock's origin is unknown.
started = state.get("gauntlet_started_at") or 0
within_budget = (time.time() - float(started)) <= MAX_RETRY_ELAPSED_SECS
retry = [lane for lane in faulted if within_budget and attempts[lane] < MAX_ATTEMPTS]
declined = [lane for lane in faulted if lane not in retry]
incomplete = sorted(set(incomplete) | set(declined))
crash_context["review_incomplete"] = incomplete
out.update(
{
"lane_attempts": attempts,
"kept_results": kept,
"retry_lanes": retry,
"review_incomplete": incomplete,
}
)
if retry:
out["_next"] = "build_items"
print(json.dumps(out))
try:
main()
except Exception as e: # noqa: BLE001 — a gate crash must fall through, never retry
fault = f"PIPELINE-FAULT: retry gate crashed — {e}"
if crash_context["signals_error"]:
fault = f"{crash_context['signals_error']}; {fault}"
print(
json.dumps(
{
"retry_lanes": [],
"review_incomplete": crash_context["review_incomplete"],
"signals_error": fault,
}
)
)
assets/agents/review-gauntlet/scripts/signals.py
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Deterministic diff signals for the review gauntlet.
Pure git — no LLM judgment. Every signal is computed from the changed-file
list and the added lines of the diff. Failures never sink the gauntlet:
on any error the script reports `signals_error` and leaves the defaults
(false/empty) in place, which biases lane selection toward the caller's
context and forced lanes.
"""
import json
import os
import re
import subprocess
import time
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
state = load_state()
proj = os.path.expanduser(
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
)
spec = (state.get("diff_spec") or "worktree").strip()
def git(*args):
r = subprocess.run(
["git", "-C", proj, *args], capture_output=True, text=True, timeout=90
)
if r.returncode != 0:
raise RuntimeError((r.stderr or r.stdout).strip()[:400])
return r.stdout
out = {"project_dir": proj}
# retry_gate measures its wall-clock budget from this stamp; it is set
# before the git work so a failed diff still starts the clock.
if not state.get("gauntlet_started_at"):
out["gauntlet_started_at"] = time.time()
try:
if spec in ("", "worktree"):
names = git("diff", "--name-only", "HEAD")
difftext = git("diff", "HEAD")
else:
names = git("diff", "--name-only", spec)
difftext = git("diff", spec)
files = [f.strip() for f in names.splitlines() if f.strip()]
added = "\n".join(
l for l in difftext.splitlines() if l.startswith("+") and not l.startswith("+++")
)
def any_path(pattern):
return any(re.search(pattern, f, re.I) for f in files)
out.update(
{
"changed_files": files[:200],
"file_count": len(files),
"touches_proto": any_path(r"\.proto$|buf\.(yaml|gen\.yaml)$"),
"touches_migrations": any_path(r"migrations?/|\.sql$"),
# auth(?!or) admits auth/authn/authz/oauth but not "author".
# Over-firing here is the safe direction: it only ADDS a lane.
"touches_auth": any_path(
r"auth(?!or)|login|logout|token|session|permission|rbac|\bacl\b|credential|secret"
)
or bool(
re.search(
r"auth(?!or)|password|token|secret|credential|jwt|session",
added,
re.I,
)
),
"touches_deps": any_path(
r"go\.(mod|sum)$|package(-lock)?\.json$|yarn\.lock$|pnpm-lock\.yaml$"
r"|Cargo\.(toml|lock)$|requirements[^/]*\.txt$|pyproject\.toml$"
r"|Gemfile(\.lock)?$|pom\.xml$|build\.gradle|composer\.(json|lock)$"
),
"touches_exec": bool(
re.search(
r"exec\.Command|subprocess|os/exec|Runtime\.getRuntime"
r"|shell_exec|\beval\(|\bsystem\(|child_process",
added,
)
),
"consumer_surface": any_path(
r"routes?|handlers?|\bapi\b|endpoints?|\bcmd/|\bcli\b|\.proto$"
r"|openapi|swagger|controllers?|graphql"
),
"docs_only": bool(files)
and all(re.search(r"\.(md|rst|txt|adoc)$", f, re.I) for f in files),
"signals_summary": f"{len(files)} file(s) changed (diff: {spec})",
}
)
except Exception as e: # noqa: BLE001 — degraded signals must not sink the gauntlet
out["signals_error"] = (
f"NOTE: diff signals could not be computed ({e}); "
"lane selection falls back to caller context and forced lanes."
)
out["signals_summary"] = "signals unavailable"
print(json.dumps(out))
assets/agents/review-gauntlet/scripts/verdict_gate.py
+397
View File
@@ -0,0 +1,397 @@
#!/usr/bin/env python3
"""Deterministic verdict gate for the review gauntlet.
Parses each configured lane's verdict SENTINEL with a regex — no LLM
re-reads, no judgment. The gate's hard rules:
- a missing sentinel is a LANE FAILURE and blocks — never a pass;
- a non-string lane result (the map collected an object instead of the
lane's rendered text) blocks — it cannot carry a sentinel;
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
detected only when the report IS the lane_fault.py marker or
retry_gate.py's no-report marker, i.e. it STARTS with it — a real
review merely quoting the marker flows to the normal rules) or a
PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
degraded pipeline is never a pass;
- a code-review lane whose OWN verdict line records a pipeline fault
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
the nested graph completed, but with a dead domain lane, verifier, or
synthesis behind it, so its critical count is not trustworthy;
- an adversary lane whose OWN header line (the line directly under its
sentinel) records a degraded run — a pipeline fault, died criterion
checks, or its verdict script's crash stub — blocks as a PIPELINE-FAULT,
not as a DIVERGES finding: the nested graph completed with a dead stage
behind it, so its verdict is not a judgment on the code;
- a non-zero critical count in the code-review report blocks regardless
of the lane's own verdict line — derived from render.py's deterministic
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
count is used only when that summary line is missing;
- NEEDS-HUMAN without 🔴 passes but is surfaced as attention required;
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
treated as PASS or FAIL (fix the environment per the plan's local-run
recipe, then re-run);
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
re-ran it, so the reader can tell a first-pass death from an exhausted
retry budget;
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
incomplete" section: that is a fault to re-run, never a code finding.
Every lane named in review_incomplete is forced to a BLOCKED row, so a
restored earlier report can never turn an incomplete review into a
pass: a non-empty machine line structurally implies GAUNTLET: BLOCKED.
Malformed review_incomplete bookkeeping (not a list, or non-str
entries) is itself recorded as a `pipeline` fault, never dropped;
- the gate itself must never crash into a silent pass: any internal
error emits GAUNTLET: BLOCKED naming the error, with the pipeline
marked incomplete.
"""
import json
import os
import re
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
ADVERSARY_DEGRADED_HEADER = re.compile(
r"Criteria: .*("
r"degraded run: pipeline fault recorded"
r"|degraded run: \d+ criterion check\(s\) died"
r"|verdict computation error:"
r")"
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def adversary_degraded(report):
# The adversary's verdict.py emits its degraded-run header on the line
# directly under the FIRST sentinel; anchoring there keeps a report that
# merely quotes the wording later (observations, complaints) a real verdict.
lines = report.splitlines()
for i, ln in enumerate(lines):
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
nxt = lines[i + 1] if i + 1 < len(lines) else ""
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
return False
def main():
state = load_state()
blockers, attention, reports = [], [], []
rows = {}
malformed = set()
lane_attempts = state.get("lane_attempts")
if not isinstance(lane_attempts, dict):
lane_attempts = {}
def lane_report(key):
arr = state.get(key) or []
if not arr:
return None
item = arr[0]
if isinstance(item, str):
return item
malformed.add(key)
return json.dumps(item)
def record(lane, status, detail=""):
rows[lane] = (status, detail)
def attempt_count(lane):
try:
return int(lane_attempts.get(lane) or 0)
except (TypeError, ValueError):
return 0
def attempts_suffix(lane):
n = attempt_count(lane)
return f" ×{n}" if n > 1 else ""
def fault_excerpt(report):
line = next(
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
)
return line.strip()[:200]
def is_lane_fault(report):
# A genuine lane fault is the ENTIRE report — the lane_fault.py marker
# or retry_gate.py's no-report marker — which always begins with
# "PIPELINE-FAULT:". Matching the marker anywhere would let a real
# review that merely QUOTES it bypass 🔴 counting and sentinel
# parsing.
return report.strip().startswith("PIPELINE-FAULT:")
def lane_fault_blocker(lane, report):
blockers.append(
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
)
record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
def malformed_blocker(lane, report):
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
record(lane, "BLOCKED", "malformed lane result (non-string)")
reports.append((lane, report))
# --- code-review ------------------------------------------------------
cr = lane_report("code_review_results")
if cr is None:
record("code-review", "SKIPPED", "not selected")
elif "code_review_results" in malformed:
malformed_blocker("code-review", cr)
elif is_lane_fault(cr):
lane_fault_blocker("code-review", cr)
reports.append(("code-review", cr))
else:
reports.append(("code-review", cr))
m = re.search(r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)", cr)
# code-reviewer's verdict.py forces NEEDS-HUMAN on its own internal
# faults; render.py emits that as `**Verdict: NEEDS-HUMAN** — <reason>`
# (CONTRACT comments at both emitters). Scoped to the FIRST rendered
# verdict line — render.py emits it before any finding body — so a
# finding that quotes the wording cannot trip it.
verdict_line = re.search(r"^\*\*Verdict: .*$", cr, re.MULTILINE)
m_fault = verdict_line and re.search(
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
verdict_line.group(0),
)
# Same quoted-marker class as the PIPELINE-FAULT prefix anchoring: a
# 🔴 quoted inside finding text or a Changes-table row is not a
# finding. render.py's summary line is the authoritative count; the
# raw count is the fail-closed fallback when that line is absent. The
# LAST match wins: render.py emits the line after every finding body,
# so a quoted look-alike earlier in the report cannot shadow it. The
# pattern is the FULL shape of render.py's line (CONTRACT comment
# there) so a partial look-alike in LLM-authored text never matches.
sums = re.findall(
r"^\*Reviewed \d+ files, found (\d+) critical, \d+ warnings, \d+ suggestions, "
r"\d+ nitpicks \(\d+ deferred by quality bar\)\*$",
cr,
re.MULTILINE,
)
reds = int(sums[-1]) if sums else cr.count("🔴")
if m_fault:
blockers.append(
"code-review: the lane reported an internal PIPELINE-FAULT "
"(degraded review) — a degraded lane is never a pass"
)
if reds:
blockers.append(
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
)
suffix = attempts_suffix("code-review")
if m_fault and reds:
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
elif m_fault:
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
elif reds:
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
elif m and m.group(1) == "MERGE-READY":
record("code-review", "GREEN", "MERGE-READY")
elif m:
attention.append(
"code-review: NEEDS-HUMAN — see the lane report's "
"'Human attention required' section"
)
record("code-review", "GREEN (attention)", "NEEDS-HUMAN, no 🔴")
else:
blockers.append(
"code-review: no Review Verdict sentinel found — the lane failed; "
"a missing verdict is never a pass"
)
record("code-review", "BLOCKED", "missing verdict sentinel")
# --- adversary --------------------------------------------------------
adv = lane_report("adversary_results")
if adv is None:
record("adversary", "SKIPPED", "not selected")
elif "adversary_results" in malformed:
malformed_blocker("adversary", adv)
elif is_lane_fault(adv):
lane_fault_blocker("adversary", adv)
reports.append(("adversary", adv))
else:
reports.append(("adversary", adv))
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
if adversary_degraded(adv):
blockers.append(
"adversary: PIPELINE-FAULT — degraded run, pipeline fault recorded "
"inside the adversary; a degraded lane is never a pass"
)
record(
"adversary",
"BLOCKED",
f"PIPELINE-FAULT (degraded lane{attempts_suffix('adversary')})",
)
elif m and m.group(1) == "CONFORMS":
record("adversary", "GREEN", "CONFORMS")
elif m:
blockers.append(
"adversary: DIVERGES — the implementation does not conform to the plan"
)
record("adversary", "BLOCKED", "DIVERGES")
else:
blockers.append(
"adversary: no ADVERSARIAL_REVIEW sentinel found — lane failed; never a pass"
)
record("adversary", "BLOCKED", "missing verdict sentinel")
# --- security ---------------------------------------------------------
sec = lane_report("security_results")
if sec is None:
record("security", "SKIPPED", "not selected")
elif "security_results" in malformed:
malformed_blocker("security", sec)
elif is_lane_fault(sec):
lane_fault_blocker("security", sec)
reports.append(("security", sec))
else:
reports.append(("security", sec))
m = re.search(r"SECURITY_REVIEW:\s*(PASS|FAIL)", sec)
if m and m.group(1) == "PASS":
record("security", "GREEN", "PASS")
elif m:
blockers.append("security: SECURITY_REVIEW: FAIL — exploitable finding(s)")
record("security", "BLOCKED", "FAIL")
else:
blockers.append(
"security: no SECURITY_REVIEW sentinel found — lane failed; never a pass"
)
record("security", "BLOCKED", "missing verdict sentinel")
# --- probe ------------------------------------------------------------
pb = lane_report("probe_results")
if pb is None:
record("probe", "SKIPPED", "not selected")
elif "probe_results" in malformed:
malformed_blocker("probe", pb)
elif is_lane_fault(pb):
lane_fault_blocker("probe", pb)
reports.append(("probe", pb))
else:
reports.append(("probe", pb))
m = re.search(r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)", pb)
if m and m.group(1) == "PASS":
record("probe", "GREEN", "PASS")
elif m and m.group(1) == "FAIL":
blockers.append("probe: USAGE_PROBE: FAIL — observed behavior contradicts the contract")
record("probe", "BLOCKED", "FAIL")
elif m:
blockers.append(
"probe: INCONCLUSIVE — the ENVIRONMENT could not be established; this is "
"not a code verdict. Follow the plan's local-run recipe to fix the "
"environment and re-run. INCONCLUSIVE is never treated as PASS or FAIL."
)
record("probe", "BLOCKED", "INCONCLUSIVE (environment)")
else:
blockers.append(
"probe: no USAGE_PROBE sentinel found — lane failed; never a pass"
)
record("probe", "BLOCKED", "missing verdict sentinel")
# retry_gate restores an exhausted lane's LAST report, which may be an
# earlier clean verdict — an incomplete lane must block regardless of
# what its row parsed to.
declared = state.get("review_incomplete")
incomplete = set()
malformed_bookkeeping = None
if isinstance(declared, list):
incomplete = {name for name in declared if isinstance(name, str)}
if any(not isinstance(name, str) for name in declared):
malformed_bookkeeping = repr(declared)[:80]
elif declared is not None:
malformed_bookkeeping = repr(declared)[:80]
for lane in sorted(incomplete - {"pipeline"}):
if rows.get(lane, ("", ""))[0] == "BLOCKED":
continue
after = f" after {n} attempt(s)" if (n := attempt_count(lane)) else ""
blockers.append(
f"{lane}: review incomplete{after} — the lane never produced a usable verdict"
)
record(lane, "BLOCKED", f"review incomplete{after}")
# A gauntlet-level fault (build_items crash, dead parse stage) means the
# lanes above were never built — SKIPPED rows alone must not read as PASS.
signals_error = state.get("signals_error")
pipeline_fault = isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error
if pipeline_fault:
blockers.append(f"pipeline: {signals_error}")
incomplete.add("pipeline")
if malformed_bookkeeping is not None:
blockers.append(
"pipeline: PIPELINE-FAULT — malformed review_incomplete bookkeeping "
f"({malformed_bookkeeping}); never a pass"
)
incomplete.add("pipeline")
if "pipeline" in incomplete and not any(b.startswith("pipeline:") for b in blockers):
blockers.append(
"pipeline: review incomplete — a pipeline stage never completed "
"(no fault text recorded)"
)
verdict = "BLOCKED" if blockers or incomplete else "PASS"
incomplete_lines = []
for name in sorted(incomplete):
if name == "pipeline":
incomplete_lines.append(
f"- pipeline: {signals_error}"
if pipeline_fault
else "- pipeline: review incomplete (no fault text recorded)"
)
elif n := attempt_count(name):
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
else:
incomplete_lines.append(f"- {name}: no completed review")
incomplete_line = (
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
if incomplete
else ""
)
md = "\n".join(
["| lane | status | detail |", "|------|--------|--------|"]
+ [f"| {lane} | {status} | {detail} |" for lane, (status, detail) in rows.items()]
)
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
if blockers:
md += "\n\n## Blockers\n" + "\n".join(f"- {b}" for b in blockers)
if attention:
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
if state.get("signals_error"):
md += f"\n\n> {state['signals_error']}"
if incomplete_lines:
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
md += "\n\n## Full lane reports\n"
for lane, rep in reports:
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
print(
json.dumps(
{
"gauntlet_verdict": verdict,
"gauntlet_report": md,
"gauntlet_incomplete_line": incomplete_line,
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the gate must never crash into a silent pass
print(
json.dumps(
{
"gauntlet_verdict": "BLOCKED",
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
}
)
)
assets/agents/security-reviewer/config.yaml
+12 -1
View File
@@ -1,6 +1,6 @@
name: security-reviewer
description: Security analyst - hunts exploitable flaws in a code change (injection, secrets, authz gaps, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Returns a posture-gated PASS/FAIL verdict so POCs aren't held to production strictness. Complements code-reviewer (quality) and adversary (plan conformance). Designed to be delegated to by sisyphus.
version: 1.0.0
version: 1.1.0
auto_continue: true
max_auto_continues: 15
@@ -61,6 +61,17 @@ instructions: |
1. Load `security-review`.
2. Get the diff (inline or via `get_diff`) and identify the changed files.
2b. **Mechanized pre-pass (deterministic, read-only).** Before manual tracing, run whichever of
these exist on PATH and fit the repo — never install anything, skip each silently when absent,
time-box each to ~60s via `execute_command`:
- `gitleaks` on the changed paths (committed-secret candidates);
- `semgrep --config auto --error --quiet` scoped to the changed files (or the repo's own
`.semgrep.yml` when present);
- a dependency audit when a manifest/lockfile changed: `osv-scanner --lockfile <it>`,
`npm audit --omit dev`, `cargo audit`, or `pip-audit` — whichever matches.
Treat every mechanized hit as a LEAD, not a finding: trace it per steps 3-5 and report it only
with the confirmed source→sink path (or secret location) and your own severity calibration.
Note which tools ran/were skipped in the report. Raw scanner output is never pasted as findings.
3. For EACH hunk: identify untrusted-data sources, dangerous sinks, and the mediation (or lack of
it) between them. Apply the skill's hunt checklist (secrets, injection, paths, authn/authz,
deserialization, network, supply chain, crypto, data exposure, resource abuse).
assets/agents/sisyphus/README.md
+4
View File
@@ -47,7 +47,9 @@ Spawnable sub-agents (from `config.yaml`):
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
- **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. A `GAUNTLET_REVIEW_INCOMPLETE:` line under BLOCKED is an infrastructure fault (a lane exhausted its retries, or the pipeline faulted) that Sisyphus escalates to the user — retry / accept NEEDS-HUMAN / abort — never a finding to fix. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed.
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
- **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion.
## Features
@@ -55,7 +57,9 @@ Spawnable sub-agents (from `config.yaml`):
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
- 🔄 **Task Management**: Tracks progress and context across complex operations.
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
- 🧵 **Task-queue bursts**: Dependency-shaped fan-outs (explore → code → verify) may be encoded as `agent__task_create` chains — dependents auto-dispatch when `agent__task_complete` unblocks them; todos remain the tracking source of truth.
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
- 🚦 **Escalation policy**: `escalation_policy=autonomous` pre-delegates decision-making — standing rulings (`~/.config/coyote/share/standing-rulings.md`, recorded via `coyote --macro record-ruling`) and evidence-backed recommended options are taken without pausing and batched into a “Decisions taken autonomously” digest; secrets, money movement, and destructive actions always escalate.
## Plan-Driven Workflows
assets/agents/sisyphus/config.yaml
+134 -3
View File
@@ -1,6 +1,6 @@
name: sisyphus
description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
version: 3.10.0
version: 3.13.0
agent_session: temp
auto_continue: true
@@ -17,10 +17,12 @@ spawnable_agents:
- adversary
- security-reviewer
- probe
- review-gauntlet
- architecture-reviewer
- step-runner
- whetstone
max_concurrent_agents: 40
max_agent_depth: 3
max_agent_depth: 5
inject_spawn_instructions: true
summarization_threshold: 80000
@@ -54,6 +56,9 @@ variables:
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
- name: escalation_policy
description: "How escalation-worthy decisions are handled mid-run: 'interactive' pauses and asks the user; 'autonomous' self-resolves decisions covered by a standing ruling or a clearly recommended option, reports them in a final-report digest, and only pings on hard triggers or novel conflicts"
default: 'interactive'
mcp_servers:
- ddg-search
@@ -229,6 +234,11 @@ instructions: |
Never artificially cap at a small number. If there are 10 distinct things to find, spawn 10 agents. The system limit is the only ceiling that matters.
For dependency-shaped bursts (explore → code → verify), `agent__task_create` chains MAY encode the
ordering — dependents with `agent`+`prompt` auto-dispatch when `agent__task_complete` unblocks them.
Same rules as direct spawning: collect every auto-dispatched agent via `agent__collect`, and an agent
finishing does NOT complete its task — call `agent__task_complete` yourself. Todos remain the tracking source of truth.
If the slices are likely to FEED each other (adjacent modules; one lane's discovery redirects another's search), run the burst in team mode per `parallel-research`: after spawning, send
every explorer the roster of its siblings (`agent__send_message --id <id> --message "Teammate roster: <slice>=<id>, ... Message a teammate directly when you find a fact in THEIR slice;
check your inbox after each search round and before finalizing."`) so cross-slice facts move directly between lanes instead of waiting for your synthesis. Never do this with reviewer
@@ -291,6 +301,60 @@ instructions: |
- Never reuse a verification claim from an earlier report (yours or another agent's) without re-running the command yourself. Prior reports are unverified context, not evidence.
- An honest failure — "gate X failed / could not run, here is the verbatim error" — is an acceptable, preferable deliverable. A success-shaped report with missing evidence poisons every downstream consumer.
### The review gauntlet (preferred): one spawn, every review lane, machine-parsed verdicts
When the `review-gauntlet` agent is installed, run the post-implementation review passes through
it instead of spawning the individual reviewers yourself. Spawn it ONCE whenever the work was
non-trivial (any trigger in the reviewer sections below would fire, or you are unsure whether one
would):
```
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
Project: <absolute repo path>
Diff: <git range like main...HEAD, or 'worktree' for uncommitted changes>
Rigor: <rigor> Security posture: <posture>
Plan / acceptance criteria: <pasted verbatim — enables the adversary lane; omit when none>
Local-run recipe / usage suites: <from the plan — enables the probe lane; omit when none>"
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
```
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the prompt
also carries pasted plan text from the repo under review, and whatever is declared here is
executed with a shell — a trust boundary the prose must not cross.
It computes diff signals deterministically, selects lanes (code-review always; adversary when a
spec/plan is present; security on attack-surface signals; probe on consumer-facing surface with a
run recipe — judgment can only WIDEN the selection, never narrow it), runs the selected lanes in
parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`
with a lane table, blockers, attention items, and every lane's full report.
- `GAUNTLET: BLOCKED` blocks completion. Route each blocker that is a real review finding through
the findings-handling rules in the reviewer sections below (🔴 → coder fix in the SAME session;
DIVERGES → the adversary rules; security FAIL → the security rules; probe INCONCLUSIVE → fix the
ENVIRONMENT per the plan's local-run recipe first — it is never a code verdict). Then spawn a
FRESH gauntlet run; lane selection re-runs from the new diff. A BLOCKED whose only blockers are
review faults (`PIPELINE-FAULT` / `review incomplete` entries) is NOT a code failure — the next
rule owns it.
- `GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]` (the line directly under `GAUNTLET: BLOCKED`)
means the review could NOT be completed: the named lanes still faulted after 3 attempts, or the
gauntlet pipeline itself faulted (`pipeline`). This is an infrastructure fault, not a code
finding. NEVER route it through the findings-handling rules. NEVER claim done while it stands.
Escalate to the user via `user__select` with exactly these three options:
- "Retry the review again"
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
- "Abort this task"
Retry → spawn a FRESH gauntlet run. Accept → finish the work and record the skipped review
(lane names or `pipeline`) in your final report and in the PR body when you author one. Abort →
stop; leave the code in a working state and report. Findings AND incomplete on the same output →
fix the findings first, then re-run the gauntlet (the re-run retries the lane) — escalate only if
the line persists. Under `escalation_policy: autonomous` the standard ladder applies (a standing
ruling may pre-answer this decision).
- `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those
items into your final report verbatim.
- The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the
spawn fallback when `review-gauntlet` is not installed.
### Independent code review (post-coder, non-trivial work)
After completing delegated `coder` work, spawn `code-reviewer` for an independent review pass if ANY of these are true:
@@ -350,8 +414,13 @@ instructions: |
PLAN — acceptance criteria to check against:
<paste the task/step spec + acceptance criteria VERBATIM — not a summary>"
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
```
Same trust boundary as the gauntlet spawn: verification commands are declared via `--variables`,
never in the prompt, because the prompt also carries untrusted pasted text.
### Handling adversary findings
- **`ADVERSARIAL_REVIEW: DIVERGES` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <complaints pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `adversary` ONCE to confirm it now CONFORMS; if it still DIVERGES on the same criteria after one fix cycle, STOP and escalate to the user (the plan or the approach may be wrong — consider `oracle`).
@@ -477,6 +546,22 @@ instructions: |
Do not block completion on observability findings — the failure mode is skipping the pass on applicable surface, not shipping without an alert. Threshold and paging decisions belong to humans; your job is to make them informed and cheap.
### Continuous improvement (whetstone) — after addressing human review feedback
When a run's work included addressing review feedback that did NOT come from your own
reviewer lanes — human comments and requested changes, but also a teammate's review
agent or another team's bot (an independent reviewer catching what your suite missed is
exactly the signal) — spawn `whetstone` ONCE at the very end, after all work is
complete. Pass it: the PR reference,
the human feedback items you addressed (verbatim), and what your own review passes
(code-reviewer/adversary/security-reviewer/probe) flagged for the same change — that
ground truth is what makes its miss-classification precise. Collect its result; its
proposals go to the USER (it escalates its own approval questions), not to you — never
apply suite changes on its behalf and never block run completion on it (if it errors,
note that and finish). Skip with a one-line note ONLY when every piece of feedback you
addressed came from your own reviewer lanes: whetstone never learns from its own suite's
output, but it learns from everyone else — human or agent.
## File Operations (Direct Edits)
When you write or modify files yourself (rather than delegating to coder):
@@ -570,9 +655,49 @@ instructions: |
- Writing non-trivial code → always `coder` (graph agent runs verification internally)
- Multi-angle exploration → fan out `explore` agents
## Escalation policy: {{escalation_policy}}
Every "ASK the user" / "STOP and escalate" instruction in this config is subject to this policy.
- **interactive** (default) — pause and ask exactly as written throughout this config.
- **autonomous** — the user pre-delegated decision-making for this run ("here's the spec — go").
When an escalation-worthy decision arises, work this ladder IN ORDER:
1. **Hard triggers — ALWAYS escalate; no ruling or policy can delegate these:** decisions about
secrets/credentials, money movement, destructive or irreversible actions (data loss,
force-push, dropping data or public API without a migration path), anything the Hard Blocks
section forbids — and any decision that already failed once after an autonomous resolution
(a ruling that did not survive one fix cycle goes to a human, never into a retry loop).
2. **Standing rulings** — read `$HOME/.config/coyote/share/standing-rulings.md` (skip silently
if absent). If a ruling's trigger GENUINELY matches the decision at hand, apply its
resolution. Match on substance, not resemblance — a ruling about security-vs-spec conflicts
does not authorize scope changes. Unsure whether it matches? Then it does not.
3. **Clearly recommended option** — if you would have presented options with one marked
`(Recommended)` backed by concrete evidence (a reviewer's proposed fix, a failing test that
names the cause, a repo convention), take the recommended option.
4. **Otherwise — escalate** exactly as interactive mode would. A genuinely novel conflict or a
decision with no defensible recommendation still reaches the user; autonomy never converts
"I don't know" into a guess.
**Plan amendments taken autonomously.** When a resolution amends the spec/plan/acceptance
criteria (e.g. a security fix conflicts with a criterion's wording), REWRITE the affected
criterion text in your working plan context BEFORE re-running any reviewer or the gauntlet —
the adversary must verify the plan-as-amended, not the stale wording. The amendment is always
a visible artifact, never implied.
**The digest (MANDATORY when any autonomous decision was taken).** Your final report — and the
PR body, when you open one — MUST include a `## Decisions taken autonomously` section: one line
per decision stating what was decided, the basis (`ruling: <id/title>` or `recommended option`),
and what changed (including any plan amendment). Zero autonomous decisions → omit the section.
This converts interrupt-driven review into digest-driven review — never into silence.
**Close the loop.** When a decision DID reach the user (step 4), suggest in your final report
that they record the resolution with `coyote --macro record-ruling` so that class of decision
never asks again.
## User Interaction (get buy-in before major decisions)
Use `user__ask`, `user__confirm`, `user__checkbox`, `user__input` to clarify ambiguities interactively. **Do NOT guess when you can ask.**
Use `user__ask`, `user__confirm`, `user__checkbox`, `user__input` to clarify ambiguities interactively. **Do NOT guess when you can ask.** When `escalation_policy` is `autonomous`, the Escalation policy section above governs which of these actually pause the run — autonomy substitutes standing rulings and evidence-backed recommendations for the pause; it never substitutes guessing.
| Situation | Tool |
|-----------|------|
@@ -603,6 +728,11 @@ instructions: |
If you see `pending_escalations` in tool results, a child agent needs user input and is blocked. Reply promptly via `agent__reply_escalation`. You can answer from context, or prompt the user yourself first and relay the answer.
In `autonomous` escalation policy, work the Escalation-policy ladder before prompting the user: a
matching standing ruling or a clearly recommended option in the child's escalation answers it
directly via `agent__reply_escalation` (hard triggers always go to the user). Replies that amend a
plan/spec follow the amendment rule above, and every autonomous reply lands in the digest.
## Anti-Patterns (BLOCKING)
- Skipping intent verbalization → unclear routing, wasted turns
@@ -630,6 +760,7 @@ instructions: |
## Context
- Project: {{project_dir}}
- Escalation policy: {{escalation_policy}}
- OS: {{__os__}}
- Shell: {{__shell__}}
- CWD: {{__cwd__}}
assets/agents/step-runner/README.md
+32 -2
View File
@@ -27,6 +27,7 @@ flowchart TD
gate_blocked -->|"yes"| orient
gate_blocked -->|"no"| end_blocked
orient["orient<br/>llm, read-only"] --> route_staleness
orient -. "LLM fault" .-> note_llm_fault
route_staleness{"route_staleness<br/>script"}
route_staleness -->|"major deviation"| gate_deviation
route_staleness -->|"else"| implement
@@ -35,6 +36,7 @@ flowchart TD
gate_deviation -->|"abort"| end_rejected
gate_deviation -->|"other (user guidance)"| implement
implement[["implement<br/>agent → coder"]] --> route_coder_result
implement -. "coder crashed<br/>(no retries)" .-> end_failure
route_coder_result{"route_coder_result<br/>script"}
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
route_coder_result -->|"REJECTED / FAILED"| end_failure
@@ -51,14 +53,18 @@ flowchart TD
fix_loop_gate -->|"budget left"| implement
fix_loop_gate -->|"budget spent"| end_failure
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
edge_case_sweep -. "LLM fault" .-> write_handoff
route_sweep{"route_sweep<br/>script"}
route_sweep -->|"5+ files or boundary"| independent_review
route_sweep -->|"else"| write_handoff
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
independent_review -. "2 attempts failed" .-> write_handoff
route_review{"route_review<br/>script"}
route_review -->|"🔴 critical findings"| implement
route_review -->|"else"| write_handoff
write_handoff["write_handoff<br/>llm"] --> check_handoff
write_handoff -. "LLM fault" .-> note_llm_fault
note_llm_fault{"note_llm_fault<br/>script"} --> end_failure
check_handoff{"check_handoff<br/>script"}
check_handoff -->|"schema valid"| gate_user_review
check_handoff -->|"one retry"| write_handoff
@@ -81,8 +87,32 @@ End nodes emit sentinel outcomes for the caller:
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
was rejected at its approval gate.
- `STEP_FAILED` — coder failed, the step-level fix budget was exhausted, or
the handoff failed validation twice.
- `STEP_FAILED` — the coder failed or crashed, the step-level fix budget was
exhausted, the handoff failed validation twice, or an orient/handoff LLM
fault was recorded (rendered as `Pipeline fault:` in the output).
## Fault handling
Node crashes degrade the run visibly instead of masquerading as success:
- `implement` has **no retries** — rerunning a coder that died mid-edit
against an already-mutated tree is unsafe. Its failure text lands in
`coder_result` and the run falls straight to `end_failure`.
- `independent_review` retries once (`max_attempts: 2`); if both attempts
fail it falls back to `write_handoff`, which flags "PIPELINE-FAULT:
independent review did not run" in the handoff rather than presenting the
failure text as findings. `route_review` carries the same guard before its
🔴 grep, so a reviewer fault never spends a fix-loop attempt.
- `edge_case_sweep` faults likewise fall through to `write_handoff` and are
flagged as "PIPELINE-FAULT: edge-case sweep did not run".
- `orient` and `write_handoff` faults route through `note_llm_fault`, which
distills the engine's failure text into a `fault_note` rendered in the
`STEP_FAILED` output — never an empty shell.
Degraded is never passing (a skipped review or sweep is recorded as NOT RUN,
never paraphrased as covered), and the sentinel is always emitted — every
path terminates at one of the four end nodes, so the caller always gets a
parseable outcome.
## Usage
assets/agents/step-runner/graph.yaml
+62 -2
View File
@@ -6,6 +6,12 @@ description: |
review -> evidence-backed handoff -> user approval gate. Designed to be
delegated to by sisyphus.
version: '1.0'
# The limit is consulted by the SPAWNER against the child's absolute depth
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
# step-runner(2) → code-reviewer(3) →
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
# This agent spawns depth-3 children; 4 = one level of headroom (default 3 had none).
max_agent_depth: 4
global_tools:
- ast_grep.sh
- fs_cat.sh
@@ -64,6 +70,7 @@ initial_state:
plan_summary: ''
implementation_brief: ''
staleness_report: ''
orient_failure: ''
has_major_deviation: false
deviation_summary: ''
user_feedback: ''
@@ -80,12 +87,15 @@ initial_state:
tests_output: ''
edge_case_report: ''
downstream_updates: ''
sweep_failure: ''
needs_independent_review: false
review_report: ''
review_attempts: 0
max_review_attempts: 1
handoff_attempts: 0
handoff_fix: ''
handoff_failure: ''
fault_note: ''
step_summary: ''
start: resolve_step
@@ -191,7 +201,9 @@ nodes:
has_major_deviation,
deviation_summary,
]
fallback: end_failure
state_updates:
orient_failure: '{{output}}'
fallback: note_llm_fault
next: route_staleness
route_staleness:
@@ -270,6 +282,12 @@ nodes:
timeout: 3600
state_updates:
coder_result: '{{output}}'
# No max_attempts: rerunning a coder that died mid-edit against an
# already-mutated tree is unsafe. On failure the "Agent node failed:"
# text lands in coder_result (via state_updates above) and end_failure
# renders it; end_failure's other keys all have initial_state defaults,
# so the STEP_FAILED output degrades to empty build/tests sections.
fallback: end_failure
next: route_coder_result
route_coder_result:
@@ -376,6 +394,8 @@ nodes:
needs_independent_review:
type: boolean
required: [edge_case_report, downstream_updates, needs_independent_review]
state_updates:
sweep_failure: '{{output}}'
fallback: write_handoff
next: route_sweep
@@ -405,8 +425,10 @@ nodes:
Review the changed files against the step plan's acceptance criteria.
Preserve severity tags in your findings.
timeout: 1200
max_attempts: 2
state_updates:
review_report: '{{output}}'
fallback: write_handoff
next: route_review
route_review:
@@ -450,6 +472,25 @@ nodes:
If "Gate feedback" below is non-empty, a previous handoff attempt
failed validation - fix exactly what it lists.
PIPELINE-FAULT VISIBILITY: an "Independent review report" below that
starts with "Agent node failed:" means the independent review DID NOT
RUN - the graph routed here through the reviewer's failure fallback.
Never present that text as review findings. Flag it prominently
instead: add a Deviations entry "PIPELINE-FAULT: independent review
did not run - <failure text>", repeat it under Notes for next step,
and carry it into `step_summary` so the user sees it at the approval
gate.
The same rule covers the "Edge-case sweep result" below: text starting
with "LLM node" means the edge-case sweep DID NOT RUN - the graph
routed here through the sweep's failure fallback. Never present it as
sweep findings. Add a Deviations entry "PIPELINE-FAULT: edge-case
sweep did not run - <failure text>" and carry it into `step_summary`
so the user sees it at the approval gate. If it holds successful
structured output or is empty, ignore it - the sweep's real results
already flow through the "Edge cases discovered" / "Downstream plan
updates" sections above.
prompt: |
## Step
{{step_number}} ({{step_title}}) - plan at {{step_plan_path}}
@@ -473,6 +514,9 @@ nodes:
## Downstream plan updates
{{downstream_updates}}
## Edge-case sweep result (fault detection)
{{sweep_failure}}
## Independent review report (if any)
{{review_report}}
@@ -501,9 +545,24 @@ nodes:
type: string
description: 3-6 sentence summary of the step for the user's approval decision - what was done, deviations, anything needing their attention
required: [step_summary]
fallback: end_failure
state_updates:
handoff_failure: '{{output}}'
fallback: note_llm_fault
next: check_handoff
note_llm_fault:
id: note_llm_fault
type: script
description: |
Make LLM-node fallback captures visible: when orient or write_handoff
failed (their capture keys hold the engine's "LLM node..." failure
text), distill it into fault_note so end_failure can render WHY the
step died instead of an empty STEP_FAILED shell.
script: scripts/note_llm_fault.sh
timeout: 5
fallback: end_failure
next: end_failure
check_handoff:
id: check_handoff
type: script
@@ -601,6 +660,7 @@ nodes:
Step: {{step_number}} ({{step_title}})
Fix attempts: {{fix_attempts}}/{{max_fix_attempts}}
Blocking reason (if resolution failed): {{blocking_reason}}
Pipeline fault (if any): {{fault_note}}
Coder result:
{{coder_result}}
assets/agents/step-runner/scripts/note_llm_fault.sh
Executable
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ -n "${GRAPH_STATE_FILE:-}" ]]; then
state=$(cat "$GRAPH_STATE_FILE")
elif [[ -n "${GRAPH_STATE:-}" ]]; then
state="$GRAPH_STATE"
else
state='{}'
fi
orient_failure=$(echo "$state" | jq -r '.orient_failure // ""')
handoff_failure=$(echo "$state" | jq -r '.handoff_failure // ""')
# Engine failure text is always prefix-anchored "LLM node" (llm.rs); on
# success these keys hold the node's structured JSON output, which never
# starts with that prefix.
if [[ "$orient_failure" == "LLM node"* ]]; then
fault_note="PIPELINE-FAULT: orient stage failed — ${orient_failure:0:300}"
elif [[ "$handoff_failure" == "LLM node"* ]]; then
fault_note="PIPELINE-FAULT: handoff stage failed — ${handoff_failure:0:300}"
else
fault_note=""
fi
jq -nc --arg note "$fault_note" '{"fault_note": $note}'
assets/agents/step-runner/scripts/route_review.sh
+10
View File
@@ -13,6 +13,16 @@ review_report=$(echo "$state" | jq -r '.review_report // ""')
review_attempts=$(echo "$state" | jq -r '.review_attempts // 0')
max_review_attempts=$(echo "$state" | jq -r '.max_review_attempts // 1')
# Fail-visible guard: a review_report holding the engine's "Agent node
# failed:" text means the independent review DID NOT RUN (its fallback
# normally routes straight to write_handoff, bypassing this script). Never
# treat that text as findings or spend a fix-loop attempt on it -
# write_handoff flags the fault prominently in the handoff instead.
if [[ "$review_report" == "Agent node failed:"* ]]; then
jq -nc '{"_next": "write_handoff"}'
exit 0
fi
if ! grep -qF "🔴" <<< "$review_report"; then
jq -nc '{"_next": "write_handoff"}'
exit 0
assets/agents/whetstone/README.md
+67
View File
@@ -0,0 +1,67 @@
# whetstone
> The review suite reviews code. Whetstone reviews the review.
A continuous-improvement agent for the review stack: it takes the human review feedback a
PR actually received, compares it against what the automated suite flagged for the same
change, and turns every generalizable miss into a permanent upgrade — a new rule in the
generic review skills when the class is portable, a review-miss ledger entry when it's
org-specific. Nothing lands without the user approving each proposal.
## Why it exists
An empirical gap study of automated-review misses vs human review feedback showed the suite's misses
cluster into learnable classes — and that a miss, once *recorded*, converts to a catch on
the next review. The study was a one-time manual harvest; whetstone is the same loop run
continuously, at the moment the signal is strongest: when human feedback on a PR is being
addressed, the spawner (sisyphus/architect) knows BOTH what the humans said and what the
suite's own lanes flagged — ground truth that offline comment mining could only re-infer from formats.
## How it fits
- **Spawned by** sisyphus (after a run that addressed human review feedback) or architect
(same trigger at run end), or invoked directly against a PR. Runs AFTER review, never
during — it is not a reviewer lane, never posts to the PR, and never judges the code.
- **Classifies** each retained finding as CATCH / PARTIAL / MISS against the suite's
findings. Lane hygiene excludes only what would be circular: author self-replies and
the suite's OWN posted output. A teammate's review agent or another team's bot is an
independent reviewer — its findings count as miss signal when the author actually
addressed them (ignored agent comments are unvalidated noise and are dropped).
- **Generalization gate** for each miss, in order: taste → dropped; one-off → appended to
the candidates inbox (`share/review-miss-candidates.jsonl`) for recurrence tracking;
org-specific class → drafted as a ledger entry; general class → drafted as a suite edit
in house style. General rules must pass the noise firewall: concrete check, crisp
trigger, zero org nouns, not already covered (sharpen over duplicate).
- **Propose → approve → apply**: every proposal shows the quoted evidence, the class
statement, and the full draft; the user checkboxes what to apply. Ledger entries append
idempotently. Suite edits go where the suite lives: with a `suite_repo` configured (a
source-of-truth bundle repo), edits land in its working tree — never
committed (that stays a human act) — and one further approval mirrors them into the
live runtime config, drift-guarded (runtime files diverging from the pre-edit repo
version are reported and skipped; runtime-only changes are never made). Without a
`suite_repo` — most users, whose live config IS the suite — edits apply directly under
`runtime_config`, with per-file backups to `share/whetstone-backups/` as the revert
path. Empty proposal lists are a fine outcome; whetstone never invents findings to
justify its spawn.
## Relationship to the other ledger tools
| Tool | Sees | When |
|------|------|------|
| `whetstone` | Human feedback on PRs YOUR runs addressed, with suite ground truth | Real-time, per run |
| Review-round reconciliation | Human-caught, panel-missed defects on any PR the panel re-reviews → candidates inbox (`share/review-miss-candidates.jsonl`) | Automatic, every round |
| `record-miss` | Defects that escaped review entirely (prod, incidents) | One-off, when it bites |
Collection is automatic end to end: whetstone captures misses the moment feedback is
addressed, round reconciliation captures them on any PR the panel re-reviews (teammates'
PRs included), and `record-miss` covers the misses no one ever commented on. Nothing
auto-appends to the LEDGER itself — candidates wait in the inbox until a human promotes them.
## Rules that keep it safe
Additive only (never weakens or removes an existing rule); writes only to the suite
skills/review-agent configs (repo when configured, live config otherwise — always
backed up first), the ledger, the candidates inbox, and — with its own approval,
drift-guarded — the runtime mirror of this run's applied repo edits; never commits,
never pushes, never touches source repos, never makes runtime-only changes when a
suite repo is configured; one proposal per defect class; verbatim citations only.
assets/agents/whetstone/config.yaml
+158
View File
@@ -0,0 +1,158 @@
name: whetstone
description: Continuous-improvement agent for the review suite — classifies human review feedback the suite missed, generalizes misses into defect classes, and proposes suite edits (general) or ledger entries (org-specific), applying them only with user approval
version: 1.3.0
skills_enabled: true
enabled_skills:
- review-misses
enabled_tools:
- fs_read
- fs_grep
- fs_glob
- fs_cat
- fs_ls
- fs_write
- fs_patch
- execute_command
variables:
- name: suite_repo
description: OPTIONAL source-of-truth bundle repo (a working tree containing skills/ and agents/). When set, approved GENERAL improvements are written here first, then optionally mirrored to the runtime config. When empty, improvements are applied DIRECTLY to runtime_config — for most users the live config IS the suite.
default: ''
- name: runtime_config
description: The live coyote config dir the review suite actually runs from. Direct write target when suite_repo is empty (with per-file backups for revert); drift-guarded mirror target when suite_repo is set. Empty = never touch it (propose-only).
default: '~/.config/coyote'
- name: pr
description: The PR to analyze (owner/repo#N or a full URL). May be omitted when the spawner passes the feedback inline.
default: ''
instructions: |
You are WHETSTONE — the review suite reviews code; you review the REVIEW. Your input is a
PR's human review feedback plus what the automated review suite flagged; your output is
the suite made sharper: every generalizable miss becomes either a rule in the generic
review skills (portable to any repo) or a review-miss ledger entry (org-specific), so the
same class is never missed twice. You are NOT a reviewer lane: you never post to the PR,
never judge the author's code, and never run during a review — only after humans have
spoken.
## Inputs
Your spawner (sisyphus, architect, or the user directly) provides some or all of:
- The PR reference (`{{pr}}` or in the prompt).
- The human review feedback that was addressed (verbatim, if the spawner has it).
- What the suite's own review passes flagged for the same change (ground truth — the
spawner has this when it ran the reviewers; use it verbatim and do not re-infer).
Whatever is missing, gather yourself with gh (ALWAYS `export GH_NO_UPDATE_NOTIFIER=1;
unset CLICOLOR_FORCE` first — the sandbox forces ANSI color into piped JSON otherwise):
inline review comments, review bodies, and the commits that addressed them. When no
suite-findings ground truth was passed, identify the suite's own posted output on the PR
by FORMAT (severity emoji like 🔴/🟡/🟢, lane tags like `[adversary]`, panel/verdict
tables) — automated reviews are often posted from a personal account, so judge by
format, not username.
## Workflow
1. **Lane hygiene.** The exclusion rule is about CIRCULARITY, not automation. Drop: the
PR author's self-replies, and YOUR OWN suite's output (by ground truth from the
spawner, or by the suite's format markers + posting account — learning from your own
lanes teaches nothing). KEEP third-party automated reviewers (a teammate's review
agent, other orgs' bots): they are independent reviewers, and what they catch that
your suite didn't is exactly the miss signal you exist for — but only their findings
the author actually ADDRESSED (a fix commit or an accepting reply). An addressed
third-party finding is validated the way a human comment implicitly is; an ignored
one is unvalidated noise — drop it. Report what you excluded and why. Nothing
retained → report "nothing to learn from" and STOP. Never invent findings to justify
having been spawned.
2. **Classify each retained finding** (human, or validated third-party) against the
suite's findings for the same change:
- `CATCH` — a suite lane flagged the same issue (any wording). Skip.
- `PARTIAL` — a lane flagged the area but missed the human's specific point.
- `MISS` — no lane flagged it. PARTIAL and MISS proceed to step 3.
3. **The generalization gate** — for each MISS/PARTIAL, in this order:
a. **Taste?** (naming preference, style debate, "I'd structure it differently") →
DROP. List dropped items in the report; never encode taste anywhere.
b. **One-off?** (real defect, but no articulable class — a typo-level slip, a
this-code-only fact) → append one line to the candidates inbox
`$HOME/.config/coyote/share/review-miss-candidates.jsonl`
(`{"date","pr","quote","note"}`, plain shell append) and move on. If later runs
show the same shape, THAT is your recurrence evidence for a class.
c. **Org-specific class?** (org conventions — especially ones repo prior art
contradicts — live org state, product/domain semantics not derivable from any
repo) → draft a LEDGER ENTRY in the `review-misses` skill's exact format; the
detection pattern is the payload.
d. **General class?** (portable to any repo, any org) → draft a SUITE EDIT: name the
exact file and section in the suite repo (a review skill's checklist, an agent
config's trigger list), and write the edit in HOUSE STYLE — read the surrounding
section first and match its severity markers, exemption language, and
"reports even when clean" conventions. Additive only.
4. **The generality bar** (the noise firewall — a general rule that fires on every diff
will get the whole suite ignored). A suite-edit proposal must pass ALL of:
- It states the CONCRETE check a reviewer runs (a grep shape or a decidable
question), not advice ("be careful with X" fails).
- It has a crisp TRIGGER and would stay silent on diffs where the class is
impossible.
- It contains zero org/product nouns.
- It is not already covered: grep the suite's skills first; if an existing rule
half-covers it, propose SHARPENING that rule instead of adding a sibling.
- Check the ledger too: if the miss matches an already-RECORDED entry, the finding
is "recorded pattern failed to fire" — a suite-wiring problem to report, not a new
entry.
5. **Propose — never write unbidden.** Present every surviving proposal to the user:
the human's feedback quoted verbatim with the PR link, the class statement (one
sentence: what general thing was missed), and the full remediation draft (ledger
entry or suite diff). Ask which to apply via a checkbox selection. An empty proposal
list is a fine outcome — say so and stop.
6. **Apply what was approved.**
- Ledger entries: idempotent append to `$HOME/.config/coyote/share/review-misses.md`
(title-match skip; numbering via `grep -c '^## MISS-'` — the `review-misses` skill
has the recipe).
- Suite edits — the target depends on where the suite lives:
- **`suite_repo` set and exists** (a source-of-truth bundle repo): snapshot each
original to a temp dir (needed for the runtime drift check below), write the
edit into the repo's working tree with fs_patch, and NEVER commit or push — the
user reviews and commits. Then offer the runtime mirror (next bullet).
- **`suite_repo` empty — the live config IS the suite.** Apply the edit directly
under `{{runtime_config}}` (the review skills and review-agent configs there)
with fs_patch. FIRST back up each original to
`{{runtime_config}}/share/whetstone-backups/<UTC timestamp>/<relative path>` so
any edit can be reverted, and state the backup location in your report. If
`{{runtime_config}}` is itself a git repo, list the files you changed so the
user can commit them. No extra approval beyond the proposal checkbox — there is
no repo/runtime split to guard. Edits take effect on the NEXT session load; say
so.
- Both empty or missing: emit the exact edits (file, section, full text) for
manual application instead.
- **Runtime mirror (repo mode only, separately approved).** Repo edits are inert until the
runtime config is synced. After suite edits are applied, if `{{runtime_config}}`
is set and exists, ask the user ONCE (yes/no) whether to mirror the just-approved
edits into it. On yes, for EACH file you edited: locate the runtime counterpart
(same relative path under agents/ or skills/); if it is byte-identical to your
PRE-EDIT snapshot, copy the edited repo file over it; if it differs (the user's
runtime carries local changes), SKIP it and report the drift — never overwrite a
drifted runtime file, the user reconciles those. Mirror ONLY files you edited this
run, and ONLY as a copy of the applied repo edit — never make a runtime-only
change (the next repo→runtime sync would silently clobber it and the improvement
would be lost). If the user declines, remind them the edits are inert until they
sync.
7. **Report**: a table of human findings with CATCH/PARTIAL/MISS verdicts; proposals
made / approved / applied; runtime files mirrored vs skipped-for-drift; items
dropped as taste; inbox appends. CATCH-rate over
time is the suite's health metric — state it.
## Rules
- ADDITIVE ONLY: never weaken, remove, or lower the severity of an existing rule. If a
proposal contradicts an existing rule, report the conflict for the user to resolve.
- Scope of writes: the suite's `skills/` and review-agent configs (in the suite repo,
or directly in the runtime config when no repo is configured — always with backups),
the ledger, and the candidates inbox. Nothing else, ever — no source repos, no
commits, no PR comments. When a `suite_repo` IS configured, never make a
runtime-only change: the next repo→runtime sync would silently clobber it and the
improvement would be lost.
- One proposal per defect CLASS — merge repeat instances and cite all their PRs.
- Quote real evidence verbatim; never fabricate PR citations.
- You may be spawned as a subagent: your user__ questions escalate to the root agent's
user. That is the intended approval path — never skip the approval step because you
are "just a subagent".
assets/config-template.yaml
+10 -1
View File
@@ -173,10 +173,12 @@ agent_session: null # Set a session to use when starting an agent (
save_session: null # Controls the persistence of the session. If true, auto save; if false, don't auto-save; if null, ask the user what to do
compression_threshold: 4000 # Compress the session when the token count reaches or exceeds this threshold
compression_keep_last: 0 # Number of most-recent messages to keep visible after compression (0 = compress all messages)
compression_model: null # Model to use for session compression (e.g. 'openai:gpt-4o-mini'); defaults to the session's current model
summarization_prompt: null # The text prompt used for creating a concise summary of session messages. If null, uses built-in default
summary_context_prompt: null # The text prompt used for including the summary of the entire session as context to the model. If null, uses built-in default
max_tool_result_chars: null # Cap on tool result characters forwarded to the model per call (null = no cap)
max_concurrent_jobs: null # Max background jobs (`job__*` tools) running at once per context (null = 5; 0 disables background jobs entirely)
tool_timeout: null # Max seconds a tool call may run before being killed (null = 1800; 0 = unlimited). COYOTE_TOOL_TIMEOUT env var overrides
# ---- Memory ----
# See the [Memory documentation](https://github.com/Dark-Alex-17/coyote/wiki/Memory) for more information.
@@ -239,7 +241,11 @@ theme: null # null = the built-in dark theme; set to `light
left_prompt: null # If null, uses the built-in default:
# '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
right_prompt: null # If null, uses the built-in default:
# '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
# '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?total_cost ${total_cost} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
# Token-usage variables (set only when the provider reports usage, so guard them with {?var ...}):
# last_input_tokens, last_output_tokens, last_cache_read_tokens, last_cache_creation_tokens (most recent API call)
# total_input_tokens, total_output_tokens, total_cache_read_tokens, total_cache_creation_tokens (session cumulative)
# last_cost, total_cost (USD, formatted like 0.0142; set only when nonzero and every billed token type has a price)
# ---- Miscellaneous ----
user_agent: null # Set User-Agent HTTP header, use `auto` for coyote/<current-version>
@@ -281,4 +287,7 @@ sync_models_ttl_hours: 24 # Hours before the synced model catalog is cons
# proxy: socks5://127.0.0.1:1080 # Set proxy
# connect_timeout: 10 # Set timeout in seconds for connect to api
# read_timeout: 300 # Set timeout in seconds for a read stall (no bytes received); 0 disables (default: 300)
#
# The claude, bedrock, and vertexai clients also support:
# prompt_cache: true # Enable Anthropic prompt caching (default: true)
__CLIENTS_BLOCK__
assets/functions/tools/fs_grep.sh
Regular → Executable
+5 -1
View File
@@ -59,7 +59,11 @@ main() {
# (they only matter when recursing into a directory tree).
local results
results=$(grep "${grep_args[@]}" -E "$search_pattern" "$search_path" 2>/dev/null | head -n "$MAX_RESULTS") || true
# tr strips NUL bytes from matches in real binary files (text mode searches
# them), which would otherwise trigger bash's "ignored null byte in input"
# warning in the command substitution. -I can't be used instead: it overrides
# --binary-files=text and re-skips the misdetected UTF-8 files noted above.
results=$(grep "${grep_args[@]}" -E "$search_pattern" "$search_path" 2>/dev/null | tr -d '\0' | head -n "$MAX_RESULTS") || true
if [[ -z "$results" ]]; then
echo "No matches found for: $search_pattern" >> "$LLM_OUTPUT"
assets/functions/tools/web_search_coyote.sh
+65 -13
View File
@@ -8,10 +8,17 @@ set -e
# @meta require-tools coyote
# @env WEB_SEARCH_MODEL=gemini:gemini-2.5-flash The model for web-searching.
# @env WEB_SEARCH_MODEL=gemini:gemini-2.5-flash The fallback model for web-searching.
#
# supported coyote models:
# - gemini:gemini-2.0-*
# Model selection: when WEB_SEARCH_USE_CURRENT_MODEL is true (the default) and
# the running session's model (COYOTE_CURRENT_MODEL, injected by coyote for
# every tool call) natively supports web search, the search runs on the current
# model first. If the current model is unsupported, or if the search fails at
# runtime, the tool falls back to WEB_SEARCH_MODEL. If that also fails, the
# tool fails.
#
# supported coyote models (native web search):
# - gemini:*
# - vertexai:gemini-*
# - perplexity:*
# - ernie:*
@@ -20,29 +27,74 @@ set -e
# client — the codex OAuth path uses the
# Responses API where this parameter
# does not exist)
# @env WEB_SEARCH_USE_CURRENT_MODEL=true Prefer the running session's model when it natively supports web search.
# @env LLM_OUTPUT=/dev/stdout The output path
# shellcheck disable=SC2154
main() {
client="${WEB_SEARCH_MODEL%%:*}"
# Returns 0 when the model natively supports web search via a known mechanism.
supports_native_search() {
case "$1" in
gemini:* | vertexai:gemini-* | perplexity:* | ernie:* | claude:*) return 0 ;;
openai:gpt-4o*search-preview*) return 0 ;;
*) return 1 ;;
esac
}
if [[ "$client" == "gemini" ]]; then
# Exports the request-body patch that enables native web search for the
# given model's client (clearing any patch left over from a prior attempt).
export_search_patch() {
unset COYOTE_PATCH_GEMINI_CHAT_COMPLETIONS \
COYOTE_PATCH_VERTEXAI_CHAT_COMPLETIONS \
COYOTE_PATCH_ERNIE_CHAT_COMPLETIONS \
COYOTE_PATCH_CLAUDE_CHAT_COMPLETIONS \
COYOTE_PATCH_OPENAI_CHAT_COMPLETIONS
case "${1%%:*}" in
gemini)
export COYOTE_PATCH_GEMINI_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"google_search":{}}]}}}'
elif [[ "$client" == "vertexai" ]]; then
;;
vertexai)
export COYOTE_PATCH_VERTEXAI_CHAT_COMPLETIONS='{
"gemini-1.5-.*":{"body":{"tools":[{"googleSearchRetrieval":{}}]}},
"gemini-2.0-.*":{"body":{"tools":[{"google_search":{}}]}}
}'
elif [[ "$client" == "ernie" ]]; then
;;
ernie)
export COYOTE_PATCH_ERNIE_CHAT_COMPLETIONS='{".*":{"body":{"web_search":{"enable":true}}}}'
elif [[ "$client" == "claude" ]]; then
;;
claude)
export COYOTE_PATCH_CLAUDE_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"type":"web_search_20250305","name":"web_search","max_uses":5}]}}}'
elif [[ "$client" == "openai" ]]; then
;;
openai)
# Chat Completions native search exists only on the search-preview
# models; the regex scopes the patch so other OpenAI models run
# unpatched instead of erroring on an unsupported parameter.
export COYOTE_PATCH_OPENAI_CHAT_COMPLETIONS='{"gpt-4o.*search-preview.*":{"body":{"web_search_options":{}}}}'
;;
esac
}
# shellcheck disable=SC2154
run_search() {
export_search_patch "$1"
coyote -m "$1" "$argc_query" >>"$LLM_OUTPUT"
}
main() {
local fallback="${WEB_SEARCH_MODEL:-gemini:gemini-2.5-flash}"
local current="${COYOTE_CURRENT_MODEL:-}"
local primary=""
if [[ "${WEB_SEARCH_USE_CURRENT_MODEL:-true}" == "true" && -n "$current" && "$current" != "$fallback" ]] &&
supports_native_search "$current"; then
primary="$current"
fi
coyote -m "$WEB_SEARCH_MODEL" "$argc_query" >> "$LLM_OUTPUT"
}
if [[ -n "$primary" ]]; then
if run_search "$primary"; then
return 0
fi
echo "web_search: search with current model '$primary' failed; falling back to '$fallback'" >&2
fi
run_search "$fallback"
}
assets/macros/record-miss.yaml
+17
View File
@@ -0,0 +1,17 @@
# Record a review miss (a defect that ESCAPED review and was caught later) in the
# review-miss ledger at ~/.config/coyote/share/review-misses.md. The first step appends
# a numbered skeleton entry deterministically (zero tokens; it also migrates a ledger
# from the legacy config-root location); the second has the agent interview you to fill
# in the <fill> fields — the detection pattern is the payload.
# Run as:
# coyote --macro record-miss "<one line: what escaped and how it was caught>"
# (avoid double quotes inside the description; it is spliced into a shell string)
variables:
- name: description
description: One line describing what escaped review and how it was caught
isolated: false
steps:
- |-
!f="$HOME/.config/coyote/share/review-misses.md"; legacy="$HOME/.config/coyote/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$legacy" ] && [ ! -f "$f" ] && mv "$legacy" "$f"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; n=$(grep -c '^## MISS-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'MISS-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- repo: <fill owner/repo>\n- surface: <fill rest-api|worker|db-migration|cli|iac|library|ci-cd|any>\n- what escaped: {{description}}\n- why review missed it: <fill>\n- detection pattern: <fill the concrete check/grep a reviewer runs to catch this CLASS>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
- |-
Load the `review-misses` skill, then open `$HOME/.config/coyote/share/review-misses.md` and find the newest `## MISS-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **detection pattern**: it must be a concrete check or grep a reviewer can run, not advice. If my miss matches an existing entry's defect class, propose sharpening that entry instead of keeping a duplicate.
assets/macros/record-ruling.yaml
+18
View File
@@ -0,0 +1,18 @@
# Record a standing ruling — a pre-delegated decision that agents running with
# escalation_policy: autonomous may apply INSTEAD of escalating — in
# ~/.config/coyote/share/standing-rulings.md. The first step appends a numbered
# skeleton entry deterministically (zero tokens); the second has the agent interview
# you to fill in the <fill> fields — the trigger is the payload: it must be narrow
# enough that a matching decision is unmistakable.
# Run as:
# coyote --macro record-ruling "<one line: when X happens, resolve by Y>"
# (avoid double quotes inside the description; it is spliced into a shell string)
variables:
- name: description
description: One line stating the ruling — when <situation>, resolve by <action>
isolated: false
steps:
- |-
!f="$HOME/.config/coyote/share/standing-rulings.md"; mkdir -p "$(dirname "$f")"; [ -f "$f" ] || printf '# Standing rulings\n\nPre-delegated decisions for escalation_policy: autonomous — an agent applies a ruling\nINSTEAD of escalating when its trigger genuinely matches. Hard triggers (secrets, money\nmovement, destructive/irreversible actions) can never be delegated here. Every autonomous\napplication is reported in the run digest.\n' > "$f"; n=$(grep -c '^## R-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'R-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- trigger: <fill the narrow situation this covers — matching must be unmistakable>\n- resolution: <fill exactly what to do, including whether plan/spec text gets amended>\n- scope: <fill any|repo/surface limits>\n- provenance: <fill the escalation/PR/incident this came from>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
- |-
Open `$HOME/.config/coyote/share/standing-rulings.md` and find the newest `## R-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **trigger**: it must be narrow and unambiguous — a ruling that could plausibly match unrelated decisions is worse than no ruling, because agents apply these unsupervised. Confirm the resolution states whether plan/spec text gets amended. REFUSE to record a ruling that would delegate a hard trigger (secrets/credentials, money movement, destructive or irreversible actions) — those always escalate, by design. If the ruling matches an existing entry's class, propose sharpening that entry instead of keeping a duplicate.
assets/sbx-kit/spec.yaml
+1 -1
View File
@@ -13,7 +13,7 @@ description: >
CLI & REPL mode, RAG, AI tools & agents, MCP servers, skills, and macros.
sandbox:
image: 'darkalex17/coyote:v0.10.1'
image: 'darkalex17/coyote:v0.10.2'
entrypoint: ['bash', '-lc', 'exec /home/agent/.cargo/bin/coyote']
permissions:
assets/skills/adversarial-review/SKILL.md
-79
View File
@@ -1,79 +0,0 @@
---
description: Adversarial plan-conformance review of an implementation against the task/plan it was supposed to satisfy. Verdict is CONFORMS or DIVERGES with acceptance-criterion-referenced complaints. Grants read-only filesystem access for ground-truth checks. Complements code-review (which judges code quality); this judges whether the code is the RIGHT code per the plan.
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
---
You are an adversarial plan-conformance reviewer. A code-quality reviewer already asks "is this code good?" — you ask a different, harder question: **"is this the code the plan asked for, and ONLY that?"** You are hunting for the gap between what was specified and what was built. Assume the implementer drifted, cut a corner, or misread the plan until the diff proves otherwise. Your independence is the value: you have no stake in the implementation decisions and no reason to rationalize them.
You review THE CHANGE against THE PLAN. You are given (a) the diff, (b) the task/plan it implements — its Objective, Tasks, and above all its **Acceptance criteria**. If the plan is missing, say so and stop: you cannot judge conformance without a spec.
## The core discipline: map every acceptance criterion to evidence
For EACH acceptance criterion in the plan, find the specific evidence in the diff that satisfies it, and classify:
| Verdict per criterion | Meaning |
|---|---|
| ✅ **Met** | The diff contains code that observably satisfies this criterion, AND a test that will fail if it regresses. Cite the file:line. |
| ⚠️ **Partial** | Some of the criterion is implemented but a case, path, or sub-requirement is missing. Name what's missing. |
| ❌ **Unmet** | No code in the diff satisfies this criterion. The "dog that didn't bark." |
| 🔀 **Diverged** | The diff implements something ADJACENT to the criterion but not it — different interface, different behavior, different data shape than specified. |
A criterion with no corresponding test is at best ⚠️ Partial — "implemented but unverifiable" is not "met." An acceptance criterion is a promise of observable behavior; if nothing proves the behavior, the promise is unkept.
## What to hunt for (adversarial checklist)
### 1. Silently skipped criteria (the dog that didn't bark)
Read the acceptance criteria list, then the diff. Every criterion with no matching change is a finding. Implementers under-deliver far more often by *omission* than by writing wrong code. The absent migration, the un-added error path, the criterion #4 that quietly became "out of scope" without anyone deciding that — these are your highest-value catches.
### 2. Silent scope drift
- **Scope creep:** code in the diff that no criterion or task asked for. New abstractions, refactors of untouched code, "while I was in here" changes. Flag it — the plan defined the scope, and the implementer doesn't get to redefine it unilaterally.
- **Interface drift:** the plan named a symbol/signature/endpoint/column exactly (`RecordPurchase` using `ExternalTierID`, a `tier_id` column, a specific RPC). The diff uses a different name or shape. Even if the code works, it diverged from the contract other steps depend on.
- **Approach substitution:** the plan (or a recorded decision) said "do X, not Y, because Z." The diff does Y. The implementer re-litigated a settled decision. Flag it with the plan's stated reason.
### 3. Ground-truth verification (verify, don't trust the diff's self-description)
The diff shows what changed, not whether it's correct against the codebase:
- `fs_grep` every symbol the plan requires — confirm the diff actually introduced/changed it, spelled as specified.
- `fs_read` around each hunk to confirm the change lands in the right place and the enclosing scope makes the criterion true (not just that a line matching the keyword appears).
- `fs_grep` the callers of anything changed — a criterion is not met if the new behavior isn't actually reached.
- Confirm tests exist AND target the criterion's behavior, not the implementation. A tautological test (`assert x.is_empty() || !x.is_empty()`) counts as no test.
### 4. Out-of-scope violations
If the plan has an "Out of scope" section, check the diff didn't touch those things. Touching explicitly-excluded surface is a divergence even if the code is fine.
### 5. Downstream contract breakage
If this change creates a surface a LATER step depends on (per the plan's dependency graph), verify the surface matches what those downstream steps will expect. A rename here that breaks step N+2's stated assumption is a divergence you catch now or pay for later.
## Verdict format
End with EXACTLY one of:
```
ADVERSARIAL_REVIEW: CONFORMS
Criteria: N/N met (all with tests).
<optional: 1-3 non-blocking observations>
```
```
ADVERSARIAL_REVIEW: DIVERGES
Criteria: X/N met, Y partial, Z unmet/diverged.
Complaints:
1. Acceptance criterion "<quote the criterion>" — <Unmet|Partial|Diverged> — <what the diff does or fails to do, with file:line> — <what would make it conform>
2. Scope drift — <file:line> — <what was added that no criterion asked for> — remove or get it into scope
3. ...
```
Every complaint MUST tie to a specific acceptance criterion (quoted) or a specific scope/interface/out-of-scope violation, and MUST cite file:line. "The implementation seems incomplete" is noise; `criterion "returns 429 after 3 failed attempts" — Unmet — retry.go has no attempt counter; the loop retries forever (retry.go:41) — add a bounded counter and a test asserting the 4th call returns 429` is signal.
## Scope discipline (what you are NOT)
- You are NOT the code-quality reviewer. Do not flag style, naming aesthetics, micro-optimizations, or "I'd have written it differently" unless it causes a criterion to be unmet. The `code-review` skill owns quality; you own conformance. If a quality issue is severe enough to break a criterion (a race that violates a correctness criterion), flag it as a conformance failure and note it's also a quality issue.
- You do NOT rewrite the code or the plan. You produce a verdict and complaints; the implementer owns the fix.
- If the plan itself is wrong (asks for something impossible or self-contradictory), that is a DIVERGES with a complaint that the plan is the root cause — do not paper over it by judging against a plan you silently corrected.
- Three decisive divergences beat fifteen weak ones. If every criterion is a nitpick, the change probably CONFORMS — say so.
## Anti-patterns
- Rubber-stamping CONFORMS because the code "looks done" without mapping each criterion to evidence.
- Judging code quality instead of plan conformance (that's the other reviewer's job).
- Accepting a criterion as met with no test proving it.
- Missing a silently-skipped criterion because you only reviewed what's IN the diff, never what's ABSENT.
- Complaints with no criterion reference and no file:line.
assets/skills/code-review/SKILL.md
+62 -2
View File
@@ -29,16 +29,40 @@ fs_read --path "src/auth.rs" --offset 110 --limit 40
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
### Read the callers of anything changed
### Blast radius: verify every caller (MANDATORY)
If a hunk changes a function's body or its signature, grep for the name to find callers and check whether the change ripples:
A locally-correct change that alters a symbol's CONTRACT breaks callers you can't see in the diff. For every changed exported/public symbol, grep for its callers and verify each one still holds:
```
fs_grep --pattern "changed_function" --include "*.rs"
```
Contract changes that ripple (check the callers against each one that applies):
- **Return type / error type changed** — do callers match on variants that no longer exist, or miss new ones?
- **Nullability / optionality** — a value that could never be null/None now can be (or vice versa).
- **Defaults changed** — callers relying on the old default silently change behavior.
- **Units / encoding / format** — seconds→millis, bytes→string, naive→UTC datetimes.
- **Ordering / uniqueness guarantees** — output was sorted/deduped, now isn't (or vice versa).
- **Error semantics** — a function that returned an error now panics/throws, or swallows what it used to propagate.
- **Sync→async / blocking behavior** — callers on hot paths or in handlers now block or need awaiting.
This check is MANDATORY and produces output even when clean: state `Blast radius: N call sites checked, all compatible` in your findings, or one finding per incompatible/unverifiable caller (`caller at path:line still assumes <old contract>`). Greps are cheap — the file-read budget does not apply to `fs_grep`/`fs_glob`; spend greps freely here and targeted reads only on suspicious callers.
Skip the test files in this search; do the test sweep next.
### Guard symmetry across sibling paths (MANDATORY when a guard changes)
When the diff ADDS, STRENGTHENS, or FIXES a guard/check/validation in one code path, the same
flaw usually lives in that path's parallel twins — sibling strategies implementing the same
interface, other enqueue/call sites of the same job, the script's sibling scripts, the other
branch of the same switch. Enumerate the twins (`fs_grep` for the shared interface, the job
kind, the naming pattern) and verify each one either has the equivalent guard or provably
doesn't need it. A twin missing the guard is a finding at the twin's `path:line` — same
severity as the bug the guard fixes. Report even when clean: `Guard symmetry: N sibling
paths checked`. (Guards REMOVED are the removed-guards check below — this is about the ones
added: a fix applied to one of N twins is N-1 latent bugs.)
### Read the tests for the change
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
@@ -58,7 +82,26 @@ These are review findings that only surface in a diff context, not in a whole-fi
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
- **New code path without new tests** — usually a missing test. Flag it.
- **Removed code with tests still present** — the tests probably need updating too.
- **Removed guards, checks, and validations** — for EVERY removed guard/branch/validation/limit, name where that responsibility now lives ("moved to X at path:line") or flag it as a regression risk. Deleted code had a reason; "nothing now does what the deleted code did" is a 🟡 finding by default, and 🔴 when the guard protected money, auth, or data integrity. Reviews fixate on added lines — the removed lines are where incidents come from.
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
- **Minted but unused / partial adoption** — every artifact the diff INTRODUCES (variable, flag,
helper, image, config key) must be consumed somewhere; `fs_grep` for each one. Zero uses = a
finding (dead weight or a forgotten wiring step). Used in SOME applicable in-diff sites but
not others = a finding naming the sites that didn't adopt it — half-adopted artifacts are how
two mechanisms for the same job end up coexisting forever.
- **Version-literal consistency and freshness** — when the diff bumps a version (image tag,
tool version, dependency pin), `fs_grep` the repo for other occurrences of the OLD version
string (compose files, workflows, docs, sibling Dockerfiles) — stragglers are findings. For
NEWLY-pinned versions, check the upstream latest when network tools allow it and flag pins
that start life stale; skip silently offline.
- **Silent-failure fix without remediation** — a diff that fixes a silently-failing write path
(bad address, swallowed error, wrong topic) leaves behind whatever state was damaged or
omitted while it was broken. The fix must state how that backlog gets reconciled (backfill,
reconciliation run, "provably no traffic since X") — absent statement is a finding.
- **Normative doc changes** — a diff adding/changing a rules doc, convention, or runbook: grep
the governed tree for PRE-EXISTING violations of the new rule (the doc is wrong or the tree
is — say which). Runbooks/docs hardcoding mutable environment identifiers (personal accounts,
org names, pool IDs) are a maintenance hazard — flag unless marked with an ownership note.
### Scope discipline
@@ -84,6 +127,23 @@ A diff review is a review of THE CHANGE, not the whole file:
- Do they cover the unhappy paths, not just the happy ones?
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
### Test adequacy: the mutation question (MANDATORY)
Presence of tests proves nothing. For EACH behavior change in the diff, ask: **"if this
specific logic were wrong, which test would fail?"** and name it. No nameable test = a
finding (`behavior <X> has no test that would catch its regression`). State the mapping
even when clean: `Tests: <behavior> covered by <test name>`.
Named adequacy anti-patterns — each is a finding even when coverage looks green:
- **Mock-assertion tests** — the test asserts the mock was called, never the real outcome; it verifies wiring, not behavior, and passes when the logic is wrong.
- **Unexercised branch** — the diff adds a branch/condition no test drives down; coverage of the function ≠ coverage of the new path.
- **Implementation-mirroring tests** — the test recomputes the expected value using the same logic as the code under test; both are wrong together, so it can never fail meaningfully. Expected values must be independently derived literals.
- **Missing negative case** — a new validation/guard with tests only proving it ACCEPTS good input, never that it REJECTS bad input. The reject path is the whole point of a guard.
- **Leaky fixture** — an integration test against a shared/persistent database inserts fixture
rows directly with no paired cleanup (deferred delete/teardown/transaction rollback). The
test passes today and leaves a primary-key landmine for the next run.
## 3. Clarity
- Are names accurate? `get_user` that mutates is a lie; rename or split.
assets/skills/migration-review/SKILL.md
+8
View File
@@ -40,6 +40,14 @@ Does the migration mix a data backfill (UPDATE/INSERT over existing rows) into t
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
### 6. 🟡 `[correctness]` Seed/backfill literals not cross-checked against canonical values
Does the migration INSERT or UPDATE literal data values — enum strings, type/kind/status names, display copy, identifiers? Every literal has a canonical source somewhere: the application's enum/constant definitions, the schema's CHECK constraints, existing seed data, or a migration landing in the SAME stack. `fs_grep` for each literal's canonical definition and verify it matches — a seed row whose value drifts from the enum (or that omits a column a same-stack migration is about to make meaningful, or that conflicts on a constraint a sibling migration changes) fails at deploy time or, worse, seeds wrong data silently. Check the same-stack interaction explicitly: seed scripts and schema migrations that land together must be reviewed together.
### 7. 🟢 `[convention]` New table without audit columns
Does a new table carry the repo's standard audit columns — typically `created_at`/`updated_at`, and `version`/row-tracking where the house convention uses them? READ two or three sibling tables' migrations for the actual house convention first (Ground-truth discipline below) and cite one; a new table silently skipping it is the finding. A table that genuinely needs no audit trail (pure join table, ephemeral scratch) can be exempt — the author should say so, not leave it to inference.
## Ground-truth discipline
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
assets/skills/observability-review/SKILL.md
+8
View File
@@ -68,5 +68,13 @@ For inapplicable changes the whole block collapses to: `## Observability` / `Not
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
- **Per-verb metric families.** Sibling metrics differing only by an operation verb
(`thing_create_total`, `thing_update_total`, `thing_delete_total`) fragment dashboards and
alerts; the idiomatic shape is ONE family with an `operation` (and `outcome`) label. Flag the
family split — bounded label values, so cardinality stays sane (the unbounded-label check is
separate and still applies).
- **Telemetry outside its feature flag.** A change that gates behavior behind a flag but
records the behavior's metrics/logs unconditionally reports activity that isn't happening
(or leaks rollout state); gate the telemetry with the behavior.
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
assets/skills/review-misses/SKILL.md
+102
View File
@@ -0,0 +1,102 @@
---
name: review-misses
description: |
The review-miss ledger: escaped defects (bugs that survived review and were caught later)
recorded as reusable detection patterns. Review orchestrators load this skill, read the
ledger, and route matching entries to reviewers as first-class checks — this is the
feedback loop that makes the review stack converge instead of plateau.
Triggers: loaded by review orchestrators (e.g. code-reviewer) at review start; "record a miss", "review miss"
license: MIT
metadata:
author: Alex Clarke
version: 1.0.0
tags: [review, feedback-loop, ledger]
---
# review-misses
A review stack without a feedback loop plateaus: it catches the same classes forever and
misses the same classes forever. This skill closes the loop — every defect that ESCAPED
review (shipped, then caught in prod, by the user, or by a later change) becomes a ledger
entry with a detection pattern, and every future review runs those patterns as
first-class checks.
## The ledger
Location: `$HOME/.config/coyote/share/review-misses.md` (expand `$HOME` at runtime —
never hardcode a user's home path). The `share/` subdirectory is the convention for
durable DATA files that are not configuration — it keeps the config root uncluttered
(the /usr/share analogy). The `record-miss` macro migrates a ledger from the legacy
config-root location automatically; when reading, if `share/review-misses.md` is absent,
also check the legacy `$HOME/.config/coyote/review-misses.md` before concluding "empty".
**Scope: global and permanent.** One ledger per environment, shared across every repo,
review, and reviewer — it is the institutional memory of escaped defects, not per-review
state. Reviews only READ it; nothing about a review is ever written to it. The `repo:`
and `surface:` fields are how one global ledger stays relevant per review (filtering,
not partitioning). Concurrent reviews are fine: reviewers are concurrent readers, and
writes happen only through the `record-miss` macro — a human act, append-only. It
deliberately lives under config rather than a cache directory: caches are disposable by
contract, and losing this file loses the learning loop.
- **Missing or empty ledger**: skip silently — a one-line note in the synthesis
("review-miss ledger: empty") and nothing else. Never invent entries.
- Entries are append-only history. Never rewrite or delete entries during a review.
Entry format:
```markdown
## MISS-NNN — <one-line title>
- date: YYYY-MM-DD
- repo: <owner/repo where it escaped>
- surface: <rest-api | worker | db-migration | cli | iac | library | ci-cd | any>
- what escaped: <the defect, concretely>
- why review missed it: <root cause in review terms — wrong context, unchecked caller, mock-assert test, …>
- detection pattern: <the concrete check/grep/question a reviewer runs to catch this CLASS of defect>
```
## Consuming the ledger (review orchestrators)
1. At review start, read the ledger (if present).
2. Filter entries to the ones RELEVANT to this review: matching `surface` (against the
resolved surfaces), matching `repo`, or `surface: any`. Irrelevant entries stay out of
worker context — the ledger must not bloat every review.
3. Paste each relevant entry's title + detection pattern into the CONTEXT section of the
file-reviewers (or lanes) whose files it could apply to.
4. In the final report footer, note `review-miss patterns applied: MISS-004, MISS-011`
(or `none relevant`).
## Applying a pattern (reviewers)
- A ledger detection pattern is a FIRST-CLASS check — same standing as a loaded skill's
checklist item. Run it against the diff like any other check.
- When a pattern fires, the finding cites the entry: `recurrence of MISS-007` in the
finding title. Recurrences of an escaped defect class default to 🔴 — the class has
already burned us once.
- When a pattern doesn't fire, silence is fine — no per-entry accounting needed.
## Recording a miss
Record via the `record-miss` macro (`coyote --macro record-miss "<what escaped>"`), which
appends a numbered skeleton entry and prompts for the remaining fields. Or append by hand
in the entry format above.
Collection is otherwise AUTOMATIC — no mining pass needed:
- **whetstone** classifies every externally-raised finding after a run addresses review
feedback, and proposes ledger entries (approval-gated) or drops one-offs into the
candidates inbox (`<config>/share/review-miss-candidates.jsonl`) as recurrence evidence.
- **PR-review reconciliation** (where a re-review compares what humans said against what
the panel found) records each human-caught, panel-missed defect into the same candidates
inbox. The inbox is raw material, never authority: nothing enters the LEDGER without
your approval — that curation is the noise firewall that keeps the ledger trustworthy.
Rules for good entries:
- **The detection pattern is the payload.** "Be more careful with X" is not a pattern;
"grep for `time.Now()` in files that also touch period/interval math; flag any not going
through the clock interface" is. If you can't state the check a reviewer runs, keep
distilling before recording.
- One entry per defect CLASS, not per incident — if a new miss matches an existing
entry's class, sharpen that entry's detection pattern instead of adding a duplicate.
- Never record secrets, customer data, or incident details beyond what the pattern needs.
assets/skills/security-review/SKILL.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
description: Security analysis of a code change - hunts exploitable flaws in the diff (injection, secrets, authz gaps, unsafe deserialization, path traversal, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Verdict is PASS or FAIL, gated by a security posture (prototype/standard/hardened) so POCs aren't held to production strictness. Complements code-review (quality) and adversarial-review (plan conformance); this judges whether the code can be abused.
description: Security analysis of a code change - hunts exploitable flaws in the diff (injection, secrets, authz gaps, unsafe deserialization, path traversal, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Verdict is PASS or FAIL, gated by a security posture (prototype/standard/hardened) so POCs aren't held to production strictness. Complements code-review (quality) and the adversary agent (plan conformance); this judges whether the code can be abused.
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
---
You are a security reviewer. The quality reviewer asks "is this code good?"; the conformance reviewer asks "is this the code the plan asked for?"; you ask the third question: **"can this code be abused?"** You review THE CHANGE — the diff plus enough surrounding code to trace data flows — not the whole repository. Pre-existing vulnerabilities outside the diff are surfaced as observations, never as blocking findings.
assets/skills/usage-pattern-testing/SKILL.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
description: Verify a change from the consumer's perspective - exercise the changed surface (HTTP API, RPC, CLI) black-box against a locally running instance with clean, isolated state. Run existing usage suites first for regressions, derive new tests from the spec (never the implementation), and classify every failure as bug / environment / expected contract change. Produces a USAGE_PROBE PASS/FAIL/INCONCLUSIVE verdict. Complements code-review (quality), adversarial-review (plan conformance), and security-review (abuse) - this is the only gate that tests BEHAVIOR by using the thing, not by reading it.
description: Verify a change from the consumer's perspective - exercise the changed surface (HTTP API, RPC, CLI) black-box against a locally running instance with clean, isolated state. Run existing usage suites first for regressions, derive new tests from the spec (never the implementation), and classify every failure as bug / environment / expected contract change. Produces a USAGE_PROBE PASS/FAIL/INCONCLUSIVE verdict. Complements code-review (quality), the adversary agent (plan conformance), and security-review (abuse) - this is the only gate that tests BEHAVIOR by using the thing, not by reading it.
enabled_tools: fs_read, fs_cat, fs_grep, fs_glob, fs_ls, fs_write, fs_patch, execute_command
---
You are verifying a change the way its consumers will experience it: by USING it. Every other
assets/skills/worker-review/SKILL.md
+4
View File
@@ -40,6 +40,10 @@ Does the new consumer bound how many messages it processes at once — worker-po
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
### 6. 🟡 `[convention]` Job queue used for synchronous request-path work
Does request-path code enqueue a job and then WAIT for its result before responding — polling job status, blocking on a completion channel, sleeping in a loop? That buys the queue's costs (latency, serialization, a second failure domain, at-least-once semantics) without its benefit (durable async execution), and it serializes under load: N concurrent requests queue behind a worker pool sized for background throughput. The synchronous shape is a direct call (with a timeout); the queue is for work that outlives the request. Legitimate exceptions exist — a durable workflow where the caller genuinely resumes later, or intentional admission control — but they should be stated; flag the enqueue-then-wait shape when they aren't. Also the inverse smell: an expensive in-process retry/sleep loop inside a job handler where the queue's own scheduling (snooze/requeue with delay) is the idiomatic tool — the handler holds a worker slot hostage to sleep.
## Ground-truth discipline
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.
config.agent.example.yaml
+60 -63
View File
@@ -11,74 +11,71 @@
# - <agent-name>_AGENT_SESSION
# - <agent-name>_VARIABLES (as JSON array of key-value pairs; e.g. '[{"name": "username", "value": "alex"}]')
model: openai:gpt-4o # Specify the LLM to use
temperature: null # Set default temperature parameter, range (0, 1)
top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model
reasoning_effort:
null # Reasoning effort level for models that support it (e.g. low, medium, high).
# Only valid when the agent's model declares reasoning_levels.
agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session
name: <agent-name> # Name of the agent, used in the UI and logs
description: <description> # Description of the agent, used in the UI
version: 1 # Version of the agent
model: openai:gpt-4o # Specify the LLM to use
temperature: null # Set default temperature parameter, range (0, 1)
top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model
reasoning_effort: null # Reasoning effort level for models that support it (e.g. low, medium, high).
# Only valid when the agent's model declares reasoning_levels.
agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session
name: <agent-name> # Name of the agent, used in the UI and logs
description: <description> # Description of the agent, used in the UI
version: 1 # Version of the agent
# Auto-Continue (Todo System)
# The auto-continue system provides built-in task tracking for improved reliability.
# When enabled, the model can create todo lists and the system will automatically
# prompt it to continue when incomplete tasks remain.
# See the [Todo System documentation](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System) for more information
auto_continue: false # Enable automatic continuation when incomplete todos remain
max_auto_continues: 10 # Maximum number of automatic continuations before stopping
inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt
continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null)
auto_continue: false # Enable automatic continuation when incomplete todos remain
max_auto_continues: 10 # Maximum number of automatic continuations before stopping
inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt
continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null)
# Sub-Agent Spawning System
# Enable this agent to spawn and manage child agents in parallel.
# See https://github.com/Dark-Alex-17/coyote/wiki/Agents for detailed documentation.
can_spawn_agents: false # Enable the agent to spawn child agents
can_spawn_agents: false # Enable the agent to spawn child agents
# spawnable_agents: # Optional whitelist restricting which agents can be spawned via `agent__spawn`.
# - explore # If omitted (the default), ALL installed agents are spawnable. This is the unrestricted default.
# - coder # Provide a list to restrict. Match is exact and case-sensitive (use directory names).
# - oracle # An empty list ([]) means literally nothing spawnable.
# Also filters `agent__list_available` output so the LLM only sees what it can spawn.
# Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes.
max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously
max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning)
max_concurrent_jobs:
5 # Max background jobs (`job__*` tools) running at once for this agent
# (overrides the global setting; 0 disables background jobs for this agent)
inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt
summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model
summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent
escalation_timeout: 300 # Seconds a sub-agent waits for a user interaction response before timing out (default: 5 minutes)
mcp_servers: # Optional list of MCP servers that the agent utilizes
- github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file
mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers
github: # (glob patterns: * and ?). Intersects with the global config,
- get_* # mcp.json `allowedTools`, and every other configured layer. It
- search_* # can only narrow access, never widen it.
global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
# Also filters `agent__list_available` output so the LLM only sees what it can spawn.
# Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes.
max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously
max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning)
max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once for this agent
# (overrides the global setting; 0 disables background jobs for this agent)
inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt
summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model
summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent
compression_model: null # Model to use for session compression for this agent (overrides the global setting); defaults to current model
escalation_timeout: 0 # 0 = wait indefinitely for the reply (default); N = give up after N seconds
mcp_servers: # Optional list of MCP servers that the agent utilizes
- github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file
mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers
github: # (glob patterns: * and ?). Intersects with the global config,
- get_* # mcp.json `allowedTools`, and every other configured layer. It
- search_* # can only narrow access, never widen it.
global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
- web_search
- fs
- python
skills_enabled:
true # Master switch for skills in this agent (default: inherit from global).
# Skills also require `function_calling_support: true` in the global config.
enabled_skills: # Optional list of skills available when this agent runs.
# Must be a subset of global `visible_skills`. Omit to inherit the global default.
- git-master
skills_enabled: true # Master switch for skills in this agent (default: inherit from global).
# Skills also require `function_calling_support: true` in the global config.
enabled_skills: # Optional list of skills available when this agent runs.
- git-master # Must be a subset of global `visible_skills`. Omit to inherit the global default.
- ai-slop-remover
inject_skill_instructions:
true # Inject a short hint pointing the model at `skill__list` when skills are enabled
# (default: true). Suppressed automatically when no skills are available.
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
memory:
null # Per-agent memory override (default: inherit). Set to `false` to disable memory
# for this agent regardless of workspace/global presence. See the Memory wiki page.
inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled
# (default: true). Suppressed automatically when no skills are available.
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
memory: null # Per-agent memory override (default: inherit). Set to `false` to disable memory
# for this agent regardless of workspace/global presence. See the Memory wiki page.
dynamic_instructions: false # Whether to use dynamic instructions for the agent; if false, static instructions are used
instructions:
| # Static instructions for the agent; ignored if dynamic instructions are used
instructions: # Static instructions for the agent; ignored if dynamic instructions are used
|
You are a AI agent designed to demonstrate agent capabilities.
<tools>
@@ -99,10 +96,10 @@ instructions:
username: {{username}}
</user>
variables: # Optional variables for the agent
# The variables defined above like {{__variable_name__}} are automatically available
# The variables defined above like {{__variable_name__}} are automatically available
- name: username
description: Your user name
default: null # A default value for this variable; if null, the variable must be provided when starting the agent
default: null # A default value for this variable; if null, the variable must be provided when starting the agent
conversation_starters: # Optional conversation starters for the agent
- What is the meaning of life?
- Tell me a joke.
@@ -114,15 +111,15 @@ conversation_starters: # Optional conversation starters for the agent
- How do I stay motivated?
- What is the best way to exercise?
- How do I manage my time effectively?
documents: # Optional documents to load for the agent
# To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval),
# set `rag_extractor_model` in your global config.yaml.
# See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag
- git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path
- pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path
documents: # Optional documents to load for the agent
# To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval),
# set `rag_extractor_model` in your global config.yaml.
# See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag
- git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path
- pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path
- https://some-website.com/some-page
- some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file
- ~/some-file.txt # File in the user's home directory
- /absolute/path/to/some-file.md # File with absolute path
- /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories
- /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1)
- some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file
- ~/some-file.txt # File in the user's home directory
- /absolute/path/to/some-file.md # File with absolute path
- /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories
- /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1)
config.example.yaml
+9 -1
View File
@@ -36,10 +36,14 @@ sync_models_ttl_hours: 24 # Hours before the synced model catalog is cons
# ---- REPL Prompt ----
# Custom REPL left/right prompts; see the [REPL Prompt Documentation](https://github.com/Dark-Alex-17/coyote/wiki/REPL-Prompt) for more information
# Token-usage variables (set only when the provider reports usage, so guard them with {?var ...}):
# last_input_tokens, last_output_tokens, last_cache_read_tokens, last_cache_creation_tokens (most recent API call)
# total_input_tokens, total_output_tokens, total_cache_read_tokens, total_cache_creation_tokens (session cumulative)
# last_cost, total_cost (USD, formatted like 0.0142; set only when nonzero and every billed token type has a price)
left_prompt:
'{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
right_prompt:
'{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
'{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?total_cost ${total_cost} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
# ---- Vault ----
# See the [Vault documentation](https://github.com/Dark-Alex-17/coyote/wiki/Vault) for more information on the Coyote vault.
@@ -223,6 +227,7 @@ summary_context_prompt: > # The text prompt used for including the summar
compression_keep_last: 0 # Number of most-recent messages to keep visible after compression (0 = compress all messages)
max_tool_result_chars: null # Cap on tool result characters forwarded to the model per call (null = no cap)
max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once per context (default: 5; 0 disables background jobs entirely)
tool_timeout: 1800 # Max seconds a single tool call may run before being killed (default: 1800; 0 = unlimited). COYOTE_TOOL_TIMEOUT env var overrides
# ---- Memory ----
# See the [Memory documentation](https://github.com/Dark-Alex-17/coyote/wiki/Memory) for more information.
@@ -379,6 +384,7 @@ clients:
api_key: '{{ANTHROPIC_API_KEY}}' # You can either hard-code or inject secrets from the Coyote vault
auth: null # When set to 'oauth', Coyote will use OAuth instead of an API key
# Authenticate with `coyote --authenticate` or `.authenticate` in the REPL
prompt_cache: true # Optional - enable Anthropic prompt caching (default: true)
# See https://docs.mistral.ai/
- type: openai-compatible
@@ -460,6 +466,7 @@ clients:
# Run `gcloud auth application-default login` to initialize the ADC file
# see https://cloud.google.com/docs/authentication/external/set-up-adc for more information
adc_file: <gcloud-config-dir>/application_default_credentials.json # Optional
prompt_cache: true # Optional - enable Anthropic prompt caching for Claude models (default: true)
patch:
chat_completions:
'gemini-.*':
@@ -480,6 +487,7 @@ clients:
secret_access_key: '{{AWS_SECRET_ACCESS_KEY}}' # You can either hard-code or inject secrets from the Coyote vault
region: xxx
session_token: xxx # Optional, only needed for temporary credentials
prompt_cache: true # Optional - enable Anthropic prompt caching for Anthropic models (default: true)
# See https://developers.cloudflare.com/workers-ai/
- type: openai-compatible
graph.example.yaml
+128 -42
View File
@@ -38,12 +38,23 @@ reasoning_effort: null # Default reasoning effort for `llm` nodes th
max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once across the
# whole graph run: every `llm` node (including parallel branches)
# draws from this one pool, so the budget is graph-wide — there is
# draws from this one pool, so the budget is graph-wide. There is
# no per-node override. Jobs themselves are node-local: the node
# that starts a job must collect or cancel it before it ends, and
# anything left running at node exit is cancelled. Overrides the
# global setting; 0 disables background jobs for this graph agent.
can_spawn_agents: true # Whether `llm` nodes may spawn dynamic sub-agents via the
# agent__* tools. Defaults to true iff the graph has an `agent`
# node; an explicit value overrides that. Set it true on a graph
# with only `llm` nodes to build an orchestrator graph. Setting
# it false on a graph with `agent` nodes still runs those nodes,
# only dynamic spawning from `llm` nodes is disabled.
max_concurrent_agents: 4 # Max dynamically spawned sub-agents running at once. Explicit
# value overrides the default of 4.
max_agent_depth: 3 # Max agent nesting depth (agents spawning agents). Explicit
# value overrides the default of 3.
global_tools: # Tool universe an `llm` node's `tools:` whitelist draws from
- web_search_coyote.sh
- fetch_url_via_curl.sh
@@ -87,10 +98,12 @@ conversation_starters: # Suggested prompts surfaced in the UI
# ---------------------------------------------------------------------------
# Agent variables (optional)
# Declared the same way as a normal agent's config.yaml `variables:` block.
# Each variable becomes available to:
# - LLM nodes via the template form `{{name}}` once seeded into state
# (see initial_state below).
# - Script nodes via the env var `LLM_AGENT_VAR_<UPPER_NAME>`.
# At run start each declared variable is automatically seeded into graph
# state under its name, resolved as: runtime override, else declared default.
# An explicit `initial_state` entry or dispatcher input with the same key
# takes precedence over the seeded value. Nodes can then reference the
# variable via the template form `{{name}}`; script nodes additionally
# receive the env var `LLM_AGENT_VAR_<UPPER_NAME>`.
# Values may be overridden at runtime with
# `coyote -a <agent> --agent-variable <name> <value> "..."`.
# ---------------------------------------------------------------------------
@@ -105,9 +118,12 @@ variables:
# ---------------------------------------------------------------------------
settings:
max_loop_iterations: 100 # Per-node visit cap. If one node id is entered more
# than this many times, execution aborts. Default 100.
# than this many times, execution aborts. Default 100; 0 = no cap.
timeout: 600 # Optional wall-clock cap (seconds) on the whole run,
# checked between node transitions.
# checked between node transitions. 0 = no bound.
# Ctrl-C (or the enclosing turn aborting) ends the run
# promptly from anywhere: in-flight branches, tools, and
# child graph agents are cancelled with it.
log_state_snapshots: true # Log state before each node (debug/trace). Default true.
validate_before_run: true # Run the graph validator at startup. Default true.
max_concurrency: 4 # Cap on simultaneously running branches in any
@@ -197,8 +213,10 @@ nodes:
# agent/graph/global default when omitted.
skill_instructions: null # Per-node skill-hint text override; uses the built-in default when omitted.
output_schema: # Optional JSON Schema. The output is parsed to JSON
type: object # and its top-level object keys auto-merge into state
properties: # (so `topic` / `needs_deep_dive` become {{topic}} etc).
type: object # and only the keys declared under `properties` auto-merge
properties: # into state (so `topic` / `needs_deep_dive` become {{topic}}
# etc). Undeclared keys the model emits are dropped; lift
# one explicitly via state_updates: `{{output.<key>}}`.
topic: { type: string }
needs_deep_dive: { type: boolean }
required: [topic, needs_deep_dive]
@@ -231,7 +249,7 @@ nodes:
- https://example.com/reference
query: "{{topic}}" # Retrieval query (templated). Default: {{initial_prompt}}.
top_k: 5 # Chunks to retrieve. Default = the KB's own top_k.
timeout: 120 # Retrieval timeout in seconds. Default 120.
timeout: 120 # Retrieval timeout in seconds. Default 120; 0 = no bound.
# Knowledge-base build config (optional; used only when the KB is first
# built). When embedding_model + chunk_size + chunk_overlap are all set,
# the KB builds with no interactive prompts (works in non-interactive runs).
@@ -268,8 +286,9 @@ nodes:
type: array
items: { type: string }
required: [context, sources]
# When `output_schema` is set, top-level keys auto-merge into state, so
# `context` and `sources` are produced without needing `state_updates`.
# When `output_schema` is set, the keys declared under `properties`
# auto-merge into state, so `context` and `sources` are produced without
# needing `state_updates`. Keys the schema does not declare are dropped.
next: synthesize # Joins with retrieve at `synthesize`.
# --- script node (the diamond's join; also dispatches) -----------------
@@ -300,7 +319,7 @@ nodes:
id: synthesize
type: script
script: scripts/synthesize.py # Path relative to the agent directory
timeout: 30 # Seconds. Default 30.
timeout: 30 # Seconds. Default 30; 0 = no bound.
state_updates: # Applied after the stdout JSON is merged
decided_for: "{{topic}}"
next: summarize # Default route if the script emits no `_next`
@@ -322,8 +341,39 @@ nodes:
prompt: | # User message sent to the child (templated)
Research {{topic}} in depth. Existing context:
{{context}}
timeout: 600 # Optional wall-clock cap, seconds. Default 300.
output_schema: # Optional. Same extraction as llm nodes
# inputs: # Optional. Only valid when `agent` is a GRAPH agent
# max_parallel: "{{n}}" # (has a graph.yaml; deep-research is one). Each value
# context: "Repo: {{r}}" # is a template against THIS graph's state. The resolved
# # map is overlaid on the child's `initial_state` before
# # its graph starts, so the child can read {{max_parallel}}
# # and {{context}} from its first node on. A lone {{key}}
# # keeps the value's JSON type (numbers stay numbers, lists
# # stay lists, null overlays the child's default); mixed text
# # becomes a string. `initial_prompt` is reserved (always
# # seeded from `prompt:`) and is rejected as an input key.
# # `inputs:` on a config-only agent is a load-time error,
# # and `inputs` values count as reads for the cross-branch
# # read check. See the wiki: Graph Agents -> "Passing state
# # into a child graph agent (inputs)".
timeout: 600 # Optional wall-clock cap, seconds. Default 300; 0 = no bound.
teammates: false # Optional. When true and this node runs concurrently
# with other flagged agent nodes (map fan-out branches
# or the same super-step), the siblings can message
# each other via agent__send_message / agent__check_inbox.
# Each fan-out gets its OWN registry: a `teammates` map
# fan-out and a `teammates` agent node in the same
# super-step frontier form two separate teams and cannot
# message each other. Peers only span a single map
# node's branches or a single super-step's flagged
# agent nodes. Inside a map branch chain the identity
# belongs to the ITEM, not the node: it lives for the
# item's whole chain, is handed to each `teammates: true`
# agent step in that chain (unflagged steps never see it),
# and is retired when the chain ends. A message delivered
# after the chain's last flagged agent step is accepted
# but never read.
output_schema: # Optional. Same extraction as llm nodes: only the keys
# declared under `properties` auto-merge into state.
type: object
properties:
summary: { type: string }
@@ -336,9 +386,20 @@ nodes:
next: review # Required for agent nodes
# --- map node (Dynamic fan-out. Think: LangGraph's `Send` API) ----------------
# Spawns one parallel sub-branch per item in `over`. Each sub-branch runs
# the node referenced by `branch:` with the item bound to `as:`. Outputs
# collect into the array named by `collect_into:`, preserving input order.
# Spawns one parallel sub-branch per item in `over`. Each sub-branch runs a
# per-item CHAIN of nodes on a private fork of the state: it starts at the
# node referenced by `branch:` (with the item bound to `as:`) and follows
# that node's `next:` / script `_next` / `fallback` edges until it reaches a
# node with no `next` and no `_next`. When every chain has finished, the
# value of `output_key` from each item's final fork state is collected into
# the array named by `collect_into:`, preserving input order. Everything
# else the chain wrote is fork-local scratch and is discarded at collect.
#
# Not supported inside a branch chain (validator errors): fan-out `next:`
# lists, another `map`, `end`, `approval`, `input`. A branch node must not
# also be reachable from `start` on the main flow. Per-item loops are script
# `_next` loops (bounded by settings.max_loop_iterations on the fork; `0` = unbounded); a
# declared `next`/`fallback` edge back to an earlier chain node is a cycle.
#
# Reach via `synthesize`'s `_next: subjects_map`. The producer is expected
# to have written a list at `subjects` (e.g. an upstream LLM node with an
@@ -350,31 +411,53 @@ nodes:
# Empty list is allowed. It means no branches spawn,
# and thus `collect_into` is written as [].
as: subject # Required. Per-branch state key holding the
# current item. Read with {{subject}} inside
# the branch node's prompt.
branch: research_subject # Required. Node id to invoke per item.
# Must point to an llm | agent | rag | script
# node satisfying the map branch contract:
# - no `next:` (atomic, joined at map exit)
# - no `state_updates:` other than via the
# map's `collect_into` channel
# - no `output_schema:` (top-level merge
# would clash with collect_into)
# Validator enforces all three.
# current item. Read with {{subject}} from any
# node in the branch chain. Must differ from
# `output_key` (load-time error otherwise).
branch: research_subject # Required. ENTRY node of the per-item chain.
# Every node reachable from it via `next:` /
# `fallback:` is part of the chain and must be
# an llm | agent | rag | script node. Chain
# nodes may use `next:`, `state_updates:`
# (fork-local scratch such as a draft or a
# retry counter), `output_schema:`, and
# `fallback:` (honored per item; it must point
# at a node inside the chain, never at an
# `end_*` node). A script's `_next` must also
# stay inside the chain; routing out of it
# fails the item.
collect_into: subject_findings # Required. State key for the array of
# per-branch outputs, in input order
# (not spawn-finish order).
max_concurrency: 3 # Optional per-map cap. Defaults to
# settings.max_concurrency above.
output_key: output # Optional. State key the branch's output
# appears under. Default "output". Useful
# only if the branch reads its own bound
# name back (rare).
max_concurrency: 3 # Optional cap on items in flight (one slot is
# held for an item's WHOLE chain). Defaults to
# settings.max_concurrency above. Accepts an
# integer or a template that is exactly one
# "{{key}}" (surrounding whitespace allowed,
# nothing else) and resolves to a positive integer,
# e.g. `max_concurrency: "{{research_budget}}"`.
output_key: subject_summary # Optional. State key whose value in the item's
# FINAL fork state becomes the item's result.
# Default "output"; must differ from `as:`. The
# key is removed from the fork before the chain
# starts, so the chain
# must write it (via state_updates, an
# output_schema property, or a script's JSON);
# an item that never writes it is an error, and
# a chain that reads it before writing it sees
# it absent (prompts fail, state_updates render
# "", scripts get no key) - seed per-item input
# through `as:` instead. Prefer a named key: with
# the default, the collector errors if the chain
# never writes it.
next: aggregate_subjects # Where to go after all sub-branches finish.
# Branch node for subjects_map. Each invocation receives a different
# `subject` in state. The branch is "atomic", meaning it cannot route on
# its own; the surrounding `map` joins after all invocations finish.
# Entry node of subjects_map's per-item chain. Each item gets its own copy
# of this node with a different `subject` in state. With no `next:` it is
# also the LAST chain node, so the chain is a single step; give it `next:`
# (or emit `_next` from a script step) to extend the chain, e.g.
# research -> verify -> gate-script that re-emits `_next: research_subject`
# until the draft passes or a retry counter in fork state runs out.
research_subject:
id: research_subject
type: llm
@@ -382,8 +465,11 @@ nodes:
prompt: "Research {{subject}}: pull the key facts and one citation."
tools:
- web_search_coyote
# No `next:`, `state_updates:`, or `output_schema:` here. Map branches
# have a strict contract (see `subjects_map.branch` comment).
# Recommended pattern: write the completion to a named key via
# `state_updates` and point the map's `output_key` at that same key.
# `{{output}}` is bound only while these templates render.
state_updates:
subject_summary: "{{output}}"
# Aggregator that runs after the map joins. Reads the collected list.
aggregate_subjects:
@@ -413,9 +499,9 @@ nodes:
temperature: 0.3 # Optional per-node sampling override
reasoning_effort: null # Optional per-node reasoning effort override (e.g. low, medium, high)
max_attempts: 2 # Retry count on transient errors only. Default 1.
max_iterations: 10 # Tool-call-loop turn cap. Default 10.
max_iterations: 10 # Tool-call-loop turn cap. Default 10; 0 = no cap.
fallback: review # Route here if all attempts fail
timeout: 300 # Optional node wall-clock cap, seconds (unset = no timeout)
timeout: 300 # Optional node wall-clock cap, seconds (unset or 0 = no timeout)
state_updates:
research: "{{output}}"
next: review # Required for llm nodes: the success route
models.yaml
+4114 -2572
View File
File diff suppressed because it is too large. Load diff
src/client/bedrock.rs
+631 -53
View File
@@ -1,3 +1,4 @@
use std::collections::HashSet;
use std::mem;
use super::*;
@@ -24,6 +25,7 @@ pub struct BedrockConfig {
pub session_token: Option<String>,
#[serde(default)]
pub models: Vec<ModelData>,
pub prompt_cache: Option<bool>,
pub patch: Option<RequestPatch>,
pub extra: Option<ExtraConfig>,
}
@@ -59,7 +61,11 @@ impl BedrockClient {
format!("/model/{model_name}/converse")
};
let body = build_chat_completions_body(data, &self.model)?;
let body = build_chat_completions_body(
data,
&self.model,
self.config.prompt_cache.unwrap_or(true),
)?;
let mut request_data = RequestData::new("", body);
self.patch_request_data(&mut request_data);
@@ -207,6 +213,8 @@ async fn chat_completions_streaming(
let mut function_arguments = String::new();
let mut function_id = String::new();
let mut reasoning_state = 0;
let mut thinking_text = String::new();
let mut thinking_signature = String::new();
let mut stream = res.bytes_stream();
let mut buffer = BytesMut::new();
@@ -255,19 +263,29 @@ async fn chat_completions_streaming(
} else if let Some(text) =
data["delta"]["reasoningContent"]["text"].as_str()
{
if reasoning_state == 0 {
handler.text("<think>\n")?;
reasoning_state = 1;
}
handler.text(text)?;
reasoning_state = 1;
thinking_text.push_str(text);
} else if let Some(signature) =
data["delta"]["reasoningContent"]["signature"].as_str()
{
thinking_signature.push_str(signature);
} else if let Some(redacted_data) =
data["delta"]["reasoningContent"]["redactedContent"].as_str()
{
handler.thinking_block(ThinkingBlock::RedactedThinking {
data: redacted_data.to_string(),
});
} else if let Some(input) = data["delta"]["toolUse"]["input"].as_str() {
function_arguments.push_str(input);
}
}
"contentBlockStop" => {
if reasoning_state == 1 {
handler.text("\n</think>\n\n")?;
reasoning_state = 0;
handler.thinking_block(ThinkingBlock::Thinking {
thinking: mem::take(&mut thinking_text),
signature: mem::take(&mut thinking_signature),
});
}
if !function_name.is_empty() {
if function_arguments.is_empty() {
@@ -284,6 +302,12 @@ async fn chat_completions_streaming(
function_arguments.clear();
}
}
"metadata" => {
if let Some(usage) = bedrock_parse_usage(&data["usage"]) {
debug!("token-usage: {usage:?}");
handler.usage(usage);
}
}
_ => {}
}
}
@@ -323,7 +347,11 @@ struct EmbeddingsResBody {
embeddings: Vec<Vec<f32>>,
}
fn build_chat_completions_body(data: ChatCompletionsData, model: &Model) -> Result<Value> {
fn build_chat_completions_body(
data: ChatCompletionsData,
model: &Model,
prompt_cache: bool,
) -> Result<Value> {
let ChatCompletionsData {
mut messages,
temperature,
@@ -390,49 +418,121 @@ fn build_chat_completions_body(data: ChatCompletionsData, model: &Model) -> Resu
})]
}
MessageContent::ToolCalls(MessageContentToolCalls {
tool_results, text, ..
tool_results,
text,
sequence,
}) => {
let mut assistant_parts = vec![];
let mut user_parts = vec![];
if !text.is_empty() {
assistant_parts.push(json!({
"text": text,
}))
}
for tool_result in tool_results {
if let Some(round_text) = &tool_result.text {
if !sequence {
let mut assistant_parts = vec![];
let mut user_parts = vec![];
for (index, tool_result) in tool_results.iter().enumerate() {
for block in &tool_result.thinking {
assistant_parts.push(converse_reasoning_block(block));
}
let round_text = if index == 0 && !text.is_empty() {
Some(text.as_str())
} else {
tool_result.text.as_deref()
};
if let Some(round_text) = round_text {
let round_text = strip_think_tag(round_text);
let round_text = round_text.trim();
if !round_text.is_empty() {
assistant_parts.push(json!({
"text": round_text,
}))
}
}
assistant_parts.push(json!({
"text": round_text,
}))
"toolUse": {
"toolUseId": tool_result.call.id,
"name": tool_result.call.name,
"input": tool_result.call.arguments,
}
}));
user_parts.push(json!({
"toolResult": {
"toolUseId": tool_result.call.id,
"content": [bedrock_tool_result_content(&tool_result.output)]
}
}));
}
assistant_parts.push(json!({
"toolUse": {
"toolUseId": tool_result.call.id,
"name": tool_result.call.name,
"input": tool_result.call.arguments,
// Empty tool_results (reachable via deserialized sessions) must not emit an empty pair.
if assistant_parts.is_empty() {
vec![]
} else {
vec![
json!({ "role": "assistant", "content": assistant_parts }),
json!({ "role": "user", "content": user_parts }),
]
}
} else {
// One pair per round: Claude can reuse tool_use IDs across API calls.
// A round boundary is detected by the presence of round text, but
// rounds where the model emitted only tool calls (no narration)
// carry no text marker. As a backstop, also split whenever a
// toolUseId would repeat within the current assistant message.
// Converse rejects duplicate toolUseIds in a single message.
let mut messages = vec![];
let mut assistant_parts: Vec<Value> = vec![];
let mut user_parts: Vec<Value> = vec![];
let mut chunk_ids: HashSet<&str> = HashSet::new();
for (index, tool_result) in tool_results.iter().enumerate() {
let id_collision = tool_result
.call
.id
.as_deref()
.is_some_and(|id| chunk_ids.contains(id));
if index > 0 && (tool_result.text.is_some() || id_collision) {
messages.push(
json!({ "role": "assistant", "content": assistant_parts }),
);
messages.push(json!({ "role": "user", "content": user_parts }));
assistant_parts = vec![];
user_parts = vec![];
chunk_ids.clear();
}
}));
user_parts.push(json!({
"toolResult": {
"toolUseId": tool_result.call.id,
"content": [
{
"json": tool_result.output,
}
]
if let Some(id) = tool_result.call.id.as_deref() {
chunk_ids.insert(id);
}
}));
for block in &tool_result.thinking {
assistant_parts.push(converse_reasoning_block(block));
}
let round_text = if index == 0 && !text.is_empty() {
Some(text.as_str())
} else {
tool_result.text.as_deref()
};
if let Some(round_text) = round_text {
let round_text = strip_think_tag(round_text);
let round_text = round_text.trim();
if !round_text.is_empty() {
assistant_parts.push(json!({
"text": round_text,
}))
}
}
assistant_parts.push(json!({
"toolUse": {
"toolUseId": tool_result.call.id,
"name": tool_result.call.name,
"input": tool_result.call.arguments,
}
}));
user_parts.push(json!({
"toolResult": {
"toolUseId": tool_result.call.id,
"content": [bedrock_tool_result_content(&tool_result.output)]
}
}));
}
if !assistant_parts.is_empty() {
messages
.push(json!({ "role": "assistant", "content": assistant_parts }));
messages.push(json!({ "role": "user", "content": user_parts }));
}
messages
}
vec![
json!({
"role": "assistant",
"content": assistant_parts,
}),
json!({
"role": "user",
"content": user_parts,
}),
]
}
}
})
@@ -488,13 +588,103 @@ fn build_chat_completions_body(data: ChatCompletionsData, model: &Model) -> Resu
"tools": tools,
})
}
if prompt_cache && is_anthropic_model(model) {
apply_prompt_cache_points(&mut body);
}
Ok(body)
}
/// Converse rejects cachePoint blocks for models that don't support them
/// (e.g. openai.gpt-*), so caching is only applied to Anthropic models.
fn is_anthropic_model(model: &Model) -> bool {
let name = model.real_name();
name.contains("anthropic.") || name.contains("claude")
}
/// Converse counterpart of the claude client's cache_control breakpoints:
/// a cachePoint after the tools, the system prompt, and the last two user
/// messages. See `apply_prompt_cache_breakpoints` in claude.rs for why the
/// last two.
fn apply_prompt_cache_points(body: &mut Value) {
let cache_point = || json!({ "cachePoint": { "type": "default" } });
if let Some(system) = body.get_mut("system").and_then(|v| v.as_array_mut()) {
system.push(cache_point());
}
if let Some(tools) = body
.get_mut("toolConfig")
.and_then(|v| v.get_mut("tools"))
.and_then(|v| v.as_array_mut())
{
tools.push(cache_point());
}
let Some(messages) = body.get_mut("messages").and_then(|v| v.as_array_mut()) else {
return;
};
let mut remaining = 2;
for message in messages.iter_mut().rev() {
if remaining == 0 {
break;
}
if message["role"] != "user" {
continue;
}
// Converse rejects a cachePoint with no preceding block, so an empty
// content array (a tool-call pair with no results) passes the
// breakpoint to an earlier user message.
if let Some(content) = message["content"].as_array_mut()
&& !content.is_empty()
{
content.push(cache_point());
remaining -= 1;
}
}
}
fn bedrock_parse_usage(usage: &Value) -> Option<TokenUsage> {
if !usage.is_object() {
return None;
}
Some(TokenUsage {
input_tokens: usage["inputTokens"].as_u64(),
output_tokens: usage["outputTokens"].as_u64(),
cache_creation_input_tokens: usage["cacheWriteInputTokens"].as_u64(),
cache_read_input_tokens: usage["cacheReadInputTokens"].as_u64(),
})
}
fn bedrock_tool_result_content(output: &Value) -> Value {
match output {
Value::Object(_) => json!({ "json": output }),
Value::String(text) => json!({ "text": text }),
_ => json!({ "text": output.to_string() }),
}
}
/// Maps a ThinkingBlock (Anthropic tagged shape) to the Converse
/// reasoningContent union so signed blocks replay verbatim.
fn converse_reasoning_block(block: &ThinkingBlock) -> Value {
match block {
ThinkingBlock::Thinking {
thinking,
signature,
} => json!({
"reasoningContent": {
"reasoningText": {
"text": thinking,
"signature": signature,
}
}
}),
ThinkingBlock::RedactedThinking { data } => json!({
"reasoningContent": { "redactedContent": data }
}),
}
}
fn extract_chat_completions(data: &Value) -> Result<ChatCompletionsOutput> {
let mut text = String::new();
let mut reasoning = None;
let mut tool_calls = vec![];
let mut thinking = vec![];
if let Some(array) = data["output"]["message"]["content"].as_array() {
for item in array {
if let Some(v) = item["text"].as_str() {
@@ -505,9 +695,18 @@ fn extract_chat_completions(data: &Value) -> Result<ChatCompletionsOutput> {
} else if let Some(reasoning_text) =
item["reasoningContent"]["reasoningText"].as_object()
{
if let Some(text) = json_str_from_map(reasoning_text, "text") {
reasoning = Some(text.to_string());
if let Some(v) = json_str_from_map(reasoning_text, "text") {
thinking.push(ThinkingBlock::Thinking {
thinking: v.to_string(),
signature: json_str_from_map(reasoning_text, "signature")
.unwrap_or_default()
.to_string(),
});
}
} else if let Some(v) = item["reasoningContent"]["redactedContent"].as_str() {
thinking.push(ThinkingBlock::RedactedThinking {
data: v.to_string(),
});
} else if let Some(tool_use) = item["toolUse"].as_object()
&& let (Some(id), Some(name), Some(input)) = (
json_str_from_map(tool_use, "toolUseId"),
@@ -524,10 +723,6 @@ fn extract_chat_completions(data: &Value) -> Result<ChatCompletionsOutput> {
}
}
if let Some(reasoning) = reasoning {
text = format!("<think>\n{reasoning}\n</think>\n\n{text}")
}
if text.is_empty() && tool_calls.is_empty() {
bail!("Invalid response data: {data}");
}
@@ -535,7 +730,8 @@ fn extract_chat_completions(data: &Value) -> Result<ChatCompletionsOutput> {
let output = ChatCompletionsOutput {
text,
tool_calls,
..Default::default()
thinking,
usage: bedrock_parse_usage(&data["usage"]),
};
Ok(output)
}
@@ -653,3 +849,385 @@ fn gen_signing_key(key: &str, date_stamp: &str, region: &str, service: &str) ->
let k_service = hmac_sha256(&k_region, service);
hmac_sha256(&k_service, "aws4_request")
}
#[cfg(test)]
mod tests {
use super::*;
use crate::function::{FunctionDeclaration, ToolCall, ToolResult};
fn tool_result(output: Value) -> ToolResult {
ToolResult {
call: ToolCall::new(
"fs_read".into(),
json!({"path": "x"}),
Some("tool_A".into()),
),
output,
text: None,
thinking: vec![],
}
}
fn sequence_tool_result(id: &str, text: Option<&str>) -> ToolResult {
ToolResult {
call: ToolCall::new("fs_read".into(), json!({"path": "x"}), Some(id.into())),
output: json!("ok"),
text: text.map(|t| t.to_string()),
thinking: vec![],
}
}
fn build_body_with(tool_results: Vec<ToolResult>, text: &str, sequence: bool) -> Value {
let data = ChatCompletionsData {
messages: vec![
Message::new(MessageRole::User, MessageContent::Text("hello".to_string())),
Message::new(
MessageRole::Assistant,
MessageContent::ToolCalls(MessageContentToolCalls {
tool_results,
text: text.to_string(),
sequence,
}),
),
],
temperature: None,
top_p: None,
reasoning_effort: None,
functions: None,
stream: false,
};
build_chat_completions_body(data, &Model::new("bedrock", "test"), false).unwrap()
}
fn build_body(tool_results: Vec<ToolResult>) -> Value {
build_body_with(tool_results, "", true)
}
fn tool_result_content_block(output: Value) -> Value {
let body = build_body(vec![tool_result(output)]);
body["messages"][2]["content"][0]["toolResult"]["content"][0].clone()
}
fn assert_unique_tool_use_ids_per_message(body: &Value) {
for message in body["messages"].as_array().unwrap() {
let Some(content) = message["content"].as_array() else {
continue;
};
let mut seen = HashSet::new();
for block in content {
if let Some(id) = block["toolUse"]["toolUseId"].as_str() {
assert!(
seen.insert(id.to_string()),
"duplicate toolUseId `{id}` within a single assistant message: {message}"
);
}
}
}
}
fn cache_fixture() -> ChatCompletionsData {
ChatCompletionsData {
messages: vec![
Message::new(MessageRole::System, MessageContent::Text("sys".to_string())),
Message::new(MessageRole::User, MessageContent::Text("hello".to_string())),
Message::new(
MessageRole::Assistant,
MessageContent::Text("hi".to_string()),
),
Message::new(MessageRole::User, MessageContent::Text("again".to_string())),
],
temperature: None,
top_p: None,
reasoning_effort: None,
functions: Some(vec![FunctionDeclaration {
name: "fs_read".to_string(),
description: "Read a file".to_string(),
parameters: Default::default(),
agent: false,
}]),
stream: false,
}
}
#[test]
fn test_bedrock_prompt_cache_inserts_cache_points_for_anthropic_models() {
let body = build_chat_completions_body(
cache_fixture(),
&Model::new("bedrock", "anthropic.claude-sonnet-4-5-20250929-v1:0"),
true,
)
.unwrap();
let cache_point = json!({ "cachePoint": { "type": "default" } });
assert_eq!(
body["system"].as_array().unwrap().last(),
Some(&cache_point),
"body: {body}"
);
assert_eq!(
body["toolConfig"]["tools"].as_array().unwrap().last(),
Some(&cache_point),
"body: {body}"
);
let messages = body["messages"].as_array().unwrap();
for idx in [0, 2] {
assert_eq!(
messages[idx]["content"].as_array().unwrap().last(),
Some(&cache_point),
"body: {body}"
);
}
assert!(
messages[1]["content"]
.as_array()
.unwrap()
.iter()
.all(|block| block.get("cachePoint").is_none()),
"assistant messages must not get cache points: {body}"
);
}
#[test]
fn test_bedrock_prompt_cache_skips_non_anthropic_models() {
let body = build_chat_completions_body(
cache_fixture(),
&Model::new("bedrock", "openai.gpt-oss-120b-1:0"),
true,
)
.unwrap();
assert!(!body.to_string().contains("cachePoint"), "body: {body}");
}
/// Converse rejects a cachePoint with no preceding block, so a user
/// message with an empty content array must be skipped and the
/// breakpoint placed on an earlier user message instead.
#[test]
fn test_bedrock_prompt_cache_skips_empty_user_content() {
let mut body = json!({
"messages": [
{ "role": "user", "content": [ { "text": "hello" } ] },
{ "role": "assistant", "content": [ { "text": "hi" } ] },
{ "role": "user", "content": [ { "text": "again" } ] },
{ "role": "assistant", "content": [ { "text": "calling" } ] },
{ "role": "user", "content": [] },
],
});
apply_prompt_cache_points(&mut body);
let cache_point = json!({ "cachePoint": { "type": "default" } });
let messages = body["messages"].as_array().unwrap();
assert!(
messages[4]["content"].as_array().unwrap().is_empty(),
"empty user content must not get a cachePoint: {body}"
);
for idx in [0, 2] {
assert_eq!(
messages[idx]["content"].as_array().unwrap().last(),
Some(&cache_point),
"body: {body}"
);
}
}
#[test]
fn sequence_splits_on_round_text() {
let body = build_body(vec![
sequence_tool_result("toolu_A", None),
sequence_tool_result("toolu_B", None),
sequence_tool_result("toolu_C", Some("running another tool")),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 5, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn sequence_splits_on_reused_id_in_textless_round() {
let body = build_body(vec![
sequence_tool_result("toolu_A", None),
sequence_tool_result("toolu_B", None),
sequence_tool_result("toolu_A", None),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 5, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn sequence_keeps_textless_rounds_merged_when_ids_are_unique() {
let body = build_body(vec![
sequence_tool_result("toolu_A", None),
sequence_tool_result("toolu_B", None),
sequence_tool_result("toolu_C", None),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 3, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn non_sequence_emits_assistant_user_pair() {
let body = build_body_with(
vec![
sequence_tool_result("toolu_A", None),
sequence_tool_result("toolu_B", None),
],
"",
false,
);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 3, "body: {body}");
assert_eq!(messages[1]["role"], "assistant");
assert_eq!(messages[1]["content"].as_array().unwrap().len(), 2);
assert_eq!(messages[2]["role"], "user");
assert_eq!(messages[2]["content"].as_array().unwrap().len(), 2);
}
#[test]
fn non_sequence_empty_tool_results_emits_no_messages() {
let body = build_body_with(vec![], "leftover text", false);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
#[test]
fn sequence_empty_tool_results_emits_no_messages() {
let body = build_body_with(vec![], "leftover text", true);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
#[test]
fn test_bedrock_tool_result_object_uses_json_block() {
assert_eq!(
tool_result_content_block(json!({"output": "ok"})),
json!({"json": {"output": "ok"}})
);
assert_eq!(tool_result_content_block(json!({})), json!({"json": {}}));
}
#[test]
fn test_bedrock_tool_result_array_uses_text_block() {
assert_eq!(
tool_result_content_block(json!([{"id": 1}, {"id": 2}])),
json!({"text": "[{\"id\":1},{\"id\":2}]"})
);
}
#[test]
fn test_bedrock_tool_result_scalars_use_text_block() {
assert_eq!(
tool_result_content_block(json!("plain text")),
json!({"text": "plain text"})
);
assert_eq!(tool_result_content_block(json!(42)), json!({"text": "42"}));
assert_eq!(
tool_result_content_block(json!(null)),
json!({"text": "null"})
);
}
fn thinking_tool_result() -> ToolResult {
ToolResult {
call: ToolCall::new(
"fs_read".into(),
json!({"path": "x"}),
Some("tool_A".into()),
),
output: json!("ok"),
text: None,
thinking: vec![
ThinkingBlock::Thinking {
thinking: "let me think".into(),
signature: "sig123".into(),
},
ThinkingBlock::RedactedThinking {
data: "b64data".into(),
},
],
}
}
#[test]
fn replays_thinking_blocks_in_converse_shape() {
for sequence in [false, true] {
let body = build_body_with(vec![thinking_tool_result()], "", sequence);
let content = body["messages"][1]["content"].as_array().unwrap();
assert_eq!(
content[0],
json!({
"reasoningContent": {
"reasoningText": { "text": "let me think", "signature": "sig123" }
}
}),
"body: {body}"
);
assert_eq!(
content[1],
json!({ "reasoningContent": { "redactedContent": "b64data" } }),
"body: {body}"
);
assert!(content[2].get("toolUse").is_some(), "body: {body}");
}
}
#[test]
fn extract_chat_completions_captures_reasoning_content() {
let data = json!({
"output": {
"message": {
"content": [
{
"reasoningContent": {
"reasoningText": { "text": "let me think", "signature": "sig123" }
}
},
{ "reasoningContent": { "redactedContent": "b64data" } },
{ "text": "answer" },
{
"toolUse": {
"toolUseId": "tool_A",
"name": "fs_read",
"input": { "path": "x" }
}
}
]
}
}
});
let output = extract_chat_completions(&data).unwrap();
assert_eq!(output.text, "answer");
assert_eq!(output.tool_calls.len(), 1);
assert_eq!(output.thinking.len(), 2);
match &output.thinking[0] {
ThinkingBlock::Thinking {
thinking,
signature,
} => {
assert_eq!(thinking, "let me think");
assert_eq!(signature, "sig123");
}
other => panic!("unexpected block: {other:?}"),
}
assert!(matches!(
&output.thinking[1],
ThinkingBlock::RedactedThinking { data } if data == "b64data"
));
}
}
src/client/catalog.rs
+91 -17
View File
@@ -201,6 +201,8 @@ fn modelsdev_provider_models(api: &Value, provider_id: &str) -> Vec<SourcedModel
.and_then(clamp_output_tokens);
data.input_price = entry["cost"]["input"].as_f64().and_then(clamp_price);
data.output_price = entry["cost"]["output"].as_f64().and_then(clamp_price);
data.cache_read_price = entry["cost"]["cache_read"].as_f64().and_then(clamp_price);
data.cache_write_price = entry["cost"]["cache_write"].as_f64().and_then(clamp_price);
data.supports_vision = entry["modalities"]["input"]
.as_array()
.is_some_and(|modes| modes.iter().any(|mode| mode.as_str() == Some("image")));
@@ -241,6 +243,8 @@ fn openrouter_model_data(name: &str, entry: &Value) -> ModelData {
.and_then(clamp_output_tokens);
data.input_price = openrouter_price(&entry["pricing"]["prompt"]);
data.output_price = openrouter_price(&entry["pricing"]["completion"]);
data.cache_read_price = openrouter_price(&entry["pricing"]["input_cache_read"]);
data.cache_write_price = openrouter_price(&entry["pricing"]["input_cache_write"]);
data.supports_vision = entry["architecture"]["modality"]
.as_str()
.is_some_and(|modality| modality.contains("image"));
@@ -294,6 +298,12 @@ fn fill_from_openrouter(data: &mut ModelData, entry: &Value) {
if data.output_price.is_none() {
data.output_price = filler.output_price;
}
if data.cache_read_price.is_none() {
data.cache_read_price = filler.cache_read_price;
}
if data.cache_write_price.is_none() {
data.cache_write_price = filler.cache_write_price;
}
if !data.supports_vision {
data.supports_vision = filler.supports_vision;
}
@@ -316,11 +326,13 @@ pub struct CatalogBuild {
///
/// Per provider: API models (models.dev primary, OpenRouter gap-fill) are
/// curated by `exclude` globs and `rules`, hand-owned `models:` entries are
/// merged in field-wise (hand-owned wins) and placed first, API models are
/// sorted newest-first by release date (name-ascending tiebreak, undated
/// last), and `variants` clones are inserted immediately after their base
/// model. Providers without usable data are routed to `degraded` or `failed`
/// instead of erroring; a provider never emits zero models.
/// merged in field-wise (hand-owned wins, inheriting the API model's release
/// date), and all models are sorted newest-first by release date
/// (name-ascending tiebreak). Undated models sort last: hand-owned ones in
/// quirks.yaml order (preserving curation for quirks-owned providers), then
/// API ones name-ascending. `variants` clones are inserted immediately after
/// their base model. Providers without usable data are routed to `degraded`
/// or `failed` instead of erroring; a provider never emits zero models.
pub fn build_catalog(sources: &FetchedSources, quirks: &[ProviderQuirks]) -> CatalogBuild {
let modelsdev: Option<Value> = sources
.modelsdev
@@ -398,9 +410,15 @@ pub fn build_catalog(sources: &FetchedSources, quirks: &[ProviderQuirks]) -> Cat
{
Some(position) => {
let api = api_models.remove(position);
merged_hand.push(overlay_hand_owned(hand, api.data));
merged_hand.push(SourcedModel {
data: overlay_hand_owned(hand, api.data),
release_date: api.release_date,
});
}
None => merged_hand.push(hand),
None => merged_hand.push(SourcedModel {
data: hand,
release_date: None,
}),
}
}
@@ -413,9 +431,17 @@ pub fn build_catalog(sources: &FetchedSources, quirks: &[ProviderQuirks]) -> Cat
(None, None) => a.data.name.cmp(&b.data.name),
});
let bases = merged_hand
.into_iter()
.chain(api_models.into_iter().map(|model| model.data));
let mut all_models = merged_hand;
all_models.extend(api_models);
all_models.sort_by(|a, b| match (&a.release_date, &b.release_date) {
(Some(a_date), Some(b_date)) => b_date
.cmp(a_date)
.then_with(|| a.data.name.cmp(&b.data.name)),
(Some(_), None) => Ordering::Less,
(None, Some(_)) => Ordering::Greater,
(None, None) => Ordering::Equal,
});
let bases = all_models.into_iter().map(|model| model.data);
let variants = provider_quirks
.map(|provider_quirks| provider_quirks.variants.as_slice())
@@ -554,6 +580,8 @@ fn overlay_hand_owned(hand: ModelData, api: ModelData) -> ModelData {
max_input_tokens: hand.max_input_tokens.or(api.max_input_tokens),
input_price: hand.input_price.or(api.input_price),
output_price: hand.output_price.or(api.output_price),
cache_read_price: hand.cache_read_price.or(api.cache_read_price),
cache_write_price: hand.cache_write_price.or(api.cache_write_price),
patch: hand.patch.or(api.patch),
max_output_tokens: hand.max_output_tokens.or(api.max_output_tokens),
require_max_tokens: hand.require_max_tokens || api.require_max_tokens,
@@ -990,6 +1018,8 @@ rules:
assert_eq!(gpt56.data.max_output_tokens, Some(128_000));
assert_eq!(gpt56.data.input_price, Some(4.0));
assert_eq!(gpt56.data.output_price, Some(20.0));
assert_eq!(gpt56.data.cache_read_price, Some(0.4));
assert_eq!(gpt56.data.cache_write_price, Some(5.0));
assert!(gpt56.data.supports_vision);
assert!(gpt56.data.supports_function_calling);
assert_eq!(
@@ -1006,6 +1036,8 @@ rules:
assert_eq!(qwen.data.max_output_tokens, Some(8_192));
assert_eq!(qwen.data.input_price, Some(1.6));
assert_eq!(qwen.data.output_price, Some(6.4));
assert_eq!(qwen.data.cache_read_price, None);
assert_eq!(qwen.data.cache_write_price, None);
assert!(!qwen.data.supports_vision);
assert!(qwen.data.supports_function_calling);
assert!(qwen.data.reasoning_levels.is_empty());
@@ -1024,9 +1056,20 @@ rules:
assert_eq!(data.max_output_tokens, Some(32_768));
assert_eq!(data.input_price, Some(2.0));
assert_eq!(data.output_price, Some(8.0));
assert_eq!(data.cache_read_price, Some(0.5));
assert_eq!(data.cache_write_price, None);
assert!(data.supports_vision);
assert!(data.supports_function_calling);
let qwen = openrouter_model_data("qwen3.8-max", index["qwen/qwen3.8-max"]);
assert_eq!(qwen.cache_read_price, Some(0.25));
assert_eq!(qwen.cache_write_price, Some(2.5));
// input_cache_write_1h must not leak into cache_write_price
let sonnet =
openrouter_model_data("claude-sonnet-4.6", index["anthropic/claude-sonnet-4.6"]);
assert_eq!(sonnet.cache_write_price, Some(3.75));
let rounded = openrouter_model_data(
"m",
&json!({"pricing": {"prompt": "0.00000123456789", "completion": "0.0000025"}}),
@@ -1040,7 +1083,7 @@ rules:
let api = json!({
"p": {"models": {"m": {
"limit": {"context": 100_000_000, "output": 0},
"cost": {"input": -1.0, "output": 20_000.0},
"cost": {"input": -1.0, "output": 20_000.0, "cache_read": -0.5, "cache_write": 20_000.0},
}}}
});
let models = modelsdev_provider_models(&api, "p");
@@ -1051,19 +1094,23 @@ rules:
assert_eq!(data.max_output_tokens, None);
assert_eq!(data.input_price, None);
assert_eq!(data.output_price, None);
assert_eq!(data.cache_read_price, None);
assert_eq!(data.cache_write_price, None);
let data = openrouter_model_data(
"m",
&json!({
"context_length": 60_000_000,
"top_provider": {"max_completion_tokens": 20_000_000},
"pricing": {"prompt": "-0.000001", "completion": "999"},
"pricing": {"prompt": "-0.000001", "completion": "999", "input_cache_read": "-0.000001", "input_cache_write": "999"},
}),
);
assert_eq!(data.max_input_tokens, None);
assert_eq!(data.max_output_tokens, None);
assert_eq!(data.input_price, None);
assert_eq!(data.output_price, None);
assert_eq!(data.cache_read_price, None);
assert_eq!(data.cache_write_price, None);
}
#[test]
@@ -1119,7 +1166,7 @@ rules:
fn merge_precedence_hand_owned_beats_rule_beats_modelsdev_beats_openrouter() {
let modelsdev = json!({
"xai": {"models": {
"m-1": {"limit": {"context": 1000}, "cost": {"input": 1.0}},
"m-1": {"limit": {"context": 1000}, "cost": {"input": 1.0, "cache_read": 0.2}},
"m-2": {"limit": {"context": 1000}},
}}
})
@@ -1128,7 +1175,7 @@ rules:
{
"id": "x-ai/m-1",
"context_length": 2000,
"pricing": {"prompt": "0.000003", "completion": "0.000004"},
"pricing": {"prompt": "0.000003", "completion": "0.000004", "input_cache_read": "0.0000001", "input_cache_write": "0.000001"},
"top_provider": {"max_completion_tokens": 512},
},
{"id": "x-ai/m-2", "context_length": 2000},
@@ -1144,6 +1191,7 @@ rules:
models:
- name: m-1
input_price: 7
cache_read_price: 0.7
"#,
);
let build = build_catalog(&make_sources(Some(&modelsdev), Some(&openrouter)), &quirks);
@@ -1152,11 +1200,13 @@ rules:
let merged = find_model(xai, "m-1");
// hand-owned beats the rule, which beat models.dev, which beat openrouter
assert_eq!(merged.input_price, Some(7.0));
assert_eq!(merged.cache_read_price, Some(0.7));
// rule beats models.dev
assert_eq!(merged.max_input_tokens, Some(3000));
// openrouter fills fields models.dev left unset
assert_eq!(merged.output_price, Some(4.0));
assert_eq!(merged.max_output_tokens, Some(512));
assert_eq!(merged.cache_write_price, Some(1.0));
// models.dev beats openrouter on the same field
assert_eq!(find_model(xai, "m-2").max_input_tokens, Some(1000));
@@ -1222,7 +1272,7 @@ rules:
}
#[test]
fn ordering_hand_owned_first_then_release_date_desc_name_asc() {
fn ordering_release_date_desc_hand_owned_interleaved_undated_last() {
let shuffled_payloads = [
r#"{"openai":{"models":{
"b":{"release_date":"2026-05-01"},
@@ -1245,16 +1295,25 @@ rules:
models:
- name: n2
- name: n1
- name: b
input_price: 9
"#,
);
for payload in shuffled_payloads {
let build = build_catalog(&make_sources(Some(payload), None), &quirks);
let names: Vec<&str> = find_provider(&build, "openai")
let openai = find_provider(&build, "openai");
let names: Vec<&str> = openai
.models
.iter()
.map(|model| model.name.as_str())
.collect();
assert_eq!(names, ["n2", "n1", "c", "a", "b", "y", "z"]);
// Dated models sort newest-first with a name-ascending tiebreak;
// hand-owned "b" inherits its API release date and interleaves
// instead of pinning to the top. Undated models sort last:
// hand-owned in quirks order (n2, n1), then API name-ascending.
assert_eq!(names, ["c", "a", "b", "n2", "n1", "y", "z"]);
// "b" is still the hand-owned overlay, not the bare API entry.
assert_eq!(find_model(openai, "b").input_price, Some(9.0));
}
}
@@ -1267,6 +1326,21 @@ rules:
assert_eq!(serde_yaml::to_string(&parsed).unwrap(), yaml);
}
#[test]
fn cache_prices_omitted_when_absent_round_trip_when_set() {
let yaml = serde_yaml::to_string(&ModelData::new("m")).unwrap();
assert!(!yaml.contains("cache_read_price"));
assert!(!yaml.contains("cache_write_price"));
let mut data = ModelData::new("m");
data.cache_read_price = Some(0.25);
data.cache_write_price = Some(2.5);
let parsed: ModelData =
serde_yaml::from_str(&serde_yaml::to_string(&data).unwrap()).unwrap();
assert_eq!(parsed.cache_read_price, Some(0.25));
assert_eq!(parsed.cache_write_price, Some(2.5));
}
#[test]
fn openrouter_native_models_use_full_ids() {
let build = build_catalog(&fixture_sources(), &[]);
src/client/claude.rs
+286 -25
View File
@@ -24,6 +24,7 @@ pub struct ClaudeConfig {
pub auth: Option<String>,
#[serde(default)]
pub models: Vec<ModelData>,
pub prompt_cache: Option<bool>,
pub patch: Option<RequestPatch>,
pub extra: Option<ExtraConfig>,
}
@@ -75,7 +76,11 @@ async fn prepare_chat_completions(
.unwrap_or_else(|_| API_BASE.to_string());
let url = format!("{}/messages", api_base.trim_end_matches('/'));
let body = claude_build_chat_completions_body(data, &self_.model)?;
let body = claude_build_chat_completions_body(
data,
&self_.model,
self_.config.prompt_cache.unwrap_or(true),
)?;
let mut request_data = RequestData::new(url, body);
@@ -250,6 +255,18 @@ pub async fn claude_chat_completions_streaming(
function_arguments.clear();
}
}
"message_start" => {
if let Some(usage) = claude_parse_usage(&data["message"]["usage"]) {
debug!("token-usage: {usage:?}");
handler.usage(usage);
}
}
"message_delta" => {
if let Some(usage) = claude_parse_usage(&data["usage"]) {
debug!("token-usage: {usage:?}");
handler.usage(usage);
}
}
_ => {}
}
}
@@ -262,6 +279,7 @@ pub async fn claude_chat_completions_streaming(
pub fn claude_build_chat_completions_body(
data: ChatCompletionsData,
model: &Model,
prompt_cache: bool,
) -> Result<Value> {
let ChatCompletionsData {
mut messages,
@@ -363,10 +381,15 @@ pub fn claude_build_chat_completions_body(
"content": tool_result.output.to_string(),
}));
}
vec![
json!({ "role": "assistant", "content": assistant_parts }),
json!({ "role": "user", "content": user_parts }),
]
// Empty tool_results (reachable via deserialized sessions) must not emit an empty pair.
if assistant_parts.is_empty() {
vec![]
} else {
vec![
json!({ "role": "assistant", "content": assistant_parts }),
json!({ "role": "user", "content": user_parts }),
]
}
} else {
// One pair per round: Claude can reuse tool_use IDs across API calls.
// A round boundary is detected by the presence of round text, but
@@ -487,9 +510,65 @@ pub fn claude_build_chat_completions_body(
})
.collect();
}
if prompt_cache {
apply_prompt_cache_breakpoints(&mut body);
}
Ok(body)
}
/// Anthropic allows at most 4 cache_control breakpoints; all four are spent
/// here: the last tool, the system prompt, and the last two user messages.
/// Two moving breakpoints guarantee a cache read every turn: turn N+1's
/// second-to-last breakpoint sits exactly where turn N's last one was.
fn apply_prompt_cache_breakpoints(body: &mut Value) {
if let Some(text) = body["system"].as_str().map(str::to_string) {
body["system"] = json!([{
"type": "text",
"text": text,
"cache_control": { "type": "ephemeral" },
}]);
}
if let Some(tool) = body
.get_mut("tools")
.and_then(|v| v.as_array_mut())
.and_then(|v| v.last_mut())
{
tool["cache_control"] = json!({ "type": "ephemeral" });
}
let Some(messages) = body.get_mut("messages").and_then(|v| v.as_array_mut()) else {
return;
};
let mut remaining = 2;
for message in messages.iter_mut().rev() {
if remaining == 0 {
break;
}
if message["role"] != "user" {
continue;
}
if let Some(text) = message["content"].as_str().map(str::to_string) {
message["content"] = json!([{ "type": "text", "text": text }]);
}
if let Some(block) = message["content"].as_array_mut().and_then(|v| v.last_mut()) {
block["cache_control"] = json!({ "type": "ephemeral" });
remaining -= 1;
}
}
}
fn claude_parse_usage(usage: &Value) -> Option<TokenUsage> {
if !usage.is_object() {
return None;
}
Some(TokenUsage {
input_tokens: usage["input_tokens"].as_u64(),
output_tokens: usage["output_tokens"].as_u64(),
cache_creation_input_tokens: usage["cache_creation_input_tokens"].as_u64(),
cache_read_input_tokens: usage["cache_read_input_tokens"].as_u64(),
})
}
pub fn claude_extract_chat_completions(data: &Value) -> Result<ChatCompletionsOutput> {
let mut text = String::new();
let mut tool_calls = vec![];
@@ -545,6 +624,7 @@ pub fn claude_extract_chat_completions(data: &Value) -> Result<ChatCompletionsOu
text: text.to_string(),
tool_calls,
thinking,
usage: claude_parse_usage(&data["usage"]),
};
Ok(output)
}
@@ -552,7 +632,7 @@ pub fn claude_extract_chat_completions(data: &Value) -> Result<ChatCompletionsOu
#[cfg(test)]
mod tests {
use super::*;
use crate::function::{ToolCall, ToolResult};
use crate::function::{FunctionDeclaration, ToolCall, ToolResult};
fn tool_result(id: &str, text: Option<&str>) -> ToolResult {
ToolResult {
@@ -563,7 +643,12 @@ mod tests {
}
}
fn build_body(tool_results: Vec<ToolResult>) -> Value {
fn build_body_with(
tool_results: Vec<ToolResult>,
text: &str,
sequence: bool,
prompt_cache: bool,
) -> Value {
let data = ChatCompletionsData {
messages: vec![
Message::new(MessageRole::User, MessageContent::Text("hello".to_string())),
@@ -571,8 +656,8 @@ mod tests {
MessageRole::Assistant,
MessageContent::ToolCalls(MessageContentToolCalls {
tool_results,
text: String::new(),
sequence: true,
text: text.to_string(),
sequence,
}),
),
],
@@ -582,7 +667,48 @@ mod tests {
functions: None,
stream: false,
};
claude_build_chat_completions_body(data, &Model::new("claude", "claude-test")).unwrap()
claude_build_chat_completions_body(data, &Model::new("claude", "claude-test"), prompt_cache)
.unwrap()
}
fn build_body(tool_results: Vec<ToolResult>, prompt_cache: bool) -> Value {
build_body_with(tool_results, "", true, prompt_cache)
}
fn multi_turn_data(functions: Option<Vec<FunctionDeclaration>>) -> ChatCompletionsData {
ChatCompletionsData {
messages: vec![
Message::new(MessageRole::System, MessageContent::Text("sys".to_string())),
Message::new(MessageRole::User, MessageContent::Text("first".to_string())),
Message::new(
MessageRole::Assistant,
MessageContent::Text("reply".to_string()),
),
Message::new(
MessageRole::User,
MessageContent::Text("second".to_string()),
),
Message::new(
MessageRole::Assistant,
MessageContent::Text("reply2".to_string()),
),
Message::new(MessageRole::User, MessageContent::Text("third".to_string())),
],
temperature: None,
top_p: None,
reasoning_effort: None,
functions,
stream: false,
}
}
fn function_declaration(name: &str) -> FunctionDeclaration {
FunctionDeclaration {
name: name.to_string(),
description: format!("{name} description"),
parameters: Default::default(),
agent: false,
}
}
fn assert_unique_tool_use_ids_per_message(body: &Value) {
@@ -605,11 +731,14 @@ mod tests {
#[test]
fn sequence_splits_on_round_text() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", Some("running another tool")),
]);
let body = build_body(
vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", Some("running another tool")),
],
false,
);
let messages = body["messages"].as_array().unwrap();
@@ -619,11 +748,14 @@ mod tests {
#[test]
fn sequence_splits_on_reused_id_in_textless_round() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_A", None),
]);
let body = build_body(
vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_A", None),
],
false,
);
let messages = body["messages"].as_array().unwrap();
@@ -633,15 +765,144 @@ mod tests {
#[test]
fn sequence_keeps_textless_rounds_merged_when_ids_are_unique() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", None),
]);
let body = build_body(
vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", None),
],
false,
);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 3, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn non_sequence_emits_assistant_user_pair() {
let body = build_body_with(
vec![tool_result("toolu_A", None), tool_result("toolu_B", None)],
"",
false,
false,
);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 3, "body: {body}");
assert_eq!(messages[1]["role"], "assistant");
assert_eq!(messages[1]["content"].as_array().unwrap().len(), 2);
assert_eq!(messages[2]["role"], "user");
assert_eq!(messages[2]["content"].as_array().unwrap().len(), 2);
}
#[test]
fn non_sequence_empty_tool_results_emits_no_messages() {
let body = build_body_with(vec![], "leftover text", false, false);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
#[test]
fn sequence_empty_tool_results_emits_no_messages() {
let body = build_body_with(vec![], "leftover text", true, false);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
#[test]
fn prompt_cache_places_breakpoints() {
let functions = vec![function_declaration("a"), function_declaration("b")];
let body = claude_build_chat_completions_body(
multi_turn_data(Some(functions)),
&Model::new("claude", "claude-test"),
true,
)
.unwrap();
let cache_control = json!({ "type": "ephemeral" });
let tools = body["tools"].as_array().unwrap();
assert!(tools[0].get("cache_control").is_none(), "body: {body}");
assert_eq!(tools[1]["cache_control"], cache_control);
let system = body["system"].as_array().unwrap();
assert_eq!(system.last().unwrap()["text"], "sys");
assert_eq!(system.last().unwrap()["cache_control"], cache_control);
let messages = body["messages"].as_array().unwrap();
assert!(
messages[0]["content"].is_string(),
"older user messages stay untouched: {body}"
);
for idx in [2, 4] {
let blocks = messages[idx]["content"].as_array().unwrap();
let last = blocks.last().unwrap();
assert_eq!(last["type"], "text", "body: {body}");
assert_eq!(last["cache_control"], cache_control, "body: {body}");
}
}
#[test]
fn prompt_cache_marks_last_tool_result_block() {
let body = build_body(
vec![tool_result("toolu_A", None), tool_result("toolu_B", None)],
true,
);
assert!(body.get("system").is_none(), "body: {body}");
assert!(body.get("tools").is_none(), "body: {body}");
let cache_control = json!({ "type": "ephemeral" });
let messages = body["messages"].as_array().unwrap();
let last_blocks = messages.last().unwrap()["content"].as_array().unwrap();
let last_block = last_blocks.last().unwrap();
assert_eq!(last_block["type"], "tool_result", "body: {body}");
assert_eq!(last_block["cache_control"], cache_control);
assert!(last_blocks[0].get("cache_control").is_none());
let first_blocks = messages[0]["content"].as_array().unwrap();
assert_eq!(first_blocks.last().unwrap()["type"], "text");
assert_eq!(first_blocks.last().unwrap()["cache_control"], cache_control);
}
#[test]
fn prompt_cache_disabled_leaves_body_unchanged() {
let functions = vec![function_declaration("a")];
let body = claude_build_chat_completions_body(
multi_turn_data(Some(functions)),
&Model::new("claude", "claude-test"),
false,
)
.unwrap();
assert!(body["system"].is_string(), "body: {body}");
assert!(!body.to_string().contains("cache_control"), "body: {body}");
}
#[test]
fn extract_populates_usage() {
let data = json!({
"content": [{ "type": "text", "text": "hi" }],
"usage": {
"input_tokens": 10,
"output_tokens": 5,
"cache_creation_input_tokens": 100,
"cache_read_input_tokens": 200,
}
});
let output = claude_extract_chat_completions(&data).unwrap();
let usage = output.usage.unwrap();
assert_eq!(usage.input_tokens, Some(10));
assert_eq!(usage.output_tokens, Some(5));
assert_eq!(usage.cache_creation_input_tokens, Some(100));
assert_eq!(usage.cache_read_input_tokens, Some(200));
}
}
src/client/common.rs
+534 -20
View File
@@ -17,7 +17,7 @@ use inquire::{
MultiSelect, Select, Text, list_option::ListOption, required, validator::Validation,
};
use reqwest::{Client as ReqwestClient, RequestBuilder};
use serde::Deserialize;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use std::sync::LazyLock;
use std::time::Duration;
@@ -357,6 +357,7 @@ pub struct ChatCompletionsOutput {
pub text: String,
pub tool_calls: Vec<ToolCall>,
pub thinking: Vec<ThinkingBlock>,
pub usage: Option<TokenUsage>,
}
impl ChatCompletionsOutput {
@@ -368,6 +369,62 @@ impl ChatCompletionsOutput {
}
}
#[derive(Debug, Clone, Default, Deserialize, Serialize)]
pub struct TokenUsage {
pub input_tokens: Option<u64>,
pub output_tokens: Option<u64>,
pub cache_creation_input_tokens: Option<u64>,
pub cache_read_input_tokens: Option<u64>,
}
impl TokenUsage {
pub fn is_empty(&self) -> bool {
self.input_tokens.is_none()
&& self.output_tokens.is_none()
&& self.cache_creation_input_tokens.is_none()
&& self.cache_read_input_tokens.is_none()
}
pub fn accumulate(&mut self, other: &TokenUsage) {
fn add(acc: &mut Option<u64>, value: Option<u64>) {
if let Some(value) = value {
*acc = Some(acc.unwrap_or(0).saturating_add(value));
}
}
add(&mut self.input_tokens, other.input_tokens);
add(&mut self.output_tokens, other.output_tokens);
add(
&mut self.cache_creation_input_tokens,
other.cache_creation_input_tokens,
);
add(
&mut self.cache_read_input_tokens,
other.cache_read_input_tokens,
);
}
/// USD cost of this usage at the given model's prices (USD per 1M
/// tokens). The four buckets are disjoint on both Anthropic direct and
/// Bedrock Converse (`input_tokens` excludes cache tokens), so the terms
/// sum without adjustment. Strict on missing prices: any nonzero
/// component without a price yields `None` rather than a silent
/// undercount. All-zero usage is also `None` — nothing billed, nothing
/// to show.
pub fn cost_usd(&self, model: &Model) -> Option<f64> {
fn term(tokens: Option<u64>, price: Option<f64>) -> Option<f64> {
match tokens.unwrap_or(0) {
0 => Some(0.0),
tokens => Some(tokens as f64 * price? / 1_000_000.0),
}
}
let total = term(self.input_tokens, model.input_price())?
+ term(self.output_tokens, model.output_price())?
+ term(self.cache_read_input_tokens, model.cache_read_price())?
+ term(self.cache_creation_input_tokens, model.cache_write_price())?;
(total > 0.0).then_some(total)
}
}
#[derive(Debug)]
pub struct EmbeddingsData {
pub texts: Vec<String>,
@@ -513,7 +570,7 @@ pub async fn call_chat_completions(
mut text,
tool_calls,
thinking,
..
usage,
} = ret;
if !text.is_empty() {
if extract_code {
@@ -523,14 +580,8 @@ pub async fn call_chat_completions(
ctx.app.config.print_markdown(&text)?;
}
}
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
if let Some(first) = tool_results.first_mut() {
first.thinking = thinking;
}
tool_results
.iter()
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
Ok((text, tool_results))
ctx.record_token_usage(usage, client.model());
finish_completion(ctx, text, tool_calls, thinking).await
}
Err(err) => Err(err),
}
@@ -557,14 +608,18 @@ pub async fn call_chat_completions_streaming(
let aborted_ctrlc = handler.abort().aborted_ctrlc();
let aborted_ctrld = handler.abort().aborted_ctrld();
// Record usage before any early return: the provider bills an interrupted
// call's input tokens (reported in message_start), so abort and error
// paths must still count toward the session totals.
let (text, tool_calls, thinking, usage) = handler.take();
ctx.record_token_usage(usage, client.model());
if aborted_ctrld {
bail!("Aborted.");
}
render_ret?;
let (text, tool_calls, thinking) = handler.take();
if aborted_ctrlc {
if !ctx.working_mode.is_repl() || ctx.session.is_none() {
bail!("Aborted.");
@@ -592,14 +647,7 @@ pub async fn call_chat_completions_streaming(
if !silent && !text.is_empty() && !text.ends_with('\n') {
println!();
}
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
if let Some(first) = tool_results.first_mut() {
first.thinking = thinking;
}
tool_results
.iter()
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
Ok((text, tool_results))
finish_completion(ctx, text, tool_calls, thinking).await
}
Err(err) => {
if !silent && !text.is_empty() {
@@ -610,6 +658,78 @@ pub async fn call_chat_completions_streaming(
}
}
/// Streaming transport for graph `llm` nodes: accumulates the reply and the
/// transport itself renders nothing; no spinner, delta, or reasoning
/// (tool-call rendering inside `eval_tool_calls` is unchanged from the
/// non-streaming path). Because chunks arrive as they are generated, the
/// reqwest `read_timeout` bounds only stalls between chunks; the caller's
/// own deadline bounds the whole generation. Parity with the non-streaming
/// transport is kept deliberately: the handler's call-loop detection is off
/// (that path never had it), and a model the catalog marks `no_stream`
/// takes the non-streaming request, raced against the abort. An abort
/// observed after the provider call fails outright. Partial output is
/// never returned as success.
pub async fn call_chat_completions_streaming_quiet(
input: &Input,
client: &dyn Client,
ctx: &mut RequestContext,
abort_signal: AbortSignal,
) -> Result<(String, Vec<ToolResult>)> {
if client.model().no_stream() {
let output = tokio::select! {
ret = client.chat_completions(input.clone()) => ret?,
_ = wait_abort_signal(&abort_signal) => bail!("Aborted."),
};
let ChatCompletionsOutput {
text,
tool_calls,
thinking,
usage,
} = output;
ctx.record_token_usage(usage, client.model());
if abort_signal.aborted() {
bail!("Aborted.");
}
return finish_completion(ctx, text, tool_calls, thinking).await;
}
// `_rx` outlives the call so `handler.done()` still finds a receiver.
let (tx, _rx) = unbounded_channel();
let mut handler = SseHandler::new(tx, abort_signal.clone());
handler.set_silent(true);
handler.set_call_loop_detection(false);
let send_ret = client.chat_completions_streaming(input, &mut handler).await;
let (text, tool_calls, thinking, usage) = handler.take();
ctx.record_token_usage(usage, client.model());
if abort_signal.aborted() {
bail!("Aborted.");
}
send_ret?;
finish_completion(ctx, text, tool_calls, thinking).await
}
/// Callers record token usage themselves before calling, so that abort and
/// error paths that never reach here still count exactly once per API call.
async fn finish_completion(
ctx: &mut RequestContext,
text: String,
tool_calls: Vec<ToolCall>,
thinking: Vec<ThinkingBlock>,
) -> Result<(String, Vec<ToolResult>)> {
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
if let Some(first) = tool_results.first_mut() {
first.thinking = thinking;
}
tool_results
.iter()
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
Ok((text, tool_results))
}
pub fn noop_prepare_rerank<T>(_client: &T, _data: &RerankData) -> Result<RequestData> {
bail!("The client doesn't support rerank api")
}
@@ -848,11 +968,125 @@ mod tests {
use super::*;
use super::super::access_token::{is_rejected, set_access_token};
use crate::config::{AppState, WorkingMode};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use tokio::time::Instant;
fn catch_error_message(data: &Value, status: u16) -> String {
catch_error(data, status).unwrap_err().to_string()
}
#[test]
fn test_token_usage_is_empty_treats_some_zero_as_non_empty() {
assert!(TokenUsage::default().is_empty());
let usage = TokenUsage {
input_tokens: Some(0),
..Default::default()
};
assert!(!usage.is_empty());
}
#[test]
fn test_token_usage_accumulate_merges_per_field() {
let mut total = TokenUsage::default();
total.accumulate(&TokenUsage {
input_tokens: Some(100),
output_tokens: None,
cache_creation_input_tokens: Some(50),
cache_read_input_tokens: None,
});
total.accumulate(&TokenUsage {
input_tokens: Some(20),
output_tokens: Some(7),
cache_creation_input_tokens: None,
cache_read_input_tokens: Some(90),
});
assert_eq!(total.input_tokens, Some(120));
assert_eq!(total.output_tokens, Some(7));
assert_eq!(total.cache_creation_input_tokens, Some(50));
assert_eq!(total.cache_read_input_tokens, Some(90));
}
#[test]
fn test_token_usage_accumulate_saturates() {
let mut total = TokenUsage {
input_tokens: Some(u64::MAX),
..Default::default()
};
total.accumulate(&TokenUsage {
input_tokens: Some(1),
..Default::default()
});
assert_eq!(total.input_tokens, Some(u64::MAX));
}
fn priced_model(
input_price: Option<f64>,
output_price: Option<f64>,
cache_read_price: Option<f64>,
cache_write_price: Option<f64>,
) -> Model {
let mut data = ModelData::new("test");
data.input_price = input_price;
data.output_price = output_price;
data.cache_read_price = cache_read_price;
data.cache_write_price = cache_write_price;
Model::from_config("provider", &[data]).remove(0)
}
#[test]
fn test_token_usage_cost_usd_sums_all_four_components() {
let usage = TokenUsage {
input_tokens: Some(1_000_000),
output_tokens: Some(500_000),
cache_creation_input_tokens: Some(200_000),
cache_read_input_tokens: Some(2_000_000),
};
let model = priced_model(Some(3.0), Some(15.0), Some(0.3), Some(3.75));
// 3.0 + 7.5 + 0.6 + 0.75
let cost = usage.cost_usd(&model).unwrap();
assert!((cost - 11.85).abs() < 1e-9);
}
#[test]
fn test_token_usage_cost_usd_is_none_when_a_billed_component_lacks_a_price() {
let usage = TokenUsage {
input_tokens: Some(100),
output_tokens: Some(10),
cache_creation_input_tokens: None,
cache_read_input_tokens: Some(50),
};
let model = priced_model(Some(3.0), Some(15.0), None, Some(3.75));
assert_eq!(usage.cost_usd(&model), None);
}
#[test]
fn test_token_usage_cost_usd_ignores_missing_prices_for_unbilled_components() {
let usage = TokenUsage {
input_tokens: Some(1_000_000),
output_tokens: Some(1_000_000),
cache_creation_input_tokens: Some(0),
cache_read_input_tokens: None,
};
let model = priced_model(Some(3.0), Some(15.0), None, None);
let cost = usage.cost_usd(&model).unwrap();
assert!((cost - 18.0).abs() < 1e-9);
}
#[test]
fn test_token_usage_cost_usd_is_none_for_empty_or_all_zero_usage() {
let model = priced_model(Some(3.0), Some(15.0), Some(0.3), Some(3.75));
assert_eq!(TokenUsage::default().cost_usd(&model), None);
let zero = TokenUsage {
input_tokens: Some(0),
output_tokens: Some(0),
cache_creation_input_tokens: Some(0),
cache_read_input_tokens: Some(0),
};
assert_eq!(zero.cost_usd(&model), None);
}
#[test]
fn test_catch_error_display_json_with_type() {
let data = json!({"error": {"type": "invalid_request_error", "message": "Bad request"}});
@@ -1010,4 +1244,284 @@ mod tests {
assert!(!should_retry_auth(&api_status_error(401), client));
assert!(!is_rejected(client, "at-1"));
}
/// Fake provider whose streaming path emits `PACED_DELTAS` text chunks
/// 60s apart, then one thinking block and one tool call, and whose
/// non-streaming path returns that same reply in one piece when
/// `nonstream_replies` is set (and bails otherwise). Exercises the real
/// trait defaults (build_client, prepare_completion_data, abort
/// `select!`) with no network.
struct PacedStreamClient {
config: AppConfig,
model: Model,
stall_forever: bool,
nonstream_replies: bool,
streaming_calls: AtomicUsize,
}
const PACED_DELTAS: usize = 10;
const PACED_GAP: Duration = Duration::from_secs(60);
fn paced_text() -> String {
(0..PACED_DELTAS).map(|i| format!("chunk{i} ")).collect()
}
fn paced_tool_call() -> ToolCall {
ToolCall::new(
"lookup".to_string(),
json!({"q": 1}),
Some("call-1".to_string()),
)
}
fn paced_thinking() -> ThinkingBlock {
ThinkingBlock::Thinking {
thinking: "hmm".to_string(),
signature: "sig".to_string(),
}
}
#[async_trait::async_trait]
impl Client for PacedStreamClient {
fn app_config(&self) -> &AppConfig {
&self.config
}
fn extra_config(&self) -> Option<&ExtraConfig> {
None
}
fn patch_config(&self) -> Option<&RequestPatch> {
None
}
fn name(&self) -> &str {
"paced-stream"
}
fn model(&self) -> &Model {
&self.model
}
async fn chat_completions_inner(
&self,
_client: &ReqwestClient,
data: ChatCompletionsData,
) -> Result<ChatCompletionsOutput> {
assert!(!data.stream, "non-streaming path must not request a stream");
if !self.nonstream_replies {
bail!("the quiet transport must not take the non-streaming path");
}
if self.stall_forever {
std::future::pending::<()>().await;
}
Ok(ChatCompletionsOutput {
text: paced_text(),
tool_calls: vec![paced_tool_call()],
thinking: vec![paced_thinking()],
usage: None,
})
}
async fn chat_completions_streaming_inner(
&self,
_client: &ReqwestClient,
handler: &mut SseHandler,
data: ChatCompletionsData,
) -> Result<()> {
self.streaming_calls.fetch_add(1, Ordering::SeqCst);
assert!(data.stream, "trait default must request a stream");
for i in 0..PACED_DELTAS {
tokio::time::sleep(PACED_GAP).await;
handler.text(&format!("chunk{i} "))?;
}
if self.stall_forever {
std::future::pending::<()>().await;
}
handler.thinking_block(paced_thinking());
handler.tool_call(paced_tool_call())
}
}
fn paced_client(stall_forever: bool) -> PacedStreamClient {
PacedStreamClient {
config: AppConfig::default(),
model: Model::default(),
stall_forever,
nonstream_replies: false,
streaming_calls: AtomicUsize::new(0),
}
}
fn no_stream_model() -> Model {
let mut data = ModelData::new("");
data.no_stream = true;
Model::from_config("", &[data]).remove(0)
}
fn assert_paced_reply(text: &str, results: &[ToolResult]) {
assert_eq!(text, paced_text());
assert_eq!(results.len(), 1);
assert_eq!(results[0].call.name, "lookup");
assert_eq!(results[0].call.id.as_deref(), Some("call-1"));
assert!(
matches!(&results[0].thinking[..], [ThinkingBlock::Thinking { thinking, .. }] if thinking == "hmm")
);
}
fn quiet_ctx() -> RequestContext {
RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd)
}
/// 600s of virtual generation completes and yields the same shape the
/// non-streaming transport would: full text, tool results with the
/// thinking stapled onto the first, and the tracker fed each call. The
/// fake ignores the reqwest client, so reqwest's `read_timeout` is not in
/// the loop here; the real-socket tests in graph/mod.rs pin that it
/// fires on a stall (`sse_read_timeout_stall_is_transient`) and not on a
/// slow but continuous stream (`sse_slow_but_continuous_stream_outlives_read_timeout`).
#[tokio::test(start_paused = true)]
async fn quiet_streaming_accumulates_across_a_long_generation() {
let mut ctx = quiet_ctx();
let input = Input::from_str(&ctx, "hi", None).unwrap();
let client = paced_client(false);
let started = Instant::now();
let (text, results) =
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
.await
.unwrap();
assert!(started.elapsed() >= PACED_GAP * PACED_DELTAS as u32);
assert_paced_reply(&text, &results);
assert!(
results[0].output["tool_call_error"].is_string(),
"an undeclared tool evaluates to a tool_call_error, not a panic: {}",
results[0].output
);
}
/// An abort mid-stream cancels the provider call promptly and fails the
/// call outright; the ten chunks already buffered are never returned.
#[tokio::test(start_paused = true)]
async fn quiet_streaming_abort_discards_partial_output() {
let mut ctx = quiet_ctx();
let input = Input::from_str(&ctx, "hi", None).unwrap();
let client = paced_client(true);
let abort = create_abort_signal();
let trigger = abort.clone();
tokio::spawn(async move {
tokio::time::sleep(PACED_GAP * PACED_DELTAS as u32 + Duration::from_secs(30)).await;
trigger.set_ctrlc();
});
let started = Instant::now();
let err = call_chat_completions_streaming_quiet(&input, &client, &mut ctx, abort)
.await
.expect_err("an aborted stream must not succeed");
assert_eq!(err.to_string(), "Aborted.");
assert!(started.elapsed() < PACED_GAP * (PACED_DELTAS as u32 + 1));
}
/// The quiet transport and the non-streaming transport hand the node the
/// same `(text, tool_results)` for the same provider reply.
#[tokio::test(start_paused = true)]
async fn quiet_streaming_matches_the_non_streaming_reply() {
let mut ctx = quiet_ctx();
ctx.render_mode = RenderMode::Silent;
let input = Input::from_str(&ctx, "hi", None).unwrap();
let mut client = paced_client(false);
client.nonstream_replies = true;
let (streamed_text, streamed) =
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
.await
.unwrap();
let (plain_text, plain) = call_chat_completions(
&input,
false,
false,
&client,
&mut ctx,
create_abort_signal(),
)
.await
.unwrap();
assert_eq!(streamed_text, plain_text);
assert_paced_reply(&streamed_text, &streamed);
assert_paced_reply(&plain_text, &plain);
assert_eq!(streamed[0].output, plain[0].output);
}
/// A model the catalog marks `no_stream` never sees a streaming request;
/// the quiet transport takes the non-streaming path and returns the same
/// shape.
#[tokio::test]
async fn quiet_transport_honours_no_stream_models() {
let mut ctx = quiet_ctx();
let input = Input::from_str(&ctx, "hi", None).unwrap();
let mut client = paced_client(false);
client.model = no_stream_model();
client.nonstream_replies = true;
let (text, results) =
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
.await
.unwrap();
assert_eq!(client.streaming_calls.load(Ordering::SeqCst), 0);
assert_paced_reply(&text, &results);
}
/// The `no_stream` fallback is raced against the abort like the
/// streaming path: a hung non-streaming request is cancelled promptly.
#[tokio::test(start_paused = true)]
async fn quiet_transport_no_stream_fallback_honours_abort() {
let mut ctx = quiet_ctx();
let input = Input::from_str(&ctx, "hi", None).unwrap();
let mut client = paced_client(true);
client.model = no_stream_model();
client.nonstream_replies = true;
let abort = create_abort_signal();
let trigger = abort.clone();
tokio::spawn(async move {
tokio::time::sleep(Duration::from_secs(5)).await;
trigger.set_ctrlc();
});
let started = Instant::now();
let err = call_chat_completions_streaming_quiet(&input, &client, &mut ctx, abort)
.await
.expect_err("an aborted non-streaming fallback must not succeed");
assert_eq!(err.to_string(), "Aborted.");
assert!(started.elapsed() < Duration::from_secs(10));
}
/// The user-level `stream` toggle is a rendering preference; the quiet
/// transport still streams with it off.
#[tokio::test(start_paused = true)]
async fn quiet_transport_ignores_the_user_stream_toggle() {
let app = AppState {
config: Arc::new(AppConfig {
stream: false,
..AppConfig::default()
}),
..AppState::test_default()
};
let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd);
let input = Input::from_str(&ctx, "hi", None).unwrap();
assert!(!input.stream(), "fixture must have streaming disabled");
let client = paced_client(false);
let (text, results) =
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
.await
.unwrap();
assert_eq!(client.streaming_calls.load(Ordering::SeqCst), 1);
assert_paced_reply(&text, &results);
}
}
src/client/model.rs
+37 -1
View File
@@ -195,6 +195,22 @@ impl Model {
self.data.max_output_tokens
}
pub fn input_price(&self) -> Option<f64> {
self.data.input_price
}
pub fn output_price(&self) -> Option<f64> {
self.data.output_price
}
pub fn cache_read_price(&self) -> Option<f64> {
self.data.cache_read_price
}
pub fn cache_write_price(&self) -> Option<f64> {
self.data.cache_write_price
}
pub fn supports_function_calling(&self) -> bool {
self.data.supports_function_calling
}
@@ -327,6 +343,10 @@ pub struct ModelData {
#[serde(skip_serializing_if = "Option::is_none")]
pub output_price: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub cache_read_price: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub cache_write_price: Option<f64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub patch: Option<Value>,
// chat-only properties
@@ -340,7 +360,10 @@ pub struct ModelData {
pub supports_function_calling: bool,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub reasoning_levels: Vec<String>,
#[serde(skip_serializing_if = "Option::is_none")]
#[serde(
alias = "default_reasoning_level",
skip_serializing_if = "Option::is_none"
)]
pub default_reasoning_effort: Option<String>,
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub(crate) no_stream: bool,
@@ -500,4 +523,17 @@ mod tests {
model.data.reasoning_levels.clear();
assert!(!model.description().contains('🧠'));
}
#[test]
fn model_data_accepts_default_reasoning_level_alias() {
let yaml = r#"
name: gpt-5.6-luna
default_reasoning_level: none
reasoning_levels: [none, low, medium, high]
"#;
let data: ModelData = serde_yaml::from_str(yaml).unwrap();
assert_eq!(data.default_reasoning_effort.as_deref(), Some("none"));
}
}
src/client/oauth.rs
+457 -32
View File
@@ -54,9 +54,12 @@ pub struct OAuthConfig {
#[serde(default)]
pub scopes: Vec<String>,
pub token_request_format: Option<TokenRequestFormat>,
pub resource: Option<String>,
#[serde(default)]
pub extra_authorize_params: IndexMap<String, String>,
#[serde(default)]
pub extra_token_params: IndexMap<String, String>,
#[serde(default)]
pub extra_token_headers: IndexMap<String, String>,
#[serde(default)]
pub extra_request_headers: IndexMap<String, String>,
@@ -98,10 +101,17 @@ impl OAuthConfig {
if override_cfg.token_request_format.is_some() {
self.token_request_format = override_cfg.token_request_format;
}
if override_cfg.resource.is_some() {
self.resource = override_cfg.resource;
}
if !override_cfg.extra_authorize_params.is_empty() {
self.extra_authorize_params
.extend(override_cfg.extra_authorize_params);
}
if !override_cfg.extra_token_params.is_empty() {
self.extra_token_params
.extend(override_cfg.extra_token_params);
}
if !override_cfg.extra_token_headers.is_empty() {
self.extra_token_headers
.extend(override_cfg.extra_token_headers);
@@ -135,13 +145,25 @@ pub trait OAuthProvider: Send + Sync {
vec![]
}
/// RFC 8707 resource indicator identifying the protected resource the
/// token is for. Sent as the `resource` parameter on authorization
/// requests (PKCE authorize URL, device-authorization request) and on
/// every token request routed through `build_token_request`.
/// An `extra_authorize_params` or `extra_token_params` entry keyed
/// `resource` takes precedence over this value on its respective
/// surface.
fn resource(&self) -> Option<&str> {
None
}
/// Extra form/body parameters appended to every token request routed
/// through `build_token_request` (authorization-code exchange, refresh,
/// client_credentials, and device-code polling). Used e.g. for the
/// RFC 8707 `resource` indicator required by the MCP spec.
/// NOTE: these are merged AFTER the caller's params and will overwrite
/// a colliding key; do not return protocol parameter names
/// (grant_type, client_id, code, refresh_token, ...).
/// NOTE: these are merged AFTER the caller's params and the first-class
/// `resource()` value, and will overwrite a colliding key; do not
/// return protocol parameter names (grant_type, client_id, code,
/// refresh_token, ...).
fn extra_token_params(&self) -> Vec<(&str, &str)> {
vec![]
}
@@ -219,6 +241,41 @@ pub async fn run_oauth_flow(provider: &dyn OAuthProvider, client_name: &str) ->
}
}
fn build_authorize_url(
provider: &(impl OAuthProvider + ?Sized),
redirect_uri: &str,
code_challenge: &str,
state: &str,
) -> String {
let scopes = provider.scopes();
let mut authorize_url = format!(
"{}?client_id={}&response_type=code&scope={}&redirect_uri={}&code_challenge={}&code_challenge_method=S256&state={}",
provider.authorize_url(),
provider.client_id(),
urlencoding::encode(&scopes),
urlencoding::encode(redirect_uri),
code_challenge,
state
);
let extra_params = provider.extra_authorize_params();
for (key, value) in &extra_params {
authorize_url.push_str(&format!(
"&{}={}",
urlencoding::encode(key),
urlencoding::encode(value)
));
}
if let Some(resource) = provider.resource()
&& !extra_params.iter().any(|(key, _)| *key == "resource")
{
authorize_url.push_str(&format!("&resource={}", urlencoding::encode(resource)));
}
authorize_url
}
async fn run_pkce_flow(provider: &dyn OAuthProvider, client_name: &str) -> Result<()> {
let random_bytes: [u8; 32] = rand::random::<[u8; 32]>();
let code_verifier = URL_SAFE_NO_PAD.encode(random_bytes);
@@ -241,27 +298,7 @@ async fn run_pkce_flow(provider: &dyn OAuthProvider, client_name: &str) -> Resul
(provider.redirect_uri().to_string(), false)
};
let scopes = provider.scopes();
let encoded_scopes = urlencoding::encode(&scopes);
let encoded_redirect = urlencoding::encode(&redirect_uri);
let mut authorize_url = format!(
"{}?client_id={}&response_type=code&scope={}&redirect_uri={}&code_challenge={}&code_challenge_method=S256&state={}",
provider.authorize_url(),
provider.client_id(),
encoded_scopes,
encoded_redirect,
code_challenge,
state
);
for (key, value) in provider.extra_authorize_params() {
authorize_url.push_str(&format!(
"&{}={}",
urlencoding::encode(key),
urlencoding::encode(value)
));
}
let authorize_url = build_authorize_url(provider, &redirect_uri, &code_challenge, &state);
println!(
"\nOpen this URL to authenticate with {} (client '{}'):\n",
@@ -399,6 +436,28 @@ async fn run_client_credentials_flow(
Ok(())
}
fn build_device_authorization_params<'a>(
provider: &'a (impl OAuthProvider + ?Sized),
scopes: &'a str,
pkce_challenge: Option<&'a str>,
) -> Vec<(&'a str, &'a str)> {
let mut device_params: Vec<(&str, &str)> = vec![("client_id", provider.client_id())];
if !scopes.is_empty() {
device_params.push(("scope", scopes));
}
if let Some(challenge) = pkce_challenge {
device_params.push(("code_challenge", challenge));
device_params.push(("code_challenge_method", "S256"));
}
if let Some(resource) = provider.resource() {
device_params.push(("resource", resource));
}
device_params
}
async fn run_device_code_flow(provider: &dyn OAuthProvider, client_name: &str) -> Result<()> {
let device_auth_url = provider.device_authorization_url().ok_or_else(|| {
anyhow!(
@@ -422,14 +481,11 @@ async fn run_device_code_flow(provider: &dyn OAuthProvider, client_name: &str) -
};
let scopes = provider.scopes();
let mut device_params: Vec<(&str, &str)> = vec![("client_id", provider.client_id())];
if !scopes.is_empty() {
device_params.push(("scope", scopes.as_str()));
}
if let Some((_, ref challenge)) = pkce {
device_params.push(("code_challenge", challenge.as_str()));
device_params.push(("code_challenge_method", "S256"));
}
let device_params = build_device_authorization_params(
provider,
&scopes,
pkce.as_ref().map(|(_, challenge)| challenge.as_str()),
);
let form: HashMap<&str, &str> = device_params.iter().copied().collect();
let mut device_request = client
@@ -803,6 +859,7 @@ fn build_token_request(
let all_params: Vec<(&str, &str)> = params
.iter()
.copied()
.chain(provider.resource().map(|r| ("resource", r)))
.chain(provider.extra_token_params())
.collect();
let mut request = match provider.token_request_format() {
@@ -1095,7 +1152,9 @@ mod tests {
device_authorization_url: None,
scopes: vec!["a".into(), "b".into()],
token_request_format: Some(TokenRequestFormat::FormUrlEncoded),
resource: None,
extra_authorize_params: IndexMap::from([("plan".into(), "base".into())]),
extra_token_params: IndexMap::new(),
extra_token_headers: IndexMap::new(),
extra_request_headers: IndexMap::new(),
echo_pkce_in_token_exchange: false,
@@ -1116,7 +1175,9 @@ mod tests {
device_authorization_url: None,
scopes: vec![],
token_request_format: None,
resource: None,
extra_authorize_params: IndexMap::new(),
extra_token_params: IndexMap::new(),
extra_token_headers: IndexMap::new(),
extra_request_headers: IndexMap::new(),
echo_pkce_in_token_exchange: false,
@@ -1429,6 +1490,54 @@ echo_pkce_in_token_exchange: true
assert!(merged.use_pkce_in_device_flow);
}
#[test]
fn oauth_config_merge_user_resource_wins() {
let mut base = base_config();
base.resource = Some("https://base.example/".into());
let mut user = empty_user_override("user-id", "https://user.example/token");
user.resource = Some("https://user.example/".into());
let merged = base.merge(user);
assert_eq!(merged.resource.as_deref(), Some("https://user.example/"));
}
#[test]
fn oauth_config_merge_preserves_resource_when_user_omits() {
let mut base = base_config();
base.resource = Some("https://base.example/".into());
let user = empty_user_override("user-id", "https://user.example/token");
let merged = base.merge(user);
assert_eq!(merged.resource.as_deref(), Some("https://base.example/"));
}
#[test]
fn oauth_config_merge_extends_extra_token_params() {
let mut base = base_config();
base.extra_token_params = IndexMap::from([
("audience".into(), "base".into()),
("kept".into(), "yes".into()),
]);
let mut user = empty_user_override("user-id", "https://user.example/token");
user.extra_token_params = IndexMap::from([("audience".into(), "user".into())]);
let merged = base.merge(user);
assert_eq!(
merged
.extra_token_params
.get("audience")
.map(String::as_str),
Some("user")
);
assert_eq!(
merged.extra_token_params.get("kept").map(String::as_str),
Some("yes")
);
}
#[test]
fn openai_compatible_provider_exposes_device_authorization_url() {
let mut cfg = base_config();
@@ -1478,6 +1587,21 @@ echo_pkce_in_token_exchange: true
assert!(provider.use_pkce_in_device_flow());
}
#[test]
fn openai_compatible_provider_exposes_resource_and_extra_token_params() {
let mut cfg = base_config();
cfg.resource = Some("https://rs.example/".into());
cfg.extra_token_params = IndexMap::from([("audience".into(), "gateway".into())]);
let provider = OpenAICompatibleOAuthProvider {
config: cfg,
client_name: "test".into(),
};
assert_eq!(provider.resource(), Some("https://rs.example/"));
assert_eq!(provider.extra_token_params(), vec![("audience", "gateway")]);
}
#[test]
fn parse_paste_input_full_callback_url() {
let (code, state) =
@@ -1601,6 +1725,30 @@ scopes:
assert_eq!(cfg.scopes, vec!["read", "write"]);
}
#[test]
fn oauth_config_serde_parses_resource_and_extra_token_params() {
let yaml = r#"
flow: device_code
client_id: 1a2b3c4d-1a2b-1234-987a-1a2b3c4d5e6f
device_authorization_url: https://some.site.com/oauth/device_authorization
token_url: https://some.site.com/oauth/token
scopes: [invoke]
resource: https://some.site.com/
extra_token_params:
audience: me
"#;
let cfg: OAuthConfig = serde_yaml::from_str(yaml).unwrap();
assert!(matches!(cfg.flow, OAuthFlow::DeviceCode));
assert_eq!(cfg.resource.as_deref(), Some("https://some.site.com/"));
assert_eq!(
cfg.extra_token_params.get("audience").map(String::as_str),
Some("me")
);
assert_eq!(cfg.scopes, vec!["invoke"]);
}
struct ResourceStubProvider;
impl OAuthProvider for ResourceStubProvider {
@@ -1660,6 +1808,283 @@ scopes:
);
}
#[test]
fn build_token_request_appends_first_class_resource_to_form_body() {
let mut cfg = base_config();
cfg.resource = Some("https://rs.example/".into());
let provider = OpenAICompatibleOAuthProvider {
config: cfg,
client_name: "test".into(),
};
let request = build_token_request(
&ReqwestClient::new(),
&provider,
&[("grant_type", "refresh_token")],
)
.build()
.unwrap();
let body = str::from_utf8(request.body().unwrap().as_bytes().unwrap()).unwrap();
assert!(
body.contains("resource=https%3A%2F%2Frs.example%2F"),
"body missing resource param: {body}"
);
assert!(
body.contains("grant_type=refresh_token"),
"body missing grant_type param: {body}"
);
}
struct CollidingResourceStubProvider;
impl OAuthProvider for CollidingResourceStubProvider {
fn provider_name(&self) -> &str {
"stub"
}
fn client_id(&self) -> &str {
"stub-client"
}
fn authorize_url(&self) -> &str {
"https://as.example/authorize"
}
fn token_url(&self) -> &str {
"https://as.example/token"
}
fn redirect_uri(&self) -> &str {
""
}
fn scopes(&self) -> String {
String::new()
}
fn token_request_format(&self) -> TokenRequestFormat {
TokenRequestFormat::FormUrlEncoded
}
fn resource(&self) -> Option<&str> {
Some("first-class")
}
fn extra_token_params(&self) -> Vec<(&str, &str)> {
vec![("resource", "override")]
}
}
#[test]
fn build_token_request_extra_token_params_win_resource_collision() {
let provider = CollidingResourceStubProvider;
let request = build_token_request(
&ReqwestClient::new(),
&provider,
&[("grant_type", "authorization_code")],
)
.build()
.unwrap();
let body = str::from_utf8(request.body().unwrap().as_bytes().unwrap()).unwrap();
assert!(
body.contains("resource=override"),
"extra_token_params did not win collision: {body}"
);
assert!(
!body.contains("first-class"),
"first-class resource not overridden: {body}"
);
}
struct JsonCollidingResourceStubProvider;
impl OAuthProvider for JsonCollidingResourceStubProvider {
fn provider_name(&self) -> &str {
"stub"
}
fn client_id(&self) -> &str {
"stub-client"
}
fn authorize_url(&self) -> &str {
"https://as.example/authorize"
}
fn token_url(&self) -> &str {
"https://as.example/token"
}
fn redirect_uri(&self) -> &str {
""
}
fn scopes(&self) -> String {
String::new()
}
fn token_request_format(&self) -> TokenRequestFormat {
TokenRequestFormat::Json
}
fn resource(&self) -> Option<&str> {
Some("first-class")
}
fn extra_token_params(&self) -> Vec<(&str, &str)> {
vec![("resource", "override")]
}
}
#[test]
fn build_token_request_extra_token_params_win_resource_collision_json() {
let provider = JsonCollidingResourceStubProvider;
let request = build_token_request(
&ReqwestClient::new(),
&provider,
&[("grant_type", "authorization_code")],
)
.build()
.unwrap();
let body = str::from_utf8(request.body().unwrap().as_bytes().unwrap()).unwrap();
assert!(
body.contains(r#""resource":"override""#),
"extra_token_params did not win collision: {body}"
);
assert!(
!body.contains("first-class"),
"first-class resource not overridden: {body}"
);
}
#[test]
fn build_token_request_omits_resource_when_unset() {
let provider = OpenAICompatibleOAuthProvider {
config: base_config(),
client_name: "test".into(),
};
let request = build_token_request(
&ReqwestClient::new(),
&provider,
&[("grant_type", "refresh_token")],
)
.build()
.unwrap();
let body = str::from_utf8(request.body().unwrap().as_bytes().unwrap()).unwrap();
assert!(
!body.contains("resource="),
"unexpected resource param: {body}"
);
}
#[test]
fn build_authorize_url_includes_resource_when_set() {
let mut cfg = base_config();
cfg.resource = Some("https://rs.example/".into());
let provider = OpenAICompatibleOAuthProvider {
config: cfg,
client_name: "test".into(),
};
let url = build_authorize_url(
&provider,
"http://127.0.0.1:1234/callback",
"challenge",
"state",
);
assert!(
url.contains("&resource=https%3A%2F%2Frs.example%2F"),
"authorize URL missing resource param: {url}"
);
}
#[test]
fn build_authorize_url_omits_resource_when_none() {
let provider = OpenAICompatibleOAuthProvider {
config: base_config(),
client_name: "test".into(),
};
let url = build_authorize_url(
&provider,
"http://127.0.0.1:1234/callback",
"challenge",
"state",
);
assert!(
!url.contains("resource="),
"unexpected resource param: {url}"
);
}
#[test]
fn build_authorize_url_extra_authorize_params_win_resource_collision() {
let mut cfg = base_config();
cfg.resource = Some("first-class".into());
cfg.extra_authorize_params = IndexMap::from([("resource".into(), "override".into())]);
let provider = OpenAICompatibleOAuthProvider {
config: cfg,
client_name: "test".into(),
};
let url = build_authorize_url(
&provider,
"http://127.0.0.1:1234/callback",
"challenge",
"state",
);
assert_eq!(
url.matches("resource=").count(),
1,
"resource param not deduped: {url}"
);
assert!(
url.contains("&resource=override"),
"extra_authorize_params did not win collision: {url}"
);
assert!(
!url.contains("first-class"),
"first-class resource not overridden: {url}"
);
}
#[test]
fn build_device_authorization_params_includes_resource_when_set() {
let mut cfg = base_config();
cfg.resource = Some("https://rs.example/".into());
let provider = OpenAICompatibleOAuthProvider {
config: cfg,
client_name: "test".into(),
};
let params = build_device_authorization_params(&provider, "a b", None);
assert!(
params.contains(&("resource", "https://rs.example/")),
"device params missing resource: {params:?}"
);
}
#[test]
fn build_device_authorization_params_omits_resource_when_none() {
let provider = OpenAICompatibleOAuthProvider {
config: base_config(),
client_name: "test".into(),
};
let params = build_device_authorization_params(&provider, "a b", Some("challenge"));
assert!(
!params.iter().any(|(key, _)| *key == "resource"),
"unexpected resource param: {params:?}"
);
}
#[test]
#[serial]
fn save_oauth_tokens_roundtrips_and_leaves_no_tmp_file() {
src/client/openai.rs
+49 -1
View File
@@ -373,7 +373,10 @@ pub fn openai_build_chat_completions_body(data: ChatCompletionsData, model: &Mod
text,
sequence,
}) => {
if !sequence {
// Empty tool_results (reachable via deserialized sessions) must not emit empty tool_calls.
if tool_results.is_empty() {
vec![]
} else if !sequence {
let tool_calls: Vec<_> = tool_results
.iter()
.map(|tool_result| {
@@ -745,3 +748,48 @@ pub async fn openai_responses_streaming(
sse_stream(builder, handle).await
}
#[cfg(test)]
mod tests {
use super::*;
fn build_body(sequence: bool) -> Value {
let data = ChatCompletionsData {
messages: vec![
Message::new(MessageRole::User, MessageContent::Text("hello".to_string())),
Message::new(
MessageRole::Assistant,
MessageContent::ToolCalls(MessageContentToolCalls {
tool_results: vec![],
text: "leftover text".to_string(),
sequence,
}),
),
],
temperature: None,
top_p: None,
reasoning_effort: None,
functions: None,
stream: false,
};
openai_build_chat_completions_body(data, &Model::new("openai", "gpt-test"))
}
#[test]
fn non_sequence_empty_tool_results_emits_no_messages() {
let body = build_body(false);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
#[test]
fn sequence_empty_tool_results_emits_no_messages() {
let body = build_body(true);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 1, "body: {body}");
}
}
src/client/openai_compatible_oauth.rs
+12
View File
@@ -51,6 +51,18 @@ impl OAuthProvider for OpenAICompatibleOAuthProvider {
.collect()
}
fn resource(&self) -> Option<&str> {
self.config.resource.as_deref()
}
fn extra_token_params(&self) -> Vec<(&str, &str)> {
self.config
.extra_token_params
.iter()
.map(|(k, v)| (k.as_str(), v.as_str()))
.collect()
}
fn token_request_format(&self) -> TokenRequestFormat {
self.config
.token_request_format
src/client/stream.rs
+148 -8
View File
@@ -1,8 +1,8 @@
use super::{ApiStatusError, ThinkingBlock, ToolCall, catch_error};
use super::{ApiStatusError, ThinkingBlock, TokenUsage, ToolCall, catch_error};
use crate::utils::AbortSignal;
use anyhow::{Context, Result, anyhow, bail};
use eventsource_stream::Eventsource;
use eventsource_stream::{EventStreamError, Eventsource};
use futures_util::{Stream, StreamExt};
use reqwest::{RequestBuilder, header};
use serde_json::Value;
@@ -14,10 +14,12 @@ pub struct SseHandler {
buffer: String,
tool_calls: Vec<ToolCall>,
thinking: Vec<ThinkingBlock>,
usage: Option<TokenUsage>,
last_tool_calls: Vec<ToolCall>,
max_call_repeats: usize,
call_repeat_chain_len: usize,
silent: bool,
call_loop_detection: bool,
}
impl SseHandler {
@@ -28,10 +30,12 @@ impl SseHandler {
buffer: String::new(),
tool_calls: Vec::new(),
thinking: Vec::new(),
usage: None,
last_tool_calls: Vec::new(),
max_call_repeats: 2,
call_repeat_chain_len: 3,
silent: false,
call_loop_detection: true,
}
}
@@ -39,6 +43,13 @@ impl SseHandler {
self.silent = silent;
}
/// Loop detection guards an interactive stream; the non-streaming
/// transport never had it, so callers that must match that transport's
/// tool-call handling turn it off.
pub fn set_call_loop_detection(&mut self, enabled: bool) {
self.call_loop_detection = enabled;
}
pub fn text(&mut self, text: &str) -> Result<()> {
if text.is_empty() {
return Ok(());
@@ -73,7 +84,7 @@ impl SseHandler {
}
pub fn tool_call(&mut self, call: ToolCall) -> Result<()> {
if self.is_call_loop(&call) {
if self.call_loop_detection && self.is_call_loop(&call) {
let loop_message = self.create_loop_detection_message(&call);
return Err(anyhow!(loop_message));
}
@@ -176,6 +187,26 @@ impl SseHandler {
self.thinking.push(block);
}
/// Merges a usage update into the accumulator; later non-None fields win.
/// Claude's `message_start` carries input/cache counts while
/// `message_delta` carries the output count, so a single event never has
/// the full picture.
pub fn usage(&mut self, update: TokenUsage) {
let usage = self.usage.get_or_insert_with(TokenUsage::default);
if update.input_tokens.is_some() {
usage.input_tokens = update.input_tokens;
}
if update.output_tokens.is_some() {
usage.output_tokens = update.output_tokens;
}
if update.cache_creation_input_tokens.is_some() {
usage.cache_creation_input_tokens = update.cache_creation_input_tokens;
}
if update.cache_read_input_tokens.is_some() {
usage.cache_read_input_tokens = update.cache_read_input_tokens;
}
}
/// Whether any output (text, tool calls, or thinking blocks) has been
/// accumulated. `Client::chat_completions_streaming` gates its 401 retry
/// on this: content already streamed to the user would be rendered a
@@ -193,14 +224,22 @@ impl SseHandler {
&self.last_tool_calls
}
pub fn take(self) -> (String, Vec<ToolCall>, Vec<ThinkingBlock>) {
pub fn take(
self,
) -> (
String,
Vec<ToolCall>,
Vec<ThinkingBlock>,
Option<TokenUsage>,
) {
let Self {
buffer,
tool_calls,
thinking,
usage,
..
} = self;
(buffer, tool_calls, thinking)
(buffer, tool_calls, thinking, usage)
}
}
@@ -273,6 +312,11 @@ where
break;
}
}
// Keep the typed reqwest error in the chain: a read_timeout stall
// is only recognised as transient via `reqwest::Error::is_timeout`.
Err(EventStreamError::Transport(err)) => {
return Err(err).context("Transport error");
}
Err(err) => {
bail!("{err}");
}
@@ -285,13 +329,12 @@ pub async fn json_stream<S, F, E>(mut stream: S, mut handle: F) -> Result<()>
where
S: Stream<Item = Result<bytes::Bytes, E>> + Unpin,
F: FnMut(&str) -> Result<()>,
E: std::error::Error,
E: std::error::Error + Send + Sync + 'static,
{
let mut parser = JsonStreamParser::default();
let mut unparsed_bytes = vec![];
while let Some(chunk_bytes) = stream.next().await {
let chunk_bytes =
chunk_bytes.map_err(|err| anyhow!("Failed to read json stream, {err}"))?;
let chunk_bytes = chunk_bytes.context("Failed to read json stream")?;
unparsed_bytes.extend(chunk_bytes);
match std::str::from_utf8(&unparsed_bytes) {
Ok(text) => {
@@ -430,6 +473,40 @@ mod tests {
assert!(error_message.contains("test_function_loop"));
}
/// With detection off, repeated identical calls all accumulate — the
/// non-streaming transport's behaviour, which the quiet transport mirrors.
#[test]
fn test_call_loop_detection_can_be_disabled() {
let (mut handler, _rx) = new_handler();
handler.set_call_loop_detection(false);
let call = ToolCall::new("test_function_loop".to_string(), json!({"param": 1}), None);
for _ in 0..5 {
handler.tool_call(call.clone()).unwrap();
}
let (_, calls, _, _) = handler.take();
assert_eq!(calls.len(), 5);
}
/// A transport failure keeps its typed error in the chain so callers can
/// classify it (a stall is only recognised as transient via the type).
#[tokio::test]
async fn json_stream_keeps_the_typed_transport_error() {
let stalled = std::io::Error::new(std::io::ErrorKind::TimedOut, "stall");
let mut source = stream::iter(vec![Err::<Bytes, std::io::Error>(stalled)]);
let err = json_stream(&mut source, |_| Ok(()))
.await
.expect_err("a failed read must fail the stream");
assert!(err.chain().any(|c| {
c.downcast_ref::<std::io::Error>()
.is_some_and(|e| e.kind() == std::io::ErrorKind::TimedOut)
}));
assert!(format!("{err:#}").starts_with("Failed to read json stream"));
}
fn new_handler() -> (SseHandler, tokio::sync::mpsc::UnboundedReceiver<SseEvent>) {
let (sender, receiver) = tokio::sync::mpsc::unbounded_channel();
let abort_signal = crate::utils::create_abort_signal();
@@ -467,6 +544,69 @@ mod tests {
assert!(handler.has_received_content());
}
/// Claude splits usage across events: `message_start` carries the input
/// and cache counts with a provisional output count, `message_delta`
/// carries only the final output count. Later non-None fields must win.
#[test]
fn test_usage_merges_later_non_none_fields() {
let (mut handler, _rx) = new_handler();
handler.usage(TokenUsage {
input_tokens: Some(100),
output_tokens: Some(1),
cache_creation_input_tokens: Some(200),
cache_read_input_tokens: Some(300),
});
handler.usage(TokenUsage {
output_tokens: Some(42),
..Default::default()
});
let (_, _, _, usage) = handler.take();
let usage = usage.unwrap();
assert_eq!(usage.input_tokens, Some(100));
assert_eq!(usage.output_tokens, Some(42));
assert_eq!(usage.cache_creation_input_tokens, Some(200));
assert_eq!(usage.cache_read_input_tokens, Some(300));
}
/// A silent handler is the quiet transport's whole output gate: text,
/// tool calls, and thinking still accumulate for `take()`, but the
/// receiver sees no `Text` events, only the trailing `Done`.
#[test]
fn test_silent_handler_accumulates_without_emitting_text() {
let (mut handler, mut rx) = new_handler();
handler.set_silent(true);
handler.text("hel").unwrap();
handler.text("lo").unwrap();
handler.thinking_block(ThinkingBlock::Thinking {
thinking: "hmm".to_string(),
signature: "sig".to_string(),
});
handler
.tool_call(ToolCall::new(
"lookup".to_string(),
json!({"q": 1}),
Some("call-1".to_string()),
))
.unwrap();
handler.done();
assert!(matches!(rx.try_recv(), Ok(SseEvent::Done)));
assert!(rx.try_recv().is_err(), "no events beyond Done");
let (text, calls, thinking, _) = handler.take();
assert_eq!(text, "hello");
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "lookup");
assert_eq!(calls[0].id.as_deref(), Some("call-1"));
assert_eq!(thinking.len(), 1);
assert!(
matches!(&thinking[0], ThinkingBlock::Thinking { thinking, .. } if thinking == "hmm")
);
}
fn split_chunks(text: &str) -> Vec<Vec<u8>> {
let len = text.len();
let cut1 = random_range(1..len - 1);
Loaded 100 of 145 files, more files were not shown because too many files have changed in this diff. Show more