Fire agent.started before the supervisor spawn and agent.completed/failed
from the spawn task's result arms, resolved against the child context so
global hooks stay gated by the child's own whitelist. Graph AgentNode
failures fire agent.failed at the point of failure, before a declared
fallback converts them into a successful route. The graph executor fires
graph.node.started before each branch spawn and graph.node.completed/failed
beside the narrate calls, with per-branch pre-resolved snapshots and
metadata so parallel super-step branches never cross-contaminate. Base
envs for detached dispatches are rebuilt at fire time via a shared
hook_base_envs helper.
Introduce src/hooks.rs as a types-only module (HookDef, HooksMap) and
plumb hooks:/global_hooks: config fields through every scope:
- Config + AppConfig hooks field, both Default impls, the manual
from_config copy, and a COYOTE_HOOKS env override
- Role frontmatter: hand-parsed hooks arm, private field, getter, and
JSON-inline re-serialization in Role::export (malformed maps degrade
to None)
- AgentConfig + Graph: hooks + global_hooks on both structs, copied in
AgentConfig::from_graph, with a per-agent global_hooks env override
- paths::hooks_dir() = <config_dir>/hooks with COYOTE_HOOKS_DIR override
- Commented hooks: block in the global config template (global scope
only; no global_hooks key)
No engine or dispatch logic yet; existing configs parse unchanged.
Give the review gauntlet a deterministic retry loop for lanes that FAULT
(PIPELINE-FAULT marker, missing verdict sentinel, or a nested graph's own
degraded-run wording) without ever re-running a lane that reached a real
verdict.
- graph.yaml: the four lane maps route to a new `retry_gate` script node
(static `next: verdict_gate`; emits `_next: build_items` to retry).
initial_state gains lane_attempts / kept_results / retry_lanes /
review_incomplete / gauntlet_started_at. settings.timeout 24000 -> 69600
(3 passes x 2 attempts x 11400 + 1200 margin).
- scripts/retry_gate.py (new): stashes every lane report that ran
(completed and faulted alike) and restores reports a re-run pass
overwrote with []; classifies with verdict_gate.py's sentinel regexes
verbatim; counts attempts; retries while attempts < 3 and inside a
45600 s wall-clock budget; otherwise records the lane in
review_incomplete and falls through. Crash guard appends to
signals_error and never retries.
- scripts/signals.py: stamps gauntlet_started_at before the git work so a
failed diff still starts the clock.
- scripts/build_items.py: retry pass emits items only for retry_lanes
(with attempt numbers), adds a `retried:` note to lanes_summary, resets
retry_lanes; falsy-guarded stamp fallback; crash guard appends to
signals_error and preserves lanes_summary.
- Tests: hermetic python3-guarded script tests for retry_gate /
build_items / signals, graph wiring + initial_state pins, sentinel and
LANE_KEYS cross-pins, retry-budget envelope vs settings.timeout, a
first-pass golden payload, and an engine-level executor test proving a
script `_next` back-edge re-fires a static fan-out and an empty map
re-run clears its collect_into.
Plan: PLAN-review-retry section A items 1-3 and 5. Verdict/report surface
(section A items 4 and 6) is TASK-032.
is_transient_error: a typed crate::client::ApiStatusError anywhere in the
chain is now authoritative (positive and negative) — catch_error renders
429s as "… (type: rate_limit_error)" / "… (status: 429)", which none of
the old string anchors matched, so provider rate limits were never
retried, while a 400 whose body mentioned "rate limit" was. Transient
statuses: 429, 500, 502, 503, 504, 529; message text only classifies
untyped errors, and those anchors are now case-insensitive.
validator: warn for an llm/agent node whose output_schema lacks a
non-empty object `properties` map (auto-merge writes nothing since
402d25a) unless the node lifts keys via a non-empty state_updates; the
same absent-or-empty predicate now also gates validate_fallback_capture.
Bundled and graph.example.yaml warning baselines are unchanged.
Tests build the 429 fixture through real catch_error output and a
.context() chain, cover the status table, typed-overrides-text, and all
validator branches.
state_updates::apply merged EVERY top-level key of an llm/agent node's JSON
output into graph state whenever the node had an output_schema, and
structured::extract never validated the output against that schema. Any
prompt text able to steer the model — including plan/diff text pasted from a
repo under review — could therefore plant an undeclared key (e.g.
verification_commands) into state and overwrite a caller-declared value
before a downstream script consumed it.
- apply() now takes the node's output_schema and merges only keys present in
schema.properties; undeclared keys are dropped. No schema, or a schema
without an object properties map, merges nothing.
- Nodes without output_schema are unchanged (None == the old false path);
explicit state_updates are unchanged and still see the full output via
{{output.<key>}}, which remains the author-controlled way to lift a key.
- Callers: llm.rs, agent.rs (x2), rag.rs (None).
- Sweep test bundled_graph_output_schemas_declare_every_key_relied_on: for
every bundled graph, every consumed state key has a declared writer
(initial_state, variable, state_updates target, map key, script-emitted key,
or an output_schema property), and no schema node's field list names a key
its schema omits — so this restriction cannot silently break a shipped
graph. It passes with no graph.yaml changes.
- graph.example.yaml documents the declared-keys-only merge; the residual
disclosed in 665fee9's README/docstring notes is now recorded as closed.
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.
- adversary + review-gauntlet graphs declare verification_commands as a graph
variable (default '[]'); the key leaves initial_state (or_insert seeding
would otherwise shadow it) and leaves both parse nodes' instructions,
output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
arrive as strings); '[]'/[]/missing = none declared; any other shape fails
closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
--variables / --agent-variable form and the trust boundary. Known residual
(llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
prompt-prose injection proof, verdict wiring, asset pins for both graphs and
orchestrator configs, dispatch-inputs > variable-override > default
precedence.
Graph llm nodes (and the structured-output extractor they share with agent
nodes) now obtain the model reply through a quiet streaming transport,
`call_chat_completions_streaming_quiet`, instead of the non-streaming call.
Because chunks arrive as they are generated, reqwest's client-level
`read_timeout` (default 300s) bounds only a genuine stall between chunks and
the node's own `timeout` becomes the effective bound on total generation.
Previously zero bytes arrived until the whole reply finished, so a 300s
read_timeout silently capped every llm node regardless of its `timeout`
(seen live: code-reviewer's `synthesize` node dying on a 61-file review).
The quiet transport renders nothing (silent SseHandler, no spinner), keeps
tool-call handling identical to the non-streaming path (call-loop detection
off), falls back to the non-streaming request for catalog `no_stream` models
(raced against the abort), and never returns partial output as success.
Supporting changes:
- `sse_stream`/`json_stream` keep the typed `reqwest::Error` in the anyhow
chain so `is_transient_error` still classifies read_timeout stalls as
transient and `max_attempts` retries them (previously stringified).
- `structured::extract` takes the graph/session AbortSignal and refuses to
issue an extraction request for an already-aborted graph.
- `run_chat_loop` gains an injectable CompletionRunner seam (owning client
construction + the before-hook) so the timeout/retry/abort behaviour is
unit-testable over a paused clock; a fake `impl Client` drives the real
trait-default streaming for 600s of virtual time; real-socket tests pin
that read_timeout fires on a stall but not on a slow continuous stream.
agent.rs: the retry-after-timeout test ran its second attempt with no
timeout bound, so it passed whether or not the retry owned a fresh
budget. Attempt 2 now runs under the same 1s bounded_attempt and sleeps
900ms under the paused clock; it only succeeds if the deadline is
per-attempt rather than shared with the attempt that already used it.
llm.rs: the `LLM node structured-extraction failed: {e:#}` branch had no
test because `run` cannot succeed without a model. Split the post-run
half of execute into `finish(node, state, ctx, result)` (byte-identical
body, mirrors agent.rs's outcome_from seam) and add tests feeding
Ok("not json") through it, with and without a fallback, asserting the
captured text prefix and the extractor's full context chain.
A live code-reviewer run reported "LLM node failed: Failed to call
chat-completions api" with the root cause invisible, because the
state_updates fault text used `{e}` (top context only). Format with
`{e:#}` so fault scripts and warn! logs see the whole anyhow chain.
The adversary conformance review flagged that the fallback warn! at
outcome_from (src/graph/agent.rs:71) was never asserted — only the retry
warn had captured-text coverage. Adds one test installing the shared warn
collector, filtered by the unique node id 'fallback_warn_capture' for
parallel safety, asserting the warn names both the node and the fallback
target and carries the error chain.
Agent nodes gain an optional fallback target with llm-node parity: on any
failure (retries exhausted, hard error, or structured-extraction failure)
the failure chain is written into state as 'Agent node failed: <chain>'
and the chain routes to the fallback node. Without a fallback the error
propagates unchanged. Validator collects agent fallback edges for
missing-target checks, reachability, and map-branch locality.
Hoist the llm node's transient-error matcher to graph::is_transient_error,
extend it with broken-pipe / connection-error / error-sending-request
signatures, and give AgentNode an opt-in max_attempts retry loop with
llm-parity semantics: contexts applied before the transient check, fresh
wall-clock budget per attempt, peer identity (assignment + registry)
re-armed before each retry, retirement exactly once after the final
attempt, and structured extraction outside the loop.
A task aborted before its first poll never runs its body, so a guard
constructed inside the future did not exist yet and the pre-provisioned
teammate identity stayed messageable. Built outside and moved in, the
guard drops with the unpolled future on both the frontier and map items.
Also rewords the two timeout: 0 validator warnings: the graph-level one
no longer claims max_loop_iterations always applies (it may be 0 too),
and the node-level one names the enclosing bounds that still end the
node instead of saying it runs until it returns.
Semaphore::new panics above tokio's MAX_PERMITS, so a fixed or templated
map cap (or settings.max_concurrency) that large crashed the run instead
of erroring. The validator rejects both at load, resolve_max_concurrency
rejects every arm at run time, and the executor checks the graph setting
before building its own semaphore, so validation mode makes no difference.
Also corrects run_child_agent_with_graph_inputs' contract comment: inputs
for a config-only agent are rejected, not ignored.
On slow runners bash start-up alone can exceed the fixed 300 ms delay, so
the abort landed before the worker tasks (and their retirement guards)
existed and the observer saw nothing. The dispatcher now leaves a marker
and the trigger waits for it.
A string max_concurrency passed the load-time check whenever it merely
contained a {{...}} pair, so forms like "n={{k}}" or "{{ key }}" that
can never resolve at runtime were only caught after the map's inputs had
been computed. The check now requires the trimmed string to be exactly
one template reference, matching what interpolate_raw resolves as a pure
reference; padded forms like " {{budget}} " stay valid.
Also: the bash tool build-failure context names the source file instead
of the destination path, and the map-subgraph validator no longer
shadows its member id list with the resolved member nodes.
A map node whose item binding shares its name with output_key would have
the binding collected as the chain's result whenever the chain never
wrote it, while the missing-writer warning stayed silent. The collision
is now a validator error attributed to the map node and, with validation
off, a runtime error in run_map before any item forks.
LlmNodeExecutor::execute now takes the graph's abort signal and threads
it through run -> run_with_retries -> run_chat_loop, replacing the
loop-private signal, so an in-flight model call is interrupted and every
turn starts with an abort check that bails 'llm node aborted' before any
client is created.
run_llm_function gains an optional abort signal: the synchronous tool
wait loop kills the child and returns a 'Tool call ... aborted' error
within one poll interval when it is set. The tool dispatcher passes the
session abort; the agent _instructions runner passes None.
A graph run now observes user interrupts wherever it is: GraphExecutor::run
spawns a run-scoped bridge that maps SIGINT or the enclosing session abort
onto the graph's own abort flag (only when a session signal exists; a
pre-set session trips a synchronous check). The frontier join selects on
the abort and, like the timeout path, cancels every in-flight branch task,
failing the run with 'Graph ... aborted during super-step'.
Branch and map item tasks are cancellation-owned via TaskCancelGuard, so
dropping a map task's future no longer detaches its item chains. Peer
retirement is a Drop guard (PeerRetireGuard) constructed before the
semaphore acquire and dropped explicitly right after step(), so the
success-path ordering is unchanged while task abort, panic, and the
pre-step abort return all still retire the identity.
GraphState.history was pushed to on every node visit and deep-cloned on
every super-step snapshot and map fork, but its only reader is the
cfg(test) current_node accessor. Gate the field, its initializer, and
the push behind cfg(test) so production builds no longer carry an
unbounded Vec<String> that nothing observes.
validate_timeouts now also flags settings.max_loop_iterations: 0, ahead
of the settings.timeout: 0 warning so both opt-outs read top-to-bottom
as they appear in settings. (design §3.14 / C22)
Both enforcement sites (frontier loop and map chain step) skip the
comparison when the cap is 0; visits are still counted and logged.
Adds an executor test that loops past the default cap with 0, a test
pinning the frontier cap message for a non-zero value, and a map-chain
twin at 0. (design §3.14 / C22)
A timeout: 0 on settings.timeout or on an agent/script/llm/rag node is
the author's explicit opt-out of the wall-clock bound. It stays legal;
the validator now emits one warning per zero so the choice is visible in
--validate output. Non-zero and unset values do not warn (unset already
means no bound for settings.timeout and llm nodes, matching the
explicit-zero-only convention of the orchestration-limit warnings).
Design §3.13 / C21.
A zero on any of the five timeout fields (settings.timeout, agent, script,
llm, rag) now disables the bound instead of firing an immediate
"timed out after 0s" or graph bail. Unset values keep their defaults
(agent 300, script 30, rag 120, llm/settings none); non-zero values are
unchanged. One resolution helper, wall_clock(secs) -> Option<Duration>,
feeds all five call sites; the C17 peer-retirement mark and every error
message stay where they were.
The one existing test that relied on timeout: 0 firing
(run_item_chain_marks_identity_finished_when_agent_step_times_out) moves
to timeout: 1 with its 5s hold unchanged.
Design §3.13 / C21.
Two offline chain tests materialize an echo-inputs graph agent whose
script reports the width it sees: a parent with inputs.width gets the
parent's value back, a parent without inputs gets the child's YAML
default. Two agents.rs tests pin the exact runtime guard texts for a
config-only target and a reserved initial_prompt key.
Task T5 of plans/map-subgraph-branches-design.md §3.12.
Inside validate_agents: an agent node with inputs whose target has a
config.yaml but no graph.yaml is an error, and an inputs key named
initial_prompt is rejected. primary_templated_fields now includes every
inputs value for agent nodes, so validate_parallel_reads treats them as
cross-branch reads with no new pass.
Design: plans/map-subgraph-branches-design.md §3.12 V10a–V10c, task T5.
Agent nodes gain an optional `inputs:` map of templates resolved against
the parent state and overlaid on the child graph agent's initial_state
before its executor starts. A lone {{key}} keeps the state value's JSON
type; mixed text renders to a string; initial_prompt is still owned by the
dispatcher. run_child_agent and run_active_agent_graph keep their
signatures as wrappers over the new *_with_inputs variants, and
run_agent_for_graph rejects inputs aimed at a config-only agent or
containing initial_prompt.
Design: plans/map-subgraph-branches-design.md §3.12 (C18), task T5.
PR #24 follow-up 10. The identity was only retired by the frontier and
chain-exit safety nets, so a peer stayed messageable during
structured::extract after its agent had already stopped. Mark it
finished the moment the agent future resolves, on success and on both
error paths.
Copilot review on PR #24. The validator already rejects 0 and the
template path errors on it, but a literal 0 loaded with
validate_before_run off was silently clamped to 1. Fail with the same
error shape as the template path; only the absent-cap default clamps.
Copilot review on PR #24. The fields were only ever cleared by
run_agent_for_graph taking them, so a failure between arming and that
take (e.g. prompt interpolation) left them set on the item's ctx.
Reset both unconditionally once the step returns.
Dark-Alex-17