A live code-reviewer run degraded through synthesize's fallback; the
graph already retries once (max_attempts: 2) and falls back to verdict
with synth_failure captured, but nothing pinned that wiring. Add the
wiring test so a future edit cannot silently drop the retry or route.
A live run died at map_criteria: one heavy criterion exhausted
max_iterations (10) with no fallback and sank the whole 17-criterion
review. check_criterion now has a 30-iteration budget, a tool-budget
instruction, and falls back to criterion_fault, which records a
schema-shaped UNMET verdict with a PIPELINE-FAULT evidence marker for
that one criterion; verdict.py renders it as DIED rather than judged.
A live code-reviewer run reported "LLM node failed: Failed to call
chat-completions api" with the root cause invisible, because the
state_updates fault text used `{e}` (top context only). Format with
`{e:#}` so fault scripts and warn! logs see the whole anyhow chain.
R3 guards: parse_request (crash → ask_topic), bootstrap_research,
combine_findings (crash → fault-text findings; also lifts branch-local
PIPELINE-FAULT findings into pipeline_faults, deduped), reflexion_gate
(crash fails forward to synthesize), incorporate_feedback (crash still
loops with a generic feedback note), verify_sources (top-level guard +
pipeline_notes fold).
Fallbacks per the plan's naming pins: plan → plan_fault → end_fault
(fail-closed; gains state_updates plan_failure per the TASK-016 rule);
research_one_question → branch-local question_fault (map collects the
fault finding); vet_sources/critique → degrade-visibly markers that
continue to the stage each node's own next pointed to; synthesize →
synth_fault → end_fault. end_fault renders
'DEEP_RESEARCH FAILED — PIPELINE-FAULT: <fault text>'.
Accepted-path visibility: verify_sources folds pipeline_faults into
pipeline_notes ('' when fault-free), appended to end_accepted's output
and the approval question — happy-path output stays byte-identical.
initial_state seeds questions: [] so a dying plan inside the static
fan-out leaves the sibling knowledge_lookup lane mapping over zero
items while end_fault renders.
R5: 19 python3-guarded regression tests + GraphParser wiring test;
bundled warning baseline unchanged.
- prompts.rs DEFAULT_SPAWN_INSTRUCTIONS: example IDs task_1/task_2 -> "1"/"2"
(real numeric-string IDs; old format silently produced dep_error + unblocked
task), plus task-queue rules: root tasks with agent are never auto-dispatched,
task_complete is what dispatches dependents, auto-dispatched agents must be
collected, failed tasks permanently block dependents.
- architect config: optional task-queue mirroring guidance for parallel mode
(disk task files stay the durable store; queue is in-memory).
- sisyphus config: short note on encoding dependency-shaped bursts as
agent__task_create chains; todos remain the tracking source of truth.
The three fallback-capture keys (orient_failure, sweep_failure,
handoff_failure) were write-only dead state: captured on LLM-node failure
but interpolated nowhere, so end_failure rendered an empty STEP_FAILED
shell and write_handoff carried no trace a dead sweep.
- new crash-proof script node note_llm_fault distills orient/handoff
'LLM node...' failure text into fault_note (prefix-anchored; empty when
neither key holds a fault); orient and write_handoff fallbacks now route
through it before end_failure
- end_failure renders 'Pipeline fault (if any): {{fault_note}}' (sentinel
and all existing fields preserved; fault_note defaulted in initial_state)
- write_handoff prompt interpolates {{sweep_failure}} with a
PIPELINE-FAULT VISIBILITY instruction mirroring the review_report
treatment (fault -> Deviations entry + step_summary; success/empty
ignored)
- tests: note_llm_fault fault/clean/crash/truncation paths via the
GRAPH_STATE harness; wiring test pins the retargeted fallbacks,
note_llm_fault routing, {{fault_note}} in end_failure, and the
sweep-fault prompt/instruction anchors; warning baseline unchanged
finding-verifier: parse gains max_attempts 2 + parse_failure capture +
fallback to a new parse_fault marker that rewrites verdicts to a single
UNVERIFIABLE PIPELINE-FAULT entry and continues to done (sentinel
preserved — the caller's verify lane degrades instead of dying);
verify_one gains a branch-local verify_fault marker normalizing the
engine failure text into a schema-conformant UNVERIFIABLE verdict.
step-runner: implement gains fallback end_failure (no retry — reruns
against a mutated tree are unsafe; STEP_FAILED renders the failure via
coder_result); independent_review gains fallback write_handoff with the
handoff instructions and route_review treating an "Agent node failed:"
review_report as review-did-not-run (fail-visible, guard ahead of the
🔴 grep); orient/edge_case_sweep/write_handoff capture their failure
text via state_updates, shrinking the bundled fallback-capture warning
baseline back to zero.
R5 tests: fault-marker normalization + crash paths (raw non-JSON
GRAPH_STATE), route_review.sh fault/critical/clean routing via bash+jq,
and structural wiring tests pinning fallbacks, sentinels, and
end_failure's initial_state-backed template keys.
render.py's empty-diff stub bypass is anchored on the 'PIPELINE-FAULT:'
prefix in reason/attention, but verdict.py's crash guard emitted
'verdict computation error: …' / 'verdict script error: …' without it —
on an empty diff a crashed verdict gate collapsed into the
'No changes to review.' stub and the fail-closed NEEDS-HUMAN verdict
never reached the final report.
- prefix the crash-guard reason and attention entry with PIPELINE-FAULT:
- strengthen the existing crash test to assert the prefix on both fields
- add R5 chained test: empty diff + crash-inducing state piped
verdict.py -> render.py must render the fault, never the stub
- add crash-guard symmetry tests for parse_fault.py / aux_fault.py
(raw non-JSON GRAPH_STATE keeps the fault marker) and a test that a
fault survives the attention[:5] cap alongside 6 user flags
Every llm/agent stage that can die persistently now falls back to a
machine-readable PIPELINE-FAULT sentinel instead of killing the graph:
- review_domain -> branch-local domain_fault marker (banner-prefixed
domain_report the map collects; completeness_gate precedent)
- verify -> verdict (verifier fault renders findings unverified + a
PIPELINE-FAULT attention line)
- parse -> parse_fault -> render (complete NEEDS-HUMAN verdict_out;
render's no-changes stub is bypassed for fault verdicts)
- refine_groups -> cover_gate (deterministic-proposal fallback absorbs
the failure; degradation noted in groups_note)
- aux_lanes -> aux_fault -> synthesize (aux_note fault marker; visible,
never blocking - aux is enrichment)
- synthesize -> verdict (synth_failure marker; keyed off the failure
prefix + non-empty diff, never findings==[], so clean zero-finding
reviews stay MERGE-READY)
verdict.py gains unified fault accounting (R2): any PIPELINE-FAULT in
domain_reports / verifier_output / state fault keys forces NEEDS-HUMAN,
prefix-anchored so a report merely quoting the marker mid-text never
trips it; old 'PIPELINE FAULT' spelling migrated. Faults are prepended
to attention so the [:5] cap can never hide one. Dedicated *_failure
state keys capture engine failure text on every fallback+output_schema
node, keeping the bundled warning baseline at zero.
15 python3-guarded R5 tests: fault paths, quoted-marker precedence,
happy-path byte-compat regression, marker normalization, crash-path
fail-closed (verdict/cover_gate/domain_fault), verifier-fault-on-empty-
diff render, and the no-changes stub regression.
Adversary finding 1: verdict_gate treated any lane report CONTAINING
'PIPELINE-FAULT:' as a lane fault, so a real review quoting the marker
silently bypassed red-finding counting and sentinel parsing. Lane-fault
detection is now precise — only a report that IS the lane_fault.py
marker (starts with it) routes to the fault rule. Adds the missing
regression tests: reds block despite MERGE-READY, NEEDS-HUMAN without
reds passes with attention, probe INCONCLUSIVE blocks with the
environment note, a gate crash fails closed, and a quoting report is
routed to the red rule, not the fault rule.
Adversary finding 2: default_lanes overwrote caller-forced lanes and
dropped the security lane the deterministic signals guaranteed. The
fallback now unions its defaults with caller-forced lanes (canonicalized
via the same aliases build_items uses) and forces security on
auth/deps/exec signals or hardened posture — selection degrades wider,
never narrower, and stays deterministic via a fixed lane order. Tests
cover forced-lane survival, alias canonicalization, unknown-name
tolerance, each security signal, and determinism.
- lane_fault.py + fallback on all four run_* lane nodes: an agent-node
failure is normalized into a PIPELINE-FAULT lane result instead of
killing the graph
- verdict_gate.py: PIPELINE-FAULT lane results block distinctly from
SKIPPED and missing-sentinel; a PIPELINE-FAULT in signals_error blocks
at the gauntlet level
- build_items.py: restructured into main() + fail-closed guard (crash
emits empty lanes + PIPELINE-FAULT) and folds lanes_degraded into
lanes_summary
- parse: state_updates/fallback -> parse_fault.py, which records the
fault in signals_error and jumps to the verdict gate
- select_lanes: state_updates/fallback -> default_lanes.py, a
deterministic wider-never-narrower lane fallback
- verification_commands threaded through parse into run_adversary's
prompt (declared-only, never invented)
- run_probe prompt gains the aborted-attempt reconcile line
- 8 gauntlet_* R5 tests in src/config/request_context.rs (script harness
+ graph interpolation)
Adversary conformance-review fixes for the TASK-017 exec-evidence stage:
- run_checks.py: total-runtime deadline (3300s = node timeout minus margin);
each command gets min(900s, remaining budget), commands past the deadline
are recorded as {cmd, skipped: deadline} entries so in-script handling
stays authoritative instead of the node timeout killing the script from
outside. POSIX hardening: per-command process group (start_new_session)
+ killpg on timeout so hung grandchildren cannot linger; non-POSIX
limitation documented in the docstring.
- graph.yaml: run_checks gains fallback: pipeline_fault; check_criterion
treats SKIPPED records as at-best PARTIAL (unproven, never failed).
- pipeline_fault.py: run_checks-stage attribution grounded in real engine
behavior (script-node fallback writes nothing into state, executor.rs
script arm) — nonempty diff_text proves facts completed, so the fault is
the verification runner; emits 'PIPELINE-FAULT: verification runner
killed'. Empty diff_text still attributes to diff resolution.
- verdict.py: renders skipped records as [SKIPPED] instead of a bogus
[FAIL] exit-? line; non-degraded verdict semantics untouched.
- tests: outer-except ENVIRONMENT degradation, deadline timeout+skip (env
seam ADVERSARY_RUN_CHECKS_DEADLINE_SECS), pipeline_fault stage
attribution (parse / facts / run_checks), check_criterion prompt cites
{{exec_results}}, verdict SKIPPED rendering; run_adversary_script_env
helper for env-carrying invocations.
- cargo fmt violations in 3d7dea9 test code fixed (fmt --check exit 0).
The adversary conformance review flagged that the fallback warn! at
outcome_from (src/graph/agent.rs:71) was never asserted — only the retry
warn had captured-text coverage. Adds one test installing the shared warn
collector, filtered by the unique node id 'fallback_warn_capture' for
parallel safety, asserting the warn names both the node and the fallback
target and carries the error chain.
Agent nodes gain an optional fallback target with llm-node parity: on any
failure (retries exhausted, hard error, or structured-extraction failure)
the failure chain is written into state as 'Agent node failed: <chain>'
and the chain routes to the fallback node. Without a fallback the error
propagates unchanged. Validator collects agent fallback edges for
missing-target checks, reachability, and map-branch locality.
Hoist the llm node's transient-error matcher to graph::is_transient_error,
extend it with broken-pipe / connection-error / error-sending-request
signatures, and give AgentNode an opt-in max_attempts retry loop with
llm-parity semantics: contexts applied before the transient check, fresh
wall-clock budget per attempt, peer identity (assignment + registry)
re-armed before each retry, retirement exactly once after the final
attempt, and structured extraction outside the loop.
- agent__spawn accepts an optional variables object (string->string);
values overlay the parent-inherited agent variables per key, with
inheritance preserved as the fallback layer
- unknown variable names, non-string values, and non-object shapes
return self-correcting status:error JSON listing the agent's
declared variables
- spawned children can no longer trigger the interactive variable
prompt: a new interactive gate in Agent::init_agent_variables routes
missing required variables to the existing bail path while leaving
CLI/REPL prompting and --info skip semantics unchanged
- agent__task_create persists variables on the task, agent__task_list
echoes them, and task auto-dispatch threads them into the spawn
- tool schemas and built-in spawn docs updated; unit tests extended
in agents.rs and taskqueue.rs
A task aborted before its first poll never runs its body, so a guard
constructed inside the future did not exist yet and the pre-provisioned
teammate identity stayed messageable. Built outside and moved in, the
guard drops with the unpolled future on both the frontier and map items.
Also rewords the two timeout: 0 validator warnings: the graph-level one
no longer claims max_loop_iterations always applies (it may be 0 too),
and the node-level one names the enclosing bounds that still end the
node instead of saying it runs until it returns.
Semaphore::new panics above tokio's MAX_PERMITS, so a fixed or templated
map cap (or settings.max_concurrency) that large crashed the run instead
of erroring. The validator rejects both at load, resolve_max_concurrency
rejects every arm at run time, and the executor checks the graph setting
before building its own semaphore, so validation mode makes no difference.
Also corrects run_child_agent_with_graph_inputs' contract comment: inputs
for a config-only agent are rejected, not ignored.
On slow runners bash start-up alone can exceed the fixed 300 ms delay, so
the abort landed before the worker tasks (and their retirement guards)
existed and the observer saw nothing. The dispatcher now leaves a marker
and the trigger waits for it.
A string max_concurrency passed the load-time check whenever it merely
contained a {{...}} pair, so forms like "n={{k}}" or "{{ key }}" that
can never resolve at runtime were only caught after the map's inputs had
been computed. The check now requires the trimmed string to be exactly
one template reference, matching what interpolate_raw resolves as a pure
reference; padded forms like " {{budget}} " stay valid.
Also: the bash tool build-failure context names the source file instead
of the destination path, and the map-subgraph validator no longer
shadows its member id list with the resolved member nodes.
A map node whose item binding shares its name with output_key would have
the binding collected as the chain's result whenever the chain never
wrote it, while the missing-writer warning stayed silent. The collision
is now a validator error attributed to the map node and, with validation
off, a runtime error in run_map before any item forks.
LlmNodeExecutor::execute now takes the graph's abort signal and threads
it through run -> run_with_retries -> run_chat_loop, replacing the
loop-private signal, so an in-flight model call is interrupted and every
turn starts with an abort check that bails 'llm node aborted' before any
client is created.
run_llm_function gains an optional abort signal: the synchronous tool
wait loop kills the child and returns a 'Tool call ... aborted' error
within one poll interval when it is set. The tool dispatcher passes the
session abort; the agent _instructions runner passes None.
A graph run now observes user interrupts wherever it is: GraphExecutor::run
spawns a run-scoped bridge that maps SIGINT or the enclosing session abort
onto the graph's own abort flag (only when a session signal exists; a
pre-set session trips a synchronous check). The frontier join selects on
the abort and, like the timeout path, cancels every in-flight branch task,
failing the run with 'Graph ... aborted during super-step'.
Branch and map item tasks are cancellation-owned via TaskCancelGuard, so
dropping a map task's future no longer detaches its item chains. Peer
retirement is a Drop guard (PeerRetireGuard) constructed before the
semaphore acquire and dropped explicitly right after step(), so the
success-path ordering is unchanged while task abort, panic, and the
pre-step abort return all still retire the identity.
A child agent spawned from a graph node now carries its parent's session
abort signal, so a user interrupt is observable from every sync wait in
the agent tree. Background agent__spawn children are unchanged.
run_child_agent_with_graph_inputs also guards the child's own supervisor
with a disarmable CancelOnDrop: if the future is dropped mid-flight or
the child fails before its normal cleanup, its spawned subagents are
cancelled instead of running on invisibly.
Dark-Alex-17