SystemTime nanos alone collide on coarse clocks, so concurrent tests shared
a scratch dir and read each other's generated sbx mixins (flaky
generated_sbx_mixin_* failures in CI).
verdict_gate.py now names every incomplete review on a machine line the
done template renders immediately after the verdict:
`GAUNTLET_REVIEW_INCOMPLETE: <name>[, <name>]` (sorted). Names are the
lanes retry_gate gave up on (`review_incomplete`) plus the `pipeline`
pseudo-lane whenever `signals_error` carries a `PIPELINE-FAULT:` —
parse death, builder / retry-gate / default-lanes crash guards — and the
gate's own crash guard emits the `pipeline` line too. The report gains a
`## Review incomplete` section (after the signals_error blockquote,
immediately before `## Full lane reports`) and the three PIPELINE-FAULT
lane rows carry ` ×N` when the lane was re-run (N = 1 texts unchanged).
retry_gate.py stands in a synthetic `PIPELINE-FAULT: <lane> lane produced
no report` marker for a lane that ran but collected nothing, so it is
stashed, classified FAULTED and never renders SKIPPED downstream.
MAX_RETRY_ELAPSED_SECS trimmed 45600 → 45000 for margin slack.
graph.yaml: `gauntlet_incomplete_line` seeded in initial_state and
spliced into the done output. README + mermaid document retry_gate, the
dotted _next back-edge, the 3-attempt / wall-clock budget, that retry
never narrows selection, the crash-guard append semantics, and the
machine-line contract (present whenever any lane or the pipeline was
incomplete; it can accompany real findings).
Tests: ×N on all three branches (N>1, N=1 byte-identical, no leak onto
non-fault rows), exact incomplete line / section placement, pipeline
pseudo-lane alone and alongside real lanes (via default_lanes.py's real
crash guard), mixed finding+fault, crash-guard chaining from
build_items / retry_gate / parse_fault / verdict_gate, synthetic
no-report marker, done-template wiring, README pins (positive + stale
negative).
Give the review gauntlet a deterministic retry loop for lanes that FAULT
(PIPELINE-FAULT marker, missing verdict sentinel, or a nested graph's own
degraded-run wording) without ever re-running a lane that reached a real
verdict.
- graph.yaml: the four lane maps route to a new `retry_gate` script node
(static `next: verdict_gate`; emits `_next: build_items` to retry).
initial_state gains lane_attempts / kept_results / retry_lanes /
review_incomplete / gauntlet_started_at. settings.timeout 24000 -> 69600
(3 passes x 2 attempts x 11400 + 1200 margin).
- scripts/retry_gate.py (new): stashes every lane report that ran
(completed and faulted alike) and restores reports a re-run pass
overwrote with []; classifies with verdict_gate.py's sentinel regexes
verbatim; counts attempts; retries while attempts < 3 and inside a
45600 s wall-clock budget; otherwise records the lane in
review_incomplete and falls through. Crash guard appends to
signals_error and never retries.
- scripts/signals.py: stamps gauntlet_started_at before the git work so a
failed diff still starts the clock.
- scripts/build_items.py: retry pass emits items only for retry_lanes
(with attempt numbers), adds a `retried:` note to lanes_summary, resets
retry_lanes; falsy-guarded stamp fallback; crash guard appends to
signals_error and preserves lanes_summary.
- Tests: hermetic python3-guarded script tests for retry_gate /
build_items / signals, graph wiring + initial_state pins, sentinel and
LANE_KEYS cross-pins, retry-budget envelope vs settings.timeout, a
first-pass golden payload, and an engine-level executor test proving a
script `_next` back-edge re-fires a static fan-out and an empty map
re-run clears its collect_into.
Plan: PLAN-review-retry section A items 1-3 and 5. Verdict/report surface
(section A items 4 and 6) is TASK-032.
Child escalations (coder plan approvals, deviation gates, user__* calls
from subagents) previously expired after the escalating agent's
escalation_timeout, defaulting to 300 s, and 0 timed out instantly. In
autonomous runs the orchestrator often could not reply within that
window, so approvals silently fell through to the best-judgment
fallback.
- Extract await_escalation_reply as the wait seam: 0 awaits the oneshot
with no deadline; nonzero keeps tokio::time::timeout with the same
timeout message. A dropped sender still resolves to the existing
cancelled reply either way.
- DEFAULT_ESCALATION_TIMEOUT_SECS and default_escalation_timeout() -> 0,
so graph agents (which inherit the default via from_graph) are covered.
- Document the new semantics in config.agent.example.yaml and drop the
5-minute-timeout claim from the default spawn instructions.
- Tests: paused-clock unlimited wait (3600 s), nonzero expiry message,
dropped-sender completion, default pin, and a grep pin that the stale
timeout wording is gone from src/, assets/, and the example config.
Plan: PLAN-review-retry §C (D3).
verdict.py: a recorded verification run with exit≠0 (including the -1
TIMEOUT sentinel) or a skipped record now forces DIVERGES with a numbered
complaint naming the command — previously a [FAIL] line sat in the report
appendix while the sentinel read CONFORMS, and callers route on the
sentinel alone. Fail-closed by the same rule: an ENVIRONMENT runner-error
marker when commands WERE declared is one red complaint (the declared
commands never ran, so the run is unproven); the "none declared" marker
and an empty declaration stay non-red. Headers name the red-run count.
run_checks.py: decode command output with encoding=utf-8/errors=replace
(a single non-UTF-8 byte used to raise inside communicate() and collapse
the whole evidence record to an ENVIRONMENT marker); bound the post-kill
pipe drain by DRAIN_TIMEOUT_SECS so a session-escaped grandchild holding
the pipe cannot turn a TIMEOUT record into a node kill.
graph.yaml: settings.timeout 5400 → 9000 = 3600 (run_checks node timeout)
+ 5400 (the previous whole-graph budget, kept for every other stage); the
old value predated run_checks and a graph-level timeout kills from
outside, losing the DIVERGES sentinel. Descriptions and README doctrine
updated (CONFORMS = all MET + zero holistic complaints + zero pipeline
faults + zero red runs).
Tests: envelope pin (settings >= run_checks + 5400), gauntlet lane
timeouts > child graph timeouts and gauntlet >= max_attempts × lane +
margin over every agent lane, non-UTF-8 output, bounded drain (unix),
red/skipped/timeout/none-declared/ENVIRONMENT×declaration matrix,
numbering continuity, header punctuation, wording pins on graph.yaml,
run_checks.py and both READMEs. The deadline test's hung command is now
python3-based instead of GNU sleep (cmd.exe on the Windows CI leg).
code-reviewer verdict.py: finding-verifier now completes "successfully"
with fault markers INSIDE its FINDING_VERIFIER_RESULTS payload
(parse_fault's `pipeline-fault` sentinel id, verify_fault / verdict_gate
crash-guard notes prefixed PIPELINE-FAULT:), which the old
"Agent node failed:" prefix check never saw — a dead parse degraded
silently to "unverified" with no fault recorded. Detect them, anchored on
the sentinel id / note prefix only (never a substring over the report),
counted over the raw entry list so duplicate crash-guard ids are not
collapsed. Faults lead the attention list uncapped; the display cap of 5
applies to non-fault entries; the "always-human trigger(s) fired" reason
fires only when non-fault triggers exist.
code-reviewer render.py: CONTRACT comments at the verdict and summary
lines review-gauntlet's gate parses; every LLM-influenced value rendered
after the summary line (dropped_titles, rigor, bar_provenance,
resolved_surfaces, reason) is newline-sanitised so none can forge a
second summary line.
finding-verifier verdict_gate.py: the outer crash-guard note carries the
PIPELINE-FAULT: prefix (the after-retry UNVERIFIABLE path stays
unprefixed — model misbehaviour, not infrastructure); README names all
three emitters.
deep-research: vet_fault's UNVETTED note tells the critic not to request
revision solely because the assessment is unavailable (it matched
critique's PREPRINT/UNVERIFIED REVISE trigger verbatim, burning the
revision budget on an infra fault); vet_fault and critique_fault dedupe
repeated faults and preserve accumulated faults on crash via
safe_faults(); question_fault keeps its plan-pinned prefix and appends the
lane's question identity so distinct dead lanes no longer collapse in
combine_findings' dedupe.
verdict_gate.py: the code-review critical count is read from render.py's
deterministic summary line (full-shape regex, LAST match wins — a 🔴
quoted inside a finding body or Changes-table row is not a finding); the
raw 🔴 count remains the fail-closed fallback when the line is absent. A
code-review lane whose own rendered verdict line records a pipeline
fault (degraded nested run) now BLOCKS instead of passing as GREEN
(attention), and fault + reds append both blockers. A non-string lane
result item is BLOCKED as malformed, never routed to GREEN.
default_lanes.py: the crash handler now emits the WIDEST safe selection
(code-review ∪ adversary ∪ security) and records a PIPELINE-FAULT in
signals_error so the gate blocks — previously it repaired malformed state
into a NARROWER passable selection with no fault. Probe is added only
when consumer_surface AND a probe_context (local-run recipe) are present,
mirroring build_items' guard, so degraded selection cannot spawn an
INCONCLUSIVE-by-construction probe.
graph.yaml: re-derive the timeout envelope. Agent-node retries each get a
fresh per-attempt budget (agent.rs retry_transient/bounded_attempt) and
the child's "Graph … timed out" matches the transient anchor, so lanes
exceed their child graph's settings.timeout (code-review 11400 > 10800,
adversary 9600 > 9000) and the gauntlet's own timeout covers
max_attempts × lane timeout + margin (2 × 11400 + 1200 = 24000). The
child graphs' internal retry envelopes are deliberately not derived here
(tracked as a follow-up). README/descriptions updated and pinned.
Tests: summary-count/quoted-🔴/fallback/last-match/nested-fault/both-
blockers/malformed-lane, crash-widens-and-records-fault, probe guard,
ALIASES parity with build_items, and e2e render.py → verdict_gate runs.
is_transient_error: a typed crate::client::ApiStatusError anywhere in the
chain is now authoritative (positive and negative) — catch_error renders
429s as "… (type: rate_limit_error)" / "… (status: 429)", which none of
the old string anchors matched, so provider rate limits were never
retried, while a 400 whose body mentioned "rate limit" was. Transient
statuses: 429, 500, 502, 503, 504, 529; message text only classifies
untyped errors, and those anchors are now case-insensitive.
validator: warn for an llm/agent node whose output_schema lacks a
non-empty object `properties` map (auto-merge writes nothing since
402d25a) unless the node lifts keys via a non-empty state_updates; the
same absent-or-empty predicate now also gates validate_fallback_capture.
Bundled and graph.example.yaml warning baselines are unchanged.
Tests build the 429 fixture through real catch_error output and a
.context() chain, cover the status table, typed-overrides-text, and all
validator branches.
Adversary re-review follow-ups to 665fee9/402d25a:
- architect/sisyphus spawn templates: the pinned cargo commands are only an
EXAMPLE in a language-agnostic orchestrator — a literal follower in a
non-Rust repo would record 'cargo: command not found' and yield a false
DIVERGES. Each --variables line now carries a substitution cue (pinned).
- run_checks.py: a non-list prior pipeline_faults value is no longer iterated
character by character (test case added).
- Pins: both graphs' verification_commands description wording; the
run_adversary prompt states where the commands come from; the replacement
sentence sits on one physical line.
state_updates::apply merged EVERY top-level key of an llm/agent node's JSON
output into graph state whenever the node had an output_schema, and
structured::extract never validated the output against that schema. Any
prompt text able to steer the model — including plan/diff text pasted from a
repo under review — could therefore plant an undeclared key (e.g.
verification_commands) into state and overwrite a caller-declared value
before a downstream script consumed it.
- apply() now takes the node's output_schema and merges only keys present in
schema.properties; undeclared keys are dropped. No schema, or a schema
without an object properties map, merges nothing.
- Nodes without output_schema are unchanged (None == the old false path);
explicit state_updates are unchanged and still see the full output via
{{output.<key>}}, which remains the author-controlled way to lift a key.
- Callers: llm.rs, agent.rs (x2), rag.rs (None).
- Sweep test bundled_graph_output_schemas_declare_every_key_relied_on: for
every bundled graph, every consumed state key has a declared writer
(initial_state, variable, state_updates target, map key, script-emitted key,
or an output_schema property), and no schema node's field list names a key
its schema omits — so this restriction cannot silently break a shipped
graph. It passes with no graph.yaml changes.
- graph.example.yaml documents the declared-keys-only merge; the residual
disclosed in 665fee9's README/docstring notes is now recorded as closed.
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.
- adversary + review-gauntlet graphs declare verification_commands as a graph
variable (default '[]'); the key leaves initial_state (or_insert seeding
would otherwise shadow it) and leaves both parse nodes' instructions,
output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
arrive as strings); '[]'/[]/missing = none declared; any other shape fails
closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
--variables / --agent-variable form and the trust boundary. Known residual
(llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
prompt-prose injection proof, verdict wiring, asset pins for both graphs and
orchestrator configs, dispatch-inputs > variable-override > default
precedence.
Graph llm nodes (and the structured-output extractor they share with agent
nodes) now obtain the model reply through a quiet streaming transport,
`call_chat_completions_streaming_quiet`, instead of the non-streaming call.
Because chunks arrive as they are generated, reqwest's client-level
`read_timeout` (default 300s) bounds only a genuine stall between chunks and
the node's own `timeout` becomes the effective bound on total generation.
Previously zero bytes arrived until the whole reply finished, so a 300s
read_timeout silently capped every llm node regardless of its `timeout`
(seen live: code-reviewer's `synthesize` node dying on a 61-file review).
The quiet transport renders nothing (silent SseHandler, no spinner), keeps
tool-call handling identical to the non-streaming path (call-loop detection
off), falls back to the non-streaming request for catalog `no_stream` models
(raced against the abort), and never returns partial output as success.
Supporting changes:
- `sse_stream`/`json_stream` keep the typed `reqwest::Error` in the anyhow
chain so `is_transient_error` still classifies read_timeout stalls as
transient and `max_attempts` retries them (previously stringified).
- `structured::extract` takes the graph/session AbortSignal and refuses to
issue an extraction request for an already-aborted graph.
- `run_chat_loop` gains an injectable CompletionRunner seam (owning client
construction + the before-hook) so the timeout/retry/abort behaviour is
unit-testable over a paused clock; a fake `impl Client` drives the real
trait-default streaming for 600s of virtual time; real-socket tests pin
that read_timeout fires on a stall but not on a slow continuous stream.
run_checks.py SIGKILLs the timed-out command's whole process group so a
backgrounded grandchild cannot outlive the record; nothing proved it.
Add a unix-only test that backgrounds `sleep 30`, records its pid, and
probes `kill -0` after the 1s deadline fires.
TOTAL_DEADLINE_SECS (run_checks.py) and the run_checks node's `timeout:`
(graph.yaml) live in different files and could drift until the node
executor kills the script before its in-script degradation runs. Parse
the script's default and assert it stays strictly below the node
timeout, without hardcoding either value.
github-actions[bot]
Dark-Alex-17