Merge pull request #26 from Dark-Alex-17/feat/PLAN-agent-node-resilience
PLAN-agent-node-resilience: graph agent-node max_attempts + fallback
This commit is contained in:
68 files changed
+11192
-445
No files matched your search
@@ -146,6 +146,7 @@ ctor = "1.0.13"
|
||||
pretty_assertions = "1.4.0"
|
||||
rmcp = { version = "3.1.2", features = ["server"] }
|
||||
serial_test = "3"
|
||||
tokio = { version = "1.34.0", features = ["test-util"] }
|
||||
|
||||
[[bin]]
|
||||
name = "coyote"
|
||||
|
||||
@@ -6,8 +6,9 @@
|
||||
> is structurally impossible. Each verdict passes a deterministic contract gate (MET requires
|
||||
> evidence including a proving test; no test = at best PARTIAL), a holistic ADDITIVE-only pass hunts
|
||||
> absence/scope-drift/substitution/gamed-tests, and the CONFORMS/DIVERGES sentinel is computed by a
|
||||
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is unchanged — now
|
||||
> structurally enforced.
|
||||
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is structurally
|
||||
> enforced: CONFORMS requires ALL MET + zero holistic complaints + zero pipeline faults +
|
||||
> zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
|
||||
|
||||
|
||||
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
|
||||
@@ -41,6 +42,9 @@ ADVERSARIAL_REVIEW: CONFORMS
|
||||
Criteria: N/N met (all with tests).
|
||||
```
|
||||
|
||||
CONFORMS is deterministic: ALL MET + zero holistic complaints + zero pipeline faults +
|
||||
zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
|
||||
|
||||
```
|
||||
ADVERSARIAL_REVIEW: DIVERGES
|
||||
Criteria: X/N met, Y partial, Z unmet/diverged.
|
||||
@@ -49,6 +53,12 @@ Complaints:
|
||||
2. ...
|
||||
```
|
||||
|
||||
Every verdict — CONFORMS or DIVERGES — ends with a `Verification runs:` section: the recorded
|
||||
PASS/FAIL results (command, exit code, duration, failure tail) of the caller-declared verification
|
||||
commands, or an explicit marker when none were declared or the run never reached that stage. The
|
||||
one exception is the verdict script's own crash guard, which emits a bare DIVERGES with a
|
||||
"verdict computation error" line — and no section — when verdict computation itself dies.
|
||||
|
||||
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
|
||||
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
|
||||
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
|
||||
@@ -60,28 +70,41 @@ violation) and cites `file:line`. Vague complaints are not emitted.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["parse (llm): extract acceptance criteria"] --> B["diff_facts (script): resolve + bound the diff"]
|
||||
B --> C["map over criteria: one fresh-context branch each (parallel)"]
|
||||
A["parse (llm): extract acceptance criteria"] --> B["facts (diff_facts.py): resolve + bound the diff"]
|
||||
V["verification_commands (declared graph variable — never parsed from the prompt)"] --> R
|
||||
B --> R["run_checks (script): run the declared verification commands ONCE, record results"]
|
||||
R --> C["map over criteria: one fresh-context branch each (parallel)"]
|
||||
C --> D["check_criterion (llm + fs/ast tools)"]
|
||||
D --> E["crit_gate (script): MET needs change+test evidence"]
|
||||
E -. "reject-retry once; exhausted = PARTIAL" .-> D
|
||||
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
|
||||
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras; no criteria = DIVERGES"]
|
||||
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults + zero red runs; no criteria = DIVERGES"]
|
||||
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
|
||||
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
|
||||
B -. "fallback" .-> P
|
||||
R -. "fallback" .-> P
|
||||
D -. "fallback" .-> CF["criterion_fault (script): UNMET + PIPELINE-FAULT evidence for that ONE criterion"]
|
||||
CF -. "map join" .-> F
|
||||
F -. "fallback" .-> G
|
||||
P --> G
|
||||
```
|
||||
|
||||
The conformance doctrine is baked into the graph's own nodes:
|
||||
|
||||
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
|
||||
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
||||
3. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL. Independently of the criterion judgments, the verdict script blocks CONFORMS on any red run (exit≠0/timeout/skipped) and on an ENVIRONMENT runner-error marker when commands were declared — the declared commands never ran, so the run is unproven. The commands arrive through the declared `verification_commands` graph variable (see [Declaring verification commands](#declaring-verification-commands)) — `parse` never extracts them from the prompt, so the prompt is no longer an extraction source for the commands.
|
||||
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
|
||||
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT: criterion ` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. That exact marker prefix (the one `criterion_fault` emits) forces the criterion into unmet regardless of the status it carries, so a verdict that echoes it with `MET` can never count toward CONFORMS; `check_criterion` is told never to begin its own evidence with `PIPELINE-FAULT:`.
|
||||
|
||||
It is **read-only** — it produces a verdict, never a fix.
|
||||
|
||||
## Usage
|
||||
|
||||
Typically spawned by `sisyphus` (or `architect`) alongside `code-reviewer`. The spawn prompt IS its
|
||||
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria:
|
||||
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria.
|
||||
Verification commands travel separately, as a declared variable:
|
||||
|
||||
```sh
|
||||
agent__spawn --agent adversary --prompt "
|
||||
@@ -93,19 +116,41 @@ Run get_diff (or --base main), or: <paste diff>
|
||||
|
||||
## PLAN — acceptance criteria to check against
|
||||
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
|
||||
"
|
||||
" --variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
```
|
||||
|
||||
Direct invocation for ad-hoc use:
|
||||
|
||||
```sh
|
||||
coyote -a adversary --agent-variable project_dir /path/to/repo \
|
||||
--agent-variable verification_commands '["cargo test --all"]' \
|
||||
"Review staged changes against these criteria: <paste criteria>"
|
||||
```
|
||||
|
||||
### Declaring verification commands
|
||||
|
||||
`verification_commands` is a declared graph variable (default `'[]'`): a JSON array of shell
|
||||
commands, passed as a string because agent variables always arrive as strings. `run_checks` runs
|
||||
exactly those commands with a shell — so the declaration is a **trust boundary**. The prompt is the
|
||||
wrong side of it: it also carries plan and diff text pasted from the repo under review, and an LLM
|
||||
`parse` node extracting commands from that prose would let anything in the repo's text run under
|
||||
the reviewer's shell. Hence the commands are never in the prompt and never in `parse`'s output
|
||||
schema; `review-gauntlet` forwards its own `verification_commands` variable to the adversary lane
|
||||
as a structured `inputs:` passthrough, not as prompt text.
|
||||
|
||||
- `'[]'` (or omitted) ⇒ the "none declared" marker; execution criteria are at best PARTIAL.
|
||||
- Anything that is not a JSON array of strings (unparsable JSON, an object, non-string items) is
|
||||
an invalid declaration: `run_checks` executes **nothing** and records a
|
||||
`PIPELINE-FAULT: verification_commands declaration invalid — …` marker, which forces DIVERGES:
|
||||
it fails closed as a PIPELINE-FAULT (complaint #1) — it never degrades to an ENVIRONMENT marker.
|
||||
- The engine merges only the keys declared under `parse`'s `output_schema.properties` into state,
|
||||
so an injected instruction that makes the parse LLM emit an extra `verification_commands` key
|
||||
cannot reach state: the undeclared key is dropped and the declared variable stands.
|
||||
|
||||
### Tools
|
||||
|
||||
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
|
||||
- The `run_checks` script executes the commands declared via the `verification_commands` variable and records the results — criterion branches cite the record and never run anything themselves.
|
||||
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
|
||||
|
||||
## Related
|
||||
|
||||
@@ -11,8 +11,15 @@ description: |
|
||||
silently-overridden recorded decisions, out-of-scope touches, and gamed
|
||||
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
|
||||
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
|
||||
requires every criterion MET and zero holistic complaints. Missing plan =
|
||||
DIVERGES (fail-closed) naming the missing spec.
|
||||
requires every criterion MET, zero holistic complaints, zero pipeline faults,
|
||||
and zero red verification runs (exit≠0/timeout/skipped, or a runner error
|
||||
when commands were declared). Missing plan = DIVERGES (fail-closed) naming
|
||||
the missing spec. Caller-declared verification
|
||||
commands are executed ONCE by a deterministic stage and recorded — criterion
|
||||
branches cite the record and never run anything themselves. Any pipeline
|
||||
fault (parse, diff resolution, verification runner, or holistic failure)
|
||||
lands a PIPELINE-FAULT marker and fails closed into DIVERGES: degraded is
|
||||
never passing.
|
||||
|
||||
version: '1.0'
|
||||
|
||||
@@ -26,11 +33,22 @@ variables:
|
||||
- name: project_dir
|
||||
description: Project directory containing the changes under review
|
||||
default: '.'
|
||||
- name: verification_commands
|
||||
description: |
|
||||
JSON array of shell commands the CALLER declares for the deterministic
|
||||
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
||||
prompt; empty array = none declared.
|
||||
default: '[]'
|
||||
|
||||
settings:
|
||||
max_loop_iterations: 10
|
||||
log_state_snapshots: false
|
||||
timeout: 5400
|
||||
# 3600 (run_checks node timeout, worst case) + 5400 (the previous
|
||||
# whole-graph budget, kept as the envelope for every stage other than
|
||||
# run_checks: parse/facts before it, criteria map/holistic/verdict after)
|
||||
# = 9000. A graph timeout kills from outside — no fallback fires and the
|
||||
# sentinel is lost.
|
||||
timeout: 9000
|
||||
|
||||
initial_state:
|
||||
project_dir_in: ''
|
||||
@@ -42,12 +60,16 @@ initial_state:
|
||||
changed_files: []
|
||||
diff_text: ''
|
||||
diff_note: ''
|
||||
exec_results: ''
|
||||
crit_verdicts: []
|
||||
extra_complaints: []
|
||||
observations: ''
|
||||
adv_report: ''
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
parse_failure: ''
|
||||
holistic_failure: ''
|
||||
pipeline_faults: []
|
||||
|
||||
start: parse
|
||||
|
||||
@@ -106,6 +128,9 @@ nodes:
|
||||
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: pipeline_fault
|
||||
next: facts
|
||||
|
||||
facts:
|
||||
@@ -114,6 +139,16 @@ nodes:
|
||||
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
|
||||
script: scripts/diff_facts.py
|
||||
timeout: 120
|
||||
fallback: pipeline_fault
|
||||
next: run_checks
|
||||
|
||||
run_checks:
|
||||
id: run_checks
|
||||
type: script
|
||||
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker (criterion branches judge it PARTIAL/unproven; the verdict additionally blocks CONFORMS when commands were declared); a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
|
||||
script: scripts/run_checks.py
|
||||
timeout: 3600
|
||||
fallback: pipeline_fault
|
||||
next: map_criteria
|
||||
|
||||
map_criteria:
|
||||
@@ -130,7 +165,7 @@ nodes:
|
||||
check_criterion:
|
||||
id: check_criterion
|
||||
type: llm
|
||||
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree.
|
||||
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree (ENVIRONMENT/SKIPPED runs judge PARTIAL here; the deterministic verdict blocks CONFORMS when declared commands did not run).
|
||||
instructions: |
|
||||
You verify ONE acceptance criterion against a code change. Fresh context,
|
||||
one criterion, full suspicion: assume the implementer drifted, cut a
|
||||
@@ -158,6 +193,25 @@ nodes:
|
||||
values shaped to the criterion's example, TODO/stub presented as done —
|
||||
those are PARTIAL or DIVERGED with the gaming named in the complaint.
|
||||
|
||||
Execution criteria ("tests pass", "lint clean", "build succeeds") are
|
||||
judged ONLY against the recorded verification runs in your prompt — you
|
||||
have no execution tools BY DESIGN and must never claim to have run
|
||||
anything yourself:
|
||||
- a recorded green run (exit 0) of a command covering the criterion is
|
||||
the proving evidence — MET, citing that run (the command + exit 0).
|
||||
- a recorded failing run covering the criterion ⇒ UNMET, citing the
|
||||
recorded exit code and output tail.
|
||||
- "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
|
||||
ENVIRONMENT note in the complaint naming the exact command the caller
|
||||
should declare via the `verification_commands` variable (structured
|
||||
input, e.g. "cargo test --all") — commands are never declared in the
|
||||
prompt.
|
||||
- a SKIPPED record (the runner's total deadline expired before that
|
||||
command ran) ⇒ at best PARTIAL for the criteria it covers, same
|
||||
treatment as an ENVIRONMENT marker: unproven at the criterion level.
|
||||
The deterministic verdict additionally blocks CONFORMS when declared
|
||||
commands did not run — that is its job, not yours.
|
||||
|
||||
Output format — respond with ONLY this JSON object, no prose around it:
|
||||
{"id": "<the criterion's id, echoed>",
|
||||
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
|
||||
@@ -169,6 +223,10 @@ nodes:
|
||||
Your response is machine-validated: a malformed response is rejected back
|
||||
to you exactly once with the reason, then recorded as PARTIAL (unproven).
|
||||
The id is stamped from the criterion itself — your echo is informative only.
|
||||
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
|
||||
whole-file reads, and conclude as soon as the evidence is decisive.
|
||||
Never begin `evidence` with the literal `PIPELINE-FAULT:` — that prefix is
|
||||
reserved for the graph's own fault markers.
|
||||
prompt: |
|
||||
Repository: {{project_dir}}
|
||||
Changed files: {{changed_files}}
|
||||
@@ -181,16 +239,20 @@ nodes:
|
||||
|
||||
The diff{{diff_note}}:
|
||||
{{diff_text}}
|
||||
|
||||
Verification runs (recorded by the deterministic run_checks stage):
|
||||
{{exec_results}}
|
||||
{{gate_feedback}}
|
||||
tools:
|
||||
- fs_read
|
||||
- fs_cat
|
||||
- fs_grep
|
||||
- ast_grep
|
||||
max_iterations: 10
|
||||
max_iterations: 30
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
crit_verdict: '{{output}}'
|
||||
fallback: criterion_fault
|
||||
next: crit_gate
|
||||
|
||||
crit_gate:
|
||||
@@ -200,6 +262,13 @@ nodes:
|
||||
script: scripts/crit_gate.py
|
||||
timeout: 30
|
||||
|
||||
criterion_fault:
|
||||
id: criterion_fault
|
||||
type: script
|
||||
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met.
|
||||
script: scripts/criterion_fault.py
|
||||
timeout: 30
|
||||
|
||||
holistic:
|
||||
id: holistic
|
||||
type: llm
|
||||
@@ -271,14 +340,25 @@ nodes:
|
||||
observations:
|
||||
type: string
|
||||
required: [extra_complaints, observations]
|
||||
max_iterations: 10
|
||||
max_iterations: 20
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
holistic_failure: '{{output}}'
|
||||
fallback: verdict
|
||||
next: verdict
|
||||
|
||||
pipeline_fault:
|
||||
id: pipeline_fault
|
||||
type: script
|
||||
description: Fail-closed fault marker (reached only via fallback routes from parse/facts/run_checks) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
|
||||
script: scripts/pipeline_fault.py
|
||||
timeout: 30
|
||||
next: verdict
|
||||
|
||||
verdict:
|
||||
id: verdict
|
||||
type: script
|
||||
description: Deterministic verdict — CONFORMS requires every criterion MET and zero holistic complaints; no criteria at all fails closed. Assembles the exact sentinel format.
|
||||
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, zero pipeline faults, and zero recorded red verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error marker when commands were declared); no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
|
||||
script: scripts/verdict.py
|
||||
timeout: 30
|
||||
next: done
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed per-criterion fault marker for the adversary graph.
|
||||
|
||||
Reached ONLY via check_criterion's `fallback:` route, after the engine has
|
||||
written its "LLM node …failed: <chain>" text into `crit_verdict` through
|
||||
state_updates. Converts that text into a schema-shaped UNMET verdict whose
|
||||
evidence carries a "PIPELINE-FAULT:" marker, so one criterion check that dies
|
||||
(max_iterations exhausted, API failure) is recorded as DIED rather than
|
||||
judged, the other criteria still get verdicts, and the run is DIVERGES. The
|
||||
map collects it like any other verdict — the branch never sinks the map. The
|
||||
engine applies state_updates before routing to the fallback (src/graph/llm.rs:
|
||||
`apply_state_updates_with_output`, then `outcome_from`), so the "no failure
|
||||
text captured" branch is belt-and-braces rather than an expected path.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def criterion_fields(state):
|
||||
c = state.get("criterion")
|
||||
if isinstance(c, dict):
|
||||
cid = c.get("id")
|
||||
text = c.get("text")
|
||||
return (
|
||||
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
|
||||
text.strip() if isinstance(text, str) else "",
|
||||
)
|
||||
return "unknown", ""
|
||||
|
||||
|
||||
def failure_detail(state):
|
||||
captured = state.get("crit_verdict")
|
||||
detail = captured.strip() if isinstance(captured, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
return "no failure text captured"
|
||||
detail = detail.replace("\n", " ")
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
return detail
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
cid, ctext = criterion_fields(state)
|
||||
detail = failure_detail(state)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"crit_verdict": {
|
||||
"id": cid,
|
||||
"text": ctext,
|
||||
"status": "UNMET",
|
||||
"evidence": f"PIPELINE-FAULT: criterion check failed — {detail}",
|
||||
"complaint": (
|
||||
f"criterion check DIED (pipeline fault) — {detail}; the "
|
||||
"criterion was NOT verified and is treated as unmet "
|
||||
"(fail-closed)"
|
||||
),
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"crit_verdict": {
|
||||
"id": "unknown",
|
||||
"text": "",
|
||||
"status": "UNMET",
|
||||
"evidence": f"PIPELINE-FAULT: criterion fault-marker script error: {e}",
|
||||
"complaint": (
|
||||
f"criterion check DIED (pipeline fault) — fault-marker script "
|
||||
f"error: {e}; the criterion was NOT verified and is treated as "
|
||||
"unmet (fail-closed)"
|
||||
),
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed fault marker for the adversary graph.
|
||||
|
||||
Reached ONLY via `fallback:` routes (parse, facts, run_checks). Appends a
|
||||
machine-readable "PIPELINE-FAULT:" marker to `pipeline_faults` and continues
|
||||
to `verdict`, which turns any recorded fault into DIVERGES — the graph never
|
||||
dies without its sentinel, and a degraded run can never read as CONFORMS.
|
||||
|
||||
Stage attribution — grounded in engine behavior: a SCRIPT node's fallback
|
||||
route writes NOTHING into state (src/graph/executor.rs script arm routes to
|
||||
the fallback without recording the error), so the two script stages are
|
||||
discriminated by which stage outputs are present:
|
||||
- parse (llm): its `state_updates` wrote an "LLM node …failed: <chain>"
|
||||
string into `parse_failure` — attribute to parse.
|
||||
- run_checks vs facts (scripts): diff_facts.py unconditionally writes a
|
||||
NONEMPTY `diff_text` (even its internal-failure path writes
|
||||
"(diff unavailable)"), while the graph's initial value is ''. A nonempty
|
||||
`diff_text` proves facts completed, so the fault is the verification
|
||||
runner (run_checks killed at the node level, e.g. by the node timeout);
|
||||
an empty `diff_text` means facts itself died.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
parse_failure = state.get("parse_failure")
|
||||
detail = parse_failure.strip() if isinstance(parse_failure, str) else ""
|
||||
diff_text = state.get("diff_text")
|
||||
facts_completed = isinstance(diff_text, str) and bool(diff_text.strip())
|
||||
if detail.startswith("LLM node"):
|
||||
detail = detail.replace("\n", " ")
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
faults.append(f"PIPELINE-FAULT: parse failed — cannot review: {detail}")
|
||||
elif facts_completed:
|
||||
faults.append(
|
||||
"PIPELINE-FAULT: verification runner killed — the run_checks stage "
|
||||
"died at the node level before recording exec_results; execution "
|
||||
"criteria are unproven"
|
||||
)
|
||||
else:
|
||||
faults.append(
|
||||
"PIPELINE-FAULT: diff resolution failed — changed files and diff text "
|
||||
"could not be resolved; criterion verification never ran"
|
||||
)
|
||||
print(json.dumps({"pipeline_faults": faults}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pipeline_faults": [
|
||||
f"PIPELINE-FAULT: fault-marker script error: {e} — treat the "
|
||||
"run as degraded"
|
||||
]
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic execution-evidence recorder for the adversary graph.
|
||||
|
||||
Runs ONLY the caller-declared verification commands (verification_commands)
|
||||
in the project dir and records the outcomes as `exec_results` — a list of
|
||||
{cmd, exit, duration_s, tail} records. Criterion branches cite this record;
|
||||
they never execute anything themselves.
|
||||
|
||||
NO auto-detection: undeclared commands mean a "none declared" marker
|
||||
(fail-visible, not fail-guessed). Any runner error degrades into an
|
||||
ENVIRONMENT marker — this script never fails the node.
|
||||
|
||||
Trust boundary: `verification_commands` is a DECLARED graph variable (or the
|
||||
review-gauntlet's structured `inputs:` passthrough), never a field an LLM
|
||||
extracts from the spawn prompt — that prose also carries plan/diff text pasted
|
||||
from the repo under review, and whatever is declared here runs with
|
||||
shell=True. Variables land in state as strings, so a JSON-encoded list is
|
||||
accepted alongside a real list. A declaration that is not a JSON array of
|
||||
strings is a caller contract violation, not a runner hiccup: it executes
|
||||
NOTHING and fails closed as a PIPELINE-FAULT (complaint #1) — it never
|
||||
degrades to an ENVIRONMENT marker. The engine merges only the keys an
|
||||
llm node's `output_schema.properties` declares, so a parse LLM coaxed into
|
||||
emitting an extra `verification_commands` key cannot overwrite the declared
|
||||
value: the undeclared key is dropped before it reaches state.
|
||||
|
||||
Budget: commands run sequentially, so TOTAL runtime is bounded by a deadline
|
||||
(TOTAL_DEADLINE_SECS = 3300s, the run_checks node's 3600s timeout minus
|
||||
margin). Without it, a handful of hanging commands (4 x 900s) would blow the
|
||||
NODE timeout and the executor would kill this script from OUTSIDE — the
|
||||
in-script degradation would never fire. Each command gets
|
||||
min(PER_COMMAND_TIMEOUT_SECS, remaining budget); commands the deadline
|
||||
leaves no budget for are recorded as {"cmd", "skipped": "deadline"} entries,
|
||||
so the in-script handling stays authoritative. The env var
|
||||
ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam).
|
||||
|
||||
Process hygiene: commands run through the shell. On POSIX each command gets
|
||||
its own session (start_new_session) and a timeout SIGKILLs the whole process
|
||||
group, so grandchildren of a hung command cannot linger. The post-kill
|
||||
reap/drain is itself bounded by DRAIN_TIMEOUT_SECS: a grandchild that escaped
|
||||
the session (its own start_new_session) but still holds the stdout pipe
|
||||
would otherwise block the drain until it exits. On non-POSIX platforms only
|
||||
the direct shell child is killed — grandchildren of a timed-out command may
|
||||
survive (documented limitation).
|
||||
|
||||
Verification commands run with GRAPH_STATE* (GRAPH_STATE, GRAPH_STATE_FILE)
|
||||
scrubbed from their environment: the reviewed repo's own tests may exec these very
|
||||
scripts with their own GRAPH_STATE, and load_state() prefers
|
||||
GRAPH_STATE_FILE, so an inherited live state file would silently win.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
PER_COMMAND_TIMEOUT_SECS = 900
|
||||
TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
|
||||
DRAIN_TIMEOUT_SECS = 10
|
||||
TAIL_LINES = 50
|
||||
MAX_TAIL_CHARS = 8000
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
class InvalidDeclaration(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def declared_commands(declared):
|
||||
"""Normalize the verification_commands declaration to a list of commands.
|
||||
|
||||
Accepts a JSON list or a JSON-encoded string of one; missing/None means
|
||||
none declared. Anything else raises InvalidDeclaration.
|
||||
"""
|
||||
if declared is None:
|
||||
return []
|
||||
if isinstance(declared, str):
|
||||
if not declared.strip():
|
||||
raise InvalidDeclaration("empty string (declare '[]' for none)")
|
||||
try:
|
||||
declared = json.loads(declared)
|
||||
except json.JSONDecodeError as e:
|
||||
raise InvalidDeclaration(f"not valid JSON: {e}") from e
|
||||
if not isinstance(declared, list):
|
||||
raise InvalidDeclaration(
|
||||
f"expected a JSON array of strings, got {type(declared).__name__}"
|
||||
)
|
||||
bad = [c for c in declared if not isinstance(c, str)]
|
||||
if bad:
|
||||
raise InvalidDeclaration(f"non-string item(s): {json.dumps(bad)}")
|
||||
return [c.strip() for c in declared if c.strip()]
|
||||
|
||||
|
||||
def tail_of(text):
|
||||
tail = "\n".join(text.splitlines()[-TAIL_LINES:])
|
||||
if len(tail) > MAX_TAIL_CHARS:
|
||||
tail = "…" + tail[-MAX_TAIL_CHARS:]
|
||||
return tail
|
||||
|
||||
|
||||
def run_one(cmd, proj, budget):
|
||||
"""Run one command, hard-bounded by `budget` seconds; return its record."""
|
||||
start = time.monotonic()
|
||||
popen_kwargs = {"start_new_session": True} if os.name == "posix" else {}
|
||||
env = {k: v for k, v in os.environ.items() if not k.startswith("GRAPH_STATE")}
|
||||
p = subprocess.Popen(
|
||||
cmd,
|
||||
shell=True,
|
||||
cwd=proj,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
env=env,
|
||||
**popen_kwargs,
|
||||
)
|
||||
try:
|
||||
out_s, err_s = p.communicate(timeout=budget)
|
||||
except subprocess.TimeoutExpired:
|
||||
if os.name == "posix":
|
||||
try:
|
||||
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass
|
||||
else:
|
||||
p.kill()
|
||||
try:
|
||||
p.communicate(timeout=DRAIN_TIMEOUT_SECS)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
return {
|
||||
"cmd": cmd,
|
||||
"exit": -1,
|
||||
"duration_s": round(time.monotonic() - start, 1),
|
||||
"tail": f"TIMEOUT: no exit within {budget:.0f}s — recorded as a failing run",
|
||||
}
|
||||
combined = (out_s or "") + (("\n" + err_s) if err_s else "")
|
||||
return {
|
||||
"cmd": cmd,
|
||||
"exit": p.returncode,
|
||||
"duration_s": round(time.monotonic() - start, 1),
|
||||
"tail": tail_of(combined),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
try:
|
||||
cmds = declared_commands(state.get("verification_commands"))
|
||||
except InvalidDeclaration as e:
|
||||
msg = f"PIPELINE-FAULT: verification_commands declaration invalid — {e}"
|
||||
prior = state.get("pipeline_faults")
|
||||
faults = [f for f in prior if isinstance(f, str)] if isinstance(prior, list) else []
|
||||
faults.append(msg)
|
||||
print(json.dumps({"exec_results": msg, "pipeline_faults": faults}))
|
||||
return
|
||||
if not cmds:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"exec_results": (
|
||||
"none declared — the caller supplied no verification_commands; "
|
||||
"execution criteria (build/test/lint) cannot be independently "
|
||||
"confirmed. Declare the exact commands via the "
|
||||
"verification_commands variable to get recorded runs."
|
||||
)
|
||||
}
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
proj = os.path.expanduser(
|
||||
(state.get("project_dir") or "").strip() or "."
|
||||
)
|
||||
deadline = time.monotonic() + TOTAL_DEADLINE_SECS
|
||||
results = []
|
||||
for cmd in cmds:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
results.append({"cmd": cmd, "skipped": "deadline"})
|
||||
continue
|
||||
results.append(run_one(cmd, proj, min(PER_COMMAND_TIMEOUT_SECS, remaining)))
|
||||
print(json.dumps({"exec_results": results}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — never fail the node; degrade visibly
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"exec_results": (
|
||||
f"ENVIRONMENT: verification runner error: {e} — the declared "
|
||||
"commands could not be executed; execution criteria are unproven "
|
||||
"(PARTIAL at the criterion level; the verdict blocks CONFORMS when "
|
||||
"commands were declared)"
|
||||
)
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -1,15 +1,35 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic conformance verdict for the adversary graph.
|
||||
|
||||
CONFORMS requires: at least one criterion, every criterion MET, and zero
|
||||
holistic complaints. Everything else — including a missing plan — is
|
||||
DIVERGES (fail-closed). Assembles the exact sentinel format the callers
|
||||
route on. Never crashes into a silent verdict.
|
||||
CONFORMS requires: at least one criterion, every criterion MET, zero
|
||||
holistic complaints, zero pipeline faults, and zero recorded red
|
||||
verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error
|
||||
marker when commands were declared — the declared commands never ran, so the
|
||||
run is unproven). Everything else — including a missing plan or a degraded
|
||||
(faulted) pipeline — is DIVERGES (fail-closed).
|
||||
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
|
||||
llm-node failure captured in `holistic_failure`) becomes complaint #1;
|
||||
per-criterion results are still reported. A criterion whose check DIED
|
||||
(criterion_fault's UNMET verdict, evidence prefixed DIED_EVIDENCE_PREFIX) is
|
||||
rendered as died rather than judged and counted as unmet regardless of the
|
||||
status it carries. Assembles the exact sentinel format the callers route on.
|
||||
Never crashes into a silent verdict.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_FAULT_DETAIL_CHARS = 300
|
||||
MAX_MARKER_CHARS = 200
|
||||
DIED_MARKER = "criterion check DIED (pipeline fault)"
|
||||
DIED_EVIDENCE_PREFIX = "PIPELINE-FAULT: criterion "
|
||||
|
||||
|
||||
def died(v):
|
||||
evidence = v.get("evidence")
|
||||
return isinstance(evidence, str) and evidence.startswith(DIED_EVIDENCE_PREFIX)
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
@@ -17,19 +37,194 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def collect_pipeline_faults(state):
|
||||
"""Recorded fault markers, plus a holistic llm-node failure if one occurred.
|
||||
|
||||
`holistic_failure` holds the "LLM node …failed: <chain>" string when the
|
||||
holistic pass fell back to verdict; on a successful pass it holds the
|
||||
rendered schema output (never that prefix).
|
||||
"""
|
||||
faults = [
|
||||
f.strip()
|
||||
for f in (state.get("pipeline_faults") or [])
|
||||
if isinstance(f, str) and f.strip()
|
||||
]
|
||||
hf = state.get("holistic_failure")
|
||||
if isinstance(hf, str) and hf.strip().startswith("LLM node"):
|
||||
detail = hf.strip().replace("\n", " ")
|
||||
if len(detail) > MAX_FAULT_DETAIL_CHARS:
|
||||
detail = detail[: MAX_FAULT_DETAIL_CHARS - 1] + "…"
|
||||
faults.append(
|
||||
"PIPELINE-FAULT: holistic pass failed — criterion verdicts stand "
|
||||
f"but cross-cutting hunt did not run ({detail})"
|
||||
)
|
||||
return faults
|
||||
|
||||
|
||||
def append_criterion_complaints(lines, verdicts, extra, start):
|
||||
"""Numbered complaints for non-MET verdicts and holistic extras."""
|
||||
i = start
|
||||
for v in verdicts:
|
||||
if v.get("status") == "MET" and not died(v):
|
||||
continue
|
||||
i += 1
|
||||
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||
if len(text) > 140:
|
||||
text = text[:137] + "…"
|
||||
complaint = (v.get("complaint") or "").strip()
|
||||
if died(v):
|
||||
if not complaint.startswith(DIED_MARKER):
|
||||
complaint = f"{DIED_MARKER} — {complaint}" if complaint else DIED_MARKER
|
||||
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
|
||||
else:
|
||||
lines.append(
|
||||
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
|
||||
f"{complaint}"
|
||||
)
|
||||
for c in extra:
|
||||
i += 1
|
||||
lines.append(
|
||||
f"{i}. {c.get('kind', 'violation')} — {c.get('location', '?')} — "
|
||||
f"{c.get('violation', '')} — {c.get('fix', '')}"
|
||||
)
|
||||
return i
|
||||
|
||||
|
||||
def declared_nonempty(declared):
|
||||
"""True when verification_commands names at least one command.
|
||||
|
||||
Accepts a list or a JSON-encoded string of one. None, unparseable, or
|
||||
empty declarations count as not declared — run_checks already records
|
||||
a PIPELINE-FAULT for the invalid ones.
|
||||
"""
|
||||
if isinstance(declared, str):
|
||||
try:
|
||||
declared = json.loads(declared)
|
||||
except json.JSONDecodeError:
|
||||
return False
|
||||
if not isinstance(declared, list):
|
||||
return False
|
||||
return any(isinstance(c, str) and c.strip() for c in declared)
|
||||
|
||||
|
||||
def exec_run_complaints(exec_results, declared):
|
||||
"""One complaint per red verification run recorded by run_checks.py.
|
||||
|
||||
A red run is a record that was skipped (deadline) or exited nonzero
|
||||
(including the -1 TIMEOUT sentinel). An "ENVIRONMENT: …" runner-error
|
||||
marker with a non-empty declaration is one complaint: the declared
|
||||
commands never ran. Other string markers ("none declared",
|
||||
"PIPELINE-FAULT…") are not runs and yield nothing.
|
||||
"""
|
||||
if isinstance(exec_results, str):
|
||||
marker = exec_results.strip().replace("\n", " ")
|
||||
if not marker.startswith("ENVIRONMENT:") or not declared_nonempty(declared):
|
||||
return []
|
||||
if len(marker) > MAX_MARKER_CHARS:
|
||||
marker = marker[: MAX_MARKER_CHARS - 1] + "…"
|
||||
return [
|
||||
f"Verification runner error — the declared command(s) never ran ({marker}); "
|
||||
"unproven runs block CONFORMS"
|
||||
]
|
||||
if not isinstance(exec_results, list):
|
||||
return []
|
||||
out = []
|
||||
for r in exec_results:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
cmd = r.get("cmd", "?")
|
||||
if r.get("skipped"):
|
||||
out.append(f"Verification run `{cmd}` — never ran: deadline expired (unproven)")
|
||||
continue
|
||||
code = r.get("exit")
|
||||
if code == 0:
|
||||
continue
|
||||
outcome = "TIMEOUT (exit -1)" if code == -1 else f"exit {code}"
|
||||
out.append(
|
||||
f"Verification run `{cmd}` — {outcome} (recorded FAIL; a red declared "
|
||||
"command blocks CONFORMS regardless of criterion judgments)"
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def render_exec_results(exec_results):
|
||||
"""Lines for the "Verification runs:" report section.
|
||||
|
||||
exec_results is a list of {cmd, exit, duration_s, tail} records (or
|
||||
{cmd, skipped} for commands the runner's total deadline left no budget
|
||||
for) from run_checks.py, or a string marker ("none declared —…" /
|
||||
"ENVIRONMENT: …"), or empty when the run never reached the stage.
|
||||
"""
|
||||
if isinstance(exec_results, list):
|
||||
out = []
|
||||
for r in exec_results:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
if r.get("skipped"):
|
||||
out.append(
|
||||
f"- [SKIPPED] `{r.get('cmd', '?')}` — never ran: the runner's "
|
||||
"total deadline expired first; covered criteria are unproven"
|
||||
)
|
||||
continue
|
||||
status = "PASS" if r.get("exit") == 0 else "FAIL"
|
||||
out.append(
|
||||
f"- [{status}] `{r.get('cmd', '?')}` — exit {r.get('exit', '?')}, "
|
||||
f"{r.get('duration_s', '?')}s"
|
||||
)
|
||||
if status == "FAIL":
|
||||
tail = (r.get("tail") or "").strip()
|
||||
out.extend(" " + t for t in tail.splitlines()[-15:])
|
||||
return out or ["- (no verification commands were run)"]
|
||||
if isinstance(exec_results, str) and exec_results.strip():
|
||||
return [f"- {exec_results.strip()}"]
|
||||
return ["- none recorded (the run did not reach the verification stage)"]
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
verdicts = [v for v in (state.get("crit_verdicts") or []) if isinstance(v, dict)]
|
||||
extra = [c for c in (state.get("extra_complaints") or []) if isinstance(c, dict)]
|
||||
observations = (state.get("observations") or "").strip()
|
||||
faults = collect_pipeline_faults(state)
|
||||
reds = exec_run_complaints(
|
||||
state.get("exec_results"), state.get("verification_commands")
|
||||
)
|
||||
n = len(verdicts)
|
||||
|
||||
met = [v for v in verdicts if v.get("status") == "MET"]
|
||||
partial = [v for v in verdicts if v.get("status") == "PARTIAL"]
|
||||
bad = [v for v in verdicts if v.get("status") in ("UNMET", "DIVERGED")]
|
||||
met = [v for v in verdicts if v.get("status") == "MET" and not died(v)]
|
||||
partial = [v for v in verdicts if v.get("status") == "PARTIAL" and not died(v)]
|
||||
bad = [
|
||||
v for v in verdicts if died(v) or v.get("status") in ("UNMET", "DIVERGED")
|
||||
]
|
||||
|
||||
lines = []
|
||||
if n == 0:
|
||||
if faults:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
if n:
|
||||
lines.append(
|
||||
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, "
|
||||
f"{len(bad)} unmet/diverged — degraded run: pipeline fault recorded "
|
||||
"(fail-closed)."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
"Criteria: none verified — degraded run: pipeline fault recorded "
|
||||
"(fail-closed)."
|
||||
)
|
||||
lines.append("Complaints:")
|
||||
i = 0
|
||||
for f in faults:
|
||||
i += 1
|
||||
lines.append(f"{i}. {f}")
|
||||
i = append_criterion_complaints(lines, verdicts, extra, i)
|
||||
for c in reds:
|
||||
i += 1
|
||||
lines.append(f"{i}. {c}")
|
||||
if observations:
|
||||
lines.append("")
|
||||
lines.append("Non-blocking observations:")
|
||||
lines.append(observations)
|
||||
elif n == 0:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
lines.append("Criteria: none provided.")
|
||||
lines.append("Complaints:")
|
||||
@@ -38,7 +233,9 @@ def main():
|
||||
"be judged without a spec (fail-closed). Supply the plan's Objective/Tasks/"
|
||||
"Acceptance criteria and re-run."
|
||||
)
|
||||
elif not partial and not bad and not extra:
|
||||
for i, c in enumerate(reds, start=2):
|
||||
lines.append(f"{i}. {c}")
|
||||
elif not partial and not bad and not extra and not reds:
|
||||
lines.append("ADVERSARIAL_REVIEW: CONFORMS")
|
||||
lines.append(f"Criteria: {n}/{n} met (all with tests).")
|
||||
if observations:
|
||||
@@ -47,28 +244,18 @@ def main():
|
||||
lines.append(observations)
|
||||
else:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
lines.append(
|
||||
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged."
|
||||
)
|
||||
header = f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged"
|
||||
notes = []
|
||||
if died_count := sum(1 for v in verdicts if died(v)):
|
||||
notes.append(f"degraded run: {died_count} criterion check(s) died (fail-closed)")
|
||||
if reds:
|
||||
notes.append(f"{len(reds)} red verification run(s) (fail-closed)")
|
||||
lines.append(header + "".join(f" — {note}" for note in notes) + ".")
|
||||
lines.append("Complaints:")
|
||||
i = 0
|
||||
for v in verdicts:
|
||||
if v.get("status") == "MET":
|
||||
continue
|
||||
i = append_criterion_complaints(lines, verdicts, extra, 0)
|
||||
for c in reds:
|
||||
i += 1
|
||||
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||
if len(text) > 140:
|
||||
text = text[:137] + "…"
|
||||
lines.append(
|
||||
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
|
||||
f'{(v.get("complaint") or "").strip()}'
|
||||
)
|
||||
for c in extra:
|
||||
i += 1
|
||||
lines.append(
|
||||
f"{i}. {c.get('kind', 'violation')} — {c.get('location', '?')} — "
|
||||
f"{c.get('violation', '')} — {c.get('fix', '')}"
|
||||
)
|
||||
lines.append(f"{i}. {c}")
|
||||
if observations:
|
||||
lines.append("")
|
||||
lines.append("Non-blocking observations:")
|
||||
@@ -84,6 +271,10 @@ def main():
|
||||
text = text[:97] + "…"
|
||||
lines.append(f'- "{text}" — {(v.get("evidence") or "").strip()}')
|
||||
|
||||
lines.append("")
|
||||
lines.append("Verification runs:")
|
||||
lines.extend(render_exec_results(state.get("exec_results")))
|
||||
|
||||
print(json.dumps({"adv_report": "\n".join(lines)}))
|
||||
|
||||
|
||||
|
||||
@@ -160,6 +160,9 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
||||
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
|
||||
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
|
||||
rebase and re-verify.
|
||||
- Optionally, the `agent__task_*` queue may mirror the on-disk `blocked_by` graph so completing a
|
||||
task auto-dispatches its unblocked dependents. The queue is in-memory convenience only — disk task
|
||||
files remain the durable source of truth.
|
||||
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
|
||||
CI-check watch) is unchanged in both modes.
|
||||
|
||||
|
||||
@@ -333,8 +333,14 @@ instructions: |
|
||||
Forced lanes: adversary<, probe — add probe when the task changed consumer-facing surface and the plan carries a local-run recipe>
|
||||
Plan / acceptance criteria: <the task's acceptance criteria, pasted verbatim>
|
||||
Local-run recipe / usage suites: <from the plan, when present>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||
```
|
||||
|
||||
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the
|
||||
prompt also carries pasted plan text from the repo under review, and whatever is declared
|
||||
here is executed with a shell — a trust boundary the prose must not cross.
|
||||
|
||||
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED apply the same fix-and-rerun rules as below
|
||||
(resume the SAME Sisyphus session with the findings verbatim, then a FRESH gauntlet run).
|
||||
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed.
|
||||
@@ -351,8 +357,13 @@ instructions: |
|
||||
<paste the task index.md body + the relevant PLAN-<slug>.md section VERBATIM>
|
||||
<paste the plan's ## Quality bar section VERBATIM — recorded dropped practices are
|
||||
conformance facts, not divergences>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||
```
|
||||
|
||||
Same trust boundary as the gauntlet spawn: verification commands are declared via
|
||||
`--variables`, never in the prompt, because the prompt also carries untrusted pasted text.
|
||||
|
||||
- **`ADVERSARIAL_REVIEW: DIVERGES`** → treat it as a blocker: resume the SAME Sisyphus session
|
||||
(`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <adversary
|
||||
complaints, verbatim>"`) — do not spawn a fresh one. Re-run `adversary` ONCE after the fix to
|
||||
@@ -456,6 +467,20 @@ instructions: |
|
||||
the task (step 5), and clean up — `git worktree remove .worktrees/task-NNN` and delete the task
|
||||
branch (local + remote).
|
||||
|
||||
**Task-queue mirroring (optional).** For a parallel batch, the `agent__task_*` queue MAY mirror
|
||||
the disk `blocked_by` graph — one queue task per TASK-NNN with `agent: sisyphus`, the delegation
|
||||
prompt, and its variables — so completing a task auto-dispatches its unblocked dependents. This is
|
||||
guidance, not a mandate; if you use it, these rules are HARD:
|
||||
- Disk task files remain the durable store. The queue is in-memory and lost on
|
||||
restart/compression — it is NEVER the source of truth.
|
||||
- Root tasks (no blockers) are NOT auto-dispatched on create — spawn them yourself.
|
||||
- Call `agent__task_complete` ONLY after this task's verification gate (adversary check, merge,
|
||||
integration build+tests) passes — completing is what dispatches dependents.
|
||||
- Collect every auto-dispatched agent via `agent__collect`; nothing is collected for you, and an
|
||||
agent finishing does NOT complete its task.
|
||||
- A failed spawn (capacity/whitelist) strands the queue task InProgress with no reset — recreate
|
||||
the task rather than retrying it.
|
||||
|
||||
Phase F is UNCHANGED (same single draft PR from the run branch). Before opening it, verify no
|
||||
stale worktrees or task branches remain (`git worktree list`); clean up any leftovers.
|
||||
|
||||
|
||||
@@ -12,7 +12,11 @@ spawnable_agents:
|
||||
- explore
|
||||
- oracle
|
||||
max_concurrent_agents: 4
|
||||
max_agent_depth: 2
|
||||
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||
# architecture-reviewer(2) → {explore | oracle}(3). This agent spawns depth-3
|
||||
# children; 4 = one level of headroom.
|
||||
max_agent_depth: 4
|
||||
inject_spawn_instructions: true
|
||||
|
||||
skills_enabled: true
|
||||
|
||||
@@ -14,21 +14,44 @@
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["parse (llm)"] --> B["facts (script): diff, quality bar, ledger, history, trigger signals"]
|
||||
A -. "fallback" .-> PF["parse_fault (script): PIPELINE-FAULT verdict"]
|
||||
PF --> L
|
||||
B --> C["linters (script): tflint / hadolint / actionlint / buf breaking"]
|
||||
C --> D["group_domains (script) --> refine_groups (llm) --> cover_gate (script: exact cover)"]
|
||||
D --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on"]
|
||||
D --> F["aux_lanes (llm): org-context + prior-art spawns"]
|
||||
D --> G["downstream_sweep (script)"]
|
||||
C --> D1["group_propose (script): deterministic domain proposal"]
|
||||
D1 --> D2["refine_groups (llm)"]
|
||||
D2 --> D3["cover_gate (script: exact cover)"]
|
||||
D2 -. "fallback" .-> D3
|
||||
D3 --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on, max_attempts: 2"]
|
||||
D3 --> F["aux_lanes (llm): org-context + prior-art spawns"]
|
||||
D3 --> G["downstream_sweep (script)"]
|
||||
E --> H["completeness_gate (script): required sections even when clean"]
|
||||
E -. "fallback" .-> DF["domain_fault (script): ⚠️ PIPELINE-FAULT slice banner"]
|
||||
H -. "reject-retry once" .-> E
|
||||
H --> I["synthesize (llm: dedup + prose, NO verdict authority)"]
|
||||
F --> I
|
||||
F -. "fallback" .-> AF["aux_fault (script): PIPELINE-FAULT lane-skip note"]
|
||||
AF --> I
|
||||
G --> I
|
||||
I --> J["verify: finding-verifier (agent node, unskippable)"]
|
||||
J --> K["verdict (script): FALSE-drops, rigor folding, MERGE-READY / NEEDS-HUMAN arithmetic"]
|
||||
I --> J["verify: finding-verifier (agent node, unskippable, max_attempts: 2)"]
|
||||
I -. "fallback" .-> K
|
||||
J --> K["verdict (script): FALSE-drops, rigor folding, fault accounting, MERGE-READY / NEEDS-HUMAN arithmetic"]
|
||||
J -. "fallback" .-> K
|
||||
K --> L["render (script): the standard report"]
|
||||
```
|
||||
|
||||
**Failure fails closed.** Every LLM/agent node retries once (`max_attempts: 2` — the
|
||||
domain-reviewer lanes and the verifier included), then routes to a `fallback:` instead of killing
|
||||
the graph, and every fallback that loses review coverage leaves a `PIPELINE-FAULT` marker: a dead
|
||||
`parse` emits a complete fault verdict and jumps straight to render; a dead domain lane becomes a
|
||||
`> ⚠️ PIPELINE-FAULT` banner where its slice report would be; a dead `refine_groups` falls
|
||||
through to the cover gate, which keeps the deterministic proposal; a dead synthesis or verifier
|
||||
falls through to the verdict script. The verdict script counts every fault and forces
|
||||
`NEEDS-HUMAN` — a degraded run never reads `MERGE-READY`. The aux context lanes are the one
|
||||
exception: they are enrichment, so their fault (`aux_fault`) is surfaced as a lane-skip note but
|
||||
never blocks. Fault detection is prefix-anchored (a report that merely *quotes* a marker doesn't
|
||||
trip it), and the report is always emitted: the verdict script's own crash guard emits a
|
||||
`PIPELINE-FAULT` NEEDS-HUMAN, and render bypasses its "No changes to review." stub for fault
|
||||
verdicts — a fault renders as a full report, never silence.
|
||||
|
||||
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
|
||||
report.
|
||||
|
||||
@@ -18,7 +18,12 @@ description: |
|
||||
version: '1.0'
|
||||
|
||||
can_spawn_agents: true
|
||||
max_agent_depth: 3
|
||||
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||
# review-gauntlet | step-runner(2) → code-reviewer(3) →
|
||||
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||
# This agent spawns depth-4 children; 5 = one level of headroom.
|
||||
max_agent_depth: 5
|
||||
max_concurrent_agents: 16
|
||||
|
||||
variables:
|
||||
@@ -91,6 +96,10 @@ initial_state:
|
||||
final_report: ''
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
parse_failure: ''
|
||||
refine_failure: ''
|
||||
aux_failure: ''
|
||||
synth_failure: ''
|
||||
|
||||
start: parse
|
||||
|
||||
@@ -139,8 +148,19 @@ nodes:
|
||||
required: [diff_spec, intent, rigor_in, surfaces_in, project_dir_in, extra_context]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: parse_fault
|
||||
next: facts
|
||||
|
||||
parse_fault:
|
||||
id: parse_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — emits a complete PIPELINE-FAULT verdict_out and jumps straight to render; a review that could not parse its request never reads as MERGE-READY.
|
||||
script: scripts/parse_fault.py
|
||||
timeout: 30
|
||||
next: render
|
||||
|
||||
facts:
|
||||
id: facts
|
||||
type: script
|
||||
@@ -209,6 +229,9 @@ nodes:
|
||||
required: [groups]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
refine_failure: '{{output}}'
|
||||
fallback: cover_gate
|
||||
next: cover_gate
|
||||
|
||||
cover_gate:
|
||||
@@ -272,8 +295,10 @@ nodes:
|
||||
Linter output (extract the lines for your files):
|
||||
{{linter_notes}}
|
||||
{{extra_context}}
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
domain_report: '{{output}}'
|
||||
fallback: domain_fault
|
||||
next: completeness_gate
|
||||
|
||||
completeness_gate:
|
||||
@@ -283,6 +308,13 @@ nodes:
|
||||
script: scripts/completeness_gate.py
|
||||
timeout: 30
|
||||
|
||||
domain_fault:
|
||||
id: domain_fault
|
||||
type: script
|
||||
description: Fail-closed domain lane fault marker (reached only via review_domain's fallback route inside the map branch). Normalizes the engine's "Agent node failed" text in domain_report into a > ⚠️ PIPELINE-FAULT banner the map collects; verdict.py counts any faulted lane and forces NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
|
||||
script: scripts/domain_fault.py
|
||||
timeout: 30
|
||||
|
||||
aux_lanes:
|
||||
id: aux_lanes
|
||||
type: llm
|
||||
@@ -338,6 +370,17 @@ nodes:
|
||||
max_iterations: 10
|
||||
max_attempts: 2
|
||||
timeout: 3600
|
||||
state_updates:
|
||||
aux_failure: '{{output}}'
|
||||
fallback: aux_fault
|
||||
next: synthesize
|
||||
|
||||
aux_fault:
|
||||
id: aux_fault
|
||||
type: script
|
||||
description: Fail-closed marker for dead aux context lanes (reached only via aux_lanes' fallback) — records the failure as a PIPELINE-FAULT in aux_note, which synthesize surfaces as a lane skip. Aux lanes are enrichment; the degradation is visible but never blocks.
|
||||
script: scripts/aux_fault.py
|
||||
timeout: 30
|
||||
next: synthesize
|
||||
|
||||
downstream_sweep:
|
||||
@@ -456,6 +499,9 @@ nodes:
|
||||
required: [walkthrough, changes_rows, findings, cross_file, operational_history, org_context_section, attention_flags, placement_notes]
|
||||
max_iterations: 6
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
synth_failure: '{{output}}'
|
||||
fallback: verdict
|
||||
next: verify
|
||||
|
||||
verify:
|
||||
@@ -472,8 +518,10 @@ nodes:
|
||||
Findings (JSON; verify every 🔴 and 🟡 entry — id, severity, path, lines, claim
|
||||
are the fields; `title` + `block` together are the claim):
|
||||
{{findings}}
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
verifier_output: '{{output}}'
|
||||
fallback: verdict
|
||||
next: verdict
|
||||
|
||||
verdict:
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for dead aux context lanes of the code-reviewer graph.
|
||||
|
||||
Reached ONLY via aux_lanes' `fallback:` route. aux_lanes' state_updates wrote
|
||||
an "LLM node …failed: <chain>" string into `aux_failure` before routing here;
|
||||
this node records the failure as a PIPELINE-FAULT in `aux_note`, which flows
|
||||
into synthesize's "lane skips" prompt line. Aux lanes are ENRICHMENT, not
|
||||
review coverage: the degradation is made VISIBLE in the report's inputs but
|
||||
never blocks — verdict.py deliberately does NOT scan aux_note for faults.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("aux_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(
|
||||
json.dumps(
|
||||
{"aux_note": f"PIPELINE-FAULT: aux context lanes failed after retries — {detail}"}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"aux_note": "PIPELINE-FAULT: aux context lanes failed after retries — "
|
||||
f"fault-marker script error: {e}"
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -54,6 +54,11 @@ def main():
|
||||
else:
|
||||
chosen = [{"domain": g["domain"], "files": g["files"]} for g in refined]
|
||||
if chosen is None:
|
||||
# refine_groups' fallback lands here with `groups` never set; make the
|
||||
# degradation visible in the synthesis prompt via groups_note.
|
||||
refine_failure = state.get("refine_failure")
|
||||
if not refined and isinstance(refine_failure, str) and refine_failure.startswith("LLM node"):
|
||||
note = "refinement lane failed after retries — fell back to the deterministic grouping"
|
||||
chosen = [{"domain": g["domain"], "files": g["files"]} for g in proposed if isinstance(g, dict)]
|
||||
|
||||
print(json.dumps({"group_items": chosen, "groups_note": note}))
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed domain lane fault marker for the code-reviewer graph.
|
||||
|
||||
Reached ONLY via review_domain's `fallback:` route inside the map branch.
|
||||
Grounded in engine behavior: on agent-node failure the engine writes
|
||||
"Agent node failed: {e:#}" into `domain_report` via the node's state_updates
|
||||
BEFORE routing to the fallback, so the raw failure text is already in state.
|
||||
This node normalizes it into a "> ⚠️ PIPELINE-FAULT: domain review lane
|
||||
failed after retries — …" banner the map collects as the lane's report;
|
||||
verdict.py counts any banner-prefixed report as a fault and forces
|
||||
NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
|
||||
|
||||
Slice identity comes from `domain_group` (the map item cover_gate emitted,
|
||||
which carries `domain` and `files`); a missing/malformed item degrades to
|
||||
the failure detail alone.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
FAULT_PREFIX = "> ⚠️ PIPELINE-FAULT: domain review lane failed after retries — "
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
slice_desc = ""
|
||||
group = state.get("domain_group")
|
||||
if isinstance(group, dict):
|
||||
domain = group.get("domain")
|
||||
files = [f for f in (group.get("files") or []) if isinstance(f, str)]
|
||||
if isinstance(domain, str) and domain.strip():
|
||||
slice_desc = f"domain '{domain.strip()}' (files: {', '.join(files)}): "
|
||||
raw = state.get("domain_report")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps({"domain_report": f"{FAULT_PREFIX}{slice_desc}{detail}"}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"domain_report": "> ⚠️ PIPELINE-FAULT: domain review lane failed "
|
||||
f"after retries — fault-marker script error: {e}"
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead parse stage of the code-reviewer graph.
|
||||
|
||||
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||
this node emits a complete PIPELINE-FAULT `verdict_out` and the graph jumps
|
||||
straight to render — a review that could not even parse its request never
|
||||
reads as MERGE-READY. render.py renders a PIPELINE-FAULT verdict despite the
|
||||
empty changed-file list (its "No changes to review." early return skips
|
||||
fault-reason verdicts).
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
ATTENTION_LINE = (
|
||||
"PIPELINE-FAULT: parse failed — the review request could not be parsed; "
|
||||
"the review did not run"
|
||||
)
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def fault_verdict(detail):
|
||||
return {
|
||||
"verdict": "NEEDS-HUMAN",
|
||||
"reason": f"PIPELINE-FAULT: parse failed — {detail}",
|
||||
"counts": {},
|
||||
"deferred_count": 0,
|
||||
"dropped_count": 0,
|
||||
"dropped_titles": [],
|
||||
"attention": [ATTENTION_LINE],
|
||||
"findings_final": [],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("parse_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the parse stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps({"verdict_out": fault_verdict(detail)}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(json.dumps({"verdict_out": fault_verdict(f"fault-marker script error: {e}")}))
|
||||
@@ -16,15 +16,28 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
# LLM-influenced values rendered after the summary line: a newline could
|
||||
# forge a second summary line that review-gauntlet/verdict_gate.py parses.
|
||||
def one_line(x):
|
||||
return str(x).replace("\n", " ")
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
v = state.get("verdict_out") or {}
|
||||
findings = v.get("findings_final") or []
|
||||
counts = v.get("counts") or {}
|
||||
changed = state.get("changed_files") or []
|
||||
rigor = state.get("resolved_rigor") or "production"
|
||||
rigor = one_line(state.get("resolved_rigor") or "production")
|
||||
|
||||
if not changed:
|
||||
# A fault can leave changed_files empty (a parse fault dies before the
|
||||
# diff is resolved; a verifier fault fires even on an empty diff) — a
|
||||
# fault-degraded verdict must still render as a full report. parse_fault
|
||||
# puts the marker in the reason; verdict.py prepends it to attention.
|
||||
faulted = str(v.get("reason") or "").startswith("PIPELINE-FAULT:") or any(
|
||||
isinstance(a, str) and a.startswith("PIPELINE-FAULT:") for a in v.get("attention") or []
|
||||
)
|
||||
if not changed and not faulted:
|
||||
print(json.dumps({"final_report": "# Code Review Summary\n\nNo changes to review."}))
|
||||
return
|
||||
|
||||
@@ -32,7 +45,8 @@ def main():
|
||||
lines = [
|
||||
"# Code Review Summary",
|
||||
"",
|
||||
f"**Verdict: {verdict}** — {v.get('reason', '')}",
|
||||
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
|
||||
f"**Verdict: {verdict}** — {one_line(v.get('reason', ''))}",
|
||||
"",
|
||||
"## Walkthrough",
|
||||
state.get("walkthrough") or "(no walkthrough provided)",
|
||||
@@ -89,15 +103,17 @@ def main():
|
||||
m for x in findings for m in
|
||||
__import__("re").findall(r"MISS-\d+", (x.get("title") or "") + (x.get("block") or ""))
|
||||
})
|
||||
dropped = ", ".join(one_line(t) for t in v.get("dropped_titles", [])[:5])
|
||||
lines += [
|
||||
"---",
|
||||
# CONTRACT: review-gauntlet/verdict_gate.py parses this line — keep the format in sync
|
||||
f"*Reviewed {len(changed)} files, found {counts.get('🔴', 0)} critical, "
|
||||
f"{counts.get('🟡', 0)} warnings, {counts.get('🟢', 0)} suggestions, "
|
||||
f"{counts.get('💡', 0)} nitpicks ({v.get('deferred_count', 0)} deferred by quality bar)*",
|
||||
f"*Quality bar: {rigor} — provenance: {state.get('bar_provenance', '?')}; "
|
||||
f"surfaces: {state.get('resolved_surfaces') or 'none'}*",
|
||||
f"*Quality bar: {rigor} — provenance: {one_line(state.get('bar_provenance', '?'))}; "
|
||||
f"surfaces: {one_line(state.get('resolved_surfaces') or 'none')}*",
|
||||
f"*Verdict: {verdict}; verifier: dropped {v.get('dropped_count', 0)} provably-false"
|
||||
+ (f" ({', '.join(v.get('dropped_titles', [])[:5])})" if v.get("dropped_count") else "")
|
||||
+ (f" ({dropped})" if v.get("dropped_count") else "")
|
||||
+ f"; review-miss patterns applied: {', '.join(miss_ids) if miss_ids else 'none relevant'}*",
|
||||
]
|
||||
print(json.dumps({"final_report": "\n".join(lines)}))
|
||||
|
||||
@@ -8,6 +8,12 @@ The model that wrote the findings never grades its own homework:
|
||||
markers, exactly per the published rules;
|
||||
- MERGE-READY/NEEDS-HUMAN is computed from counts + always-human triggers
|
||||
(deterministic signals ∪ the synthesis's ADDITIVE-only flags).
|
||||
Fail-closed: any PIPELINE-FAULT recorded upstream (dead domain lane, dead
|
||||
verifier, verifier verdicts carrying a PIPELINE-FAULT note or the
|
||||
`pipeline-fault` sentinel id, dead synthesis, missing domain reports)
|
||||
forces NEEDS-HUMAN — a degraded run can never read MERGE-READY. Faults
|
||||
always lead the attention list; the display cap applies only to the
|
||||
non-fault entries so a degraded run can never hide a fault.
|
||||
The gate never crashes into a silent verdict: any internal error emits
|
||||
NEEDS-HUMAN naming the error.
|
||||
"""
|
||||
@@ -16,6 +22,9 @@ import json
|
||||
import os
|
||||
import re
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
@@ -30,35 +39,43 @@ def sev_icon(severity):
|
||||
return "🟢"
|
||||
|
||||
|
||||
def parse_verifier(text):
|
||||
"""Extract {id: {verdict, evidence, note}} from finding-verifier output."""
|
||||
verdicts = {}
|
||||
def parse_verifier_entries(text):
|
||||
"""Extract the raw list of {id, verdict, evidence, note} dicts from
|
||||
finding-verifier output. Order and duplicates are preserved so fault
|
||||
accounting can count every entry (crash-guard entries all carry the
|
||||
same "unknown" id)."""
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
return verdicts
|
||||
return []
|
||||
start, end = text.find("["), text.rfind("]")
|
||||
if start != -1 and end > start:
|
||||
try:
|
||||
for v in json.loads(text[start : end + 1]):
|
||||
if isinstance(v, dict) and v.get("id"):
|
||||
verdicts[str(v["id"])] = v
|
||||
return verdicts
|
||||
return [
|
||||
v for v in json.loads(text[start : end + 1]) if isinstance(v, dict) and v.get("id")
|
||||
]
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
entries = []
|
||||
for m in re.finditer(r'\{[^{}]*"id"[^{}]*\}', text):
|
||||
try:
|
||||
v = json.loads(m.group(0))
|
||||
if v.get("id"):
|
||||
verdicts[str(v["id"])] = v
|
||||
entries.append(v)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
return verdicts
|
||||
return entries
|
||||
|
||||
|
||||
def parse_verifier(entries):
|
||||
"""Index verifier entries by finding id; the last entry per id wins."""
|
||||
return {str(v["id"]): v for v in entries}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
findings = [f for f in (state.get("findings") or []) if isinstance(f, dict)]
|
||||
rigor = (state.get("resolved_rigor") or "production").lower()
|
||||
verdicts = parse_verifier(state.get("verifier_output"))
|
||||
verifier_entries = parse_verifier_entries(state.get("verifier_output"))
|
||||
verdicts = parse_verifier(verifier_entries)
|
||||
|
||||
kept, dropped_titles = [], []
|
||||
for f in findings:
|
||||
@@ -117,17 +134,73 @@ def main():
|
||||
seen = set()
|
||||
attention = [a for a in attention if not (a.lower() in seen or seen.add(a.lower()))]
|
||||
|
||||
if (state.get("changed_files") or []) and not (state.get("domain_reports") or []):
|
||||
attention.append(
|
||||
"PIPELINE FAULT: the diff is non-empty but no domain reports were produced — "
|
||||
# --- pipeline-fault accounting (fail closed) -----------------------------
|
||||
# PREFIX-anchored: a report that merely QUOTES a marker mid-text must not
|
||||
# trip it. Any fault forces NEEDS-HUMAN.
|
||||
faults = []
|
||||
changed = state.get("changed_files") or []
|
||||
reports = state.get("domain_reports") or []
|
||||
faulted = sum(
|
||||
1 for r in reports if isinstance(r, str) and r.lstrip().startswith("> ⚠️ PIPELINE-FAULT:")
|
||||
)
|
||||
if faulted:
|
||||
faults.append(
|
||||
f"PIPELINE-FAULT: {faulted} of {len(reports)} domain review lane(s) "
|
||||
"failed after retries — their slices were not reviewed"
|
||||
)
|
||||
verifier_raw = state.get("verifier_output")
|
||||
if isinstance(verifier_raw, str) and verifier_raw.lstrip().startswith("Agent node failed:"):
|
||||
faults.append(
|
||||
"PIPELINE-FAULT: finding verification failed — findings render as unverified"
|
||||
)
|
||||
# finding-verifier completes "successfully" with faults INSIDE the payload:
|
||||
# parse_fault's `pipeline-fault` sentinel entry, or a per-finding
|
||||
# PIPELINE-FAULT note from verify_fault or verdict_gate's crash guard.
|
||||
# Anchored on the sentinel id / note prefix only. Counted over the raw
|
||||
# entries, not the id-indexed dict, so N crash-guard entries sharing the
|
||||
# "unknown" id count as N.
|
||||
fv_faults = [
|
||||
v
|
||||
for v in verifier_entries
|
||||
if v.get("id") == "pipeline-fault"
|
||||
or (isinstance(v.get("note"), str) and v["note"].lstrip().startswith("PIPELINE-FAULT:"))
|
||||
]
|
||||
if fv_faults:
|
||||
first = fv_faults[0].get("note")
|
||||
detail = first.strip().replace("\n", " ") if isinstance(first, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
faults.append(
|
||||
f"PIPELINE-FAULT: finding verification degraded — {len(fv_faults)} verifier "
|
||||
f"verdict(s) carry a pipeline fault ({detail})"
|
||||
)
|
||||
# Keyed off synth_failure + a non-empty diff, NEVER off findings == [] —
|
||||
# a clean review legitimately has zero findings and stays MERGE-READY.
|
||||
synth_failure = state.get("synth_failure")
|
||||
if isinstance(synth_failure, str) and synth_failure.startswith("LLM node") and changed:
|
||||
detail = synth_failure.strip().replace("\n", " ")
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
faults.append(f"PIPELINE-FAULT: synthesis failed — findings unavailable; {detail}")
|
||||
if changed and not reports:
|
||||
faults.append(
|
||||
"PIPELINE-FAULT: the diff is non-empty but no domain reports were produced — "
|
||||
"the review did not actually run; do not trust this verdict."
|
||||
)
|
||||
# faults go FIRST and uncapped so the display cap can never hide one;
|
||||
# the trigger reason keys off the human-facing triggers alone
|
||||
has_triggers = bool(attention)
|
||||
attention = faults + attention[:5]
|
||||
|
||||
reasons = []
|
||||
if counts["🔴"]:
|
||||
reasons.append(f"{counts['🔴']} 🔴 CRITICAL finding(s)")
|
||||
if yellow_correctness:
|
||||
reasons.append(f"{yellow_correctness} 🟡 [correctness] finding(s) outside the deferred section")
|
||||
if attention:
|
||||
if faults:
|
||||
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
|
||||
reasons.append("pipeline fault(s) recorded — degraded run")
|
||||
if has_triggers:
|
||||
reasons.append("always-human trigger(s) fired")
|
||||
verdict = "NEEDS-HUMAN" if reasons else "MERGE-READY"
|
||||
reason_line = "; ".join(reasons) if reasons else "no blocking findings and no always-human triggers"
|
||||
@@ -142,7 +215,7 @@ def main():
|
||||
"deferred_count": len(deferred),
|
||||
"dropped_count": len(dropped_titles),
|
||||
"dropped_titles": dropped_titles,
|
||||
"attention": attention[:5],
|
||||
"attention": attention,
|
||||
"findings_final": final,
|
||||
}
|
||||
}
|
||||
@@ -158,9 +231,13 @@ except Exception as e: # noqa: BLE001 — never crash into a silent verdict
|
||||
{
|
||||
"verdict_out": {
|
||||
"verdict": "NEEDS-HUMAN",
|
||||
"reason": f"verdict computation error: {e} — treat as needing human review",
|
||||
# PIPELINE-FAULT prefix is load-bearing: render.py's
|
||||
# empty-diff stub bypass anchors on it — without it a
|
||||
# crashed gate on an empty diff collapses into the
|
||||
# "No changes to review." stub.
|
||||
"reason": f"PIPELINE-FAULT: verdict computation error: {e} — human review required",
|
||||
"counts": {}, "deferred_count": 0, "dropped_count": 0,
|
||||
"dropped_titles": [], "attention": [f"verdict script error: {e}"],
|
||||
"dropped_titles": [], "attention": [f"PIPELINE-FAULT: verdict script error: {e}"],
|
||||
"findings_final": [],
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,11 +22,13 @@ agent, this is the file to read alongside the
|
||||
|
||||
## Workflow
|
||||
|
||||
17 nodes. Solid arrows are static `next` / `routes` edges declared in
|
||||
23 nodes. Solid arrows are static `next` / `routes` edges declared in
|
||||
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
|
||||
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
|
||||
`research_each_question` node spawns one `research_one_question` branch per
|
||||
sub-question and joins them before continuing.
|
||||
sub-question and joins them before continuing — and `fallback` edges
|
||||
(labeled "fallback"), taken when a node fails outright (see *Fault paths*
|
||||
below).
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
@@ -38,18 +40,27 @@ flowchart TD
|
||||
bootstrap_research --> plan
|
||||
bootstrap_research --> knowledge_lookup
|
||||
plan["plan<br/>llm + output_schema"] --> research_each_question
|
||||
plan -. "fallback" .-> plan_fault{"plan_fault<br/>script"}
|
||||
plan_fault --> end_fault
|
||||
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
|
||||
research_each_question[\research_each_question<br/>map/]
|
||||
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
|
||||
research_one_question -. "fallback" .-> question_fault{"question_fault<br/>script"}
|
||||
research_each_question --> combine_findings
|
||||
combine_findings{"combine_findings<br/>script"} --> vet_sources
|
||||
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
|
||||
vet_sources -. "fallback" .-> vet_fault{"vet_fault<br/>script"}
|
||||
vet_fault --> critique
|
||||
critique["critique<br/>llm"] --> reflexion_gate
|
||||
critique -. "fallback" .-> critique_fault{"critique_fault<br/>script"}
|
||||
critique_fault --> reflexion_gate
|
||||
reflexion_gate{"reflexion_gate<br/>script"}
|
||||
reflexion_gate -->|"PASS"| synthesize
|
||||
reflexion_gate -->|"REVISE (budget left)"| research_each_question
|
||||
reflexion_gate -->|"REVISE (budget spent)"| synthesize
|
||||
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
|
||||
synthesize -. "fallback" .-> synth_fault{"synth_fault<br/>script"}
|
||||
synth_fault --> end_fault
|
||||
verify_sources{"verify_sources<br/>script"} --> approve
|
||||
approve{{"approve<br/>approval"}}
|
||||
approve -->|"accept"| end_accepted
|
||||
@@ -59,13 +70,14 @@ flowchart TD
|
||||
|
||||
end_accepted(["end_accepted<br/>report"])
|
||||
end_rejected(["end_rejected"])
|
||||
end_fault(["end_fault<br/>DEEP_RESEARCH FAILED"])
|
||||
```
|
||||
|
||||
### Node-type breakdown
|
||||
|
||||
| Type | Nodes |
|
||||
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
||||
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback` |
|
||||
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback`, plus the fault markers `plan_fault`, `question_fault`, `vet_fault`, `critique_fault`, `synth_fault` |
|
||||
| `llm` (tools: `[]`) | `plan`, `critique` |
|
||||
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
|
||||
| `rag` | `knowledge_lookup` — local corpus retrieval |
|
||||
@@ -73,7 +85,7 @@ flowchart TD
|
||||
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
|
||||
| `input` | `ask_topic` |
|
||||
| `approval` | `approve` |
|
||||
| `end` | `end_accepted`, `end_rejected` |
|
||||
| `end` | `end_accepted`, `end_rejected`, `end_fault` |
|
||||
|
||||
## Parallel execution
|
||||
|
||||
@@ -217,6 +229,46 @@ folds that text into `research_feedback` and loops back to
|
||||
`settings.max_loop_iterations` (40) is the engine's infinite-loop
|
||||
backstop: it caps the total visits to any single node.
|
||||
|
||||
## Fault paths
|
||||
|
||||
Every stage that can die outright has a `fallback:` edge to a small
|
||||
fault-marker script — the dashed edges in the diagram. A fallback fires
|
||||
after a node's retries are exhausted (`research_one_question` and
|
||||
`synthesize` each get two attempts via `max_attempts: 2`); the engine
|
||||
writes its failure text into the node's state key before routing, and
|
||||
the marker normalizes that text into a `PIPELINE-FAULT: ...` entry in
|
||||
`pipeline_faults`. The five routes:
|
||||
|
||||
- `plan -> plan_fault -> end_fault` — fail-closed: no plan means
|
||||
nothing to research.
|
||||
- `research_one_question -> question_fault` — branch-local, inside the
|
||||
map branch: one dead research lane becomes a fault finding instead of
|
||||
sinking the whole map.
|
||||
- `vet_sources -> vet_fault -> critique` — degrade-visibly: vetting is
|
||||
a quality lane, not the product; `source_assessment` becomes a
|
||||
neutral "unvetted" note.
|
||||
- `critique -> critique_fault -> reflexion_gate` — degrade-visibly: the
|
||||
rewritten critique deliberately has no `VERDICT:` line, so the gate's
|
||||
malformed-critique PASS default proceeds to synthesis.
|
||||
- `synthesize -> synth_fault -> end_fault` — fail-closed: no report
|
||||
means no product.
|
||||
|
||||
On the fail-closed routes, `end_fault` renders the
|
||||
`DEEP_RESEARCH FAILED — PIPELINE-FAULT: ...` sentinel. On the
|
||||
degrade-visibly routes, `verify_sources` folds the accumulated faults
|
||||
into a `## Pipeline notes` section shown at the approval step and
|
||||
appended to the accepted report (`{{report}}{{pipeline_notes}}`;
|
||||
empty when there were no faults). Degraded output is never a silent
|
||||
pass — every fault is visible to the reviewer or the caller — and some
|
||||
end node always renders its sentinel, so the graph degrades instead of
|
||||
dying.
|
||||
|
||||
All 11 scripts also carry a top-level crash guard: a script that
|
||||
throws records a fault and fails forward where the flow allows it (a
|
||||
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
|
||||
`verify_sources` reports that sources were NOT checked) instead of
|
||||
killing the run.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
@@ -241,6 +293,9 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
||||
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
|
||||
request and reports which are unreachable, so the human reviewer
|
||||
sees broken citations before approving.
|
||||
- Fault paths degrade visibly, never silently: a `PIPELINE-FAULT`
|
||||
can only ever surface as a failure sentinel or a `## Pipeline
|
||||
notes` section, never as a clean report.
|
||||
|
||||
## Customizing
|
||||
|
||||
@@ -270,7 +325,7 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
||||
|
||||
```
|
||||
assets/agents/deep-research/
|
||||
graph.yaml - agent config + 17-node workflow
|
||||
graph.yaml - agent config + 23-node workflow
|
||||
tools.sh - classify_source custom tool
|
||||
README.md - this file
|
||||
knowledge/
|
||||
@@ -279,9 +334,15 @@ assets/agents/deep-research/
|
||||
scripts/
|
||||
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
|
||||
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
|
||||
plan_fault.py - fail-closed fault marker: dead plan -> end_fault
|
||||
question_fault.py - branch-local fault marker for a dead research lane
|
||||
combine_findings.py - joins map output (question_findings) into findings
|
||||
vet_fault.py - degrade-visibly fault marker: unvetted sources
|
||||
critique_fault.py - degrade-visibly fault marker: skipped critique
|
||||
reflexion_gate.py - _next: research_each_question (revise) or synthesize
|
||||
verify_sources.py - HTTP HEAD on cited URLs / DOIs
|
||||
synth_fault.py - fail-closed fault marker: no report -> end_fault
|
||||
verify_sources.py - HTTP HEAD on cited URLs / DOIs; folds pipeline_faults
|
||||
into the report's "## Pipeline notes" section
|
||||
incorporate_feedback.py - _next: research_each_question, with user feedback
|
||||
```
|
||||
|
||||
|
||||
@@ -38,6 +38,14 @@ initial_state:
|
||||
research_attempts: 0
|
||||
local_context: ''
|
||||
local_sources: ''
|
||||
# Fault-path defaults. `questions: []` also keeps the
|
||||
# knowledge_lookup lane alive for one super-step when `plan` dies inside the
|
||||
# static fan-out: research_each_question maps over zero items while the
|
||||
# plan_fault → end_fault route renders the failure sentinel.
|
||||
pipeline_faults: []
|
||||
pipeline_notes: ''
|
||||
fault_text: ''
|
||||
questions: []
|
||||
|
||||
start: parse_request
|
||||
|
||||
@@ -91,8 +99,22 @@ nodes:
|
||||
maxItems: 6
|
||||
description: 3-5 specific, self-contained sub-questions.
|
||||
required: [research_plan, questions]
|
||||
# On success plan_failure holds the node's structured JSON output; on
|
||||
# failure the engine writes "LLM node ...failed: <chain>" into it BEFORE
|
||||
# routing to the fallback (schema properties are only merged on success).
|
||||
state_updates:
|
||||
plan_failure: '{{output}}'
|
||||
fallback: plan_fault
|
||||
next: research_each_question
|
||||
|
||||
plan_fault:
|
||||
id: plan_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead planning stage (reached only via plan's fallback) — normalizes the failure text captured in plan_failure into a PIPELINE-FAULT and continues to end_fault; no plan means nothing to research, so the DEEP_RESEARCH FAILED sentinel always renders. The sibling knowledge_lookup lane survives one more super-step by mapping over the empty initial_state `questions` list.
|
||||
script: scripts/plan_fault.py
|
||||
fallback: end_fault
|
||||
next: end_fault
|
||||
|
||||
knowledge_lookup:
|
||||
id: knowledge_lookup
|
||||
type: rag
|
||||
@@ -153,6 +175,16 @@ nodes:
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
finding: '{{output}}'
|
||||
# On failure the engine writes "LLM node ...failed: <chain>" into `finding`
|
||||
# via the state_updates above BEFORE routing to the fallback. The marker
|
||||
# is branch-local: a map branch's fallback must stay inside the branch.
|
||||
fallback: question_fault
|
||||
|
||||
question_fault:
|
||||
id: question_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead research lane (reached only via research_one_question's fallback route inside the map branch). Normalizes the engine's failure text in `finding` into a PIPELINE-FAULT finding the map collects — one dead lane never sinks the whole map. combine_findings later lifts PIPELINE-FAULT findings into pipeline_faults for the report's Pipeline notes.
|
||||
script: scripts/question_fault.py
|
||||
|
||||
combine_findings:
|
||||
id: combine_findings
|
||||
@@ -179,6 +211,15 @@ nodes:
|
||||
max_iterations: 15
|
||||
state_updates:
|
||||
source_assessment: '{{output}}'
|
||||
fallback: vet_fault
|
||||
next: critique
|
||||
|
||||
vet_fault:
|
||||
id: vet_fault
|
||||
type: script
|
||||
description: Degrade-visibly marker for a dead vetting stage (reached only via vet_sources' fallback) — vetting is a quality lane, not the product. Records a PIPELINE-FAULT, rewrites source_assessment to a neutral "unvetted" note, and continues to critique, the stage vet_sources' own next pointed to.
|
||||
script: scripts/vet_fault.py
|
||||
fallback: critique
|
||||
next: critique
|
||||
|
||||
critique:
|
||||
@@ -219,6 +260,15 @@ nodes:
|
||||
tools: []
|
||||
state_updates:
|
||||
critique: '{{output}}'
|
||||
fallback: critique_fault
|
||||
next: reflexion_gate
|
||||
|
||||
critique_fault:
|
||||
id: critique_fault
|
||||
type: script
|
||||
description: Degrade-visibly marker for a dead critique stage (reached only via critique's fallback) — critique is a quality lane, not the product. Records a PIPELINE-FAULT and rewrites `critique` to a neutral note with deliberately NO "VERDICT:" line, so reflexion_gate's malformed-critique PASS default proceeds to synthesis; then continues to reflexion_gate, the stage critique's own next pointed to.
|
||||
script: scripts/critique_fault.py
|
||||
fallback: reflexion_gate
|
||||
next: reflexion_gate
|
||||
|
||||
reflexion_gate:
|
||||
@@ -242,10 +292,22 @@ nodes:
|
||||
|
||||
Produce the final report following your instructions.
|
||||
timeout: 300
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
report: '{{output}}'
|
||||
# On failure the engine writes "Agent node failed: <chain>" into `report`
|
||||
# via the state_updates above BEFORE routing to the fallback.
|
||||
fallback: synth_fault
|
||||
next: verify_sources
|
||||
|
||||
synth_fault:
|
||||
id: synth_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead synthesis stage (reached only via synthesize's fallback) — no report means no product. Normalizes the "Agent node failed:" text captured in `report` into a PIPELINE-FAULT and continues to end_fault, which renders the DEEP_RESEARCH FAILED sentinel.
|
||||
script: scripts/synth_fault.py
|
||||
fallback: end_fault
|
||||
next: end_fault
|
||||
|
||||
verify_sources:
|
||||
id: verify_sources
|
||||
type: script
|
||||
@@ -261,7 +323,7 @@ nodes:
|
||||
{{report}}
|
||||
|
||||
----
|
||||
{{source_check}}
|
||||
{{source_check}}{{pipeline_notes}}
|
||||
----
|
||||
|
||||
Accept this report? Pick "accept" or "reject", or type specific
|
||||
@@ -284,9 +346,14 @@ nodes:
|
||||
end_accepted:
|
||||
id: end_accepted
|
||||
type: end
|
||||
output: '{{report}}'
|
||||
output: '{{report}}{{pipeline_notes}}'
|
||||
|
||||
end_rejected:
|
||||
id: end_rejected
|
||||
type: end
|
||||
output: "Research on '{{topic}}' was rejected and discarded."
|
||||
|
||||
end_fault:
|
||||
id: end_fault
|
||||
type: end
|
||||
output: 'DEEP_RESEARCH FAILED — PIPELINE-FAULT: {{fault_text}}'
|
||||
@@ -6,6 +6,12 @@ list on this node fans out into two parallel branches (the LLM planner and
|
||||
the RAG knowledge lookup) as a single super-step. The validator requires
|
||||
declared parallel-branch script outputs, so we emit an empty JSON object
|
||||
explicitly here.
|
||||
|
||||
Fail-safe: the node reads no state and computes nothing, so its
|
||||
happy-path output IS the sane degraded output — the guard emits the same
|
||||
empty object on any crash. No PIPELINE-FAULT is recorded: appending one
|
||||
would require reading state (the very thing that crashed) or clobbering
|
||||
faults recorded upstream by parse_request, and a crash here loses nothing.
|
||||
"""
|
||||
import json
|
||||
|
||||
@@ -14,5 +20,7 @@ def main():
|
||||
print(json.dumps({}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception: # noqa: BLE001 — the fan-out source must never crash the graph
|
||||
print(json.dumps({}))
|
||||
@@ -6,6 +6,18 @@ one entry per sub-question, in input order). Downstream nodes
|
||||
(`vet_sources`, `critique`, `synthesize`) read `{{findings}}` as a
|
||||
single block, so this script renders the array as a Markdown document
|
||||
with one section per question.
|
||||
|
||||
Fault visibility: a research lane that died lands here as a finding
|
||||
starting with "PIPELINE-FAULT:" (written by question_fault inside the map
|
||||
branch, where `pipeline_faults` itself is out of reach — map branches only
|
||||
surface their collected output_key). This script lifts such findings into
|
||||
`pipeline_faults` (deduplicated, since a reflexion or feedback loop can
|
||||
re-run the map) so dead lanes surface in the report's Pipeline notes.
|
||||
|
||||
Fail-safe: on a crash (e.g. malformed GRAPH_STATE), `findings`
|
||||
becomes the fault text itself and a PIPELINE-FAULT entry is recorded —
|
||||
the pipeline proceeds visibly degraded, and the human approval gate is
|
||||
the backstop. Existing faults are preserved best-effort.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@@ -19,21 +31,40 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
questions = state.get("questions") or []
|
||||
per_question = state.get("question_findings") or []
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
|
||||
sections = []
|
||||
for idx, q in enumerate(questions):
|
||||
body = per_question[idx] if idx < len(per_question) else ""
|
||||
if isinstance(body, dict) or isinstance(body, list):
|
||||
body = json.dumps(body, indent=2)
|
||||
if isinstance(body, str) and body.startswith("PIPELINE-FAULT:") and body not in faults:
|
||||
faults.append(body)
|
||||
sections.append(f"## {q}\n\n{body}")
|
||||
|
||||
findings = "\n\n".join(sections) if sections else "No findings gathered."
|
||||
print(json.dumps({"findings": findings}))
|
||||
print(json.dumps({"findings": findings, "pipeline_faults": faults}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a crashed join must degrade, not die
|
||||
faults = safe_faults()
|
||||
fault = (
|
||||
f"PIPELINE-FAULT: combining findings crashed — {e}; per-question "
|
||||
"findings may exist in state but could not be joined"
|
||||
)
|
||||
faults.append(fault)
|
||||
print(json.dumps({"findings": fault, "pipeline_faults": faults}))
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Degrade-visibly marker for a dead critique stage of deep-research.
|
||||
|
||||
Reached ONLY via critique's `fallback:` route. Grounded in engine behavior:
|
||||
on llm-node failure the engine writes "LLM node …failed: <chain>" into
|
||||
`critique` via the node's state_updates BEFORE routing to the fallback.
|
||||
Critique is a quality lane, not the product, so this path degrades instead
|
||||
of failing closed: it records a PIPELINE-FAULT (surfaced later in the
|
||||
report's Pipeline notes) and rewrites `critique` to a neutral note, then
|
||||
continues to `reflexion_gate` — the stage critique's own `next` pointed to.
|
||||
|
||||
The rewritten note deliberately contains NO "VERDICT:" line:
|
||||
reflexion_gate's parse_verdict defaults to PASS when no verdict line is
|
||||
found (its documented malformed-critique behavior), so a dead critique
|
||||
proceeds to synthesis instead of burning the revision budget on a fault.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
CRITIQUE_NOTE = (
|
||||
"Critique unavailable — the critique stage failed (pipeline fault); "
|
||||
"proceeding to synthesis without an automated review pass."
|
||||
)
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
raw = state.get("critique")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the critique stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
fault = f"PIPELINE-FAULT: critique failed — {detail}"
|
||||
if fault not in faults:
|
||||
faults.append(fault)
|
||||
print(json.dumps({"pipeline_faults": faults, "critique": CRITIQUE_NOTE}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
faults = safe_faults()
|
||||
faults.append(f"PIPELINE-FAULT: critique failed — fault-marker script error: {e}")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pipeline_faults": faults,
|
||||
"critique": CRITIQUE_NOTE,
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -10,6 +10,13 @@ reflexion counter is reset so the user-driven pass gets a fresh revision
|
||||
budget.
|
||||
|
||||
Routing (`_next`): always research_each_question.
|
||||
|
||||
Fail-safe: a crash here means the user's feedback text cannot be
|
||||
recovered, but their INTENT — another research pass — is unambiguous
|
||||
(they chose neither accept nor reject). The degraded route therefore
|
||||
still loops to `research_each_question`, with a generic feedback note
|
||||
naming the loss and a PIPELINE-FAULT entry (best-effort preserving
|
||||
existing faults) that surfaces in the next approval's Pipeline notes.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@@ -23,6 +30,14 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
feedback = (state.get("decision") or "").strip()
|
||||
@@ -37,5 +52,26 @@ def main():
|
||||
print(json.dumps(output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a crashed feedback fold must degrade, not die
|
||||
faults = safe_faults()
|
||||
faults.append(
|
||||
f"PIPELINE-FAULT: feedback incorporation crashed — {e}; the reviewer's "
|
||||
"feedback text could not be recovered"
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"_next": "research_each_question",
|
||||
"research_attempts": 0,
|
||||
"research_feedback": (
|
||||
"The user reviewed the report and asked for changes, but "
|
||||
"the feedback text could not be recovered (pipeline "
|
||||
"fault). Re-run the research pass with extra care and "
|
||||
"note this loss."
|
||||
),
|
||||
"pipeline_faults": faults,
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -9,6 +9,13 @@ supply one interactively.
|
||||
Routing (`_next`):
|
||||
- prompt present -> (no _next; static next: plan)
|
||||
- prompt empty -> ask_topic
|
||||
|
||||
Fail-safe: a crash here (e.g. malformed GRAPH_STATE) means the
|
||||
caller's prompt cannot be read at all, so proceeding silently is
|
||||
meaningless — the degraded route is `ask_topic`, which asks the user for
|
||||
the topic directly, plus a PIPELINE-FAULT entry so the loss is visible in
|
||||
the final report's Pipeline notes. Clobbering `pipeline_faults` is safe
|
||||
here: this is the start node, so the list can only hold its initial [].
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@@ -31,5 +38,17 @@ def main():
|
||||
print(json.dumps({"_next": "ask_topic"}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a crashed entry router must degrade, not die
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"_next": "ask_topic",
|
||||
"pipeline_faults": [
|
||||
f"PIPELINE-FAULT: request parsing crashed — {e}; asked the "
|
||||
"user for the research topic instead"
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead planning stage of the deep-research graph.
|
||||
|
||||
Reached ONLY via plan's `fallback:` route. Grounded in engine behavior: on
|
||||
llm-node failure the engine writes "LLM node …failed: <chain>" into
|
||||
`plan_failure` via the node's state_updates BEFORE routing to the fallback
|
||||
(on success the same key holds the node's structured JSON output, which
|
||||
never starts with the "LLM node" prefix). No plan means no sub-questions —
|
||||
there is nothing to research — so this path is fail-closed: record the
|
||||
PIPELINE-FAULT and continue to `end_fault`, which renders the
|
||||
DEEP_RESEARCH FAILED sentinel.
|
||||
|
||||
The planner runs inside the static fan-out [plan, knowledge_lookup]. While
|
||||
this marker routes to end_fault, the sibling branch proceeds one more
|
||||
super-step into `research_each_question`, whose `{{questions}}` resolves to
|
||||
the initial_state [] — the doomed lane maps over zero items and the graph
|
||||
returns end_fault's output at the next join.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
raw = state.get("plan_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the planning stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
fault_text = f"planning failed — nothing to research: {detail}"
|
||||
faults.append(f"PIPELINE-FAULT: {fault_text}")
|
||||
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
fault_text = f"planning failed — fault-marker script error: {e}"
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"fault_text": fault_text,
|
||||
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead research lane of the deep-research graph.
|
||||
|
||||
Reached ONLY via research_one_question's `fallback:` route inside the
|
||||
`research_each_question` map branch (branch-local — a map branch's fallback
|
||||
must stay inside the branch subgraph). Grounded in engine behavior: on
|
||||
llm-node failure the engine writes "LLM node …failed: <chain>" into
|
||||
`finding` via the node's state_updates BEFORE routing to the fallback (on
|
||||
success the same key holds the researched findings, which never start with
|
||||
the "LLM node" prefix). This node normalizes the failure text into a
|
||||
PIPELINE-FAULT finding and the chain ends here — the map collects the fault
|
||||
as the lane's `finding`, so one dead research lane never sinks the whole
|
||||
map. combine_findings later lifts PIPELINE-FAULT findings into
|
||||
`pipeline_faults` so the fault also surfaces in the report's Pipeline notes.
|
||||
The question text is part of the fault so distinct dead lanes with identical
|
||||
engine failure text do not collapse into one entry there.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
MAX_QUESTION_CHARS = 200
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def fault_finding(question, detail):
|
||||
return {
|
||||
"finding": (
|
||||
"PIPELINE-FAULT: question research failed — finding unavailable — "
|
||||
f'{detail} (question: "{question}")'
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
def question_label(state):
|
||||
raw = state.get("question")
|
||||
question = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not question:
|
||||
return "(unknown question)"
|
||||
if len(question) > MAX_QUESTION_CHARS:
|
||||
question = question[: MAX_QUESTION_CHARS - 1] + "…"
|
||||
return question
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("finding")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the research lane died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps(fault_finding(question_label(state), detail)))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
fault_finding("(unknown question)", f"fault-marker script error: {e}")
|
||||
)
|
||||
)
|
||||
@@ -15,6 +15,12 @@ Routing (`_next`):
|
||||
Reflexion is a best-effort quality booster, not a hard gate: once the
|
||||
budget is spent the workflow proceeds anyway, and the human approval
|
||||
step is the final backstop.
|
||||
|
||||
Fail-safe: a crash here fails FORWARD to `synthesize` — the same
|
||||
direction as the existing malformed-critique PASS default — with a
|
||||
PIPELINE-FAULT entry recorded (best-effort preserving existing faults),
|
||||
so a broken gate costs the run its automated critique routing, never the
|
||||
run itself.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@@ -33,6 +39,14 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def as_int(value, default=0):
|
||||
try:
|
||||
return int(value)
|
||||
@@ -72,5 +86,12 @@ def main():
|
||||
print(json.dumps(output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a crashed gate fails forward, never dies
|
||||
faults = safe_faults()
|
||||
faults.append(
|
||||
f"PIPELINE-FAULT: reflexion gate crashed — {e}; proceeded to synthesis "
|
||||
"without automated critique routing"
|
||||
)
|
||||
print(json.dumps({"_next": "synthesize", "pipeline_faults": faults}))
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead synthesis stage of the deep-research graph.
|
||||
|
||||
Reached ONLY via synthesize's `fallback:` route. Grounded in engine
|
||||
behavior: on agent-node failure the engine writes "Agent node failed:
|
||||
{e:#}" into `report` via the node's state_updates BEFORE routing to the
|
||||
fallback (on success the same key holds the report-writer's report, which
|
||||
never starts with that prefix). No report means no product, so this path is
|
||||
fail-closed: normalize the failure text, record the PIPELINE-FAULT, and
|
||||
continue to `end_fault`, which renders the DEEP_RESEARCH FAILED sentinel.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
raw = state.get("report")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("Agent node failed:"):
|
||||
detail = "the synthesis stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
fault_text = f"report synthesis failed — no report was produced: {detail}"
|
||||
faults.append(f"PIPELINE-FAULT: {fault_text}")
|
||||
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
fault_text = f"report synthesis failed — fault-marker script error: {e}"
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"fault_text": fault_text,
|
||||
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -6,6 +6,19 @@ request, and writes a `source_check` summary into state so the human
|
||||
reviewer sees broken citations at the approval step.
|
||||
|
||||
Times out per request so a slow source cannot stall the graph.
|
||||
|
||||
Pipeline-notes fold: end-node templates render unconditionally, so the
|
||||
conditional "## Pipeline notes" section of the accepted report is built
|
||||
HERE — the last stop before the approval gate on every accepted path,
|
||||
including reflexion and feedback re-loops. `pipeline_notes` is the
|
||||
rendered section when `pipeline_faults` is non-empty and the empty
|
||||
string otherwise, keeping the happy-path `{{report}}{{pipeline_notes}}`
|
||||
output byte-identical to the pre-hardening `{{report}}`.
|
||||
|
||||
Fail-safe: per-URL probes were already guarded; the top level now
|
||||
is too. On a crash the summary says sources were NOT checked, a
|
||||
PIPELINE-FAULT is recorded (best-effort preserving existing faults), and
|
||||
pipeline_notes still renders — the approval gate remains the backstop.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@@ -25,6 +38,21 @@ def load_state():
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def render_notes(faults):
|
||||
entries = [f for f in (faults or []) if isinstance(f, str) and f.strip()]
|
||||
if not entries:
|
||||
return ""
|
||||
return "\n\n## Pipeline notes\n\n" + "\n".join(f"- {e}" for e in entries)
|
||||
|
||||
|
||||
def reachable(url, timeout=5.0):
|
||||
req = urllib.request.Request(url, method="HEAD")
|
||||
try:
|
||||
@@ -62,8 +90,27 @@ def main():
|
||||
+ "\n".join(results)
|
||||
)
|
||||
|
||||
print(json.dumps({"source_check": summary}))
|
||||
notes = render_notes(state.get("pipeline_faults"))
|
||||
print(json.dumps({"source_check": summary, "pipeline_notes": notes}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a crashed source check must degrade, not die
|
||||
faults = safe_faults()
|
||||
faults.append(
|
||||
f"PIPELINE-FAULT: source verification crashed — {e}; cited sources "
|
||||
"were not checked for reachability"
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"source_check": (
|
||||
"Source verification crashed (pipeline fault) — cited "
|
||||
"sources were NOT checked for reachability."
|
||||
),
|
||||
"pipeline_notes": render_notes(faults),
|
||||
"pipeline_faults": faults,
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Degrade-visibly marker for a dead source-vetting stage of deep-research.
|
||||
|
||||
Reached ONLY via vet_sources' `fallback:` route. Grounded in engine
|
||||
behavior: on llm-node failure the engine writes "LLM node …failed: <chain>"
|
||||
into `source_assessment` via the node's state_updates BEFORE routing to the
|
||||
fallback. Vetting is a quality lane, not the product, so this path degrades
|
||||
instead of failing closed: it records a PIPELINE-FAULT (surfaced later in
|
||||
the report's Pipeline notes) and rewrites `source_assessment` to a neutral
|
||||
"unvetted" note so downstream prompts (critique, synthesize) never carry
|
||||
raw engine error text, then continues to `critique` — the stage
|
||||
vet_sources' own `next` pointed to.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
UNVETTED_NOTE = (
|
||||
"Source credibility assessment unavailable — the vetting stage failed "
|
||||
"(pipeline fault); treat every cited source as unvetted and needing "
|
||||
"corroboration. Do not request revision solely because this assessment "
|
||||
"is unavailable — re-research cannot restore it; judge the findings on "
|
||||
"their own evidence."
|
||||
)
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def safe_faults():
|
||||
try:
|
||||
state = load_state()
|
||||
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||
return []
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||
raw = state.get("source_assessment")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the vetting stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
fault = f"PIPELINE-FAULT: source vetting failed — {detail}"
|
||||
if fault not in faults:
|
||||
faults.append(fault)
|
||||
print(json.dumps({"pipeline_faults": faults, "source_assessment": UNVETTED_NOTE}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
faults = safe_faults()
|
||||
faults.append(f"PIPELINE-FAULT: source vetting failed — fault-marker script error: {e}")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pipeline_faults": faults,
|
||||
"source_assessment": UNVETTED_NOTE,
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -23,7 +23,13 @@ enabled_skills:
|
||||
spawnable_agents:
|
||||
- file-reviewer
|
||||
max_concurrent_agents: 8
|
||||
max_agent_depth: 2
|
||||
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||
# review-gauntlet | step-runner(2) → code-reviewer(3) → domain-reviewer(4) →
|
||||
# file-reviewer(5). This agent spawns depth-5 children; 6 = one level of headroom.
|
||||
# NOTE: can_spawn_agents is not yet set, so the file-reviewer hop is documented
|
||||
# but inert; 6 keeps the limit forward-compatible for when it is wired.
|
||||
max_agent_depth: 6
|
||||
|
||||
variables:
|
||||
- name: project_dir
|
||||
|
||||
@@ -7,13 +7,23 @@ or confirmation-bias the check. A graph can't: the structure IS the discipline.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
parse (llm) ──▶ verify_each (map, max_concurrency 4) ──▶ done (end)
|
||||
└─▶ verify_one (llm + fs tools) ──▶ verdict_gate (script), one CHAIN per finding
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["parse (llm, max_attempts 2)"] --> B["verify_each (map, max_concurrency 4): one branch per finding"]
|
||||
A -. "fallback" .-> PF["parse_fault (script): verdicts = one UNVERIFIABLE PIPELINE-FAULT entry"]
|
||||
PF --> D
|
||||
B -. "spawns × N" .-> C["verify_one (llm + fs tools, max_attempts 2)"]
|
||||
C --> G["verdict_gate (script): machine-validates the JSON contract"]
|
||||
G -. "reject-retry once" .-> C
|
||||
C -. "fallback" .-> VF["verify_fault (script): branch-local UNVERIFIABLE marker"]
|
||||
B --> D(["done (end): FINDING_VERIFIER_RESULTS"])
|
||||
```
|
||||
|
||||
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
|
||||
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
|
||||
Gets two attempts (`max_attempts: 2`); if both fail, its `fallback` routes to
|
||||
`parse_fault`, which rewrites `verdicts` to a single UNVERIFIABLE entry naming the fault
|
||||
and jumps straight to `done`.
|
||||
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
|
||||
finding is structurally impossible; results collect in input order.
|
||||
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
|
||||
@@ -28,12 +38,20 @@ parse (llm) ──▶ verify_each (map, max_concurrency 4) ──▶ done (end)
|
||||
contract (verdict enum; evidence required for VERIFIED/FALSE; note for UNVERIFIABLE),
|
||||
stamps the authoritative id FROM the finding item (the model's echo is never trusted),
|
||||
rejects back to `verify_one` exactly once with the reason, and on a second failure records
|
||||
UNVERIFIABLE — a malformed branch never sinks the map.
|
||||
Kept, marked unverified.
|
||||
UNVERIFIABLE — a malformed branch never sinks the map. If `verify_one` itself dies (both
|
||||
`max_attempts: 2` attempts fail), its `fallback` routes to `verify_fault`, which
|
||||
normalizes the engine's failure text into a branch-local UNVERIFIABLE verdict the map
|
||||
still collects.
|
||||
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
|
||||
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
|
||||
marked). No final LLM step that could editorialize.
|
||||
|
||||
The dashed edges are fault paths. All three fault emitters (`parse_fault`, `verify_fault`,
|
||||
and `verdict_gate`'s own crash guard) surface as `PIPELINE-FAULT:` text in a verdict's
|
||||
`note`, and a fault can only ever produce UNVERIFIABLE — degraded output is never a
|
||||
passing verdict. Every path ends at `done`, so the `FINDING_VERIFIER_RESULTS`
|
||||
sentinel is always emitted and the caller's verify lane degrades instead of dying.
|
||||
|
||||
## What it is NOT
|
||||
|
||||
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
|
||||
@@ -50,3 +68,6 @@ installed, code-reviewer runs the same contract inline as a fallback.
|
||||
|
||||
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
|
||||
per node).
|
||||
- `scripts/verdict_gate.py` — the deterministic contract gate.
|
||||
- `scripts/parse_fault.py`, `scripts/verify_fault.py` — the fault markers behind the
|
||||
dashed fallback edges.
|
||||
@@ -32,6 +32,7 @@ initial_state:
|
||||
verdicts: []
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
parse_failure: ''
|
||||
|
||||
start: parse
|
||||
|
||||
@@ -92,8 +93,20 @@ nodes:
|
||||
type: string
|
||||
description: Repository directory named in the request (echo the prompt default when none).
|
||||
required: [findings_list, project_dir]
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: parse_fault
|
||||
next: verify_each
|
||||
|
||||
parse_fault:
|
||||
id: parse_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — rewrites `verdicts` to a single UNVERIFIABLE PIPELINE-FAULT entry and jumps straight to done, so the FINDING_VERIFIER_RESULTS sentinel still renders and the caller's verify lane degrades instead of dying.
|
||||
script: scripts/parse_fault.py
|
||||
timeout: 30
|
||||
next: done
|
||||
|
||||
verify_each:
|
||||
id: verify_each
|
||||
type: map
|
||||
@@ -165,6 +178,7 @@ nodes:
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
verdict: '{{output}}'
|
||||
fallback: verify_fault
|
||||
next: verdict_gate
|
||||
|
||||
verdict_gate:
|
||||
@@ -174,6 +188,13 @@ nodes:
|
||||
script: scripts/verdict_gate.py
|
||||
timeout: 30
|
||||
|
||||
verify_fault:
|
||||
id: verify_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead verify branch (reached only via verify_one's fallback; branch-local). Normalizes the engine's failure text in `verdict` into a schema-conformant UNVERIFIABLE verdict the map collects — a dead verifier lane never sinks the whole map or masquerades as verified.
|
||||
script: scripts/verify_fault.py
|
||||
timeout: 30
|
||||
|
||||
done:
|
||||
id: done
|
||||
type: end
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead parse stage of the finding-verifier graph.
|
||||
|
||||
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||
this node rewrites `verdicts` to a single schema-conformant UNVERIFIABLE
|
||||
entry naming the fault and the graph jumps straight to `done` — the
|
||||
FINDING_VERIFIER_RESULTS sentinel still renders, so the caller (e.g.
|
||||
code-reviewer's verify lane) degrades instead of dying. Findings that were
|
||||
never extracted cannot be verified; the caller's "UNVERIFIABLE = keep the
|
||||
finding, marked unverified" semantics do the right thing with the entry.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def fault_verdicts(detail):
|
||||
return {
|
||||
"verdicts": [
|
||||
{
|
||||
"id": "pipeline-fault",
|
||||
"verdict": "UNVERIFIABLE",
|
||||
"evidence": "",
|
||||
"note": (
|
||||
"PIPELINE-FAULT: parse failed — findings could not be extracted "
|
||||
f"from the verification request; no finding was verified: {detail}"
|
||||
),
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("parse_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the parse stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps(fault_verdicts(detail)))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(json.dumps(fault_verdicts(f"fault-marker script error: {e}")))
|
||||
@@ -104,6 +104,7 @@ def main():
|
||||
)
|
||||
return
|
||||
|
||||
# Deliberately not PIPELINE-FAULT-prefixed: model misbehavior, not infrastructure.
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
@@ -128,7 +129,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash the chain
|
||||
"id": "unknown",
|
||||
"verdict": "UNVERIFIABLE",
|
||||
"evidence": "",
|
||||
"note": f"verdict gate error: {e}",
|
||||
"note": f"PIPELINE-FAULT: verdict gate error: {e}",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead verify branch of the finding-verifier graph.
|
||||
|
||||
Reached ONLY via verify_one's `fallback:` route inside the map branch.
|
||||
Grounded in engine behavior: on llm-node failure the engine writes
|
||||
"LLM node failed: <chain>" into `verdict` via the node's state_updates
|
||||
BEFORE routing to the fallback, so the raw failure text is already in
|
||||
state. This node normalizes it into a schema-conformant UNVERIFIABLE
|
||||
verdict (authoritative id stamped from the finding item, mirroring
|
||||
verdict_gate.py) and the chain ends there — the map collects a valid
|
||||
verdict, so a dead verifier lane never sinks the whole map or masquerades
|
||||
as verified.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def finding_id(state):
|
||||
finding = state.get("finding")
|
||||
if isinstance(finding, dict):
|
||||
fid = finding.get("id")
|
||||
if isinstance(fid, str) and fid.strip():
|
||||
return fid.strip()
|
||||
return "unknown"
|
||||
|
||||
|
||||
def fault_verdict(fid, detail):
|
||||
return {
|
||||
"verdict": {
|
||||
"id": fid,
|
||||
"verdict": "UNVERIFIABLE",
|
||||
"evidence": "",
|
||||
"note": f"PIPELINE-FAULT: verifier lane failed after retries — {detail}",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
fid = finding_id(state)
|
||||
raw = state.get("verdict")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the verifier lane died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps(fault_verdict(fid, detail)))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(json.dumps(fault_verdict("unknown", f"fault-marker script error: {e}")))
|
||||
@@ -11,13 +11,21 @@ gets the chance to rationalize past it.
|
||||
```mermaid
|
||||
flowchart TD
|
||||
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
|
||||
parse -. "fallback" .-> pfault["parse_fault<br/>(script: PIPELINE-FAULT marker)"]
|
||||
pfault --> gate
|
||||
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
|
||||
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes override exactly)"]
|
||||
select -. "fallback" .-> dlanes["default_lanes<br/>(script: deterministic degraded<br/>selection — wider, never narrower)"]
|
||||
dlanes --> build
|
||||
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
|
||||
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer"]]
|
||||
madv --> radv[["adversary"]]
|
||||
msec --> rsec[["security-reviewer"]]
|
||||
mpb --> rpb[["probe"]]
|
||||
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer<br/>(max_attempts: 2)"]]
|
||||
madv --> radv[["adversary<br/>(max_attempts: 2)"]]
|
||||
msec --> rsec[["security-reviewer<br/>(max_attempts: 2)"]]
|
||||
mpb --> rpb[["probe<br/>(max_attempts: 2)"]]
|
||||
rcr -. "fallback" .-> lfault["lane_fault<br/>(script: PIPELINE-FAULT lane result,<br/>collected by the map)"]
|
||||
radv -. "fallback" .-> lfault
|
||||
rsec -. "fallback" .-> lfault
|
||||
rpb -. "fallback" .-> lfault
|
||||
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
||||
```
|
||||
@@ -43,18 +51,37 @@ those rules are *structure*:
|
||||
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
||||
the verdict gate on equal-length paths.
|
||||
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
||||
sentinel is a lane FAILURE, never a pass; any 🔴 in the code-review report
|
||||
blocks regardless of its verdict line; `NEEDS-HUMAN` without 🔴 passes but
|
||||
sentinel is a lane FAILURE, never a pass; the critical count from the
|
||||
code-review report's summary line (raw 🔴 count if absent) blocks
|
||||
regardless of its verdict line; a code-review lane whose own verdict line
|
||||
records a pipeline fault blocks (a degraded review is never a pass);
|
||||
`NEEDS-HUMAN` without 🔴 passes but
|
||||
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
|
||||
with an environment note — it is never treated as PASS or FAIL.
|
||||
- **Verdict independence is structural**: lanes run as isolated sub-agents
|
||||
with no `teammates:` flag — no cross-lane messaging, by construction.
|
||||
- **Failure fails closed, structurally** (`fallback:` routes + `PIPELINE-FAULT`
|
||||
markers): every LLM node and lane retries once (`max_attempts: 2`), then a
|
||||
fallback fires instead of killing the graph — and every fallback degrades
|
||||
toward BLOCKED, never toward a pass. A dead lane becomes a `PIPELINE-FAULT:`
|
||||
lane result (`lane_fault.py`) that the gate reports as BLOCKED *naming the
|
||||
lane* — distinct from SKIPPED (not selected). A dead `parse` records a fault
|
||||
and jumps straight to the gate (`parse_fault.py`). A dead `select_lanes`
|
||||
degrades to a deterministic selection (`default_lanes.py`): code-review +
|
||||
adversary, plus probe on consumer surface *with a local-run recipe*, plus
|
||||
security on auth/deps/exec
|
||||
signals or hardened posture, unioned with caller-forced lanes — wider, never
|
||||
narrower. A crashed `build_items` records a fault so an all-SKIPPED gate
|
||||
still blocks. Fault detection is prefix-anchored — a real review that merely
|
||||
*quotes* "PIPELINE-FAULT:" flows to the normal sentinel rules. And the
|
||||
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
||||
`GAUNTLET: BLOCKED`, never silence.
|
||||
|
||||
## Sentinels parsed
|
||||
|
||||
| Lane | Sentinel | Blocks on |
|
||||
|------|----------|-----------|
|
||||
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | any 🔴 finding; missing sentinel |
|
||||
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
|
||||
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; missing sentinel |
|
||||
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
|
||||
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
|
||||
@@ -74,7 +101,18 @@ Plan / acceptance criteria:
|
||||
Local-run recipe / usage suites:
|
||||
<how to boot the service locally + where existing suites live — enables the probe lane>
|
||||
|
||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)"
|
||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)" \
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
```
|
||||
|
||||
Verification commands are NOT part of the prompt. They are the declared
|
||||
`verification_commands` graph variable (default `'[]'`; a JSON array of shell
|
||||
commands as a string, since agent variables arrive as strings), also settable
|
||||
from the CLI:
|
||||
|
||||
```sh
|
||||
coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
|
||||
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
||||
```
|
||||
|
||||
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
|
||||
@@ -89,9 +127,24 @@ final report verbatim.
|
||||
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||
non-trivial work — that's why code-review is unconditionally on); the
|
||||
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet).
|
||||
A lane that dies or times out produces no sentinel — which is a BLOCKED,
|
||||
never a silent pass.
|
||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet)
|
||||
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
|
||||
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
|
||||
pass.
|
||||
- The `verification_commands` variable is forwarded to the adversary lane as a
|
||||
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
||||
lands the raw value in the child's state) — never through the lane prompt
|
||||
and never extracted by `parse`. The adversary's `run_checks` stage executes
|
||||
those commands with a shell, which makes the declaration a trust boundary:
|
||||
the prompt also carries plan text pasted from the repo under review, so
|
||||
commands must never be inferred from it, and the engine merges only the keys
|
||||
`parse`'s `output_schema.properties` declares, so an extra
|
||||
`verification_commands` key emitted by the LLM is dropped before it reaches
|
||||
state.
|
||||
Undeclared stays undeclared
|
||||
(`'[]'` ⇒ "none declared"); a declaration that is not a JSON array of
|
||||
strings executes nothing and records a `PIPELINE-FAULT`, which the
|
||||
adversary lane reports as DIVERGES and the gate as BLOCKED.
|
||||
- Signals failures degrade gracefully: if git can't compute the diff, lane
|
||||
selection falls back to caller context + forced lanes, and the report notes
|
||||
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
||||
|
||||
@@ -19,6 +19,11 @@ description: |
|
||||
|
||||
version: '1.0'
|
||||
|
||||
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||
# review-gauntlet(2) → code-reviewer(3) →
|
||||
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||
# This agent spawns depth-3 children; 4 = one level of headroom.
|
||||
max_agent_depth: 4
|
||||
max_concurrent_agents: 12
|
||||
|
||||
@@ -26,11 +31,23 @@ variables:
|
||||
- name: project_dir
|
||||
description: Repository the change lives in; signals and every review lane run against it.
|
||||
default: '.'
|
||||
- name: verification_commands
|
||||
description: |
|
||||
JSON array of shell commands the CALLER declares for the deterministic
|
||||
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
||||
prompt; empty array = none declared.
|
||||
default: '[]'
|
||||
|
||||
settings:
|
||||
max_loop_iterations: 10
|
||||
log_state_snapshots: true
|
||||
timeout: 10800
|
||||
# max over agent lanes of (max_attempts × lane timeout) = 2 × 11400 = 22800
|
||||
# (agent-node retries each get a fresh per-attempt budget,
|
||||
# src/graph/agent.rs retry_transient/bounded_attempt) + 1200 margin for
|
||||
# the script stages (signals/select/build/gate) = 24000; the parse and
|
||||
# select_lanes llm nodes carry no `timeout:` and are bounded by
|
||||
# max_iterations.
|
||||
timeout: 24000
|
||||
|
||||
initial_state:
|
||||
diff_spec: worktree
|
||||
@@ -56,6 +73,9 @@ initial_state:
|
||||
add_security: false
|
||||
add_probe: false
|
||||
lane_reasons: ''
|
||||
parse_failure: ''
|
||||
select_lanes_failure: ''
|
||||
lanes_degraded: ''
|
||||
code_review_items: []
|
||||
adversary_items: []
|
||||
security_items: []
|
||||
@@ -129,8 +149,19 @@ nodes:
|
||||
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: parse_fault
|
||||
next: signals
|
||||
|
||||
parse_fault:
|
||||
id: parse_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — records a PIPELINE-FAULT in signals_error and jumps straight to the verdict gate, which blocks on it. Lane item lists stay their initial [] so every lane records SKIPPED.
|
||||
script: scripts/parse_fault.py
|
||||
timeout: 30
|
||||
next: verdict_gate
|
||||
|
||||
signals:
|
||||
id: signals
|
||||
type: script
|
||||
@@ -204,6 +235,17 @@ nodes:
|
||||
required: [add_code_review, add_adversary, add_security, add_probe, lane_reasons]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
select_lanes_failure: '{{output}}'
|
||||
fallback: default_lanes
|
||||
next: build_items
|
||||
|
||||
default_lanes:
|
||||
id: default_lanes
|
||||
type: script
|
||||
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface AND a probe_context (local-run recipe) is present, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
|
||||
script: scripts/default_lanes.py
|
||||
timeout: 30
|
||||
next: build_items
|
||||
|
||||
build_items:
|
||||
@@ -263,7 +305,12 @@ nodes:
|
||||
type: agent
|
||||
description: Independent code-quality review lane.
|
||||
agent: code-reviewer
|
||||
timeout: 7200
|
||||
# Exceeds the code-reviewer graph's own settings.timeout (10800) by 600 so
|
||||
# the child's graph timeout, not this lane timeout, is the binding bound.
|
||||
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
|
||||
# timeouts) is not derived here — deliberately out of scope for this
|
||||
# change; tracked as a follow-up.
|
||||
timeout: 11400
|
||||
prompt: |
|
||||
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
|
||||
independent post-implementation review.
|
||||
@@ -276,15 +323,24 @@ nodes:
|
||||
{{extra_context}}
|
||||
|
||||
Close your report with your Review Verdict line.
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
fallback: lane_fault
|
||||
|
||||
run_adversary:
|
||||
id: run_adversary
|
||||
type: agent
|
||||
description: Plan-conformance review lane.
|
||||
agent: adversary
|
||||
timeout: 7200
|
||||
# Exceeds the adversary graph's own settings.timeout (9000) by 600 so the
|
||||
# child's graph timeout, not this lane timeout, is the binding bound.
|
||||
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
|
||||
# timeouts) is not derived here — deliberately out of scope for this
|
||||
# change; tracked as a follow-up.
|
||||
timeout: 9600
|
||||
inputs:
|
||||
verification_commands: '{{verification_commands}}'
|
||||
prompt: |
|
||||
Adversarial plan-conformance review in {{project_dir}}.
|
||||
|
||||
@@ -293,11 +349,15 @@ nodes:
|
||||
The spec / plan / acceptance criteria it must satisfy:
|
||||
{{plan_context}}
|
||||
|
||||
Verification commands arrive as a structured input (the verification_commands variable), not in this prompt.
|
||||
|
||||
{{extra_context}}
|
||||
|
||||
Return your ADVERSARIAL_REVIEW verdict.
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
fallback: lane_fault
|
||||
|
||||
run_security:
|
||||
id: run_security
|
||||
@@ -313,8 +373,10 @@ nodes:
|
||||
{{extra_context}}
|
||||
|
||||
Return your SECURITY_REVIEW verdict.
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
fallback: lane_fault
|
||||
|
||||
run_probe:
|
||||
id: run_probe
|
||||
@@ -332,16 +394,28 @@ nodes:
|
||||
Plan / acceptance-criteria context:
|
||||
{{plan_context}}
|
||||
|
||||
The workspace may contain test files from an aborted earlier attempt of
|
||||
this lane; reconcile rather than duplicate.
|
||||
|
||||
Derive expected behaviors from the plan/contract BEFORE reading
|
||||
implementation code, run existing suites first, then author spec-first
|
||||
tests for the uncovered delta. Return your USAGE_PROBE verdict.
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
fallback: lane_fault
|
||||
|
||||
lane_fault:
|
||||
id: lane_fault
|
||||
type: script
|
||||
description: Fail-closed lane fault marker (reached only via the run_* lanes' fallback routes; shared across the four map branches). Normalizes the engine's "Agent node failed" text in lane_out into a PIPELINE-FAULT lane result the map collects; verdict_gate turns it into BLOCKED — a degraded lane is never a pass.
|
||||
script: scripts/lane_fault.py
|
||||
timeout: 30
|
||||
|
||||
verdict_gate:
|
||||
id: verdict_gate
|
||||
type: script
|
||||
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; 🔴 findings block regardless of the lane's own verdict line.
|
||||
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane.
|
||||
script: scripts/verdict_gate.py
|
||||
timeout: 60
|
||||
next: done
|
||||
|
||||
@@ -8,20 +8,17 @@ skip mechanism — no dynamic routing anywhere.
|
||||
|
||||
The LLM's add_* flags can only ADD lanes. A flag that is false never
|
||||
removes a lane the hard rules selected.
|
||||
|
||||
Fail-closed: a builder crash emits all four lists empty plus a
|
||||
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
|
||||
never kills the graph.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
state = load_state()
|
||||
|
||||
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
|
||||
# canonicalize the same way on the degraded path as it does here.
|
||||
ALIASES = {
|
||||
"code-review": "code-review",
|
||||
"code_review": "code-review",
|
||||
@@ -39,80 +36,113 @@ ALIASES = {
|
||||
}
|
||||
|
||||
|
||||
def text(key):
|
||||
v = state.get(key)
|
||||
return v.strip() if isinstance(v, str) else ""
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
forced_raw = state.get("forced_lanes") or []
|
||||
forced, unknown = [], []
|
||||
for lane in forced_raw:
|
||||
canon = ALIASES.get(str(lane).strip().lower())
|
||||
(forced.append(canon) if canon else unknown.append(str(lane)))
|
||||
def main():
|
||||
state = load_state()
|
||||
|
||||
reasons = []
|
||||
if forced:
|
||||
lanes = set(forced)
|
||||
reasons.append(f"forced lanes honored exactly: {sorted(lanes)}")
|
||||
else:
|
||||
lanes = {"code-review"}
|
||||
reasons.append(
|
||||
"code-review: always on — the gauntlet is only spawned for non-trivial work"
|
||||
)
|
||||
if text("plan_context"):
|
||||
lanes.add("adversary")
|
||||
reasons.append("adversary: a spec/plan/acceptance-criteria context is present")
|
||||
if (
|
||||
state.get("touches_auth")
|
||||
or state.get("touches_deps")
|
||||
or state.get("touches_exec")
|
||||
or state.get("security_posture") == "hardened"
|
||||
):
|
||||
lanes.add("security")
|
||||
def text(key):
|
||||
v = state.get(key)
|
||||
return v.strip() if isinstance(v, str) else ""
|
||||
|
||||
forced_raw = state.get("forced_lanes") or []
|
||||
forced, unknown = [], []
|
||||
for lane in forced_raw:
|
||||
canon = ALIASES.get(str(lane).strip().lower())
|
||||
(forced.append(canon) if canon else unknown.append(str(lane)))
|
||||
|
||||
reasons = []
|
||||
if forced:
|
||||
lanes = set(forced)
|
||||
reasons.append(f"forced lanes honored exactly: {sorted(lanes)}")
|
||||
else:
|
||||
lanes = {"code-review"}
|
||||
reasons.append(
|
||||
"security: attack-surface signals (auth/deps/exec) or hardened posture"
|
||||
"code-review: always on — the gauntlet is only spawned for non-trivial work"
|
||||
)
|
||||
if state.get("consumer_surface") and text("probe_context"):
|
||||
lanes.add("probe")
|
||||
reasons.append(
|
||||
"probe: consumer-facing surface changed and a local-run recipe/suite pointer exists"
|
||||
if text("plan_context"):
|
||||
lanes.add("adversary")
|
||||
reasons.append("adversary: a spec/plan/acceptance-criteria context is present")
|
||||
if (
|
||||
state.get("touches_auth")
|
||||
or state.get("touches_deps")
|
||||
or state.get("touches_exec")
|
||||
or state.get("security_posture") == "hardened"
|
||||
):
|
||||
lanes.add("security")
|
||||
reasons.append(
|
||||
"security: attack-surface signals (auth/deps/exec) or hardened posture"
|
||||
)
|
||||
if state.get("consumer_surface") and text("probe_context"):
|
||||
lanes.add("probe")
|
||||
reasons.append(
|
||||
"probe: consumer-facing surface changed and a local-run recipe/suite pointer exists"
|
||||
)
|
||||
|
||||
# LLM judgment is additive-only.
|
||||
llm_why = text("lane_reasons")[:300]
|
||||
for flag, lane in (
|
||||
("add_code_review", "code-review"),
|
||||
("add_adversary", "adversary"),
|
||||
("add_security", "security"),
|
||||
("add_probe", "probe"),
|
||||
):
|
||||
if state.get(flag) and lane not in lanes:
|
||||
if lane == "probe" and not text("probe_context"):
|
||||
reasons.append(
|
||||
"probe: requested by lane judgment but skipped — no local-run "
|
||||
"recipe/suite pointer (a probe without one is guaranteed INCONCLUSIVE)"
|
||||
)
|
||||
continue
|
||||
if lane == "adversary" and not text("plan_context"):
|
||||
reasons.append(
|
||||
"adversary: requested by lane judgment but skipped — no spec/plan "
|
||||
"to check conformance against"
|
||||
)
|
||||
continue
|
||||
lanes.add(lane)
|
||||
reasons.append(f"{lane}: added by lane-selection judgment — {llm_why}")
|
||||
|
||||
if unknown:
|
||||
reasons.append(f"ignored unknown forced lane name(s): {unknown}")
|
||||
|
||||
# A degraded lane selection (default_lanes fallback) must survive into the
|
||||
# final report; lanes_summary is the only channel this script doesn't
|
||||
# overwrite silently.
|
||||
degraded = text("lanes_degraded")
|
||||
if degraded:
|
||||
reasons.insert(0, degraded)
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"code_review_items": [{"lane": "code-review"}] if "code-review" in lanes else [],
|
||||
"adversary_items": [{"lane": "adversary"}] if "adversary" in lanes else [],
|
||||
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
||||
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
||||
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"code_review_items": [],
|
||||
"adversary_items": [],
|
||||
"security_items": [],
|
||||
"probe_items": [],
|
||||
"lanes_summary": "",
|
||||
"signals_error": f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run",
|
||||
}
|
||||
)
|
||||
|
||||
# LLM judgment is additive-only.
|
||||
llm_why = text("lane_reasons")[:300]
|
||||
for flag, lane in (
|
||||
("add_code_review", "code-review"),
|
||||
("add_adversary", "adversary"),
|
||||
("add_security", "security"),
|
||||
("add_probe", "probe"),
|
||||
):
|
||||
if state.get(flag) and lane not in lanes:
|
||||
if lane == "probe" and not text("probe_context"):
|
||||
reasons.append(
|
||||
"probe: requested by lane judgment but skipped — no local-run "
|
||||
"recipe/suite pointer (a probe without one is guaranteed INCONCLUSIVE)"
|
||||
)
|
||||
continue
|
||||
if lane == "adversary" and not text("plan_context"):
|
||||
reasons.append(
|
||||
"adversary: requested by lane judgment but skipped — no spec/plan "
|
||||
"to check conformance against"
|
||||
)
|
||||
continue
|
||||
lanes.add(lane)
|
||||
reasons.append(f"{lane}: added by lane-selection judgment — {llm_why}")
|
||||
|
||||
if unknown:
|
||||
reasons.append(f"ignored unknown forced lane name(s): {unknown}")
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"code_review_items": [{"lane": "code-review"}] if "code-review" in lanes else [],
|
||||
"adversary_items": [{"lane": "adversary"}] if "adversary" in lanes else [],
|
||||
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
||||
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
||||
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic lane-selection fallback for the review gauntlet.
|
||||
|
||||
Reached ONLY via select_lanes' `fallback:` route, when the additive LLM
|
||||
judgment dies after retries. Emits `forced_lanes` — code-review + adversary,
|
||||
plus probe when the deterministic signals flagged a consumer-facing surface
|
||||
AND a local-run recipe exists (build_items gates probe the same way — a
|
||||
probe without a recipe is INCONCLUSIVE by construction),
|
||||
plus security when the attack-surface signals (auth/deps/exec) or a hardened
|
||||
posture demand it, unioned with any lanes the caller forced (canonicalized
|
||||
with the same aliases build_items uses) — which build_items honors exactly,
|
||||
so selection degrades WIDER, never narrower. The degradation note goes into
|
||||
`lanes_degraded`; build_items folds it into lanes_summary so the final
|
||||
report names the degradation.
|
||||
|
||||
Fail-closed: if the fallback itself crashes it emits the widest safe
|
||||
selection (defaults ∪ security) plus a PIPELINE-FAULT in signals_error so
|
||||
verdict_gate blocks — a crashed fallback never narrows selection silently.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
# Keep in sync with build_items.ALIASES — a caller-forced lane must
|
||||
# canonicalize the same way here as it would on the non-degraded path.
|
||||
ALIASES = {
|
||||
"code-review": "code-review",
|
||||
"code_review": "code-review",
|
||||
"code-reviewer": "code-review",
|
||||
"adversary": "adversary",
|
||||
"adversarial": "adversary",
|
||||
"adversarial-review": "adversary",
|
||||
"security": "security",
|
||||
"security-review": "security",
|
||||
"security-reviewer": "security",
|
||||
"probe": "probe",
|
||||
"usage": "probe",
|
||||
"usage-pattern": "probe",
|
||||
"usage-pattern-testing": "probe",
|
||||
}
|
||||
|
||||
# Stable emission order keeps the fallback deterministic.
|
||||
LANE_ORDER = ["code-review", "adversary", "security", "probe"]
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
lanes = {"code-review", "adversary"}
|
||||
probe_context = state.get("probe_context")
|
||||
if (
|
||||
state.get("consumer_surface")
|
||||
and isinstance(probe_context, str)
|
||||
and probe_context.strip()
|
||||
):
|
||||
lanes.add("probe")
|
||||
# The deterministic security signals are already in state when this
|
||||
# fallback runs; a degraded selection must never drop the security lane
|
||||
# build_items' hard rules would have selected.
|
||||
if (
|
||||
state.get("touches_auth")
|
||||
or state.get("touches_deps")
|
||||
or state.get("touches_exec")
|
||||
or state.get("security_posture") == "hardened"
|
||||
):
|
||||
lanes.add("security")
|
||||
# Caller-forced lanes survive degradation — union, never overwrite.
|
||||
for lane in state.get("forced_lanes") or []:
|
||||
if canon := ALIASES.get(str(lane).strip().lower()):
|
||||
lanes.add(canon)
|
||||
forced = [lane for lane in LANE_ORDER if lane in lanes]
|
||||
raw = state.get("select_lanes_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
note = "lane selection degraded to deterministic defaults"
|
||||
if detail:
|
||||
note += f" — {detail}"
|
||||
print(json.dumps({"forced_lanes": forced, "lanes_degraded": note}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fallback itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"forced_lanes": ["code-review", "adversary", "security"],
|
||||
"lanes_degraded": "lane selection degraded to deterministic "
|
||||
f"defaults (fallback script error: {e})",
|
||||
"signals_error": (
|
||||
f"PIPELINE-FAULT: lane-selection fallback crashed — {e}; "
|
||||
"lanes widened to defaults ∪ security"
|
||||
),
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed lane fault marker for the review gauntlet.
|
||||
|
||||
Reached ONLY via the run_* lane nodes' `fallback:` routes inside their map
|
||||
branches. Grounded in engine behavior: on agent-node failure the engine
|
||||
writes "Agent node failed: {e:#}" into `lane_out` via the node's
|
||||
state_updates BEFORE routing to the fallback, so the raw failure text is
|
||||
already in state. This node normalizes it into a
|
||||
"PIPELINE-FAULT: <lane> lane failed after retries — …" marker that the map
|
||||
collects as the lane result; verdict_gate turns any PIPELINE-FAULT lane
|
||||
result into BLOCKED — a degraded lane can never read as a pass.
|
||||
|
||||
Lane identity comes from `lane_ctx` (the map item build_items emitted, which
|
||||
carries a `lane` field); a missing/malformed item degrades to "unknown".
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
lane = "unknown"
|
||||
lane_ctx = state.get("lane_ctx")
|
||||
if isinstance(lane_ctx, dict):
|
||||
name = lane_ctx.get("lane")
|
||||
if isinstance(name, str) and name.strip():
|
||||
lane = name.strip()
|
||||
raw = state.get("lane_out")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(
|
||||
json.dumps(
|
||||
{"lane_out": f"PIPELINE-FAULT: {lane} lane failed after retries — {detail}"}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"lane_out": "PIPELINE-FAULT: lane failed after retries — "
|
||||
f"fault-marker script error: {e}"
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead parse stage of the review gauntlet.
|
||||
|
||||
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||
this node records the failure as a PIPELINE-FAULT in `signals_error` and the
|
||||
graph jumps straight to the verdict gate. Every lane item list is still its
|
||||
initial [] so every lane records SKIPPED, and verdict_gate's signals_error
|
||||
rule makes the verdict BLOCKED — a gauntlet that could not even parse its
|
||||
request never reads as a pass.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("parse_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
|
||||
f"gauntlet: {detail}"
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
|
||||
f"gauntlet: fault-marker script error: {e}"
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -5,8 +5,21 @@ Parses each configured lane's verdict SENTINEL with a regex — no LLM
|
||||
re-reads, no judgment. The gate's hard rules:
|
||||
|
||||
- a missing sentinel is a LANE FAILURE and blocks — never a pass;
|
||||
- any 🔴 finding in the code-review report blocks regardless of the
|
||||
lane's own verdict line;
|
||||
- a non-string lane result (the map collected an object instead of the
|
||||
lane's rendered text) blocks — it cannot carry a sentinel;
|
||||
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
||||
detected only when the report IS the lane_fault.py marker, i.e. it
|
||||
STARTS with it — a real review merely quoting the marker flows to the
|
||||
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse
|
||||
fault) blocks — a degraded pipeline is never a pass;
|
||||
- a code-review lane whose OWN verdict line records a pipeline fault
|
||||
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
||||
the nested graph completed, but with a dead domain lane, verifier, or
|
||||
synthesis behind it, so its critical count is not trustworthy;
|
||||
- a non-zero critical count in the code-review report blocks regardless
|
||||
of the lane's own verdict line — derived from render.py's deterministic
|
||||
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
|
||||
count is used only when that summary line is missing;
|
||||
- NEEDS-HUMAN without 🔴 passes but is surfaced as attention required;
|
||||
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
||||
treated as PASS or FAIL (fix the environment per the plan's local-run
|
||||
@@ -29,29 +42,98 @@ def load_state():
|
||||
def main():
|
||||
state = load_state()
|
||||
blockers, attention, lane_lines, reports = [], [], [], []
|
||||
malformed = set()
|
||||
|
||||
def lane_report(key):
|
||||
arr = state.get(key) or []
|
||||
if not arr:
|
||||
return None
|
||||
item = arr[0]
|
||||
return item if isinstance(item, str) else json.dumps(item)
|
||||
if isinstance(item, str):
|
||||
return item
|
||||
malformed.add(key)
|
||||
return json.dumps(item)
|
||||
|
||||
def record(lane, status, detail=""):
|
||||
lane_lines.append(f"| {lane} | {status} | {detail} |")
|
||||
|
||||
def fault_excerpt(report):
|
||||
line = next(
|
||||
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
|
||||
)
|
||||
return line.strip()[:200]
|
||||
|
||||
def is_lane_fault(report):
|
||||
# A genuine lane fault is the ENTIRE report emitted by lane_fault.py,
|
||||
# which always begins with the marker. Matching the marker anywhere
|
||||
# would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass
|
||||
# 🔴 counting and sentinel parsing.
|
||||
return report.strip().startswith("PIPELINE-FAULT:")
|
||||
|
||||
def lane_fault_blocker(lane, report):
|
||||
blockers.append(
|
||||
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
||||
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
||||
)
|
||||
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)")
|
||||
|
||||
def malformed_blocker(lane, report):
|
||||
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
||||
record(lane, "BLOCKED", "malformed lane result (non-string)")
|
||||
reports.append((lane, report))
|
||||
|
||||
# --- code-review ------------------------------------------------------
|
||||
cr = lane_report("code_review_results")
|
||||
if cr is None:
|
||||
record("code-review", "SKIPPED", "not selected")
|
||||
elif "code_review_results" in malformed:
|
||||
malformed_blocker("code-review", cr)
|
||||
elif is_lane_fault(cr):
|
||||
lane_fault_blocker("code-review", cr)
|
||||
reports.append(("code-review", cr))
|
||||
else:
|
||||
reports.append(("code-review", cr))
|
||||
m = re.search(r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)", cr)
|
||||
reds = cr.count("🔴")
|
||||
# code-reviewer's verdict.py forces NEEDS-HUMAN on its own internal
|
||||
# faults; render.py emits that as `**Verdict: NEEDS-HUMAN** — <reason>`
|
||||
# (CONTRACT comments at both emitters). Scoped to the FIRST rendered
|
||||
# verdict line — render.py emits it before any finding body — so a
|
||||
# finding that quotes the wording cannot trip it.
|
||||
verdict_line = re.search(r"^\*\*Verdict: .*$", cr, re.MULTILINE)
|
||||
m_fault = verdict_line and re.search(
|
||||
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
|
||||
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
|
||||
verdict_line.group(0),
|
||||
)
|
||||
# Same quoted-marker class as the PIPELINE-FAULT prefix anchoring: a
|
||||
# 🔴 quoted inside finding text or a Changes-table row is not a
|
||||
# finding. render.py's summary line is the authoritative count; the
|
||||
# raw count is the fail-closed fallback when that line is absent. The
|
||||
# LAST match wins: render.py emits the line after every finding body,
|
||||
# so a quoted look-alike earlier in the report cannot shadow it. The
|
||||
# pattern is the FULL shape of render.py's line (CONTRACT comment
|
||||
# there) so a partial look-alike in LLM-authored text never matches.
|
||||
sums = re.findall(
|
||||
r"^\*Reviewed \d+ files, found (\d+) critical, \d+ warnings, \d+ suggestions, "
|
||||
r"\d+ nitpicks \(\d+ deferred by quality bar\)\*$",
|
||||
cr,
|
||||
re.MULTILINE,
|
||||
)
|
||||
reds = int(sums[-1]) if sums else cr.count("🔴")
|
||||
if m_fault:
|
||||
blockers.append(
|
||||
"code-review: the lane reported an internal PIPELINE-FAULT "
|
||||
"(degraded review) — a degraded lane is never a pass"
|
||||
)
|
||||
if reds:
|
||||
blockers.append(
|
||||
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
||||
)
|
||||
if m_fault and reds:
|
||||
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴")
|
||||
elif m_fault:
|
||||
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)")
|
||||
elif reds:
|
||||
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
||||
elif m and m.group(1) == "MERGE-READY":
|
||||
record("code-review", "GREEN", "MERGE-READY")
|
||||
@@ -72,6 +154,11 @@ def main():
|
||||
adv = lane_report("adversary_results")
|
||||
if adv is None:
|
||||
record("adversary", "SKIPPED", "not selected")
|
||||
elif "adversary_results" in malformed:
|
||||
malformed_blocker("adversary", adv)
|
||||
elif is_lane_fault(adv):
|
||||
lane_fault_blocker("adversary", adv)
|
||||
reports.append(("adversary", adv))
|
||||
else:
|
||||
reports.append(("adversary", adv))
|
||||
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
|
||||
@@ -92,6 +179,11 @@ def main():
|
||||
sec = lane_report("security_results")
|
||||
if sec is None:
|
||||
record("security", "SKIPPED", "not selected")
|
||||
elif "security_results" in malformed:
|
||||
malformed_blocker("security", sec)
|
||||
elif is_lane_fault(sec):
|
||||
lane_fault_blocker("security", sec)
|
||||
reports.append(("security", sec))
|
||||
else:
|
||||
reports.append(("security", sec))
|
||||
m = re.search(r"SECURITY_REVIEW:\s*(PASS|FAIL)", sec)
|
||||
@@ -110,6 +202,11 @@ def main():
|
||||
pb = lane_report("probe_results")
|
||||
if pb is None:
|
||||
record("probe", "SKIPPED", "not selected")
|
||||
elif "probe_results" in malformed:
|
||||
malformed_blocker("probe", pb)
|
||||
elif is_lane_fault(pb):
|
||||
lane_fault_blocker("probe", pb)
|
||||
reports.append(("probe", pb))
|
||||
else:
|
||||
reports.append(("probe", pb))
|
||||
m = re.search(r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)", pb)
|
||||
@@ -131,6 +228,12 @@ def main():
|
||||
)
|
||||
record("probe", "BLOCKED", "missing verdict sentinel")
|
||||
|
||||
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
||||
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
||||
signals_error = state.get("signals_error")
|
||||
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error:
|
||||
blockers.append(f"pipeline: {signals_error}")
|
||||
|
||||
verdict = "BLOCKED" if blockers else "PASS"
|
||||
|
||||
md = "\n".join(
|
||||
|
||||
@@ -57,6 +57,7 @@ Spawnable sub-agents (from `config.yaml`):
|
||||
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
|
||||
- 🔄 **Task Management**: Tracks progress and context across complex operations.
|
||||
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
|
||||
- 🧵 **Task-queue bursts**: Dependency-shaped fan-outs (explore → code → verify) may be encoded as `agent__task_create` chains — dependents auto-dispatch when `agent__task_complete` unblocks them; todos remain the tracking source of truth.
|
||||
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
|
||||
- 🚦 **Escalation policy**: `escalation_policy=autonomous` pre-delegates decision-making — standing rulings (`~/.config/coyote/share/standing-rulings.md`, recorded via `coyote --macro record-ruling`) and evidence-backed recommended options are taken without pausing and batched into a “Decisions taken autonomously” digest; secrets, money movement, and destructive actions always escalate.
|
||||
|
||||
|
||||
@@ -234,6 +234,11 @@ instructions: |
|
||||
|
||||
Never artificially cap at a small number. If there are 10 distinct things to find, spawn 10 agents. The system limit is the only ceiling that matters.
|
||||
|
||||
For dependency-shaped bursts (explore → code → verify), `agent__task_create` chains MAY encode the
|
||||
ordering — dependents with `agent`+`prompt` auto-dispatch when `agent__task_complete` unblocks them.
|
||||
Same rules as direct spawning: collect every auto-dispatched agent via `agent__collect`, and an agent
|
||||
finishing does NOT complete its task — call `agent__task_complete` yourself. Todos remain the tracking source of truth.
|
||||
|
||||
If the slices are likely to FEED each other (adjacent modules; one lane's discovery redirects another's search), run the burst in team mode per `parallel-research`: after spawning, send
|
||||
every explorer the roster of its siblings (`agent__send_message --id <id> --message "Teammate roster: <slice>=<id>, ... Message a teammate directly when you find a fact in THEIR slice;
|
||||
check your inbox after each search round and before finalizing."`) so cross-slice facts move directly between lanes instead of waiting for your synthesis. Never do this with reviewer
|
||||
@@ -310,8 +315,14 @@ instructions: |
|
||||
Rigor: <rigor> Security posture: <posture>
|
||||
Plan / acceptance criteria: <pasted verbatim — enables the adversary lane; omit when none>
|
||||
Local-run recipe / usage suites: <from the plan — enables the probe lane; omit when none>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||
```
|
||||
|
||||
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the prompt
|
||||
also carries pasted plan text from the repo under review, and whatever is declared here is
|
||||
executed with a shell — a trust boundary the prose must not cross.
|
||||
|
||||
It computes diff signals deterministically, selects lanes (code-review always; adversary when a
|
||||
spec/plan is present; security on attack-surface signals; probe on consumer-facing surface with a
|
||||
run recipe — judgment can only WIDEN the selection, never narrow it), runs the selected lanes in
|
||||
@@ -387,8 +398,13 @@ instructions: |
|
||||
|
||||
PLAN — acceptance criteria to check against:
|
||||
<paste the task/step spec + acceptance criteria VERBATIM — not a summary>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||
```
|
||||
|
||||
Same trust boundary as the gauntlet spawn: verification commands are declared via `--variables`,
|
||||
never in the prompt, because the prompt also carries untrusted pasted text.
|
||||
|
||||
### Handling adversary findings
|
||||
|
||||
- **`ADVERSARIAL_REVIEW: DIVERGES` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <complaints pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `adversary` ONCE to confirm it now CONFORMS; if it still DIVERGES on the same criteria after one fix cycle, STOP and escalate to the user (the plan or the approach may be wrong — consider `oracle`).
|
||||
|
||||
@@ -27,6 +27,7 @@ flowchart TD
|
||||
gate_blocked -->|"yes"| orient
|
||||
gate_blocked -->|"no"| end_blocked
|
||||
orient["orient<br/>llm, read-only"] --> route_staleness
|
||||
orient -. "LLM fault" .-> note_llm_fault
|
||||
route_staleness{"route_staleness<br/>script"}
|
||||
route_staleness -->|"major deviation"| gate_deviation
|
||||
route_staleness -->|"else"| implement
|
||||
@@ -35,6 +36,7 @@ flowchart TD
|
||||
gate_deviation -->|"abort"| end_rejected
|
||||
gate_deviation -->|"other (user guidance)"| implement
|
||||
implement[["implement<br/>agent → coder"]] --> route_coder_result
|
||||
implement -. "coder crashed<br/>(no retries)" .-> end_failure
|
||||
route_coder_result{"route_coder_result<br/>script"}
|
||||
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
|
||||
route_coder_result -->|"REJECTED / FAILED"| end_failure
|
||||
@@ -51,14 +53,18 @@ flowchart TD
|
||||
fix_loop_gate -->|"budget left"| implement
|
||||
fix_loop_gate -->|"budget spent"| end_failure
|
||||
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
|
||||
edge_case_sweep -. "LLM fault" .-> write_handoff
|
||||
route_sweep{"route_sweep<br/>script"}
|
||||
route_sweep -->|"5+ files or boundary"| independent_review
|
||||
route_sweep -->|"else"| write_handoff
|
||||
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
|
||||
independent_review -. "2 attempts failed" .-> write_handoff
|
||||
route_review{"route_review<br/>script"}
|
||||
route_review -->|"🔴 critical findings"| implement
|
||||
route_review -->|"else"| write_handoff
|
||||
write_handoff["write_handoff<br/>llm"] --> check_handoff
|
||||
write_handoff -. "LLM fault" .-> note_llm_fault
|
||||
note_llm_fault{"note_llm_fault<br/>script"} --> end_failure
|
||||
check_handoff{"check_handoff<br/>script"}
|
||||
check_handoff -->|"schema valid"| gate_user_review
|
||||
check_handoff -->|"one retry"| write_handoff
|
||||
@@ -81,8 +87,32 @@ End nodes emit sentinel outcomes for the caller:
|
||||
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
|
||||
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
|
||||
was rejected at its approval gate.
|
||||
- `STEP_FAILED` — coder failed, the step-level fix budget was exhausted, or
|
||||
the handoff failed validation twice.
|
||||
- `STEP_FAILED` — the coder failed or crashed, the step-level fix budget was
|
||||
exhausted, the handoff failed validation twice, or an orient/handoff LLM
|
||||
fault was recorded (rendered as `Pipeline fault:` in the output).
|
||||
|
||||
## Fault handling
|
||||
|
||||
Node crashes degrade the run visibly instead of masquerading as success:
|
||||
|
||||
- `implement` has **no retries** — rerunning a coder that died mid-edit
|
||||
against an already-mutated tree is unsafe. Its failure text lands in
|
||||
`coder_result` and the run falls straight to `end_failure`.
|
||||
- `independent_review` retries once (`max_attempts: 2`); if both attempts
|
||||
fail it falls back to `write_handoff`, which flags "PIPELINE-FAULT:
|
||||
independent review did not run" in the handoff rather than presenting the
|
||||
failure text as findings. `route_review` carries the same guard before its
|
||||
🔴 grep, so a reviewer fault never spends a fix-loop attempt.
|
||||
- `edge_case_sweep` faults likewise fall through to `write_handoff` and are
|
||||
flagged as "PIPELINE-FAULT: edge-case sweep did not run".
|
||||
- `orient` and `write_handoff` faults route through `note_llm_fault`, which
|
||||
distills the engine's failure text into a `fault_note` rendered in the
|
||||
`STEP_FAILED` output — never an empty shell.
|
||||
|
||||
Degraded is never passing (a skipped review or sweep is recorded as NOT RUN,
|
||||
never paraphrased as covered), and the sentinel is always emitted — every
|
||||
path terminates at one of the four end nodes, so the caller always gets a
|
||||
parseable outcome.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -6,6 +6,12 @@ description: |
|
||||
review -> evidence-backed handoff -> user approval gate. Designed to be
|
||||
delegated to by sisyphus.
|
||||
version: '1.0'
|
||||
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||
# step-runner(2) → code-reviewer(3) →
|
||||
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||
# This agent spawns depth-3 children; 4 = one level of headroom (default 3 had none).
|
||||
max_agent_depth: 4
|
||||
global_tools:
|
||||
- ast_grep.sh
|
||||
- fs_cat.sh
|
||||
@@ -64,6 +70,7 @@ initial_state:
|
||||
plan_summary: ''
|
||||
implementation_brief: ''
|
||||
staleness_report: ''
|
||||
orient_failure: ''
|
||||
has_major_deviation: false
|
||||
deviation_summary: ''
|
||||
user_feedback: ''
|
||||
@@ -80,12 +87,15 @@ initial_state:
|
||||
tests_output: ''
|
||||
edge_case_report: ''
|
||||
downstream_updates: ''
|
||||
sweep_failure: ''
|
||||
needs_independent_review: false
|
||||
review_report: ''
|
||||
review_attempts: 0
|
||||
max_review_attempts: 1
|
||||
handoff_attempts: 0
|
||||
handoff_fix: ''
|
||||
handoff_failure: ''
|
||||
fault_note: ''
|
||||
step_summary: ''
|
||||
|
||||
start: resolve_step
|
||||
@@ -191,7 +201,9 @@ nodes:
|
||||
has_major_deviation,
|
||||
deviation_summary,
|
||||
]
|
||||
fallback: end_failure
|
||||
state_updates:
|
||||
orient_failure: '{{output}}'
|
||||
fallback: note_llm_fault
|
||||
next: route_staleness
|
||||
|
||||
route_staleness:
|
||||
@@ -270,6 +282,12 @@ nodes:
|
||||
timeout: 3600
|
||||
state_updates:
|
||||
coder_result: '{{output}}'
|
||||
# No max_attempts: rerunning a coder that died mid-edit against an
|
||||
# already-mutated tree is unsafe. On failure the "Agent node failed:"
|
||||
# text lands in coder_result (via state_updates above) and end_failure
|
||||
# renders it; end_failure's other keys all have initial_state defaults,
|
||||
# so the STEP_FAILED output degrades to empty build/tests sections.
|
||||
fallback: end_failure
|
||||
next: route_coder_result
|
||||
|
||||
route_coder_result:
|
||||
@@ -376,6 +394,8 @@ nodes:
|
||||
needs_independent_review:
|
||||
type: boolean
|
||||
required: [edge_case_report, downstream_updates, needs_independent_review]
|
||||
state_updates:
|
||||
sweep_failure: '{{output}}'
|
||||
fallback: write_handoff
|
||||
next: route_sweep
|
||||
|
||||
@@ -405,8 +425,10 @@ nodes:
|
||||
Review the changed files against the step plan's acceptance criteria.
|
||||
Preserve severity tags in your findings.
|
||||
timeout: 1200
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
review_report: '{{output}}'
|
||||
fallback: write_handoff
|
||||
next: route_review
|
||||
|
||||
route_review:
|
||||
@@ -450,6 +472,25 @@ nodes:
|
||||
|
||||
If "Gate feedback" below is non-empty, a previous handoff attempt
|
||||
failed validation - fix exactly what it lists.
|
||||
|
||||
PIPELINE-FAULT VISIBILITY: an "Independent review report" below that
|
||||
starts with "Agent node failed:" means the independent review DID NOT
|
||||
RUN - the graph routed here through the reviewer's failure fallback.
|
||||
Never present that text as review findings. Flag it prominently
|
||||
instead: add a Deviations entry "PIPELINE-FAULT: independent review
|
||||
did not run - <failure text>", repeat it under Notes for next step,
|
||||
and carry it into `step_summary` so the user sees it at the approval
|
||||
gate.
|
||||
|
||||
The same rule covers the "Edge-case sweep result" below: text starting
|
||||
with "LLM node" means the edge-case sweep DID NOT RUN - the graph
|
||||
routed here through the sweep's failure fallback. Never present it as
|
||||
sweep findings. Add a Deviations entry "PIPELINE-FAULT: edge-case
|
||||
sweep did not run - <failure text>" and carry it into `step_summary`
|
||||
so the user sees it at the approval gate. If it holds successful
|
||||
structured output or is empty, ignore it - the sweep's real results
|
||||
already flow through the "Edge cases discovered" / "Downstream plan
|
||||
updates" sections above.
|
||||
prompt: |
|
||||
## Step
|
||||
{{step_number}} ({{step_title}}) - plan at {{step_plan_path}}
|
||||
@@ -473,6 +514,9 @@ nodes:
|
||||
## Downstream plan updates
|
||||
{{downstream_updates}}
|
||||
|
||||
## Edge-case sweep result (fault detection)
|
||||
{{sweep_failure}}
|
||||
|
||||
## Independent review report (if any)
|
||||
{{review_report}}
|
||||
|
||||
@@ -501,9 +545,24 @@ nodes:
|
||||
type: string
|
||||
description: 3-6 sentence summary of the step for the user's approval decision - what was done, deviations, anything needing their attention
|
||||
required: [step_summary]
|
||||
fallback: end_failure
|
||||
state_updates:
|
||||
handoff_failure: '{{output}}'
|
||||
fallback: note_llm_fault
|
||||
next: check_handoff
|
||||
|
||||
note_llm_fault:
|
||||
id: note_llm_fault
|
||||
type: script
|
||||
description: |
|
||||
Make LLM-node fallback captures visible: when orient or write_handoff
|
||||
failed (their capture keys hold the engine's "LLM node..." failure
|
||||
text), distill it into fault_note so end_failure can render WHY the
|
||||
step died instead of an empty STEP_FAILED shell.
|
||||
script: scripts/note_llm_fault.sh
|
||||
timeout: 5
|
||||
fallback: end_failure
|
||||
next: end_failure
|
||||
|
||||
check_handoff:
|
||||
id: check_handoff
|
||||
type: script
|
||||
@@ -601,6 +660,7 @@ nodes:
|
||||
Step: {{step_number}} ({{step_title}})
|
||||
Fix attempts: {{fix_attempts}}/{{max_fix_attempts}}
|
||||
Blocking reason (if resolution failed): {{blocking_reason}}
|
||||
Pipeline fault (if any): {{fault_note}}
|
||||
|
||||
Coder result:
|
||||
{{coder_result}}
|
||||
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -n "${GRAPH_STATE_FILE:-}" ]]; then
|
||||
state=$(cat "$GRAPH_STATE_FILE")
|
||||
elif [[ -n "${GRAPH_STATE:-}" ]]; then
|
||||
state="$GRAPH_STATE"
|
||||
else
|
||||
state='{}'
|
||||
fi
|
||||
|
||||
orient_failure=$(echo "$state" | jq -r '.orient_failure // ""')
|
||||
handoff_failure=$(echo "$state" | jq -r '.handoff_failure // ""')
|
||||
|
||||
# Engine failure text is always prefix-anchored "LLM node" (llm.rs); on
|
||||
# success these keys hold the node's structured JSON output, which never
|
||||
# starts with that prefix.
|
||||
if [[ "$orient_failure" == "LLM node"* ]]; then
|
||||
fault_note="PIPELINE-FAULT: orient stage failed — ${orient_failure:0:300}"
|
||||
elif [[ "$handoff_failure" == "LLM node"* ]]; then
|
||||
fault_note="PIPELINE-FAULT: handoff stage failed — ${handoff_failure:0:300}"
|
||||
else
|
||||
fault_note=""
|
||||
fi
|
||||
|
||||
jq -nc --arg note "$fault_note" '{"fault_note": $note}'
|
||||
@@ -13,6 +13,16 @@ review_report=$(echo "$state" | jq -r '.review_report // ""')
|
||||
review_attempts=$(echo "$state" | jq -r '.review_attempts // 0')
|
||||
max_review_attempts=$(echo "$state" | jq -r '.max_review_attempts // 1')
|
||||
|
||||
# Fail-visible guard: a review_report holding the engine's "Agent node
|
||||
# failed:" text means the independent review DID NOT RUN (its fallback
|
||||
# normally routes straight to write_handoff, bypassing this script). Never
|
||||
# treat that text as findings or spend a fix-loop attempt on it -
|
||||
# write_handoff flags the fault prominently in the handoff instead.
|
||||
if [[ "$review_report" == "Agent node failed:"* ]]; then
|
||||
jq -nc '{"_next": "write_handoff"}'
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! grep -qF "🔴" <<< "$review_report"; then
|
||||
jq -nc '{"_next": "write_handoff"}'
|
||||
exit 0
|
||||
|
||||
+9
-5
@@ -213,8 +213,10 @@ nodes:
|
||||
# agent/graph/global default when omitted.
|
||||
skill_instructions: null # Per-node skill-hint text override; uses the built-in default when omitted.
|
||||
output_schema: # Optional JSON Schema. The output is parsed to JSON
|
||||
type: object # and its top-level object keys auto-merge into state
|
||||
properties: # (so `topic` / `needs_deep_dive` become {{topic}} etc).
|
||||
type: object # and only the keys declared under `properties` auto-merge
|
||||
properties: # into state (so `topic` / `needs_deep_dive` become {{topic}}
|
||||
# etc). Undeclared keys the model emits are dropped; lift
|
||||
# one explicitly via state_updates: `{{output.<key>}}`.
|
||||
topic: { type: string }
|
||||
needs_deep_dive: { type: boolean }
|
||||
required: [topic, needs_deep_dive]
|
||||
@@ -284,8 +286,9 @@ nodes:
|
||||
type: array
|
||||
items: { type: string }
|
||||
required: [context, sources]
|
||||
# When `output_schema` is set, top-level keys auto-merge into state, so
|
||||
# `context` and `sources` are produced without needing `state_updates`.
|
||||
# When `output_schema` is set, the keys declared under `properties`
|
||||
# auto-merge into state, so `context` and `sources` are produced without
|
||||
# needing `state_updates`. Keys the schema does not declare are dropped.
|
||||
next: synthesize # Joins with retrieve at `synthesize`.
|
||||
|
||||
# --- script node (the diamond's join; also dispatches) -----------------
|
||||
@@ -369,7 +372,8 @@ nodes:
|
||||
# and is retired when the chain ends. A message delivered
|
||||
# after the chain's last flagged agent step is accepted
|
||||
# but never read.
|
||||
output_schema: # Optional. Same extraction as llm nodes
|
||||
output_schema: # Optional. Same extraction as llm nodes: only the keys
|
||||
# declared under `properties` auto-merge into state.
|
||||
type: object
|
||||
properties:
|
||||
summary: { type: string }
|
||||
|
||||
+351
-16
@@ -523,14 +523,7 @@ pub async fn call_chat_completions(
|
||||
ctx.app.config.print_markdown(&text)?;
|
||||
}
|
||||
}
|
||||
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
|
||||
if let Some(first) = tool_results.first_mut() {
|
||||
first.thinking = thinking;
|
||||
}
|
||||
tool_results
|
||||
.iter()
|
||||
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
|
||||
Ok((text, tool_results))
|
||||
finish_completion(ctx, text, tool_calls, thinking).await
|
||||
}
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
@@ -592,14 +585,7 @@ pub async fn call_chat_completions_streaming(
|
||||
if !silent && !text.is_empty() && !text.ends_with('\n') {
|
||||
println!();
|
||||
}
|
||||
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
|
||||
if let Some(first) = tool_results.first_mut() {
|
||||
first.thinking = thinking;
|
||||
}
|
||||
tool_results
|
||||
.iter()
|
||||
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
|
||||
Ok((text, tool_results))
|
||||
finish_completion(ctx, text, tool_calls, thinking).await
|
||||
}
|
||||
Err(err) => {
|
||||
if !silent && !text.is_empty() {
|
||||
@@ -610,6 +596,72 @@ pub async fn call_chat_completions_streaming(
|
||||
}
|
||||
}
|
||||
|
||||
/// Streaming transport for graph `llm` nodes: accumulates the reply and the
|
||||
/// transport itself renders nothing; no spinner, delta, or reasoning
|
||||
/// (tool-call rendering inside `eval_tool_calls` is unchanged from the
|
||||
/// non-streaming path). Because chunks arrive as they are generated, the
|
||||
/// reqwest `read_timeout` bounds only stalls between chunks; the caller's
|
||||
/// own deadline bounds the whole generation. Parity with the non-streaming
|
||||
/// transport is kept deliberately: the handler's call-loop detection is off
|
||||
/// (that path never had it), and a model the catalog marks `no_stream`
|
||||
/// takes the non-streaming request, raced against the abort. An abort
|
||||
/// observed after the provider call fails outright. Partial output is
|
||||
/// never returned as success.
|
||||
pub async fn call_chat_completions_streaming_quiet(
|
||||
input: &Input,
|
||||
client: &dyn Client,
|
||||
ctx: &mut RequestContext,
|
||||
abort_signal: AbortSignal,
|
||||
) -> Result<(String, Vec<ToolResult>)> {
|
||||
if client.model().no_stream() {
|
||||
let output = tokio::select! {
|
||||
ret = client.chat_completions(input.clone()) => ret?,
|
||||
_ = wait_abort_signal(&abort_signal) => bail!("Aborted."),
|
||||
};
|
||||
if abort_signal.aborted() {
|
||||
bail!("Aborted.");
|
||||
}
|
||||
let ChatCompletionsOutput {
|
||||
text,
|
||||
tool_calls,
|
||||
thinking,
|
||||
} = output;
|
||||
return finish_completion(ctx, text, tool_calls, thinking).await;
|
||||
}
|
||||
|
||||
// `_rx` outlives the call so `handler.done()` still finds a receiver.
|
||||
let (tx, _rx) = unbounded_channel();
|
||||
let mut handler = SseHandler::new(tx, abort_signal.clone());
|
||||
handler.set_silent(true);
|
||||
handler.set_call_loop_detection(false);
|
||||
|
||||
let send_ret = client.chat_completions_streaming(input, &mut handler).await;
|
||||
|
||||
if abort_signal.aborted() {
|
||||
bail!("Aborted.");
|
||||
}
|
||||
send_ret?;
|
||||
|
||||
let (text, tool_calls, thinking) = handler.take();
|
||||
finish_completion(ctx, text, tool_calls, thinking).await
|
||||
}
|
||||
|
||||
async fn finish_completion(
|
||||
ctx: &mut RequestContext,
|
||||
text: String,
|
||||
tool_calls: Vec<ToolCall>,
|
||||
thinking: Vec<ThinkingBlock>,
|
||||
) -> Result<(String, Vec<ToolResult>)> {
|
||||
let mut tool_results = eval_tool_calls(ctx, tool_calls).await?;
|
||||
if let Some(first) = tool_results.first_mut() {
|
||||
first.thinking = thinking;
|
||||
}
|
||||
tool_results
|
||||
.iter()
|
||||
.for_each(|res| ctx.tool_scope.tool_tracker.record_call(res.call.clone()));
|
||||
Ok((text, tool_results))
|
||||
}
|
||||
|
||||
pub fn noop_prepare_rerank<T>(_client: &T, _data: &RerankData) -> Result<RequestData> {
|
||||
bail!("The client doesn't support rerank api")
|
||||
}
|
||||
@@ -848,6 +900,10 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
use super::super::access_token::{is_rejected, set_access_token};
|
||||
use crate::config::{AppState, WorkingMode};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use tokio::time::Instant;
|
||||
|
||||
fn catch_error_message(data: &Value, status: u16) -> String {
|
||||
catch_error(data, status).unwrap_err().to_string()
|
||||
@@ -1010,4 +1066,283 @@ mod tests {
|
||||
assert!(!should_retry_auth(&api_status_error(401), client));
|
||||
assert!(!is_rejected(client, "at-1"));
|
||||
}
|
||||
|
||||
/// Fake provider whose streaming path emits `PACED_DELTAS` text chunks
|
||||
/// 60s apart, then one thinking block and one tool call, and whose
|
||||
/// non-streaming path returns that same reply in one piece when
|
||||
/// `nonstream_replies` is set (and bails otherwise). Exercises the real
|
||||
/// trait defaults (build_client, prepare_completion_data, abort
|
||||
/// `select!`) with no network.
|
||||
struct PacedStreamClient {
|
||||
config: AppConfig,
|
||||
model: Model,
|
||||
stall_forever: bool,
|
||||
nonstream_replies: bool,
|
||||
streaming_calls: AtomicUsize,
|
||||
}
|
||||
|
||||
const PACED_DELTAS: usize = 10;
|
||||
const PACED_GAP: Duration = Duration::from_secs(60);
|
||||
|
||||
fn paced_text() -> String {
|
||||
(0..PACED_DELTAS).map(|i| format!("chunk{i} ")).collect()
|
||||
}
|
||||
|
||||
fn paced_tool_call() -> ToolCall {
|
||||
ToolCall::new(
|
||||
"lookup".to_string(),
|
||||
json!({"q": 1}),
|
||||
Some("call-1".to_string()),
|
||||
)
|
||||
}
|
||||
|
||||
fn paced_thinking() -> ThinkingBlock {
|
||||
ThinkingBlock::Thinking {
|
||||
thinking: "hmm".to_string(),
|
||||
signature: "sig".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Client for PacedStreamClient {
|
||||
fn app_config(&self) -> &AppConfig {
|
||||
&self.config
|
||||
}
|
||||
|
||||
fn extra_config(&self) -> Option<&ExtraConfig> {
|
||||
None
|
||||
}
|
||||
|
||||
fn patch_config(&self) -> Option<&RequestPatch> {
|
||||
None
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
"paced-stream"
|
||||
}
|
||||
|
||||
fn model(&self) -> &Model {
|
||||
&self.model
|
||||
}
|
||||
|
||||
async fn chat_completions_inner(
|
||||
&self,
|
||||
_client: &ReqwestClient,
|
||||
data: ChatCompletionsData,
|
||||
) -> Result<ChatCompletionsOutput> {
|
||||
assert!(!data.stream, "non-streaming path must not request a stream");
|
||||
if !self.nonstream_replies {
|
||||
bail!("the quiet transport must not take the non-streaming path");
|
||||
}
|
||||
if self.stall_forever {
|
||||
std::future::pending::<()>().await;
|
||||
}
|
||||
Ok(ChatCompletionsOutput {
|
||||
text: paced_text(),
|
||||
tool_calls: vec![paced_tool_call()],
|
||||
thinking: vec![paced_thinking()],
|
||||
})
|
||||
}
|
||||
|
||||
async fn chat_completions_streaming_inner(
|
||||
&self,
|
||||
_client: &ReqwestClient,
|
||||
handler: &mut SseHandler,
|
||||
data: ChatCompletionsData,
|
||||
) -> Result<()> {
|
||||
self.streaming_calls.fetch_add(1, Ordering::SeqCst);
|
||||
assert!(data.stream, "trait default must request a stream");
|
||||
for i in 0..PACED_DELTAS {
|
||||
tokio::time::sleep(PACED_GAP).await;
|
||||
handler.text(&format!("chunk{i} "))?;
|
||||
}
|
||||
if self.stall_forever {
|
||||
std::future::pending::<()>().await;
|
||||
}
|
||||
handler.thinking_block(paced_thinking());
|
||||
handler.tool_call(paced_tool_call())
|
||||
}
|
||||
}
|
||||
|
||||
fn paced_client(stall_forever: bool) -> PacedStreamClient {
|
||||
PacedStreamClient {
|
||||
config: AppConfig::default(),
|
||||
model: Model::default(),
|
||||
stall_forever,
|
||||
nonstream_replies: false,
|
||||
streaming_calls: AtomicUsize::new(0),
|
||||
}
|
||||
}
|
||||
|
||||
fn no_stream_model() -> Model {
|
||||
let mut data = ModelData::new("");
|
||||
data.no_stream = true;
|
||||
Model::from_config("", &[data]).remove(0)
|
||||
}
|
||||
|
||||
fn assert_paced_reply(text: &str, results: &[ToolResult]) {
|
||||
assert_eq!(text, paced_text());
|
||||
assert_eq!(results.len(), 1);
|
||||
assert_eq!(results[0].call.name, "lookup");
|
||||
assert_eq!(results[0].call.id.as_deref(), Some("call-1"));
|
||||
assert!(
|
||||
matches!(&results[0].thinking[..], [ThinkingBlock::Thinking { thinking, .. }] if thinking == "hmm")
|
||||
);
|
||||
}
|
||||
|
||||
fn quiet_ctx() -> RequestContext {
|
||||
RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd)
|
||||
}
|
||||
|
||||
/// 600s of virtual generation completes and yields the same shape the
|
||||
/// non-streaming transport would: full text, tool results with the
|
||||
/// thinking stapled onto the first, and the tracker fed each call. The
|
||||
/// fake ignores the reqwest client, so reqwest's `read_timeout` is not in
|
||||
/// the loop here; the real-socket tests in graph/mod.rs pin that it
|
||||
/// fires on a stall (`sse_read_timeout_stall_is_transient`) and not on a
|
||||
/// slow but continuous stream (`sse_slow_but_continuous_stream_outlives_read_timeout`).
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn quiet_streaming_accumulates_across_a_long_generation() {
|
||||
let mut ctx = quiet_ctx();
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let client = paced_client(false);
|
||||
let started = Instant::now();
|
||||
|
||||
let (text, results) =
|
||||
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(started.elapsed() >= PACED_GAP * PACED_DELTAS as u32);
|
||||
assert_paced_reply(&text, &results);
|
||||
assert!(
|
||||
results[0].output["tool_call_error"].is_string(),
|
||||
"an undeclared tool evaluates to a tool_call_error, not a panic: {}",
|
||||
results[0].output
|
||||
);
|
||||
}
|
||||
|
||||
/// An abort mid-stream cancels the provider call promptly and fails the
|
||||
/// call outright; the ten chunks already buffered are never returned.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn quiet_streaming_abort_discards_partial_output() {
|
||||
let mut ctx = quiet_ctx();
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let client = paced_client(true);
|
||||
let abort = create_abort_signal();
|
||||
let trigger = abort.clone();
|
||||
tokio::spawn(async move {
|
||||
tokio::time::sleep(PACED_GAP * PACED_DELTAS as u32 + Duration::from_secs(30)).await;
|
||||
trigger.set_ctrlc();
|
||||
});
|
||||
let started = Instant::now();
|
||||
|
||||
let err = call_chat_completions_streaming_quiet(&input, &client, &mut ctx, abort)
|
||||
.await
|
||||
.expect_err("an aborted stream must not succeed");
|
||||
|
||||
assert_eq!(err.to_string(), "Aborted.");
|
||||
assert!(started.elapsed() < PACED_GAP * (PACED_DELTAS as u32 + 1));
|
||||
}
|
||||
|
||||
/// The quiet transport and the non-streaming transport hand the node the
|
||||
/// same `(text, tool_results)` for the same provider reply.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn quiet_streaming_matches_the_non_streaming_reply() {
|
||||
let mut ctx = quiet_ctx();
|
||||
ctx.render_mode = RenderMode::Silent;
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let mut client = paced_client(false);
|
||||
client.nonstream_replies = true;
|
||||
|
||||
let (streamed_text, streamed) =
|
||||
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
|
||||
.await
|
||||
.unwrap();
|
||||
let (plain_text, plain) = call_chat_completions(
|
||||
&input,
|
||||
false,
|
||||
false,
|
||||
&client,
|
||||
&mut ctx,
|
||||
create_abort_signal(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(streamed_text, plain_text);
|
||||
assert_paced_reply(&streamed_text, &streamed);
|
||||
assert_paced_reply(&plain_text, &plain);
|
||||
assert_eq!(streamed[0].output, plain[0].output);
|
||||
}
|
||||
|
||||
/// A model the catalog marks `no_stream` never sees a streaming request;
|
||||
/// the quiet transport takes the non-streaming path and returns the same
|
||||
/// shape.
|
||||
#[tokio::test]
|
||||
async fn quiet_transport_honours_no_stream_models() {
|
||||
let mut ctx = quiet_ctx();
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let mut client = paced_client(false);
|
||||
client.model = no_stream_model();
|
||||
client.nonstream_replies = true;
|
||||
|
||||
let (text, results) =
|
||||
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(client.streaming_calls.load(Ordering::SeqCst), 0);
|
||||
assert_paced_reply(&text, &results);
|
||||
}
|
||||
|
||||
/// The `no_stream` fallback is raced against the abort like the
|
||||
/// streaming path: a hung non-streaming request is cancelled promptly.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn quiet_transport_no_stream_fallback_honours_abort() {
|
||||
let mut ctx = quiet_ctx();
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let mut client = paced_client(true);
|
||||
client.model = no_stream_model();
|
||||
client.nonstream_replies = true;
|
||||
let abort = create_abort_signal();
|
||||
let trigger = abort.clone();
|
||||
tokio::spawn(async move {
|
||||
tokio::time::sleep(Duration::from_secs(5)).await;
|
||||
trigger.set_ctrlc();
|
||||
});
|
||||
let started = Instant::now();
|
||||
|
||||
let err = call_chat_completions_streaming_quiet(&input, &client, &mut ctx, abort)
|
||||
.await
|
||||
.expect_err("an aborted non-streaming fallback must not succeed");
|
||||
|
||||
assert_eq!(err.to_string(), "Aborted.");
|
||||
assert!(started.elapsed() < Duration::from_secs(10));
|
||||
}
|
||||
|
||||
/// The user-level `stream` toggle is a rendering preference; the quiet
|
||||
/// transport still streams with it off.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn quiet_transport_ignores_the_user_stream_toggle() {
|
||||
let app = AppState {
|
||||
config: Arc::new(AppConfig {
|
||||
stream: false,
|
||||
..AppConfig::default()
|
||||
}),
|
||||
..AppState::test_default()
|
||||
};
|
||||
let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd);
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
assert!(!input.stream(), "fixture must have streaming disabled");
|
||||
let client = paced_client(false);
|
||||
|
||||
let (text, results) =
|
||||
call_chat_completions_streaming_quiet(&input, &client, &mut ctx, create_abort_signal())
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(client.streaming_calls.load(Ordering::SeqCst), 1);
|
||||
assert_paced_reply(&text, &results);
|
||||
}
|
||||
}
|
||||
+89
-5
@@ -2,7 +2,7 @@ use super::{ApiStatusError, ThinkingBlock, ToolCall, catch_error};
|
||||
use crate::utils::AbortSignal;
|
||||
|
||||
use anyhow::{Context, Result, anyhow, bail};
|
||||
use eventsource_stream::Eventsource;
|
||||
use eventsource_stream::{EventStreamError, Eventsource};
|
||||
use futures_util::{Stream, StreamExt};
|
||||
use reqwest::{RequestBuilder, header};
|
||||
use serde_json::Value;
|
||||
@@ -18,6 +18,7 @@ pub struct SseHandler {
|
||||
max_call_repeats: usize,
|
||||
call_repeat_chain_len: usize,
|
||||
silent: bool,
|
||||
call_loop_detection: bool,
|
||||
}
|
||||
|
||||
impl SseHandler {
|
||||
@@ -32,6 +33,7 @@ impl SseHandler {
|
||||
max_call_repeats: 2,
|
||||
call_repeat_chain_len: 3,
|
||||
silent: false,
|
||||
call_loop_detection: true,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,6 +41,13 @@ impl SseHandler {
|
||||
self.silent = silent;
|
||||
}
|
||||
|
||||
/// Loop detection guards an interactive stream; the non-streaming
|
||||
/// transport never had it, so callers that must match that transport's
|
||||
/// tool-call handling turn it off.
|
||||
pub fn set_call_loop_detection(&mut self, enabled: bool) {
|
||||
self.call_loop_detection = enabled;
|
||||
}
|
||||
|
||||
pub fn text(&mut self, text: &str) -> Result<()> {
|
||||
if text.is_empty() {
|
||||
return Ok(());
|
||||
@@ -73,7 +82,7 @@ impl SseHandler {
|
||||
}
|
||||
|
||||
pub fn tool_call(&mut self, call: ToolCall) -> Result<()> {
|
||||
if self.is_call_loop(&call) {
|
||||
if self.call_loop_detection && self.is_call_loop(&call) {
|
||||
let loop_message = self.create_loop_detection_message(&call);
|
||||
return Err(anyhow!(loop_message));
|
||||
}
|
||||
@@ -273,6 +282,11 @@ where
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Keep the typed reqwest error in the chain: a read_timeout stall
|
||||
// is only recognised as transient via `reqwest::Error::is_timeout`.
|
||||
Err(EventStreamError::Transport(err)) => {
|
||||
return Err(err).context("Transport error");
|
||||
}
|
||||
Err(err) => {
|
||||
bail!("{err}");
|
||||
}
|
||||
@@ -285,13 +299,12 @@ pub async fn json_stream<S, F, E>(mut stream: S, mut handle: F) -> Result<()>
|
||||
where
|
||||
S: Stream<Item = Result<bytes::Bytes, E>> + Unpin,
|
||||
F: FnMut(&str) -> Result<()>,
|
||||
E: std::error::Error,
|
||||
E: std::error::Error + Send + Sync + 'static,
|
||||
{
|
||||
let mut parser = JsonStreamParser::default();
|
||||
let mut unparsed_bytes = vec![];
|
||||
while let Some(chunk_bytes) = stream.next().await {
|
||||
let chunk_bytes =
|
||||
chunk_bytes.map_err(|err| anyhow!("Failed to read json stream, {err}"))?;
|
||||
let chunk_bytes = chunk_bytes.context("Failed to read json stream")?;
|
||||
unparsed_bytes.extend(chunk_bytes);
|
||||
match std::str::from_utf8(&unparsed_bytes) {
|
||||
Ok(text) => {
|
||||
@@ -430,6 +443,40 @@ mod tests {
|
||||
assert!(error_message.contains("test_function_loop"));
|
||||
}
|
||||
|
||||
/// With detection off, repeated identical calls all accumulate — the
|
||||
/// non-streaming transport's behaviour, which the quiet transport mirrors.
|
||||
#[test]
|
||||
fn test_call_loop_detection_can_be_disabled() {
|
||||
let (mut handler, _rx) = new_handler();
|
||||
handler.set_call_loop_detection(false);
|
||||
let call = ToolCall::new("test_function_loop".to_string(), json!({"param": 1}), None);
|
||||
|
||||
for _ in 0..5 {
|
||||
handler.tool_call(call.clone()).unwrap();
|
||||
}
|
||||
|
||||
let (_, calls, _) = handler.take();
|
||||
assert_eq!(calls.len(), 5);
|
||||
}
|
||||
|
||||
/// A transport failure keeps its typed error in the chain so callers can
|
||||
/// classify it (a stall is only recognised as transient via the type).
|
||||
#[tokio::test]
|
||||
async fn json_stream_keeps_the_typed_transport_error() {
|
||||
let stalled = std::io::Error::new(std::io::ErrorKind::TimedOut, "stall");
|
||||
let mut source = stream::iter(vec![Err::<Bytes, std::io::Error>(stalled)]);
|
||||
|
||||
let err = json_stream(&mut source, |_| Ok(()))
|
||||
.await
|
||||
.expect_err("a failed read must fail the stream");
|
||||
|
||||
assert!(err.chain().any(|c| {
|
||||
c.downcast_ref::<std::io::Error>()
|
||||
.is_some_and(|e| e.kind() == std::io::ErrorKind::TimedOut)
|
||||
}));
|
||||
assert!(format!("{err:#}").starts_with("Failed to read json stream"));
|
||||
}
|
||||
|
||||
fn new_handler() -> (SseHandler, tokio::sync::mpsc::UnboundedReceiver<SseEvent>) {
|
||||
let (sender, receiver) = tokio::sync::mpsc::unbounded_channel();
|
||||
let abort_signal = crate::utils::create_abort_signal();
|
||||
@@ -467,6 +514,43 @@ mod tests {
|
||||
assert!(handler.has_received_content());
|
||||
}
|
||||
|
||||
/// A silent handler is the quiet transport's whole output gate: text,
|
||||
/// tool calls, and thinking still accumulate for `take()`, but the
|
||||
/// receiver sees no `Text` events, only the trailing `Done`.
|
||||
#[test]
|
||||
fn test_silent_handler_accumulates_without_emitting_text() {
|
||||
let (mut handler, mut rx) = new_handler();
|
||||
handler.set_silent(true);
|
||||
|
||||
handler.text("hel").unwrap();
|
||||
handler.text("lo").unwrap();
|
||||
handler.thinking_block(ThinkingBlock::Thinking {
|
||||
thinking: "hmm".to_string(),
|
||||
signature: "sig".to_string(),
|
||||
});
|
||||
handler
|
||||
.tool_call(ToolCall::new(
|
||||
"lookup".to_string(),
|
||||
json!({"q": 1}),
|
||||
Some("call-1".to_string()),
|
||||
))
|
||||
.unwrap();
|
||||
handler.done();
|
||||
|
||||
assert!(matches!(rx.try_recv(), Ok(SseEvent::Done)));
|
||||
assert!(rx.try_recv().is_err(), "no events beyond Done");
|
||||
|
||||
let (text, calls, thinking) = handler.take();
|
||||
assert_eq!(text, "hello");
|
||||
assert_eq!(calls.len(), 1);
|
||||
assert_eq!(calls[0].name, "lookup");
|
||||
assert_eq!(calls[0].id.as_deref(), Some("call-1"));
|
||||
assert_eq!(thinking.len(), 1);
|
||||
assert!(
|
||||
matches!(&thinking[0], ThinkingBlock::Thinking { thinking, .. } if thinking == "hmm")
|
||||
);
|
||||
}
|
||||
|
||||
fn split_chunks(text: &str) -> Vec<Vec<u8>> {
|
||||
let len = text.len();
|
||||
let cut1 = random_range(1..len - 1);
|
||||
|
||||
+12
-3
@@ -167,17 +167,26 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {"
|
||||
```
|
||||
# Create tasks with dependencies (optional: auto-dispatch with --agent)
|
||||
agent__task_create --subject \"Explore existing patterns\"
|
||||
agent__task_create --subject \"Implement feature\" --blocked_by [\"task_1\"] --agent coder --prompt \"Implement based on patterns found\"
|
||||
agent__task_create --subject \"Write tests\" --blocked_by [\"task_2\"]
|
||||
agent__task_create --subject \"Implement feature\" --blocked_by [\"1\"] --agent coder --prompt \"Implement based on patterns found\"
|
||||
agent__task_create --subject \"Write tests\" --blocked_by [\"2\"]
|
||||
|
||||
# Check what's runnable
|
||||
agent__task_list
|
||||
|
||||
# After completing a task, mark it done to unblock dependents
|
||||
# If dependents have --agent set, they auto-dispatch
|
||||
agent__task_complete --task_id task_1
|
||||
agent__task_complete --task_id 1
|
||||
```
|
||||
|
||||
Task-queue rules (NOT obvious — read carefully):
|
||||
- Task IDs are numeric strings (\"1\", \"2\", ...) assigned in creation order.
|
||||
- A task created with `agent` but NO blockers is NEVER auto-dispatched — spawn it yourself.
|
||||
- Auto-dispatch happens only inside `agent__task_complete`: completing a task is what
|
||||
dispatches its newly-unblocked dependents.
|
||||
- Auto-dispatched agents must still be collected via `agent__collect`, and an agent finishing
|
||||
does NOT complete its task — call `agent__task_complete` yourself to advance the chain.
|
||||
- A failed task permanently blocks its dependents — recreate the chain rather than retrying.
|
||||
|
||||
### Escalation Handling
|
||||
|
||||
Child agents may need user input but cannot prompt the user directly. When this happens,
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -918,42 +918,8 @@ mod tests {
|
||||
use super::*;
|
||||
use crate::config::mcp_tool_policy::LayerSource;
|
||||
use crate::function::ToolCall;
|
||||
use log::{Level, LevelFilter, Log, Metadata, Record};
|
||||
use crate::testing::{install_warn_collector, warn_messages};
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::sync::{Mutex, Once, OnceLock};
|
||||
|
||||
struct WarnCollector;
|
||||
|
||||
static WARN_MESSAGES: OnceLock<Mutex<Vec<String>>> = OnceLock::new();
|
||||
|
||||
fn warn_messages() -> &'static Mutex<Vec<String>> {
|
||||
WARN_MESSAGES.get_or_init(Mutex::default)
|
||||
}
|
||||
|
||||
impl Log for WarnCollector {
|
||||
fn enabled(&self, metadata: &Metadata) -> bool {
|
||||
metadata.level() <= Level::Warn
|
||||
}
|
||||
|
||||
fn log(&self, record: &Record) {
|
||||
if self.enabled(record.metadata()) {
|
||||
warn_messages()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(record.args().to_string());
|
||||
}
|
||||
}
|
||||
|
||||
fn flush(&self) {}
|
||||
}
|
||||
|
||||
fn install_warn_collector() {
|
||||
static INSTALL: Once = Once::new();
|
||||
INSTALL.call_once(|| {
|
||||
log::set_logger(&WarnCollector).expect("no other logger should be installed");
|
||||
log::set_max_level(LevelFilter::Warn);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mcp_runtime_new_is_empty() {
|
||||
|
||||
+818
-63
@@ -1,3 +1,4 @@
|
||||
use super::is_transient_error;
|
||||
use super::state::StateManager;
|
||||
use super::state_updates;
|
||||
use super::structured;
|
||||
@@ -5,78 +6,283 @@ use super::types::AgentNode;
|
||||
use super::wall_clock;
|
||||
use crate::config::RequestContext;
|
||||
use crate::function::agents::run_agent_for_graph;
|
||||
use anyhow::{Context, Result};
|
||||
use log::debug;
|
||||
use crate::supervisor::mailbox::{Inbox, PeerRegistry};
|
||||
use crate::utils::create_abort_signal;
|
||||
use anyhow::{Context, Error, Result, anyhow};
|
||||
use log::{debug, warn};
|
||||
use serde_json::Value;
|
||||
use std::collections::HashMap;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use tokio::time::timeout;
|
||||
|
||||
const DEFAULT_TIMEOUT_SECS: u64 = 300;
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub(super) enum AgentExecutionOutcome {
|
||||
Continue(String),
|
||||
FellBack(String),
|
||||
}
|
||||
|
||||
pub struct AgentNodeExecutor;
|
||||
|
||||
impl AgentNodeExecutor {
|
||||
pub async fn execute(
|
||||
pub(super) async fn execute(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
state_manager: &mut StateManager,
|
||||
parent_ctx: &mut RequestContext,
|
||||
retire_peer_on_return: bool,
|
||||
) -> Result<String> {
|
||||
let prompt = state_manager
|
||||
.interpolate(&node.prompt)
|
||||
.with_context(|| format!("Failed to interpolate prompt for agent '{}'", node.agent))?;
|
||||
|
||||
let graph_inputs = resolve_inputs(node, state_manager)?;
|
||||
if let Some(inputs) = &graph_inputs {
|
||||
let mut keys: Vec<&String> = inputs.keys().collect();
|
||||
keys.sort_unstable();
|
||||
debug!("Agent '{}' graph inputs: {keys:?}", node.agent);
|
||||
}
|
||||
|
||||
let secs = node.timeout.unwrap_or(DEFAULT_TIMEOUT_SECS);
|
||||
let bound = wall_clock(secs);
|
||||
|
||||
// run_agent_for_graph takes the identity off the ctx; keep a handle so
|
||||
// a frontier peer is retired the moment the agent stops, not after
|
||||
// extraction. Chains re-arm the same identity for later steps and
|
||||
// leave retirement to the chain runner.
|
||||
let peer = parent_ctx.peer_registry.clone().zip(
|
||||
parent_ctx
|
||||
.peer_assignment
|
||||
.as_ref()
|
||||
.map(|(id, _)| id.clone()),
|
||||
);
|
||||
|
||||
let fut = run_agent_for_graph(parent_ctx, &node.agent, &prompt, graph_inputs);
|
||||
let raw_result = match bound {
|
||||
Some(d) => timeout(d, fut).await,
|
||||
None => Ok(fut.await),
|
||||
};
|
||||
if retire_peer_on_return && let Some((registry, id)) = &peer {
|
||||
registry.mark_finished(id);
|
||||
}
|
||||
let raw = raw_result
|
||||
.with_context(|| format!("Agent '{}' timed out after {}s", node.agent, secs))?
|
||||
.with_context(|| format!("Agent '{}' failed", node.agent))?;
|
||||
|
||||
let output_value = match &node.output_schema {
|
||||
Some(schema) => structured::extract(&raw, schema, parent_ctx)
|
||||
.await
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"Agent '{}' output failed structured-output extraction",
|
||||
node.agent
|
||||
)
|
||||
})?,
|
||||
None => Value::String(raw.clone()),
|
||||
};
|
||||
|
||||
apply_state_updates(node, state_manager, &output_value);
|
||||
|
||||
Ok(raw)
|
||||
) -> Result<AgentExecutionOutcome> {
|
||||
let result = run(
|
||||
node_id,
|
||||
node,
|
||||
state_manager,
|
||||
parent_ctx,
|
||||
retire_peer_on_return,
|
||||
)
|
||||
.await;
|
||||
outcome_from(node_id, node, state_manager, result)
|
||||
}
|
||||
}
|
||||
|
||||
/// Turns the node run's final result into a routing outcome. With a
|
||||
/// `fallback` declared, any failure (e.g. retries exhausted, hard error, or
|
||||
/// extraction failure) is written into state as
|
||||
/// `"Agent node failed: <chain>"` and routes to the fallback. Only the
|
||||
/// node's `state_updates` can capture that string (bound as `{{output}}`
|
||||
/// during interpolation); the output-schema auto-merge never applies to it,
|
||||
/// because it only merges object values and the failure is a plain string.
|
||||
/// Without a fallback the error propagates unchanged; unlike llm nodes
|
||||
/// there is no teaching bail, because agent nodes always failed loudly and
|
||||
/// callers assert on the existing "Agent 'X' failed" chains.
|
||||
fn outcome_from(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
state_manager: &mut StateManager,
|
||||
result: Result<String>,
|
||||
) -> Result<AgentExecutionOutcome> {
|
||||
match result {
|
||||
Ok(raw) => Ok(AgentExecutionOutcome::Continue(raw)),
|
||||
Err(e) => match &node.fallback {
|
||||
Some(fb) => {
|
||||
warn!("agent node '{node_id}' failed, routing to fallback '{fb}': {e:#}");
|
||||
state_updates::apply(
|
||||
state_manager,
|
||||
&Value::String(format!("Agent node failed: {e:#}")),
|
||||
node.output_schema.as_ref(),
|
||||
node.state_updates.as_ref(),
|
||||
);
|
||||
Ok(AgentExecutionOutcome::FellBack(fb.clone()))
|
||||
}
|
||||
None => Err(e),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
async fn run(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
state_manager: &mut StateManager,
|
||||
parent_ctx: &mut RequestContext,
|
||||
retire_peer_on_return: bool,
|
||||
) -> Result<String> {
|
||||
let prompt = state_manager
|
||||
.interpolate(&node.prompt)
|
||||
.with_context(|| format!("Failed to interpolate prompt for agent '{}'", node.agent))?;
|
||||
|
||||
let graph_inputs = resolve_inputs(node, state_manager)?;
|
||||
if let Some(inputs) = &graph_inputs {
|
||||
let mut keys: Vec<&String> = inputs.keys().collect();
|
||||
keys.sort_unstable();
|
||||
debug!("Agent '{}' graph inputs: {keys:?}", node.agent);
|
||||
}
|
||||
|
||||
let secs = node.timeout.unwrap_or(DEFAULT_TIMEOUT_SECS);
|
||||
let agent_name = node.agent.clone();
|
||||
|
||||
let mut run_attempt = attempt_runner(move |ctx| {
|
||||
let agent_name = agent_name.clone();
|
||||
let prompt = prompt.clone();
|
||||
let graph_inputs = graph_inputs.clone();
|
||||
boxed_attempt(async move {
|
||||
bounded_attempt(
|
||||
&agent_name,
|
||||
secs,
|
||||
run_agent_for_graph(ctx, &agent_name, &prompt, graph_inputs),
|
||||
)
|
||||
.await
|
||||
})
|
||||
});
|
||||
attempt_and_extract(
|
||||
node_id,
|
||||
node,
|
||||
state_manager,
|
||||
parent_ctx,
|
||||
retire_peer_on_return,
|
||||
&mut run_attempt,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Drives the retry loop, then extraction, then state updates. Extraction
|
||||
/// stays outside the retry loop: by the time it runs, the agent itself
|
||||
/// succeeded, so an extraction failure is never retried.
|
||||
async fn attempt_and_extract(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
state_manager: &mut StateManager,
|
||||
parent_ctx: &mut RequestContext,
|
||||
retire_peer_on_return: bool,
|
||||
run_attempt: &mut AttemptRunner<'_>,
|
||||
) -> Result<String> {
|
||||
let raw = run_with_retries(
|
||||
node_id,
|
||||
node,
|
||||
parent_ctx,
|
||||
retire_peer_on_return,
|
||||
run_attempt,
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Agent nodes carry no graph abort; the session signal is the interrupt
|
||||
// the executor bridges onto the graph abort anyway.
|
||||
let abort = parent_ctx
|
||||
.session_abort
|
||||
.clone()
|
||||
.unwrap_or_else(create_abort_signal);
|
||||
let output_value = match &node.output_schema {
|
||||
Some(schema) => structured::extract(&raw, schema, parent_ctx, &abort)
|
||||
.await
|
||||
.with_context(|| {
|
||||
format!(
|
||||
"Agent '{}' output failed structured-output extraction",
|
||||
node.agent
|
||||
)
|
||||
})?,
|
||||
None => Value::String(raw.clone()),
|
||||
};
|
||||
|
||||
apply_state_updates(node, state_manager, &output_value);
|
||||
|
||||
Ok(raw)
|
||||
}
|
||||
|
||||
/// One boxed attempt against the ctx. Boxed (rather than an opaque
|
||||
/// `AsyncFnMut` future) because agent nodes run inside `tokio::spawn`ed map
|
||||
/// branches, where higher-ranked opaque futures trip the `Send` auto-trait
|
||||
/// solver ("implementation of `Send` is not general enough").
|
||||
type AttemptFuture<'a> = Pin<Box<dyn Future<Output = Result<String>> + Send + 'a>>;
|
||||
|
||||
/// The per-attempt runner the retry loop drives. The attempt future may only
|
||||
/// borrow the ctx it is handed, so runners move owned clones of everything
|
||||
/// else into the future.
|
||||
type AttemptRunner<'f> = dyn for<'a> FnMut(&'a mut RequestContext) -> AttemptFuture<'a> + Send + 'f;
|
||||
|
||||
/// Identity funnel that pins a closure to the runner's higher-ranked
|
||||
/// signature, so inline closures at call sites infer the right lifetimes.
|
||||
fn attempt_runner<F>(f: F) -> F
|
||||
where
|
||||
F: for<'a> FnMut(&'a mut RequestContext) -> AttemptFuture<'a> + Send,
|
||||
{
|
||||
f
|
||||
}
|
||||
|
||||
fn boxed_attempt<'a>(fut: impl Future<Output = Result<String>> + Send + 'a) -> AttemptFuture<'a> {
|
||||
Box::pin(fut)
|
||||
}
|
||||
|
||||
/// Runs the node's agent up to `node.max_attempts` times, retrying only
|
||||
/// failures that `is_transient_error` recognizes.
|
||||
///
|
||||
/// `run_agent_for_graph` takes the peer identity off the ctx and never
|
||||
/// restores it, so BOTH halves, the (id, inbox) assignment and the registry,
|
||||
/// are captured up front and re-armed before every retry; re-arming only
|
||||
/// the assignment would give a retried attempt its identity back but no
|
||||
/// roster and a broken `agent__send_message`. A frontier peer is retired
|
||||
/// exactly once, after the final attempt, the moment the agent stops, not
|
||||
/// after extraction. Chains re-arm the same identity for later steps and
|
||||
/// leave retirement to the chain runner.
|
||||
async fn run_with_retries(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
parent_ctx: &mut RequestContext,
|
||||
retire_peer_on_return: bool,
|
||||
run_attempt: &mut AttemptRunner<'_>,
|
||||
) -> Result<String> {
|
||||
let assignment = parent_ctx.peer_assignment.clone();
|
||||
let registry = parent_ctx.peer_registry.clone();
|
||||
|
||||
let result = retry_transient(
|
||||
node_id,
|
||||
node,
|
||||
parent_ctx,
|
||||
&assignment,
|
||||
®istry,
|
||||
run_attempt,
|
||||
)
|
||||
.await;
|
||||
|
||||
if retire_peer_on_return && let (Some(registry), Some((id, _))) = (®istry, &assignment) {
|
||||
registry.mark_finished(id);
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
async fn retry_transient(
|
||||
node_id: &str,
|
||||
node: &AgentNode,
|
||||
parent_ctx: &mut RequestContext,
|
||||
assignment: &Option<(String, Arc<Inbox>)>,
|
||||
registry: &Option<Arc<PeerRegistry>>,
|
||||
run_attempt: &mut AttemptRunner<'_>,
|
||||
) -> Result<String> {
|
||||
let mut last_err: Option<Error> = None;
|
||||
for attempt in 1..=node.max_attempts {
|
||||
if attempt > 1 {
|
||||
if let Some(assignment) = assignment {
|
||||
parent_ctx.peer_assignment = Some(assignment.clone());
|
||||
}
|
||||
if let Some(registry) = registry {
|
||||
parent_ctx.peer_registry = Some(Arc::clone(registry));
|
||||
}
|
||||
}
|
||||
|
||||
match run_attempt(parent_ctx).await {
|
||||
Ok(out) => return Ok(out),
|
||||
Err(e) if is_transient_error(&e) && attempt < node.max_attempts => {
|
||||
warn!(
|
||||
"agent node '{node_id}' attempt {attempt} failed (transient): {e:#}; retrying"
|
||||
);
|
||||
last_err = Some(e);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
Err(last_err.unwrap_or_else(|| anyhow!("agent node exhausted retries")))
|
||||
}
|
||||
|
||||
/// Applies the wall-clock bound and the human-readable failure contexts to a
|
||||
/// single attempt. The contexts land here, before the caller inspects the
|
||||
/// error, because `tokio::time::error::Elapsed` renders as "deadline has
|
||||
/// elapsed", which the transient matcher would not recognize as a timeout.
|
||||
async fn bounded_attempt(
|
||||
agent_name: &str,
|
||||
secs: u64,
|
||||
fut: impl Future<Output = Result<String>>,
|
||||
) -> Result<String> {
|
||||
let raw_result = match wall_clock(secs) {
|
||||
Some(d) => timeout(d, fut).await,
|
||||
None => Ok(fut.await),
|
||||
};
|
||||
raw_result
|
||||
.with_context(|| format!("Agent '{agent_name}' timed out after {secs}s"))?
|
||||
.with_context(|| format!("Agent '{agent_name}' failed"))
|
||||
}
|
||||
|
||||
/// Resolves each `inputs` template against the parent state. A lone
|
||||
/// `{{key}}` yields the state value as-is (numbers, arrays, objects and
|
||||
/// `null` all survive); anything else renders to a string. Every key is
|
||||
@@ -105,7 +311,7 @@ fn apply_state_updates(node: &AgentNode, state_manager: &mut StateManager, outpu
|
||||
state_updates::apply(
|
||||
state_manager,
|
||||
output,
|
||||
node.output_schema.is_some(),
|
||||
node.output_schema.as_ref(),
|
||||
node.state_updates.as_ref(),
|
||||
);
|
||||
}
|
||||
@@ -116,6 +322,7 @@ mod tests {
|
||||
use super::*;
|
||||
use crate::config::{AppState, WorkingMode, default_max_agent_depth};
|
||||
use crate::supervisor::mailbox::{Inbox, PeerRegistry, graph_agent_id};
|
||||
use crate::testing::{install_warn_collector, warn_messages};
|
||||
use serde_json::json;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
@@ -135,6 +342,8 @@ mod tests {
|
||||
state_updates: updates,
|
||||
output_schema: None,
|
||||
timeout: None,
|
||||
max_attempts: 1,
|
||||
fallback: None,
|
||||
inputs: None,
|
||||
teammates: false,
|
||||
}
|
||||
@@ -157,9 +366,10 @@ mod tests {
|
||||
node.teammates = true;
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let err = AgentNodeExecutor::execute(&node, &mut state, ctx, retire_peer_on_return)
|
||||
.await
|
||||
.expect_err("agent past max depth should fail before running");
|
||||
let err =
|
||||
AgentNodeExecutor::execute("test_node", &node, &mut state, ctx, retire_peer_on_return)
|
||||
.await
|
||||
.expect_err("agent past max depth should fail before running");
|
||||
|
||||
let chain = format!("{err:#}");
|
||||
assert!(chain.contains("Agent 'test_agent' failed"), "{chain}");
|
||||
@@ -184,6 +394,539 @@ mod tests {
|
||||
assert!(!registry.is_finished(&id));
|
||||
}
|
||||
|
||||
fn plain_ctx() -> RequestContext {
|
||||
RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd)
|
||||
}
|
||||
|
||||
fn retryable_node(max_attempts: u32) -> AgentNode {
|
||||
let mut node = node_with("hi", None);
|
||||
node.max_attempts = max_attempts;
|
||||
node
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_retries_transient_failure_and_succeeds() {
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(2);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let out = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
let result = if attempts == 1 {
|
||||
Err(anyhow!("error sending request for url"))
|
||||
} else {
|
||||
Ok("recovered".to_string())
|
||||
};
|
||||
boxed_attempt(async move { result })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(out, "recovered");
|
||||
assert_eq!(attempts, 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_does_not_retry_non_transient_failures() {
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(3);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let err = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(async move { Err::<String, _>(anyhow!("Unknown model 'foo'")) })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("non-transient failure must propagate immediately");
|
||||
|
||||
assert_eq!(attempts, 1);
|
||||
assert!(format!("{err:#}").contains("Unknown model 'foo'"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_stops_at_max_attempts_and_keeps_failure_context() {
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(2);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let err = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(bounded_attempt("test_agent", 0, async {
|
||||
Err(anyhow!("connection error: unexpected end of stream"))
|
||||
}))
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("exhausted retries must propagate the last error");
|
||||
|
||||
assert_eq!(attempts, 2);
|
||||
let chain = format!("{err:#}");
|
||||
assert!(chain.contains("Agent 'test_agent' failed"), "{chain}");
|
||||
assert!(chain.contains("connection error"), "{chain}");
|
||||
}
|
||||
|
||||
/// Attempt 2 is bounded by the same 1s wall clock as attempt 1 and
|
||||
/// sleeps for most of it; it can only succeed if the retry owns a fresh
|
||||
/// budget rather than the deadline attempt 1 already exhausted.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn run_with_retries_gives_each_attempt_a_fresh_timeout_budget() {
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(2);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let out = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
if attempts == 1 {
|
||||
boxed_attempt(bounded_attempt(
|
||||
"test_agent",
|
||||
1,
|
||||
std::future::pending::<Result<String>>(),
|
||||
))
|
||||
} else {
|
||||
boxed_attempt(bounded_attempt("test_agent", 1, async move {
|
||||
tokio::time::sleep(std::time::Duration::from_millis(900)).await;
|
||||
Ok("recovered".to_string())
|
||||
}))
|
||||
}
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(out, "recovered");
|
||||
assert_eq!(attempts, 2);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn bounded_attempt_timeout_context_is_transient() {
|
||||
let err = bounded_attempt("test_agent", 1, std::future::pending::<Result<String>>())
|
||||
.await
|
||||
.expect_err("pending future must hit the wall clock");
|
||||
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains("Agent 'test_agent' timed out after 1s"),
|
||||
"{chain}"
|
||||
);
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_rearms_the_same_peer_identity_for_each_retry() {
|
||||
let (mut ctx, registry, id) = ctx_at_max_depth_with_peer();
|
||||
let expected_inbox = Arc::clone(&ctx.peer_assignment.as_ref().unwrap().1);
|
||||
let node = retryable_node(2);
|
||||
let mut attempts = 0u32;
|
||||
let mut observations: Vec<(String, bool, bool)> = Vec::new();
|
||||
|
||||
let out = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
true,
|
||||
&mut attempt_runner(|ctx| {
|
||||
attempts += 1;
|
||||
// Mirror run_agent_for_graph: consume both identity halves.
|
||||
let (peer_id, inbox) = ctx.peer_assignment.take().expect("assignment armed");
|
||||
let reg = ctx.peer_registry.take().expect("registry armed");
|
||||
observations.push((
|
||||
peer_id,
|
||||
Arc::ptr_eq(&inbox, &expected_inbox),
|
||||
Arc::ptr_eq(®, ®istry),
|
||||
));
|
||||
assert!(
|
||||
!registry.is_finished(&id),
|
||||
"peer must not be retired between attempts"
|
||||
);
|
||||
let result = if attempts == 1 {
|
||||
Err(anyhow!("stream closed because of a broken pipe"))
|
||||
} else {
|
||||
Ok("done".to_string())
|
||||
};
|
||||
boxed_attempt(async move { result })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(out, "done");
|
||||
assert_eq!(observations.len(), 2);
|
||||
for (peer_id, same_inbox, same_registry) in &observations {
|
||||
assert_eq!(peer_id, &id);
|
||||
assert!(same_inbox, "retry must reuse the same inbox Arc");
|
||||
assert!(same_registry, "retry must reuse the same registry Arc");
|
||||
}
|
||||
assert!(registry.is_finished(&id));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_retires_frontier_peer_after_the_final_failed_attempt() {
|
||||
let (mut ctx, registry, id) = ctx_at_max_depth_with_peer();
|
||||
let node = retryable_node(2);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let err = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
true,
|
||||
&mut attempt_runner(|ctx| {
|
||||
attempts += 1;
|
||||
ctx.peer_assignment.take();
|
||||
ctx.peer_registry.take();
|
||||
assert!(
|
||||
!registry.is_finished(&id),
|
||||
"peer must not be retired between attempts"
|
||||
);
|
||||
boxed_attempt(async move { Err::<String, _>(anyhow!("Connection reset by peer")) })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("both attempts fail");
|
||||
|
||||
assert_eq!(attempts, 2);
|
||||
assert!(registry.is_finished(&id));
|
||||
assert!(format!("{err:#}").contains("Connection reset"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_leaves_peer_live_when_not_retiring() {
|
||||
let (mut ctx, registry, id) = ctx_at_max_depth_with_peer();
|
||||
let node = retryable_node(1);
|
||||
|
||||
run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|ctx| {
|
||||
ctx.peer_assignment.take();
|
||||
ctx.peer_registry.take();
|
||||
boxed_attempt(async move { Ok("done".to_string()) })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(!registry.is_finished(&id));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn run_with_retries_zero_attempts_reports_exhausted_retries() {
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(0);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let err = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(async move { Ok("never".to_string()) })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("zero attempts cannot succeed");
|
||||
|
||||
assert_eq!(attempts, 0);
|
||||
assert!(format!("{err:#}").contains("agent node exhausted retries"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn retry_transient_warns_once_per_retried_attempt_with_the_context_chain() {
|
||||
install_warn_collector();
|
||||
let mut ctx = plain_ctx();
|
||||
let node = retryable_node(3);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
run_with_retries(
|
||||
"warn_capture",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(async move {
|
||||
Err::<String, _>(
|
||||
anyhow!("connection error: warn-capture reset")
|
||||
.context("Agent 'test_agent' failed"),
|
||||
)
|
||||
})
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("all attempts fail");
|
||||
|
||||
assert_eq!(attempts, 3);
|
||||
let warns: Vec<String> = warn_messages()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|m| m.contains("'warn_capture'"))
|
||||
.cloned()
|
||||
.collect();
|
||||
assert_eq!(
|
||||
warns.len(),
|
||||
2,
|
||||
"one warn per retried attempt (the final attempt propagates instead): {warns:?}"
|
||||
);
|
||||
for (i, warn) in warns.iter().enumerate() {
|
||||
let attempt = i + 1;
|
||||
assert!(
|
||||
warn.contains(&format!(
|
||||
"agent node 'warn_capture' attempt {attempt} failed (transient)"
|
||||
)),
|
||||
"{warn}"
|
||||
);
|
||||
assert!(
|
||||
warn.contains("Agent 'test_agent' failed: connection error: warn-capture reset"),
|
||||
"warn must carry the full context chain: {warn}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn extraction_failure_after_a_successful_run_is_never_retried() {
|
||||
let mut ctx = plain_ctx();
|
||||
let mut node = retryable_node(2);
|
||||
node.output_schema = Some(json!({"type": "object"}));
|
||||
let mut state = manager_with(&[]);
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let err = attempt_and_extract(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut state,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(async move { Ok("not json".to_string()) })
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("extraction must fail without an extractor model");
|
||||
|
||||
assert_eq!(
|
||||
attempts, 1,
|
||||
"an extraction failure must not re-run the agent"
|
||||
);
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains("output failed structured-output extraction"),
|
||||
"{chain}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Agent-node extraction runs on the session abort: a fired signal
|
||||
/// stops the extractor before it issues a request.
|
||||
#[tokio::test]
|
||||
async fn extraction_honours_a_fired_session_abort() {
|
||||
let mut ctx = plain_ctx();
|
||||
let session = create_abort_signal();
|
||||
session.set_ctrlc();
|
||||
ctx.session_abort = Some(session);
|
||||
let mut node = retryable_node(1);
|
||||
node.output_schema = Some(json!({"type": "object"}));
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let err = attempt_and_extract(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut state,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| boxed_attempt(async move { Ok("not json".to_string()) })),
|
||||
)
|
||||
.await
|
||||
.expect_err("an aborted session must stop the extractor");
|
||||
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains("output failed structured-output extraction"),
|
||||
"{chain}"
|
||||
);
|
||||
assert!(chain.contains("Aborted."), "{chain}");
|
||||
assert!(!chain.contains("Invalid model"), "{chain}");
|
||||
}
|
||||
|
||||
fn failure_capture_updates() -> HashMap<String, String> {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("failure".into(), "{{output}}".into());
|
||||
u
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn execute_failure_with_fallback_routes_and_records_the_failure() {
|
||||
let mut ctx = plain_ctx();
|
||||
ctx.current_depth = default_max_agent_depth();
|
||||
let mut node = node_with("hi", Some(failure_capture_updates()));
|
||||
node.fallback = Some("recover".into());
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let outcome = AgentNodeExecutor::execute("test_node", &node, &mut state, &mut ctx, false)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(outcome, AgentExecutionOutcome::FellBack("recover".into()));
|
||||
let failure = state
|
||||
.state()
|
||||
.get("failure")
|
||||
.and_then(Value::as_str)
|
||||
.expect("failure text must land in state for the fallback node")
|
||||
.to_string();
|
||||
assert!(failure.starts_with("Agent node failed: "), "{failure}");
|
||||
assert!(failure.contains("Agent 'test_agent' failed"), "{failure}");
|
||||
assert!(failure.contains("Max agent depth exceeded"), "{failure}");
|
||||
}
|
||||
|
||||
fn extraction_error() -> Error {
|
||||
anyhow!("no JSON object found in output")
|
||||
.context("Agent 'test_agent' output failed structured-output extraction")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn outcome_from_success_is_continue() {
|
||||
let node = node_with("hi", None);
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let outcome = outcome_from("test_node", &node, &mut state, Ok("raw".into())).unwrap();
|
||||
|
||||
assert_eq!(outcome, AgentExecutionOutcome::Continue("raw".into()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn outcome_from_extraction_failure_with_fallback_falls_back() {
|
||||
let mut node = node_with("hi", Some(failure_capture_updates()));
|
||||
node.fallback = Some("recover".into());
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let outcome =
|
||||
outcome_from("test_node", &node, &mut state, Err(extraction_error())).unwrap();
|
||||
|
||||
assert_eq!(outcome, AgentExecutionOutcome::FellBack("recover".into()));
|
||||
let failure = state
|
||||
.state()
|
||||
.get("failure")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap();
|
||||
assert!(
|
||||
failure.contains("structured-output extraction"),
|
||||
"{failure}"
|
||||
);
|
||||
assert!(failure.contains("no JSON object found"), "{failure}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn outcome_from_fallback_warn_names_the_node_and_the_fallback_target() {
|
||||
install_warn_collector();
|
||||
let mut node = node_with("hi", None);
|
||||
node.fallback = Some("recover".into());
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let outcome = outcome_from(
|
||||
"fallback_warn_capture",
|
||||
&node,
|
||||
&mut state,
|
||||
Err(extraction_error()),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(outcome, AgentExecutionOutcome::FellBack("recover".into()));
|
||||
let warns: Vec<String> = warn_messages()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|m| m.contains("'fallback_warn_capture'"))
|
||||
.cloned()
|
||||
.collect();
|
||||
assert_eq!(warns.len(), 1, "exactly one fallback warn: {warns:?}");
|
||||
assert!(
|
||||
warns[0].contains(
|
||||
"agent node 'fallback_warn_capture' failed, routing to fallback 'recover'"
|
||||
),
|
||||
"{}",
|
||||
warns[0]
|
||||
);
|
||||
assert!(
|
||||
warns[0].contains("Agent 'test_agent' output failed structured-output extraction"),
|
||||
"warn must carry the error chain: {}",
|
||||
warns[0]
|
||||
);
|
||||
assert!(
|
||||
warns[0].contains("no JSON object found in output"),
|
||||
"fallback warn must carry the full error chain: {}",
|
||||
warns[0]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn outcome_from_failure_without_fallback_propagates_the_error_unchanged() {
|
||||
let node = node_with("hi", Some(failure_capture_updates()));
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
let err = outcome_from("test_node", &node, &mut state, Err(extraction_error()))
|
||||
.expect_err("no fallback must propagate");
|
||||
|
||||
assert_eq!(
|
||||
format!("{err:#}"),
|
||||
"Agent 'test_agent' output failed structured-output extraction: \
|
||||
no JSON object found in output"
|
||||
);
|
||||
assert!(state.state().get("failure").is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fallback_engages_only_after_transient_retries_exhaust() {
|
||||
let mut ctx = plain_ctx();
|
||||
let mut node = retryable_node(2);
|
||||
node.fallback = Some("recover".into());
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let result = run_with_retries(
|
||||
"test_node",
|
||||
&node,
|
||||
&mut ctx,
|
||||
false,
|
||||
&mut attempt_runner(|_ctx| {
|
||||
attempts += 1;
|
||||
boxed_attempt(async move {
|
||||
Err::<String, _>(anyhow!("connection error: reset by peer"))
|
||||
})
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
let mut state = manager_with(&[]);
|
||||
let outcome = outcome_from("test_node", &node, &mut state, result).unwrap();
|
||||
|
||||
assert_eq!(attempts, 2, "fallback must not preempt the retry budget");
|
||||
assert_eq!(outcome, AgentExecutionOutcome::FellBack("recover".into()));
|
||||
}
|
||||
|
||||
fn node_with_inputs(pairs: &[(&str, &str)]) -> AgentNode {
|
||||
let mut node = node_with("hi", None);
|
||||
node.inputs = Some(
|
||||
@@ -349,7 +1092,11 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn output_schema_auto_merges_top_level_keys() {
|
||||
let node = node_with_schema("hi", None, json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
"hi",
|
||||
None,
|
||||
json!({"type": "object", "properties": {"goal": {}, "summary": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({"goal": "do X", "summary": "details"});
|
||||
|
||||
@@ -361,7 +1108,11 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn output_schema_preserves_nested_value_types() {
|
||||
let node = node_with_schema("hi", None, json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
"hi",
|
||||
None,
|
||||
json!({"type": "object", "properties": {"tags": {}, "config": {}, "count": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({
|
||||
"tags": ["a", "b"],
|
||||
@@ -380,7 +1131,11 @@ mod tests {
|
||||
fn output_schema_explicit_state_updates_override_auto_merge() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("goal".into(), "renamed-{{output.goal}}".into());
|
||||
let node = node_with_schema("hi", Some(u), json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
"hi",
|
||||
Some(u),
|
||||
json!({"type": "object", "properties": {"goal": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({"goal": "do X"});
|
||||
|
||||
|
||||
+65
-3
@@ -1,4 +1,4 @@
|
||||
use super::agent::AgentNodeExecutor;
|
||||
use super::agent::{AgentExecutionOutcome, AgentNodeExecutor};
|
||||
use super::llm::{LlmExecutionOutcome, LlmNodeExecutor};
|
||||
use super::logging::{GraphLogger, narrate_node_complete, narrate_node_failed};
|
||||
use super::map::MapNodeExecutor;
|
||||
@@ -574,8 +574,15 @@ pub(super) async fn step(
|
||||
) -> Result<StepResult> {
|
||||
match &node.node_type {
|
||||
NodeType::Agent(agent_node) => {
|
||||
AgentNodeExecutor::execute(agent_node, state, ctx, !step_ctx.branch_mode).await?;
|
||||
let targets = static_next_targets(node, current, "agent", step_ctx.branch_mode)?;
|
||||
let outcome =
|
||||
AgentNodeExecutor::execute(current, agent_node, state, ctx, !step_ctx.branch_mode)
|
||||
.await?;
|
||||
let targets = match outcome {
|
||||
AgentExecutionOutcome::Continue(_) => {
|
||||
static_next_targets(node, current, "agent", step_ctx.branch_mode)?
|
||||
}
|
||||
AgentExecutionOutcome::FellBack(target) => vec![target],
|
||||
};
|
||||
Ok(StepResult::Continue(targets))
|
||||
}
|
||||
NodeType::Script(script_node) => {
|
||||
@@ -817,6 +824,59 @@ mod tests {
|
||||
assert!(!state.contains_key("foo"));
|
||||
}
|
||||
|
||||
// The dispatcher seeds a child's `inputs:` before the executor seeds
|
||||
// variables, so a structured input outranks both a spawn-time override
|
||||
// and the declared default — the contract the review-gauntlet →
|
||||
// adversary `verification_commands` passthrough relies on.
|
||||
fn seeded_after_dispatch(
|
||||
inputs: Option<HashMap<String, Value>>,
|
||||
agent_vars: Option<&AgentVariables>,
|
||||
) -> HashMap<String, Value> {
|
||||
let mut state = HashMap::new();
|
||||
crate::graph::dispatch::seed_initial_state(&mut state, "prompt", inputs);
|
||||
seed_variables(&mut state, &[variable("foo", Some("default"))], agent_vars);
|
||||
state
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_inputs_win_over_variable_override_and_default() {
|
||||
let agent_vars: AgentVariables =
|
||||
IndexMap::from([("foo".to_string(), "override".to_string())]);
|
||||
let inputs = HashMap::from([("foo".to_string(), json!("from-inputs"))]);
|
||||
|
||||
let state = seeded_after_dispatch(Some(inputs), Some(&agent_vars));
|
||||
|
||||
assert_eq!(state.get("foo"), Some(&json!("from-inputs")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_without_inputs_lets_variable_override_beat_default() {
|
||||
let agent_vars: AgentVariables =
|
||||
IndexMap::from([("foo".to_string(), "override".to_string())]);
|
||||
|
||||
let state = seeded_after_dispatch(None, Some(&agent_vars));
|
||||
|
||||
assert_eq!(state.get("foo"), Some(&json!("override")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_without_inputs_or_override_lands_declared_default() {
|
||||
let state = seeded_after_dispatch(None, None);
|
||||
|
||||
assert_eq!(state.get("foo"), Some(&json!("default")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dispatch_lone_array_input_survives_variable_seeding_as_array() {
|
||||
let agent_vars: AgentVariables =
|
||||
IndexMap::from([("foo".to_string(), "override".to_string())]);
|
||||
let inputs = HashMap::from([("foo".to_string(), json!(["echo a", "echo b"]))]);
|
||||
|
||||
let state = seeded_after_dispatch(Some(inputs), Some(&agent_vars));
|
||||
|
||||
assert_eq!(state.get("foo"), Some(&json!(["echo a", "echo b"])));
|
||||
}
|
||||
|
||||
fn agent_node(id: &str, teammates: bool) -> Node {
|
||||
Node {
|
||||
id: id.into(),
|
||||
@@ -827,6 +887,8 @@ mod tests {
|
||||
state_updates: None,
|
||||
output_schema: None,
|
||||
timeout: None,
|
||||
max_attempts: 1,
|
||||
fallback: None,
|
||||
inputs: None,
|
||||
teammates,
|
||||
}),
|
||||
|
||||
+478
-76
@@ -2,12 +2,13 @@ use super::state::StateManager;
|
||||
use super::state_updates;
|
||||
use super::structured;
|
||||
use super::types::LlmNode;
|
||||
use super::wall_clock;
|
||||
use crate::client::{Model, ModelType, call_chat_completions};
|
||||
use super::{is_transient_error, wall_clock};
|
||||
use crate::client::{Model, ModelType, call_chat_completions_streaming_quiet};
|
||||
use crate::config::prompts::DEFAULT_SKILL_INSTRUCTIONS;
|
||||
use crate::config::{
|
||||
Input, RequestContext, Role, RoleLike, SkillPolicy, should_inject_skill_instructions,
|
||||
};
|
||||
use crate::function::ToolResult;
|
||||
use crate::function::agents::{GuardrailAction, check_pending_tasks_guardrail};
|
||||
use crate::function::jobs::reap_jobs;
|
||||
use crate::function::skill::skill_function_declarations;
|
||||
@@ -16,6 +17,8 @@ use anyhow::{Context, Error, Result, anyhow, bail};
|
||||
use log::warn;
|
||||
use serde_json::Value;
|
||||
use std::collections::HashSet;
|
||||
use std::future::Future;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use tokio::time::timeout;
|
||||
|
||||
@@ -36,34 +39,46 @@ impl LlmNodeExecutor {
|
||||
abort: &AbortSignal,
|
||||
) -> Result<LlmExecutionOutcome> {
|
||||
let result = run(node_id, node, state_manager, parent_ctx, abort).await;
|
||||
let (output, failure_reason) = match result {
|
||||
Ok(raw) => match &node.output_schema {
|
||||
Some(schema) => match structured::extract(&raw, schema, parent_ctx).await {
|
||||
Ok(value) => (value, None),
|
||||
Err(e) => {
|
||||
warn!("llm node structured extraction failed: {e}");
|
||||
(
|
||||
Value::String(format!("LLM node structured-extraction failed: {e}")),
|
||||
Some(format!("structured-extraction failed: {e}")),
|
||||
)
|
||||
}
|
||||
},
|
||||
None => (Value::String(raw), None),
|
||||
},
|
||||
Err(e) => {
|
||||
warn!("llm node failed: {e}");
|
||||
(
|
||||
Value::String(format!("LLM node failed: {e}")),
|
||||
Some(format!("LLM call failed: {e:#}")),
|
||||
)
|
||||
}
|
||||
};
|
||||
|
||||
apply_state_updates_with_output(node, state_manager, &output);
|
||||
outcome_from(failure_reason.as_deref(), node.fallback.as_deref())
|
||||
finish(node, state_manager, parent_ctx, result, abort).await
|
||||
}
|
||||
}
|
||||
|
||||
/// Extraction, state_updates, and the outcome for an already-finished run.
|
||||
/// Split from `execute` so tests can feed a raw run result without a model.
|
||||
async fn finish(
|
||||
node: &LlmNode,
|
||||
state_manager: &mut StateManager,
|
||||
parent_ctx: &mut RequestContext,
|
||||
result: Result<String>,
|
||||
abort: &AbortSignal,
|
||||
) -> Result<LlmExecutionOutcome> {
|
||||
let (output, failure_reason) = match result {
|
||||
Ok(raw) => match &node.output_schema {
|
||||
Some(schema) => match structured::extract(&raw, schema, parent_ctx, abort).await {
|
||||
Ok(value) => (value, None),
|
||||
Err(e) => {
|
||||
warn!("llm node structured extraction failed: {e:#}");
|
||||
(
|
||||
Value::String(format!("LLM node structured-extraction failed: {e:#}")),
|
||||
Some(format!("structured-extraction failed: {e:#}")),
|
||||
)
|
||||
}
|
||||
},
|
||||
None => (Value::String(raw), None),
|
||||
},
|
||||
Err(e) => {
|
||||
warn!("llm node failed: {e:#}");
|
||||
(
|
||||
Value::String(format!("LLM node failed: {e:#}")),
|
||||
Some(format!("LLM call failed: {e:#}")),
|
||||
)
|
||||
}
|
||||
};
|
||||
|
||||
apply_state_updates_with_output(node, state_manager, &output);
|
||||
outcome_from(failure_reason.as_deref(), node.fallback.as_deref())
|
||||
}
|
||||
|
||||
fn outcome_from(
|
||||
failure_reason: Option<&str>,
|
||||
fallback: Option<&str>,
|
||||
@@ -187,13 +202,14 @@ async fn run(
|
||||
node.mcp_tools.clone().map(|map| (node_id.to_string(), map)),
|
||||
);
|
||||
parent_ctx.refresh_mcp_tool_filters();
|
||||
let result = match node.timeout.and_then(wall_clock) {
|
||||
Some(d) => match timeout(d, run_with_retries(node, &prompt, parent_ctx, abort)).await {
|
||||
Ok(r) => r,
|
||||
Err(_) => Err(anyhow!("llm node timed out after {}s", d.as_secs())),
|
||||
},
|
||||
None => run_with_retries(node, &prompt, parent_ctx, abort).await,
|
||||
};
|
||||
let result = bounded_run(
|
||||
node,
|
||||
&prompt,
|
||||
parent_ctx,
|
||||
abort,
|
||||
&mut default_completion_runner(),
|
||||
)
|
||||
.await;
|
||||
parent_ctx.role = saved_role;
|
||||
let node_jobs =
|
||||
std::mem::replace(&mut parent_ctx.node_job_scope, saved_job_scope).unwrap_or_default();
|
||||
@@ -240,18 +256,63 @@ fn restore_agent_skill_policy(ctx: &mut RequestContext, saved: Option<SavedAgent
|
||||
agent.set_enabled_skills(saved.enabled_skills);
|
||||
}
|
||||
|
||||
/// One boxed model call against the input and ctx. Boxed for the same
|
||||
/// reason as `AttemptFuture` in agent.rs: llm nodes run inside
|
||||
/// `tokio::spawn`ed map branches, where higher-ranked opaque futures trip
|
||||
/// the `Send` auto-trait solver.
|
||||
type CompletionFuture<'a> =
|
||||
Pin<Box<dyn Future<Output = Result<(String, Vec<ToolResult>)>> + Send + 'a>>;
|
||||
|
||||
/// The per-call runner the chat loop drives; production uses the quiet
|
||||
/// streaming transport, tests substitute paced or aborting fakes. The runner
|
||||
/// owns client construction and the before-hook so a failed `create_client`
|
||||
/// never records a live `last_message`.
|
||||
type CompletionRunner<'f> = dyn for<'a> FnMut(&'a Input, &'a mut RequestContext, AbortSignal) -> CompletionFuture<'a>
|
||||
+ Send
|
||||
+ 'f;
|
||||
|
||||
fn default_completion_runner()
|
||||
-> impl for<'a> FnMut(&'a Input, &'a mut RequestContext, AbortSignal) -> CompletionFuture<'a> + Send
|
||||
{
|
||||
|input, ctx, abort| {
|
||||
Box::pin(async move {
|
||||
let client = input.create_client()?;
|
||||
ctx.before_chat_completion(input)?;
|
||||
call_chat_completions_streaming_quiet(input, client.as_ref(), ctx, abort).await
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Applies the node's wall-clock bound around the whole retry loop.
|
||||
async fn bounded_run(
|
||||
node: &LlmNode,
|
||||
prompt: &str,
|
||||
ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
runner: &mut CompletionRunner<'_>,
|
||||
) -> Result<String> {
|
||||
match node.timeout.and_then(wall_clock) {
|
||||
Some(d) => match timeout(d, run_with_retries(node, prompt, ctx, abort, runner)).await {
|
||||
Ok(r) => r,
|
||||
Err(_) => Err(anyhow!("llm node timed out after {}s", d.as_secs())),
|
||||
},
|
||||
None => run_with_retries(node, prompt, ctx, abort, runner).await,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_with_retries(
|
||||
node: &LlmNode,
|
||||
prompt: &str,
|
||||
ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
runner: &mut CompletionRunner<'_>,
|
||||
) -> Result<String> {
|
||||
let mut last_err: Option<Error> = None;
|
||||
for attempt in 1..=node.max_attempts {
|
||||
match run_chat_loop(node, prompt, ctx, abort).await {
|
||||
match run_chat_loop(node, prompt, ctx, abort, runner).await {
|
||||
Ok(out) => return Ok(out),
|
||||
Err(e) if is_transient(&e) && attempt < node.max_attempts => {
|
||||
warn!("llm node attempt {attempt} failed (transient): {e}; retrying");
|
||||
Err(e) if is_transient_error(&e) && attempt < node.max_attempts => {
|
||||
warn!("llm node attempt {attempt} failed (transient): {e:#}; retrying");
|
||||
last_err = Some(e);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
@@ -270,6 +331,7 @@ async fn run_chat_loop(
|
||||
prompt: &str,
|
||||
ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
runner: &mut CompletionRunner<'_>,
|
||||
) -> Result<String> {
|
||||
let abort = abort.clone();
|
||||
let app_cfg = Arc::clone(&ctx.app.config);
|
||||
@@ -282,11 +344,12 @@ async fn run_chat_loop(
|
||||
if abort.aborted() {
|
||||
bail!("llm node aborted");
|
||||
}
|
||||
let client = input.create_client()?;
|
||||
ctx.before_chat_completion(&input)?;
|
||||
let (output, tool_results) =
|
||||
call_chat_completions(&input, false, false, client.as_ref(), ctx, abort.clone())
|
||||
.await?;
|
||||
let (output, tool_results) = runner(&input, ctx, abort.clone()).await?;
|
||||
// Defence in depth: the transport bails on abort itself; this closes
|
||||
// the window between its check and ours.
|
||||
if abort.aborted() {
|
||||
bail!("llm node aborted");
|
||||
}
|
||||
ctx.after_chat_completion(app_cfg.as_ref(), &input, &output, &tool_results)?;
|
||||
|
||||
if !output.is_empty() {
|
||||
@@ -449,16 +512,6 @@ fn validate_tools_subset(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn is_transient(err: &Error) -> bool {
|
||||
let s = format!("{err:#}");
|
||||
s.contains("timed out")
|
||||
|| s.contains("rate limit")
|
||||
|| s.contains("429")
|
||||
|| s.contains("Connection reset")
|
||||
|| s.contains("Connection refused")
|
||||
|| s.contains("produced no output")
|
||||
}
|
||||
|
||||
fn apply_state_updates_with_output(
|
||||
node: &LlmNode,
|
||||
state_manager: &mut StateManager,
|
||||
@@ -467,7 +520,7 @@ fn apply_state_updates_with_output(
|
||||
state_updates::apply(
|
||||
state_manager,
|
||||
output,
|
||||
node.output_schema.is_some(),
|
||||
node.output_schema.as_ref(),
|
||||
node.state_updates.as_ref(),
|
||||
);
|
||||
}
|
||||
@@ -618,7 +671,10 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn output_schema_auto_merges_top_level_keys() {
|
||||
let node = node_with_schema(None, json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
None,
|
||||
json!({"type": "object", "properties": {"goal": {}, "summary": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({"goal": "do X", "summary": "details"});
|
||||
|
||||
@@ -630,7 +686,10 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn output_schema_preserves_nested_value_types() {
|
||||
let node = node_with_schema(None, json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
None,
|
||||
json!({"type": "object", "properties": {"tags": {}, "config": {}, "count": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({
|
||||
"tags": ["a", "b"],
|
||||
@@ -649,7 +708,10 @@ mod tests {
|
||||
fn output_schema_explicit_state_updates_override_auto_merge() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("goal".into(), "renamed-{{output.goal}}".into());
|
||||
let node = node_with_schema(Some(u), json!({"type": "object"}));
|
||||
let node = node_with_schema(
|
||||
Some(u),
|
||||
json!({"type": "object", "properties": {"goal": {}}}),
|
||||
);
|
||||
let mut state = manager_with(&[]);
|
||||
let output = json!({"goal": "do X"});
|
||||
|
||||
@@ -724,26 +786,6 @@ mod tests {
|
||||
assert!(mcp.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_matches_expected_signatures() {
|
||||
assert!(is_transient(&anyhow!("request timed out after 30s")));
|
||||
assert!(is_transient(&anyhow!("rate limit reached")));
|
||||
assert!(is_transient(&anyhow!("429 too many requests")));
|
||||
assert!(is_transient(&anyhow!("Connection reset by peer")));
|
||||
assert!(is_transient(&anyhow!("Connection refused")));
|
||||
assert!(is_transient(&anyhow!("llm produced no output")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_rejects_non_transient_errors() {
|
||||
assert!(!is_transient(&anyhow!("Unknown model 'foo'")));
|
||||
assert!(!is_transient(&anyhow!(
|
||||
"llm node references unknown tool 'bad'"
|
||||
)));
|
||||
assert!(!is_transient(&anyhow!("hit max_iterations")));
|
||||
assert!(!is_transient(&anyhow!("authentication failed")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zero_timeout_resolves_to_no_bound() {
|
||||
assert!(Some(0u64).and_then(wall_clock).is_none());
|
||||
@@ -774,6 +816,165 @@ mod tests {
|
||||
assert!(!is_last_turn(u32::MAX, u32::MAX));
|
||||
}
|
||||
|
||||
fn plain_ctx() -> RequestContext {
|
||||
RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd)
|
||||
}
|
||||
|
||||
/// Identity funnel that pins a closure to the runner's higher-ranked
|
||||
/// signature, so inline closures at call sites infer the right lifetimes.
|
||||
fn completion_runner<F>(f: F) -> F
|
||||
where
|
||||
F: for<'a> FnMut(&'a Input, &'a mut RequestContext, AbortSignal) -> CompletionFuture<'a>
|
||||
+ Send,
|
||||
{
|
||||
f
|
||||
}
|
||||
|
||||
fn boxed_completion<'a>(
|
||||
fut: impl Future<Output = Result<(String, Vec<ToolResult>)>> + Send + 'a,
|
||||
) -> CompletionFuture<'a> {
|
||||
Box::pin(fut)
|
||||
}
|
||||
|
||||
fn paced_reply(secs: u64) -> CompletionFuture<'static> {
|
||||
boxed_completion(async move {
|
||||
tokio::time::sleep(Duration::from_secs(secs)).await;
|
||||
Ok(("done".to_string(), vec![]))
|
||||
})
|
||||
}
|
||||
|
||||
/// A 600s generation finishes under a 900s node timeout: the node's
|
||||
/// own deadline is what bounds a reply. The runner is a fake, so this
|
||||
/// does not exercise reqwest's `read_timeout`; the real-socket tests in
|
||||
/// graph/mod.rs pin that it fires on a stall
|
||||
/// (`sse_read_timeout_stall_is_transient`) and not on a slow but
|
||||
/// continuous stream (`sse_slow_but_continuous_stream_outlives_read_timeout`).
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn node_timeout_above_generation_time_lets_the_reply_complete() {
|
||||
let mut node = node_with(None);
|
||||
node.timeout = Some(900);
|
||||
let mut ctx = plain_ctx();
|
||||
let abort = create_abort_signal();
|
||||
|
||||
let out = bounded_run(
|
||||
&node,
|
||||
"user",
|
||||
&mut ctx,
|
||||
&abort,
|
||||
&mut completion_runner(|_input, _ctx, _abort| paced_reply(600)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(out, "done");
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn node_timeout_below_generation_time_fails_transiently() {
|
||||
let mut node = node_with(None);
|
||||
node.timeout = Some(300);
|
||||
let mut ctx = plain_ctx();
|
||||
let abort = create_abort_signal();
|
||||
|
||||
let err = bounded_run(
|
||||
&node,
|
||||
"user",
|
||||
&mut ctx,
|
||||
&abort,
|
||||
&mut completion_runner(|_input, _ctx, _abort| paced_reply(600)),
|
||||
)
|
||||
.await
|
||||
.expect_err("a 600s reply must not survive a 300s node timeout");
|
||||
|
||||
assert_eq!(err.to_string(), "llm node timed out after 300s");
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
/// A typed stall (an `Elapsed` under the provider context, as the
|
||||
/// transport surfaces it) is retried; the second attempt's reply wins.
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn run_with_retries_retries_a_typed_stall_once() {
|
||||
let mut node = node_with(None);
|
||||
node.max_attempts = 2;
|
||||
let mut ctx = plain_ctx();
|
||||
let abort = create_abort_signal();
|
||||
let mut attempts = 0u32;
|
||||
|
||||
let out = run_with_retries(
|
||||
&node,
|
||||
"user",
|
||||
&mut ctx,
|
||||
&abort,
|
||||
&mut completion_runner(|_input, _ctx, _abort| {
|
||||
attempts += 1;
|
||||
if attempts == 1 {
|
||||
boxed_completion(async {
|
||||
let elapsed = tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
std::future::pending::<()>(),
|
||||
)
|
||||
.await
|
||||
.expect_err("pending future must time out");
|
||||
Err(Error::new(elapsed).context("Failed to call chat-completions api"))
|
||||
})
|
||||
} else {
|
||||
boxed_completion(async { Ok(("recovered".to_string(), vec![])) })
|
||||
}
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(out, "recovered");
|
||||
assert_eq!(attempts, 2);
|
||||
}
|
||||
|
||||
/// The production runner builds the client before running the
|
||||
/// before-hook, so a request that never leaves records no live
|
||||
/// `last_message`.
|
||||
#[tokio::test]
|
||||
async fn default_completion_runner_leaves_no_last_message_when_create_client_fails() {
|
||||
let mut ctx = plain_ctx();
|
||||
let input = Input::from_str(&ctx, "hi", None).unwrap();
|
||||
let mut runner = default_completion_runner();
|
||||
|
||||
let err = runner(&input, &mut ctx, create_abort_signal())
|
||||
.await
|
||||
.expect_err("the test AppState has no model to build a client for");
|
||||
|
||||
assert!(
|
||||
format!("{err:#}").contains("Invalid model"),
|
||||
"expected create_client to fail, got: {err:#}"
|
||||
);
|
||||
assert!(ctx.last_message.is_none());
|
||||
}
|
||||
|
||||
/// A call that observes the abort but still returns text must not be
|
||||
/// treated as a completed turn; the loop discards it and fails.
|
||||
#[tokio::test]
|
||||
async fn run_chat_loop_discards_output_from_a_call_aborted_midway() {
|
||||
let node = node_with(None);
|
||||
let mut ctx = plain_ctx();
|
||||
let abort = create_abort_signal();
|
||||
|
||||
let err = run_chat_loop(
|
||||
&node,
|
||||
"user",
|
||||
&mut ctx,
|
||||
&abort,
|
||||
&mut completion_runner(|_input, _ctx, abort| {
|
||||
boxed_completion(async move {
|
||||
abort.set_ctrlc();
|
||||
Ok(("partial".to_string(), vec![]))
|
||||
})
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect_err("partial output from an aborted call is not a success");
|
||||
|
||||
assert_eq!(err.to_string(), "llm node aborted");
|
||||
}
|
||||
|
||||
/// The turn-top abort check runs before any client is created, so an
|
||||
/// already-aborted graph never issues a model call and the node fails
|
||||
/// with the abort reason rather than a connection error.
|
||||
@@ -803,4 +1004,205 @@ mod tests {
|
||||
"no output is recorded for an aborted node without state_updates"
|
||||
);
|
||||
}
|
||||
|
||||
/// The fault text written through state_updates must carry the full
|
||||
/// anyhow chain, not just the outermost context, so downstream fault
|
||||
/// scripts can surface the root cause.
|
||||
#[tokio::test]
|
||||
async fn failure_text_in_state_updates_carries_full_error_chain() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("captured".into(), "{{output}}".into());
|
||||
let mut node = node_with(Some(u));
|
||||
node.prompt = "{{nope}}".into();
|
||||
node.fallback = Some("fb".into());
|
||||
let mut state = manager_with(&[]);
|
||||
let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd);
|
||||
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
|
||||
let abort = create_abort_signal();
|
||||
|
||||
let inner = state
|
||||
.interpolate(&node.prompt)
|
||||
.context("Failed to interpolate llm node prompt")
|
||||
.expect_err("missing reference must fail interpolation");
|
||||
assert_ne!(
|
||||
format!("{inner:#}"),
|
||||
format!("{inner}"),
|
||||
"the chosen error must have more than one level of context"
|
||||
);
|
||||
|
||||
let outcome = LlmNodeExecutor::execute("think", &node, &mut state, &mut ctx, &abort)
|
||||
.await
|
||||
.expect("a declared fallback turns the failure into a route");
|
||||
|
||||
assert_eq!(outcome, LlmExecutionOutcome::FellBack("fb".into()));
|
||||
let captured = state
|
||||
.state()
|
||||
.get("captured")
|
||||
.and_then(Value::as_str)
|
||||
.expect("captured failure text is a string")
|
||||
.to_string();
|
||||
assert!(captured.starts_with("LLM node failed: "), "{captured}");
|
||||
assert!(
|
||||
captured.contains(
|
||||
"Failed to interpolate llm node prompt: Template interpolation failed: 'nope' not found in state"
|
||||
),
|
||||
"{captured}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Without a fallback the node bails, and that bail must wrap the same
|
||||
/// full chain the state_updates text carries.
|
||||
#[tokio::test]
|
||||
async fn bail_without_fallback_carries_full_error_chain() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("captured".into(), "{{output}}".into());
|
||||
let mut node = node_with(Some(u));
|
||||
node.prompt = "{{nope}}".into();
|
||||
node.fallback = None;
|
||||
let mut state = manager_with(&[]);
|
||||
let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd);
|
||||
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
|
||||
let abort = create_abort_signal();
|
||||
|
||||
let err = LlmNodeExecutor::execute("think", &node, &mut state, &mut ctx, &abort)
|
||||
.await
|
||||
.expect_err("no fallback means the failure bails");
|
||||
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains(
|
||||
"LLM node failed and no fallback declared: LLM call failed: Failed to interpolate llm node prompt: Template interpolation failed: 'nope' not found in state"
|
||||
),
|
||||
"{chain}"
|
||||
);
|
||||
let captured = state
|
||||
.state()
|
||||
.get("captured")
|
||||
.and_then(Value::as_str)
|
||||
.expect("state_updates still run before the bail")
|
||||
.to_string();
|
||||
assert!(captured.starts_with("LLM node failed: "), "{captured}");
|
||||
}
|
||||
|
||||
/// A run that succeeds but yields unparseable output must fault through
|
||||
/// the structured-extraction branch with the extractor's full chain, so
|
||||
/// fallback scripts can tell "model spoke prose" from "model call failed".
|
||||
#[tokio::test]
|
||||
async fn structured_extraction_failure_text_carries_full_error_chain() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("captured".into(), "{{output}}".into());
|
||||
let mut node = node_with_schema(Some(u), json!({"type": "object"}));
|
||||
node.fallback = Some("fb".into());
|
||||
let mut state = manager_with(&[]);
|
||||
let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd);
|
||||
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
|
||||
|
||||
let outcome = finish(
|
||||
&node,
|
||||
&mut state,
|
||||
&mut ctx,
|
||||
Ok("not json".to_string()),
|
||||
&create_abort_signal(),
|
||||
)
|
||||
.await
|
||||
.expect("a declared fallback turns the failure into a route");
|
||||
|
||||
assert_eq!(outcome, LlmExecutionOutcome::FellBack("fb".into()));
|
||||
let captured = state
|
||||
.state()
|
||||
.get("captured")
|
||||
.and_then(Value::as_str)
|
||||
.expect("captured failure text is a string")
|
||||
.to_string();
|
||||
assert!(
|
||||
captured.starts_with("LLM node structured-extraction failed: "),
|
||||
"{captured}"
|
||||
);
|
||||
assert!(
|
||||
captured.contains("Structured-output extractor LLM call failed: "),
|
||||
"{captured}"
|
||||
);
|
||||
assert!(captured.contains("Invalid model"), "{captured}");
|
||||
}
|
||||
|
||||
/// The extractor runs on the graph abort: an already-aborted graph
|
||||
/// never issues the extraction request, and the failure text still
|
||||
/// carries the structured-extraction prefix the fallback scripts anchor on.
|
||||
#[tokio::test]
|
||||
async fn structured_extraction_observes_the_graph_abort() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("captured".into(), "{{output}}".into());
|
||||
let mut node = node_with_schema(Some(u), json!({"type": "object"}));
|
||||
node.fallback = Some("fb".into());
|
||||
let mut state = manager_with(&[]);
|
||||
let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd);
|
||||
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
|
||||
let abort = create_abort_signal();
|
||||
abort.set_ctrlc();
|
||||
|
||||
let outcome = finish(
|
||||
&node,
|
||||
&mut state,
|
||||
&mut ctx,
|
||||
Ok("not json".to_string()),
|
||||
&abort,
|
||||
)
|
||||
.await
|
||||
.expect("a declared fallback turns the failure into a route");
|
||||
|
||||
assert_eq!(outcome, LlmExecutionOutcome::FellBack("fb".into()));
|
||||
let captured = state
|
||||
.state()
|
||||
.get("captured")
|
||||
.and_then(Value::as_str)
|
||||
.expect("captured failure text is a string")
|
||||
.to_string();
|
||||
assert!(
|
||||
captured.starts_with("LLM node structured-extraction failed: "),
|
||||
"{captured}"
|
||||
);
|
||||
assert!(captured.contains("Aborted."), "{captured}");
|
||||
assert!(!captured.contains("Invalid model"), "{captured}");
|
||||
}
|
||||
|
||||
/// Without a fallback the extraction fault bails, wrapping the same
|
||||
/// extractor chain the state_updates text carries.
|
||||
#[tokio::test]
|
||||
async fn structured_extraction_failure_without_fallback_bails_with_full_chain() {
|
||||
let mut u = HashMap::new();
|
||||
u.insert("captured".into(), "{{output}}".into());
|
||||
let mut node = node_with_schema(Some(u), json!({"type": "object"}));
|
||||
node.fallback = None;
|
||||
let mut state = manager_with(&[]);
|
||||
let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd);
|
||||
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
|
||||
|
||||
let err = finish(
|
||||
&node,
|
||||
&mut state,
|
||||
&mut ctx,
|
||||
Ok("not json".to_string()),
|
||||
&create_abort_signal(),
|
||||
)
|
||||
.await
|
||||
.expect_err("no fallback means the failure bails");
|
||||
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains(
|
||||
"LLM node failed and no fallback declared: structured-extraction failed: Structured-output extractor LLM call failed"
|
||||
),
|
||||
"{chain}"
|
||||
);
|
||||
let captured = state
|
||||
.state()
|
||||
.get("captured")
|
||||
.and_then(Value::as_str)
|
||||
.expect("state_updates still run before the bail")
|
||||
.to_string();
|
||||
assert!(
|
||||
captured.starts_with("LLM node structured-extraction failed: "),
|
||||
"{captured}"
|
||||
);
|
||||
}
|
||||
}
|
||||
+84
-3
@@ -294,8 +294,8 @@ async fn run_chain_step(
|
||||
branch_nodes.sort_unstable();
|
||||
bail!(
|
||||
"routed to '{target}' which is outside the branch subgraph rooted at \
|
||||
'{}' (branch nodes: {}). Script `_next` targets inside a map branch \
|
||||
must stay within the branch.",
|
||||
'{}' (branch nodes: {}). Routing targets inside a map branch (script \
|
||||
`_next` or node fallback) must stay within the branch.",
|
||||
chain.entry,
|
||||
branch_nodes.join(", ")
|
||||
);
|
||||
@@ -440,6 +440,7 @@ fn first_flagged_agent_by_bfs<'g>(
|
||||
match &node.node_type {
|
||||
NodeType::Script(s) => edges.extend(s.fallback.as_ref()),
|
||||
NodeType::Llm(l) => edges.extend(l.fallback.as_ref()),
|
||||
NodeType::Agent(a) => edges.extend(a.fallback.as_ref()),
|
||||
_ => {}
|
||||
}
|
||||
for next in edges {
|
||||
@@ -464,6 +465,8 @@ mod tests {
|
||||
state_updates: None,
|
||||
output_schema: None,
|
||||
timeout: None,
|
||||
max_attempts: 1,
|
||||
fallback: None,
|
||||
inputs: None,
|
||||
teammates,
|
||||
})
|
||||
@@ -1279,7 +1282,8 @@ nodes:
|
||||
);
|
||||
assert!(
|
||||
chain.contains(
|
||||
"Script `_next` targets inside a map branch must stay within the branch."
|
||||
"Routing targets inside a map branch (script `_next` or node fallback) \
|
||||
must stay within the branch."
|
||||
),
|
||||
"{chain}"
|
||||
);
|
||||
@@ -1608,6 +1612,55 @@ nodes:
|
||||
assert_eq!(collected(&result), vec![json!("fb")]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn map_chain_agent_failure_routes_to_fallback() {
|
||||
if !cmd_available("python3") {
|
||||
eprintln!("skipping: python3 not available");
|
||||
return;
|
||||
}
|
||||
let ws = TestWorkspace::new();
|
||||
ws.write_py(
|
||||
"recover.py",
|
||||
r#"print(json.dumps({"output": "recovered"}))"#,
|
||||
);
|
||||
|
||||
let yaml = r#"
|
||||
name: chain
|
||||
start: fan_out
|
||||
settings:
|
||||
validate_before_run: false
|
||||
initial_state:
|
||||
items: [1, 2]
|
||||
nodes:
|
||||
fan_out:
|
||||
type: map
|
||||
over: "{{items}}"
|
||||
as: item
|
||||
branch: worker
|
||||
collect_into: results
|
||||
next: done
|
||||
worker:
|
||||
type: agent
|
||||
agent: no-such-agent
|
||||
prompt: "hi"
|
||||
fallback: recover
|
||||
recover:
|
||||
type: script
|
||||
script: recover.py
|
||||
done:
|
||||
type: end
|
||||
output: "{{results}}"
|
||||
"#;
|
||||
let result = run_graph(yaml, &ws)
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("executor failed: {e:#}"));
|
||||
|
||||
assert_eq!(
|
||||
collected(&result),
|
||||
vec![json!("recovered"), json!("recovered")]
|
||||
);
|
||||
}
|
||||
|
||||
struct Harness {
|
||||
ws: TestWorkspace,
|
||||
graph: Arc<Graph>,
|
||||
@@ -2433,6 +2486,34 @@ nodes:
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn provision_map_peers_finds_flagged_agent_behind_agent_fallback_edge() {
|
||||
let h = Harness::new(
|
||||
r#"
|
||||
name: t
|
||||
start: gate
|
||||
nodes:
|
||||
gate:
|
||||
type: agent
|
||||
agent: gatekeeper
|
||||
prompt: "p"
|
||||
fallback: rescue
|
||||
rescue:
|
||||
type: agent
|
||||
agent: rescue-agent
|
||||
prompt: "p"
|
||||
teammates: true
|
||||
"#,
|
||||
);
|
||||
|
||||
let (_, assignments) = h.provision("gate", 2);
|
||||
|
||||
assert_eq!(assignments.len(), 2);
|
||||
for (id, _) in &assignments {
|
||||
assert!(id.starts_with("graph_agent_rescue-agent_"), "{id}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn provision_map_peers_ignores_flagged_agent_outside_subgraph() {
|
||||
let h = Harness::new(GATE_WORKER_FINISH_GRAPH);
|
||||
|
||||
@@ -16,6 +16,8 @@ pub mod types;
|
||||
pub mod user_interaction;
|
||||
pub mod validator;
|
||||
|
||||
use crate::client::ApiStatusError;
|
||||
use anyhow::Error;
|
||||
pub use dispatch::{
|
||||
active_agent_graph_name, run_active_agent_graph, run_active_agent_graph_with_inputs,
|
||||
};
|
||||
@@ -34,6 +36,41 @@ pub(crate) fn wall_clock(secs: u64) -> Option<Duration> {
|
||||
(secs != 0).then(|| Duration::from_secs(secs))
|
||||
}
|
||||
|
||||
/// An HTTP status in the chain is authoritative — message text never
|
||||
/// overrides it. String anchors are only consulted for untyped errors.
|
||||
pub(crate) fn is_transient_error(err: &Error) -> bool {
|
||||
if err.chain().any(|c| {
|
||||
c.downcast_ref::<reqwest::Error>()
|
||||
.is_some_and(|e| e.is_connect() || e.is_timeout())
|
||||
|| c.is::<tokio::time::error::Elapsed>()
|
||||
}) {
|
||||
return true;
|
||||
}
|
||||
if let Some(api) = err.chain().find_map(|c| c.downcast_ref::<ApiStatusError>()) {
|
||||
return is_transient_status(api.status);
|
||||
}
|
||||
let s = format!("{err:#}").to_lowercase();
|
||||
s.contains("timed out")
|
||||
|| s.contains("rate limit")
|
||||
|| s.contains("http 429")
|
||||
|| s.contains("status 429")
|
||||
|| s.contains("429 too many")
|
||||
|| s.contains("connection reset")
|
||||
|| s.contains("connection refused")
|
||||
|| s.contains("produced no output")
|
||||
|| s.contains("broken pipe")
|
||||
|| s.contains("connection error")
|
||||
|| s.contains("error sending request")
|
||||
}
|
||||
|
||||
/// Transient HTTP statuses worth a from-scratch retry: 429 (rate limit),
|
||||
/// 500/502/503/504 (upstream hiccups), 529 (Anthropic overloaded).
|
||||
/// Deterministic statuses (4xx other than 429 (including 408) and
|
||||
/// 501/505-class capability errors) are excluded.
|
||||
fn is_transient_status(status: u16) -> bool {
|
||||
matches!(status, 429 | 500 | 502 | 503 | 504 | 529)
|
||||
}
|
||||
|
||||
pub const MAX_STATE_SIZE_BYTES: usize = 32 * 1024;
|
||||
|
||||
pub(in crate::graph) fn type_name(value: &Value) -> &'static str {
|
||||
@@ -50,6 +87,8 @@ pub(in crate::graph) fn type_name(value: &Value) -> &'static str {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::client;
|
||||
use anyhow::anyhow;
|
||||
|
||||
#[test]
|
||||
fn wall_clock_zero_is_no_bound() {
|
||||
@@ -60,4 +99,225 @@ mod tests {
|
||||
fn wall_clock_nonzero_is_that_many_seconds() {
|
||||
assert_eq!(wall_clock(7), Some(Duration::from_secs(7)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_matches_expected_signatures() {
|
||||
assert!(is_transient_error(&anyhow!("request timed out after 30s")));
|
||||
assert!(is_transient_error(&anyhow!("rate limit reached")));
|
||||
assert!(is_transient_error(&anyhow!("HTTP 429")));
|
||||
assert!(is_transient_error(&anyhow!("status 429")));
|
||||
assert!(is_transient_error(&anyhow!("429 Too Many Requests")));
|
||||
assert!(is_transient_error(&anyhow!("Connection reset by peer")));
|
||||
assert!(is_transient_error(&anyhow!("Connection refused")));
|
||||
assert!(is_transient_error(&anyhow!("llm produced no output")));
|
||||
assert!(is_transient_error(&anyhow!(
|
||||
"stream closed because of a broken pipe"
|
||||
)));
|
||||
assert!(is_transient_error(&anyhow!(
|
||||
"connection error: unexpected end of stream"
|
||||
)));
|
||||
assert!(is_transient_error(&anyhow!(
|
||||
"error sending request for url"
|
||||
)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_sees_through_context_chains() {
|
||||
let err = anyhow!("stream closed because of a broken pipe")
|
||||
.context("Failed to call chat-completions api")
|
||||
.context("Agent 'domain-reviewer' failed");
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_rejects_non_transient_errors() {
|
||||
assert!(!is_transient_error(&anyhow!("Unknown model 'foo'")));
|
||||
assert!(!is_transient_error(&anyhow!(
|
||||
"llm node references unknown tool 'bad'"
|
||||
)));
|
||||
assert!(!is_transient_error(&anyhow!("hit max_iterations")));
|
||||
assert!(!is_transient_error(&anyhow!("authentication failed")));
|
||||
assert!(!is_transient_error(&anyhow!(
|
||||
"script exited 1: see line 429 in a comment"
|
||||
)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_sees_typed_api_status_429_through_context_chain() {
|
||||
let data = serde_json::json!({
|
||||
"error": {
|
||||
"type": "rate_limit_error",
|
||||
"message": "Too many requests, please slow down"
|
||||
}
|
||||
});
|
||||
let err = client::catch_error(&data, 429)
|
||||
.unwrap_err()
|
||||
.context("Failed to call chat-completions api")
|
||||
.context("Agent 'domain-reviewer' failed");
|
||||
|
||||
let rendered = format!("{err:#}");
|
||||
assert!(
|
||||
!is_transient_error(&anyhow!("{rendered}")),
|
||||
"the rendered text alone must not be transient — proves the typed arm is what catches it"
|
||||
);
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_typed_status_overrides_message_text() {
|
||||
let err = client::catch_error(
|
||||
&serde_json::json!({ "message": "rate limit reached — connection reset" }),
|
||||
400,
|
||||
)
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
!is_transient_error(&err),
|
||||
"typed 400 must not be retried even when the body matches string anchors"
|
||||
);
|
||||
|
||||
let err = client::catch_error(&serde_json::json!({ "message": "x" }), 503)
|
||||
.unwrap_err()
|
||||
.context("Agent 'x' failed");
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_typed_status_table() {
|
||||
let data = serde_json::json!({ "message": "x" });
|
||||
for status in [429, 500, 502, 503, 504, 529] {
|
||||
let err = client::catch_error(&data, status).unwrap_err();
|
||||
assert!(
|
||||
is_transient_error(&err),
|
||||
"status {status} should be transient"
|
||||
);
|
||||
}
|
||||
for status in [400, 401, 403, 404, 408, 422, 501, 505] {
|
||||
let err = client::catch_error(&data, status).unwrap_err();
|
||||
assert!(
|
||||
!is_transient_error(&err),
|
||||
"status {status} should not be transient"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_transient_error_string_anchors_are_case_insensitive() {
|
||||
assert!(is_transient_error(&anyhow!("Rate Limit reached")));
|
||||
assert!(is_transient_error(&anyhow!("Request Timed Out")));
|
||||
assert!(is_transient_error(&anyhow!("connection reset by peer")));
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn is_transient_error_recognizes_typed_elapsed_without_matching_text() {
|
||||
let elapsed = tokio::time::timeout(Duration::from_secs(1), std::future::pending::<()>())
|
||||
.await
|
||||
.expect_err("pending future must time out");
|
||||
let err = Error::new(elapsed).context("calling provider");
|
||||
|
||||
// "deadline has elapsed" contains no recognized substring, so only
|
||||
// the typed pre-check can classify this as transient.
|
||||
let rendered = format!("{err:#}");
|
||||
assert!(!rendered.contains("timed out"), "{rendered}");
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn is_transient_error_recognizes_typed_reqwest_connect_error() {
|
||||
// `no_proxy` keeps environments with HTTP(S)_PROXY set from turning
|
||||
// the refused connection into a proxied response.
|
||||
let client = reqwest::Client::builder().no_proxy().build().unwrap();
|
||||
let err = client
|
||||
.get("http://127.0.0.1:1")
|
||||
.send()
|
||||
.await
|
||||
.expect_err("closed local port must refuse the connection");
|
||||
|
||||
assert!(err.is_connect());
|
||||
assert!(is_transient_error(&Error::new(err)));
|
||||
}
|
||||
|
||||
/// A `read_timeout` stall on the SSE path keeps its typed reqwest error,
|
||||
/// so the classifier's typed branch fires rather than depending on the
|
||||
/// rendered text. The server sends headers then never writes a body.
|
||||
#[tokio::test]
|
||||
async fn sse_read_timeout_stall_is_transient() {
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
let server = tokio::spawn(async move {
|
||||
let (mut sock, _) = listener.accept().await.unwrap();
|
||||
let mut buf = [0u8; 4096];
|
||||
let _ = sock.read(&mut buf).await;
|
||||
sock.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n\r\n")
|
||||
.await
|
||||
.unwrap();
|
||||
std::future::pending::<()>().await;
|
||||
});
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.no_proxy()
|
||||
.read_timeout(Duration::from_millis(50))
|
||||
.build()
|
||||
.unwrap();
|
||||
let err = client::sse_stream(client.get(format!("http://{addr}/")), |_| Ok(false))
|
||||
.await
|
||||
.expect_err("a body that never arrives must trip the read timeout");
|
||||
server.abort();
|
||||
|
||||
assert!(
|
||||
err.chain().any(|c| {
|
||||
c.downcast_ref::<reqwest::Error>()
|
||||
.is_some_and(|e| e.is_timeout())
|
||||
}),
|
||||
"typed reqwest timeout missing from chain: {err:#}"
|
||||
);
|
||||
assert!(is_transient_error(&err));
|
||||
}
|
||||
|
||||
/// `read_timeout` resets on every read: a stream whose events keep
|
||||
/// arriving well under the limit completes even though the whole body
|
||||
/// takes several times longer than the limit.
|
||||
#[tokio::test]
|
||||
async fn sse_slow_but_continuous_stream_outlives_read_timeout() {
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
|
||||
const EVENTS: usize = 15;
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
let server = tokio::spawn(async move {
|
||||
let (mut sock, _) = listener.accept().await.unwrap();
|
||||
let mut buf = [0u8; 4096];
|
||||
let _ = sock.read(&mut buf).await;
|
||||
sock.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n\r\n")
|
||||
.await
|
||||
.unwrap();
|
||||
for i in 0..EVENTS {
|
||||
tokio::time::sleep(Duration::from_millis(20)).await;
|
||||
sock.write_all(format!("data: chunk{i}\n\n").as_bytes())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
sock.write_all(b"data: [DONE]\n\n").await.unwrap();
|
||||
});
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.no_proxy()
|
||||
.read_timeout(Duration::from_millis(100))
|
||||
.build()
|
||||
.unwrap();
|
||||
let mut received = 0usize;
|
||||
client::sse_stream(client.get(format!("http://{addr}/")), |message| {
|
||||
if message.data == "[DONE]" {
|
||||
return Ok(true);
|
||||
}
|
||||
received += 1;
|
||||
Ok(false)
|
||||
})
|
||||
.await
|
||||
.expect("events every 20ms must never trip a 100ms read timeout");
|
||||
server.abort();
|
||||
|
||||
assert_eq!(received, EVENTS);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -68,7 +68,7 @@ fn build_rag_output(context: String, sources_str: &str) -> Value {
|
||||
}
|
||||
|
||||
fn apply_state_updates(node: &RagNode, state_manager: &mut StateManager, output: &Value) {
|
||||
state_updates::apply(state_manager, output, false, node.state_updates.as_ref());
|
||||
state_updates::apply(state_manager, output, None, node.state_updates.as_ref());
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
+101
-16
@@ -4,19 +4,32 @@ use std::collections::HashMap;
|
||||
|
||||
pub(super) const OUTPUT_KEY: &str = "output";
|
||||
|
||||
/// Merge `output_schema` top-level keys (when `has_schema`) and apply `state_updates` with
|
||||
/// `{{output}}` bound to `output` for the duration of interpolation only. An explicit `output`
|
||||
/// key in `updates` is honored; otherwise `output` is restored to its prior value, or removed
|
||||
/// again if it was absent.
|
||||
/// Merge the `output` keys declared under `schema["properties"]` and apply `state_updates` with
|
||||
/// `{{output}}` bound to the full, unfiltered `output` for the duration of interpolation only.
|
||||
///
|
||||
/// The schema is the graph author's declaration of what the model may write to state; any other
|
||||
/// key in the output is model- (and therefore prompt-) controlled and is dropped rather than
|
||||
/// merged implicitly. No `schema`, or a schema without an object `properties`, merges nothing.
|
||||
/// Authors who want an undeclared key can still lift it explicitly via
|
||||
/// `state_updates: { key: '{{output.key}}' }`.
|
||||
///
|
||||
/// An explicit `output` key in `updates` is honored; otherwise `output` is restored to its prior
|
||||
/// value, or removed again if it was absent.
|
||||
pub(super) fn apply(
|
||||
state_manager: &mut StateManager,
|
||||
output: &Value,
|
||||
has_schema: bool,
|
||||
schema: Option<&Value>,
|
||||
updates: Option<&HashMap<String, String>>,
|
||||
) {
|
||||
if has_schema && let Some(obj) = output.as_object() {
|
||||
if let Some(declared) = schema
|
||||
.and_then(|s| s.get("properties"))
|
||||
.and_then(Value::as_object)
|
||||
&& let Some(obj) = output.as_object()
|
||||
{
|
||||
for (k, v) in obj {
|
||||
state_manager.state_mut().set(k.clone(), v.clone());
|
||||
if declared.contains_key(k) {
|
||||
state_manager.state_mut().set(k.clone(), v.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,7 +88,7 @@ mod tests {
|
||||
let u = updates(&[("output", "{{output}}")]);
|
||||
let mut state = manager_with(&[("output", json!("stale"))]);
|
||||
|
||||
apply(&mut state, &json!("the completion"), false, Some(&u));
|
||||
apply(&mut state, &json!("the completion"), None, Some(&u));
|
||||
|
||||
assert_eq!(
|
||||
state.state().get(OUTPUT_KEY),
|
||||
@@ -88,7 +101,7 @@ mod tests {
|
||||
let u = updates(&[("output", "{{output}}"), ("summary", "got: {{output}}")]);
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(&mut state, &json!("done"), false, Some(&u));
|
||||
apply(&mut state, &json!("done"), None, Some(&u));
|
||||
|
||||
assert_eq!(state.state().get(OUTPUT_KEY), Some(&json!("done")));
|
||||
assert_eq!(state.state().get("summary"), Some(&json!("got: done")));
|
||||
@@ -98,7 +111,7 @@ mod tests {
|
||||
fn no_updates_is_a_noop() {
|
||||
let mut state = manager_with(&[("keep", json!(1))]);
|
||||
|
||||
apply(&mut state, &json!("ignored"), false, None);
|
||||
apply(&mut state, &json!("ignored"), None, None);
|
||||
|
||||
assert_eq!(state.state().get("keep"), Some(&json!(1)));
|
||||
assert!(state.state().get(OUTPUT_KEY).is_none());
|
||||
@@ -109,7 +122,7 @@ mod tests {
|
||||
let u = updates(&[("response", "{{output}}")]);
|
||||
let mut state = manager_with(&[("output", json!("preserved"))]);
|
||||
|
||||
apply(&mut state, &json!("new"), false, Some(&u));
|
||||
apply(&mut state, &json!("new"), None, Some(&u));
|
||||
|
||||
assert_eq!(state.state().get("response"), Some(&json!("new")));
|
||||
assert_eq!(state.state().get(OUTPUT_KEY), Some(&json!("preserved")));
|
||||
@@ -120,7 +133,7 @@ mod tests {
|
||||
let u = updates(&[("response", "{{output}}")]);
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(&mut state, &json!("new"), false, Some(&u));
|
||||
apply(&mut state, &json!("new"), None, Some(&u));
|
||||
|
||||
assert_eq!(state.state().get("response"), Some(&json!("new")));
|
||||
assert!(state.state().get(OUTPUT_KEY).is_none());
|
||||
@@ -131,23 +144,95 @@ mod tests {
|
||||
let u = updates(&[("response", "{{output}}")]);
|
||||
let mut state = manager_with(&[("output", json!(null))]);
|
||||
|
||||
apply(&mut state, &json!("new"), false, Some(&u));
|
||||
apply(&mut state, &json!("new"), None, Some(&u));
|
||||
|
||||
assert_eq!(state.state().get("response"), Some(&json!("new")));
|
||||
assert_eq!(state.state().get(OUTPUT_KEY), Some(&json!(null)));
|
||||
}
|
||||
|
||||
fn schema_with(keys: &[&str]) -> Value {
|
||||
let props: serde_json::Map<String, Value> = keys
|
||||
.iter()
|
||||
.map(|k| ((*k).to_string(), json!({"type": "string"})))
|
||||
.collect();
|
||||
json!({"type": "object", "properties": props})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schema_merges_top_level_keys_when_has_schema() {
|
||||
fn schema_merges_declared_top_level_keys() {
|
||||
let schema = schema_with(&["summary", "score"]);
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(&mut state, &json!({"summary": "s", "score": 7}), true, None);
|
||||
apply(
|
||||
&mut state,
|
||||
&json!({"summary": "s", "score": 7}),
|
||||
Some(&schema),
|
||||
None,
|
||||
);
|
||||
|
||||
assert_eq!(state.state().get("summary"), Some(&json!("s")));
|
||||
assert_eq!(state.state().get("score"), Some(&json!(7)));
|
||||
assert!(state.state().get(OUTPUT_KEY).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schema_drops_undeclared_keys_and_preserves_existing_state() {
|
||||
let schema = schema_with(&["summary"]);
|
||||
let mut state = manager_with(&[("verification_commands", json!(["cargo test"]))]);
|
||||
|
||||
apply(
|
||||
&mut state,
|
||||
&json!({"summary": "s", "verification_commands": ["rm -rf /"]}),
|
||||
Some(&schema),
|
||||
None,
|
||||
);
|
||||
|
||||
assert_eq!(state.state().get("summary"), Some(&json!("s")));
|
||||
assert_eq!(
|
||||
state.state().get("verification_commands"),
|
||||
Some(&json!(["cargo test"]))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn none_schema_merges_nothing() {
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(&mut state, &json!({"summary": "s"}), None, None);
|
||||
|
||||
assert!(state.state().get("summary").is_none());
|
||||
assert!(state.state().get(OUTPUT_KEY).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schema_without_properties_merges_nothing() {
|
||||
let schema = json!({"type": "object"});
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(&mut state, &json!({"summary": "s"}), Some(&schema), None);
|
||||
|
||||
assert!(state.state().get("summary").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn state_updates_still_see_undeclared_output_keys() {
|
||||
let schema = schema_with(&["summary"]);
|
||||
let u = updates(&[("lifted", "{{output.extra}}")]);
|
||||
let mut state = manager_with(&[]);
|
||||
|
||||
apply(
|
||||
&mut state,
|
||||
&json!({"summary": "s", "extra": "e"}),
|
||||
Some(&schema),
|
||||
Some(&u),
|
||||
);
|
||||
|
||||
assert_eq!(state.state().get("summary"), Some(&json!("s")));
|
||||
assert!(state.state().get("extra").is_none());
|
||||
assert_eq!(state.state().get("lifted"), Some(&json!("e")));
|
||||
assert!(state.state().get(OUTPUT_KEY).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_output_is_not_merged_without_schema() {
|
||||
let u = updates(&[("ctx", "{{output.context}}")]);
|
||||
@@ -156,7 +241,7 @@ mod tests {
|
||||
apply(
|
||||
&mut state,
|
||||
&json!({"context": "c", "sources": ["a"]}),
|
||||
false,
|
||||
None,
|
||||
Some(&u),
|
||||
);
|
||||
|
||||
|
||||
+26
-9
@@ -1,6 +1,6 @@
|
||||
use crate::client::call_chat_completions;
|
||||
use crate::client::call_chat_completions_streaming_quiet;
|
||||
use crate::config::{Input, RequestContext, Role, RoleLike};
|
||||
use crate::utils::create_abort_signal;
|
||||
use crate::utils::AbortSignal;
|
||||
use anyhow::{Context, Result, bail};
|
||||
use serde_json::Value;
|
||||
use std::sync::Arc;
|
||||
@@ -19,18 +19,24 @@ Rules:
|
||||
- If a field cannot be determined from the input, use `null` (when allowed) or your best inferred value.
|
||||
- Do NOT invent fields not present in the schema.";
|
||||
|
||||
pub async fn extract(raw: &str, schema: &Value, parent_ctx: &mut RequestContext) -> Result<Value> {
|
||||
pub async fn extract(
|
||||
raw: &str,
|
||||
schema: &Value,
|
||||
parent_ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
) -> Result<Value> {
|
||||
if let Some(parsed) = try_parse_json(raw) {
|
||||
return Ok(parsed);
|
||||
}
|
||||
|
||||
extract_via_extractor(raw, schema, parent_ctx, false).await
|
||||
extract_via_extractor(raw, schema, parent_ctx, abort, false).await
|
||||
}
|
||||
|
||||
async fn extract_via_extractor(
|
||||
raw: &str,
|
||||
schema: &Value,
|
||||
parent_ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
is_repair: bool,
|
||||
) -> Result<Value> {
|
||||
let role = build_extractor_role(parent_ctx);
|
||||
@@ -38,7 +44,7 @@ async fn extract_via_extractor(
|
||||
|
||||
let saved_role = parent_ctx.role.clone();
|
||||
parent_ctx.role = Some(role);
|
||||
let result = run_one_shot(&prompt, parent_ctx).await;
|
||||
let result = run_one_shot(&prompt, parent_ctx, abort).await;
|
||||
parent_ctx.role = saved_role;
|
||||
|
||||
let output = result.context("Structured-output extractor LLM call failed")?;
|
||||
@@ -49,7 +55,12 @@ async fn extract_via_extractor(
|
||||
"Structured-output extractor failed to produce valid JSON after repair retry. \
|
||||
Last response:\n{output}"
|
||||
),
|
||||
None => Box::pin(extract_via_extractor(&output, schema, parent_ctx, true)).await,
|
||||
None => {
|
||||
Box::pin(extract_via_extractor(
|
||||
&output, schema, parent_ctx, abort, true,
|
||||
))
|
||||
.await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,15 +84,21 @@ fn build_extractor_prompt(raw: &str, schema: &Value, is_repair: bool) -> String
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_one_shot(prompt: &str, ctx: &mut RequestContext) -> Result<String> {
|
||||
let abort = create_abort_signal();
|
||||
async fn run_one_shot(
|
||||
prompt: &str,
|
||||
ctx: &mut RequestContext,
|
||||
abort: &AbortSignal,
|
||||
) -> Result<String> {
|
||||
if abort.aborted() {
|
||||
bail!("Aborted.");
|
||||
}
|
||||
let app_cfg = Arc::clone(&ctx.app.config);
|
||||
let role_for_input = ctx.role.clone();
|
||||
let input = Input::from_str(ctx, prompt, role_for_input)?;
|
||||
let client = input.create_client()?;
|
||||
ctx.before_chat_completion(&input)?;
|
||||
let (output, tool_results) =
|
||||
call_chat_completions(&input, false, false, client.as_ref(), ctx, abort).await?;
|
||||
call_chat_completions_streaming_quiet(&input, client.as_ref(), ctx, abort.clone()).await?;
|
||||
ctx.after_chat_completion(app_cfg.as_ref(), &input, &output, &tool_results)?;
|
||||
|
||||
Ok(output)
|
||||
|
||||
+78
-2
@@ -240,6 +240,12 @@ pub struct AgentNode {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub timeout: Option<u64>,
|
||||
|
||||
#[serde(default = "default_max_attempts")]
|
||||
pub max_attempts: u32,
|
||||
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub fallback: Option<String>,
|
||||
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub inputs: Option<HashMap<String, String>>,
|
||||
|
||||
@@ -321,7 +327,7 @@ pub struct LlmNode {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub fallback: Option<String>,
|
||||
|
||||
#[serde(default = "default_llm_max_attempts")]
|
||||
#[serde(default = "default_max_attempts")]
|
||||
pub max_attempts: u32,
|
||||
|
||||
#[serde(default = "default_llm_max_iterations")]
|
||||
@@ -349,7 +355,7 @@ pub struct LlmNode {
|
||||
pub skill_instructions: Option<String>,
|
||||
}
|
||||
|
||||
fn default_llm_max_attempts() -> u32 {
|
||||
fn default_max_attempts() -> u32 {
|
||||
1
|
||||
}
|
||||
|
||||
@@ -640,6 +646,76 @@ nodes:
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_node_max_attempts_parses_and_defaults_to_one() {
|
||||
let yaml = r#"
|
||||
name: g
|
||||
start: a
|
||||
nodes:
|
||||
a:
|
||||
id: a
|
||||
type: agent
|
||||
agent: helper
|
||||
prompt: hi
|
||||
max_attempts: 2
|
||||
next: b
|
||||
b:
|
||||
id: b
|
||||
type: agent
|
||||
agent: helper
|
||||
prompt: hi
|
||||
next: e
|
||||
e:
|
||||
id: e
|
||||
type: end
|
||||
output: done
|
||||
"#;
|
||||
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
|
||||
match &graph.get_node("a").unwrap().node_type {
|
||||
NodeType::Agent(n) => assert_eq!(n.max_attempts, 2),
|
||||
_ => panic!("expected Agent variant"),
|
||||
}
|
||||
match &graph.get_node("b").unwrap().node_type {
|
||||
NodeType::Agent(n) => assert_eq!(n.max_attempts, 1),
|
||||
_ => panic!("expected Agent variant"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_node_fallback_parses_and_defaults_to_none() {
|
||||
let yaml = r#"
|
||||
name: g
|
||||
start: a
|
||||
nodes:
|
||||
a:
|
||||
id: a
|
||||
type: agent
|
||||
agent: helper
|
||||
prompt: hi
|
||||
fallback: b
|
||||
next: b
|
||||
b:
|
||||
id: b
|
||||
type: agent
|
||||
agent: helper
|
||||
prompt: hi
|
||||
next: e
|
||||
e:
|
||||
id: e
|
||||
type: end
|
||||
output: done
|
||||
"#;
|
||||
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
|
||||
match &graph.get_node("a").unwrap().node_type {
|
||||
NodeType::Agent(n) => assert_eq!(n.fallback.as_deref(), Some("b")),
|
||||
_ => panic!("expected Agent variant"),
|
||||
}
|
||||
match &graph.get_node("b").unwrap().node_type {
|
||||
NodeType::Agent(n) => assert!(n.fallback.is_none()),
|
||||
_ => panic!("expected Agent variant"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_node_inputs_parse_optionally_and_empty_map_is_some() {
|
||||
let yaml = r#"
|
||||
|
||||
+556
-7
@@ -5,7 +5,7 @@ use crate::config;
|
||||
use crate::config::{Agent, AppConfig, paths};
|
||||
use crate::rag::{GraphRagConfig, RagData};
|
||||
use anyhow::{Result, bail};
|
||||
use std::collections::{BTreeMap, HashSet, VecDeque};
|
||||
use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use tokio::sync::Semaphore;
|
||||
@@ -176,6 +176,9 @@ impl GraphValidator {
|
||||
self.validate_rag_nodes(graph, &mut result);
|
||||
self.validate_llm_nodes(graph, &mut result);
|
||||
self.validate_llm_skills(graph, &mut result);
|
||||
self.validate_max_attempts(graph, &mut result);
|
||||
self.validate_fallback_capture(graph, &mut result);
|
||||
self.validate_output_schema_properties(graph, &mut result);
|
||||
self.validate_max_concurrency(graph, &mut result);
|
||||
self.validate_max_concurrency_template(graph, &mut result);
|
||||
self.validate_orchestration_limits(graph, &mut result);
|
||||
@@ -374,6 +377,87 @@ impl GraphValidator {
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_max_attempts(&self, graph: &Graph, result: &mut ValidationResult) {
|
||||
for (node_id, node) in &graph.nodes {
|
||||
let kind = match &node.node_type {
|
||||
NodeType::Llm(l) if l.max_attempts == 0 => "llm",
|
||||
NodeType::Agent(a) if a.max_attempts == 0 => "agent",
|
||||
_ => continue,
|
||||
};
|
||||
result.error(ValidationError::with_node(
|
||||
node_id,
|
||||
format!(
|
||||
"{kind} node's `max_attempts` must be >= 1 (got 0); the node \
|
||||
would exhaust retries without ever attempting a run"
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_fallback_capture(&self, graph: &Graph, result: &mut ValidationResult) {
|
||||
for (node_id, node) in &graph.nodes {
|
||||
let kind = match &node.node_type {
|
||||
NodeType::Llm(l)
|
||||
if l.fallback.is_some()
|
||||
&& l.output_schema.is_some()
|
||||
&& !lifts_via_state_updates(&l.state_updates) =>
|
||||
{
|
||||
"llm"
|
||||
}
|
||||
NodeType::Agent(a)
|
||||
if a.fallback.is_some()
|
||||
&& a.output_schema.is_some()
|
||||
&& !lifts_via_state_updates(&a.state_updates) =>
|
||||
{
|
||||
"agent"
|
||||
}
|
||||
_ => continue,
|
||||
};
|
||||
result.warning(ValidationError::with_node(
|
||||
node_id,
|
||||
format!(
|
||||
"{kind} node declares `fallback` with `output_schema` but no \
|
||||
`state_updates`; on failure the \"...failed: <chain>\" string is \
|
||||
only written through `state_updates`, so the fallback node \
|
||||
cannot see why it was reached"
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_output_schema_properties(&self, graph: &Graph, result: &mut ValidationResult) {
|
||||
for (node_id, node) in &graph.nodes {
|
||||
let (kind, schema) = match &node.node_type {
|
||||
NodeType::Llm(l) if !lifts_via_state_updates(&l.state_updates) => {
|
||||
("llm", l.output_schema.as_ref())
|
||||
}
|
||||
NodeType::Agent(a) if !lifts_via_state_updates(&a.state_updates) => {
|
||||
("agent", a.output_schema.as_ref())
|
||||
}
|
||||
_ => continue,
|
||||
};
|
||||
let Some(schema) = schema else {
|
||||
continue;
|
||||
};
|
||||
if schema
|
||||
.get("properties")
|
||||
.and_then(serde_json::Value::as_object)
|
||||
.is_some_and(|m| !m.is_empty())
|
||||
{
|
||||
continue;
|
||||
}
|
||||
result.warning(ValidationError::with_node(
|
||||
node_id,
|
||||
format!(
|
||||
"{kind} node's `output_schema` declares no (non-empty) object `properties`; \
|
||||
the engine auto-merges ONLY keys declared under `properties` (a bare \
|
||||
object schema merges nothing). Declare the keys under \
|
||||
`properties` or lift them via `state_updates`"
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_node_references(&self, graph: &Graph, result: &mut ValidationResult) {
|
||||
for (node_id, node) in &graph.nodes {
|
||||
for (target, label) in declared_targets(node) {
|
||||
@@ -719,6 +803,7 @@ impl GraphValidator {
|
||||
.filter_map(|(_, n)| match &n.node_type {
|
||||
NodeType::Script(s) => s.fallback.as_deref(),
|
||||
NodeType::Llm(l) => l.fallback.as_deref(),
|
||||
NodeType::Agent(a) => a.fallback.as_deref(),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
@@ -982,12 +1067,17 @@ fn declared_targets(node: &Node) -> Vec<(String, &'static str)> {
|
||||
out.push((t.clone(), "llm 'fallback'"));
|
||||
}
|
||||
}
|
||||
NodeType::Agent(a) => {
|
||||
if let Some(t) = &a.fallback {
|
||||
out.push((t.clone(), "agent 'fallback'"));
|
||||
}
|
||||
}
|
||||
NodeType::Map(m) => {
|
||||
out.push((m.branch.clone(), "map 'branch'"));
|
||||
}
|
||||
// `agent`/`input`/`rag` route only via `next` (already collected
|
||||
// above); `end` is terminal. No type-specific routing edges to add.
|
||||
NodeType::Agent(_) | NodeType::Input(_) | NodeType::Rag(_) | NodeType::End(_) => {}
|
||||
// `input`/`rag` route only via `next` (already collected above);
|
||||
// `end` is terminal. No type-specific routing edges to add.
|
||||
NodeType::Input(_) | NodeType::Rag(_) | NodeType::End(_) => {}
|
||||
}
|
||||
out
|
||||
}
|
||||
@@ -1056,8 +1146,8 @@ fn main_flow_reachable(graph: &Graph) -> HashSet<String> {
|
||||
}
|
||||
|
||||
/// Nodes reachable from `entry` over the edges a chain can follow at run
|
||||
/// time: `next` targets and script/llm `fallback`s. Approval routes and a
|
||||
/// nested map's `branch` are not followed. Neither may appear inside a
|
||||
/// time: `next` targets and script/llm/agent `fallback`s. Approval routes
|
||||
/// and a nested map's `branch` are not followed. Neither may appear inside a
|
||||
/// branch, and the validator reports them separately.
|
||||
pub(super) fn branch_subgraph(graph: &Graph, entry: &str) -> HashSet<String> {
|
||||
let mut reachable: HashSet<String> = HashSet::new();
|
||||
@@ -1082,6 +1172,7 @@ pub(super) fn branch_subgraph(graph: &Graph, entry: &str) -> HashSet<String> {
|
||||
match &node.node_type {
|
||||
NodeType::Script(s) => edges.extend(s.fallback.as_ref()),
|
||||
NodeType::Llm(l) => edges.extend(l.fallback.as_ref()),
|
||||
NodeType::Agent(a) => edges.extend(a.fallback.as_ref()),
|
||||
_ => {}
|
||||
}
|
||||
for next in edges {
|
||||
@@ -1206,7 +1297,7 @@ fn primary_templated_fields(node: &Node) -> Vec<String> {
|
||||
}
|
||||
}
|
||||
|
||||
fn node_state_updates_map(node: &Node) -> Option<&std::collections::HashMap<String, String>> {
|
||||
fn node_state_updates_map(node: &Node) -> Option<&HashMap<String, String>> {
|
||||
match &node.node_type {
|
||||
NodeType::Llm(n) => n.state_updates.as_ref(),
|
||||
NodeType::Agent(n) => n.state_updates.as_ref(),
|
||||
@@ -1219,6 +1310,10 @@ fn node_state_updates_map(node: &Node) -> Option<&std::collections::HashMap<Stri
|
||||
}
|
||||
}
|
||||
|
||||
fn lifts_via_state_updates(updates: &Option<HashMap<String, String>>) -> bool {
|
||||
updates.as_ref().is_some_and(|m| !m.is_empty())
|
||||
}
|
||||
|
||||
fn node_state_updates_keys(node: &Node) -> Option<HashSet<String>> {
|
||||
let updates = match &node.node_type {
|
||||
NodeType::Agent(n) => n.state_updates.as_ref(),
|
||||
@@ -1924,6 +2019,8 @@ mod tests {
|
||||
state_updates: None,
|
||||
output_schema: None,
|
||||
timeout: None,
|
||||
max_attempts: 1,
|
||||
fallback: None,
|
||||
inputs: None,
|
||||
teammates: false,
|
||||
}),
|
||||
@@ -2017,6 +2114,51 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flags_missing_agent_fallback_target() {
|
||||
let mut a = agent_node("a", "helper", Some("end"));
|
||||
if let NodeType::Agent(ref mut an) = a.node_type {
|
||||
an.fallback = Some("ghost".into());
|
||||
}
|
||||
let graph = graph_with(vec![("a", a), ("end", end_node("end"))], "a");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(!result.is_valid());
|
||||
assert!(
|
||||
result
|
||||
.errors
|
||||
.iter()
|
||||
.any(|e| e.message.contains("non-existent node 'ghost'")
|
||||
&& e.message.contains("agent 'fallback'"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_fallback_target_is_not_flagged_unreachable() {
|
||||
let mut a = agent_node("a", "helper", Some("end"));
|
||||
if let NodeType::Agent(ref mut an) = a.node_type {
|
||||
an.fallback = Some("recover".into());
|
||||
}
|
||||
let recover = agent_node("recover", "helper", Some("end"));
|
||||
let graph = graph_with(
|
||||
vec![("a", a), ("recover", recover), ("end", end_node("end"))],
|
||||
"a",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
!result
|
||||
.warnings
|
||||
.iter()
|
||||
.any(|w| w.node_id.as_deref() == Some("recover")
|
||||
&& w.message.contains("unreachable")),
|
||||
"fallback target 'recover' incorrectly marked unreachable: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detects_two_node_cycle() {
|
||||
let mut a = end_node("a");
|
||||
@@ -2856,6 +2998,90 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn map_branch_agent_fallback_escaping_the_branch_gets_hint() {
|
||||
let map = map_node_basic("m", "br", Some("end"));
|
||||
let mut branch = agent_node("br", "helper", None);
|
||||
if let NodeType::Agent(ref mut an) = branch.node_type {
|
||||
an.fallback = Some("end".into());
|
||||
}
|
||||
let graph = graph_with(
|
||||
vec![("m", map), ("br", branch), ("end", end_node("end"))],
|
||||
"m",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result.errors.iter().any(|e| e.message.contains("'end'")
|
||||
&& e.message.contains("end node")
|
||||
&& e.message.contains("never honored")
|
||||
&& e.node_id.as_deref() == Some("m")),
|
||||
"expected end-node error with the fallback hint: {:?}",
|
||||
result.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn map_branch_agent_fallback_to_branch_local_node_passes() {
|
||||
let map = map_node_basic("m", "br", Some("end"));
|
||||
let mut branch = agent_node("br", "helper", None);
|
||||
if let NodeType::Agent(ref mut an) = branch.node_type {
|
||||
an.fallback = Some("recover".into());
|
||||
}
|
||||
let recover = agent_node("recover", "helper", None);
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("m", map),
|
||||
("br", branch),
|
||||
("recover", recover),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"m",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
// Agent-existence errors on 'br'/'recover' are expected (no agent
|
||||
// dirs in the test cwd); branch-locality errors land on the map.
|
||||
assert!(
|
||||
!result
|
||||
.errors
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("m")),
|
||||
"a branch-local agent fallback is allowed: {:?}",
|
||||
result.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn map_branch_llm_fallback_to_branch_local_script_passes() {
|
||||
let map = map_node_basic("m", "br", Some("end"));
|
||||
let branch = llm_node("br", Some("recover"), None);
|
||||
let recover = script_node("recover", "Cargo.toml", None);
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("m", map),
|
||||
("br", branch),
|
||||
("recover", recover),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"m",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(result.errors.is_empty(), "{:?}", result.errors);
|
||||
assert!(
|
||||
!result
|
||||
.warnings
|
||||
.iter()
|
||||
.any(|w| w.node_id.as_deref() == Some("m")),
|
||||
"a branch-local script fallback raises nothing on the map: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn map_branch_deep_approval_node_errors_on_the_map() {
|
||||
let map = map_node_basic("m", "br", Some("end"));
|
||||
@@ -4308,4 +4534,327 @@ mod tests {
|
||||
|
||||
assert_eq!(ids(&branch_subgraph(&graph, "a")), vec!["a"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn llm_max_attempts_zero_is_an_error() {
|
||||
let mut node = llm_node("l", None, Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.max_attempts = 0;
|
||||
}
|
||||
let graph = graph_with(vec![("l", node), ("end", end_node("end"))], "l");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result
|
||||
.errors
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("l")
|
||||
&& e.message.contains("`max_attempts` must be >= 1")),
|
||||
"expected max_attempts error: {:?}",
|
||||
result.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_max_attempts_zero_is_an_error() {
|
||||
let mut node = agent_node("a", "helper", Some("end"));
|
||||
if let NodeType::Agent(ref mut n) = node.node_type {
|
||||
n.max_attempts = 0;
|
||||
}
|
||||
let graph = graph_with(vec![("a", node), ("end", end_node("end"))], "a");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result
|
||||
.errors
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("a")
|
||||
&& e.message.contains("`max_attempts` must be >= 1")),
|
||||
"expected max_attempts error: {:?}",
|
||||
result.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn max_attempts_of_one_is_not_flagged() {
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", llm_node("l", None, Some("end"))),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
!result
|
||||
.errors
|
||||
.iter()
|
||||
.any(|e| e.message.contains("`max_attempts`")),
|
||||
"unexpected max_attempts error: {:?}",
|
||||
result.errors
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn llm_fallback_with_output_schema_and_no_state_updates_warns() {
|
||||
let mut node = llm_with_output_schema("l", &["summary"], Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.fallback = Some("fb".into());
|
||||
}
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", node),
|
||||
("fb", end_node("fb")),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result
|
||||
.warnings
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("l")
|
||||
&& e.message
|
||||
.contains("declares `fallback` with `output_schema`")),
|
||||
"expected fallback capture warning: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_fallback_with_output_schema_and_no_state_updates_warns() {
|
||||
let mut node = agent_node("a", "helper", Some("end"));
|
||||
if let NodeType::Agent(ref mut n) = node.node_type {
|
||||
n.fallback = Some("fb".into());
|
||||
n.output_schema = Some(serde_json::json!({ "type": "object" }));
|
||||
}
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("a", node),
|
||||
("fb", end_node("fb")),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"a",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result
|
||||
.warnings
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("a")
|
||||
&& e.message
|
||||
.contains("declares `fallback` with `output_schema`")),
|
||||
"expected fallback capture warning: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fallback_with_output_schema_and_state_updates_is_not_flagged() {
|
||||
let mut node = llm_with_output_schema("l", &["summary"], Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.fallback = Some("fb".into());
|
||||
n.state_updates = Some(HashMap::from([("summary".into(), "{{output}}".into())]));
|
||||
}
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", node),
|
||||
("fb", end_node("fb")),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
!result.warnings.iter().any(|e| e
|
||||
.message
|
||||
.contains("declares `fallback` with `output_schema`")),
|
||||
"unexpected fallback capture warning: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fallback_with_output_schema_and_empty_state_updates_warns() {
|
||||
let mut node = llm_with_output_schema("l", &["summary"], Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.fallback = Some("fb".into());
|
||||
n.state_updates = Some(HashMap::new());
|
||||
}
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", node),
|
||||
("fb", end_node("fb")),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
result
|
||||
.warnings
|
||||
.iter()
|
||||
.any(|e| e.node_id.as_deref() == Some("l")
|
||||
&& e.message
|
||||
.contains("declares `fallback` with `output_schema`")),
|
||||
"expected fallback capture warning: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn output_schema_without_fallback_is_not_flagged() {
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", llm_with_output_schema("l", &["summary"], Some("end"))),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
!result.warnings.iter().any(|e| e
|
||||
.message
|
||||
.contains("declares `fallback` with `output_schema`")),
|
||||
"unexpected fallback capture warning: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
fn bare_object_schema_warnings(result: &ValidationResult) -> Vec<&ValidationError> {
|
||||
result
|
||||
.warnings
|
||||
.iter()
|
||||
.filter(|e| {
|
||||
e.message
|
||||
.contains("declares no (non-empty) object `properties`")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn llm_output_schema_without_properties_warns() {
|
||||
let mut node = llm_node("l", None, Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.output_schema = Some(serde_json::json!({ "type": "object" }));
|
||||
}
|
||||
let graph = graph_with(vec![("l", node), ("end", end_node("end"))], "l");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
let warnings = bare_object_schema_warnings(&result);
|
||||
assert_eq!(warnings.len(), 1, "{:?}", result.warnings);
|
||||
assert_eq!(warnings[0].node_id.as_deref(), Some("l"));
|
||||
assert!(warnings[0].message.starts_with("llm node's"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_output_schema_without_properties_warns() {
|
||||
let mut node = agent_node("a", "helper", Some("end"));
|
||||
if let NodeType::Agent(ref mut n) = node.node_type {
|
||||
n.output_schema = Some(serde_json::json!({ "type": "object" }));
|
||||
}
|
||||
let graph = graph_with(vec![("a", node), ("end", end_node("end"))], "a");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
let warnings = bare_object_schema_warnings(&result);
|
||||
assert_eq!(warnings.len(), 1, "{:?}", result.warnings);
|
||||
assert_eq!(warnings[0].node_id.as_deref(), Some("a"));
|
||||
assert!(warnings[0].message.starts_with("agent node's"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn output_schema_with_empty_or_non_object_properties_warns() {
|
||||
for properties in [serde_json::json!({}), serde_json::json!([])] {
|
||||
let mut node = llm_node("l", None, Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.output_schema = Some(serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
}));
|
||||
}
|
||||
let graph = graph_with(vec![("l", node), ("end", end_node("end"))], "l");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
let warnings = bare_object_schema_warnings(&result);
|
||||
assert_eq!(
|
||||
warnings.len(),
|
||||
1,
|
||||
"properties={properties}: {:?}",
|
||||
result.warnings
|
||||
);
|
||||
assert_eq!(warnings[0].node_id.as_deref(), Some("l"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bare_output_schema_with_state_updates_is_not_flagged() {
|
||||
let mut node = llm_node("l", None, Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.output_schema = Some(serde_json::json!({ "type": "object" }));
|
||||
n.state_updates = Some(HashMap::from([("k".into(), "{{output}}".into())]));
|
||||
}
|
||||
let graph = graph_with(vec![("l", node), ("end", end_node("end"))], "l");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
bare_object_schema_warnings(&result).is_empty(),
|
||||
"{:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bare_output_schema_with_empty_state_updates_still_warns() {
|
||||
let mut node = llm_node("l", None, Some("end"));
|
||||
if let NodeType::Llm(ref mut n) = node.node_type {
|
||||
n.output_schema = Some(serde_json::json!({ "type": "object" }));
|
||||
n.state_updates = Some(HashMap::new());
|
||||
}
|
||||
let graph = graph_with(vec![("l", node), ("end", end_node("end"))], "l");
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
let warnings = bare_object_schema_warnings(&result);
|
||||
assert_eq!(warnings.len(), 1, "{:?}", result.warnings);
|
||||
assert_eq!(warnings[0].node_id.as_deref(), Some("l"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn output_schema_with_properties_or_absent_is_not_flagged() {
|
||||
let graph = graph_with(
|
||||
vec![
|
||||
("l", llm_with_output_schema("l", &["k"], Some("plain"))),
|
||||
("plain", llm_node("plain", None, Some("end"))),
|
||||
("end", end_node("end")),
|
||||
],
|
||||
"l",
|
||||
);
|
||||
|
||||
let result = validator().validate(&graph);
|
||||
|
||||
assert!(
|
||||
bare_object_schema_warnings(&result).is_empty(),
|
||||
"{:?}",
|
||||
result.warnings
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,8 @@ mod mcp;
|
||||
mod parsers;
|
||||
mod sandbox;
|
||||
mod supervisor;
|
||||
#[cfg(test)]
|
||||
mod testing;
|
||||
mod vault;
|
||||
|
||||
#[macro_use]
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
//! Test-only helpers shared across modules' unit tests.
|
||||
|
||||
use log::{Level, LevelFilter, Log, Metadata, Record};
|
||||
use std::sync::{Mutex, Once, OnceLock};
|
||||
|
||||
struct WarnCollector;
|
||||
|
||||
static WARN_MESSAGES: OnceLock<Mutex<Vec<String>>> = OnceLock::new();
|
||||
|
||||
/// Every warn-level message captured since the collector was installed. The
|
||||
/// buffer is process-global and shared by all tests, so assert by filtering
|
||||
/// for a marker unique to the test rather than on the whole buffer.
|
||||
pub(crate) fn warn_messages() -> &'static Mutex<Vec<String>> {
|
||||
WARN_MESSAGES.get_or_init(Mutex::default)
|
||||
}
|
||||
|
||||
impl Log for WarnCollector {
|
||||
fn enabled(&self, metadata: &Metadata) -> bool {
|
||||
metadata.level() <= Level::Warn
|
||||
}
|
||||
|
||||
fn log(&self, record: &Record) {
|
||||
if self.enabled(record.metadata()) {
|
||||
warn_messages()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(record.args().to_string());
|
||||
}
|
||||
}
|
||||
|
||||
fn flush(&self) {}
|
||||
}
|
||||
|
||||
/// Installs the process-wide warn collector. `log::set_logger` accepts one
|
||||
/// logger per process, so every test that captures warns must install through
|
||||
/// this shared entry point.
|
||||
pub(crate) fn install_warn_collector() {
|
||||
static INSTALL: Once = Once::new();
|
||||
INSTALL.call_once(|| {
|
||||
log::set_logger(&WarnCollector).expect("no other logger should be installed");
|
||||
log::set_max_level(LevelFilter::Warn);
|
||||
});
|
||||
}
|
||||
Reference in new issue
Block a user