diff --git a/assets/agents/review-gauntlet/README.md b/assets/agents/review-gauntlet/README.md index 39785f6..d4aae09 100644 --- a/assets/agents/review-gauntlet/README.md +++ b/assets/agents/review-gauntlet/README.md @@ -26,8 +26,10 @@ flowchart TD radv -. "fallback" .-> lfault rsec -. "fallback" .-> lfault rpb -. "fallback" .-> lfault - mcr & madv & msec & mpb --> gate["verdict_gate
(script: sentinel regex — deterministic)"] - gate --> done(["GAUNTLET: PASS | BLOCKED"]) + mcr & madv & msec & mpb --> rgate["retry_gate
(script: fault-only re-run gate;
stash + restore, ≤3 attempts,
wall-clock budget)"] + rgate -. "_next: retry faulted lanes" .-> build + rgate --> gate["verdict_gate
(script: sentinel regex — deterministic)"] + gate --> done(["GAUNTLET: PASS | BLOCKED
(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"]) ``` ## Why a graph @@ -49,7 +51,8 @@ those rules are *structure*: - **Conditional lanes without dynamic routing**: each lane is a `map` node over a 0-or-1-item list — an empty list means the map runs zero branches (map-over-nothing), which *is* the skip mechanism. All four maps join at - the verdict gate on equal-length paths. + the retry gate on equal-length paths; it falls through to the verdict + gate, or routes back to `build_items` for faulted lanes only. - **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing sentinel is a lane FAILURE, never a pass; the critical count from the code-review report's summary line (raw 🔴 count if absent) blocks @@ -77,6 +80,45 @@ those rules are *structure*: `GAUNTLET:` sentinel is always emitted: even a crashed gate prints `GAUNTLET: BLOCKED`, never silence. +## Fault-only re-runs + +`retry_gate.py` sits between the lane maps and the verdict gate and re-runs +only lanes that **faulted** — a missing report (stood in for by a synthetic +`PIPELINE-FAULT: lane produced no report` marker so it never renders as +SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less +report, or a nested graph's own degraded-run wording. A real verdict +(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's +judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock +budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout. + +- **Stash + restore**: every lane report that ran this pass is stashed in + `kept_results`; a re-run pass overwrites the untouched lanes' collected + results with `[]`, so their stashed reports are restored before the gate. + The verdict gate always sees each lane's *last* report. +- **Retry never narrows selection**: lane selection is not re-run. A re-run + pass rebuilds items only for the retried lanes, and the stashed results + stand in for the rest. +- **Exhaustion is recorded, not hidden**: a lane still faulted after its + attempts or the budget lands in `review_incomplete`, and its fault flows to + the verdict gate as-is. +- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash + guards append to an existing `signals_error` rather than overwrite it, and + `build_items.py` preserves the prior `lanes_summary`. + +Whenever at least one lane (or the pipeline) could not complete, the first +line reads `GAUNTLET: BLOCKED` followed immediately by +`GAUNTLET_REVIEW_INCOMPLETE: [, ]`. The names are the exhausted +lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a +pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the +builder, the retry gate, the default-lanes fallback, or the verdict gate +itself (the default-lanes fallback's crash still runs the widened lanes, so +`pipeline` appears alongside real lane rows). The report gains a +`## Review incomplete` section naming each exhausted lane with its +attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was +re-run (N > 1). The *named* entries are faults to re-run, never code +findings; the line can accompany real findings from the other lanes, which +still appear under `## Blockers` and must be fixed as usual. + ## Sentinels parsed | Lane | Sentinel | Blocks on | @@ -115,22 +157,29 @@ coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \ --agent-variable verification_commands '["cargo test --all"]' "" ``` -The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status +The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the +`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status table, the lane-selection reasons, blockers, attention items, and each lane's -full report in a collapsible section. On BLOCKED: fix the findings per your -own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with -*Human attention required*: complete the work but carry those items into your -final report verbatim. +full report in a collapsible section. On BLOCKED with a +`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — fix any +non-fault blockers listed under `## Blockers`, then spawn a **fresh** gauntlet +run (a fault is never a code finding). On BLOCKED without it: fix the findings +per your own findings-handling rules, then spawn a **fresh** gauntlet run. On +PASS with *Human attention required*: complete the work but carry those items +into your final report verbatim. ## Notes - The caller decides *whether* review is warranted at all (the gauntlet is for non-trivial work — that's why code-review is unconditionally on); the gauntlet decides *which* of the other lanes apply and what the verdicts mean. -- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet) - and one retry (`max_attempts: 2`). A lane that still dies or times out falls - back to the `lane_fault` marker — BLOCKED naming the lane, never a silent - pass. +- Lane sub-agents get generous finite per-attempt timeouts (see the + `timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside + the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while + the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout` + is derived from that envelope in graph.yaml's comment. A lane that still + dies or times out falls back to the `lane_fault` marker — BLOCKED naming the + lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass. - The `verification_commands` variable is forwarded to the adversary lane as a structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template lands the raw value in the child's state) — never through the lane prompt diff --git a/assets/agents/review-gauntlet/graph.yaml b/assets/agents/review-gauntlet/graph.yaml index e2aeaf5..cc58f67 100644 --- a/assets/agents/review-gauntlet/graph.yaml +++ b/assets/agents/review-gauntlet/graph.yaml @@ -93,6 +93,7 @@ initial_state: lanes_summary: '' gauntlet_verdict: '' gauntlet_report: '' + gauntlet_incomplete_line: '' start: parse @@ -429,7 +430,7 @@ nodes: verdict_gate: id: verdict_gate type: script - description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. + description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one. script: scripts/verdict_gate.py timeout: 60 next: done @@ -438,6 +439,6 @@ nodes: id: done type: end output: | - GAUNTLET: {{gauntlet_verdict}} + GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}} {{gauntlet_report}} diff --git a/assets/agents/review-gauntlet/scripts/retry_gate.py b/assets/agents/review-gauntlet/scripts/retry_gate.py index 2831e47..45f1ecd 100644 --- a/assets/agents/review-gauntlet/scripts/retry_gate.py +++ b/assets/agents/review-gauntlet/scripts/retry_gate.py @@ -10,7 +10,9 @@ hard rules: and faulted alike, and a lane whose map ran nothing this pass gets its stashed report restored — the verdict gate always sees each lane's LAST report; - - a lane is FAULTED only when its report is missing, non-string, the + - a lane is FAULTED only when its report is missing (the map ran but + collected nothing — replaced by a synthetic PIPELINE-FAULT marker so + the verdict gate never reads it as SKIPPED), non-string, the lane_fault.py marker, sentinel-less, or carries the nested graph's own degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run — the review's judgment stands; @@ -27,7 +29,9 @@ import os import re import time -MAX_RETRY_ELAPSED_SECS = 45600 +# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages) +# = 45600; 600 s of slack below that. +MAX_RETRY_ELAPSED_SECS = 45000 MAX_ATTEMPTS = 3 # lane name -> (items key, results key) @@ -114,8 +118,10 @@ def main(): out[results_key] = [kept[lane]] continue report = results[0] if results else None - if report is not None: - kept[lane] = report + if report is None: + report = f"PIPELINE-FAULT: {lane} lane produced no report" + out[results_key] = [report] + kept[lane] = report if is_faulted(lane, report): faulted.append(lane) attempts[lane] = int(attempts.get(lane) or 0) + 1 diff --git a/assets/agents/review-gauntlet/scripts/verdict_gate.py b/assets/agents/review-gauntlet/scripts/verdict_gate.py index f995a0d..cbeab90 100644 --- a/assets/agents/review-gauntlet/scripts/verdict_gate.py +++ b/assets/agents/review-gauntlet/scripts/verdict_gate.py @@ -8,10 +8,11 @@ re-reads, no judgment. The gate's hard rules: - a non-string lane result (the map collected an object instead of the lane's rendered text) blocks — it cannot carry a sentinel; - a PIPELINE-FAULT lane result (the lane's agent died after retries; - detected only when the report IS the lane_fault.py marker, i.e. it - STARTS with it — a real review merely quoting the marker flows to the - normal rules) or a PIPELINE-FAULT in signals_error (builder/parse - fault) blocks — a degraded pipeline is never a pass; + detected only when the report IS the lane_fault.py marker or + retry_gate.py's no-report marker, i.e. it STARTS with it — a real + review merely quoting the marker flows to the normal rules) or a + PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a + degraded pipeline is never a pass; - a code-review lane whose OWN verdict line records a pipeline fault (code-reviewer's verdict.py / render.py degraded-run wording) blocks — the nested graph completed, but with a dead domain lane, verifier, or @@ -24,8 +25,16 @@ re-reads, no judgment. The gate's hard rules: - probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never treated as PASS or FAIL (fix the environment per the plan's local-run recipe, then re-run); + - a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate + re-ran it, so the reader can tell a first-pass death from an exhausted + retry budget; + - an incomplete review — lanes retry_gate gave up on (review_incomplete) + or a pipeline-level fault (reported under the `pipeline` pseudo-lane) — + emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review + incomplete" section: that is a fault to re-run, never a code finding; - the gate itself must never crash into a silent pass: any internal - error emits GAUNTLET: BLOCKED naming the error. + error emits GAUNTLET: BLOCKED naming the error, with the pipeline + marked incomplete. """ import json @@ -43,6 +52,7 @@ def main(): state = load_state() blockers, attention, lane_lines, reports = [], [], [], [] malformed = set() + lane_attempts = state.get("lane_attempts") or {} def lane_report(key): arr = state.get(key) or [] @@ -57,6 +67,10 @@ def main(): def record(lane, status, detail=""): lane_lines.append(f"| {lane} | {status} | {detail} |") + def attempts_suffix(lane): + n = int(lane_attempts.get(lane) or 0) + return f" ×{n}" if n > 1 else "" + def fault_excerpt(report): line = next( (ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report @@ -64,10 +78,11 @@ def main(): return line.strip()[:200] def is_lane_fault(report): - # A genuine lane fault is the ENTIRE report emitted by lane_fault.py, - # which always begins with the marker. Matching the marker anywhere - # would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass - # 🔴 counting and sentinel parsing. + # A genuine lane fault is the ENTIRE report — the lane_fault.py marker + # or retry_gate.py's no-report marker — which always begins with + # "PIPELINE-FAULT:". Matching the marker anywhere would let a real + # review that merely QUOTES it bypass 🔴 counting and sentinel + # parsing. return report.strip().startswith("PIPELINE-FAULT:") def lane_fault_blocker(lane, report): @@ -75,7 +90,7 @@ def main(): f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced " f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})" ) - record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)") + record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})") def malformed_blocker(lane, report): blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass") @@ -129,10 +144,11 @@ def main(): blockers.append( f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done" ) + suffix = attempts_suffix("code-review") if m_fault and reds: - record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴") + record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴") elif m_fault: - record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)") + record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})") elif reds: record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)") elif m and m.group(1) == "MERGE-READY": @@ -231,11 +247,27 @@ def main(): # A gauntlet-level fault (build_items crash, dead parse stage) means the # lanes above were never built — SKIPPED rows alone must not read as PASS. signals_error = state.get("signals_error") + incomplete = set(state.get("review_incomplete") or []) if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error: blockers.append(f"pipeline: {signals_error}") + incomplete.add("pipeline") verdict = "BLOCKED" if blockers else "PASS" + incomplete_lines = [] + for name in sorted(incomplete): + if name == "pipeline": + incomplete_lines.append(f"- pipeline: {signals_error}") + elif n := int(lane_attempts.get(name) or 0): + incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)") + else: + incomplete_lines.append(f"- {name}: no completed review") + incomplete_line = ( + "\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete)) + if incomplete + else "" + ) + md = "\n".join( ["| lane | status | detail |", "|------|--------|--------|"] + lane_lines ) @@ -246,11 +278,21 @@ def main(): md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention) if state.get("signals_error"): md += f"\n\n> {state['signals_error']}" + if incomplete_lines: + md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines) md += "\n\n## Full lane reports\n" for lane, rep in reports: md += f"\n
{lane}\n\n{rep}\n\n
\n" - print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md})) + print( + json.dumps( + { + "gauntlet_verdict": verdict, + "gauntlet_report": md, + "gauntlet_incomplete_line": incomplete_line, + } + ) + ) try: @@ -261,6 +303,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash into a sile { "gauntlet_verdict": "BLOCKED", "gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.", + "gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", } ) ) diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 7710ef5..7261f8e 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -8335,6 +8335,50 @@ mod tests { ); } + #[test] + fn review_gauntlet_readme_states_retry_and_incomplete_contract() { + let readmes = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("assets/agents"); + let gauntlet = read_to_string(readmes.join("review-gauntlet/README.md")).unwrap(); + for edge in [ + "mcr & madv & msec & mpb --> rgate[\"retry_gate", + "rgate --> gate[\"verdict_gate", + "rgate -. \"_next: retry faulted lanes\" .-> build", + ] { + assert!( + gauntlet.contains(edge), + "review-gauntlet README mermaid must route the lane maps through the retry gate: {edge}" + ); + } + assert!(gauntlet.contains("the retry gate on equal-length paths")); + for claim in [ + "when any lane or the pipeline was incomplete", + "Whenever at least one lane (or the pipeline) could not complete", + "GAUNTLET_REVIEW_INCOMPLETE: [, ]", + "`pipeline` pseudo-lane", + "the named lanes never completed", + "never narrows selection", + "up to 3 attempts", + "MAX_RETRY_ELAPSED_SECS", + "preserves the prior `lanes_summary`", + "append to an existing `signals_error`", + ] { + assert!( + gauntlet.contains(claim), + "review-gauntlet README must state the retry/incomplete contract: {claim}" + ); + } + for stale in [ + "on fault-only BLOCKED", + "there is nothing to fix in the code yet", + "3h whole-gauntlet", + ] { + assert!( + !gauntlet.contains(stale), + "review-gauntlet README must not carry the stale claim: {stale}" + ); + } + } + // The spawner's own `max_agent_depth` is checked against the child's // absolute depth (root = 0), so every hop along a chain needs its own // limit >= child depth; +1 leaves one level of headroom. @@ -10916,6 +10960,10 @@ mod tests { .contains("verdict gate error"), "the internal error must be named in the report: {out}" ); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", + "a crashed gate is an incomplete review, never a code finding: {out}" + ); } #[test] @@ -11105,6 +11153,269 @@ mod tests { ); } + #[test] + fn gauntlet_verdict_gate_attempt_counts_on_faulted_rows() { + if !cmd_available("python3") { + eprintln!("skipping: python3 not available"); + return; + } + let degraded = "# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — pipeline fault(s) recorded — degraded run; always-human trigger(s) fired\n\n## Walkthrough\n(no walkthrough provided)\n\n---\n*Reviewed 3 files, found 0 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"; + let state = json!({ + "lane_attempts": {"adversary": 3, "code-review": 2}, + "code_review_results": [degraded], + "adversary_results": [GAUNTLET_ADVERSARY_FAULT], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed ×3) |"), + "a re-run faulted lane must show its attempt count: {report}" + ); + assert!( + report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane ×2) |"), + "a re-run degraded lane must show its attempt count: {report}" + ); + + let state = json!({ + "lane_attempts": {"code-review": 2}, + "code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s); pipeline fault(s) recorded — degraded run\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"], + "adversary_results": [], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane ×2); 2 🔴 |"), + "the attempt count must sit inside the degraded detail, before the reds: {report}" + ); + + let state = json!({ + "lane_attempts": {"adversary": 1}, + "code_review_results": [], + "adversary_results": [GAUNTLET_ADVERSARY_FAULT], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed) |"), + "a single attempt must render without a suffix: {report}" + ); + + let state = json!({ + "lane_attempts": {"code-review": 1}, + "code_review_results": [degraded], + "adversary_results": [], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane) |"), + "a single-attempt degraded lane must render without a suffix: {report}" + ); + + let state = json!({ + "lane_attempts": {"code-review": 1}, + "code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s); pipeline fault(s) recorded — degraded run\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"], + "adversary_results": [], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane); 2 🔴 |"), + "a single-attempt degraded lane with reds must render without a suffix: {report}" + ); + + let state = json!({ + "lane_attempts": {"adversary": 2}, + "code_review_results": [], + "adversary_results": ["ADVERSARIAL_REVIEW: DIVERGES"], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| adversary | BLOCKED | DIVERGES |"), + "a real verdict must render its own detail: {report}" + ); + assert!( + !report.contains("×2"), + "the attempt suffix must stay on PIPELINE-FAULT rows only: {report}" + ); + } + + #[test] + fn gauntlet_verdict_gate_incomplete_line_names_exhausted_lanes() { + if !cmd_available("python3") { + eprintln!("skipping: python3 not available"); + return; + } + let state = json!({ + "review_incomplete": ["probe", "adversary"], + "lane_attempts": {"adversary": 3, "probe": 3}, + "code_review_results": [GAUNTLET_MERGE_READY], + "adversary_results": [GAUNTLET_ADVERSARY_FAULT], + "security_results": [], + "probe_results": ["PIPELINE-FAULT: probe lane failed after retries — Agent node failed: boom"], + "signals_error": "git diff unavailable" + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}"); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary, probe", + "the machine line must name every exhausted lane, sorted: {out}" + ); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains( + "## Review incomplete\n- adversary: no completed review after 3 attempt(s)\n- probe: no completed review after 3 attempt(s)\n\n## Full lane reports" + ), + "the section must list the exhausted lanes sorted, with attempt counts, directly above the full lane reports: {report}" + ); + assert!( + report.find("> git diff unavailable").unwrap() + < report.find("## Review incomplete").unwrap(), + "the section must follow the signals_error blockquote: {report}" + ); + + // No lane_attempts at all (retry_gate never ran, or crashed before + // recording): the section still names the lane, without a count. + let state = json!({ + "review_incomplete": ["probe"], + "code_review_results": [GAUNTLET_MERGE_READY], + "adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"], + "security_results": [], + "probe_results": ["PIPELINE-FAULT: probe lane failed after retries — Agent node failed: boom"] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: probe", + "{out}" + ); + assert!( + out["gauntlet_report"].as_str().unwrap().contains( + "## Review incomplete\n- probe: no completed review\n\n## Full lane reports" + ), + "a lane with no recorded attempts must render without a count: {out}" + ); + + // The incomplete line must coexist with real findings, never + // suppress them. + let state = json!({ + "review_incomplete": ["adversary"], + "lane_attempts": {"adversary": 3}, + "code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s)\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"], + "adversary_results": [GAUNTLET_ADVERSARY_FAULT], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}"); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary", + "{out}" + ); + assert!( + out["gauntlet_report"] + .as_str() + .unwrap() + .contains("code-review: 2 🔴 CRITICAL finding(s) — fix before claiming done"), + "a real finding must still be named as a blocker alongside the incomplete line: {out}" + ); + + let state = json!({ + "code_review_results": [GAUNTLET_MERGE_READY], + "adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!(out["gauntlet_verdict"], "PASS", "{out}"); + assert_eq!( + out["gauntlet_incomplete_line"], "", + "a complete review must emit an empty line, not a missing key: {out}" + ); + assert!( + !out["gauntlet_report"] + .as_str() + .unwrap() + .contains("## Review incomplete"), + "a complete review must not render the section: {out}" + ); + } + + #[test] + fn gauntlet_verdict_gate_pipeline_incomplete_alongside_real_lanes() { + if !cmd_available("python3") { + eprintln!("skipping: python3 not available"); + return; + } + // default_lanes' own crash guard fired (the ordinary degraded fallback + // emits no PIPELINE-FAULT and is not incomplete): real lanes ran to a + // verdict, but the selection stage faulted, so the review is incomplete + // at the pipeline level even though no lane row is faulted. + let fault = run_gauntlet_script( + "default_lanes.py", + &json!({"forced_lanes": 42, "touches_auth": true}), + )["signals_error"] + .clone(); + let fault = fault + .as_str() + .expect("default_lanes crash guard must fire on a non-list forced_lanes"); + assert!( + fault.starts_with("PIPELINE-FAULT: lane-selection fallback crashed"), + "{fault}" + ); + let state = json!({ + "signals_error": fault, + "code_review_results": ["**Verdict: MERGE-READY**"], + "adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"], + "security_results": ["SECURITY_REVIEW: PASS"], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}"); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", + "a pipeline-level fault must name the pipeline pseudo-lane: {out}" + ); + let report = out["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| code-review | GREEN | MERGE-READY |"), + "real lane verdicts must still render: {report}" + ); + assert!( + report.contains( + "## Review incomplete\n- pipeline: PIPELINE-FAULT: lane-selection fallback crashed" + ), + "the section must carry the pipeline fault text: {report}" + ); + + let state = json!({ + "signals_error": fault, + "review_incomplete": ["adversary"], + "lane_attempts": {"adversary": 3}, + "code_review_results": ["**Verdict: MERGE-READY**"], + "adversary_results": [GAUNTLET_ADVERSARY_FAULT], + "security_results": [], + "probe_results": [] + }); + let out = run_gauntlet_script("verdict_gate.py", &state); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary, pipeline", + "exhausted lanes and the pipeline pseudo-lane must sort together: {out}" + ); + } + #[test] fn gauntlet_verdict_gate_non_string_lane_result_blocks() { if !cmd_available("python3") { @@ -11464,6 +11775,10 @@ mod tests { report.contains("provider exploded"), "the failure detail must survive into the report: {report}" ); + assert_eq!( + out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", + "a dead parse stage is an incomplete review under the pipeline pseudo-lane: {out}" + ); } #[test] @@ -11916,6 +12231,46 @@ mod tests { assert_eq!(out["review_incomplete"], json!(["adversary"]), "{out}"); } + #[test] + fn gauntlet_retry_gate_missing_report_becomes_synthetic_fault() { + if !cmd_available("python3") { + eprintln!("skipping: python3 not available"); + return; + } + // The map ran an item but collected nothing: without a synthetic + // marker the lane would have no report to stash or restore and the + // verdict gate would render it SKIPPED. + let marker = "PIPELINE-FAULT: adversary lane produced no report"; + let out = run_gauntlet_script( + "retry_gate.py", + &json!({ + "adversary_items": [{"lane": "adversary"}], + "adversary_results": [], + "gauntlet_started_at": now_secs(), + }), + ); + assert_eq!(out["adversary_results"], json!([marker]), "{out}"); + assert_eq!(out["kept_results"]["adversary"], marker, "{out}"); + assert_eq!(out["retry_lanes"], json!(["adversary"]), "{out}"); + assert_eq!(out["_next"], "build_items", "{out}"); + assert_eq!(out["lane_attempts"]["adversary"], 1, "{out}"); + + let verdict = run_gauntlet_script( + "verdict_gate.py", + &json!({ + "code_review_results": [], + "adversary_results": out["adversary_results"], + "security_results": [], + "probe_results": [] + }), + ); + let report = verdict["gauntlet_report"].as_str().unwrap(); + assert!( + report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed) |"), + "a lane that ran but produced nothing must block, never SKIPPED: {report}" + ); + } + #[test] fn gauntlet_retry_gate_crash_falls_through_without_retry() { if !cmd_available("python3") { @@ -11950,6 +12305,63 @@ mod tests { ); } + #[test] + fn gauntlet_pipeline_crash_guards_name_pipeline_incomplete() { + if !cmd_available("python3") { + eprintln!("skipping: python3 not available"); + return; + } + let out = run_gauntlet_script("build_items.py", &json!({"forced_lanes": 42})); + let gate = run_gauntlet_script( + "verdict_gate.py", + &json!({ + "code_review_results": [], + "adversary_results": [], + "security_results": [], + "probe_results": [], + "signals_error": out["signals_error"], + }), + ); + assert_eq!(gate["gauntlet_verdict"], "BLOCKED", "{gate}"); + assert_eq!( + gate["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", + "an all-SKIPPED gate downstream of a builder crash must name the pipeline: {gate}" + ); + + let out = run_gauntlet_script( + "retry_gate.py", + &json!({ + "lane_attempts": 42, + "adversary_items": [{"lane": "adversary"}], + "adversary_results": ["x"], + "signals_error": "git diff unavailable", + }), + ); + let crashed = out["signals_error"] + .as_str() + .expect("retry_gate crash guard must record signals_error"); + assert!( + crashed.contains("PIPELINE-FAULT: retry gate crashed"), + "the pipeline name below must come from the crash guard, not the seed: {out}" + ); + let gate = run_gauntlet_script( + "verdict_gate.py", + &json!({ + "code_review_results": [], + "adversary_results": [], + "security_results": [], + "probe_results": [], + "signals_error": out["signals_error"], + "review_incomplete": out["review_incomplete"], + }), + ); + assert_eq!(gate["gauntlet_verdict"], "BLOCKED", "{gate}"); + assert_eq!( + gate["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline", + "the verdict gate must name the pipeline after a retry-gate crash: {gate}" + ); + } + /// retry_gate.py decides "completed" with the same sentinel regexes /// verdict_gate.py parses; a drift would re-run lanes the verdict gate /// accepts, or accept lanes it cannot read. @@ -12247,6 +12659,32 @@ mod tests { assert!(script.exists(), "{} must exist", script.display()); } + #[test] + fn gauntlet_done_template_carries_incomplete_line() { + use crate::graph::NodeType; + let graph = load_bundled_graph("review-gauntlet"); + let NodeType::End(done) = &graph.get_node("done").unwrap().node_type else { + panic!("done must be an end node") + }; + assert!( + done.output + .contains("GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}"), + "the incomplete line must follow the verdict immediately: {}", + done.output + ); + assert!( + done.output + .contains("{{gauntlet_incomplete_line}}\n\n{{gauntlet_report}}"), + "the report must follow the incomplete line after one blank line: {}", + done.output + ); + assert_eq!( + graph.initial_state.get("gauntlet_incomplete_line"), + Some(&json!("")), + "initial_state must declare the incomplete line so the template key resolves" + ); + } + /// MAX_RETRY_ELAPSED_SECS in retry_gate.py and settings.timeout in /// graph.yaml live in different files. A re-run admitted at the very end /// of the retry budget can still burn a full lane envelope (max_attempts