feat(TASK-032): review-gauntlet incomplete-review machine line + ×N attempt details
verdict_gate.py now names every incomplete review on a machine line the done template renders immediately after the verdict: `GAUNTLET_REVIEW_INCOMPLETE: <name>[, <name>]` (sorted). Names are the lanes retry_gate gave up on (`review_incomplete`) plus the `pipeline` pseudo-lane whenever `signals_error` carries a `PIPELINE-FAULT:` — parse death, builder / retry-gate / default-lanes crash guards — and the gate's own crash guard emits the `pipeline` line too. The report gains a `## Review incomplete` section (after the signals_error blockquote, immediately before `## Full lane reports`) and the three PIPELINE-FAULT lane rows carry ` ×N` when the lane was re-run (N = 1 texts unchanged). retry_gate.py stands in a synthetic `PIPELINE-FAULT: <lane> lane produced no report` marker for a lane that ran but collected nothing, so it is stashed, classified FAULTED and never renders SKIPPED downstream. MAX_RETRY_ELAPSED_SECS trimmed 45600 → 45000 for margin slack. graph.yaml: `gauntlet_incomplete_line` seeded in initial_state and spliced into the done output. README + mermaid document retry_gate, the dotted _next back-edge, the 3-attempt / wall-clock budget, that retry never narrows selection, the crash-guard append semantics, and the machine-line contract (present whenever any lane or the pipeline was incomplete; it can accompany real findings). Tests: ×N on all three branches (N>1, N=1 byte-identical, no leak onto non-fault rows), exact incomplete line / section placement, pipeline pseudo-lane alone and alongside real lanes (via default_lanes.py's real crash guard), mixed finding+fault, crash-guard chaining from build_items / retry_gate / parse_fault / verdict_gate, synthetic no-report marker, done-template wiring, README pins (positive + stale negative).
This commit is contained in:
5 files changed
+567
-30
No files matched your search
@@ -26,8 +26,10 @@ flowchart TD
|
|||||||
radv -. "fallback" .-> lfault
|
radv -. "fallback" .-> lfault
|
||||||
rsec -. "fallback" .-> lfault
|
rsec -. "fallback" .-> lfault
|
||||||
rpb -. "fallback" .-> lfault
|
rpb -. "fallback" .-> lfault
|
||||||
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
|
||||||
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
rgate -. "_next: retry faulted lanes" .-> build
|
||||||
|
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||||
|
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
|
||||||
```
|
```
|
||||||
|
|
||||||
## Why a graph
|
## Why a graph
|
||||||
@@ -49,7 +51,8 @@ those rules are *structure*:
|
|||||||
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
||||||
over a 0-or-1-item list — an empty list means the map runs zero branches
|
over a 0-or-1-item list — an empty list means the map runs zero branches
|
||||||
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
||||||
the verdict gate on equal-length paths.
|
the retry gate on equal-length paths; it falls through to the verdict
|
||||||
|
gate, or routes back to `build_items` for faulted lanes only.
|
||||||
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
||||||
sentinel is a lane FAILURE, never a pass; the critical count from the
|
sentinel is a lane FAILURE, never a pass; the critical count from the
|
||||||
code-review report's summary line (raw 🔴 count if absent) blocks
|
code-review report's summary line (raw 🔴 count if absent) blocks
|
||||||
@@ -77,6 +80,45 @@ those rules are *structure*:
|
|||||||
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
||||||
`GAUNTLET: BLOCKED`, never silence.
|
`GAUNTLET: BLOCKED`, never silence.
|
||||||
|
|
||||||
|
## Fault-only re-runs
|
||||||
|
|
||||||
|
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
|
||||||
|
only lanes that **faulted** — a missing report (stood in for by a synthetic
|
||||||
|
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
|
||||||
|
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
|
||||||
|
report, or a nested graph's own degraded-run wording. A real verdict
|
||||||
|
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
|
||||||
|
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
|
||||||
|
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
|
||||||
|
|
||||||
|
- **Stash + restore**: every lane report that ran this pass is stashed in
|
||||||
|
`kept_results`; a re-run pass overwrites the untouched lanes' collected
|
||||||
|
results with `[]`, so their stashed reports are restored before the gate.
|
||||||
|
The verdict gate always sees each lane's *last* report.
|
||||||
|
- **Retry never narrows selection**: lane selection is not re-run. A re-run
|
||||||
|
pass rebuilds items only for the retried lanes, and the stashed results
|
||||||
|
stand in for the rest.
|
||||||
|
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
|
||||||
|
attempts or the budget lands in `review_incomplete`, and its fault flows to
|
||||||
|
the verdict gate as-is.
|
||||||
|
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
|
||||||
|
guards append to an existing `signals_error` rather than overwrite it, and
|
||||||
|
`build_items.py` preserves the prior `lanes_summary`.
|
||||||
|
|
||||||
|
Whenever at least one lane (or the pipeline) could not complete, the first
|
||||||
|
line reads `GAUNTLET: BLOCKED` followed immediately by
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
|
||||||
|
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
|
||||||
|
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
|
||||||
|
builder, the retry gate, the default-lanes fallback, or the verdict gate
|
||||||
|
itself (the default-lanes fallback's crash still runs the widened lanes, so
|
||||||
|
`pipeline` appears alongside real lane rows). The report gains a
|
||||||
|
`## Review incomplete` section naming each exhausted lane with its
|
||||||
|
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
|
||||||
|
re-run (N > 1). The *named* entries are faults to re-run, never code
|
||||||
|
findings; the line can accompany real findings from the other lanes, which
|
||||||
|
still appear under `## Blockers` and must be fixed as usual.
|
||||||
|
|
||||||
## Sentinels parsed
|
## Sentinels parsed
|
||||||
|
|
||||||
| Lane | Sentinel | Blocks on |
|
| Lane | Sentinel | Blocks on |
|
||||||
@@ -115,22 +157,29 @@ coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
|
|||||||
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
||||||
```
|
```
|
||||||
|
|
||||||
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
|
The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
|
||||||
table, the lane-selection reasons, blockers, attention items, and each lane's
|
table, the lane-selection reasons, blockers, attention items, and each lane's
|
||||||
full report in a collapsible section. On BLOCKED: fix the findings per your
|
full report in a collapsible section. On BLOCKED with a
|
||||||
own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with
|
`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — fix any
|
||||||
*Human attention required*: complete the work but carry those items into your
|
non-fault blockers listed under `## Blockers`, then spawn a **fresh** gauntlet
|
||||||
final report verbatim.
|
run (a fault is never a code finding). On BLOCKED without it: fix the findings
|
||||||
|
per your own findings-handling rules, then spawn a **fresh** gauntlet run. On
|
||||||
|
PASS with *Human attention required*: complete the work but carry those items
|
||||||
|
into your final report verbatim.
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||||
non-trivial work — that's why code-review is unconditionally on); the
|
non-trivial work — that's why code-review is unconditionally on); the
|
||||||
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet)
|
- Lane sub-agents get generous finite per-attempt timeouts (see the
|
||||||
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
|
`timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
|
||||||
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
|
the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
|
||||||
pass.
|
the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
|
||||||
|
is derived from that envelope in graph.yaml's comment. A lane that still
|
||||||
|
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
|
||||||
|
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
|
||||||
- The `verification_commands` variable is forwarded to the adversary lane as a
|
- The `verification_commands` variable is forwarded to the adversary lane as a
|
||||||
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
||||||
lands the raw value in the child's state) — never through the lane prompt
|
lands the raw value in the child's state) — never through the lane prompt
|
||||||
|
|||||||
@@ -93,6 +93,7 @@ initial_state:
|
|||||||
lanes_summary: ''
|
lanes_summary: ''
|
||||||
gauntlet_verdict: ''
|
gauntlet_verdict: ''
|
||||||
gauntlet_report: ''
|
gauntlet_report: ''
|
||||||
|
gauntlet_incomplete_line: ''
|
||||||
|
|
||||||
start: parse
|
start: parse
|
||||||
|
|
||||||
@@ -429,7 +430,7 @@ nodes:
|
|||||||
verdict_gate:
|
verdict_gate:
|
||||||
id: verdict_gate
|
id: verdict_gate
|
||||||
type: script
|
type: script
|
||||||
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane.
|
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
|
||||||
script: scripts/verdict_gate.py
|
script: scripts/verdict_gate.py
|
||||||
timeout: 60
|
timeout: 60
|
||||||
next: done
|
next: done
|
||||||
@@ -438,6 +439,6 @@ nodes:
|
|||||||
id: done
|
id: done
|
||||||
type: end
|
type: end
|
||||||
output: |
|
output: |
|
||||||
GAUNTLET: {{gauntlet_verdict}}
|
GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
|
||||||
|
|
||||||
{{gauntlet_report}}
|
{{gauntlet_report}}
|
||||||
@@ -10,7 +10,9 @@ hard rules:
|
|||||||
and faulted alike, and a lane whose map ran nothing this pass gets
|
and faulted alike, and a lane whose map ran nothing this pass gets
|
||||||
its stashed report restored — the verdict gate always sees each
|
its stashed report restored — the verdict gate always sees each
|
||||||
lane's LAST report;
|
lane's LAST report;
|
||||||
- a lane is FAULTED only when its report is missing, non-string, the
|
- a lane is FAULTED only when its report is missing (the map ran but
|
||||||
|
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
|
||||||
|
the verdict gate never reads it as SKIPPED), non-string, the
|
||||||
lane_fault.py marker, sentinel-less, or carries the nested graph's own
|
lane_fault.py marker, sentinel-less, or carries the nested graph's own
|
||||||
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
|
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
|
||||||
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
|
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
|
||||||
@@ -27,7 +29,9 @@ import os
|
|||||||
import re
|
import re
|
||||||
import time
|
import time
|
||||||
|
|
||||||
MAX_RETRY_ELAPSED_SECS = 45600
|
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
|
||||||
|
# = 45600; 600 s of slack below that.
|
||||||
|
MAX_RETRY_ELAPSED_SECS = 45000
|
||||||
MAX_ATTEMPTS = 3
|
MAX_ATTEMPTS = 3
|
||||||
|
|
||||||
# lane name -> (items key, results key)
|
# lane name -> (items key, results key)
|
||||||
@@ -114,7 +118,9 @@ def main():
|
|||||||
out[results_key] = [kept[lane]]
|
out[results_key] = [kept[lane]]
|
||||||
continue
|
continue
|
||||||
report = results[0] if results else None
|
report = results[0] if results else None
|
||||||
if report is not None:
|
if report is None:
|
||||||
|
report = f"PIPELINE-FAULT: {lane} lane produced no report"
|
||||||
|
out[results_key] = [report]
|
||||||
kept[lane] = report
|
kept[lane] = report
|
||||||
if is_faulted(lane, report):
|
if is_faulted(lane, report):
|
||||||
faulted.append(lane)
|
faulted.append(lane)
|
||||||
|
|||||||
@@ -8,10 +8,11 @@ re-reads, no judgment. The gate's hard rules:
|
|||||||
- a non-string lane result (the map collected an object instead of the
|
- a non-string lane result (the map collected an object instead of the
|
||||||
lane's rendered text) blocks — it cannot carry a sentinel;
|
lane's rendered text) blocks — it cannot carry a sentinel;
|
||||||
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
||||||
detected only when the report IS the lane_fault.py marker, i.e. it
|
detected only when the report IS the lane_fault.py marker or
|
||||||
STARTS with it — a real review merely quoting the marker flows to the
|
retry_gate.py's no-report marker, i.e. it STARTS with it — a real
|
||||||
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse
|
review merely quoting the marker flows to the normal rules) or a
|
||||||
fault) blocks — a degraded pipeline is never a pass;
|
PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
|
||||||
|
degraded pipeline is never a pass;
|
||||||
- a code-review lane whose OWN verdict line records a pipeline fault
|
- a code-review lane whose OWN verdict line records a pipeline fault
|
||||||
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
||||||
the nested graph completed, but with a dead domain lane, verifier, or
|
the nested graph completed, but with a dead domain lane, verifier, or
|
||||||
@@ -24,8 +25,16 @@ re-reads, no judgment. The gate's hard rules:
|
|||||||
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
||||||
treated as PASS or FAIL (fix the environment per the plan's local-run
|
treated as PASS or FAIL (fix the environment per the plan's local-run
|
||||||
recipe, then re-run);
|
recipe, then re-run);
|
||||||
|
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
|
||||||
|
re-ran it, so the reader can tell a first-pass death from an exhausted
|
||||||
|
retry budget;
|
||||||
|
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
|
||||||
|
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
|
||||||
|
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
|
||||||
|
incomplete" section: that is a fault to re-run, never a code finding;
|
||||||
- the gate itself must never crash into a silent pass: any internal
|
- the gate itself must never crash into a silent pass: any internal
|
||||||
error emits GAUNTLET: BLOCKED naming the error.
|
error emits GAUNTLET: BLOCKED naming the error, with the pipeline
|
||||||
|
marked incomplete.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -43,6 +52,7 @@ def main():
|
|||||||
state = load_state()
|
state = load_state()
|
||||||
blockers, attention, lane_lines, reports = [], [], [], []
|
blockers, attention, lane_lines, reports = [], [], [], []
|
||||||
malformed = set()
|
malformed = set()
|
||||||
|
lane_attempts = state.get("lane_attempts") or {}
|
||||||
|
|
||||||
def lane_report(key):
|
def lane_report(key):
|
||||||
arr = state.get(key) or []
|
arr = state.get(key) or []
|
||||||
@@ -57,6 +67,10 @@ def main():
|
|||||||
def record(lane, status, detail=""):
|
def record(lane, status, detail=""):
|
||||||
lane_lines.append(f"| {lane} | {status} | {detail} |")
|
lane_lines.append(f"| {lane} | {status} | {detail} |")
|
||||||
|
|
||||||
|
def attempts_suffix(lane):
|
||||||
|
n = int(lane_attempts.get(lane) or 0)
|
||||||
|
return f" ×{n}" if n > 1 else ""
|
||||||
|
|
||||||
def fault_excerpt(report):
|
def fault_excerpt(report):
|
||||||
line = next(
|
line = next(
|
||||||
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
|
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
|
||||||
@@ -64,10 +78,11 @@ def main():
|
|||||||
return line.strip()[:200]
|
return line.strip()[:200]
|
||||||
|
|
||||||
def is_lane_fault(report):
|
def is_lane_fault(report):
|
||||||
# A genuine lane fault is the ENTIRE report emitted by lane_fault.py,
|
# A genuine lane fault is the ENTIRE report — the lane_fault.py marker
|
||||||
# which always begins with the marker. Matching the marker anywhere
|
# or retry_gate.py's no-report marker — which always begins with
|
||||||
# would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass
|
# "PIPELINE-FAULT:". Matching the marker anywhere would let a real
|
||||||
# 🔴 counting and sentinel parsing.
|
# review that merely QUOTES it bypass 🔴 counting and sentinel
|
||||||
|
# parsing.
|
||||||
return report.strip().startswith("PIPELINE-FAULT:")
|
return report.strip().startswith("PIPELINE-FAULT:")
|
||||||
|
|
||||||
def lane_fault_blocker(lane, report):
|
def lane_fault_blocker(lane, report):
|
||||||
@@ -75,7 +90,7 @@ def main():
|
|||||||
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
||||||
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
||||||
)
|
)
|
||||||
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)")
|
record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
|
||||||
|
|
||||||
def malformed_blocker(lane, report):
|
def malformed_blocker(lane, report):
|
||||||
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
||||||
@@ -129,10 +144,11 @@ def main():
|
|||||||
blockers.append(
|
blockers.append(
|
||||||
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
||||||
)
|
)
|
||||||
|
suffix = attempts_suffix("code-review")
|
||||||
if m_fault and reds:
|
if m_fault and reds:
|
||||||
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴")
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
|
||||||
elif m_fault:
|
elif m_fault:
|
||||||
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)")
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
|
||||||
elif reds:
|
elif reds:
|
||||||
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
||||||
elif m and m.group(1) == "MERGE-READY":
|
elif m and m.group(1) == "MERGE-READY":
|
||||||
@@ -231,11 +247,27 @@ def main():
|
|||||||
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
||||||
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
||||||
signals_error = state.get("signals_error")
|
signals_error = state.get("signals_error")
|
||||||
|
incomplete = set(state.get("review_incomplete") or [])
|
||||||
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error:
|
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error:
|
||||||
blockers.append(f"pipeline: {signals_error}")
|
blockers.append(f"pipeline: {signals_error}")
|
||||||
|
incomplete.add("pipeline")
|
||||||
|
|
||||||
verdict = "BLOCKED" if blockers else "PASS"
|
verdict = "BLOCKED" if blockers else "PASS"
|
||||||
|
|
||||||
|
incomplete_lines = []
|
||||||
|
for name in sorted(incomplete):
|
||||||
|
if name == "pipeline":
|
||||||
|
incomplete_lines.append(f"- pipeline: {signals_error}")
|
||||||
|
elif n := int(lane_attempts.get(name) or 0):
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
|
||||||
|
else:
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review")
|
||||||
|
incomplete_line = (
|
||||||
|
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
|
||||||
|
if incomplete
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
|
||||||
md = "\n".join(
|
md = "\n".join(
|
||||||
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines
|
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines
|
||||||
)
|
)
|
||||||
@@ -246,11 +278,21 @@ def main():
|
|||||||
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
||||||
if state.get("signals_error"):
|
if state.get("signals_error"):
|
||||||
md += f"\n\n> {state['signals_error']}"
|
md += f"\n\n> {state['signals_error']}"
|
||||||
|
if incomplete_lines:
|
||||||
|
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
|
||||||
md += "\n\n## Full lane reports\n"
|
md += "\n\n## Full lane reports\n"
|
||||||
for lane, rep in reports:
|
for lane, rep in reports:
|
||||||
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
||||||
|
|
||||||
print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md}))
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"gauntlet_verdict": verdict,
|
||||||
|
"gauntlet_report": md,
|
||||||
|
"gauntlet_incomplete_line": incomplete_line,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -261,6 +303,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash into a sile
|
|||||||
{
|
{
|
||||||
"gauntlet_verdict": "BLOCKED",
|
"gauntlet_verdict": "BLOCKED",
|
||||||
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
||||||
|
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -8335,6 +8335,50 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn review_gauntlet_readme_states_retry_and_incomplete_contract() {
|
||||||
|
let readmes = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("assets/agents");
|
||||||
|
let gauntlet = read_to_string(readmes.join("review-gauntlet/README.md")).unwrap();
|
||||||
|
for edge in [
|
||||||
|
"mcr & madv & msec & mpb --> rgate[\"retry_gate",
|
||||||
|
"rgate --> gate[\"verdict_gate",
|
||||||
|
"rgate -. \"_next: retry faulted lanes\" .-> build",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
gauntlet.contains(edge),
|
||||||
|
"review-gauntlet README mermaid must route the lane maps through the retry gate: {edge}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(gauntlet.contains("the retry gate on equal-length paths"));
|
||||||
|
for claim in [
|
||||||
|
"when any lane or the pipeline was incomplete",
|
||||||
|
"Whenever at least one lane (or the pipeline) could not complete",
|
||||||
|
"GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]",
|
||||||
|
"`pipeline` pseudo-lane",
|
||||||
|
"the named lanes never completed",
|
||||||
|
"never narrows selection",
|
||||||
|
"up to 3 attempts",
|
||||||
|
"MAX_RETRY_ELAPSED_SECS",
|
||||||
|
"preserves the prior `lanes_summary`",
|
||||||
|
"append to an existing `signals_error`",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
gauntlet.contains(claim),
|
||||||
|
"review-gauntlet README must state the retry/incomplete contract: {claim}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
for stale in [
|
||||||
|
"on fault-only BLOCKED",
|
||||||
|
"there is nothing to fix in the code yet",
|
||||||
|
"3h whole-gauntlet",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
!gauntlet.contains(stale),
|
||||||
|
"review-gauntlet README must not carry the stale claim: {stale}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// The spawner's own `max_agent_depth` is checked against the child's
|
// The spawner's own `max_agent_depth` is checked against the child's
|
||||||
// absolute depth (root = 0), so every hop along a chain needs its own
|
// absolute depth (root = 0), so every hop along a chain needs its own
|
||||||
// limit >= child depth; +1 leaves one level of headroom.
|
// limit >= child depth; +1 leaves one level of headroom.
|
||||||
@@ -10916,6 +10960,10 @@ mod tests {
|
|||||||
.contains("verdict gate error"),
|
.contains("verdict gate error"),
|
||||||
"the internal error must be named in the report: {out}"
|
"the internal error must be named in the report: {out}"
|
||||||
);
|
);
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
"a crashed gate is an incomplete review, never a code finding: {out}"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -11105,6 +11153,269 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_verdict_gate_attempt_counts_on_faulted_rows() {
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let degraded = "# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — pipeline fault(s) recorded — degraded run; always-human trigger(s) fired\n\n## Walkthrough\n(no walkthrough provided)\n\n---\n*Reviewed 3 files, found 0 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*";
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"adversary": 3, "code-review": 2},
|
||||||
|
"code_review_results": [degraded],
|
||||||
|
"adversary_results": [GAUNTLET_ADVERSARY_FAULT],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed ×3) |"),
|
||||||
|
"a re-run faulted lane must show its attempt count: {report}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane ×2) |"),
|
||||||
|
"a re-run degraded lane must show its attempt count: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"code-review": 2},
|
||||||
|
"code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s); pipeline fault(s) recorded — degraded run\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"],
|
||||||
|
"adversary_results": [],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane ×2); 2 🔴 |"),
|
||||||
|
"the attempt count must sit inside the degraded detail, before the reds: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"adversary": 1},
|
||||||
|
"code_review_results": [],
|
||||||
|
"adversary_results": [GAUNTLET_ADVERSARY_FAULT],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed) |"),
|
||||||
|
"a single attempt must render without a suffix: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"code-review": 1},
|
||||||
|
"code_review_results": [degraded],
|
||||||
|
"adversary_results": [],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane) |"),
|
||||||
|
"a single-attempt degraded lane must render without a suffix: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"code-review": 1},
|
||||||
|
"code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s); pipeline fault(s) recorded — degraded run\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"],
|
||||||
|
"adversary_results": [],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| code-review | BLOCKED | PIPELINE-FAULT (degraded lane); 2 🔴 |"),
|
||||||
|
"a single-attempt degraded lane with reds must render without a suffix: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"lane_attempts": {"adversary": 2},
|
||||||
|
"code_review_results": [],
|
||||||
|
"adversary_results": ["ADVERSARIAL_REVIEW: DIVERGES"],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| adversary | BLOCKED | DIVERGES |"),
|
||||||
|
"a real verdict must render its own detail: {report}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!report.contains("×2"),
|
||||||
|
"the attempt suffix must stay on PIPELINE-FAULT rows only: {report}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_verdict_gate_incomplete_line_names_exhausted_lanes() {
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let state = json!({
|
||||||
|
"review_incomplete": ["probe", "adversary"],
|
||||||
|
"lane_attempts": {"adversary": 3, "probe": 3},
|
||||||
|
"code_review_results": [GAUNTLET_MERGE_READY],
|
||||||
|
"adversary_results": [GAUNTLET_ADVERSARY_FAULT],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": ["PIPELINE-FAULT: probe lane failed after retries — Agent node failed: boom"],
|
||||||
|
"signals_error": "git diff unavailable"
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}");
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary, probe",
|
||||||
|
"the machine line must name every exhausted lane, sorted: {out}"
|
||||||
|
);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains(
|
||||||
|
"## Review incomplete\n- adversary: no completed review after 3 attempt(s)\n- probe: no completed review after 3 attempt(s)\n\n## Full lane reports"
|
||||||
|
),
|
||||||
|
"the section must list the exhausted lanes sorted, with attempt counts, directly above the full lane reports: {report}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
report.find("> git diff unavailable").unwrap()
|
||||||
|
< report.find("## Review incomplete").unwrap(),
|
||||||
|
"the section must follow the signals_error blockquote: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// No lane_attempts at all (retry_gate never ran, or crashed before
|
||||||
|
// recording): the section still names the lane, without a count.
|
||||||
|
let state = json!({
|
||||||
|
"review_incomplete": ["probe"],
|
||||||
|
"code_review_results": [GAUNTLET_MERGE_READY],
|
||||||
|
"adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": ["PIPELINE-FAULT: probe lane failed after retries — Agent node failed: boom"]
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: probe",
|
||||||
|
"{out}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
out["gauntlet_report"].as_str().unwrap().contains(
|
||||||
|
"## Review incomplete\n- probe: no completed review\n\n## Full lane reports"
|
||||||
|
),
|
||||||
|
"a lane with no recorded attempts must render without a count: {out}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The incomplete line must coexist with real findings, never
|
||||||
|
// suppress them.
|
||||||
|
let state = json!({
|
||||||
|
"review_incomplete": ["adversary"],
|
||||||
|
"lane_attempts": {"adversary": 3},
|
||||||
|
"code_review_results": ["# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — 2 🔴 CRITICAL finding(s)\n\n---\n*Reviewed 3 files, found 2 critical, 0 warnings, 0 suggestions, 0 nitpicks (0 deferred by quality bar)*"],
|
||||||
|
"adversary_results": [GAUNTLET_ADVERSARY_FAULT],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}");
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary",
|
||||||
|
"{out}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
out["gauntlet_report"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("code-review: 2 🔴 CRITICAL finding(s) — fix before claiming done"),
|
||||||
|
"a real finding must still be named as a blocker alongside the incomplete line: {out}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"code_review_results": [GAUNTLET_MERGE_READY],
|
||||||
|
"adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(out["gauntlet_verdict"], "PASS", "{out}");
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "",
|
||||||
|
"a complete review must emit an empty line, not a missing key: {out}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!out["gauntlet_report"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("## Review incomplete"),
|
||||||
|
"a complete review must not render the section: {out}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_verdict_gate_pipeline_incomplete_alongside_real_lanes() {
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// default_lanes' own crash guard fired (the ordinary degraded fallback
|
||||||
|
// emits no PIPELINE-FAULT and is not incomplete): real lanes ran to a
|
||||||
|
// verdict, but the selection stage faulted, so the review is incomplete
|
||||||
|
// at the pipeline level even though no lane row is faulted.
|
||||||
|
let fault = run_gauntlet_script(
|
||||||
|
"default_lanes.py",
|
||||||
|
&json!({"forced_lanes": 42, "touches_auth": true}),
|
||||||
|
)["signals_error"]
|
||||||
|
.clone();
|
||||||
|
let fault = fault
|
||||||
|
.as_str()
|
||||||
|
.expect("default_lanes crash guard must fire on a non-list forced_lanes");
|
||||||
|
assert!(
|
||||||
|
fault.starts_with("PIPELINE-FAULT: lane-selection fallback crashed"),
|
||||||
|
"{fault}"
|
||||||
|
);
|
||||||
|
let state = json!({
|
||||||
|
"signals_error": fault,
|
||||||
|
"code_review_results": ["**Verdict: MERGE-READY**"],
|
||||||
|
"adversary_results": ["ADVERSARIAL_REVIEW: CONFORMS"],
|
||||||
|
"security_results": ["SECURITY_REVIEW: PASS"],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(out["gauntlet_verdict"], "BLOCKED", "{out}");
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
"a pipeline-level fault must name the pipeline pseudo-lane: {out}"
|
||||||
|
);
|
||||||
|
let report = out["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| code-review | GREEN | MERGE-READY |"),
|
||||||
|
"real lane verdicts must still render: {report}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
report.contains(
|
||||||
|
"## Review incomplete\n- pipeline: PIPELINE-FAULT: lane-selection fallback crashed"
|
||||||
|
),
|
||||||
|
"the section must carry the pipeline fault text: {report}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let state = json!({
|
||||||
|
"signals_error": fault,
|
||||||
|
"review_incomplete": ["adversary"],
|
||||||
|
"lane_attempts": {"adversary": 3},
|
||||||
|
"code_review_results": ["**Verdict: MERGE-READY**"],
|
||||||
|
"adversary_results": [GAUNTLET_ADVERSARY_FAULT],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
});
|
||||||
|
let out = run_gauntlet_script("verdict_gate.py", &state);
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: adversary, pipeline",
|
||||||
|
"exhausted lanes and the pipeline pseudo-lane must sort together: {out}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn gauntlet_verdict_gate_non_string_lane_result_blocks() {
|
fn gauntlet_verdict_gate_non_string_lane_result_blocks() {
|
||||||
if !cmd_available("python3") {
|
if !cmd_available("python3") {
|
||||||
@@ -11464,6 +11775,10 @@ mod tests {
|
|||||||
report.contains("provider exploded"),
|
report.contains("provider exploded"),
|
||||||
"the failure detail must survive into the report: {report}"
|
"the failure detail must survive into the report: {report}"
|
||||||
);
|
);
|
||||||
|
assert_eq!(
|
||||||
|
out["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
"a dead parse stage is an incomplete review under the pipeline pseudo-lane: {out}"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -11916,6 +12231,46 @@ mod tests {
|
|||||||
assert_eq!(out["review_incomplete"], json!(["adversary"]), "{out}");
|
assert_eq!(out["review_incomplete"], json!(["adversary"]), "{out}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_retry_gate_missing_report_becomes_synthetic_fault() {
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// The map ran an item but collected nothing: without a synthetic
|
||||||
|
// marker the lane would have no report to stash or restore and the
|
||||||
|
// verdict gate would render it SKIPPED.
|
||||||
|
let marker = "PIPELINE-FAULT: adversary lane produced no report";
|
||||||
|
let out = run_gauntlet_script(
|
||||||
|
"retry_gate.py",
|
||||||
|
&json!({
|
||||||
|
"adversary_items": [{"lane": "adversary"}],
|
||||||
|
"adversary_results": [],
|
||||||
|
"gauntlet_started_at": now_secs(),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert_eq!(out["adversary_results"], json!([marker]), "{out}");
|
||||||
|
assert_eq!(out["kept_results"]["adversary"], marker, "{out}");
|
||||||
|
assert_eq!(out["retry_lanes"], json!(["adversary"]), "{out}");
|
||||||
|
assert_eq!(out["_next"], "build_items", "{out}");
|
||||||
|
assert_eq!(out["lane_attempts"]["adversary"], 1, "{out}");
|
||||||
|
|
||||||
|
let verdict = run_gauntlet_script(
|
||||||
|
"verdict_gate.py",
|
||||||
|
&json!({
|
||||||
|
"code_review_results": [],
|
||||||
|
"adversary_results": out["adversary_results"],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": []
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let report = verdict["gauntlet_report"].as_str().unwrap();
|
||||||
|
assert!(
|
||||||
|
report.contains("| adversary | BLOCKED | PIPELINE-FAULT (lane failed) |"),
|
||||||
|
"a lane that ran but produced nothing must block, never SKIPPED: {report}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn gauntlet_retry_gate_crash_falls_through_without_retry() {
|
fn gauntlet_retry_gate_crash_falls_through_without_retry() {
|
||||||
if !cmd_available("python3") {
|
if !cmd_available("python3") {
|
||||||
@@ -11950,6 +12305,63 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_pipeline_crash_guards_name_pipeline_incomplete() {
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let out = run_gauntlet_script("build_items.py", &json!({"forced_lanes": 42}));
|
||||||
|
let gate = run_gauntlet_script(
|
||||||
|
"verdict_gate.py",
|
||||||
|
&json!({
|
||||||
|
"code_review_results": [],
|
||||||
|
"adversary_results": [],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": [],
|
||||||
|
"signals_error": out["signals_error"],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert_eq!(gate["gauntlet_verdict"], "BLOCKED", "{gate}");
|
||||||
|
assert_eq!(
|
||||||
|
gate["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
"an all-SKIPPED gate downstream of a builder crash must name the pipeline: {gate}"
|
||||||
|
);
|
||||||
|
|
||||||
|
let out = run_gauntlet_script(
|
||||||
|
"retry_gate.py",
|
||||||
|
&json!({
|
||||||
|
"lane_attempts": 42,
|
||||||
|
"adversary_items": [{"lane": "adversary"}],
|
||||||
|
"adversary_results": ["x"],
|
||||||
|
"signals_error": "git diff unavailable",
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
let crashed = out["signals_error"]
|
||||||
|
.as_str()
|
||||||
|
.expect("retry_gate crash guard must record signals_error");
|
||||||
|
assert!(
|
||||||
|
crashed.contains("PIPELINE-FAULT: retry gate crashed"),
|
||||||
|
"the pipeline name below must come from the crash guard, not the seed: {out}"
|
||||||
|
);
|
||||||
|
let gate = run_gauntlet_script(
|
||||||
|
"verdict_gate.py",
|
||||||
|
&json!({
|
||||||
|
"code_review_results": [],
|
||||||
|
"adversary_results": [],
|
||||||
|
"security_results": [],
|
||||||
|
"probe_results": [],
|
||||||
|
"signals_error": out["signals_error"],
|
||||||
|
"review_incomplete": out["review_incomplete"],
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
assert_eq!(gate["gauntlet_verdict"], "BLOCKED", "{gate}");
|
||||||
|
assert_eq!(
|
||||||
|
gate["gauntlet_incomplete_line"], "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
"the verdict gate must name the pipeline after a retry-gate crash: {gate}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// retry_gate.py decides "completed" with the same sentinel regexes
|
/// retry_gate.py decides "completed" with the same sentinel regexes
|
||||||
/// verdict_gate.py parses; a drift would re-run lanes the verdict gate
|
/// verdict_gate.py parses; a drift would re-run lanes the verdict gate
|
||||||
/// accepts, or accept lanes it cannot read.
|
/// accepts, or accept lanes it cannot read.
|
||||||
@@ -12247,6 +12659,32 @@ mod tests {
|
|||||||
assert!(script.exists(), "{} must exist", script.display());
|
assert!(script.exists(), "{} must exist", script.display());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gauntlet_done_template_carries_incomplete_line() {
|
||||||
|
use crate::graph::NodeType;
|
||||||
|
let graph = load_bundled_graph("review-gauntlet");
|
||||||
|
let NodeType::End(done) = &graph.get_node("done").unwrap().node_type else {
|
||||||
|
panic!("done must be an end node")
|
||||||
|
};
|
||||||
|
assert!(
|
||||||
|
done.output
|
||||||
|
.contains("GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}"),
|
||||||
|
"the incomplete line must follow the verdict immediately: {}",
|
||||||
|
done.output
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
done.output
|
||||||
|
.contains("{{gauntlet_incomplete_line}}\n\n{{gauntlet_report}}"),
|
||||||
|
"the report must follow the incomplete line after one blank line: {}",
|
||||||
|
done.output
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
graph.initial_state.get("gauntlet_incomplete_line"),
|
||||||
|
Some(&json!("")),
|
||||||
|
"initial_state must declare the incomplete line so the template key resolves"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// MAX_RETRY_ELAPSED_SECS in retry_gate.py and settings.timeout in
|
/// MAX_RETRY_ELAPSED_SECS in retry_gate.py and settings.timeout in
|
||||||
/// graph.yaml live in different files. A re-run admitted at the very end
|
/// graph.yaml live in different files. A re-run admitted at the very end
|
||||||
/// of the retry budget can still burn a full lane envelope (max_attempts
|
/// of the retry budget can still burn a full lane envelope (max_attempts
|
||||||
|
|||||||
Reference in new issue
Block a user