feat(TASK-032): review-gauntlet incomplete-review machine line + ×N attempt details

verdict_gate.py now names every incomplete review on a machine line the
done template renders immediately after the verdict:
`GAUNTLET_REVIEW_INCOMPLETE: <name>[, <name>]` (sorted). Names are the
lanes retry_gate gave up on (`review_incomplete`) plus the `pipeline`
pseudo-lane whenever `signals_error` carries a `PIPELINE-FAULT:` —
parse death, builder / retry-gate / default-lanes crash guards — and the
gate's own crash guard emits the `pipeline` line too. The report gains a
`## Review incomplete` section (after the signals_error blockquote,
immediately before `## Full lane reports`) and the three PIPELINE-FAULT
lane rows carry ` ×N` when the lane was re-run (N = 1 texts unchanged).

retry_gate.py stands in a synthetic `PIPELINE-FAULT: <lane> lane produced
no report` marker for a lane that ran but collected nothing, so it is
stashed, classified FAULTED and never renders SKIPPED downstream.
MAX_RETRY_ELAPSED_SECS trimmed 45600 → 45000 for margin slack.

graph.yaml: `gauntlet_incomplete_line` seeded in initial_state and
spliced into the done output. README + mermaid document retry_gate, the
dotted _next back-edge, the 3-attempt / wall-clock budget, that retry
never narrows selection, the crash-guard append semantics, and the
machine-line contract (present whenever any lane or the pipeline was
incomplete; it can accompany real findings).

Tests: ×N on all three branches (N>1, N=1 byte-identical, no leak onto
non-fault rows), exact incomplete line / section placement, pipeline
pseudo-lane alone and alongside real lanes (via default_lanes.py's real
crash guard), mixed finding+fault, crash-guard chaining from
build_items / retry_gate / parse_fault / verdict_gate, synthetic
no-report marker, done-template wiring, README pins (positive + stale
negative).
This commit is contained in:
Dark-Alex-17 committed 2026-09-14 11:13:34 -06:00
1 parent 18f5ada585
commit 786e5542d4
5 files changed
+568 -31

No files matched your search

+61 -12
View File
@@ -26,8 +26,10 @@ flowchart TD
radv -. "fallback" .-> lfault
rsec -. "fallback" .-> lfault
rpb -. "fallback" .-> lfault
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
gate --> done(["GAUNTLET: PASS | BLOCKED"])
mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
rgate -. "_next: retry faulted lanes" .-> build
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
```
## Why a graph
@@ -49,7 +51,8 @@ those rules are *structure*:
- **Conditional lanes without dynamic routing**: each lane is a `map` node
over a 0-or-1-item list — an empty list means the map runs zero branches
(map-over-nothing), which *is* the skip mechanism. All four maps join at
the verdict gate on equal-length paths.
the retry gate on equal-length paths; it falls through to the verdict
gate, or routes back to `build_items` for faulted lanes only.
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
sentinel is a lane FAILURE, never a pass; the critical count from the
code-review report's summary line (raw 🔴 count if absent) blocks
@@ -77,6 +80,45 @@ those rules are *structure*:
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
`GAUNTLET: BLOCKED`, never silence.
## Fault-only re-runs
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
only lanes that **faulted** — a missing report (stood in for by a synthetic
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
report, or a nested graph's own degraded-run wording. A real verdict
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
- **Stash + restore**: every lane report that ran this pass is stashed in
`kept_results`; a re-run pass overwrites the untouched lanes' collected
results with `[]`, so their stashed reports are restored before the gate.
The verdict gate always sees each lane's *last* report.
- **Retry never narrows selection**: lane selection is not re-run. A re-run
pass rebuilds items only for the retried lanes, and the stashed results
stand in for the rest.
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
attempts or the budget lands in `review_incomplete`, and its fault flows to
the verdict gate as-is.
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
guards append to an existing `signals_error` rather than overwrite it, and
`build_items.py` preserves the prior `lanes_summary`.
Whenever at least one lane (or the pipeline) could not complete, the first
line reads `GAUNTLET: BLOCKED` followed immediately by
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
builder, the retry gate, the default-lanes fallback, or the verdict gate
itself (the default-lanes fallback's crash still runs the widened lanes, so
`pipeline` appears alongside real lane rows). The report gains a
`## Review incomplete` section naming each exhausted lane with its
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
re-run (N > 1). The *named* entries are faults to re-run, never code
findings; the line can accompany real findings from the other lanes, which
still appear under `## Blockers` and must be fixed as usual.
## Sentinels parsed
| Lane | Sentinel | Blocks on |
@@ -115,22 +157,29 @@ coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
```
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
table, the lane-selection reasons, blockers, attention items, and each lane's
full report in a collapsible section. On BLOCKED: fix the findings per your
own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with
*Human attention required*: complete the work but carry those items into your
final report verbatim.
full report in a collapsible section. On BLOCKED with a
`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — fix any
non-fault blockers listed under `## Blockers`, then spawn a **fresh** gauntlet
run (a fault is never a code finding). On BLOCKED without it: fix the findings
per your own findings-handling rules, then spawn a **fresh** gauntlet run. On
PASS with *Human attention required*: complete the work but carry those items
into your final report verbatim.
## Notes
- The caller decides *whether* review is warranted at all (the gauntlet is for
non-trivial work — that's why code-review is unconditionally on); the
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet)
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
pass.
- Lane sub-agents get generous finite per-attempt timeouts (see the
`timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
is derived from that envelope in graph.yaml's comment. A lane that still
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
- The `verification_commands` variable is forwarded to the adversary lane as a
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
lands the raw value in the child's state) — never through the lane prompt
+3 -2
View File
@@ -93,6 +93,7 @@ initial_state:
lanes_summary: ''
gauntlet_verdict: ''
gauntlet_report: ''
gauntlet_incomplete_line: ''
start: parse
@@ -429,7 +430,7 @@ nodes:
verdict_gate:
id: verdict_gate
type: script
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane.
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
script: scripts/verdict_gate.py
timeout: 60
next: done
@@ -438,6 +439,6 @@ nodes:
id: done
type: end
output: |
GAUNTLET: {{gauntlet_verdict}}
GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
{{gauntlet_report}}
@@ -10,7 +10,9 @@ hard rules:
and faulted alike, and a lane whose map ran nothing this pass gets
its stashed report restored — the verdict gate always sees each
lane's LAST report;
- a lane is FAULTED only when its report is missing, non-string, the
- a lane is FAULTED only when its report is missing (the map ran but
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
the verdict gate never reads it as SKIPPED), non-string, the
lane_fault.py marker, sentinel-less, or carries the nested graph's own
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
@@ -27,7 +29,9 @@ import os
import re
import time
MAX_RETRY_ELAPSED_SECS = 45600
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
# = 45600; 600 s of slack below that.
MAX_RETRY_ELAPSED_SECS = 45000
MAX_ATTEMPTS = 3
# lane name -> (items key, results key)
@@ -114,8 +118,10 @@ def main():
out[results_key] = [kept[lane]]
continue
report = results[0] if results else None
if report is not None:
kept[lane] = report
if report is None:
report = f"PIPELINE-FAULT: {lane} lane produced no report"
out[results_key] = [report]
kept[lane] = report
if is_faulted(lane, report):
faulted.append(lane)
attempts[lane] = int(attempts.get(lane) or 0) + 1
@@ -8,10 +8,11 @@ re-reads, no judgment. The gate's hard rules:
- a non-string lane result (the map collected an object instead of the
lane's rendered text) blocks — it cannot carry a sentinel;
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
detected only when the report IS the lane_fault.py marker, i.e. it
STARTS with it — a real review merely quoting the marker flows to the
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse
fault) blocks — a degraded pipeline is never a pass;
detected only when the report IS the lane_fault.py marker or
retry_gate.py's no-report marker, i.e. it STARTS with it — a real
review merely quoting the marker flows to the normal rules) or a
PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
degraded pipeline is never a pass;
- a code-review lane whose OWN verdict line records a pipeline fault
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
the nested graph completed, but with a dead domain lane, verifier, or
@@ -24,8 +25,16 @@ re-reads, no judgment. The gate's hard rules:
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
treated as PASS or FAIL (fix the environment per the plan's local-run
recipe, then re-run);
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
re-ran it, so the reader can tell a first-pass death from an exhausted
retry budget;
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
incomplete" section: that is a fault to re-run, never a code finding;
- the gate itself must never crash into a silent pass: any internal
error emits GAUNTLET: BLOCKED naming the error.
error emits GAUNTLET: BLOCKED naming the error, with the pipeline
marked incomplete.
"""
import json
@@ -43,6 +52,7 @@ def main():
state = load_state()
blockers, attention, lane_lines, reports = [], [], [], []
malformed = set()
lane_attempts = state.get("lane_attempts") or {}
def lane_report(key):
arr = state.get(key) or []
@@ -57,6 +67,10 @@ def main():
def record(lane, status, detail=""):
lane_lines.append(f"| {lane} | {status} | {detail} |")
def attempts_suffix(lane):
n = int(lane_attempts.get(lane) or 0)
return f" ×{n}" if n > 1 else ""
def fault_excerpt(report):
line = next(
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
@@ -64,10 +78,11 @@ def main():
return line.strip()[:200]
def is_lane_fault(report):
# A genuine lane fault is the ENTIRE report emitted by lane_fault.py,
# which always begins with the marker. Matching the marker anywhere
# would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass
# 🔴 counting and sentinel parsing.
# A genuine lane fault is the ENTIRE report — the lane_fault.py marker
# or retry_gate.py's no-report marker — which always begins with
# "PIPELINE-FAULT:". Matching the marker anywhere would let a real
# review that merely QUOTES it bypass 🔴 counting and sentinel
# parsing.
return report.strip().startswith("PIPELINE-FAULT:")
def lane_fault_blocker(lane, report):
@@ -75,7 +90,7 @@ def main():
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
)
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)")
record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
def malformed_blocker(lane, report):
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
@@ -129,10 +144,11 @@ def main():
blockers.append(
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
)
suffix = attempts_suffix("code-review")
if m_fault and reds:
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴")
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
elif m_fault:
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)")
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
elif reds:
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
elif m and m.group(1) == "MERGE-READY":
@@ -231,11 +247,27 @@ def main():
# A gauntlet-level fault (build_items crash, dead parse stage) means the
# lanes above were never built — SKIPPED rows alone must not read as PASS.
signals_error = state.get("signals_error")
incomplete = set(state.get("review_incomplete") or [])
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error:
blockers.append(f"pipeline: {signals_error}")
incomplete.add("pipeline")
verdict = "BLOCKED" if blockers else "PASS"
incomplete_lines = []
for name in sorted(incomplete):
if name == "pipeline":
incomplete_lines.append(f"- pipeline: {signals_error}")
elif n := int(lane_attempts.get(name) or 0):
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
else:
incomplete_lines.append(f"- {name}: no completed review")
incomplete_line = (
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
if incomplete
else ""
)
md = "\n".join(
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines
)
@@ -246,11 +278,21 @@ def main():
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
if state.get("signals_error"):
md += f"\n\n> {state['signals_error']}"
if incomplete_lines:
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
md += "\n\n## Full lane reports\n"
for lane, rep in reports:
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md}))
print(
json.dumps(
{
"gauntlet_verdict": verdict,
"gauntlet_report": md,
"gauntlet_incomplete_line": incomplete_line,
}
)
)
try:
@@ -261,6 +303,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash into a sile
{
"gauntlet_verdict": "BLOCKED",
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
}
)
)