fix(review-gauntlet): summary-line critical count, nested-fault hook, fail-closed lane fallback

verdict_gate.py: the code-review critical count is read from render.py's
deterministic summary line (full-shape regex, LAST match wins — a 🔴
quoted inside a finding body or Changes-table row is not a finding); the
raw 🔴 count remains the fail-closed fallback when the line is absent. A
code-review lane whose own rendered verdict line records a pipeline
fault (degraded nested run) now BLOCKS instead of passing as GREEN
(attention), and fault + reds append both blockers. A non-string lane
result item is BLOCKED as malformed, never routed to GREEN.

default_lanes.py: the crash handler now emits the WIDEST safe selection
(code-review ∪ adversary ∪ security) and records a PIPELINE-FAULT in
signals_error so the gate blocks — previously it repaired malformed state
into a NARROWER passable selection with no fault. Probe is added only
when consumer_surface AND a probe_context (local-run recipe) are present,
mirroring build_items' guard, so degraded selection cannot spawn an
INCONCLUSIVE-by-construction probe.

graph.yaml: re-derive the timeout envelope. Agent-node retries each get a
fresh per-attempt budget (agent.rs retry_transient/bounded_attempt) and
the child's "Graph … timed out" matches the transient anchor, so lanes
exceed their child graph's settings.timeout (code-review 11400 > 10800,
adversary 9600 > 9000) and the gauntlet's own timeout covers
max_attempts × lane timeout + margin (2 × 11400 + 1200 = 24000). The
child graphs' internal retry envelopes are deliberately not derived here
(tracked as a follow-up). README/descriptions updated and pinned.

Tests: summary-count/quoted-🔴/fallback/last-match/nested-fault/both-
blockers/malformed-lane, crash-widens-and-records-fault, probe guard,
ALIASES parity with build_items, and e2e render.py → verdict_gate runs.
This commit is contained in:
Dark-Alex-17 committed 2026-09-14 10:50:34 -06:00
1 parent ea76331c49
commit 4555af8299
6 files changed
+564 -18

No files matched your search

+8 -4
View File
@@ -51,8 +51,11 @@ those rules are *structure*:
(map-over-nothing), which *is* the skip mechanism. All four maps join at
the verdict gate on equal-length paths.
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
sentinel is a lane FAILURE, never a pass; any 🔴 in the code-review report
blocks regardless of its verdict line; `NEEDS-HUMAN` without 🔴 passes but
sentinel is a lane FAILURE, never a pass; the critical count from the
code-review report's summary line (raw 🔴 count if absent) blocks
regardless of its verdict line; a code-review lane whose own verdict line
records a pipeline fault blocks (a degraded review is never a pass);
`NEEDS-HUMAN` without 🔴 passes but
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
with an environment note — it is never treated as PASS or FAIL.
- **Verdict independence is structural**: lanes run as isolated sub-agents
@@ -65,7 +68,8 @@ those rules are *structure*:
lane* — distinct from SKIPPED (not selected). A dead `parse` records a fault
and jumps straight to the gate (`parse_fault.py`). A dead `select_lanes`
degrades to a deterministic selection (`default_lanes.py`): code-review +
adversary, plus probe on consumer surface, plus security on auth/deps/exec
adversary, plus probe on consumer surface *with a local-run recipe*, plus
security on auth/deps/exec
signals or hardened posture, unioned with caller-forced lanes — wider, never
narrower. A crashed `build_items` records a fault so an all-SKIPPED gate
still blocks. Fault detection is prefix-anchored — a real review that merely
@@ -77,7 +81,7 @@ those rules are *structure*:
| Lane | Sentinel | Blocks on |
|------|----------|-----------|
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | any 🔴 finding; missing sentinel |
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; missing sentinel |
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
+21 -5
View File
@@ -41,7 +41,13 @@ variables:
settings:
max_loop_iterations: 10
log_state_snapshots: true
timeout: 10800
# max over agent lanes of (max_attempts × lane timeout) = 2 × 11400 = 22800
# (agent-node retries each get a fresh per-attempt budget,
# src/graph/agent.rs retry_transient/bounded_attempt) + 1200 margin for
# the script stages (signals/select/build/gate) = 24000; the parse and
# select_lanes llm nodes carry no `timeout:` and are bounded by
# max_iterations.
timeout: 24000
initial_state:
diff_spec: worktree
@@ -237,7 +243,7 @@ nodes:
default_lanes:
id: default_lanes
type: script
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface AND a probe_context (local-run recipe) is present, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
script: scripts/default_lanes.py
timeout: 30
next: build_items
@@ -299,7 +305,12 @@ nodes:
type: agent
description: Independent code-quality review lane.
agent: code-reviewer
timeout: 7200
# Exceeds the code-reviewer graph's own settings.timeout (10800) by 600 so
# the child's graph timeout, not this lane timeout, is the binding bound.
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
# timeouts) is not derived here — deliberately out of scope for this
# change; tracked as a follow-up.
timeout: 11400
prompt: |
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
independent post-implementation review.
@@ -322,7 +333,12 @@ nodes:
type: agent
description: Plan-conformance review lane.
agent: adversary
timeout: 7200
# Exceeds the adversary graph's own settings.timeout (9000) by 600 so the
# child's graph timeout, not this lane timeout, is the binding bound.
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
# timeouts) is not derived here — deliberately out of scope for this
# change; tracked as a follow-up.
timeout: 9600
inputs:
verification_commands: '{{verification_commands}}'
prompt: |
@@ -399,7 +415,7 @@ nodes:
verdict_gate:
id: verdict_gate
type: script
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; 🔴 findings block regardless of the lane's own verdict line.
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane.
script: scripts/verdict_gate.py
timeout: 60
next: done
@@ -17,6 +17,8 @@ never kills the graph.
import json
import os
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
# canonicalize the same way on the degraded path as it does here.
ALIASES = {
"code-review": "code-review",
"code_review": "code-review",
@@ -3,13 +3,19 @@
Reached ONLY via select_lanes' `fallback:` route, when the additive LLM
judgment dies after retries. Emits `forced_lanes` — code-review + adversary,
plus probe when the deterministic signals flagged a consumer-facing surface,
plus probe when the deterministic signals flagged a consumer-facing surface
AND a local-run recipe exists (build_items gates probe the same way — a
probe without a recipe is INCONCLUSIVE by construction),
plus security when the attack-surface signals (auth/deps/exec) or a hardened
posture demand it, unioned with any lanes the caller forced (canonicalized
with the same aliases build_items uses) — which build_items honors exactly,
so selection degrades WIDER, never narrower. The degradation note goes into
`lanes_degraded`; build_items folds it into lanes_summary so the final
report names the degradation.
Fail-closed: if the fallback itself crashes it emits the widest safe
selection (defaults ∪ security) plus a PIPELINE-FAULT in signals_error so
verdict_gate blocks — a crashed fallback never narrows selection silently.
"""
import json
@@ -49,7 +55,12 @@ def load_state():
def main():
state = load_state()
lanes = {"code-review", "adversary"}
if state.get("consumer_surface"):
probe_context = state.get("probe_context")
if (
state.get("consumer_surface")
and isinstance(probe_context, str)
and probe_context.strip()
):
lanes.add("probe")
# The deterministic security signals are already in state when this
# fallback runs; a degraded selection must never drop the security lane
@@ -82,9 +93,13 @@ except Exception as e: # noqa: BLE001 — the fallback itself must never crash
print(
json.dumps(
{
"forced_lanes": ["code-review", "adversary"],
"forced_lanes": ["code-review", "adversary", "security"],
"lanes_degraded": "lane selection degraded to deterministic "
f"defaults (fallback script error: {e})",
"signals_error": (
f"PIPELINE-FAULT: lane-selection fallback crashed — {e}; "
"lanes widened to defaults ∪ security"
),
}
)
)
@@ -5,13 +5,21 @@ Parses each configured lane's verdict SENTINEL with a regex — no LLM
re-reads, no judgment. The gate's hard rules:
- a missing sentinel is a LANE FAILURE and blocks — never a pass;
- a non-string lane result (the map collected an object instead of the
lane's rendered text) blocks — it cannot carry a sentinel;
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
detected only when the report IS the lane_fault.py marker, i.e. it
STARTS with it — a real review merely quoting the marker flows to the
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse
fault) blocks — a degraded pipeline is never a pass;
- any 🔴 finding in the code-review report blocks regardless of the
lane's own verdict line;
- a code-review lane whose OWN verdict line records a pipeline fault
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
the nested graph completed, but with a dead domain lane, verifier, or
synthesis behind it, so its critical count is not trustworthy;
- a non-zero critical count in the code-review report blocks regardless
of the lane's own verdict line — derived from render.py's deterministic
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
count is used only when that summary line is missing;
- NEEDS-HUMAN without 🔴 passes but is surfaced as attention required;
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
treated as PASS or FAIL (fix the environment per the plan's local-run
@@ -34,13 +42,17 @@ def load_state():
def main():
state = load_state()
blockers, attention, lane_lines, reports = [], [], [], []
malformed = set()
def lane_report(key):
arr = state.get(key) or []
if not arr:
return None
item = arr[0]
return item if isinstance(item, str) else json.dumps(item)
if isinstance(item, str):
return item
malformed.add(key)
return json.dumps(item)
def record(lane, status, detail=""):
lane_lines.append(f"| {lane} | {status} | {detail} |")
@@ -65,21 +77,63 @@ def main():
)
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)")
def malformed_blocker(lane, report):
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
record(lane, "BLOCKED", "malformed lane result (non-string)")
reports.append((lane, report))
# --- code-review ------------------------------------------------------
cr = lane_report("code_review_results")
if cr is None:
record("code-review", "SKIPPED", "not selected")
elif "code_review_results" in malformed:
malformed_blocker("code-review", cr)
elif is_lane_fault(cr):
lane_fault_blocker("code-review", cr)
reports.append(("code-review", cr))
else:
reports.append(("code-review", cr))
m = re.search(r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)", cr)
reds = cr.count("🔴")
# code-reviewer's verdict.py forces NEEDS-HUMAN on its own internal
# faults; render.py emits that as `**Verdict: NEEDS-HUMAN** — <reason>`
# (CONTRACT comments at both emitters). Scoped to the FIRST rendered
# verdict line — render.py emits it before any finding body — so a
# finding that quotes the wording cannot trip it.
verdict_line = re.search(r"^\*\*Verdict: .*$", cr, re.MULTILINE)
m_fault = verdict_line and re.search(
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
verdict_line.group(0),
)
# Same quoted-marker class as the PIPELINE-FAULT prefix anchoring: a
# 🔴 quoted inside finding text or a Changes-table row is not a
# finding. render.py's summary line is the authoritative count; the
# raw count is the fail-closed fallback when that line is absent. The
# LAST match wins: render.py emits the line after every finding body,
# so a quoted look-alike earlier in the report cannot shadow it. The
# pattern is the FULL shape of render.py's line (CONTRACT comment
# there) so a partial look-alike in LLM-authored text never matches.
sums = re.findall(
r"^\*Reviewed \d+ files, found (\d+) critical, \d+ warnings, \d+ suggestions, "
r"\d+ nitpicks \(\d+ deferred by quality bar\)\*$",
cr,
re.MULTILINE,
)
reds = int(sums[-1]) if sums else cr.count("🔴")
if m_fault:
blockers.append(
"code-review: the lane reported an internal PIPELINE-FAULT "
"(degraded review) — a degraded lane is never a pass"
)
if reds:
blockers.append(
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
)
if m_fault and reds:
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴")
elif m_fault:
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)")
elif reds:
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
elif m and m.group(1) == "MERGE-READY":
record("code-review", "GREEN", "MERGE-READY")
@@ -100,6 +154,8 @@ def main():
adv = lane_report("adversary_results")
if adv is None:
record("adversary", "SKIPPED", "not selected")
elif "adversary_results" in malformed:
malformed_blocker("adversary", adv)
elif is_lane_fault(adv):
lane_fault_blocker("adversary", adv)
reports.append(("adversary", adv))
@@ -123,6 +179,8 @@ def main():
sec = lane_report("security_results")
if sec is None:
record("security", "SKIPPED", "not selected")
elif "security_results" in malformed:
malformed_blocker("security", sec)
elif is_lane_fault(sec):
lane_fault_blocker("security", sec)
reports.append(("security", sec))
@@ -144,6 +202,8 @@ def main():
pb = lane_report("probe_results")
if pb is None:
record("probe", "SKIPPED", "not selected")
elif "probe_results" in malformed:
malformed_blocker("probe", pb)
elif is_lane_fault(pb):
lane_fault_blocker("probe", pb)
reports.append(("probe", pb))