fix(adversary): fail closed per criterion when check_criterion dies

A live run died at map_criteria: one heavy criterion exhausted
max_iterations (10) with no fallback and sank the whole 17-criterion
review. check_criterion now has a 30-iteration budget, a tool-budget
instruction, and falls back to criterion_fault, which records a
schema-shaped UNMET verdict with a PIPELINE-FAULT evidence marker for
that one criterion; verdict.py renders it as DIED rather than judged.
This commit is contained in:
Dark-Alex-17 committed 2026-09-14 10:50:34 -06:00
1 parent 7dac4620a8
commit 3f49db86b2
5 files changed
+391 -8

No files matched your search

+3
View File
@@ -78,6 +78,8 @@ flowchart TD
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
B -. "fallback" .-> P
R -. "fallback" .-> P
D -. "fallback" .-> CF["criterion_fault (script): UNMET + PIPELINE-FAULT evidence for that ONE criterion"]
CF --> F
F -. "fallback" .-> G
P --> G
```
@@ -89,6 +91,7 @@ The conformance doctrine is baked into the graph's own nodes:
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL.
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT:` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES.
It is **read-only** — it produces a verdict, never a fix.
+12 -2
View File
@@ -215,6 +215,8 @@ nodes:
Your response is machine-validated: a malformed response is rejected back
to you exactly once with the reason, then recorded as PARTIAL (unproven).
The id is stamped from the criterion itself — your echo is informative only.
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
whole-file reads, and conclude as soon as the evidence is decisive.
prompt: |
Repository: {{project_dir}}
Changed files: {{changed_files}}
@@ -236,10 +238,11 @@ nodes:
- fs_cat
- fs_grep
- ast_grep
max_iterations: 10
max_iterations: 30
max_attempts: 2
state_updates:
crit_verdict: '{{output}}'
fallback: criterion_fault
next: crit_gate
crit_gate:
@@ -249,6 +252,13 @@ nodes:
script: scripts/crit_gate.py
timeout: 30
criterion_fault:
id: criterion_fault
type: script
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met. Never sinks the map.
script: scripts/criterion_fault.py
timeout: 30
holistic:
id: holistic
type: llm
@@ -320,7 +330,7 @@ nodes:
observations:
type: string
required: [extra_complaints, observations]
max_iterations: 10
max_iterations: 20
max_attempts: 2
state_updates:
holistic_failure: '{{output}}'
@@ -0,0 +1,91 @@
#!/usr/bin/env python3
"""Fail-closed per-criterion fault marker for the adversary graph.
Reached ONLY via check_criterion's `fallback:` route, after the engine has
written its "LLM node …failed: <chain>" text into `crit_verdict` through
state_updates. Converts that text into a schema-shaped UNMET verdict whose
evidence carries a "PIPELINE-FAULT:" marker, so one criterion check that dies
(max_iterations exhausted, API failure) is recorded as DIED rather than
judged, the other criteria still get verdicts, and the run is DIVERGES. The
map collects it like any other verdict — the branch never sinks the map.
"""
import json
import os
MAX_DETAIL_CHARS = 500
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def criterion_fields(state):
c = state.get("criterion")
if isinstance(c, dict):
cid = c.get("id")
text = c.get("text")
return (
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
text.strip() if isinstance(text, str) else "",
)
return "unknown", ""
def failure_detail(state):
captured = state.get("crit_verdict")
detail = captured.strip() if isinstance(captured, str) else ""
if not detail.startswith("LLM node"):
return "no failure text captured"
detail = detail.replace("\n", " ")
if len(detail) > MAX_DETAIL_CHARS:
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
return detail
def main():
state = load_state()
cid, ctext = criterion_fields(state)
detail = failure_detail(state)
print(
json.dumps(
{
"crit_verdict": {
"id": cid,
"text": ctext,
"status": "UNMET",
"evidence": f"PIPELINE-FAULT: criterion check failed — {detail}",
"complaint": (
f"criterion check DIED (pipeline fault) — {detail}; the "
"criterion was NOT verified and is treated as unmet "
"(fail-closed)"
),
}
}
)
)
try:
main()
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
print(
json.dumps(
{
"crit_verdict": {
"id": "unknown",
"text": "",
"status": "UNMET",
"evidence": f"PIPELINE-FAULT: criterion fault-marker script error: {e}",
"complaint": (
f"criterion check DIED (pipeline fault) — fault-marker script "
f"error: {e}; the criterion was NOT verified and is treated as "
"unmet (fail-closed)"
),
}
}
)
)
+16 -6
View File
@@ -6,14 +6,17 @@ holistic complaints, and zero pipeline faults. Everything else — including
a missing plan or a degraded (faulted) pipeline — is DIVERGES (fail-closed).
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
llm-node failure captured in `holistic_failure`) becomes complaint #1;
per-criterion results are still reported. Assembles the exact sentinel
format the callers route on. Never crashes into a silent verdict.
per-criterion results are still reported. A criterion whose check DIED
(criterion_fault's UNMET verdict, evidence prefixed "PIPELINE-FAULT:") is
rendered as died rather than judged. Assembles the exact sentinel format the
callers route on. Never crashes into a silent verdict.
"""
import json
import os
MAX_FAULT_DETAIL_CHARS = 300
DIED_MARKER = "criterion check DIED (pipeline fault)"
def load_state():
@@ -57,10 +60,17 @@ def append_criterion_complaints(lines, verdicts, extra, start):
text = (v.get("text") or "").strip().replace("\n", " ")
if len(text) > 140:
text = text[:137] + "…"
lines.append(
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
f'{(v.get("complaint") or "").strip()}'
)
complaint = (v.get("complaint") or "").strip()
evidence = v.get("evidence")
if isinstance(evidence, str) and evidence.startswith("PIPELINE-FAULT:"):
if not complaint.startswith(DIED_MARKER):
complaint = f"{DIED_MARKER} — {complaint}"
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
else:
lines.append(
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
f"{complaint}"
)
for c in extra:
i += 1
lines.append(