fix(adversary): fail closed on PIPELINE-FAULT evidence regardless of status; pin bail-chain and validator shape
This commit is contained in:
5 files changed
+162
-10
No files matched your search
@@ -91,7 +91,7 @@ The conformance doctrine is baked into the graph's own nodes:
|
||||
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL.
|
||||
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
|
||||
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT:` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES.
|
||||
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT:` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. The evidence marker alone forces the criterion into unmet regardless of the status it carries, so a verdict that echoes the marker with `MET` can never count toward CONFORMS.
|
||||
|
||||
It is **read-only** — it produces a verdict, never a fix.
|
||||
|
||||
|
||||
@@ -8,8 +8,9 @@ Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
|
||||
llm-node failure captured in `holistic_failure`) becomes complaint #1;
|
||||
per-criterion results are still reported. A criterion whose check DIED
|
||||
(criterion_fault's UNMET verdict, evidence prefixed "PIPELINE-FAULT:") is
|
||||
rendered as died rather than judged. Assembles the exact sentinel format the
|
||||
callers route on. Never crashes into a silent verdict.
|
||||
rendered as died rather than judged and counted as unmet regardless of the
|
||||
status it carries. Assembles the exact sentinel format the callers route on.
|
||||
Never crashes into a silent verdict.
|
||||
"""
|
||||
|
||||
import json
|
||||
@@ -19,6 +20,11 @@ MAX_FAULT_DETAIL_CHARS = 300
|
||||
DIED_MARKER = "criterion check DIED (pipeline fault)"
|
||||
|
||||
|
||||
def died(v):
|
||||
evidence = v.get("evidence")
|
||||
return isinstance(evidence, str) and evidence.startswith("PIPELINE-FAULT:")
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
@@ -54,15 +60,14 @@ def append_criterion_complaints(lines, verdicts, extra, start):
|
||||
"""Numbered complaints for non-MET verdicts and holistic extras."""
|
||||
i = start
|
||||
for v in verdicts:
|
||||
if v.get("status") == "MET":
|
||||
if v.get("status") == "MET" and not died(v):
|
||||
continue
|
||||
i += 1
|
||||
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||
if len(text) > 140:
|
||||
text = text[:137] + "…"
|
||||
complaint = (v.get("complaint") or "").strip()
|
||||
evidence = v.get("evidence")
|
||||
if isinstance(evidence, str) and evidence.startswith("PIPELINE-FAULT:"):
|
||||
if died(v):
|
||||
if not complaint.startswith(DIED_MARKER):
|
||||
complaint = f"{DIED_MARKER} — {complaint}"
|
||||
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
|
||||
@@ -121,9 +126,11 @@ def main():
|
||||
faults = collect_pipeline_faults(state)
|
||||
n = len(verdicts)
|
||||
|
||||
met = [v for v in verdicts if v.get("status") == "MET"]
|
||||
partial = [v for v in verdicts if v.get("status") == "PARTIAL"]
|
||||
bad = [v for v in verdicts if v.get("status") in ("UNMET", "DIVERGED")]
|
||||
met = [v for v in verdicts if v.get("status") == "MET" and not died(v)]
|
||||
partial = [v for v in verdicts if v.get("status") == "PARTIAL" and not died(v)]
|
||||
bad = [
|
||||
v for v in verdicts if died(v) or v.get("status") in ("UNMET", "DIVERGED")
|
||||
]
|
||||
|
||||
lines = []
|
||||
if faults:
|
||||
|
||||
Reference in new issue
Block a user