fix(adversary): narrow died-marker predicate, degraded banner for died criteria, hermetic script tests
This commit is contained in:
6 files changed
+154
-21
No files matched your search
@@ -91,7 +91,7 @@ The conformance doctrine is baked into the graph's own nodes:
|
||||
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL.
|
||||
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
|
||||
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT:` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. The evidence marker alone forces the criterion into unmet regardless of the status it carries, so a verdict that echoes the marker with `MET` can never count toward CONFORMS.
|
||||
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT: criterion ` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. That exact marker prefix (the one `criterion_fault` emits) forces the criterion into unmet regardless of the status it carries, so a verdict that echoes it with `MET` can never count toward CONFORMS; `check_criterion` is told never to begin its own evidence with `PIPELINE-FAULT:`.
|
||||
|
||||
It is **read-only** — it produces a verdict, never a fix.
|
||||
|
||||
|
||||
@@ -217,6 +217,8 @@ nodes:
|
||||
The id is stamped from the criterion itself — your echo is informative only.
|
||||
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
|
||||
whole-file reads, and conclude as soon as the evidence is decisive.
|
||||
Never begin `evidence` with the literal `PIPELINE-FAULT:` — that prefix is
|
||||
reserved for the graph's own fault markers.
|
||||
prompt: |
|
||||
Repository: {{project_dir}}
|
||||
Changed files: {{changed_files}}
|
||||
@@ -255,7 +257,7 @@ nodes:
|
||||
criterion_fault:
|
||||
id: criterion_fault
|
||||
type: script
|
||||
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met. Never sinks the map.
|
||||
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met.
|
||||
script: scripts/criterion_fault.py
|
||||
timeout: 30
|
||||
|
||||
|
||||
@@ -7,7 +7,10 @@ state_updates. Converts that text into a schema-shaped UNMET verdict whose
|
||||
evidence carries a "PIPELINE-FAULT:" marker, so one criterion check that dies
|
||||
(max_iterations exhausted, API failure) is recorded as DIED rather than
|
||||
judged, the other criteria still get verdicts, and the run is DIVERGES. The
|
||||
map collects it like any other verdict — the branch never sinks the map.
|
||||
map collects it like any other verdict — the branch never sinks the map. The
|
||||
engine applies state_updates before routing to the fallback (src/graph/llm.rs:
|
||||
`apply_state_updates_with_output`, then `outcome_from`), so the "no failure
|
||||
text captured" branch is belt-and-braces rather than an expected path.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
@@ -7,7 +7,7 @@ a missing plan or a degraded (faulted) pipeline — is DIVERGES (fail-closed).
|
||||
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
|
||||
llm-node failure captured in `holistic_failure`) becomes complaint #1;
|
||||
per-criterion results are still reported. A criterion whose check DIED
|
||||
(criterion_fault's UNMET verdict, evidence prefixed "PIPELINE-FAULT:") is
|
||||
(criterion_fault's UNMET verdict, evidence prefixed DIED_EVIDENCE_PREFIX) is
|
||||
rendered as died rather than judged and counted as unmet regardless of the
|
||||
status it carries. Assembles the exact sentinel format the callers route on.
|
||||
Never crashes into a silent verdict.
|
||||
@@ -18,11 +18,12 @@ import os
|
||||
|
||||
MAX_FAULT_DETAIL_CHARS = 300
|
||||
DIED_MARKER = "criterion check DIED (pipeline fault)"
|
||||
DIED_EVIDENCE_PREFIX = "PIPELINE-FAULT: criterion "
|
||||
|
||||
|
||||
def died(v):
|
||||
evidence = v.get("evidence")
|
||||
return isinstance(evidence, str) and evidence.startswith("PIPELINE-FAULT:")
|
||||
return isinstance(evidence, str) and evidence.startswith(DIED_EVIDENCE_PREFIX)
|
||||
|
||||
|
||||
def load_state():
|
||||
@@ -69,7 +70,7 @@ def append_criterion_complaints(lines, verdicts, extra, start):
|
||||
complaint = (v.get("complaint") or "").strip()
|
||||
if died(v):
|
||||
if not complaint.startswith(DIED_MARKER):
|
||||
complaint = f"{DIED_MARKER} — {complaint}"
|
||||
complaint = f"{DIED_MARKER} — {complaint}" if complaint else DIED_MARKER
|
||||
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
|
||||
else:
|
||||
lines.append(
|
||||
@@ -174,9 +175,14 @@ def main():
|
||||
lines.append(observations)
|
||||
else:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
lines.append(
|
||||
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged."
|
||||
)
|
||||
header = f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged"
|
||||
if died_count := sum(1 for v in verdicts if died(v)):
|
||||
header += (
|
||||
f" — degraded run: {died_count} criterion check(s) died (fail-closed)."
|
||||
)
|
||||
else:
|
||||
header += "."
|
||||
lines.append(header)
|
||||
lines.append("Complaints:")
|
||||
append_criterion_complaints(lines, verdicts, extra, 0)
|
||||
if observations:
|
||||
|
||||
+133
-11
@@ -8411,16 +8411,25 @@ mod tests {
|
||||
run_adversary_script_env(script, state, &[])
|
||||
}
|
||||
|
||||
// Scripts' load_state() prefers GRAPH_STATE_FILE over GRAPH_STATE, so an
|
||||
// inherited live state file (adversary verifying this repo) must not win.
|
||||
fn adversary_script_command(script: &str, state: &serde_json::Value) -> std::process::Command {
|
||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("assets/agents/adversary/scripts")
|
||||
.join(script);
|
||||
let mut cmd = std::process::Command::new("python3");
|
||||
cmd.arg(&path)
|
||||
.env("GRAPH_STATE", state.to_string())
|
||||
.env_remove("GRAPH_STATE_FILE");
|
||||
cmd
|
||||
}
|
||||
|
||||
fn run_adversary_script_env(
|
||||
script: &str,
|
||||
state: &serde_json::Value,
|
||||
envs: &[(&str, &str)],
|
||||
) -> serde_json::Value {
|
||||
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("assets/agents/adversary/scripts")
|
||||
.join(script);
|
||||
let mut cmd = std::process::Command::new("python3");
|
||||
cmd.arg(&path).env("GRAPH_STATE", state.to_string());
|
||||
let mut cmd = adversary_script_command(script, state);
|
||||
for (k, v) in envs {
|
||||
cmd.env(k, v);
|
||||
}
|
||||
@@ -8940,6 +8949,24 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adversary_script_helper_ignores_inherited_graph_state_file() {
|
||||
let cmd = adversary_script_command("criterion_fault.py", &json!({}));
|
||||
let removed = cmd
|
||||
.get_envs()
|
||||
.any(|(k, v)| k == "GRAPH_STATE_FILE" && v.is_none());
|
||||
assert!(
|
||||
removed,
|
||||
"GRAPH_STATE_FILE must be explicitly removed so an inherited live state file cannot shadow GRAPH_STATE: {:?}",
|
||||
cmd.get_envs().collect::<Vec<_>>()
|
||||
);
|
||||
assert!(
|
||||
cmd.get_envs()
|
||||
.any(|(k, v)| k == "GRAPH_STATE" && v.is_some()),
|
||||
"GRAPH_STATE must still carry the synthetic state"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adversary_verdict_renders_died_criterion_and_never_conforms() {
|
||||
if !cmd_available("python3") {
|
||||
@@ -8966,8 +8993,8 @@ mod tests {
|
||||
"a died criterion must never conform: {report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("Criteria: 1/2 met, 0 partial, 1 unmet/diverged."),
|
||||
"the died criterion counts as unmet: {report}"
|
||||
report.contains("Criteria: 1/2 met, 0 partial, 1 unmet/diverged — degraded run: 1 criterion check(s) died (fail-closed)."),
|
||||
"the died criterion counts as unmet and the header flags the degraded run: {report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains(
|
||||
@@ -9014,12 +9041,20 @@ mod tests {
|
||||
"a fault-marked verdict must never conform, whatever status it carries: {report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("Criteria: 1/2 met, 0 partial, 1 unmet/diverged."),
|
||||
"the fault-marked verdict counts as unmet, not met: {report}"
|
||||
report.contains("Criteria: 1/2 met, 0 partial, 1 unmet/diverged — degraded run: 1 criterion check(s) died (fail-closed)."),
|
||||
"the fault-marked verdict counts as unmet, not met, and the header flags the degraded run: {report}"
|
||||
);
|
||||
let died_line = report
|
||||
.lines()
|
||||
.find(|l| l.contains("criterion check DIED (pipeline fault)"))
|
||||
.unwrap_or_else(|| panic!("a fault-marked verdict must render as died: {report}"));
|
||||
assert!(
|
||||
!died_line.ends_with("— "),
|
||||
"an empty complaint must not leave a dangling em-dash: {died_line:?}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("criterion check DIED (pipeline fault)"),
|
||||
"a fault-marked verdict must render as died: {report}"
|
||||
died_line.ends_with("criterion check DIED (pipeline fault)"),
|
||||
"with no complaint the died marker stands alone: {died_line:?}"
|
||||
);
|
||||
let met_section = report
|
||||
.split("Met criteria (evidence):")
|
||||
@@ -9031,6 +9066,87 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adversary_verdict_quoted_marker_in_met_evidence_is_not_died() {
|
||||
if !cmd_available("python3") {
|
||||
eprintln!("skipping: python3 not available");
|
||||
return;
|
||||
}
|
||||
// Free-text evidence that merely quotes the marker (not the exact
|
||||
// prefix criterion_fault emits) is a genuine MET, not a died check.
|
||||
let state = json!({
|
||||
"pipeline_faults": [],
|
||||
"crit_verdicts": [
|
||||
{"id": "c1", "text": "renders the fault marker", "status": "MET",
|
||||
"evidence": "PIPELINE-FAULT: is emitted by criterion_fault.py per verdict.py:70",
|
||||
"complaint": ""}
|
||||
],
|
||||
"extra_complaints": [],
|
||||
"observations": "",
|
||||
"exec_results": []
|
||||
});
|
||||
let out = run_adversary_script("verdict.py", &state);
|
||||
let report = out["adv_report"].as_str().unwrap();
|
||||
assert!(
|
||||
report.starts_with("ADVERSARIAL_REVIEW: CONFORMS"),
|
||||
"evidence quoting the marker must not be reclassified as died: {report}"
|
||||
);
|
||||
assert!(
|
||||
!report.contains("DIED"),
|
||||
"no died rendering for a genuine MET: {report}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adversary_verdict_died_criterion_under_pipeline_fault_banner() {
|
||||
if !cmd_available("python3") {
|
||||
eprintln!("skipping: python3 not available");
|
||||
return;
|
||||
}
|
||||
let state = json!({
|
||||
"pipeline_faults": ["PIPELINE-FAULT: parse failed — cannot review: LLM node failed: boom"],
|
||||
"crit_verdicts": [
|
||||
{"id": "c1", "text": "does X", "status": "MET",
|
||||
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""},
|
||||
{"id": "c2", "text": "does Y", "status": "UNMET",
|
||||
"evidence": "PIPELINE-FAULT: criterion check failed — LLM node failed: boom",
|
||||
"complaint": "criterion check DIED (pipeline fault) — LLM node failed: boom; the criterion was NOT verified and is treated as unmet (fail-closed)"}
|
||||
],
|
||||
"extra_complaints": [],
|
||||
"observations": "",
|
||||
"exec_results": []
|
||||
});
|
||||
let out = run_adversary_script("verdict.py", &state);
|
||||
let report = out["adv_report"].as_str().unwrap();
|
||||
assert!(
|
||||
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
|
||||
"a pipeline fault plus a died criterion must never conform: {report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains("Criteria: 1/2 met, 0 partial, 1 unmet/diverged — degraded run: pipeline fault recorded (fail-closed)."),
|
||||
"the pipeline-fault banner takes precedence in the header: {report}"
|
||||
);
|
||||
assert!(
|
||||
report
|
||||
.contains("1. PIPELINE-FAULT: parse failed — cannot review: LLM node failed: boom"),
|
||||
"the pipeline fault is complaint #1: {report}"
|
||||
);
|
||||
assert!(
|
||||
report.contains(
|
||||
"2. Acceptance criterion \"does Y\" — criterion check DIED (pipeline fault) — "
|
||||
),
|
||||
"the died criterion is complaint #2: {report}"
|
||||
);
|
||||
let met_section = report
|
||||
.split("Met criteria (evidence):")
|
||||
.nth(1)
|
||||
.expect("the genuine MET criterion produces an evidence appendix");
|
||||
assert!(
|
||||
!met_section.contains("does Y"),
|
||||
"the died criterion must not be listed as met: {report}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adversary_verdict_prepends_died_marker_once_when_complaint_lacks_it() {
|
||||
if !cmd_available("python3") {
|
||||
@@ -9133,6 +9249,12 @@ mod tests {
|
||||
.is_some_and(|s| s.contains("bounded tool budget")),
|
||||
"check_criterion must be told its tool budget is bounded"
|
||||
);
|
||||
assert!(
|
||||
llm.instructions
|
||||
.as_deref()
|
||||
.is_some_and(|s| s.contains("Never begin `evidence`")),
|
||||
"check_criterion must be told the PIPELINE-FAULT evidence prefix is reserved"
|
||||
);
|
||||
|
||||
let fault = graph
|
||||
.nodes
|
||||
|
||||
+1
-1
@@ -251,7 +251,7 @@ async fn run_with_retries(
|
||||
match run_chat_loop(node, prompt, ctx, abort).await {
|
||||
Ok(out) => return Ok(out),
|
||||
Err(e) if is_transient_error(&e) && attempt < node.max_attempts => {
|
||||
warn!("llm node attempt {attempt} failed (transient): {e}; retrying");
|
||||
warn!("llm node attempt {attempt} failed (transient): {e:#}; retrying");
|
||||
last_err = Some(e);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
|
||||
Reference in new issue
Block a user