fix(adversary): red verification runs block CONFORMS; re-derive timeout envelope; robust runner

verdict.py: a recorded verification run with exit≠0 (including the -1
TIMEOUT sentinel) or a skipped record now forces DIVERGES with a numbered
complaint naming the command — previously a [FAIL] line sat in the report
appendix while the sentinel read CONFORMS, and callers route on the
sentinel alone. Fail-closed by the same rule: an ENVIRONMENT runner-error
marker when commands WERE declared is one red complaint (the declared
commands never ran, so the run is unproven); the "none declared" marker
and an empty declaration stay non-red. Headers name the red-run count.

run_checks.py: decode command output with encoding=utf-8/errors=replace
(a single non-UTF-8 byte used to raise inside communicate() and collapse
the whole evidence record to an ENVIRONMENT marker); bound the post-kill
pipe drain by DRAIN_TIMEOUT_SECS so a session-escaped grandchild holding
the pipe cannot turn a TIMEOUT record into a node kill.

graph.yaml: settings.timeout 5400 → 9000 = 3600 (run_checks node timeout)
+ 5400 (the previous whole-graph budget, kept for every other stage); the
old value predated run_checks and a graph-level timeout kills from
outside, losing the DIVERGES sentinel. Descriptions and README doctrine
updated (CONFORMS = all MET + zero holistic complaints + zero pipeline
faults + zero red runs).

Tests: envelope pin (settings >= run_checks + 5400), gauntlet lane
timeouts > child graph timeouts and gauntlet >= max_attempts × lane +
margin over every agent lane, non-UTF-8 output, bounded drain (unix),
red/skipped/timeout/none-declared/ENVIRONMENT×declaration matrix,
numbering continuity, header punctuation, wording pins on graph.yaml,
run_checks.py and both READMEs. The deadline test's hung command is now
python3-based instead of GNU sleep (cmd.exe on the Windows CI leg).
This commit is contained in:
Dark-Alex-17 committed 2026-09-14 10:50:34 -06:00
1 parent 156f3193a9
commit 68e893d684
5 files changed
+683 -33

No files matched your search

+10 -6
View File
@@ -6,8 +6,9 @@
> is structurally impossible. Each verdict passes a deterministic contract gate (MET requires
> evidence including a proving test; no test = at best PARTIAL), a holistic ADDITIVE-only pass hunts
> absence/scope-drift/substitution/gamed-tests, and the CONFORMS/DIVERGES sentinel is computed by a
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is unchanged — now
> structurally enforced.
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is structurally
> enforced: CONFORMS requires ALL MET + zero holistic complaints + zero pipeline faults +
> zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
@@ -41,6 +42,9 @@ ADVERSARIAL_REVIEW: CONFORMS
Criteria: N/N met (all with tests).
```
CONFORMS is deterministic: ALL MET + zero holistic complaints + zero pipeline faults +
zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
```
ADVERSARIAL_REVIEW: DIVERGES
Criteria: X/N met, Y partial, Z unmet/diverged.
@@ -74,7 +78,7 @@ flowchart TD
D --> E["crit_gate (script): MET needs change+test evidence"]
E -. "reject-retry once; exhausted = PARTIAL" .-> D
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults; no criteria = DIVERGES"]
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults + zero red runs; no criteria = DIVERGES"]
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
B -. "fallback" .-> P
@@ -89,7 +93,7 @@ The conformance doctrine is baked into the graph's own nodes:
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL. The commands arrive through the declared `verification_commands` graph variable (see [Declaring verification commands](#declaring-verification-commands)) — `parse` never extracts them from the prompt, so the prompt is no longer an extraction source for the commands.
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL. Independently of the criterion judgments, the verdict script blocks CONFORMS on any red run (exit≠0/timeout/skipped) and on an ENVIRONMENT runner-error marker when commands were declared — the declared commands never ran, so the run is unproven. The commands arrive through the declared `verification_commands` graph variable (see [Declaring verification commands](#declaring-verification-commands)) — `parse` never extracts them from the prompt, so the prompt is no longer an extraction source for the commands.
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT: criterion ` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. That exact marker prefix (the one `criterion_fault` emits) forces the criterion into unmet regardless of the status it carries, so a verdict that echoes it with `MET` can never count toward CONFORMS; `check_criterion` is told never to begin its own evidence with `PIPELINE-FAULT:`.
@@ -137,8 +141,8 @@ as a structured `inputs:` passthrough, not as prompt text.
- `'[]'` (or omitted) ⇒ the "none declared" marker; execution criteria are at best PARTIAL.
- Anything that is not a JSON array of strings (unparsable JSON, an object, non-string items) is
an invalid declaration: `run_checks` executes **nothing** and records a
`PIPELINE-FAULT: verification_commands declaration invalid — …` marker, which forces DIVERGES —
fail-closed, not degraded to the soft ENVIRONMENT marker.
`PIPELINE-FAULT: verification_commands declaration invalid — …` marker, which forces DIVERGES:
it fails closed as a PIPELINE-FAULT (complaint #1) — it never degrades to an ENVIRONMENT marker.
- The engine merges only the keys declared under `parse`'s `output_schema.properties` into state,
so an injected instruction that makes the parse LLM emit an extra `verification_commands` key
cannot reach state: the undeclared key is dropped and the declared variable stands.
+16 -7
View File
@@ -11,8 +11,10 @@ description: |
silently-overridden recorded decisions, out-of-scope touches, and gamed
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
requires every criterion MET and zero holistic complaints. Missing plan =
DIVERGES (fail-closed) naming the missing spec. Caller-declared verification
requires every criterion MET, zero holistic complaints, zero pipeline faults,
and zero red verification runs (exit≠0/timeout/skipped, or a runner error
when commands were declared). Missing plan = DIVERGES (fail-closed) naming
the missing spec. Caller-declared verification
commands are executed ONCE by a deterministic stage and recorded — criterion
branches cite the record and never run anything themselves. Any pipeline
fault (parse, diff resolution, verification runner, or holistic failure)
@@ -41,7 +43,12 @@ variables:
settings:
max_loop_iterations: 10
log_state_snapshots: false
timeout: 5400
# 3600 (run_checks node timeout, worst case) + 5400 (the previous
# whole-graph budget, kept as the envelope for every stage other than
# run_checks: parse/facts before it, criteria map/holistic/verdict after)
# = 9000. A graph timeout kills from outside — no fallback fires and the
# sentinel is lost.
timeout: 9000
initial_state:
project_dir_in: ''
@@ -138,7 +145,7 @@ nodes:
run_checks:
id: run_checks
type: script
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker; a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker (criterion branches judge it PARTIAL/unproven; the verdict additionally blocks CONFORMS when commands were declared); a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
script: scripts/run_checks.py
timeout: 3600
fallback: pipeline_fault
@@ -158,7 +165,7 @@ nodes:
check_criterion:
id: check_criterion
type: llm
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree.
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree (ENVIRONMENT/SKIPPED runs judge PARTIAL here; the deterministic verdict blocks CONFORMS when declared commands did not run).
instructions: |
You verify ONE acceptance criterion against a code change. Fresh context,
one criterion, full suspicion: assume the implementer drifted, cut a
@@ -201,7 +208,9 @@ nodes:
prompt.
- a SKIPPED record (the runner's total deadline expired before that
command ran) ⇒ at best PARTIAL for the criteria it covers, same
treatment as an ENVIRONMENT marker: unproven, never failed.
treatment as an ENVIRONMENT marker: unproven at the criterion level.
The deterministic verdict additionally blocks CONFORMS when declared
commands did not run — that is its job, not yours.
Output format — respond with ONLY this JSON object, no prose around it:
{"id": "<the criterion's id, echoed>",
@@ -349,7 +358,7 @@ nodes:
verdict:
id: verdict
type: script
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, and zero pipeline faults; no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, zero pipeline faults, and zero recorded red verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error marker when commands were declared); no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
script: scripts/verdict.py
timeout: 30
next: done
+18 -8
View File
@@ -17,8 +17,8 @@ from the repo under review, and whatever is declared here runs with
shell=True. Variables land in state as strings, so a JSON-encoded list is
accepted alongside a real list. A declaration that is not a JSON array of
strings is a caller contract violation, not a runner hiccup: it executes
NOTHING and records a PIPELINE-FAULT (fail-closed into DIVERGES) instead of
degrading to the soft ENVIRONMENT marker. The engine merges only the keys an
NOTHING and fails closed as a PIPELINE-FAULT (complaint #1) — it never
degrades to an ENVIRONMENT marker. The engine merges only the keys an
llm node's `output_schema.properties` declares, so a parse LLM coaxed into
emitting an extra `verification_commands` key cannot overwrite the declared
value: the undeclared key is dropped before it reaches state.
@@ -35,9 +35,12 @@ ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam).
Process hygiene: commands run through the shell. On POSIX each command gets
its own session (start_new_session) and a timeout SIGKILLs the whole process
group, so grandchildren of a hung command cannot linger. On non-POSIX
platforms only the direct shell child is killed — grandchildren of a
timed-out command may survive (documented limitation).
group, so grandchildren of a hung command cannot linger. The post-kill
reap/drain is itself bounded by DRAIN_TIMEOUT_SECS: a grandchild that escaped
the session (its own start_new_session) but still holds the stdout pipe
would otherwise block the drain until it exits. On non-POSIX platforms only
the direct shell child is killed — grandchildren of a timed-out command may
survive (documented limitation).
Verification commands run with GRAPH_STATE* (GRAPH_STATE, GRAPH_STATE_FILE)
scrubbed from their environment: the reviewed repo's own tests may exec these very
@@ -53,6 +56,7 @@ import time
PER_COMMAND_TIMEOUT_SECS = 900
TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
DRAIN_TIMEOUT_SECS = 10
TAIL_LINES = 50
MAX_TAIL_CHARS = 8000
@@ -112,6 +116,8 @@ def run_one(cmd, proj, budget):
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
env=env,
**popen_kwargs,
)
@@ -125,7 +131,10 @@ def run_one(cmd, proj, budget):
pass
else:
p.kill()
p.communicate() # reap the child and drain the pipes
try:
p.communicate(timeout=DRAIN_TIMEOUT_SECS)
except subprocess.TimeoutExpired:
pass
return {
"cmd": cmd,
"exit": -1,
@@ -189,8 +198,9 @@ except Exception as e: # noqa: BLE001 — never fail the node; degrade visibly
{
"exec_results": (
f"ENVIRONMENT: verification runner error: {e} — the declared "
"commands could not be executed; treat execution criteria as "
"PARTIAL (unproven), not as failed"
"commands could not be executed; execution criteria are unproven "
"(PARTIAL at the criterion level; the verdict blocks CONFORMS when "
"commands were declared)"
)
}
)
+82 -11
View File
@@ -2,8 +2,11 @@
"""Deterministic conformance verdict for the adversary graph.
CONFORMS requires: at least one criterion, every criterion MET, zero
holistic complaints, and zero pipeline faults. Everything else — including
a missing plan or a degraded (faulted) pipeline — is DIVERGES (fail-closed).
holistic complaints, zero pipeline faults, and zero recorded red
verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error
marker when commands were declared — the declared commands never ran, so the
run is unproven). Everything else — including a missing plan or a degraded
(faulted) pipeline — is DIVERGES (fail-closed).
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
llm-node failure captured in `holistic_failure`) becomes complaint #1;
per-criterion results are still reported. A criterion whose check DIED
@@ -17,6 +20,7 @@ import json
import os
MAX_FAULT_DETAIL_CHARS = 300
MAX_MARKER_CHARS = 200
DIED_MARKER = "criterion check DIED (pipeline fault)"
DIED_EVIDENCE_PREFIX = "PIPELINE-FAULT: criterion "
@@ -86,6 +90,63 @@ def append_criterion_complaints(lines, verdicts, extra, start):
return i
def declared_nonempty(declared):
"""True when verification_commands names at least one command.
Accepts a list or a JSON-encoded string of one. None, unparseable, or
empty declarations count as not declared — run_checks already records
a PIPELINE-FAULT for the invalid ones.
"""
if isinstance(declared, str):
try:
declared = json.loads(declared)
except json.JSONDecodeError:
return False
if not isinstance(declared, list):
return False
return any(isinstance(c, str) and c.strip() for c in declared)
def exec_run_complaints(exec_results, declared):
"""One complaint per red verification run recorded by run_checks.py.
A red run is a record that was skipped (deadline) or exited nonzero
(including the -1 TIMEOUT sentinel). An "ENVIRONMENT: …" runner-error
marker with a non-empty declaration is one complaint: the declared
commands never ran. Other string markers ("none declared",
"PIPELINE-FAULT…") are not runs and yield nothing.
"""
if isinstance(exec_results, str):
marker = exec_results.strip().replace("\n", " ")
if not marker.startswith("ENVIRONMENT:") or not declared_nonempty(declared):
return []
if len(marker) > MAX_MARKER_CHARS:
marker = marker[: MAX_MARKER_CHARS - 1] + "…"
return [
f"Verification runner error — the declared command(s) never ran ({marker}); "
"unproven runs block CONFORMS"
]
if not isinstance(exec_results, list):
return []
out = []
for r in exec_results:
if not isinstance(r, dict):
continue
cmd = r.get("cmd", "?")
if r.get("skipped"):
out.append(f"Verification run `{cmd}` — never ran: deadline expired (unproven)")
continue
code = r.get("exit")
if code == 0:
continue
outcome = "TIMEOUT (exit -1)" if code == -1 else f"exit {code}"
out.append(
f"Verification run `{cmd}` — {outcome} (recorded FAIL; a red declared "
"command blocks CONFORMS regardless of criterion judgments)"
)
return out
def render_exec_results(exec_results):
"""Lines for the "Verification runs:" report section.
@@ -125,6 +186,9 @@ def main():
extra = [c for c in (state.get("extra_complaints") or []) if isinstance(c, dict)]
observations = (state.get("observations") or "").strip()
faults = collect_pipeline_faults(state)
reds = exec_run_complaints(
state.get("exec_results"), state.get("verification_commands")
)
n = len(verdicts)
met = [v for v in verdicts if v.get("status") == "MET" and not died(v)]
@@ -152,7 +216,10 @@ def main():
for f in faults:
i += 1
lines.append(f"{i}. {f}")
append_criterion_complaints(lines, verdicts, extra, i)
i = append_criterion_complaints(lines, verdicts, extra, i)
for c in reds:
i += 1
lines.append(f"{i}. {c}")
if observations:
lines.append("")
lines.append("Non-blocking observations:")
@@ -166,7 +233,9 @@ def main():
"be judged without a spec (fail-closed). Supply the plan's Objective/Tasks/"
"Acceptance criteria and re-run."
)
elif not partial and not bad and not extra:
for i, c in enumerate(reds, start=2):
lines.append(f"{i}. {c}")
elif not partial and not bad and not extra and not reds:
lines.append("ADVERSARIAL_REVIEW: CONFORMS")
lines.append(f"Criteria: {n}/{n} met (all with tests).")
if observations:
@@ -176,15 +245,17 @@ def main():
else:
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
header = f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged"
notes = []
if died_count := sum(1 for v in verdicts if died(v)):
header += (
f" — degraded run: {died_count} criterion check(s) died (fail-closed)."
)
else:
header += "."
lines.append(header)
notes.append(f"degraded run: {died_count} criterion check(s) died (fail-closed)")
if reds:
notes.append(f"{len(reds)} red verification run(s) (fail-closed)")
lines.append(header + "".join(f" — {note}" for note in notes) + ".")
lines.append("Complaints:")
append_criterion_complaints(lines, verdicts, extra, 0)
i = append_criterion_complaints(lines, verdicts, extra, 0)
for c in reds:
i += 1
lines.append(f"{i}. {c}")
if observations:
lines.append("")
lines.append("Non-blocking observations:")
+557 -1
View File
@@ -8280,6 +8280,15 @@ mod tests {
) && !adversary.contains("Known residual"),
"adversary README must state the schema-declared merge closes the extra-key channel"
);
assert!(
adversary.contains("zero red verification runs"),
"adversary README must state the fail-closed CONFORMS doctrine"
);
assert!(
!adversary.contains("soft ENVIRONMENT")
&& adversary.contains("never degrades to an ENVIRONMENT marker"),
"adversary README must describe a malformed declaration as fail-closed, not a soft marker"
);
let gauntlet = read_to_string(readmes.join("review-gauntlet/README.md")).unwrap();
assert!(
!gauntlet.contains("Verification commands:"),
@@ -8294,12 +8303,28 @@ mod tests {
gauntlet.contains("--agent-variable verification_commands '[\"cargo test --all\"]'"),
"review-gauntlet README must show the CLI form"
);
assert!(gauntlet.contains("plus probe on consumer surface *with a local-run recipe*"));
assert!(gauntlet.contains("critical count from the report's summary line"));
assert!(gauntlet.contains("raw 🔴 count if absent"));
assert!(
gauntlet
.contains("records a pipeline fault blocks (a degraded review is never a pass)")
);
assert!(
!gauntlet.contains("any 🔴 in the code-review report"),
"review-gauntlet README must not describe the retired raw-🔴 gate"
);
let runner = read_to_string(readmes.join("adversary/scripts/run_checks.py")).unwrap();
assert!(
runner.contains("Trust boundary:") && runner.contains("never a field an LLM"),
"run_checks.py docstring must state the declared-variable trust boundary"
);
assert!(
!runner.contains("soft ENVIRONMENT")
&& runner.contains("never\ndegrades to an ENVIRONMENT marker"),
"run_checks.py docstring must describe a malformed declaration as fail-closed, not a soft marker"
);
assert!(
!runner.contains("Known residual") && !gauntlet.contains("residual:"),
"run_checks.py and the review-gauntlet README must not disclose the closed residual"
@@ -9223,7 +9248,10 @@ mod tests {
// second must be recorded as skipped — the in-script handling stays
// authoritative instead of the NODE timeout killing the script from
// outside (which would bypass the ENVIRONMENT degradation entirely).
let state = json!({"verification_commands": ["sleep 5", "echo never"]});
let state = json!({"verification_commands": [
r#"python3 -c "import time; time.sleep(5)""#,
"echo never"
]});
let out = run_adversary_script_env(
"run_checks.py",
&state,
@@ -9249,6 +9277,30 @@ mod tests {
);
}
#[test]
fn adversary_run_checks_tolerates_non_utf8_output() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({"verification_commands": [
r#"python3 -c "import sys; sys.stdout.buffer.write(b'ok \xff\xfe bytes\n')""#
]});
let out = run_adversary_script("run_checks.py", &state);
let results = out["exec_results"].as_array().unwrap_or_else(|| {
panic!("invalid UTF-8 output must not collapse the record to a marker: {out}")
});
assert_eq!(
results[0]["exit"], 0,
"the command itself succeeded: {results:?}"
);
let tail = results[0]["tail"].as_str().unwrap();
assert!(
tail.contains("ok") && tail.contains("bytes"),
"decodable parts of the output must survive with replacement: {results:?}"
);
}
#[test]
#[cfg(unix)]
fn adversary_run_checks_timeout_kills_the_whole_process_group() {
@@ -9327,6 +9379,39 @@ mod tests {
);
}
#[test]
#[cfg(unix)]
fn adversary_run_checks_timeout_drain_is_bounded_by_session_escaped_grandchild() {
if !cmd_available("python3") || !cmd_available("sh") {
eprintln!("skipping: python3 or sh not available");
return;
}
// The grandchild starts its own session, so the SIGKILL of the
// command's process group misses it — but it inherited the stdout
// pipe, so an unbounded post-kill drain would block until it exits
// (~40s). DRAIN_TIMEOUT_SECS (10) must cap that. The orphaned `sleep
// 40` exits on its own.
let cmd = "python3 -c \"import subprocess,time; \
subprocess.Popen(['sleep','40'], start_new_session=True); time.sleep(60)\"";
let state = json!({"verification_commands": [cmd]});
let started = Instant::now();
let out = run_adversary_script_env(
"run_checks.py",
&state,
&[("ADVERSARY_RUN_CHECKS_DEADLINE_SECS", "1")],
);
let elapsed = started.elapsed();
let results = out["exec_results"].as_array().unwrap();
assert_eq!(
results[0]["exit"], -1,
"the hung command must record a timeout: {results:?}"
);
assert!(
elapsed < Duration::from_secs(30),
"the post-kill drain must be bounded by DRAIN_TIMEOUT_SECS, took {elapsed:?}: {results:?}"
);
}
/// TOTAL_DEADLINE_SECS in run_checks.py and the run_checks node's
/// `timeout:` in graph.yaml live in different files; if the script's
/// deadline ever creeps past the node timeout, the executor kills the
@@ -9362,6 +9447,103 @@ mod tests {
);
}
/// The adversary's graph-level timeout must leave room for run_checks to
/// burn its whole node timeout AND for the rest of the pipeline to still
/// finish — a graph timeout kills from outside, so no fallback fires and
/// the DIVERGES sentinel is lost.
#[test]
fn adversary_graph_timeout_covers_run_checks_budget() {
use crate::graph::NodeType;
const OTHER_STAGES_ENVELOPE_SECS: u64 = 5400;
let graph = load_bundled_graph("adversary");
let node = graph
.get_node("run_checks")
.expect("adversary graph must have a run_checks node");
let NodeType::Script(s) = &node.node_type else {
panic!("run_checks must be a script node");
};
let graph_timeout = graph
.settings
.timeout
.expect("adversary graph must set settings.timeout");
assert!(
graph_timeout >= s.timeout + OTHER_STAGES_ENVELOPE_SECS,
"assets/agents/adversary/graph.yaml settings.timeout ({graph_timeout}) must cover the run_checks node timeout ({}) plus the {OTHER_STAGES_ENVELOPE_SECS}s envelope for the other stages",
s.timeout
);
}
/// Every gauntlet lane that runs a graph-backed agent must outlive that
/// graph's own timeout so the child's own graph timeout, not the lane
/// timeout, is the binding bound; the child's internal retry envelope is
/// deliberately not derived here (follow-up). And because agent-node
/// retries each get a fresh per-attempt budget (src/graph/agent.rs
/// retry_transient/bounded_attempt), the gauntlet itself must outlive
/// `max_attempts × timeout` for every lane, with headroom for the
/// surrounding stages.
#[test]
fn gauntlet_lane_timeouts_cover_child_graphs_and_retries() {
use crate::graph::NodeType;
const OTHER_STAGES_MARGIN_SECS: u64 = 1200;
let gauntlet = load_bundled_graph("review-gauntlet");
let gauntlet_timeout = gauntlet
.settings
.timeout
.expect("review-gauntlet must set settings.timeout");
let assets = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("assets/agents");
let gauntlet_yaml = read_to_string(assets.join("review-gauntlet/graph.yaml")).unwrap();
assert!(
gauntlet_yaml.contains("internal retry envelope")
&& gauntlet_yaml.contains("not derived here"),
"review-gauntlet graph.yaml must keep the NOTE that the child's internal retry envelope is not derived from the lane timeout"
);
assert!(
!gauntlet_yaml.contains("is what arrives"),
"review-gauntlet graph.yaml must not claim the lane timeout guarantees the child's verdict arrives"
);
assert!(
gauntlet_yaml.contains("AND a probe_context (local-run recipe) is present"),
"review-gauntlet graph.yaml default_lanes description must state the probe_context guard"
);
let mut seen = 0;
let mut graph_backed = 0;
for (id, node) in &gauntlet.nodes {
let NodeType::Agent(a) = &node.node_type else {
continue;
};
seen += 1;
let lane_timeout = a
.timeout
.unwrap_or_else(|| panic!("review-gauntlet lane {id} must set a timeout"));
if assets.join(&a.agent).join("graph.yaml").exists() {
graph_backed += 1;
let child_timeout = load_bundled_graph(&a.agent)
.settings
.timeout
.unwrap_or_else(|| panic!("{} graph must set settings.timeout", a.agent));
assert!(
lane_timeout > child_timeout,
"assets/agents/review-gauntlet/graph.yaml lane {id} timeout ({lane_timeout}) must exceed assets/agents/{}/graph.yaml settings.timeout ({child_timeout})",
a.agent
);
}
let worst_case = u64::from(a.max_attempts) * lane_timeout;
assert!(
gauntlet_timeout >= worst_case + OTHER_STAGES_MARGIN_SECS,
"review-gauntlet settings.timeout ({gauntlet_timeout}) must cover lane {id}'s max_attempts × timeout ({} × {lane_timeout} = {worst_case}) plus a {OTHER_STAGES_MARGIN_SECS}s margin for the other stages",
a.max_attempts
);
}
assert!(
seen >= 4,
"review-gauntlet must have at least 4 agent lanes, saw {seen}"
);
assert!(
graph_backed >= 2,
"at least the adversary and code-reviewer lanes must be graph-backed so the child-graph timeout check runs, saw {graph_backed}"
);
}
#[test]
fn adversary_pipeline_fault_parse_stage_attribution() {
if !cmd_available("python3") {
@@ -9478,6 +9660,380 @@ mod tests {
);
}
#[test]
fn adversary_verdict_green_runs_keep_conforms() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": 0, "duration_s": 1.0, "tail": "ok"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: CONFORMS"),
"an all-green run record must not block CONFORMS: {report}"
);
}
#[test]
fn adversary_verdict_red_run_forces_diverges() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
// A criterion judged MET cannot outrank a recorded failing run of a
// declared command — the verdict is fail-closed on exec_results.
let mut state = json!({
"pipeline_faults": [],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": 101, "duration_s": 1.0, "tail": "FAILED"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"a recorded red run must force DIVERGES: {report}"
);
assert!(
report.contains("1. Verification run `cargo test` — exit 101"),
"the red run must be a numbered complaint naming the command: {report}"
);
assert!(
report.contains("Criteria: 1/1 met, 0 partial, 0 unmet/diverged — 1 red verification run(s) (fail-closed)."),
"a reds-only DIVERGES header must count the red runs: {report}"
);
state["exec_results"] = json!([{"cmd": "cargo clippy", "skipped": "deadline"}]);
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"a skipped (unproven) run must force DIVERGES: {report}"
);
assert!(
report.contains("1. Verification run `cargo clippy` — never ran"),
"the skipped run must be a numbered complaint naming the command: {report}"
);
}
#[test]
fn adversary_verdict_none_declared_marker_is_not_red() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": "none declared — the caller supplied no verification_commands; criteria relying on tests are unproven"
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: CONFORMS"),
"the none-declared marker must not block CONFORMS: {report}"
);
assert!(
!report.contains("Verification run `"),
"the none-declared marker must not yield a run complaint: {report}"
);
}
#[test]
fn adversary_verdict_environment_marker_with_declared_commands_blocks() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
// Commands were declared and the runner died before running them:
// the criteria are unproven, and an all-MET set of criterion
// judgments must not turn that into CONFORMS.
let state = json!({
"pipeline_faults": [],
"verification_commands": ["cargo test"],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": "ENVIRONMENT: verification runner error: x — the declared commands could not be executed"
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"an ENVIRONMENT marker with declared commands must block CONFORMS: {report}"
);
assert!(
report.contains("1. Verification runner error"),
"the runner error must be a numbered complaint: {report}"
);
assert!(
report.contains("0 unmet/diverged — 1 red verification run(s) (fail-closed)."),
"the header must count the runner error as a red run: {report}"
);
}
#[test]
fn adversary_verdict_environment_marker_without_declaration_is_not_red() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"verification_commands": "[]",
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": "ENVIRONMENT: verification runner error: x — the declared commands could not be executed"
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: CONFORMS"),
"with nothing declared the ENVIRONMENT marker is doctrine-PARTIAL, not a red run: {report}"
);
assert!(
!report.contains("Verification runner error"),
"the ENVIRONMENT marker must not yield a run complaint: {report}"
);
}
#[test]
fn adversary_verdict_timeout_run_named_as_timeout() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": -1, "duration_s": 900.0, "tail": "TIMEOUT after 900s"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"a timed-out run must force DIVERGES: {report}"
);
assert!(
report.contains("1. Verification run `cargo test` — TIMEOUT (exit -1)"),
"the -1 sentinel must be named as a TIMEOUT, not a bare exit code: {report}"
);
}
#[test]
fn adversary_verdict_red_runs_number_after_faults_and_criteria() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": ["PIPELINE-FAULT: x"],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "UNMET",
"evidence": "", "complaint": "no impl found"}
],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": 101, "duration_s": 1.0, "tail": "FAILED"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
let fault = report
.find("1. PIPELINE-FAULT")
.expect("the pipeline fault must be complaint 1");
let criterion = report
.find("2. Acceptance criterion")
.expect("the unmet criterion must be complaint 2");
let red = report
.find("3. Verification run")
.expect("the red run must be complaint 3");
assert!(
fault < criterion && criterion < red,
"complaints must be ordered fault, criterion, red run: {report}"
);
}
#[test]
fn adversary_verdict_no_criteria_still_lists_red_runs() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"crit_verdicts": [],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": 101, "duration_s": 1.0, "tail": "FAILED"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"no criteria must fail closed: {report}"
);
assert!(
report.contains("Criteria: none provided."),
"the no-criteria header must be kept: {report}"
);
assert!(
report.contains("2. Verification run `cargo test`"),
"the red run must be numbered after the no-criteria complaint: {report}"
);
}
#[test]
fn adversary_verdict_environment_marker_with_string_declaration_blocks() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
// Production path: graph variables land in state as strings, so the
// declaration is a JSON-encoded list, not a list.
let mut state = json!({
"pipeline_faults": [],
"verification_commands": "[\"cargo test\"]",
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "MET",
"evidence": "src/x.rs:1 + test src/x.rs:99", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": "ENVIRONMENT: verification runner error: x — the declared commands could not be executed"
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: DIVERGES"),
"a JSON-string declaration must count as declared: {report}"
);
assert!(
report.contains("1. Verification runner error"),
"the runner error must be a numbered complaint: {report}"
);
state["verification_commands"] = json!("[\" \"]");
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.starts_with("ADVERSARIAL_REVIEW: CONFORMS"),
"blank items are not a declaration, so the ENVIRONMENT marker is not a red run: {report}"
);
}
#[test]
fn adversary_verdict_died_and_red_header_has_single_period() {
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let state = json!({
"pipeline_faults": [],
"crit_verdicts": [
{"id": "c1", "text": "does X", "status": "UNMET",
"evidence": "PIPELINE-FAULT: criterion x", "complaint": ""}
],
"extra_complaints": [],
"observations": "",
"exec_results": [{"cmd": "cargo test", "exit": 101, "duration_s": 1.0, "tail": "FAILED"}]
});
let out = run_adversary_script("verdict.py", &state);
let report = out["adv_report"].as_str().unwrap();
assert!(
report.contains("died (fail-closed) — 1 red verification run(s) (fail-closed)."),
"both header notes must be joined with a single em dash: {report}"
);
assert!(
!report.contains(".."),
"the header must end with exactly one period: {report}"
);
}
/// The fail-closed CONFORMS block on unrun declared commands is described
/// in three places that must keep agreeing: the runner's ENVIRONMENT
/// marker, the node descriptions, and check_criterion's instructions.
#[test]
fn adversary_fail_closed_wording_pins() {
use crate::graph::NodeType;
let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("assets/agents/adversary");
let runner = read_to_string(dir.join("scripts/run_checks.py")).unwrap();
assert!(
runner.contains("the verdict blocks CONFORMS when"),
"run_checks.py's ENVIRONMENT marker must point at the verdict's fail-closed block"
);
let graph = load_bundled_graph("adversary");
let run_checks = graph
.get_node("run_checks")
.expect("adversary graph must have a run_checks node");
assert!(
run_checks.description.contains("blocks CONFORMS"),
"run_checks description must state the verdict blocks CONFORMS: {}",
run_checks.description
);
let verdict = graph
.get_node("verdict")
.expect("adversary graph must have a verdict node");
assert!(
verdict
.description
.contains("zero recorded red verification runs"),
"verdict description must list red runs among the CONFORMS requirements: {}",
verdict.description
);
let check = graph
.get_node("check_criterion")
.expect("adversary graph must have a check_criterion node");
assert!(
check.description.contains(
"the deterministic verdict blocks CONFORMS when declared commands did not run"
),
"check_criterion description must hand the fail-closed block to the verdict: {}",
check.description
);
let NodeType::Llm(llm) = &check.node_type else {
panic!("check_criterion must be an llm node");
};
let instructions = llm
.instructions
.as_deref()
.expect("check_criterion must have instructions");
assert!(
instructions.contains("deterministic verdict additionally blocks CONFORMS"),
"check_criterion instructions must hand the fail-closed block to the verdict"
);
}
#[test]
fn adversary_crit_gate_contract_regression() {
if !cmd_available("python3") {