fix(TASK-017): bound run_checks runtime and fail-close its node-level death

Adversary conformance-review fixes for the TASK-017 exec-evidence stage:

- run_checks.py: total-runtime deadline (3300s = node timeout minus margin);
  each command gets min(900s, remaining budget), commands past the deadline
  are recorded as {cmd, skipped: deadline} entries so in-script handling
  stays authoritative instead of the node timeout killing the script from
  outside. POSIX hardening: per-command process group (start_new_session)
  + killpg on timeout so hung grandchildren cannot linger; non-POSIX
  limitation documented in the docstring.
- graph.yaml: run_checks gains fallback: pipeline_fault; check_criterion
  treats SKIPPED records as at-best PARTIAL (unproven, never failed).
- pipeline_fault.py: run_checks-stage attribution grounded in real engine
  behavior (script-node fallback writes nothing into state, executor.rs
  script arm) — nonempty diff_text proves facts completed, so the fault is
  the verification runner; emits 'PIPELINE-FAULT: verification runner
  killed'. Empty diff_text still attributes to diff resolution.
- verdict.py: renders skipped records as [SKIPPED] instead of a bogus
  [FAIL] exit-? line; non-degraded verdict semantics untouched.
- tests: outer-except ENVIRONMENT degradation, deadline timeout+skip (env
  seam ADVERSARY_RUN_CHECKS_DEADLINE_SECS), pipeline_fault stage
  attribution (parse / facts / run_checks), check_criterion prompt cites
  {{exec_results}}, verdict SKIPPED rendering; run_adversary_script_env
  helper for env-carrying invocations.
- cargo fmt violations in 3d7dea9 test code fixed (fmt --check exit 0).

assets/agents/adversary/graph.yaml

@@ -15,8 +15,9 @@ description: |
   DIVERGES (fail-closed) naming the missing spec. Caller-declared verification
   commands are executed ONCE by a deterministic stage and recorded — criterion
   branches cite the record and never run anything themselves. Any pipeline
-  fault (parse, diff resolution, or holistic failure) lands a PIPELINE-FAULT
-  marker and fails closed into DIVERGES: degraded is never passing.
+  fault (parse, diff resolution, verification runner, or holistic failure)
+  lands a PIPELINE-FAULT marker and fails closed into DIVERGES: degraded is
+  never passing.
 
 version: '1.0'
 
@@ -140,9 +141,10 @@ nodes:
   run_checks:
     id: run_checks
     type: script
-    description: Deterministic execution evidence — runs ONLY the caller-declared verification commands in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker; never fails the node.
+    description: Deterministic execution evidence — runs ONLY the caller-declared verification commands in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker; a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
     script: scripts/run_checks.py
     timeout: 3600
+    fallback: pipeline_fault
     next: map_criteria
 
   map_criteria:
@@ -198,6 +200,9 @@ nodes:
       - "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
         ENVIRONMENT note in the complaint naming the exact command the caller
         should declare via verification_commands (e.g. "cargo test --all").
+      - a SKIPPED record (the runner's total deadline expired before that
+        command ran) ⇒ at best PARTIAL for the criteria it covers, same
+        treatment as an ENVIRONMENT marker: unproven, never failed.
 
       Output format — respond with ONLY this JSON object, no prose around it:
       {"id": "<the criterion's id, echoed>",
@@ -325,7 +330,7 @@ nodes:
   pipeline_fault:
     id: pipeline_fault
     type: script
-    description: Fail-closed fault marker (reached only via fallback routes from parse/facts) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
+    description: Fail-closed fault marker (reached only via fallback routes from parse/facts/run_checks) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
     script: scripts/pipeline_fault.py
     timeout: 30
     next: verdict

assets/agents/adversary/scripts/pipeline_fault.py

@@ -1,15 +1,23 @@
 #!/usr/bin/env python3
 """Fail-closed fault marker for the adversary graph.
 
-Reached ONLY via `fallback:` routes (parse, facts). Appends a machine-readable
-"PIPELINE-FAULT:" marker to `pipeline_faults` and continues to `verdict`,
-which turns any recorded fault into DIVERGES — the graph never dies without
-its sentinel, and a degraded run can never read as CONFORMS.
+Reached ONLY via `fallback:` routes (parse, facts, run_checks). Appends a
+machine-readable "PIPELINE-FAULT:" marker to `pipeline_faults` and continues
+to `verdict`, which turns any recorded fault into DIVERGES — the graph never
+dies without its sentinel, and a degraded run can never read as CONFORMS.
 
-Stage attribution: an llm-node failure writes an "LLM node …failed: <chain>"
-string into `parse_failure` via parse's `state_updates`; anything else means
-the deterministic facts (diff resolution) script failed, which leaves no
-failure text in state by design.
+Stage attribution — grounded in engine behavior: a SCRIPT node's fallback
+route writes NOTHING into state (src/graph/executor.rs script arm routes to
+the fallback without recording the error), so the two script stages are
+discriminated by which stage outputs are present:
+- parse (llm): its `state_updates` wrote an "LLM node …failed: <chain>"
+  string into `parse_failure` — attribute to parse.
+- run_checks vs facts (scripts): diff_facts.py unconditionally writes a
+  NONEMPTY `diff_text` (even its internal-failure path writes
+  "(diff unavailable)"), while the graph's initial value is ''. A nonempty
+  `diff_text` proves facts completed, so the fault is the verification
+  runner (run_checks killed at the node level, e.g. by the node timeout);
+  an empty `diff_text` means facts itself died.
 """
 
 import json
@@ -30,11 +38,19 @@ def main():
     faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
     parse_failure = state.get("parse_failure")
     detail = parse_failure.strip() if isinstance(parse_failure, str) else ""
+    diff_text = state.get("diff_text")
+    facts_completed = isinstance(diff_text, str) and bool(diff_text.strip())
     if detail.startswith("LLM node"):
         detail = detail.replace("\n", " ")
         if len(detail) > MAX_DETAIL_CHARS:
             detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
         faults.append(f"PIPELINE-FAULT: parse failed — cannot review: {detail}")
+    elif facts_completed:
+        faults.append(
+            "PIPELINE-FAULT: verification runner killed — the run_checks stage "
+            "died at the node level before recording exec_results; execution "
+            "criteria are unproven"
+        )
     else:
         faults.append(
             "PIPELINE-FAULT: diff resolution failed — changed files and diff text "

assets/agents/adversary/scripts/run_checks.py

@@ -9,14 +9,32 @@ they never execute anything themselves.
 NO auto-detection: undeclared commands mean a "none declared" marker
 (fail-visible, not fail-guessed). Any runner error degrades into an
 ENVIRONMENT marker — this script never fails the node.
+
+Budget: commands run sequentially, so TOTAL runtime is bounded by a deadline
+(TOTAL_DEADLINE_SECS = 3300s, the run_checks node's 3600s timeout minus
+margin). Without it, a handful of hanging commands (4 x 900s) would blow the
+NODE timeout and the executor would kill this script from OUTSIDE — the
+in-script degradation would never fire. Each command gets
+min(PER_COMMAND_TIMEOUT_SECS, remaining budget); commands the deadline
+leaves no budget for are recorded as {"cmd", "skipped": "deadline"} entries,
+so the in-script handling stays authoritative. The env var
+ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam).
+
+Process hygiene: commands run through the shell. On POSIX each command gets
+its own session (start_new_session) and a timeout SIGKILLs the whole process
+group, so grandchildren of a hung command cannot linger. On non-POSIX
+platforms only the direct shell child is killed — grandchildren of a
+timed-out command may survive (documented limitation).
 """
 
 import json
 import os
+import signal
 import subprocess
 import time
 
 PER_COMMAND_TIMEOUT_SECS = 900
+TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
 TAIL_LINES = 50
 MAX_TAIL_CHARS = 8000
 
@@ -35,6 +53,45 @@ def tail_of(text):
     return tail
 
 
+def run_one(cmd, proj, budget):
+    """Run one command, hard-bounded by `budget` seconds; return its record."""
+    start = time.monotonic()
+    popen_kwargs = {"start_new_session": True} if os.name == "posix" else {}
+    p = subprocess.Popen(
+        cmd,
+        shell=True,
+        cwd=proj,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        text=True,
+        **popen_kwargs,
+    )
+    try:
+        out_s, err_s = p.communicate(timeout=budget)
+    except subprocess.TimeoutExpired:
+        if os.name == "posix":
+            try:
+                os.killpg(os.getpgid(p.pid), signal.SIGKILL)
+            except (ProcessLookupError, PermissionError):
+                pass
+        else:
+            p.kill()
+        p.communicate()  # reap the child and drain the pipes
+        return {
+            "cmd": cmd,
+            "exit": -1,
+            "duration_s": round(time.monotonic() - start, 1),
+            "tail": f"TIMEOUT: no exit within {budget:.0f}s — recorded as a failing run",
+        }
+    combined = (out_s or "") + (("\n" + err_s) if err_s else "")
+    return {
+        "cmd": cmd,
+        "exit": p.returncode,
+        "duration_s": round(time.monotonic() - start, 1),
+        "tail": tail_of(combined),
+    }
+
+
 def main():
     state = load_state()
     declared = state.get("verification_commands")
@@ -61,39 +118,14 @@ def main():
     proj = os.path.expanduser(
         (state.get("project_dir") or "").strip() or "."
     )
+    deadline = time.monotonic() + TOTAL_DEADLINE_SECS
     results = []
     for cmd in cmds:
-        start = time.monotonic()
-        try:
-            r = subprocess.run(
-                cmd,
-                shell=True,
-                cwd=proj,
-                capture_output=True,
-                text=True,
-                timeout=PER_COMMAND_TIMEOUT_SECS,
-            )
-            combined = (r.stdout or "") + (("\n" + r.stderr) if r.stderr else "")
-            results.append(
-                {
-                    "cmd": cmd,
-                    "exit": r.returncode,
-                    "duration_s": round(time.monotonic() - start, 1),
-                    "tail": tail_of(combined),
-                }
-            )
-        except subprocess.TimeoutExpired:
-            results.append(
-                {
-                    "cmd": cmd,
-                    "exit": -1,
-                    "duration_s": round(time.monotonic() - start, 1),
-                    "tail": (
-                        f"TIMEOUT: no exit within {PER_COMMAND_TIMEOUT_SECS}s — "
-                        "recorded as a failing run"
-                    ),
-                }
-            )
+        remaining = deadline - time.monotonic()
+        if remaining <= 0:
+            results.append({"cmd": cmd, "skipped": "deadline"})
+            continue
+        results.append(run_one(cmd, proj, min(PER_COMMAND_TIMEOUT_SECS, remaining)))
     print(json.dumps({"exec_results": results}))
 
 

assets/agents/adversary/scripts/verdict.py

@@ -73,15 +73,22 @@ def append_criterion_complaints(lines, verdicts, extra, start):
 def render_exec_results(exec_results):
     """Lines for the "Verification runs:" report section.
 
-    exec_results is a list of {cmd, exit, duration_s, tail} records from
-    run_checks.py, or a string marker ("none declared —…" / "ENVIRONMENT: …"),
-    or empty when the run never reached the verification stage.
+    exec_results is a list of {cmd, exit, duration_s, tail} records (or
+    {cmd, skipped} for commands the runner's total deadline left no budget
+    for) from run_checks.py, or a string marker ("none declared —…" /
+    "ENVIRONMENT: …"), or empty when the run never reached the stage.
     """
     if isinstance(exec_results, list):
         out = []
         for r in exec_results:
             if not isinstance(r, dict):
                 continue
+            if r.get("skipped"):
+                out.append(
+                    f"- [SKIPPED] `{r.get('cmd', '?')}` — never ran: the runner's "
+                    "total deadline expired first; covered criteria are unproven"
+                )
+                continue
             status = "PASS" if r.get("exit") == 0 else "FAIL"
             out.append(
                 f"- [{status}] `{r.get('cmd', '?')}` — exit {r.get('exit', '?')}, "