fix(adversary): red verification runs block CONFORMS; re-derive timeout envelope; robust runner

verdict.py: a recorded verification run with exit≠0 (including the -1
TIMEOUT sentinel) or a skipped record now forces DIVERGES with a numbered
complaint naming the command — previously a [FAIL] line sat in the report
appendix while the sentinel read CONFORMS, and callers route on the
sentinel alone. Fail-closed by the same rule: an ENVIRONMENT runner-error
marker when commands WERE declared is one red complaint (the declared
commands never ran, so the run is unproven); the "none declared" marker
and an empty declaration stay non-red. Headers name the red-run count.

run_checks.py: decode command output with encoding=utf-8/errors=replace
(a single non-UTF-8 byte used to raise inside communicate() and collapse
the whole evidence record to an ENVIRONMENT marker); bound the post-kill
pipe drain by DRAIN_TIMEOUT_SECS so a session-escaped grandchild holding
the pipe cannot turn a TIMEOUT record into a node kill.

graph.yaml: settings.timeout 5400 → 9000 = 3600 (run_checks node timeout)
+ 5400 (the previous whole-graph budget, kept for every other stage); the
old value predated run_checks and a graph-level timeout kills from
outside, losing the DIVERGES sentinel. Descriptions and README doctrine
updated (CONFORMS = all MET + zero holistic complaints + zero pipeline
faults + zero red runs).

Tests: envelope pin (settings >= run_checks + 5400), gauntlet lane
timeouts > child graph timeouts and gauntlet >= max_attempts × lane +
margin over every agent lane, non-UTF-8 output, bounded drain (unix),
red/skipped/timeout/none-declared/ENVIRONMENT×declaration matrix,
numbering continuity, header punctuation, wording pins on graph.yaml,
run_checks.py and both READMEs. The deadline test's hung command is now
python3-based instead of GNU sleep (cmd.exe on the Windows CI leg).
This commit is contained in:
Dark-Alex-17 committed 2026-09-14 10:50:34 -06:00
1 parent 156f3193a9
commit 68e893d684
5 files changed
+683 -33

No files matched your search

+18 -8
View File
@@ -17,8 +17,8 @@ from the repo under review, and whatever is declared here runs with
shell=True. Variables land in state as strings, so a JSON-encoded list is
accepted alongside a real list. A declaration that is not a JSON array of
strings is a caller contract violation, not a runner hiccup: it executes
NOTHING and records a PIPELINE-FAULT (fail-closed into DIVERGES) instead of
degrading to the soft ENVIRONMENT marker. The engine merges only the keys an
NOTHING and fails closed as a PIPELINE-FAULT (complaint #1) — it never
degrades to an ENVIRONMENT marker. The engine merges only the keys an
llm node's `output_schema.properties` declares, so a parse LLM coaxed into
emitting an extra `verification_commands` key cannot overwrite the declared
value: the undeclared key is dropped before it reaches state.
@@ -35,9 +35,12 @@ ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam).
Process hygiene: commands run through the shell. On POSIX each command gets
its own session (start_new_session) and a timeout SIGKILLs the whole process
group, so grandchildren of a hung command cannot linger. On non-POSIX
platforms only the direct shell child is killed — grandchildren of a
timed-out command may survive (documented limitation).
group, so grandchildren of a hung command cannot linger. The post-kill
reap/drain is itself bounded by DRAIN_TIMEOUT_SECS: a grandchild that escaped
the session (its own start_new_session) but still holds the stdout pipe
would otherwise block the drain until it exits. On non-POSIX platforms only
the direct shell child is killed — grandchildren of a timed-out command may
survive (documented limitation).
Verification commands run with GRAPH_STATE* (GRAPH_STATE, GRAPH_STATE_FILE)
scrubbed from their environment: the reviewed repo's own tests may exec these very
@@ -53,6 +56,7 @@ import time
PER_COMMAND_TIMEOUT_SECS = 900
TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
DRAIN_TIMEOUT_SECS = 10
TAIL_LINES = 50
MAX_TAIL_CHARS = 8000
@@ -112,6 +116,8 @@ def run_one(cmd, proj, budget):
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
env=env,
**popen_kwargs,
)
@@ -125,7 +131,10 @@ def run_one(cmd, proj, budget):
pass
else:
p.kill()
p.communicate() # reap the child and drain the pipes
try:
p.communicate(timeout=DRAIN_TIMEOUT_SECS)
except subprocess.TimeoutExpired:
pass
return {
"cmd": cmd,
"exit": -1,
@@ -189,8 +198,9 @@ except Exception as e: # noqa: BLE001 — never fail the node; degrade visibly
{
"exec_results": (
f"ENVIRONMENT: verification runner error: {e} — the declared "
"commands could not be executed; treat execution criteria as "
"PARTIAL (unproven), not as failed"
"commands could not be executed; execution criteria are unproven "
"(PARTIAL at the criterion level; the verdict blocks CONFORMS when "
"commands were declared)"
)
}
)