feat(TASK-017): adversary execution evidence + fail-closed routing
- parse extracts caller-declared verification_commands (schema + initial_state);
never invents commands. parse gains fallback + state_updates capturing the
failure string (TASK-016 fallback-capture warning stays silent).
- new run_checks script node (facts -> run_checks -> map_criteria, timeout
3600): runs each declared command in the project dir (900s per-command
timeout), records exec_results {cmd, exit, duration_s, tail~50}; 'none
declared' / ENVIRONMENT markers instead of guessing or failing the node.
- check_criterion judges execution criteria against {{exec_results}} only:
recorded green run = MET citing it, recorded failure = UNMET, markers =
PARTIAL with an ENVIRONMENT note.
- fail-closed routing (R1/R2): parse/facts fallback -> pipeline_fault marker
script -> verdict; holistic fallback -> verdict with holistic_failure
captured. verdict.py: any PIPELINE-FAULT forces DIVERGES with the fault as
complaint #1, per-criterion results still reported; report gains a
'Verification runs:' section. Non-degraded semantics unchanged.
- R5 rust tests (python3-guarded, local cmd_available): verdict fault path,
holistic-failure fault, CONFORMS regression, run_checks markers + recorded
runs, crit_gate contract regression.
This commit is contained in:
5 files changed
+564
-26
No files matched your search
@@ -12,7 +12,11 @@ description: |
|
||||
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
|
||||
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
|
||||
requires every criterion MET and zero holistic complaints. Missing plan =
|
||||
DIVERGES (fail-closed) naming the missing spec.
|
||||
DIVERGES (fail-closed) naming the missing spec. Caller-declared verification
|
||||
commands are executed ONCE by a deterministic stage and recorded — criterion
|
||||
branches cite the record and never run anything themselves. Any pipeline
|
||||
fault (parse, diff resolution, or holistic failure) lands a PIPELINE-FAULT
|
||||
marker and fails closed into DIVERGES: degraded is never passing.
|
||||
|
||||
version: '1.0'
|
||||
|
||||
@@ -42,12 +46,17 @@ initial_state:
|
||||
changed_files: []
|
||||
diff_text: ''
|
||||
diff_note: ''
|
||||
verification_commands: []
|
||||
exec_results: ''
|
||||
crit_verdicts: []
|
||||
extra_complaints: []
|
||||
observations: ''
|
||||
adv_report: ''
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
parse_failure: ''
|
||||
holistic_failure: ''
|
||||
pipeline_faults: []
|
||||
|
||||
start: parse
|
||||
|
||||
@@ -75,6 +84,10 @@ nodes:
|
||||
verbatim — the holistic pass checks the diff didn't silently override them.
|
||||
- `scope_notes`: what the plan says NOT to touch / explicit boundaries.
|
||||
- `extra_context`: anything else conformance-relevant, condensed.
|
||||
- `verification_commands`: the exact build/test/lint shell commands the
|
||||
caller explicitly declares for verification (verbatim strings, in the
|
||||
order given). Empty array when the caller declares none — NEVER invent,
|
||||
guess, or auto-detect commands; undeclared stays undeclared.
|
||||
prompt: |
|
||||
Extract the conformance-review facts from this request:
|
||||
|
||||
@@ -103,9 +116,16 @@ nodes:
|
||||
type: string
|
||||
project_dir_in:
|
||||
type: string
|
||||
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in]
|
||||
verification_commands:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in, verification_commands]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: pipeline_fault
|
||||
next: facts
|
||||
|
||||
facts:
|
||||
@@ -114,6 +134,15 @@ nodes:
|
||||
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
|
||||
script: scripts/diff_facts.py
|
||||
timeout: 120
|
||||
fallback: pipeline_fault
|
||||
next: run_checks
|
||||
|
||||
run_checks:
|
||||
id: run_checks
|
||||
type: script
|
||||
description: Deterministic execution evidence — runs ONLY the caller-declared verification commands in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker; never fails the node.
|
||||
script: scripts/run_checks.py
|
||||
timeout: 3600
|
||||
next: map_criteria
|
||||
|
||||
map_criteria:
|
||||
@@ -158,6 +187,18 @@ nodes:
|
||||
values shaped to the criterion's example, TODO/stub presented as done —
|
||||
those are PARTIAL or DIVERGED with the gaming named in the complaint.
|
||||
|
||||
Execution criteria ("tests pass", "lint clean", "build succeeds") are
|
||||
judged ONLY against the recorded verification runs in your prompt — you
|
||||
have no execution tools BY DESIGN and must never claim to have run
|
||||
anything yourself:
|
||||
- a recorded green run (exit 0) of a command covering the criterion is
|
||||
the proving evidence — MET, citing that run (the command + exit 0).
|
||||
- a recorded failing run covering the criterion ⇒ UNMET, citing the
|
||||
recorded exit code and output tail.
|
||||
- "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
|
||||
ENVIRONMENT note in the complaint naming the exact command the caller
|
||||
should declare via verification_commands (e.g. "cargo test --all").
|
||||
|
||||
Output format — respond with ONLY this JSON object, no prose around it:
|
||||
{"id": "<the criterion's id, echoed>",
|
||||
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
|
||||
@@ -181,6 +222,9 @@ nodes:
|
||||
|
||||
The diff{{diff_note}}:
|
||||
{{diff_text}}
|
||||
|
||||
Verification runs (recorded by the deterministic run_checks stage):
|
||||
{{exec_results}}
|
||||
{{gate_feedback}}
|
||||
tools:
|
||||
- fs_read
|
||||
@@ -273,12 +317,23 @@ nodes:
|
||||
required: [extra_complaints, observations]
|
||||
max_iterations: 10
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
holistic_failure: '{{output}}'
|
||||
fallback: verdict
|
||||
next: verdict
|
||||
|
||||
pipeline_fault:
|
||||
id: pipeline_fault
|
||||
type: script
|
||||
description: Fail-closed fault marker (reached only via fallback routes from parse/facts) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
|
||||
script: scripts/pipeline_fault.py
|
||||
timeout: 30
|
||||
next: verdict
|
||||
|
||||
verdict:
|
||||
id: verdict
|
||||
type: script
|
||||
description: Deterministic verdict — CONFORMS requires every criterion MET and zero holistic complaints; no criteria at all fails closed. Assembles the exact sentinel format.
|
||||
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, and zero pipeline faults; no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
|
||||
script: scripts/verdict.py
|
||||
timeout: 30
|
||||
next: done
|
||||
|
||||
Reference in new issue
Block a user