fix(review-suite): pass verification_commands as a declared variable, never LLM-extracted from prose
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.
- adversary + review-gauntlet graphs declare verification_commands as a graph
variable (default '[]'); the key leaves initial_state (or_insert seeding
would otherwise shadow it) and leaves both parse nodes' instructions,
output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
arrive as strings); '[]'/[]/missing = none declared; any other shape fails
closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
--variables / --agent-variable form and the trust boundary. Known residual
(llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
prompt-prose injection proof, verdict wiring, asset pins for both graphs and
orchestrator configs, dispatch-inputs > variable-override > default
precedence.
This commit is contained in:
1 parent
00c3b60c2f
commit
300c7aeff0
9 files changed
+501
-47
No files matched your search
@@ -97,10 +97,18 @@ Plan / acceptance criteria:
|
||||
Local-run recipe / usage suites:
|
||||
<how to boot the service locally + where existing suites live — enables the probe lane>
|
||||
|
||||
Verification commands: (optional — exact build/test/lint commands, passed
|
||||
verbatim to the adversary lane; omitted = none declared, never invented)
|
||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)" \
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
```
|
||||
|
||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)"
|
||||
Verification commands are NOT part of the prompt. They are the declared
|
||||
`verification_commands` graph variable (default `'[]'`; a JSON array of shell
|
||||
commands as a string, since agent variables arrive as strings), also settable
|
||||
from the CLI:
|
||||
|
||||
```sh
|
||||
coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
|
||||
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
||||
```
|
||||
|
||||
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
|
||||
@@ -119,9 +127,19 @@ final report verbatim.
|
||||
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
|
||||
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
|
||||
pass.
|
||||
- Caller-declared verification commands are extracted verbatim by `parse` and
|
||||
passed through to the adversary lane's prompt — never invented or
|
||||
auto-detected; undeclared stays undeclared.
|
||||
- The `verification_commands` variable is forwarded to the adversary lane as a
|
||||
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
||||
lands the raw value in the child's state) — never through the lane prompt
|
||||
and never extracted by `parse`. The adversary's `run_checks` stage executes
|
||||
those commands with a shell, which makes the declaration a trust boundary:
|
||||
the prompt also carries plan text pasted from the repo under review, so
|
||||
commands must never be inferred from it (residual: the engine still merges
|
||||
every top-level key of `parse`'s output into state, so an extra
|
||||
`verification_commands` key emitted by the LLM is not yet blocked).
|
||||
Undeclared stays undeclared
|
||||
(`'[]'` ⇒ "none declared"); a declaration that is not a JSON array of
|
||||
strings executes nothing and records a `PIPELINE-FAULT`, which the
|
||||
adversary lane reports as DIVERGES and the gate as BLOCKED.
|
||||
- Signals failures degrade gracefully: if git can't compute the diff, lane
|
||||
selection falls back to caller context + forced lanes, and the report notes
|
||||
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
||||
|
||||
@@ -31,6 +31,12 @@ variables:
|
||||
- name: project_dir
|
||||
description: Repository the change lives in; signals and every review lane run against it.
|
||||
default: '.'
|
||||
- name: verification_commands
|
||||
description: |
|
||||
JSON array of shell commands the CALLER declares for the deterministic
|
||||
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
||||
prompt; empty array = none declared.
|
||||
default: '[]'
|
||||
|
||||
settings:
|
||||
max_loop_iterations: 10
|
||||
@@ -44,7 +50,6 @@ initial_state:
|
||||
security_posture: standard
|
||||
probe_context: ''
|
||||
extra_context: ''
|
||||
verification_commands: []
|
||||
forced_lanes: []
|
||||
changed_files: []
|
||||
file_count: 0
|
||||
@@ -109,10 +114,6 @@ nodes:
|
||||
names). Empty array = automatic selection.
|
||||
- `extra_context`: any remaining caller context reviewers should see,
|
||||
condensed.
|
||||
- `verification_commands`: the exact build/test/lint shell commands the
|
||||
caller explicitly declares for verification (verbatim strings, in the
|
||||
order given). Empty array when the caller declares none — NEVER invent,
|
||||
guess, or auto-detect commands; undeclared stays undeclared.
|
||||
prompt: |
|
||||
Extract the review-request facts from this request:
|
||||
|
||||
@@ -139,11 +140,7 @@ nodes:
|
||||
type: string
|
||||
project_dir_in:
|
||||
type: string
|
||||
verification_commands:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in, verification_commands]
|
||||
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
@@ -326,6 +323,8 @@ nodes:
|
||||
description: Plan-conformance review lane.
|
||||
agent: adversary
|
||||
timeout: 7200
|
||||
inputs:
|
||||
verification_commands: '{{verification_commands}}'
|
||||
prompt: |
|
||||
Adversarial plan-conformance review in {{project_dir}}.
|
||||
|
||||
@@ -334,7 +333,8 @@ nodes:
|
||||
The spec / plan / acceptance criteria it must satisfy:
|
||||
{{plan_context}}
|
||||
|
||||
Verification commands (declared by the caller; empty = none declared): {{verification_commands}}
|
||||
Verification commands arrive as a structured input (the
|
||||
verification_commands variable), not in this prompt.
|
||||
|
||||
{{extra_context}}
|
||||
|
||||
|
||||
Reference in new issue
Block a user