fix(review-suite): pass verification_commands as a declared variable, never LLM-extracted from prose
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.
- adversary + review-gauntlet graphs declare verification_commands as a graph
variable (default '[]'); the key leaves initial_state (or_insert seeding
would otherwise shadow it) and leaves both parse nodes' instructions,
output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
arrive as strings); '[]'/[]/missing = none declared; any other shape fails
closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
--variables / --agent-variable form and the trust boundary. Known residual
(llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
prompt-prose injection proof, verdict wiring, asset pins for both graphs and
orchestrator configs, dispatch-inputs > variable-override > default
precedence.
This commit is contained in:
1 parent
00c3b60c2f
commit
300c7aeff0
9 files changed
+501
-47
No files matched your search
@@ -315,8 +315,13 @@ instructions: |
|
||||
Rigor: <rigor> Security posture: <posture>
|
||||
Plan / acceptance criteria: <pasted verbatim — enables the adversary lane; omit when none>
|
||||
Local-run recipe / usage suites: <from the plan — enables the probe lane; omit when none>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
```
|
||||
|
||||
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the prompt
|
||||
also carries pasted plan text from the repo under review, and whatever is declared here is
|
||||
executed with a shell — a trust boundary the prose must not cross.
|
||||
|
||||
It computes diff signals deterministically, selects lanes (code-review always; adversary when a
|
||||
spec/plan is present; security on attack-surface signals; probe on consumer-facing surface with a
|
||||
run recipe — judgment can only WIDEN the selection, never narrow it), runs the selected lanes in
|
||||
@@ -392,8 +397,12 @@ instructions: |
|
||||
|
||||
PLAN — acceptance criteria to check against:
|
||||
<paste the task/step spec + acceptance criteria VERBATIM — not a summary>"
|
||||
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||
```
|
||||
|
||||
Same trust boundary as the gauntlet spawn: verification commands are declared via `--variables`,
|
||||
never in the prompt, because the prompt also carries untrusted pasted text.
|
||||
|
||||
### Handling adversary findings
|
||||
|
||||
- **`ADVERSARIAL_REVIEW: DIVERGES` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <complaints pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `adversary` ONCE to confirm it now CONFORMS; if it still DIVERGES on the same criteria after one fix cycle, STOP and escalate to the user (the plan or the approach may be wrong — consider `oracle`).
|
||||
|
||||
Reference in new issue
Block a user