Dark-Alex-17
300c7aeff0
verification_commands — the shell commands the adversary's run_checks stage
executes — were extracted by the LLM parse node from {{initial_prompt}}, a
flat prompt that also carries untrusted plan/diff text pasted from the repo
under review. A planted '## VERIFICATION' block was indistinguishable from the
caller's declaration: prompt injection -> arbitrary command execution.
- adversary + review-gauntlet graphs declare verification_commands as a graph
variable (default '[]'); the key leaves initial_state (or_insert seeding
would otherwise shadow it) and leaves both parse nodes' instructions,
output_schema.properties and required.
- review-gauntlet forwards it to run_adversary via a structured inputs:
passthrough (lone {{verification_commands}} template), not lane-prompt prose.
- run_checks.py accepts a JSON list or a JSON-encoded string of one (variables
arrive as strings); '[]'/[]/missing = none declared; any other shape fails
closed with a PIPELINE-FAULT (nothing executed) that verdict.py turns into
DIVERGES complaint #1.
- architect/sisyphus spawn templates and both READMEs document the
--variables / --agent-variable form and the trust boundary. Known residual
(llm-node auto-merge of undeclared output keys) is disclosed in the docs.
- Tests: list/JSON-string acceptance, fail-closed no-execution proof,
prompt-prose injection proof, verdict wiring, asset pins for both graphs and
orchestrator configs, dispatch-inputs > variable-override > default
precedence.
415 lines
14 KiB
YAML
415 lines
14 KiB
YAML
name: review-gauntlet
|
|
description: |
|
|
Graph-based post-implementation review gate — "the done gate". Given a diff
|
|
spec and the work's context (plan/acceptance criteria, rigor, security
|
|
posture), it deterministically computes diff signals, selects the applicable
|
|
independent review lanes — code-reviewer (always), adversary (when a
|
|
spec/plan exists), security-reviewer (attack-surface signals), probe
|
|
(consumer-facing surface with a local-run recipe) — runs the selected lanes
|
|
IN PARALLEL as isolated sub-agents, and machine-parses their verdict
|
|
sentinels into one deterministic verdict: GAUNTLET: PASS or
|
|
GAUNTLET: BLOCKED. An orchestrator that must not claim "done" without green
|
|
reviews spawns this once instead of judging trigger rules and parsing
|
|
verdicts itself — the graph structure makes skipping a lane or misreading a
|
|
verdict impossible. Invariants: lane selection can only be WIDENED by LLM
|
|
judgment, never narrowed (deterministic rules are the floor); a missing
|
|
verdict sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks
|
|
with an environment note (it is never PASS or FAIL); reviewer lanes run
|
|
verdict-independent — no cross-lane messaging.
|
|
|
|
version: '1.0'
|
|
|
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
|
# review-gauntlet(2) → code-reviewer(3) →
|
|
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
|
# This agent spawns depth-3 children; 4 = one level of headroom.
|
|
max_agent_depth: 4
|
|
max_concurrent_agents: 12
|
|
|
|
variables:
|
|
- name: project_dir
|
|
description: Repository the change lives in; signals and every review lane run against it.
|
|
default: '.'
|
|
- name: verification_commands
|
|
description: |
|
|
JSON array of shell commands the CALLER declares for the deterministic
|
|
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
|
prompt; empty array = none declared.
|
|
default: '[]'
|
|
|
|
settings:
|
|
max_loop_iterations: 10
|
|
log_state_snapshots: true
|
|
timeout: 10800
|
|
|
|
initial_state:
|
|
diff_spec: worktree
|
|
plan_context: ''
|
|
rigor: production
|
|
security_posture: standard
|
|
probe_context: ''
|
|
extra_context: ''
|
|
forced_lanes: []
|
|
changed_files: []
|
|
file_count: 0
|
|
signals_summary: ''
|
|
signals_error: ''
|
|
touches_auth: false
|
|
touches_deps: false
|
|
touches_exec: false
|
|
touches_proto: false
|
|
touches_migrations: false
|
|
consumer_surface: false
|
|
docs_only: false
|
|
add_code_review: false
|
|
add_adversary: false
|
|
add_security: false
|
|
add_probe: false
|
|
lane_reasons: ''
|
|
parse_failure: ''
|
|
select_lanes_failure: ''
|
|
lanes_degraded: ''
|
|
code_review_items: []
|
|
adversary_items: []
|
|
security_items: []
|
|
probe_items: []
|
|
code_review_results: []
|
|
adversary_results: []
|
|
security_results: []
|
|
probe_results: []
|
|
lanes_summary: ''
|
|
gauntlet_verdict: ''
|
|
gauntlet_report: ''
|
|
|
|
start: parse
|
|
|
|
nodes:
|
|
parse:
|
|
id: parse
|
|
type: llm
|
|
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging, no lane selection.
|
|
instructions: |
|
|
You extract the facts of a review request into structured fields.
|
|
EXTRACTION ONLY — do not decide which review lanes should run.
|
|
|
|
- `diff_spec`: the git diff specification the caller named — a ref
|
|
range ("main...HEAD", "HEAD~3..HEAD"), a single ref, or the literal
|
|
"worktree" for uncommitted changes against HEAD. When the caller
|
|
did not name one, use "worktree".
|
|
- `project_dir_in`: the repository directory the caller named as holding
|
|
the change (absolute path preferred); empty string when the caller
|
|
named none.
|
|
- `plan_context`: the plan / spec / acceptance-criteria text the
|
|
implementation must satisfy, verbatim (or a file path the caller
|
|
pointed at). Empty string when the caller supplied none.
|
|
- `rigor`: poc | prototype | production. Default "production" when
|
|
not stated.
|
|
- `security_posture`: prototype | standard | hardened. Default
|
|
"standard" when not stated.
|
|
- `probe_context`: the local-run recipe and/or existing usage-suite
|
|
pointers for black-box probing (how to boot the service, where the
|
|
suites live). Empty string when the caller supplied none.
|
|
- `forced_lanes`: lanes the caller explicitly demanded (verbatim
|
|
names). Empty array = automatic selection.
|
|
- `extra_context`: any remaining caller context reviewers should see,
|
|
condensed.
|
|
prompt: |
|
|
Extract the review-request facts from this request:
|
|
|
|
{{initial_prompt}}
|
|
tools: []
|
|
output_schema:
|
|
type: object
|
|
properties:
|
|
diff_spec:
|
|
type: string
|
|
plan_context:
|
|
type: string
|
|
rigor:
|
|
type: string
|
|
security_posture:
|
|
type: string
|
|
probe_context:
|
|
type: string
|
|
forced_lanes:
|
|
type: array
|
|
items:
|
|
type: string
|
|
extra_context:
|
|
type: string
|
|
project_dir_in:
|
|
type: string
|
|
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in]
|
|
max_iterations: 3
|
|
max_attempts: 2
|
|
state_updates:
|
|
parse_failure: '{{output}}'
|
|
fallback: parse_fault
|
|
next: signals
|
|
|
|
parse_fault:
|
|
id: parse_fault
|
|
type: script
|
|
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — records a PIPELINE-FAULT in signals_error and jumps straight to the verdict gate, which blocks on it. Lane item lists stay their initial [] so every lane records SKIPPED.
|
|
script: scripts/parse_fault.py
|
|
timeout: 30
|
|
next: verdict_gate
|
|
|
|
signals:
|
|
id: signals
|
|
type: script
|
|
description: Deterministic diff signals — file list, attack-surface and consumer-surface indicators. Pure git; no LLM judgment.
|
|
script: scripts/signals.py
|
|
timeout: 120
|
|
next: select_lanes
|
|
|
|
select_lanes:
|
|
id: select_lanes
|
|
type: llm
|
|
description: Additive-only lane judgment on top of the deterministic rules. It can WIDEN the gauntlet, never narrow it.
|
|
instructions: |
|
|
You look at a change's deterministic diff signals and decide whether any
|
|
review lane the hard rules might have missed should ALSO run. You can
|
|
only ADD lanes — the deterministic floor (enforced downstream, not by
|
|
you) is: code-review always runs; adversary runs when a plan/spec is
|
|
present; security runs on attack-surface signals or hardened posture;
|
|
probe runs when consumer-facing surface changed and a local-run recipe
|
|
exists. Setting a flag to false NEVER removes a lane the rules selected.
|
|
|
|
Reasons to add:
|
|
- `add_security`: the diff handles external/untrusted input, touches
|
|
crypto/secrets handling, or changes trust boundaries in ways the
|
|
path-based signals missed.
|
|
- `add_adversary`: the request references behavior it must preserve or
|
|
criteria scattered in prose that amount to a spec, even without a
|
|
formal plan.
|
|
- `add_probe`: the change alters observable API/CLI behavior even
|
|
though the path signals missed it (only useful when a local-run
|
|
recipe exists — the builder skips it otherwise, noting why).
|
|
- `add_code_review`: always selected by the rules; flag exists only
|
|
for forced-lane configurations.
|
|
|
|
Put one line per added lane into `lane_reasons`. When nothing needs
|
|
adding, all flags false and empty reasons is the correct answer.
|
|
prompt: |
|
|
Change signals (deterministic):
|
|
- diff spec: {{diff_spec}}
|
|
- files changed: {{file_count}}
|
|
- changed files (first 200): {{changed_files}}
|
|
- touches auth paths/content: {{touches_auth}}
|
|
- touches dependency manifests: {{touches_deps}}
|
|
- adds exec/shell/subprocess calls: {{touches_exec}}
|
|
- touches protobuf/contracts: {{touches_proto}}
|
|
- touches migrations: {{touches_migrations}}
|
|
- consumer-facing surface paths: {{consumer_surface}}
|
|
- docs-only diff: {{docs_only}}
|
|
- {{signals_summary}}
|
|
{{signals_error}}
|
|
|
|
Caller context:
|
|
- rigor: {{rigor}}; security posture: {{security_posture}}
|
|
- plan/spec present: {{plan_context}}
|
|
- probe recipe present: {{probe_context}}
|
|
- extra: {{extra_context}}
|
|
tools: []
|
|
output_schema:
|
|
type: object
|
|
properties:
|
|
add_code_review:
|
|
type: boolean
|
|
add_adversary:
|
|
type: boolean
|
|
add_security:
|
|
type: boolean
|
|
add_probe:
|
|
type: boolean
|
|
lane_reasons:
|
|
type: string
|
|
required: [add_code_review, add_adversary, add_security, add_probe, lane_reasons]
|
|
max_iterations: 3
|
|
max_attempts: 2
|
|
state_updates:
|
|
select_lanes_failure: '{{output}}'
|
|
fallback: default_lanes
|
|
next: build_items
|
|
|
|
default_lanes:
|
|
id: default_lanes
|
|
type: script
|
|
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
|
|
script: scripts/default_lanes.py
|
|
timeout: 30
|
|
next: build_items
|
|
|
|
build_items:
|
|
id: build_items
|
|
type: script
|
|
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map.
|
|
script: scripts/build_items.py
|
|
timeout: 30
|
|
next: [map_code_review, map_adversary, map_security, map_probe]
|
|
|
|
map_code_review:
|
|
id: map_code_review
|
|
type: map
|
|
over: '{{code_review_items}}'
|
|
as: lane_ctx
|
|
branch: run_code_review
|
|
output_key: lane_out
|
|
collect_into: code_review_results
|
|
max_concurrency: 1
|
|
next: verdict_gate
|
|
|
|
map_adversary:
|
|
id: map_adversary
|
|
type: map
|
|
over: '{{adversary_items}}'
|
|
as: lane_ctx
|
|
branch: run_adversary
|
|
output_key: lane_out
|
|
collect_into: adversary_results
|
|
max_concurrency: 1
|
|
next: verdict_gate
|
|
|
|
map_security:
|
|
id: map_security
|
|
type: map
|
|
over: '{{security_items}}'
|
|
as: lane_ctx
|
|
branch: run_security
|
|
output_key: lane_out
|
|
collect_into: security_results
|
|
max_concurrency: 1
|
|
next: verdict_gate
|
|
|
|
map_probe:
|
|
id: map_probe
|
|
type: map
|
|
over: '{{probe_items}}'
|
|
as: lane_ctx
|
|
branch: run_probe
|
|
output_key: lane_out
|
|
collect_into: probe_results
|
|
max_concurrency: 1
|
|
next: verdict_gate
|
|
|
|
run_code_review:
|
|
id: run_code_review
|
|
type: agent
|
|
description: Independent code-quality review lane.
|
|
agent: code-reviewer
|
|
timeout: 7200
|
|
prompt: |
|
|
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
|
|
independent post-implementation review.
|
|
|
|
Quality bar: rigor={{rigor}}.
|
|
|
|
Plan / acceptance-criteria context (may be empty):
|
|
{{plan_context}}
|
|
|
|
{{extra_context}}
|
|
|
|
Close your report with your Review Verdict line.
|
|
max_attempts: 2
|
|
state_updates:
|
|
lane_out: '{{output}}'
|
|
fallback: lane_fault
|
|
|
|
run_adversary:
|
|
id: run_adversary
|
|
type: agent
|
|
description: Plan-conformance review lane.
|
|
agent: adversary
|
|
timeout: 7200
|
|
inputs:
|
|
verification_commands: '{{verification_commands}}'
|
|
prompt: |
|
|
Adversarial plan-conformance review in {{project_dir}}.
|
|
|
|
The implementation to check: diff {{diff_spec}}.
|
|
|
|
The spec / plan / acceptance criteria it must satisfy:
|
|
{{plan_context}}
|
|
|
|
Verification commands arrive as a structured input (the
|
|
verification_commands variable), not in this prompt.
|
|
|
|
{{extra_context}}
|
|
|
|
Return your ADVERSARIAL_REVIEW verdict.
|
|
max_attempts: 2
|
|
state_updates:
|
|
lane_out: '{{output}}'
|
|
fallback: lane_fault
|
|
|
|
run_security:
|
|
id: run_security
|
|
type: agent
|
|
description: Security review lane.
|
|
agent: security-reviewer
|
|
timeout: 7200
|
|
prompt: |
|
|
Security review of the code changes in {{project_dir}} (diff: {{diff_spec}}).
|
|
|
|
Security posture: {{security_posture}}.
|
|
|
|
{{extra_context}}
|
|
|
|
Return your SECURITY_REVIEW verdict.
|
|
max_attempts: 2
|
|
state_updates:
|
|
lane_out: '{{output}}'
|
|
fallback: lane_fault
|
|
|
|
run_probe:
|
|
id: run_probe
|
|
type: agent
|
|
description: Black-box usage-pattern verification lane.
|
|
agent: probe
|
|
timeout: 7200
|
|
prompt: |
|
|
Usage-pattern verification (black-box probe) of the changes in
|
|
{{project_dir}} (diff: {{diff_spec}}).
|
|
|
|
Local-run recipe / existing usage suites:
|
|
{{probe_context}}
|
|
|
|
Plan / acceptance-criteria context:
|
|
{{plan_context}}
|
|
|
|
The workspace may contain test files from an aborted earlier attempt of
|
|
this lane; reconcile rather than duplicate.
|
|
|
|
Derive expected behaviors from the plan/contract BEFORE reading
|
|
implementation code, run existing suites first, then author spec-first
|
|
tests for the uncovered delta. Return your USAGE_PROBE verdict.
|
|
max_attempts: 2
|
|
state_updates:
|
|
lane_out: '{{output}}'
|
|
fallback: lane_fault
|
|
|
|
lane_fault:
|
|
id: lane_fault
|
|
type: script
|
|
description: Fail-closed lane fault marker (reached only via the run_* lanes' fallback routes; shared across the four map branches). Normalizes the engine's "Agent node failed" text in lane_out into a PIPELINE-FAULT lane result the map collects; verdict_gate turns it into BLOCKED — a degraded lane is never a pass.
|
|
script: scripts/lane_fault.py
|
|
timeout: 30
|
|
|
|
verdict_gate:
|
|
id: verdict_gate
|
|
type: script
|
|
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; 🔴 findings block regardless of the lane's own verdict line.
|
|
script: scripts/verdict_gate.py
|
|
timeout: 60
|
|
next: done
|
|
|
|
done:
|
|
id: done
|
|
type: end
|
|
output: |
|
|
GAUNTLET: {{gauntlet_verdict}}
|
|
|
|
{{gauntlet_report}}
|