402 lines
18 KiB
YAML
402 lines
18 KiB
YAML
name: adversary
|
|
description: |
|
|
Graph-based adversarial plan-conformance reviewer. The plan's acceptance
|
|
criteria are extracted into a list and the graph mechanically fans out ONE
|
|
fresh-context verification branch per criterion — "checked 8 of 12 and waved
|
|
the rest through" becomes structurally impossible. Each criterion verdict
|
|
(MET / PARTIAL / UNMET / DIVERGED) passes a deterministic contract gate
|
|
(MET requires evidence including a test proving the behavior — no test means
|
|
at best PARTIAL). A holistic pass then hunts what per-criterion checks cannot
|
|
see: absence (the dog that didn't bark), scope drift, interface substitution,
|
|
silently-overridden recorded decisions, out-of-scope touches, and gamed
|
|
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
|
|
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
|
|
requires every criterion MET, zero holistic complaints, zero pipeline faults,
|
|
and zero red verification runs (exit≠0/timeout/skipped, or a runner error
|
|
when commands were declared). Missing plan = DIVERGES (fail-closed) naming
|
|
the missing spec. Caller-declared verification
|
|
commands are executed ONCE by a deterministic stage and recorded — criterion
|
|
branches cite the record and never run anything themselves. Any pipeline
|
|
fault (parse, diff resolution, verification runner, or holistic failure)
|
|
lands a PIPELINE-FAULT marker and fails closed into DIVERGES: degraded is
|
|
never passing.
|
|
|
|
version: '1.0'
|
|
|
|
global_tools:
|
|
- ast_grep.sh
|
|
- fs_read.sh
|
|
- fs_cat.sh
|
|
- fs_grep.sh
|
|
|
|
variables:
|
|
- name: project_dir
|
|
description: Project directory containing the changes under review
|
|
default: '.'
|
|
- name: verification_commands
|
|
description: |
|
|
JSON array of shell commands the CALLER declares for the deterministic
|
|
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
|
prompt; empty array = none declared. Must not boot shared stacks or
|
|
services (compose projects, fixed ports) — those collide with the probe
|
|
lane running in parallel.
|
|
default: '[]'
|
|
|
|
settings:
|
|
max_loop_iterations: 10
|
|
log_state_snapshots: false
|
|
# 3600 (run_checks node timeout, worst case) + 5400 (the previous
|
|
# whole-graph budget, kept as the envelope for every stage other than
|
|
# run_checks: parse/facts before it, criteria map/holistic/verdict after)
|
|
# = 9000. A graph timeout kills from outside — no fallback fires and the
|
|
# sentinel is lost.
|
|
timeout: 9000
|
|
|
|
initial_state:
|
|
project_dir_in: ''
|
|
parse_retries: 0
|
|
criteria: []
|
|
diff_spec: auto
|
|
plan_notes: ''
|
|
scope_notes: ''
|
|
extra_context: ''
|
|
changed_files: []
|
|
diff_text: ''
|
|
diff_note: ''
|
|
exec_results: ''
|
|
crit_verdicts: []
|
|
extra_complaints: []
|
|
observations: ''
|
|
adv_report: ''
|
|
gate_feedback: ''
|
|
gate_attempts: 0
|
|
parse_failure: ''
|
|
holistic_failure: ''
|
|
pipeline_faults: []
|
|
|
|
start: parse
|
|
|
|
nodes:
|
|
parse:
|
|
id: parse
|
|
type: llm
|
|
description: Extract the acceptance criteria and plan facts from the spawn prompt. Extraction only — no judging, no criteria invented.
|
|
instructions: |
|
|
You extract a plan-conformance review request into structured fields.
|
|
EXTRACTION ONLY — never invent criteria, never guess intent.
|
|
|
|
- `criteria`: every acceptance criterion from the plan/task, one entry each,
|
|
with a short id (c1, c2, …) and SELF-CONTAINED text (what observable
|
|
behavior is promised, including any test expectations the plan states).
|
|
Split compound criteria ("A and B") into separate entries when they are
|
|
independently verifiable. When the caller supplied NO plan/criteria at
|
|
all, emit an empty array — downstream fails closed.
|
|
An empty array is ONLY correct when the prompt genuinely contains no
|
|
plan/criteria section — when the prompt carries one (a "Plan /
|
|
acceptance criteria:" or "PLAN —" block), returning zero criteria is
|
|
an extraction FAILURE, never a valid result.
|
|
The caller's explicit enumeration is AUTHORITATIVE: when the caller
|
|
declares "these N are the (only) criteria" or supplies a numbered
|
|
criteria list, extract EXACTLY those N — never add entries they did
|
|
not enumerate. NEVER mine criteria from sections the caller labels as
|
|
context ("FACTS", "Quality bar", "context, not criteria", background
|
|
material) — route those into plan_notes / scope_notes / extra_context
|
|
instead. Splitting applies only WITHIN a declared criterion; when in
|
|
doubt, FEWER criteria, verbatim.
|
|
- `diff_spec`: the git diff basis the caller named — a range, a ref/SHA,
|
|
"staged", "worktree", or "auto" when unnamed (a caller phrase like
|
|
"get_diff --base X" means diff_spec = "X").
|
|
- `project_dir_in`: the directory/worktree path the caller named as
|
|
holding the code under review; empty string when the caller named none.
|
|
- `plan_notes`: recorded decisions the plan states ("do X not Y because Z"),
|
|
verbatim — the holistic pass checks the diff didn't silently override them.
|
|
- `scope_notes`: what the plan says NOT to touch / explicit boundaries.
|
|
- `extra_context`: anything else conformance-relevant, condensed.
|
|
prompt: |
|
|
Extract the conformance-review facts from this request:
|
|
|
|
{{initial_prompt}}
|
|
tools: []
|
|
output_schema:
|
|
type: object
|
|
properties:
|
|
criteria:
|
|
type: array
|
|
items:
|
|
type: object
|
|
properties:
|
|
id:
|
|
type: string
|
|
text:
|
|
type: string
|
|
required: [id, text]
|
|
diff_spec:
|
|
type: string
|
|
plan_notes:
|
|
type: string
|
|
scope_notes:
|
|
type: string
|
|
extra_context:
|
|
type: string
|
|
project_dir_in:
|
|
type: string
|
|
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in]
|
|
max_iterations: 3
|
|
max_attempts: 2
|
|
state_updates:
|
|
parse_failure: '{{output}}'
|
|
fallback: pipeline_fault
|
|
next: facts
|
|
|
|
facts:
|
|
id: facts
|
|
type: script
|
|
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
|
|
script: scripts/diff_facts.py
|
|
timeout: 120
|
|
fallback: pipeline_fault
|
|
next: run_checks
|
|
|
|
run_checks:
|
|
id: run_checks
|
|
type: script
|
|
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker (criterion branches judge it PARTIAL/unproven; the verdict additionally blocks CONFORMS when commands were declared); a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
|
|
script: scripts/run_checks.py
|
|
timeout: 3600
|
|
fallback: pipeline_fault
|
|
next: map_criteria
|
|
|
|
map_criteria:
|
|
id: map_criteria
|
|
type: map
|
|
over: '{{criteria}}'
|
|
as: criterion
|
|
branch: check_criterion
|
|
output_key: crit_verdict
|
|
collect_into: crit_verdicts
|
|
max_concurrency: 6
|
|
next: holistic
|
|
|
|
check_criterion:
|
|
id: check_criterion
|
|
type: llm
|
|
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree (ENVIRONMENT/SKIPPED runs judge PARTIAL here; the deterministic verdict blocks CONFORMS when declared commands did not run).
|
|
instructions: |
|
|
You verify ONE acceptance criterion against a code change. Fresh context,
|
|
one criterion, full suspicion: assume the implementer drifted, cut a
|
|
corner, or misread the plan until the diff proves otherwise.
|
|
|
|
Classification (exactly one):
|
|
- `MET` — the diff satisfies the criterion AND a test proves the promised
|
|
behavior. Evidence MUST cite both: the satisfying change (file:line) and
|
|
the test targeting the behavior (file:line). No test ⇒ NOT MET — at best
|
|
PARTIAL (a criterion is a promise of observable behavior; unproven
|
|
behavior is unverified behavior).
|
|
- `PARTIAL` — partially satisfied, or satisfied but unproven (missing/weak
|
|
test), or you could not establish it either way (say why — doubt lands
|
|
here, never in MET).
|
|
- `UNMET` — the plan requires it and the diff does not do it. Absence is a
|
|
finding: review what SHOULD be there, not only what IS.
|
|
- `DIVERGED` — the diff does something ELSE instead (interface substituted,
|
|
approach swapped, criterion reinterpreted).
|
|
|
|
Ground-truth discipline: fs_grep the symbols the criterion names (confirm
|
|
they exist spelled as specified), fs_read around the relevant hunks to
|
|
confirm the change makes the criterion true, confirm the test targets
|
|
BEHAVIOR (not implementation echo), use ast_grep for structural claims.
|
|
Watch for gamed evidence: a test weakened/deleted so it passes, hardcoded
|
|
values shaped to the criterion's example, TODO/stub presented as done —
|
|
those are PARTIAL or DIVERGED with the gaming named in the complaint.
|
|
|
|
Retry / redelivery / idempotency demands: before you mark a criterion PARTIAL
|
|
or UNMET for lacking a retry, re-run, or "second attempt emits nothing" spec,
|
|
classify the handler. STATEFUL — it carries something across attempts (emits
|
|
events, calls an external system, performs an unconditional write, keeps an
|
|
in-memory or ledger record) — a retry spec is a legitimate demand. STATELESS —
|
|
each attempt is a pure function of the request plus a conditional store
|
|
predicate (`WHERE status = 'pending'`, `ended_at IS NULL`, `INSERT … ON
|
|
CONFLICT DO NOTHING`) and nothing is emitted or ledgered outside that
|
|
predicate — then "retry produces no event" is a tautology of the mechanism,
|
|
not a missing behavior: demand at most one test of the predicate itself,
|
|
never a retry spec. Name the classification and its file:line evidence in
|
|
the complaint either way.
|
|
|
|
Execution criteria ("tests pass", "lint clean", "build succeeds") are
|
|
judged ONLY against the recorded verification runs in your prompt — you
|
|
have no execution tools BY DESIGN and must never claim to have run
|
|
anything yourself:
|
|
- a recorded green run (exit 0) of a command covering the criterion is
|
|
the proving evidence — MET, citing that run (the command + exit 0).
|
|
- a recorded failing run covering the criterion ⇒ UNMET, citing the
|
|
recorded exit code and output tail.
|
|
- "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
|
|
ENVIRONMENT note in the complaint naming the exact command the caller
|
|
should declare via the `verification_commands` variable (structured
|
|
input, e.g. "cargo test --all") — commands are never declared in the
|
|
prompt.
|
|
- a SKIPPED record (the runner's total deadline expired before that
|
|
command ran) ⇒ at best PARTIAL for the criteria it covers, same
|
|
treatment as an ENVIRONMENT marker: unproven at the criterion level.
|
|
The deterministic verdict additionally blocks CONFORMS when declared
|
|
commands did not run — that is its job, not yours.
|
|
|
|
Output format — respond with ONLY this JSON object, no prose around it:
|
|
{"id": "<the criterion's id, echoed>",
|
|
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
|
|
"evidence": "<for MET: the satisfying change file:line AND the proving test
|
|
file:line, quoted decisively; otherwise the closest relevant citations or empty>",
|
|
"complaint": "<required for non-MET: what the diff does/omits (file:line) and
|
|
what would make it conform; empty string for MET>"}
|
|
|
|
Your response is machine-validated: a malformed response is rejected back
|
|
to you exactly once with the reason, then recorded as PARTIAL (unproven).
|
|
The id is stamped from the criterion itself — your echo is informative only.
|
|
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
|
|
whole-file reads, and conclude as soon as the evidence is decisive.
|
|
Never begin `evidence` with the literal `PIPELINE-FAULT:` — that prefix is
|
|
reserved for the graph's own fault markers.
|
|
prompt: |
|
|
Repository: {{project_dir}}
|
|
Changed files: {{changed_files}}
|
|
|
|
Criterion to verify (JSON):
|
|
{{criterion}}
|
|
|
|
Plan-recorded decisions (context): {{plan_notes}}
|
|
{{extra_context}}
|
|
|
|
The diff{{diff_note}}:
|
|
{{diff_text}}
|
|
|
|
Verification runs (recorded by the deterministic run_checks stage):
|
|
{{exec_results}}
|
|
{{gate_feedback}}
|
|
tools:
|
|
- fs_read
|
|
- fs_cat
|
|
- fs_grep
|
|
- ast_grep
|
|
max_iterations: 30
|
|
max_attempts: 2
|
|
state_updates:
|
|
crit_verdict: '{{output}}'
|
|
fallback: criterion_fault
|
|
next: crit_gate
|
|
|
|
crit_gate:
|
|
id: crit_gate
|
|
type: script
|
|
description: Deterministic contract gate per criterion verdict — status enum, MET requires evidence, non-MET requires a complaint; reject-and-retry once, then record PARTIAL (unproven). Never sinks the map.
|
|
script: scripts/crit_gate.py
|
|
timeout: 30
|
|
|
|
criterion_fault:
|
|
id: criterion_fault
|
|
type: script
|
|
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met.
|
|
script: scripts/criterion_fault.py
|
|
timeout: 30
|
|
|
|
holistic:
|
|
id: holistic
|
|
type: llm
|
|
description: Cross-criterion adversarial pass — hunts what per-criterion checks cannot see. ADDITIVE-only; cannot flip criterion verdicts.
|
|
instructions: |
|
|
You run the holistic adversarial pass over a whole change AFTER each
|
|
acceptance criterion was independently verified. You hunt what
|
|
per-criterion checks cannot see. You are ADDITIVE-ONLY: you may add
|
|
complaints; you may NOT flip, soften, or re-judge any criterion verdict.
|
|
|
|
The checklist (each hit becomes one complaint with kind, file:line
|
|
location, the violation, and the fix):
|
|
- **Absence** — something the PLAN requires that maps to no criterion and
|
|
never showed up in the diff (the dog that didn't bark). A missing
|
|
retry/redelivery spec is an absence ONLY for a handler that carries state
|
|
or side effects across attempts; a handler that is a pure function of the
|
|
request plus a conditional store predicate owes none.
|
|
- **Scope drift / out-of-scope** — the diff touches what the plan's scope
|
|
notes say not to touch, or quietly does extra unplanned work.
|
|
- **Interface/approach substitution** — the plan named an interface,
|
|
library, or approach; the diff substituted another without the plan
|
|
recording that decision.
|
|
- **Silently overridden recorded decision** — plan says "do X not Y
|
|
because Z"; diff does Y.
|
|
- **Downstream contract breakage** — the diff changes a contract the plan
|
|
promised to preserve.
|
|
- **Gamed tests** — tests weakened, deleted, or shaped (hardcoded
|
|
expectations, skipped assertions) so the criteria APPEAR met.
|
|
- **The plan itself is the culprit** — impossible or self-contradictory
|
|
requirements: that is a complaint naming the PLAN as root cause; never
|
|
silently judge against a corrected version of it.
|
|
|
|
Be terse and decisive: three real complaints beat fifteen weak ones. A
|
|
complaint with no location and no criterion/scope reference is noise — do
|
|
not emit it. When everything is a nitpick, emit zero complaints and put
|
|
1-3 non-blocking notes in `observations`.
|
|
prompt: |
|
|
Repository: {{project_dir}}
|
|
Changed files: {{changed_files}}
|
|
|
|
Acceptance criteria: {{criteria}}
|
|
Per-criterion verdicts (already final): {{crit_verdicts}}
|
|
|
|
Plan-recorded decisions: {{plan_notes}}
|
|
Scope boundaries: {{scope_notes}}
|
|
{{extra_context}}
|
|
|
|
The diff{{diff_note}}:
|
|
{{diff_text}}
|
|
tools:
|
|
- fs_read
|
|
- fs_cat
|
|
- fs_grep
|
|
- ast_grep
|
|
output_schema:
|
|
type: object
|
|
properties:
|
|
extra_complaints:
|
|
type: array
|
|
items:
|
|
type: object
|
|
properties:
|
|
kind:
|
|
type: string
|
|
location:
|
|
type: string
|
|
violation:
|
|
type: string
|
|
fix:
|
|
type: string
|
|
required: [kind, location, violation, fix]
|
|
observations:
|
|
type: string
|
|
required: [extra_complaints, observations]
|
|
max_iterations: 45
|
|
max_attempts: 2
|
|
state_updates:
|
|
holistic_failure: '{{output}}'
|
|
fallback: verdict
|
|
next: verdict
|
|
|
|
pipeline_fault:
|
|
id: pipeline_fault
|
|
type: script
|
|
description: Fail-closed fault marker (reached only via fallback routes from parse/facts/run_checks) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
|
|
script: scripts/pipeline_fault.py
|
|
timeout: 30
|
|
next: verdict
|
|
|
|
verdict:
|
|
id: verdict
|
|
type: script
|
|
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, zero pipeline faults, and zero recorded red verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error marker when commands were declared); no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
|
|
script: scripts/verdict.py
|
|
timeout: 30
|
|
next: done
|
|
|
|
done:
|
|
id: done
|
|
type: end
|
|
output: |
|
|
{{adv_report}}
|