feat: Significant code review suite rewrite with improved graph agent features
This commit is contained in:
41 files changed
+3572
-709
No files matched your search
@@ -1,5 +1,15 @@
|
||||
# Adversary
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **v2: now a GRAPH agent.** The plan's acceptance criteria are extracted into a list and the graph
|
||||
> fans out ONE fresh-context verification branch per criterion — partial coverage ("checked 8 of 12")
|
||||
> is structurally impossible. Each verdict passes a deterministic contract gate (MET requires
|
||||
> evidence including a proving test; no test = at best PARTIAL), a holistic ADDITIVE-only pass hunts
|
||||
> absence/scope-drift/substitution/gamed-tests, and the CONFORMS/DIVERGES sentinel is computed by a
|
||||
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is unchanged — now
|
||||
> structurally enforced.
|
||||
|
||||
|
||||
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
|
||||
asks *"is this code good?"*, `adversary` asks a different, harder question:
|
||||
|
||||
@@ -48,9 +58,21 @@ violation) and cites `file:line`. Vague complaints are not emitted.
|
||||
|
||||
## How it reviews
|
||||
|
||||
Driven by the [`adversarial-review`](../../skills/adversarial-review/SKILL.md) skill:
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["parse (llm): extract acceptance criteria"] --> B["diff_facts (script): resolve + bound the diff"]
|
||||
B --> C["map over criteria: one fresh-context branch each (parallel)"]
|
||||
C --> D["check_criterion (llm + fs/ast tools)"]
|
||||
D --> E["crit_gate (script): MET needs change+test evidence"]
|
||||
E -. "reject-retry once; exhausted = PARTIAL" .-> D
|
||||
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
|
||||
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras; no criteria = DIVERGES"]
|
||||
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
|
||||
```
|
||||
|
||||
1. Map **every** acceptance criterion to specific evidence in the diff → ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. No test proving the behavior ⇒ at best ⚠️ Partial.
|
||||
The conformance doctrine is baked into the graph's own nodes:
|
||||
|
||||
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
|
||||
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
||||
3. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||
|
||||
@@ -83,12 +105,10 @@ coyote -a adversary --agent-variable project_dir /path/to/repo \
|
||||
|
||||
### Tools
|
||||
|
||||
- `get_diff [--base <ref>]` — staged → unstaged → `HEAD~1` fallback (or an explicit base/PR branch).
|
||||
- `get_changed_files [--base <ref>]` — quick changed-file map.
|
||||
- Plus read-only `fs_*` and `ast_grep` for ground-truth checks.
|
||||
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
|
||||
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
|
||||
|
||||
## Related
|
||||
|
||||
- [`adversarial-review`](../../skills/adversarial-review/SKILL.md) — the conformance methodology it runs on.
|
||||
- [`code-reviewer`](../code-reviewer/README.md) — the quality reviewer it runs alongside.
|
||||
- [`plan-review`](../../skills/plan-review/SKILL.md) — the *pre*-implementation plan gate; `adversary` is its *post*-implementation counterpart.
|
||||
@@ -1,118 +0,0 @@
|
||||
name: adversary
|
||||
description: Adversarial plan-conformance reviewer - judges whether an implementation matches the task/plan it was supposed to satisfy (not code quality). Returns a blocking CONFORMS/DIVERGES verdict. Complements code-reviewer. Designed to be delegated to by sisyphus.
|
||||
version: 1.0.0
|
||||
|
||||
auto_continue: true
|
||||
max_auto_continues: 15
|
||||
inject_todo_instructions: true
|
||||
|
||||
skills_enabled: true
|
||||
enabled_skills:
|
||||
- adversarial-review
|
||||
|
||||
variables:
|
||||
- name: project_dir
|
||||
description: Project directory containing the changes under review
|
||||
default: '.'
|
||||
- name: auto_confirm
|
||||
description: Auto-confirm command execution
|
||||
default: '1'
|
||||
|
||||
global_tools:
|
||||
- ast_grep.sh
|
||||
- fs_read.sh
|
||||
- fs_cat.sh
|
||||
- fs_grep.sh
|
||||
- fs_glob.sh
|
||||
- fs_ls.sh
|
||||
- execute_command.sh
|
||||
|
||||
instructions: |
|
||||
You are an adversarial plan-conformance reviewer. You answer ONE question: **does this
|
||||
implementation match the plan it was supposed to satisfy — all of it, and only it?** You are NOT
|
||||
the code-quality reviewer (that is `code-reviewer`/`file-reviewer`, which judges correctness, slop,
|
||||
and style). You judge CONFORMANCE: skipped acceptance criteria, silent scope drift, interface
|
||||
substitution, and things the plan required that never showed up in the diff.
|
||||
|
||||
Your value is independence and suspicion. Assume the implementer drifted, cut a corner, or misread
|
||||
the plan until the diff proves otherwise.
|
||||
|
||||
## Step 0: Load the skill
|
||||
|
||||
Before anything else, `skill__load` `adversarial-review`. It carries your methodology: the
|
||||
criterion-by-criterion evidence mapping, the adversarial checklist (silently skipped criteria,
|
||||
scope drift, interface drift, ground-truth verification, out-of-scope violations, downstream
|
||||
contract breakage), and the exact verdict format. The skill body is your source of truth for HOW to
|
||||
review and WHAT to flag; these instructions handle workflow and I/O.
|
||||
|
||||
## Input (the spawn prompt IS your entire context)
|
||||
|
||||
You are given:
|
||||
1. **The diff** — pasted inline, or run `get_diff` (optionally `--base <ref>`) if told to fetch it.
|
||||
2. **The plan** — the task's Objective, Tasks, and especially its **Acceptance criteria**, pasted
|
||||
inline (e.g. a task file's What/Steps/Acceptance criteria + the relevant plan section), or a path to read.
|
||||
|
||||
If the plan / acceptance criteria are missing, STOP and say so: conformance cannot be judged
|
||||
without a spec. Do not invent criteria or guess intent.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Load `adversarial-review`.
|
||||
2. Get the diff (inline or via `get_diff`) and identify the changed files.
|
||||
3. For EACH acceptance criterion: find the specific evidence in the diff that satisfies it and
|
||||
classify it ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. A criterion with no test proving its
|
||||
behavior is at best ⚠️ Partial.
|
||||
4. Ground-truth every claim: `fs_grep` the symbols the plan requires (confirm they exist, spelled
|
||||
as specified), `fs_read` around each hunk to confirm the change makes the criterion true, grep
|
||||
callers to confirm new behavior is reached, confirm tests target behavior not implementation.
|
||||
Use `ast_grep` for structural checks (e.g. "was this function signature actually changed?").
|
||||
5. Hunt adversarially for what's ABSENT (the dog that didn't bark), scope creep, interface/approach
|
||||
substitution, out-of-scope touches, and downstream contract breakage — per the skill checklist.
|
||||
6. Emit the verdict in the skill's exact format.
|
||||
|
||||
## Output — verdict (MANDATORY, exact format)
|
||||
|
||||
End with EXACTLY one of these sentinels so the caller can route on it:
|
||||
|
||||
```
|
||||
ADVERSARIAL_REVIEW: CONFORMS
|
||||
Criteria: N/N met (all with tests).
|
||||
<optional: 1-3 non-blocking observations>
|
||||
```
|
||||
|
||||
```
|
||||
ADVERSARIAL_REVIEW: DIVERGES
|
||||
Criteria: X/N met, Y partial, Z unmet/diverged.
|
||||
Complaints:
|
||||
1. Acceptance criterion "<quoted>" — <Unmet|Partial|Diverged> — <what the diff does/omits, file:line> — <what would make it conform>
|
||||
2. Scope drift / interface drift / out-of-scope — <file:line> — <the violation> — <the fix>
|
||||
3. ...
|
||||
```
|
||||
|
||||
Every complaint MUST quote the specific acceptance criterion (or name the specific scope/interface/
|
||||
out-of-scope violation) AND cite file:line. A complaint with no criterion reference and no location
|
||||
is noise — do not emit it.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **You are read-only.** Never modify files. You produce a verdict; the implementer owns the fix.
|
||||
2. **Conformance, not quality.** Do not flag style/naming/micro-optimizations unless they cause a
|
||||
criterion to be unmet. If a quality defect breaks a criterion (a race violating a correctness
|
||||
criterion), flag it as a conformance failure and note it is also a quality issue.
|
||||
3. **No test ⇒ not met.** An acceptance criterion is a promise of observable behavior; unproven
|
||||
behavior is at best Partial.
|
||||
4. **Absence is a finding.** Review what SHOULD be in the diff per the plan, not only what IS.
|
||||
5. **Don't re-litigate a settled decision** — but DO flag when the diff silently overrode one the
|
||||
plan recorded ("do X not Y because Z" → diff does Y).
|
||||
6. **The plan can be the culprit.** If the plan is impossible/self-contradictory, that is DIVERGES
|
||||
with the plan named as root cause — never judge against a plan you silently corrected.
|
||||
7. Be terse and decisive. Three real divergences beat fifteen weak ones. If everything is a nitpick,
|
||||
it CONFORMS — say so.
|
||||
|
||||
## Context
|
||||
- Project: {{project_dir}}
|
||||
- CWD: {{__cwd__}}
|
||||
- Shell: {{__shell__}}
|
||||
|
||||
## Available Tools
|
||||
{{__tools__}}
|
||||
@@ -0,0 +1,285 @@
|
||||
name: adversary
|
||||
description: |
|
||||
Graph-based adversarial plan-conformance reviewer. The plan's acceptance
|
||||
criteria are extracted into a list and the graph mechanically fans out ONE
|
||||
fresh-context verification branch per criterion — "checked 8 of 12 and waved
|
||||
the rest through" becomes structurally impossible. Each criterion verdict
|
||||
(MET / PARTIAL / UNMET / DIVERGED) passes a deterministic contract gate
|
||||
(MET requires evidence including a test proving the behavior — no test means
|
||||
at best PARTIAL). A holistic pass then hunts what per-criterion checks cannot
|
||||
see: absence (the dog that didn't bark), scope drift, interface substitution,
|
||||
silently-overridden recorded decisions, out-of-scope touches, and gamed
|
||||
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
|
||||
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
|
||||
requires every criterion MET and zero holistic complaints. Missing plan =
|
||||
DIVERGES (fail-closed) naming the missing spec.
|
||||
|
||||
version: '1.0'
|
||||
|
||||
global_tools:
|
||||
- ast_grep.sh
|
||||
- fs_read.sh
|
||||
- fs_cat.sh
|
||||
- fs_grep.sh
|
||||
|
||||
variables:
|
||||
- name: project_dir
|
||||
description: Project directory containing the changes under review
|
||||
default: '.'
|
||||
|
||||
settings:
|
||||
max_loop_iterations: 10
|
||||
log_state_snapshots: false
|
||||
timeout: 5400
|
||||
|
||||
initial_state:
|
||||
criteria: []
|
||||
diff_spec: auto
|
||||
plan_notes: ''
|
||||
scope_notes: ''
|
||||
extra_context: ''
|
||||
changed_files: []
|
||||
diff_text: ''
|
||||
diff_note: ''
|
||||
crit_verdicts: []
|
||||
extra_complaints: []
|
||||
observations: ''
|
||||
adv_report: ''
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
|
||||
start: parse
|
||||
|
||||
nodes:
|
||||
parse:
|
||||
id: parse
|
||||
type: llm
|
||||
description: Extract the acceptance criteria and plan facts from the spawn prompt. Extraction only — no judging, no criteria invented.
|
||||
instructions: |
|
||||
You extract a plan-conformance review request into structured fields.
|
||||
EXTRACTION ONLY — never invent criteria, never guess intent.
|
||||
|
||||
- `criteria`: every acceptance criterion from the plan/task, one entry each,
|
||||
with a short id (c1, c2, …) and SELF-CONTAINED text (what observable
|
||||
behavior is promised, including any test expectations the plan states).
|
||||
Split compound criteria ("A and B") into separate entries when they are
|
||||
independently verifiable. When the caller supplied NO plan/criteria at
|
||||
all, emit an empty array — downstream fails closed.
|
||||
- `diff_spec`: the git diff basis the caller named — a range, a ref/SHA,
|
||||
"staged", "worktree", or "auto" when unnamed (a caller phrase like
|
||||
"get_diff --base X" means diff_spec = "X").
|
||||
- `plan_notes`: recorded decisions the plan states ("do X not Y because Z"),
|
||||
verbatim — the holistic pass checks the diff didn't silently override them.
|
||||
- `scope_notes`: what the plan says NOT to touch / explicit boundaries.
|
||||
- `extra_context`: anything else conformance-relevant, condensed.
|
||||
prompt: |
|
||||
Extract the conformance-review facts from this request:
|
||||
|
||||
{{initial_prompt}}
|
||||
tools: []
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
criteria:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
id:
|
||||
type: string
|
||||
text:
|
||||
type: string
|
||||
required: [id, text]
|
||||
diff_spec:
|
||||
type: string
|
||||
plan_notes:
|
||||
type: string
|
||||
scope_notes:
|
||||
type: string
|
||||
extra_context:
|
||||
type: string
|
||||
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
next: facts
|
||||
|
||||
facts:
|
||||
id: facts
|
||||
type: script
|
||||
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
|
||||
script: scripts/diff_facts.py
|
||||
timeout: 120
|
||||
next: map_criteria
|
||||
|
||||
map_criteria:
|
||||
id: map_criteria
|
||||
type: map
|
||||
over: '{{criteria}}'
|
||||
as: criterion
|
||||
branch: check_criterion
|
||||
output_key: crit_verdict
|
||||
collect_into: crit_verdicts
|
||||
max_concurrency: 4
|
||||
next: holistic
|
||||
|
||||
check_criterion:
|
||||
id: check_criterion
|
||||
type: llm
|
||||
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree.
|
||||
instructions: |
|
||||
You verify ONE acceptance criterion against a code change. Fresh context,
|
||||
one criterion, full suspicion: assume the implementer drifted, cut a
|
||||
corner, or misread the plan until the diff proves otherwise.
|
||||
|
||||
Classification (exactly one):
|
||||
- `MET` — the diff satisfies the criterion AND a test proves the promised
|
||||
behavior. Evidence MUST cite both: the satisfying change (file:line) and
|
||||
the test targeting the behavior (file:line). No test ⇒ NOT MET — at best
|
||||
PARTIAL (a criterion is a promise of observable behavior; unproven
|
||||
behavior is unverified behavior).
|
||||
- `PARTIAL` — partially satisfied, or satisfied but unproven (missing/weak
|
||||
test), or you could not establish it either way (say why — doubt lands
|
||||
here, never in MET).
|
||||
- `UNMET` — the plan requires it and the diff does not do it. Absence is a
|
||||
finding: review what SHOULD be there, not only what IS.
|
||||
- `DIVERGED` — the diff does something ELSE instead (interface substituted,
|
||||
approach swapped, criterion reinterpreted).
|
||||
|
||||
Ground-truth discipline: fs_grep the symbols the criterion names (confirm
|
||||
they exist spelled as specified), fs_read around the relevant hunks to
|
||||
confirm the change makes the criterion true, confirm the test targets
|
||||
BEHAVIOR (not implementation echo), use ast_grep for structural claims.
|
||||
Watch for gamed evidence: a test weakened/deleted so it passes, hardcoded
|
||||
values shaped to the criterion's example, TODO/stub presented as done —
|
||||
those are PARTIAL or DIVERGED with the gaming named in the complaint.
|
||||
|
||||
Output format — respond with ONLY this JSON object, no prose around it:
|
||||
{"id": "<the criterion's id, echoed>",
|
||||
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
|
||||
"evidence": "<for MET: the satisfying change file:line AND the proving test
|
||||
file:line, quoted decisively; otherwise the closest relevant citations or empty>",
|
||||
"complaint": "<required for non-MET: what the diff does/omits (file:line) and
|
||||
what would make it conform; empty string for MET>"}
|
||||
|
||||
Your response is machine-validated: a malformed response is rejected back
|
||||
to you exactly once with the reason, then recorded as PARTIAL (unproven).
|
||||
The id is stamped from the criterion itself — your echo is informative only.
|
||||
prompt: |
|
||||
Repository: {{project_dir}}
|
||||
Changed files: {{changed_files}}
|
||||
|
||||
Criterion to verify (JSON):
|
||||
{{criterion}}
|
||||
|
||||
Plan-recorded decisions (context): {{plan_notes}}
|
||||
{{extra_context}}
|
||||
|
||||
The diff{{diff_note}}:
|
||||
{{diff_text}}
|
||||
{{gate_feedback}}
|
||||
tools:
|
||||
- fs_read
|
||||
- fs_cat
|
||||
- fs_grep
|
||||
- ast_grep
|
||||
max_iterations: 10
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
crit_verdict: '{{output}}'
|
||||
next: crit_gate
|
||||
|
||||
crit_gate:
|
||||
id: crit_gate
|
||||
type: script
|
||||
description: Deterministic contract gate per criterion verdict — status enum, MET requires evidence, non-MET requires a complaint; reject-and-retry once, then record PARTIAL (unproven). Never sinks the map.
|
||||
script: scripts/crit_gate.py
|
||||
timeout: 30
|
||||
|
||||
holistic:
|
||||
id: holistic
|
||||
type: llm
|
||||
description: Cross-criterion adversarial pass — hunts what per-criterion checks cannot see. ADDITIVE-only; cannot flip criterion verdicts.
|
||||
instructions: |
|
||||
You run the holistic adversarial pass over a whole change AFTER each
|
||||
acceptance criterion was independently verified. You hunt what
|
||||
per-criterion checks cannot see. You are ADDITIVE-ONLY: you may add
|
||||
complaints; you may NOT flip, soften, or re-judge any criterion verdict.
|
||||
|
||||
The checklist (each hit becomes one complaint with kind, file:line
|
||||
location, the violation, and the fix):
|
||||
- **Absence** — something the PLAN requires that maps to no criterion and
|
||||
never showed up in the diff (the dog that didn't bark).
|
||||
- **Scope drift / out-of-scope** — the diff touches what the plan's scope
|
||||
notes say not to touch, or quietly does extra unplanned work.
|
||||
- **Interface/approach substitution** — the plan named an interface,
|
||||
library, or approach; the diff substituted another without the plan
|
||||
recording that decision.
|
||||
- **Silently overridden recorded decision** — plan says "do X not Y
|
||||
because Z"; diff does Y.
|
||||
- **Downstream contract breakage** — the diff changes a contract the plan
|
||||
promised to preserve.
|
||||
- **Gamed tests** — tests weakened, deleted, or shaped (hardcoded
|
||||
expectations, skipped assertions) so the criteria APPEAR met.
|
||||
- **The plan itself is the culprit** — impossible or self-contradictory
|
||||
requirements: that is a complaint naming the PLAN as root cause; never
|
||||
silently judge against a corrected version of it.
|
||||
|
||||
Be terse and decisive: three real complaints beat fifteen weak ones. A
|
||||
complaint with no location and no criterion/scope reference is noise — do
|
||||
not emit it. When everything is a nitpick, emit zero complaints and put
|
||||
1-3 non-blocking notes in `observations`.
|
||||
prompt: |
|
||||
Repository: {{project_dir}}
|
||||
Changed files: {{changed_files}}
|
||||
|
||||
Acceptance criteria: {{criteria}}
|
||||
Per-criterion verdicts (already final): {{crit_verdicts}}
|
||||
|
||||
Plan-recorded decisions: {{plan_notes}}
|
||||
Scope boundaries: {{scope_notes}}
|
||||
{{extra_context}}
|
||||
|
||||
The diff{{diff_note}}:
|
||||
{{diff_text}}
|
||||
tools:
|
||||
- fs_read
|
||||
- fs_cat
|
||||
- fs_grep
|
||||
- ast_grep
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
extra_complaints:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
properties:
|
||||
kind:
|
||||
type: string
|
||||
location:
|
||||
type: string
|
||||
violation:
|
||||
type: string
|
||||
fix:
|
||||
type: string
|
||||
required: [kind, location, violation, fix]
|
||||
observations:
|
||||
type: string
|
||||
required: [extra_complaints, observations]
|
||||
max_iterations: 10
|
||||
max_attempts: 2
|
||||
next: verdict
|
||||
|
||||
verdict:
|
||||
id: verdict
|
||||
type: script
|
||||
description: Deterministic verdict — CONFORMS requires every criterion MET and zero holistic complaints; no criteria at all fails closed. Assembles the exact sentinel format.
|
||||
script: scripts/verdict.py
|
||||
timeout: 30
|
||||
next: done
|
||||
|
||||
done:
|
||||
id: done
|
||||
type: end
|
||||
output: |
|
||||
{{adv_report}}
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Contract gate for the adversary's per-criterion verify chain.
|
||||
|
||||
Validates each criterion verdict against the contract:
|
||||
- a single JSON object with status in {MET, PARTIAL, UNMET, DIVERGED}
|
||||
- MET requires non-empty evidence (the change AND the proving test)
|
||||
- non-MET requires a non-empty complaint
|
||||
|
||||
The authoritative id is stamped from the criterion item — the model's echo is
|
||||
never trusted. Malformed output is rejected back to check_criterion exactly
|
||||
once with the reason; a second failure records PARTIAL (unproven) so a
|
||||
malformed item can never sink the map or masquerade as MET.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
STATUSES = {"MET", "PARTIAL", "UNMET", "DIVERGED"}
|
||||
MAX_RETRIES = 1
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def criterion_fields(state):
|
||||
c = state.get("criterion")
|
||||
if isinstance(c, dict):
|
||||
cid = c.get("id")
|
||||
text = c.get("text")
|
||||
return (
|
||||
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
|
||||
text.strip() if isinstance(text, str) else "",
|
||||
)
|
||||
return "unknown", ""
|
||||
|
||||
|
||||
def extract_json(text):
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
raise ValueError("criterion-check output is empty or not text")
|
||||
text = re.sub(r"```[a-zA-Z]*", "", text).replace("```", "")
|
||||
start, end = text.find("{"), text.rfind("}")
|
||||
if start == -1 or end <= start:
|
||||
raise ValueError("no JSON object found in criterion-check output")
|
||||
return json.loads(text[start : end + 1])
|
||||
|
||||
|
||||
def validate(obj):
|
||||
if not isinstance(obj, dict):
|
||||
raise ValueError("criterion-check output is not a JSON object")
|
||||
status = obj.get("status")
|
||||
if status not in STATUSES:
|
||||
raise ValueError(f"status must be one of {sorted(STATUSES)}, got {status!r}")
|
||||
evidence = obj.get("evidence") if isinstance(obj.get("evidence"), str) else ""
|
||||
complaint = obj.get("complaint") if isinstance(obj.get("complaint"), str) else ""
|
||||
if status == "MET" and not evidence.strip():
|
||||
raise ValueError(
|
||||
"MET requires non-empty evidence citing the satisfying change AND the "
|
||||
"test proving the behavior (file:line)"
|
||||
)
|
||||
if status != "MET" and not complaint.strip():
|
||||
raise ValueError(
|
||||
f"{status} requires a non-empty complaint (what the diff does/omits at "
|
||||
"file:line, and what would make it conform)"
|
||||
)
|
||||
return status, evidence, complaint
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
cid, ctext = criterion_fields(state)
|
||||
attempts = state.get("gate_attempts")
|
||||
if not isinstance(attempts, int):
|
||||
attempts = 0
|
||||
|
||||
try:
|
||||
status, evidence, complaint = validate(extract_json(state.get("crit_verdict")))
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"crit_verdict": {
|
||||
"id": cid,
|
||||
"text": ctext,
|
||||
"status": status,
|
||||
"evidence": evidence,
|
||||
"complaint": complaint,
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
return
|
||||
except Exception as e: # noqa: BLE001
|
||||
reason = str(e)
|
||||
|
||||
if attempts < MAX_RETRIES:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"_next": "check_criterion",
|
||||
"gate_attempts": attempts + 1,
|
||||
"gate_feedback": (
|
||||
"PREVIOUS RESPONSE REJECTED by the verdict gate: " + reason +
|
||||
". Respond with ONLY the JSON object described in your instructions."
|
||||
),
|
||||
}
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"crit_verdict": {
|
||||
"id": cid,
|
||||
"text": ctext,
|
||||
"status": "PARTIAL",
|
||||
"evidence": "",
|
||||
"complaint": (
|
||||
"criterion check output failed machine validation after retry: "
|
||||
+ reason
|
||||
+ " — treat this criterion as unproven"
|
||||
),
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the gate must never crash the chain
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"crit_verdict": {
|
||||
"id": "unknown",
|
||||
"text": "",
|
||||
"status": "PARTIAL",
|
||||
"evidence": "",
|
||||
"complaint": f"criterion gate error: {e} — treat as unproven",
|
||||
}
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic diff resolution for the adversary graph.
|
||||
|
||||
Resolves the diff spec (get_diff semantics: staged → unstaged → HEAD~1 for
|
||||
"auto"), lists changed files, and captures the bounded diff text every
|
||||
criterion branch verifies against. Failures degrade into diff_note — the
|
||||
review proceeds fail-closed rather than crashing.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
MAX_DIFF_CHARS = 60000
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
state = load_state()
|
||||
proj = state.get("project_dir") or "."
|
||||
spec = (state.get("diff_spec") or "auto").strip()
|
||||
|
||||
|
||||
def git(*args):
|
||||
r = subprocess.run(["git", "-C", proj, *args], capture_output=True, text=True, timeout=90)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError((r.stderr or r.stdout).strip()[:300])
|
||||
return r.stdout
|
||||
|
||||
|
||||
out = {"diff_note": ""}
|
||||
try:
|
||||
if spec in ("auto", ""):
|
||||
text = git("diff", "--cached")
|
||||
if not text.strip():
|
||||
text = git("diff")
|
||||
if not text.strip():
|
||||
text = git("diff", "HEAD~1")
|
||||
elif spec == "staged":
|
||||
text = git("diff", "--cached")
|
||||
elif spec == "worktree":
|
||||
text = git("diff", "HEAD")
|
||||
else:
|
||||
text = git("diff", spec)
|
||||
|
||||
files = sorted({
|
||||
line[6:].strip()
|
||||
for line in text.splitlines()
|
||||
if line.startswith("+++ b/")
|
||||
})
|
||||
out["changed_files"] = files[:300]
|
||||
if len(text) > MAX_DIFF_CHARS:
|
||||
out["diff_note"] = (
|
||||
f" (truncated to {MAX_DIFF_CHARS} chars of {len(text)} — use the fs tools "
|
||||
"to inspect files beyond the cut)"
|
||||
)
|
||||
text = text[:MAX_DIFF_CHARS]
|
||||
out["diff_text"] = text if text.strip() else "(empty diff — nothing changed)"
|
||||
except Exception as e: # noqa: BLE001
|
||||
out["changed_files"] = []
|
||||
out["diff_text"] = "(diff unavailable)"
|
||||
out["diff_note"] = f" (diff resolution failed for spec {spec!r}: {e})"
|
||||
|
||||
print(json.dumps(out))
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic conformance verdict for the adversary graph.
|
||||
|
||||
CONFORMS requires: at least one criterion, every criterion MET, and zero
|
||||
holistic complaints. Everything else — including a missing plan — is
|
||||
DIVERGES (fail-closed). Assembles the exact sentinel format the callers
|
||||
route on. Never crashes into a silent verdict.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
verdicts = [v for v in (state.get("crit_verdicts") or []) if isinstance(v, dict)]
|
||||
extra = [c for c in (state.get("extra_complaints") or []) if isinstance(c, dict)]
|
||||
observations = (state.get("observations") or "").strip()
|
||||
n = len(verdicts)
|
||||
|
||||
met = [v for v in verdicts if v.get("status") == "MET"]
|
||||
partial = [v for v in verdicts if v.get("status") == "PARTIAL"]
|
||||
bad = [v for v in verdicts if v.get("status") in ("UNMET", "DIVERGED")]
|
||||
|
||||
lines = []
|
||||
if n == 0:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
lines.append("Criteria: none provided.")
|
||||
lines.append("Complaints:")
|
||||
lines.append(
|
||||
"1. No acceptance criteria were provided or extractable — conformance cannot "
|
||||
"be judged without a spec (fail-closed). Supply the plan's Objective/Tasks/"
|
||||
"Acceptance criteria and re-run."
|
||||
)
|
||||
elif not partial and not bad and not extra:
|
||||
lines.append("ADVERSARIAL_REVIEW: CONFORMS")
|
||||
lines.append(f"Criteria: {n}/{n} met (all with tests).")
|
||||
if observations:
|
||||
lines.append("")
|
||||
lines.append("Non-blocking observations:")
|
||||
lines.append(observations)
|
||||
else:
|
||||
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||
lines.append(
|
||||
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged."
|
||||
)
|
||||
lines.append("Complaints:")
|
||||
i = 0
|
||||
for v in verdicts:
|
||||
if v.get("status") == "MET":
|
||||
continue
|
||||
i += 1
|
||||
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||
if len(text) > 140:
|
||||
text = text[:137] + "…"
|
||||
lines.append(
|
||||
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
|
||||
f'{(v.get("complaint") or "").strip()}'
|
||||
)
|
||||
for c in extra:
|
||||
i += 1
|
||||
lines.append(
|
||||
f"{i}. {c.get('kind', 'violation')} — {c.get('location', '?')} — "
|
||||
f"{c.get('violation', '')} — {c.get('fix', '')}"
|
||||
)
|
||||
if observations:
|
||||
lines.append("")
|
||||
lines.append("Non-blocking observations:")
|
||||
lines.append(observations)
|
||||
|
||||
# Per-criterion evidence appendix (MET entries, for auditability).
|
||||
if met:
|
||||
lines.append("")
|
||||
lines.append("Met criteria (evidence):")
|
||||
for v in met:
|
||||
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||
if len(text) > 100:
|
||||
text = text[:97] + "…"
|
||||
lines.append(f'- "{text}" — {(v.get("evidence") or "").strip()}')
|
||||
|
||||
print(json.dumps({"adv_report": "\n".join(lines)}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — never a silent verdict
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"adv_report": (
|
||||
"ADVERSARIAL_REVIEW: DIVERGES\n"
|
||||
f"Criteria: verdict computation error: {e} — treat as failed, not as conforming."
|
||||
)
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -1,78 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eo pipefail
|
||||
|
||||
# @env LLM_OUTPUT=/dev/stdout
|
||||
# @env LLM_AGENT_VAR_PROJECT_DIR=.
|
||||
# @describe Adversarial plan-conformance reviewer tools
|
||||
|
||||
_project_dir() {
|
||||
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
|
||||
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
|
||||
}
|
||||
|
||||
# @cmd Get the git diff to review for plan conformance. Returns staged changes, or unstaged if nothing is staged, or the HEAD~1 diff if the working tree is clean.
|
||||
# @option --base Optional base ref to diff against (e.g., "main", "HEAD~3", a commit SHA, or a PR base branch)
|
||||
get_diff() {
|
||||
local project_dir
|
||||
project_dir=$(_project_dir)
|
||||
# shellcheck disable=SC2154
|
||||
local base="${argc_base:-}"
|
||||
|
||||
local diff_output=""
|
||||
if [[ -n "${base}" ]]; then
|
||||
diff_output=$(cd "${project_dir}" && git diff "${base}" 2>&1) || true
|
||||
else
|
||||
diff_output=$(cd "${project_dir}" && git diff --cached 2>&1) || true
|
||||
if [[ -z "${diff_output}" ]]; then
|
||||
diff_output=$(cd "${project_dir}" && git diff 2>&1) || true
|
||||
fi
|
||||
if [[ -z "${diff_output}" ]]; then
|
||||
diff_output=$(cd "${project_dir}" && git diff HEAD~1 2>&1) || true
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ -z "${diff_output}" ]]; then
|
||||
echo "No changes found to review in ${project_dir}." >> "$LLM_OUTPUT"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local file_count
|
||||
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
|
||||
{
|
||||
echo "Diff contains changes to ${file_count} file(s):"
|
||||
echo ""
|
||||
echo "${diff_output}"
|
||||
} >> "$LLM_OUTPUT"
|
||||
}
|
||||
|
||||
# @cmd Get the list of changed files with stats (a quick map of what to check against the plan).
|
||||
# @option --base Optional base ref to diff against
|
||||
get_changed_files() {
|
||||
local project_dir
|
||||
project_dir=$(_project_dir)
|
||||
local base="${argc_base:-}"
|
||||
|
||||
local stat_output=""
|
||||
if [[ -n "${base}" ]]; then
|
||||
stat_output=$(cd "${project_dir}" && git diff --stat "${base}" 2>&1) || true
|
||||
else
|
||||
stat_output=$(cd "${project_dir}" && git diff --cached --stat 2>&1) || true
|
||||
if [[ -z "${stat_output}" ]]; then
|
||||
stat_output=$(cd "${project_dir}" && git diff --stat 2>&1) || true
|
||||
fi
|
||||
if [[ -z "${stat_output}" ]]; then
|
||||
stat_output=$(cd "${project_dir}" && git diff --stat HEAD~1 2>&1) || true
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ -z "${stat_output}" ]]; then
|
||||
echo "No changes found in ${project_dir}." >> "$LLM_OUTPUT"
|
||||
return 0
|
||||
fi
|
||||
|
||||
{
|
||||
echo "Changed files:"
|
||||
echo ""
|
||||
echo "${stat_output}"
|
||||
} >> "$LLM_OUTPUT"
|
||||
}
|
||||
Reference in new issue
Block a user