feat: Significant code review suite rewrite with improved graph agent features
This commit is contained in:
41 files changed
+3572
-709
No files matched your search
@@ -0,0 +1,101 @@
|
||||
# Review Gauntlet
|
||||
|
||||
A graph-based post-implementation review gate — **"the done gate"**. An
|
||||
orchestrator that must not claim *done* without green reviews spawns this agent
|
||||
once instead of judging trigger rules and parsing verdict sentinels itself. The
|
||||
graph structure makes skipping a lane or misreading a verdict impossible:
|
||||
trigger signals are computed by a script, lanes run as isolated parallel
|
||||
sub-agents, and the final verdict is a deterministic regex gate — an LLM never
|
||||
gets the chance to rationalize past it.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
|
||||
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
|
||||
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes override exactly)"]
|
||||
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
|
||||
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer"]]
|
||||
madv --> radv[["adversary"]]
|
||||
msec --> rsec[["security-reviewer"]]
|
||||
mpb --> rpb[["probe"]]
|
||||
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
||||
```
|
||||
|
||||
## Why a graph
|
||||
|
||||
The four independent review lanes existed before this agent — as **prompt
|
||||
discipline** inside the calling orchestrator: "spawn code-reviewer when the
|
||||
change is broad", "🔴 always blocks", "INCONCLUSIVE is never PASS". Prompt
|
||||
discipline degrades under context pressure: a lane gets skipped, a verdict
|
||||
line gets paraphrased, a missing report reads as "nothing to report". Here
|
||||
those rules are *structure*:
|
||||
|
||||
- **Lane selection floor is deterministic** (`build_items.py`): code-review
|
||||
always runs; adversary runs when a spec/plan is present; security runs on
|
||||
attack-surface signals (auth/deps/exec paths or diff content) or hardened
|
||||
posture; probe runs when consumer-facing surface changed *and* a local-run
|
||||
recipe exists. The `select_lanes` LLM can only **widen** the set — a false
|
||||
flag never removes a rule-selected lane. `forced_lanes` from the caller
|
||||
override everything exactly.
|
||||
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
||||
over a 0-or-1-item list — an empty list means the map runs zero branches
|
||||
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
||||
the verdict gate on equal-length paths.
|
||||
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
||||
sentinel is a lane FAILURE, never a pass; any 🔴 in the code-review report
|
||||
blocks regardless of its verdict line; `NEEDS-HUMAN` without 🔴 passes but
|
||||
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
|
||||
with an environment note — it is never treated as PASS or FAIL.
|
||||
- **Verdict independence is structural**: lanes run as isolated sub-agents
|
||||
with no `teammates:` flag — no cross-lane messaging, by construction.
|
||||
|
||||
## Sentinels parsed
|
||||
|
||||
| Lane | Sentinel | Blocks on |
|
||||
|------|----------|-----------|
|
||||
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | any 🔴 finding; missing sentinel |
|
||||
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; missing sentinel |
|
||||
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
|
||||
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
|
||||
|
||||
## Spawning it
|
||||
|
||||
```
|
||||
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
|
||||
|
||||
Project: /abs/path/to/repo
|
||||
Diff: main...HEAD (a git range, a ref, or 'worktree' for uncommitted changes)
|
||||
Rigor: production Security posture: standard
|
||||
|
||||
Plan / acceptance criteria:
|
||||
<paste the criteria the implementation must satisfy — enables the adversary lane>
|
||||
|
||||
Local-run recipe / usage suites:
|
||||
<how to boot the service locally + where existing suites live — enables the probe lane>
|
||||
|
||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)"
|
||||
```
|
||||
|
||||
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
|
||||
table, the lane-selection reasons, blockers, attention items, and each lane's
|
||||
full report in a collapsible section. On BLOCKED: fix the findings per your
|
||||
own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with
|
||||
*Human attention required*: complete the work but carry those items into your
|
||||
final report verbatim.
|
||||
|
||||
## Notes
|
||||
|
||||
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||
non-trivial work — that's why code-review is unconditionally on); the
|
||||
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet).
|
||||
A lane that dies or times out produces no sentinel — which is a BLOCKED,
|
||||
never a silent pass.
|
||||
- Signals failures degrade gracefully: if git can't compute the diff, lane
|
||||
selection falls back to caller context + forced lanes, and the report notes
|
||||
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
||||
auth/authn/authz/oauth but not "author") — the safe direction, since signals
|
||||
only ever ADD the security lane.
|
||||
- Requires the `code-reviewer`, `adversary`, `security-reviewer`, and `probe`
|
||||
agents to be installed (they ship with Coyote).
|
||||
@@ -0,0 +1,350 @@
|
||||
name: review-gauntlet
|
||||
description: |
|
||||
Graph-based post-implementation review gate — "the done gate". Given a diff
|
||||
spec and the work's context (plan/acceptance criteria, rigor, security
|
||||
posture), it deterministically computes diff signals, selects the applicable
|
||||
independent review lanes — code-reviewer (always), adversary (when a
|
||||
spec/plan exists), security-reviewer (attack-surface signals), probe
|
||||
(consumer-facing surface with a local-run recipe) — runs the selected lanes
|
||||
IN PARALLEL as isolated sub-agents, and machine-parses their verdict
|
||||
sentinels into one deterministic verdict: GAUNTLET: PASS or
|
||||
GAUNTLET: BLOCKED. An orchestrator that must not claim "done" without green
|
||||
reviews spawns this once instead of judging trigger rules and parsing
|
||||
verdicts itself — the graph structure makes skipping a lane or misreading a
|
||||
verdict impossible. Invariants: lane selection can only be WIDENED by LLM
|
||||
judgment, never narrowed (deterministic rules are the floor); a missing
|
||||
verdict sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks
|
||||
with an environment note (it is never PASS or FAIL); reviewer lanes run
|
||||
verdict-independent — no cross-lane messaging.
|
||||
|
||||
version: '1.0'
|
||||
|
||||
max_agent_depth: 4
|
||||
max_concurrent_agents: 12
|
||||
|
||||
variables:
|
||||
- name: project_dir
|
||||
description: Repository the change lives in; signals and every review lane run against it.
|
||||
default: '.'
|
||||
|
||||
settings:
|
||||
max_loop_iterations: 10
|
||||
log_state_snapshots: true
|
||||
timeout: 10800
|
||||
|
||||
initial_state:
|
||||
diff_spec: worktree
|
||||
plan_context: ''
|
||||
rigor: production
|
||||
security_posture: standard
|
||||
probe_context: ''
|
||||
extra_context: ''
|
||||
forced_lanes: []
|
||||
changed_files: []
|
||||
file_count: 0
|
||||
signals_summary: ''
|
||||
signals_error: ''
|
||||
touches_auth: false
|
||||
touches_deps: false
|
||||
touches_exec: false
|
||||
touches_proto: false
|
||||
touches_migrations: false
|
||||
consumer_surface: false
|
||||
docs_only: false
|
||||
add_code_review: false
|
||||
add_adversary: false
|
||||
add_security: false
|
||||
add_probe: false
|
||||
lane_reasons: ''
|
||||
code_review_items: []
|
||||
adversary_items: []
|
||||
security_items: []
|
||||
probe_items: []
|
||||
code_review_results: []
|
||||
adversary_results: []
|
||||
security_results: []
|
||||
probe_results: []
|
||||
lanes_summary: ''
|
||||
gauntlet_verdict: ''
|
||||
gauntlet_report: ''
|
||||
|
||||
start: parse
|
||||
|
||||
nodes:
|
||||
parse:
|
||||
id: parse
|
||||
type: llm
|
||||
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging, no lane selection.
|
||||
instructions: |
|
||||
You extract the facts of a review request into structured fields.
|
||||
EXTRACTION ONLY — do not decide which review lanes should run.
|
||||
|
||||
- `diff_spec`: the git diff specification the caller named — a ref
|
||||
range ("main...HEAD", "HEAD~3..HEAD"), a single ref, or the literal
|
||||
"worktree" for uncommitted changes against HEAD. When the caller
|
||||
did not name one, use "worktree".
|
||||
- `plan_context`: the plan / spec / acceptance-criteria text the
|
||||
implementation must satisfy, verbatim (or a file path the caller
|
||||
pointed at). Empty string when the caller supplied none.
|
||||
- `rigor`: poc | prototype | production. Default "production" when
|
||||
not stated.
|
||||
- `security_posture`: prototype | standard | hardened. Default
|
||||
"standard" when not stated.
|
||||
- `probe_context`: the local-run recipe and/or existing usage-suite
|
||||
pointers for black-box probing (how to boot the service, where the
|
||||
suites live). Empty string when the caller supplied none.
|
||||
- `forced_lanes`: lanes the caller explicitly demanded (verbatim
|
||||
names). Empty array = automatic selection.
|
||||
- `extra_context`: any remaining caller context reviewers should see,
|
||||
condensed.
|
||||
prompt: |
|
||||
Extract the review-request facts from this request:
|
||||
|
||||
{{initial_prompt}}
|
||||
tools: []
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
diff_spec:
|
||||
type: string
|
||||
plan_context:
|
||||
type: string
|
||||
rigor:
|
||||
type: string
|
||||
security_posture:
|
||||
type: string
|
||||
probe_context:
|
||||
type: string
|
||||
forced_lanes:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
extra_context:
|
||||
type: string
|
||||
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
next: signals
|
||||
|
||||
signals:
|
||||
id: signals
|
||||
type: script
|
||||
description: Deterministic diff signals — file list, attack-surface and consumer-surface indicators. Pure git; no LLM judgment.
|
||||
script: scripts/signals.py
|
||||
timeout: 120
|
||||
next: select_lanes
|
||||
|
||||
select_lanes:
|
||||
id: select_lanes
|
||||
type: llm
|
||||
description: Additive-only lane judgment on top of the deterministic rules. It can WIDEN the gauntlet, never narrow it.
|
||||
instructions: |
|
||||
You look at a change's deterministic diff signals and decide whether any
|
||||
review lane the hard rules might have missed should ALSO run. You can
|
||||
only ADD lanes — the deterministic floor (enforced downstream, not by
|
||||
you) is: code-review always runs; adversary runs when a plan/spec is
|
||||
present; security runs on attack-surface signals or hardened posture;
|
||||
probe runs when consumer-facing surface changed and a local-run recipe
|
||||
exists. Setting a flag to false NEVER removes a lane the rules selected.
|
||||
|
||||
Reasons to add:
|
||||
- `add_security`: the diff handles external/untrusted input, touches
|
||||
crypto/secrets handling, or changes trust boundaries in ways the
|
||||
path-based signals missed.
|
||||
- `add_adversary`: the request references behavior it must preserve or
|
||||
criteria scattered in prose that amount to a spec, even without a
|
||||
formal plan.
|
||||
- `add_probe`: the change alters observable API/CLI behavior even
|
||||
though the path signals missed it (only useful when a local-run
|
||||
recipe exists — the builder skips it otherwise, noting why).
|
||||
- `add_code_review`: always selected by the rules; flag exists only
|
||||
for forced-lane configurations.
|
||||
|
||||
Put one line per added lane into `lane_reasons`. When nothing needs
|
||||
adding, all flags false and empty reasons is the correct answer.
|
||||
prompt: |
|
||||
Change signals (deterministic):
|
||||
- diff spec: {{diff_spec}}
|
||||
- files changed: {{file_count}}
|
||||
- changed files (first 200): {{changed_files}}
|
||||
- touches auth paths/content: {{touches_auth}}
|
||||
- touches dependency manifests: {{touches_deps}}
|
||||
- adds exec/shell/subprocess calls: {{touches_exec}}
|
||||
- touches protobuf/contracts: {{touches_proto}}
|
||||
- touches migrations: {{touches_migrations}}
|
||||
- consumer-facing surface paths: {{consumer_surface}}
|
||||
- docs-only diff: {{docs_only}}
|
||||
- {{signals_summary}}
|
||||
{{signals_error}}
|
||||
|
||||
Caller context:
|
||||
- rigor: {{rigor}}; security posture: {{security_posture}}
|
||||
- plan/spec present: {{plan_context}}
|
||||
- probe recipe present: {{probe_context}}
|
||||
- extra: {{extra_context}}
|
||||
tools: []
|
||||
output_schema:
|
||||
type: object
|
||||
properties:
|
||||
add_code_review:
|
||||
type: boolean
|
||||
add_adversary:
|
||||
type: boolean
|
||||
add_security:
|
||||
type: boolean
|
||||
add_probe:
|
||||
type: boolean
|
||||
lane_reasons:
|
||||
type: string
|
||||
required: [add_code_review, add_adversary, add_security, add_probe, lane_reasons]
|
||||
max_iterations: 3
|
||||
max_attempts: 2
|
||||
next: build_items
|
||||
|
||||
build_items:
|
||||
id: build_items
|
||||
type: script
|
||||
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map.
|
||||
script: scripts/build_items.py
|
||||
timeout: 30
|
||||
next: [map_code_review, map_adversary, map_security, map_probe]
|
||||
|
||||
map_code_review:
|
||||
id: map_code_review
|
||||
type: map
|
||||
over: '{{code_review_items}}'
|
||||
as: lane_ctx
|
||||
branch: run_code_review
|
||||
output_key: lane_out
|
||||
collect_into: code_review_results
|
||||
max_concurrency: 1
|
||||
next: verdict_gate
|
||||
|
||||
map_adversary:
|
||||
id: map_adversary
|
||||
type: map
|
||||
over: '{{adversary_items}}'
|
||||
as: lane_ctx
|
||||
branch: run_adversary
|
||||
output_key: lane_out
|
||||
collect_into: adversary_results
|
||||
max_concurrency: 1
|
||||
next: verdict_gate
|
||||
|
||||
map_security:
|
||||
id: map_security
|
||||
type: map
|
||||
over: '{{security_items}}'
|
||||
as: lane_ctx
|
||||
branch: run_security
|
||||
output_key: lane_out
|
||||
collect_into: security_results
|
||||
max_concurrency: 1
|
||||
next: verdict_gate
|
||||
|
||||
map_probe:
|
||||
id: map_probe
|
||||
type: map
|
||||
over: '{{probe_items}}'
|
||||
as: lane_ctx
|
||||
branch: run_probe
|
||||
output_key: lane_out
|
||||
collect_into: probe_results
|
||||
max_concurrency: 1
|
||||
next: verdict_gate
|
||||
|
||||
run_code_review:
|
||||
id: run_code_review
|
||||
type: agent
|
||||
description: Independent code-quality review lane.
|
||||
agent: code-reviewer
|
||||
timeout: 7200
|
||||
prompt: |
|
||||
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
|
||||
independent post-implementation review.
|
||||
|
||||
Quality bar: rigor={{rigor}}.
|
||||
|
||||
Plan / acceptance-criteria context (may be empty):
|
||||
{{plan_context}}
|
||||
|
||||
{{extra_context}}
|
||||
|
||||
Close your report with your Review Verdict line.
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
|
||||
run_adversary:
|
||||
id: run_adversary
|
||||
type: agent
|
||||
description: Plan-conformance review lane.
|
||||
agent: adversary
|
||||
timeout: 7200
|
||||
prompt: |
|
||||
Adversarial plan-conformance review in {{project_dir}}.
|
||||
|
||||
The implementation to check: diff {{diff_spec}}.
|
||||
|
||||
The spec / plan / acceptance criteria it must satisfy:
|
||||
{{plan_context}}
|
||||
|
||||
{{extra_context}}
|
||||
|
||||
Return your ADVERSARIAL_REVIEW verdict.
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
|
||||
run_security:
|
||||
id: run_security
|
||||
type: agent
|
||||
description: Security review lane.
|
||||
agent: security-reviewer
|
||||
timeout: 7200
|
||||
prompt: |
|
||||
Security review of the code changes in {{project_dir}} (diff: {{diff_spec}}).
|
||||
|
||||
Security posture: {{security_posture}}.
|
||||
|
||||
{{extra_context}}
|
||||
|
||||
Return your SECURITY_REVIEW verdict.
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
|
||||
run_probe:
|
||||
id: run_probe
|
||||
type: agent
|
||||
description: Black-box usage-pattern verification lane.
|
||||
agent: probe
|
||||
timeout: 7200
|
||||
prompt: |
|
||||
Usage-pattern verification (black-box probe) of the changes in
|
||||
{{project_dir}} (diff: {{diff_spec}}).
|
||||
|
||||
Local-run recipe / existing usage suites:
|
||||
{{probe_context}}
|
||||
|
||||
Plan / acceptance-criteria context:
|
||||
{{plan_context}}
|
||||
|
||||
Derive expected behaviors from the plan/contract BEFORE reading
|
||||
implementation code, run existing suites first, then author spec-first
|
||||
tests for the uncovered delta. Return your USAGE_PROBE verdict.
|
||||
state_updates:
|
||||
lane_out: '{{output}}'
|
||||
|
||||
verdict_gate:
|
||||
id: verdict_gate
|
||||
type: script
|
||||
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; 🔴 findings block regardless of the lane's own verdict line.
|
||||
script: scripts/verdict_gate.py
|
||||
timeout: 60
|
||||
next: done
|
||||
|
||||
done:
|
||||
id: done
|
||||
type: end
|
||||
output: |
|
||||
GAUNTLET: {{gauntlet_verdict}}
|
||||
|
||||
{{gauntlet_report}}
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic lane builder for the review gauntlet.
|
||||
|
||||
Selection = hard rules (the floor) ∪ LLM additions; forced lanes override
|
||||
everything exactly. Emits one 0-or-1-item list per lane: an empty list makes
|
||||
that lane's map node run zero branches (map-over-nothing), which IS the
|
||||
skip mechanism — no dynamic routing anywhere.
|
||||
|
||||
The LLM's add_* flags can only ADD lanes. A flag that is false never
|
||||
removes a lane the hard rules selected.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
state = load_state()
|
||||
|
||||
ALIASES = {
|
||||
"code-review": "code-review",
|
||||
"code_review": "code-review",
|
||||
"code-reviewer": "code-review",
|
||||
"adversary": "adversary",
|
||||
"adversarial": "adversary",
|
||||
"adversarial-review": "adversary",
|
||||
"security": "security",
|
||||
"security-review": "security",
|
||||
"security-reviewer": "security",
|
||||
"probe": "probe",
|
||||
"usage": "probe",
|
||||
"usage-pattern": "probe",
|
||||
"usage-pattern-testing": "probe",
|
||||
}
|
||||
|
||||
|
||||
def text(key):
|
||||
v = state.get(key)
|
||||
return v.strip() if isinstance(v, str) else ""
|
||||
|
||||
|
||||
forced_raw = state.get("forced_lanes") or []
|
||||
forced, unknown = [], []
|
||||
for lane in forced_raw:
|
||||
canon = ALIASES.get(str(lane).strip().lower())
|
||||
(forced.append(canon) if canon else unknown.append(str(lane)))
|
||||
|
||||
reasons = []
|
||||
if forced:
|
||||
lanes = set(forced)
|
||||
reasons.append(f"forced lanes honored exactly: {sorted(lanes)}")
|
||||
else:
|
||||
lanes = {"code-review"}
|
||||
reasons.append(
|
||||
"code-review: always on — the gauntlet is only spawned for non-trivial work"
|
||||
)
|
||||
if text("plan_context"):
|
||||
lanes.add("adversary")
|
||||
reasons.append("adversary: a spec/plan/acceptance-criteria context is present")
|
||||
if (
|
||||
state.get("touches_auth")
|
||||
or state.get("touches_deps")
|
||||
or state.get("touches_exec")
|
||||
or state.get("security_posture") == "hardened"
|
||||
):
|
||||
lanes.add("security")
|
||||
reasons.append(
|
||||
"security: attack-surface signals (auth/deps/exec) or hardened posture"
|
||||
)
|
||||
if state.get("consumer_surface") and text("probe_context"):
|
||||
lanes.add("probe")
|
||||
reasons.append(
|
||||
"probe: consumer-facing surface changed and a local-run recipe/suite pointer exists"
|
||||
)
|
||||
|
||||
# LLM judgment is additive-only.
|
||||
llm_why = text("lane_reasons")[:300]
|
||||
for flag, lane in (
|
||||
("add_code_review", "code-review"),
|
||||
("add_adversary", "adversary"),
|
||||
("add_security", "security"),
|
||||
("add_probe", "probe"),
|
||||
):
|
||||
if state.get(flag) and lane not in lanes:
|
||||
if lane == "probe" and not text("probe_context"):
|
||||
reasons.append(
|
||||
"probe: requested by lane judgment but skipped — no local-run "
|
||||
"recipe/suite pointer (a probe without one is guaranteed INCONCLUSIVE)"
|
||||
)
|
||||
continue
|
||||
if lane == "adversary" and not text("plan_context"):
|
||||
reasons.append(
|
||||
"adversary: requested by lane judgment but skipped — no spec/plan "
|
||||
"to check conformance against"
|
||||
)
|
||||
continue
|
||||
lanes.add(lane)
|
||||
reasons.append(f"{lane}: added by lane-selection judgment — {llm_why}")
|
||||
|
||||
if unknown:
|
||||
reasons.append(f"ignored unknown forced lane name(s): {unknown}")
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"code_review_items": [{"lane": "code-review"}] if "code-review" in lanes else [],
|
||||
"adversary_items": [{"lane": "adversary"}] if "adversary" in lanes else [],
|
||||
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
||||
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
||||
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
||||
}
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic diff signals for the review gauntlet.
|
||||
|
||||
Pure git — no LLM judgment. Every signal is computed from the changed-file
|
||||
list and the added lines of the diff. Failures never sink the gauntlet:
|
||||
on any error the script reports `signals_error` and leaves the defaults
|
||||
(false/empty) in place, which biases lane selection toward the caller's
|
||||
context and forced lanes.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
state = load_state()
|
||||
proj = state.get("project_dir") or "."
|
||||
spec = (state.get("diff_spec") or "worktree").strip()
|
||||
|
||||
|
||||
def git(*args):
|
||||
r = subprocess.run(
|
||||
["git", "-C", proj, *args], capture_output=True, text=True, timeout=90
|
||||
)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError((r.stderr or r.stdout).strip()[:400])
|
||||
return r.stdout
|
||||
|
||||
|
||||
out = {}
|
||||
try:
|
||||
if spec in ("", "worktree"):
|
||||
names = git("diff", "--name-only", "HEAD")
|
||||
difftext = git("diff", "HEAD")
|
||||
else:
|
||||
names = git("diff", "--name-only", spec)
|
||||
difftext = git("diff", spec)
|
||||
files = [f.strip() for f in names.splitlines() if f.strip()]
|
||||
added = "\n".join(
|
||||
l for l in difftext.splitlines() if l.startswith("+") and not l.startswith("+++")
|
||||
)
|
||||
|
||||
def any_path(pattern):
|
||||
return any(re.search(pattern, f, re.I) for f in files)
|
||||
|
||||
out.update(
|
||||
{
|
||||
"changed_files": files[:200],
|
||||
"file_count": len(files),
|
||||
"touches_proto": any_path(r"\.proto$|buf\.(yaml|gen\.yaml)$"),
|
||||
"touches_migrations": any_path(r"migrations?/|\.sql$"),
|
||||
# auth(?!or) admits auth/authn/authz/oauth but not "author".
|
||||
# Over-firing here is the safe direction: it only ADDS a lane.
|
||||
"touches_auth": any_path(
|
||||
r"auth(?!or)|login|logout|token|session|permission|rbac|\bacl\b|credential|secret"
|
||||
)
|
||||
or bool(
|
||||
re.search(
|
||||
r"auth(?!or)|password|token|secret|credential|jwt|session",
|
||||
added,
|
||||
re.I,
|
||||
)
|
||||
),
|
||||
"touches_deps": any_path(
|
||||
r"go\.(mod|sum)$|package(-lock)?\.json$|yarn\.lock$|pnpm-lock\.yaml$"
|
||||
r"|Cargo\.(toml|lock)$|requirements[^/]*\.txt$|pyproject\.toml$"
|
||||
r"|Gemfile(\.lock)?$|pom\.xml$|build\.gradle|composer\.(json|lock)$"
|
||||
),
|
||||
"touches_exec": bool(
|
||||
re.search(
|
||||
r"exec\.Command|subprocess|os/exec|Runtime\.getRuntime"
|
||||
r"|shell_exec|\beval\(|\bsystem\(|child_process",
|
||||
added,
|
||||
)
|
||||
),
|
||||
"consumer_surface": any_path(
|
||||
r"routes?|handlers?|\bapi\b|endpoints?|\bcmd/|\bcli\b|\.proto$"
|
||||
r"|openapi|swagger|controllers?|graphql"
|
||||
),
|
||||
"docs_only": bool(files)
|
||||
and all(re.search(r"\.(md|rst|txt|adoc)$", f, re.I) for f in files),
|
||||
"signals_summary": f"{len(files)} file(s) changed (diff: {spec})",
|
||||
}
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 — degraded signals must not sink the gauntlet
|
||||
out["signals_error"] = (
|
||||
f"NOTE: diff signals could not be computed ({e}); "
|
||||
"lane selection falls back to caller context and forced lanes."
|
||||
)
|
||||
out["signals_summary"] = "signals unavailable"
|
||||
|
||||
print(json.dumps(out))
|
||||
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic verdict gate for the review gauntlet.
|
||||
|
||||
Parses each configured lane's verdict SENTINEL with a regex — no LLM
|
||||
re-reads, no judgment. The gate's hard rules:
|
||||
|
||||
- a missing sentinel is a LANE FAILURE and blocks — never a pass;
|
||||
- any 🔴 finding in the code-review report blocks regardless of the
|
||||
lane's own verdict line;
|
||||
- NEEDS-HUMAN without 🔴 passes but is surfaced as attention required;
|
||||
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
||||
treated as PASS or FAIL (fix the environment per the plan's local-run
|
||||
recipe, then re-run);
|
||||
- the gate itself must never crash into a silent pass: any internal
|
||||
error emits GAUNTLET: BLOCKED naming the error.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
blockers, attention, lane_lines, reports = [], [], [], []
|
||||
|
||||
def lane_report(key):
|
||||
arr = state.get(key) or []
|
||||
if not arr:
|
||||
return None
|
||||
item = arr[0]
|
||||
return item if isinstance(item, str) else json.dumps(item)
|
||||
|
||||
def record(lane, status, detail=""):
|
||||
lane_lines.append(f"| {lane} | {status} | {detail} |")
|
||||
|
||||
# --- code-review ------------------------------------------------------
|
||||
cr = lane_report("code_review_results")
|
||||
if cr is None:
|
||||
record("code-review", "SKIPPED", "not selected")
|
||||
else:
|
||||
reports.append(("code-review", cr))
|
||||
m = re.search(r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)", cr)
|
||||
reds = cr.count("🔴")
|
||||
if reds:
|
||||
blockers.append(
|
||||
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
||||
)
|
||||
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
||||
elif m and m.group(1) == "MERGE-READY":
|
||||
record("code-review", "GREEN", "MERGE-READY")
|
||||
elif m:
|
||||
attention.append(
|
||||
"code-review: NEEDS-HUMAN — see the lane report's "
|
||||
"'Human attention required' section"
|
||||
)
|
||||
record("code-review", "GREEN (attention)", "NEEDS-HUMAN, no 🔴")
|
||||
else:
|
||||
blockers.append(
|
||||
"code-review: no Review Verdict sentinel found — the lane failed; "
|
||||
"a missing verdict is never a pass"
|
||||
)
|
||||
record("code-review", "BLOCKED", "missing verdict sentinel")
|
||||
|
||||
# --- adversary --------------------------------------------------------
|
||||
adv = lane_report("adversary_results")
|
||||
if adv is None:
|
||||
record("adversary", "SKIPPED", "not selected")
|
||||
else:
|
||||
reports.append(("adversary", adv))
|
||||
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
|
||||
if m and m.group(1) == "CONFORMS":
|
||||
record("adversary", "GREEN", "CONFORMS")
|
||||
elif m:
|
||||
blockers.append(
|
||||
"adversary: DIVERGES — the implementation does not conform to the plan"
|
||||
)
|
||||
record("adversary", "BLOCKED", "DIVERGES")
|
||||
else:
|
||||
blockers.append(
|
||||
"adversary: no ADVERSARIAL_REVIEW sentinel found — lane failed; never a pass"
|
||||
)
|
||||
record("adversary", "BLOCKED", "missing verdict sentinel")
|
||||
|
||||
# --- security ---------------------------------------------------------
|
||||
sec = lane_report("security_results")
|
||||
if sec is None:
|
||||
record("security", "SKIPPED", "not selected")
|
||||
else:
|
||||
reports.append(("security", sec))
|
||||
m = re.search(r"SECURITY_REVIEW:\s*(PASS|FAIL)", sec)
|
||||
if m and m.group(1) == "PASS":
|
||||
record("security", "GREEN", "PASS")
|
||||
elif m:
|
||||
blockers.append("security: SECURITY_REVIEW: FAIL — exploitable finding(s)")
|
||||
record("security", "BLOCKED", "FAIL")
|
||||
else:
|
||||
blockers.append(
|
||||
"security: no SECURITY_REVIEW sentinel found — lane failed; never a pass"
|
||||
)
|
||||
record("security", "BLOCKED", "missing verdict sentinel")
|
||||
|
||||
# --- probe ------------------------------------------------------------
|
||||
pb = lane_report("probe_results")
|
||||
if pb is None:
|
||||
record("probe", "SKIPPED", "not selected")
|
||||
else:
|
||||
reports.append(("probe", pb))
|
||||
m = re.search(r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)", pb)
|
||||
if m and m.group(1) == "PASS":
|
||||
record("probe", "GREEN", "PASS")
|
||||
elif m and m.group(1) == "FAIL":
|
||||
blockers.append("probe: USAGE_PROBE: FAIL — observed behavior contradicts the contract")
|
||||
record("probe", "BLOCKED", "FAIL")
|
||||
elif m:
|
||||
blockers.append(
|
||||
"probe: INCONCLUSIVE — the ENVIRONMENT could not be established; this is "
|
||||
"not a code verdict. Follow the plan's local-run recipe to fix the "
|
||||
"environment and re-run. INCONCLUSIVE is never treated as PASS or FAIL."
|
||||
)
|
||||
record("probe", "BLOCKED", "INCONCLUSIVE (environment)")
|
||||
else:
|
||||
blockers.append(
|
||||
"probe: no USAGE_PROBE sentinel found — lane failed; never a pass"
|
||||
)
|
||||
record("probe", "BLOCKED", "missing verdict sentinel")
|
||||
|
||||
verdict = "BLOCKED" if blockers else "PASS"
|
||||
|
||||
md = "\n".join(
|
||||
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines
|
||||
)
|
||||
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
|
||||
if blockers:
|
||||
md += "\n\n## Blockers\n" + "\n".join(f"- {b}" for b in blockers)
|
||||
if attention:
|
||||
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
||||
if state.get("signals_error"):
|
||||
md += f"\n\n> {state['signals_error']}"
|
||||
md += "\n\n## Full lane reports\n"
|
||||
for lane, rep in reports:
|
||||
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
||||
|
||||
print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md}))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the gate must never crash into a silent pass
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"gauntlet_verdict": "BLOCKED",
|
||||
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
||||
}
|
||||
)
|
||||
)
|
||||
Reference in new issue
Block a user