208 lines
8.3 KiB
YAML
208 lines
8.3 KiB
YAML
name: finding-verifier
|
|
description: |
|
|
Graph-based independent verifier for code-review findings. Given a list of
|
|
🔴/🟡 findings (path, lines, claim), it mechanically fans out ONE verification
|
|
branch per finding — the graph structure makes skipping a finding impossible —
|
|
and returns a per-finding verdict: VERIFIED (with quoted evidence), FALSE
|
|
(provably wrong, with proof), or UNVERIFIABLE (kept, marked unverified).
|
|
Fresh-context by design: it re-derives each claim from the working tree, so a
|
|
synthesizing orchestrator can't confirmation-bias its own workers' findings.
|
|
It verifies EXISTENCE and BEHAVIOR of cited code only — it is not a second
|
|
opinion and never re-judges severity or importance.
|
|
|
|
version: '1.0'
|
|
|
|
global_tools:
|
|
- fs_read.sh
|
|
- fs_cat.sh
|
|
- fs_grep.sh
|
|
|
|
variables:
|
|
- name: project_dir
|
|
description: Repository the findings cite; relative paths in findings resolve against it.
|
|
default: '.'
|
|
|
|
settings:
|
|
max_loop_iterations: 8
|
|
log_state_snapshots: true
|
|
timeout: 3000
|
|
|
|
initial_state:
|
|
findings_list: []
|
|
verdicts: []
|
|
gate_feedback: ''
|
|
gate_attempts: 0
|
|
parse_failure: ''
|
|
|
|
start: parse
|
|
|
|
nodes:
|
|
parse:
|
|
id: parse
|
|
type: llm
|
|
description: Extract the 🔴/🟡 findings from the spawn prompt into a structured list. Extraction only — no judging.
|
|
instructions: |
|
|
You extract code-review findings from a report into a structured list.
|
|
EXTRACTION ONLY — do not judge, filter, merge, or rephrase-away any
|
|
finding. Rules:
|
|
|
|
- Include every 🔴 CRITICAL and 🟡 WARNING finding. Ignore 🟢/💡
|
|
findings (they post unverified by design).
|
|
- Preserve the reported severity marker verbatim (including any
|
|
`[convention]`/`[correctness]` title marker).
|
|
- `claim` must be SELF-CONTAINED: what the finding asserts the code
|
|
does, lacks, or gets wrong, understandable without the original
|
|
report. Include the suggested-fix context when it clarifies the claim.
|
|
- `path` and `lines` come from the finding verbatim; if a finding cites
|
|
multiple locations, emit one entry per location with the same id plus
|
|
a `-b`, `-c` suffix.
|
|
- `project_dir`: the repository directory named in the request (a
|
|
"Repository:" or "Project:" line). When none is named, echo the
|
|
default shown in the prompt.
|
|
prompt: |
|
|
Extract the findings from this verification request (default repository
|
|
directory when the request names none: {{project_dir}}):
|
|
|
|
{{initial_prompt}}
|
|
tools: []
|
|
output_schema:
|
|
type: object
|
|
properties:
|
|
findings_list:
|
|
type: array
|
|
items:
|
|
type: object
|
|
properties:
|
|
id:
|
|
type: string
|
|
description: Short stable slug for mapping the verdict back (e.g. "f1", "f2-b").
|
|
severity:
|
|
type: string
|
|
description: The severity marker verbatim (e.g. "🔴 CRITICAL", "🟡 WARNING [correctness]").
|
|
path:
|
|
type: string
|
|
description: File path the finding cites, as reported.
|
|
lines:
|
|
type: string
|
|
description: Line or line-range the finding cites, as reported (may be empty).
|
|
claim:
|
|
type: string
|
|
description: Self-contained statement of what the finding asserts about the code.
|
|
required: [id, severity, path, lines, claim]
|
|
project_dir:
|
|
type: string
|
|
description: Repository directory named in the request (echo the prompt default when none).
|
|
required: [findings_list, project_dir]
|
|
max_attempts: 2
|
|
state_updates:
|
|
parse_failure: '{{output}}'
|
|
fallback: parse_fault
|
|
next: verify_each
|
|
|
|
parse_fault:
|
|
id: parse_fault
|
|
type: script
|
|
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — rewrites `verdicts` to a single UNVERIFIABLE PIPELINE-FAULT entry and jumps straight to done, so the FINDING_VERIFIER_RESULTS sentinel still renders and the caller's verify lane degrades instead of dying.
|
|
script: scripts/parse_fault.py
|
|
timeout: 30
|
|
next: done
|
|
|
|
verify_each:
|
|
id: verify_each
|
|
type: map
|
|
over: '{{findings_list}}'
|
|
as: finding
|
|
branch: verify_one
|
|
collect_into: verdicts
|
|
output_key: verdict
|
|
max_concurrency: 4
|
|
next: done
|
|
|
|
verify_one:
|
|
id: verify_one
|
|
type: llm
|
|
description: Verify ONE finding against the working tree — existence and behavior of the cited code, nothing else.
|
|
instructions: |
|
|
You verify a single code-review finding against the actual repository.
|
|
Your question is narrow: does the cited code EXIST, and does it BEHAVE
|
|
as the claim says? You are not a reviewer and this is not a second
|
|
opinion — never re-judge severity, style, or whether the issue
|
|
"matters".
|
|
|
|
Method:
|
|
1. `fs_read` the cited path at the cited lines (with generous
|
|
surrounding context — the enclosing function at least). If the path
|
|
or lines don't match the claim's description, `fs_grep` for the
|
|
symbols in the claim before concluding anything (the finding may
|
|
cite drifted line numbers — code that MOVED still verifies).
|
|
2. Compare what the code actually does against the claim.
|
|
|
|
Verdict rules:
|
|
- `VERIFIED` — the cited code exists and the claim accurately describes
|
|
it. `evidence` MUST be the decisive line(s) quoted verbatim with
|
|
path:line.
|
|
- `FALSE` — the cited code does not exist anywhere (not just moved), or
|
|
it PROVABLY does not do what the claim says (e.g. the claim says a
|
|
nil-check is missing but it is right there). `evidence` MUST quote
|
|
the proof. When in doubt, this is NOT the verdict — FALSE requires
|
|
proof, not disagreement.
|
|
- `UNVERIFIABLE` — you could not establish it either way (claim depends
|
|
on runtime behavior, external systems, or code you cannot locate with
|
|
confidence). Explain why in `note`. Doubt lands here, never in FALSE.
|
|
|
|
Output format — respond with ONLY this JSON object, no prose before or
|
|
after it:
|
|
{"id": "<the finding's id, echoed back verbatim>",
|
|
"verdict": "VERIFIED" | "FALSE" | "UNVERIFIABLE",
|
|
"evidence": "<verbatim quoted line(s) with path:line — required for
|
|
VERIFIED and FALSE, empty string for UNVERIFIABLE>",
|
|
"note": "<one line — for FALSE the disproof in words, for UNVERIFIABLE
|
|
why it could not be established; empty string otherwise>"}
|
|
|
|
Your response is machine-validated by a deterministic gate: a malformed
|
|
response (wrong shape, unknown verdict value, missing required
|
|
evidence/note) is rejected back to you exactly once with the reason,
|
|
then recorded as UNVERIFIABLE. The `id` you echo is informative only —
|
|
the gate stamps the authoritative id from the finding itself.
|
|
prompt: |
|
|
Repository: {{project_dir}}
|
|
|
|
Verify this single finding:
|
|
{{finding}}
|
|
{{gate_feedback}}
|
|
tools:
|
|
- fs_read
|
|
- fs_cat
|
|
- fs_grep
|
|
max_iterations: 28
|
|
max_attempts: 2
|
|
state_updates:
|
|
verdict: '{{output}}'
|
|
fallback: verify_fault
|
|
next: verdict_gate
|
|
|
|
verdict_gate:
|
|
id: verdict_gate
|
|
type: script
|
|
description: Deterministic contract gate for each verdict — validates shape, verdict enum, and evidence/note rules; rejects back to verify_one exactly once with the reason, then records UNVERIFIABLE. No LLM judgment.
|
|
script: scripts/verdict_gate.py
|
|
timeout: 30
|
|
|
|
verify_fault:
|
|
id: verify_fault
|
|
type: script
|
|
description: Fail-closed marker for a dead verify branch (reached only via verify_one's fallback; branch-local). Normalizes the engine's failure text in `verdict` into a schema-conformant UNVERIFIABLE verdict the map collects — a dead verifier lane never sinks the whole map or masquerades as verified.
|
|
script: scripts/verify_fault.py
|
|
timeout: 30
|
|
|
|
done:
|
|
id: done
|
|
type: end
|
|
output: |
|
|
FINDING_VERIFIER_RESULTS
|
|
(verdict semantics for the caller: VERIFIED = keep the finding and paste its evidence
|
|
quote into the Description; FALSE = drop the finding and tally it in the footer;
|
|
UNVERIFIABLE = keep the finding, marked unverified. Severities are never rewritten.)
|
|
|
|
{{verdicts}}
|