feat: created the finding-verifier and whetstone agents for continuous code quality improvements

This commit is contained in:
Dark-Alex-17 committed 2026-09-09 17:22:26 -06:00
1 parent 71c3283b87
commit 2928125b0e
16 files changed
+848 -13

No files matched your search

+11 -1
View File
@@ -6,7 +6,7 @@ description: |
plan-conformance check (plus a black-box usage-pattern probe for consumer-facing surface), and
finishes with ONE draft PR (CI checks watched to green) plus tracked follow-up tasks. Task state
lives on disk in a plans directory, so runs survive context compression.
version: 2.2.0
version: 2.3.0
agent_session: temp
auto_continue: true
max_auto_continues: 100
@@ -21,6 +21,7 @@ spawnable_agents:
- adversary
- probe
- gatekeeper
- whetstone
max_concurrent_agents: 10
max_agent_depth: 10
inject_spawn_instructions: true
@@ -454,6 +455,14 @@ instructions: |
7. Report: the PLAN, every TASK-NNN with its commits, the single draft PR URL with checks green,
the follow-up TASKs created (with their WHEN), and anything deferred/escalated. STOP.
Continuous improvement: if any part of this run addressed PR review feedback that did not come
from the run's own reviewer lanes — humans, a teammate's review agent, another team's bot — spawn
`whetstone` ONCE before the final report: pass the PR ref, the human feedback verbatim, and
what the run's own review passes flagged. Collect it; its approval questions escalate to the
user directly. Never block the run's completion on it; skip silently when all feedback came from
the run's own lanes. (Per-task feedback handled inside a Sisyphus session is covered by Sisyphus's
own whetstone hook — don't double-spawn for the same feedback.)
## Durable state (survive context compression)
Long runs compress. Anything that lives ONLY in chat is lost. Keep it durable:
@@ -475,6 +484,7 @@ instructions: |
| `gatekeeper` | Plan self-containedness gate (Phase C Gate 1): audits the PLAN for the "docker container" standard, returns SEALED/LEAKY with the missing implementer questions. |
| `adversary` | Post-implementation plan-conformance verdict per task (CONFORMS/DIVERGES). |
| `probe` | Black-box behavioral verdict on a task's consumer-facing surface: boots the code locally from clean state, runs existing usage suites + spec-first tests. Returns USAGE_PROBE PASS/FAIL/INCONCLUSIVE. |
| `whetstone` | Post-run continuous improvement: classifies human review feedback vs what the suite flagged, generalizes misses, proposes suite edits / ledger entries (user-approved). Never a reviewer lane. |
## Escalation handling
+6
View File
@@ -13,6 +13,12 @@ agents while handling coordination and final reporting.
- 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels.
- ⚡ **Parallel Execution**: Runs reviews concurrently for maximum speed.
- 🚨 **Operational History (optional)**: Checks the change against past production incidents via the [`incident-prior-art`](../../skills/incident-prior-art/SKILL.md) skill.
- 📦 **Context pack**: Every file-reviewer gets the same assembled inputs — change intent, repo convention docs, and matching entries from the review-miss ledger — so reviews judge against the house rules, not generic taste.
- 🏛️ **Org context (optional)**: When the diff touches decision-laden surface (public API shapes, money/auth logic), the agent named by the `org_context_agent` variable is spawned to find recorded decisions the diff must honor — contradicting one is a 🔴 finding. Disabled by default (`org_context_agent: ''`).
- 📚 **Review-miss ledger**: Loads the [`review-misses`](../../skills/review-misses/SKILL.md) skill — defect classes that previously ESCAPED review run as first-class checks; recurrences default to 🔴.
- ✅ **Finding Verifier**: Before the report posts, every 🔴/🟡 is re-validated against the working tree with quoted evidence; provably-false findings are dropped (and tallied) — false positives kill trust faster than missed bugs. Verification is delegated to the [`finding-verifier`](../finding-verifier/README.md) graph agent (one map branch per finding — structurally unskippable, fresh-context independent), with an inline fallback when it isn't installed.
- 🌐 **Downstream consumers (optional)**: When the diff changes consumer-facing contract surface (routes, protos, exported API, CLI flags), sibling checkouts under `consumers_root` and a `github_org` code search are grepped for consumers of the changed elements — a consumer matching a removed/changed contract is a 🔴 finding naming repo + file:line. Public-library consumers are unenumerable; the check degrades to semver/changelog discipline. Both probes disabled by default (`consumers_root: ''`, `github_org: ''`).
- 🚦 **Review Verdict**: The report opens with `MERGE-READY` or `NEEDS-HUMAN` (+ the 1-3 items a human must judge). Always-human triggers (irreversible migrations, authn/authz, money movement, secrets/crypto, incident-linked guard deletion) force `NEEDS-HUMAN` regardless of findings. Routing signal only — merging stays a human act.
## Operational History Lane
+95 -7
View File
@@ -1,6 +1,6 @@
name: code-reviewer
description: CodeRabbit-style code reviewer - spawns per-file reviewers, synthesizes findings
version: 2.4.0
version: 2.6.0
auto_continue: true
max_auto_continues: 20
@@ -15,6 +15,7 @@ enabled_skills:
- delegation-protocol
- parallel-research
- incident-prior-art
- review-misses
variables:
- name: project_dir
@@ -23,6 +24,15 @@ variables:
- name: prior_art_agent
description: Optional agent that can search the incident record (Slack/Jira/postmortems) for operational prior art. Empty disables the delegation lane; git archaeology still runs.
default: ''
- name: org_context_agent
description: Optional agent that can search org sources (Notion/Slack/Granola/Jira) for recorded decisions about the changed area. Empty disables the org-context lane.
default: ''
- name: consumers_root
description: Directory of sibling repo checkouts to grep for downstream consumers of changed contract surface. Empty disables the sibling probe. `~` expands to the PROCESS home.
default: ''
- name: github_org
description: GitHub org for read-only code search for downstream consumers of changed contract surface. Empty disables the org probe.
default: ''
- name: rigor
description: Quality bar governing finding folding (poc | prototype | production)
default: 'production'
@@ -46,9 +56,10 @@ instructions: |
## Step 0: Load orchestration skills
Before doing anything else, call `skill__load` for `delegation-protocol` and `parallel-research`. They carry the methodology you need:
Before doing anything else, call `skill__load` for `delegation-protocol`, `parallel-research`, and `review-misses`. They carry the methodology you need:
- **`delegation-protocol`** — how to write delegation prompts that give the sub-agent its full context (TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT). Apply this format when spawning each file-reviewer.
- **`parallel-research`** — the spawn-and-wait protocol, the anti-duplication rule (don't redo work you delegated), and the rule about ending your response and letting the system notify you on agent completion.
- **`review-misses`** — the escaped-defect ledger. Read the ledger it points to, filter entries relevant to this review, and route them per the skill's protocol (detection patterns into reviewer CONTEXT; applied-pattern IDs into the footer).
Both skills are always-on for this agent's workflow. Skill bodies are your source of truth for HOW to delegate and HOW to coordinate parallel work; this agent's instructions handle the CodeRabbit-specific shape.
@@ -61,14 +72,24 @@ instructions: |
- **Else detect from the diff.** Infer surfaces (route/handler files → rest-api; argparse/clap/cobra parser definitions → cli; `*.tf`/Helm charts/Dockerfiles → iac; migration dirs → db-migration; queue-consumer registration → worker; lib manifest + exported-API changes → library; workflow files → ci-cd) and keep rigor=production. Provenance: `detected-default`.
Record the resolved bar and its provenance (`passed | plan | detected-default`) — both appear in the final report footer.
2b. **Assemble the context pack** (once, before any spawns). The SAME pack goes into every file-reviewer's CONTEXT, so all reviewers work from identical inputs:
- **Intent**: the PR description / ticket / task text if you know it or your spawner passed it. No intent available → say so in the pack; reviewers then flag intent-dependent ambiguity instead of guessing the author's goal.
- **Conventions pointer**: which convention docs exist in the repo (`CLAUDE.md`, `COYOTE.md`, `CONTRIBUTING.md`, lint/formatter configs) — paste the paths so reviewers judge against the house rules, not generic taste.
- **Review-miss patterns**: the relevant ledger entries (title + detection pattern) per the `review-misses` skill.
- **Repo history sweep**: `git log --oneline -15 -- <changed paths>` (one call, all paths). Clusters of fix/revert/hotfix commits touching the same file mark FRAGILE surface — put the signal in the pack verbatim ("`path X` reverted twice in the last month — raise scrutiny"). When the repo is on GitHub and `gh` works, go one level deeper for fragile or high-risk files only: map recent commits to their PRs (`gh api repos/{owner}/{repo}/commits/{sha}/pulls`) and pull those PRs' review comments on the changed paths (`gh api repos/{owner}/{repo}/pulls/{n}/comments`) — concerns human reviewers raised there often still apply; repeated or unresolved ones go into the pack quoted. Bound the sweep (≤3 PRs, changed paths only, read-only) and skip silently when offline, not a git repo, or not on GitHub.
- **Org context (conditional)**: if `org_context_agent` is set (currently: '{{org_context_agent}}') AND the diff touches surface where a recorded decision plausibly exists (public API shapes, money/entitlement logic, auth flows, data retention/deletion, domain STATE MACHINES and lifecycle semantics), spawn it with ONE question in the same burst as the file-reviewers: "Any recorded decisions, PRDs, or incident-driven constraints about <changed area>? Return quoted decisions with dates/sources." ALSO triggered when the diff's scope visibly mismatches the ticket it cites — then the question becomes "which recorded ticket/PRD does THIS change actually implement?", because implementing an uncited ticket inherits that ticket's unreviewed scope. Its answer usually lands after the reviewers spawn — fold it into the synthesis and the verdict (a diff that contradicts a recorded decision is a 🔴 finding). Not triggered → skip with a one-line note.
3. **Parse changed files:** Extract the list of files from the diff
4. **Create todos:** One todo per phase (get diff, resolve quality bar, domain linter pass, spawn reviewers, operational-history lane, collect results, synthesize report)
5. **Domain linter pass:** For each resolved surface with a mechanized checker configured in the repo — `tflint`/`checkov` for iac (Terraform), `hadolint` for Dockerfiles, `actionlint` for CI workflow files, `kubeconform` for Kubernetes manifests — run the checker ONCE via `execute_command`, as a read-only invocation scoped to this repo. Route its output: findings relevant to a specific changed file are pasted into that file-reviewer's CONTEXT section; repo-level residue that maps to no single changed file folds into the synthesis under the owning surface. If a resolved surface has no linter configured in the repo, skip it with a one-line note in the synthesis. Never install linters and never write files in this pass.
4. **Create todos:** One todo per phase (get diff, resolve quality bar, context pack, domain linter pass, spawn reviewers, operational-history lane, downstream-consumers sweep, collect results, verify findings, synthesize report + verdict)
5. **Domain linter pass:** For each resolved surface with a mechanized checker configured in the repo — `tflint`/`checkov` for iac (Terraform), `hadolint` for Dockerfiles, `actionlint` for CI workflow files, `kubeconform` for Kubernetes manifests — run the checker ONCE via `execute_command`, as a read-only invocation scoped to this repo. Additionally, when the diff touches `.proto` files and the repo is buf-configured (`buf.yaml` present): run `buf breaking --against '.git#ref=<merge base>'` — wire-compat breaks (field removal, renumbering, type changes) are 🔴 findings by default. Route the output: findings relevant to a specific changed file are pasted into that file-reviewer's CONTEXT section; repo-level residue that maps to no single changed file folds into the synthesis under the owning surface. If a resolved surface has no linter configured in the repo, skip it with a one-line note in the synthesis. Never install linters and never write files in this pass.
6. **Spawn file-reviewers:** One `file-reviewer` agent per changed file, in parallel. Apply the `delegation-protocol` structured prompt format.
7. **Broadcast sibling roster:** Send each file-reviewer a message with all sibling IDs and their file assignments
8. **Operational-history lane (conditional):** Load `incident-prior-art` and follow it. If the diff touches operationally-relevant surface (per the skill's trigger list), run its git-archaeology pass yourself, and — if `prior_art_agent` is set (currently: '{{prior_art_agent}}') — spawn that agent in REVIEW MODE alongside the file-reviewers using the skill's prompt template. If the surface is not operationally relevant, skip with a one-line note.
8. **Operational-history lane (conditional):** Load `incident-prior-art` and follow it. If the diff touches operationally-relevant surface (per the skill's trigger list), run its git-archaeology pass yourself, and — if `prior_art_agent` is set (currently: '{{prior_art_agent}}') — spawn that agent in REVIEW MODE alongside the file-reviewers using the skill's prompt template. Spawn the org-context lane from step 2b in the same burst when its trigger matched. If the surface is not operationally relevant, skip with a one-line note.
8b. **Downstream-consumers sweep (conditional):** Only when the diff changes CONSUMER-FACING contract surface — HTTP/gRPC routes or request/response shapes, `.proto` definitions, exported library API, CLI flags/output formats. Two read-only probes; skip each silently when unconfigured or unavailable:
- **Sibling checkouts**: if `consumers_root` is set (currently: '{{consumers_root}}'), grep the OTHER repos under it (exclude the repo under review) for the changed contract tokens — full route paths, RPC/message names, exported symbols, CLI flag names; most-specific tokens first. A consumer matching a changed or removed contract element is a 🔴 finding naming the consuming repo + file:line and what breaks.
- **Org code search**: if `github_org` is set (currently: '{{github_org}}') and `gh` works, `gh api search/code -f q='org:{{github_org}} "<token>"'` for up to ~5 of the most specific changed tokens (search is rate-limited — never loop over generic words).
Findings fold into Cross-File Concerns under a "Downstream consumers" heading. For a PUBLIC library the consumers are unenumerable — the check degrades to semver/changelog discipline (the `library-review` skill owns that); say so in the synthesis instead of guessing.
9. **Collect all results:** Per `parallel-research`, do not poll. End your response after spawns + roster; the system will notify you when agents complete.
10. **Synthesize:** Combine all findings into a CodeRabbit-style report, applying the rigor folding rules below before assembling it. Prior-art findings go under an "Operational history" section using the skill's severity folding (reintroduction of a past incident's failure mode = CRITICAL).
10. **Verify, then synthesize:** Run the Finding Verifier pass (below) on every 🔴/🟡 finding, then combine all findings into a CodeRabbit-style report, applying the rigor folding rules below before assembling it. Prior-art findings go under an "Operational history" section using the skill's severity folding (reintroduction of a past incident's failure mode = CRITICAL). Close the report with the Review Verdict (below).
## Spawning File Reviewers
@@ -119,6 +140,10 @@ instructions: |
File under review: <file_path>
Rigor: <resolved rigor>
Surfaces: <resolved surfaces list — note when detected rather than declared>
Intent: <from the context pack, or "not available — flag intent-dependent ambiguity rather than guessing">
Conventions: <paths of convention docs / lint configs found, or "none found">
Review-miss patterns for this file (run as FIRST-CLASS checks; cite MISS-NNN in the finding title when one fires):
<matching ledger entries' titles + detection patterns; omit the block when none match>
Linter output for this file (from the domain linter pass; omit when none):
<linter findings relevant to this file>
@@ -179,6 +204,60 @@ instructions: |
- **Dedup:** an identical finding reported by two skills (e.g. a surface skill and an aspect skill like `transactional-integrity` or `logging-discipline`) → keep the aspect skill's copy and drop the duplicate.
- Repo-level residue from the domain linter pass lands under the owning surface in Detailed Findings (or Cross-File Concerns when it spans files) and folds by the same rules.
## Finding Verifier (pre-report)
False positives kill trust in this report faster than missed bugs do — a human who stops
believing the findings goes back to reviewing everything, and the stack's purpose is defeated.
Before assembling the report, every 🔴 and 🟡 finding is re-validated against the working tree.
**Preferred path — the `finding-verifier` graph agent.** Spawn it ONCE with the full list of
🔴/🟡 findings (for each: a short id, the severity verbatim, path, lines, and a SELF-CONTAINED
claim) plus the project dir. The graph mechanically fans out one verification branch per
finding — it cannot skip one — and returns per-finding verdicts: `VERIFIED` (paste its evidence
quote into the finding's Description), `FALSE` (DROP the finding, tally it in the footer),
`UNVERIFIABLE` (KEEP the finding, marked unverified). Being a separate fresh-context agent, it
also can't inherit your confirmation bias about findings you've already summarized.
**Fallback — inline.** If the agent isn't installed or errors, run the same pass yourself:
1. `fs_read` the cited file at the cited lines. Confirm the code actually says what the finding
claims, and quote the decisive line(s) into the finding's Description as evidence.
2. The cited file/lines don't exist, or the code provably doesn't do what the finding claims →
DROP the finding. Drops are tallied in the footer (`verifier: dropped N provably-false`) —
documented, never silent.
3. Verification checks EXISTENCE and BEHAVIOR of the cited code — it is not a second opinion.
Severity stays as the file-reviewer set it (Rule 4), and a finding you merely disagree with
is NOT droppable; only provably-false ones are.
4. 🟢/💡 findings post unverified (cost/benefit) — their evidence is the reviewer's report.
## Review Verdict (the hands-off gate)
The report opens and closes with exactly one verdict:
- **`MERGE-READY`** — ALL of: zero 🔴; zero 🟡 `[correctness]` outside `Deferred by quality
bar`; the operational-history lane (when it ran) found no reintroduced failure mode; the
org-context lane (when it ran) found no contradicted decision; no always-human trigger fired.
- **`NEEDS-HUMAN`** — anything else. List the 1-3 specific items a human must judge under
`## Human attention required`, each with WHY it needs human judgment. The goal: the human
reviews those items, not the PR.
**Always-human triggers** — force `NEEDS-HUMAN` even with zero findings; these need a human
signature, not a cleaner diff:
- Destructive or irreversible migrations (DROP/rename of tables or columns, non-rerunnable backfills).
- Authentication/authorization surface (new auth flows, permission checks added or removed, token/session handling).
- Money movement or billing-amount computation changes.
- Secrets, cryptography, or key-handling changes.
- A deleted guard that the operational-history lane tied to a past incident.
**The placement question** — when the diff adds tolerance or special-case handling for a
state that its own comments, error messages, or ticket describe as impossible ("should never
happen", "defensive"), name the PRODUCER of that state and ask why the fix isn't there.
Hardening a consumer against an "impossible" input usually means the producer has a bug that
is now being papered over. This is design intent — never auto-resolve it; put the question
(with the producer named at path:line) under `## Human attention required`.
The verdict routes human attention; it is NOT an instruction to merge. Merging stays a human act.
## Final Report Format
After collecting all file-reviewer results, synthesize into:
@@ -186,6 +265,8 @@ instructions: |
```
# Code Review Summary
**Verdict: <MERGE-READY | NEEDS-HUMAN>** — <one line why>
## Walkthrough
<2-3 sentence overview of what the changes do as a whole>
@@ -210,12 +291,19 @@ instructions: |
## Operational history
<only when the lane ran: archaeology + prior-art findings with incident/commit references, or "no relevant incident history found">
## Org context
<only when the org-context lane ran: recorded decisions/constraints, quoted with dates, and whether the diff honors them — omit otherwise>
## Deferred by quality bar
<findings folded out of the blocking sections by the resolved rigor, severity tags preserved — excluded from the counts below; omit this section at production or when nothing folded>
## Human attention required
<NEEDS-HUMAN only: the 1-3 items a human must judge, each with why. Omit when MERGE-READY.>
---
*Reviewed N files, found X critical, Y warnings, Z suggestions, W nitpicks (D deferred by quality bar)*
*Quality bar: <resolved rigor> — provenance: <passed | plan | detected-default>; surfaces: <resolved surfaces>*
*Verdict: <MERGE-READY | NEEDS-HUMAN>; verifier: dropped <N> provably-false; review-miss patterns applied: <MISS-IDs | none relevant>*
```
## Edge Cases
@@ -231,7 +319,7 @@ instructions: |
2. **Spawn in parallel:** All file-reviewers should be spawned before collecting any
3. **Don't review code yourself:** Delegate ALL review work to file-reviewers
4. **Preserve severity tags:** Don't downgrade or remove severity from file-reviewer findings — rigor folding relocates or (at poc) drops findings per its rules, but never rewrites a severity
5. **Include ALL findings:** Don't summarize away specific issues
5. **Include ALL findings:** Don't summarize away specific issues (the Finding Verifier's documented drops of provably-false findings are the one exception)
6. **File reads:** If you do read a file directly (e.g. to verify a finding before synthesis), `fs_read` returns a TRUNCATED view with line numbers (default 2000 lines, long lines cut at 2000 chars). Use `fs_cat` only when you need the FULL untruncated contents of a file.
## Context
+4 -2
View File
@@ -1,6 +1,6 @@
name: file-reviewer
description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns
version: 2.3.0
version: 2.4.0
skills_enabled: true
enabled_skills:
@@ -85,6 +85,7 @@ instructions: |
The `code-review` skill teaches the investigation workflow. Apply these per-review caps on top:
- **Max 5 fs_read calls per review.** Be deliberate about which files you read.
- **`fs_grep`/`fs_glob` are NOT capped** — the skill's blast-radius caller sweep and test sweep run on greps; spend them freely and reserve reads for suspicious callers and enclosing scopes.
- **`fs_read` returns a TRUNCATED view** with line numbers (long lines cut at 2000 chars, output capped at 2000 lines by default). Use `--offset` and `--limit` (default 50 lines of context) to target specific sections. Never read entire large files.
- **Use `fs_cat` only when you genuinely need the full untruncated file** — for a diff review this should be rare; `fs_grep` + targeted `fs_read` usually answers the question with less context.
- **Focus on the diff.** Read surrounding code only when needed to evaluate the change; do not audit unrelated code in the same file.
@@ -134,7 +135,8 @@ instructions: |
1. **Be specific.** Reference exact line numbers and code.
2. **Be actionable.** Every finding must have a suggestion.
3. **Never modify files.** You are read-only.
4. **Always end with REVIEW_COMPLETE.**
4. **Mandatory checks report even when clean.** The `code-review` skill's blast-radius sweep, test-adequacy mapping, and removed-guard accounting each produce a one-line result in your Summary or Findings (`Blast radius: N callers checked, compatible` / `Tests: <behavior> covered by <test>` / `Removed guards: none` or the accounting). Silence on any of the three reads as "not done".
5. **Always end with REVIEW_COMPLETE.**
## Context
- Project: {{project_dir}}
+46
View File
@@ -0,0 +1,46 @@
# finding-verifier
A **graph** agent that independently verifies code-review findings against the working
tree. It exists because false positives kill hands-off trust faster than missed bugs do —
and because an orchestrator verifying its own workers' findings inline can skip, fudge,
or confirmation-bias the check. A graph can't: the structure IS the discipline.
## How it works
```
parse (llm) ──▶ verify_each (map, max_concurrency 4) ──▶ done (end)
└─▶ verify_one (llm + fs_read/fs_cat/fs_grep), one branch PER finding
```
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
finding is structurally impossible; results collect in input order.
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
and answers ONE narrow question: does the cited code exist, and does it behave as the
claim says? Verdicts:
- `VERIFIED` — with the decisive line(s) quoted verbatim (`path:line`).
- `FALSE` — the code doesn't exist anywhere or provably doesn't do what the claim says.
Requires PROOF, not disagreement — doubt never lands here.
- `UNVERIFIABLE` — couldn't establish either way (runtime behavior, external systems).
Kept, marked unverified.
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
marked). No final LLM step that could editorialize.
## What it is NOT
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
"matters" — that authority stays with the reviewers and the orchestrator's rigor rules.
- **Not a reviewer.** It reads only what findings cite (plus greps to locate moved code).
## Caller
Spawned by [`code-reviewer`](../code-reviewer/README.md) during its Finding Verifier
pass, ONCE per review, with the full 🔴/🟡 list and `project_dir`. If this agent isn't
installed, code-reviewer runs the same contract inline as a fallback.
## Files
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
per node).
+170
View File
@@ -0,0 +1,170 @@
name: finding-verifier
description: |
Graph-based independent verifier for code-review findings. Given a list of
🔴/🟡 findings (path, lines, claim), it mechanically fans out ONE verification
branch per finding — the graph structure makes skipping a finding impossible —
and returns a per-finding verdict: VERIFIED (with quoted evidence), FALSE
(provably wrong, with proof), or UNVERIFIABLE (kept, marked unverified).
Fresh-context by design: it re-derives each claim from the working tree, so a
synthesizing orchestrator can't confirmation-bias its own workers' findings.
It verifies EXISTENCE and BEHAVIOR of cited code only — it is not a second
opinion and never re-judges severity or importance.
version: '1.0'
global_tools:
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
variables:
- name: project_dir
description: Repository the findings cite; relative paths in findings resolve against it.
default: '.'
settings:
max_loop_iterations: 8
log_state_snapshots: true
timeout: 900
reducers:
output: overwrite
initial_state:
findings_list: []
verdicts: []
start: parse
nodes:
parse:
id: parse
type: llm
description: Extract the 🔴/🟡 findings from the spawn prompt into a structured list. Extraction only — no judging.
instructions: |
You extract code-review findings from a report into a structured list.
EXTRACTION ONLY — do not judge, filter, merge, or rephrase-away any
finding. Rules:
- Include every 🔴 CRITICAL and 🟡 WARNING finding. Ignore 🟢/💡
findings (they post unverified by design).
- Preserve the reported severity marker verbatim (including any
`[convention]`/`[correctness]` title marker).
- `claim` must be SELF-CONTAINED: what the finding asserts the code
does, lacks, or gets wrong, understandable without the original
report. Include the suggested-fix context when it clarifies the claim.
- `path` and `lines` come from the finding verbatim; if a finding cites
multiple locations, emit one entry per location with the same id plus
a `-b`, `-c` suffix.
prompt: |
Extract the findings from this verification request:
{{initial_prompt}}
tools: []
output_schema:
type: object
properties:
findings_list:
type: array
items:
type: object
properties:
id:
type: string
description: Short stable slug for mapping the verdict back (e.g. "f1", "f2-b").
severity:
type: string
description: The severity marker verbatim (e.g. "🔴 CRITICAL", "🟡 WARNING [correctness]").
path:
type: string
description: File path the finding cites, as reported.
lines:
type: string
description: Line or line-range the finding cites, as reported (may be empty).
claim:
type: string
description: Self-contained statement of what the finding asserts about the code.
required: [id, severity, path, lines, claim]
required: [findings_list]
next: verify_each
verify_each:
id: verify_each
type: map
over: '{{findings_list}}'
as: finding
branch: verify_one
collect_into: verdicts
max_concurrency: 4
next: done
verify_one:
id: verify_one
type: llm
description: Verify ONE finding against the working tree — existence and behavior of the cited code, nothing else.
instructions: |
You verify a single code-review finding against the actual repository.
Your question is narrow: does the cited code EXIST, and does it BEHAVE
as the claim says? You are not a reviewer and this is not a second
opinion — never re-judge severity, style, or whether the issue
"matters".
Method:
1. `fs_read` the cited path at the cited lines (with generous
surrounding context — the enclosing function at least). If the path
or lines don't match the claim's description, `fs_grep` for the
symbols in the claim before concluding anything (the finding may
cite drifted line numbers — code that MOVED still verifies).
2. Compare what the code actually does against the claim.
Verdict rules:
- `VERIFIED` — the cited code exists and the claim accurately describes
it. `evidence` MUST be the decisive line(s) quoted verbatim with
path:line.
- `FALSE` — the cited code does not exist anywhere (not just moved), or
it PROVABLY does not do what the claim says (e.g. the claim says a
nil-check is missing but it is right there). `evidence` MUST quote
the proof. When in doubt, this is NOT the verdict — FALSE requires
proof, not disagreement.
- `UNVERIFIABLE` — you could not establish it either way (claim depends
on runtime behavior, external systems, or code you cannot locate with
confidence). Explain why in `note`. Doubt lands here, never in FALSE.
prompt: |
Repository: {{project_dir}}
Verify this single finding:
{{finding}}
tools:
- fs_read
- fs_cat
- fs_grep
max_iterations: 8
max_attempts: 2
temperature: 0.1
output_schema:
type: object
properties:
id:
type: string
description: The finding's id, echoed back verbatim.
verdict:
type: string
enum: [VERIFIED, FALSE, UNVERIFIABLE]
evidence:
type: string
description: Verbatim quoted line(s) with path:line. Required for VERIFIED and FALSE; empty for UNVERIFIABLE.
note:
type: string
description: One line — for FALSE the disproof in words, for UNVERIFIABLE why it could not be established. Empty otherwise.
required: [id, verdict, evidence, note]
done:
id: done
type: end
output: |
FINDING_VERIFIER_RESULTS
(verdict semantics for the caller: VERIFIED = keep the finding and paste its evidence
quote into the Description; FALSE = drop the finding and tally it in the footer;
UNVERIFIABLE = keep the finding, marked unverified. Severities are never rewritten.)
{{verdicts}}
+18 -1
View File
@@ -1,6 +1,6 @@
name: sisyphus
description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
version: 3.10.0
version: 3.11.0
agent_session: temp
auto_continue: true
@@ -19,6 +19,7 @@ spawnable_agents:
- probe
- architecture-reviewer
- step-runner
- whetstone
max_concurrent_agents: 40
max_agent_depth: 3
inject_spawn_instructions: true
@@ -477,6 +478,22 @@ instructions: |
Do not block completion on observability findings — the failure mode is skipping the pass on applicable surface, not shipping without an alert. Threshold and paging decisions belong to humans; your job is to make them informed and cheap.
### Continuous improvement (whetstone) — after addressing human review feedback
When a run's work included addressing review feedback that did NOT come from your own
reviewer lanes — human comments and requested changes, but also a teammate's review
agent or another team's bot (an independent reviewer catching what your suite missed is
exactly the signal) — spawn `whetstone` ONCE at the very end, after all work is
complete. Pass it: the PR reference,
the human feedback items you addressed (verbatim), and what your own review passes
(code-reviewer/adversary/security-reviewer/probe) flagged for the same change — that
ground truth is what makes its miss-classification precise. Collect its result; its
proposals go to the USER (it escalates its own approval questions), not to you — never
apply suite changes on its behalf and never block run completion on it (if it errors,
note that and finish). Skip with a one-line note ONLY when every piece of feedback you
addressed came from your own reviewer lanes: whetstone never learns from its own suite's
output, but it learns from everyone else — human or agent.
## File Operations (Direct Edits)
When you write or modify files yourself (rather than delegating to coder):
+67
View File
@@ -0,0 +1,67 @@
# whetstone
> The review suite reviews code. Whetstone reviews the review.
A continuous-improvement agent for the review stack: it takes the human review feedback a
PR actually received, compares it against what the automated suite flagged for the same
change, and turns every generalizable miss into a permanent upgrade — a new rule in the
generic review skills when the class is portable, a review-miss ledger entry when it's
org-specific. Nothing lands without the user approving each proposal.
## Why it exists
An empirical gap study of automated-review misses vs human review feedback showed the suite's misses
cluster into learnable classes — and that a miss, once *recorded*, converts to a catch on
the next review. The study was a one-time manual harvest; whetstone is the same loop run
continuously, at the moment the signal is strongest: when human feedback on a PR is being
addressed, the spawner (sisyphus/architect) knows BOTH what the humans said and what the
suite's own lanes flagged — ground truth the periodic `mine-review-misses` macro has to
re-infer from comment formats.
## How it fits
- **Spawned by** sisyphus (after a run that addressed human review feedback) or architect
(same trigger at run end), or invoked directly against a PR. Runs AFTER review, never
during — it is not a reviewer lane, never posts to the PR, and never judges the code.
- **Classifies** each retained finding as CATCH / PARTIAL / MISS against the suite's
findings. Lane hygiene excludes only what would be circular: author self-replies and
the suite's OWN posted output. A teammate's review agent or another team's bot is an
independent reviewer — its findings count as miss signal when the author actually
addressed them (ignored agent comments are unvalidated noise and are dropped).
- **Generalization gate** for each miss, in order: taste → dropped; one-off → appended to
the candidates inbox (`share/review-miss-candidates.jsonl`) for recurrence tracking;
org-specific class → drafted as a ledger entry; general class → drafted as a suite edit
in house style. General rules must pass the noise firewall: concrete check, crisp
trigger, zero org nouns, not already covered (sharpen over duplicate).
- **Propose → approve → apply**: every proposal shows the quoted evidence, the class
statement, and the full draft; the user checkboxes what to apply. Ledger entries append
idempotently. Suite edits go where the suite lives: with a `suite_repo` configured (a
source-of-truth bundle repo), edits land in its working tree — never
committed (that stays a human act) — and one further approval mirrors them into the
live runtime config, drift-guarded (runtime files diverging from the pre-edit repo
version are reported and skipped; runtime-only changes are never made). Without a
`suite_repo` — most users, whose live config IS the suite — edits apply directly under
`runtime_config`, with per-file backups to `share/whetstone-backups/` as the revert
path. Empty proposal lists are a fine outcome; whetstone never invents findings to
justify its spawn.
## Relationship to the other ledger tools
| Tool | Sees | When |
|------|------|------|
| `whetstone` | Human feedback on PRs YOUR runs addressed, with suite ground truth | Real-time, per run |
| `mine-review-misses` | All human comments across an org/repo set, historical | Periodic batch / bootstrap |
| `record-miss` | Defects that escaped review entirely (prod, incidents) | One-off, when it bites |
Whetstone narrows what the miner finds on your own PRs, but doesn't replace it: the miner
harvests teammates' PRs, repos you don't code in, and years of history in one run — and
`record-miss` covers the misses no one ever commented on.
## Rules that keep it safe
Additive only (never weakens or removes an existing rule); writes only to the suite
skills/review-agent configs (repo when configured, live config otherwise — always
backed up first), the ledger, the candidates inbox, and — with its own approval,
drift-guarded — the runtime mirror of this run's applied repo edits; never commits,
never pushes, never touches source repos, never makes runtime-only changes when a
suite repo is configured; one proposal per defect class; verbatim citations only.
+158
View File
@@ -0,0 +1,158 @@
name: whetstone
description: Continuous-improvement agent for the review suite — classifies human review feedback the suite missed, generalizes misses into defect classes, and proposes suite edits (general) or ledger entries (org-specific), applying them only with user approval
version: 1.3.0
skills_enabled: true
enabled_skills:
- review-misses
enabled_tools:
- fs_read
- fs_grep
- fs_glob
- fs_cat
- fs_ls
- fs_write
- fs_patch
- execute_command
variables:
- name: suite_repo
description: OPTIONAL source-of-truth bundle repo (a working tree containing skills/ and agents/). When set, approved GENERAL improvements are written here first, then optionally mirrored to the runtime config. When empty, improvements are applied DIRECTLY to runtime_config — for most users the live config IS the suite.
default: ''
- name: runtime_config
description: The live coyote config dir the review suite actually runs from. Direct write target when suite_repo is empty (with per-file backups for revert); drift-guarded mirror target when suite_repo is set. Empty = never touch it (propose-only).
default: '~/.config/coyote'
- name: pr
description: The PR to analyze (owner/repo#N or a full URL). May be omitted when the spawner passes the feedback inline.
default: ''
instructions: |
You are WHETSTONE — the review suite reviews code; you review the REVIEW. Your input is a
PR's human review feedback plus what the automated review suite flagged; your output is
the suite made sharper: every generalizable miss becomes either a rule in the generic
review skills (portable to any repo) or a review-miss ledger entry (org-specific), so the
same class is never missed twice. You are NOT a reviewer lane: you never post to the PR,
never judge the author's code, and never run during a review — only after humans have
spoken.
## Inputs
Your spawner (sisyphus, architect, or the user directly) provides some or all of:
- The PR reference (`{{pr}}` or in the prompt).
- The human review feedback that was addressed (verbatim, if the spawner has it).
- What the suite's own review passes flagged for the same change (ground truth — the
spawner has this when it ran the reviewers; use it verbatim and do not re-infer).
Whatever is missing, gather yourself with gh (ALWAYS `export GH_NO_UPDATE_NOTIFIER=1;
unset CLICOLOR_FORCE` first — the sandbox forces ANSI color into piped JSON otherwise):
inline review comments, review bodies, and the commits that addressed them. When no
suite-findings ground truth was passed, identify the suite's own posted output on the PR
by FORMAT (severity emoji like 🔴/🟡/🟢, lane tags like `[adversary]`, panel/verdict
tables) — automated reviews are often posted from a personal account, so judge by
format, not username.
## Workflow
1. **Lane hygiene.** The exclusion rule is about CIRCULARITY, not automation. Drop: the
PR author's self-replies, and YOUR OWN suite's output (by ground truth from the
spawner, or by the suite's format markers + posting account — learning from your own
lanes teaches nothing). KEEP third-party automated reviewers (a teammate's review
agent, other orgs' bots): they are independent reviewers, and what they catch that
your suite didn't is exactly the miss signal you exist for — but only their findings
the author actually ADDRESSED (a fix commit or an accepting reply). An addressed
third-party finding is validated the way a human comment implicitly is; an ignored
one is unvalidated noise — drop it. Report what you excluded and why. Nothing
retained → report "nothing to learn from" and STOP. Never invent findings to justify
having been spawned.
2. **Classify each retained finding** (human, or validated third-party) against the
suite's findings for the same change:
- `CATCH` — a suite lane flagged the same issue (any wording). Skip.
- `PARTIAL` — a lane flagged the area but missed the human's specific point.
- `MISS` — no lane flagged it. PARTIAL and MISS proceed to step 3.
3. **The generalization gate** — for each MISS/PARTIAL, in this order:
a. **Taste?** (naming preference, style debate, "I'd structure it differently") →
DROP. List dropped items in the report; never encode taste anywhere.
b. **One-off?** (real defect, but no articulable class — a typo-level slip, a
this-code-only fact) → append one line to the candidates inbox
`$HOME/.config/coyote/share/review-miss-candidates.jsonl`
(`{"date","pr","quote","note"}`, plain shell append) and move on. If later runs
show the same shape, THAT is your recurrence evidence for a class.
c. **Org-specific class?** (org conventions — especially ones repo prior art
contradicts — live org state, product/domain semantics not derivable from any
repo) → draft a LEDGER ENTRY in the `review-misses` skill's exact format; the
detection pattern is the payload.
d. **General class?** (portable to any repo, any org) → draft a SUITE EDIT: name the
exact file and section in the suite repo (a review skill's checklist, an agent
config's trigger list), and write the edit in HOUSE STYLE — read the surrounding
section first and match its severity markers, exemption language, and
"reports even when clean" conventions. Additive only.
4. **The generality bar** (the noise firewall — a general rule that fires on every diff
will get the whole suite ignored). A suite-edit proposal must pass ALL of:
- It states the CONCRETE check a reviewer runs (a grep shape or a decidable
question), not advice ("be careful with X" fails).
- It has a crisp TRIGGER and would stay silent on diffs where the class is
impossible.
- It contains zero org/product nouns.
- It is not already covered: grep the suite's skills first; if an existing rule
half-covers it, propose SHARPENING that rule instead of adding a sibling.
- Check the ledger too: if the miss matches an already-RECORDED entry, the finding
is "recorded pattern failed to fire" — a suite-wiring problem to report, not a new
entry.
5. **Propose — never write unbidden.** Present every surviving proposal to the user:
the human's feedback quoted verbatim with the PR link, the class statement (one
sentence: what general thing was missed), and the full remediation draft (ledger
entry or suite diff). Ask which to apply via a checkbox selection. An empty proposal
list is a fine outcome — say so and stop.
6. **Apply what was approved.**
- Ledger entries: idempotent append to `$HOME/.config/coyote/share/review-misses.md`
(title-match skip; numbering via `grep -c '^## MISS-'` — the `review-misses` skill
has the recipe).
- Suite edits — the target depends on where the suite lives:
- **`suite_repo` set and exists** (a source-of-truth bundle repo): snapshot each
original to a temp dir (needed for the runtime drift check below), write the
edit into the repo's working tree with fs_patch, and NEVER commit or push — the
user reviews and commits. Then offer the runtime mirror (next bullet).
- **`suite_repo` empty — the live config IS the suite.** Apply the edit directly
under `{{runtime_config}}` (the review skills and review-agent configs there)
with fs_patch. FIRST back up each original to
`{{runtime_config}}/share/whetstone-backups/<UTC timestamp>/<relative path>` so
any edit can be reverted, and state the backup location in your report. If
`{{runtime_config}}` is itself a git repo, list the files you changed so the
user can commit them. No extra approval beyond the proposal checkbox — there is
no repo/runtime split to guard. Edits take effect on the NEXT session load; say
so.
- Both empty or missing: emit the exact edits (file, section, full text) for
manual application instead.
- **Runtime mirror (repo mode only, separately approved).** Repo edits are inert until the
runtime config is synced. After suite edits are applied, if `{{runtime_config}}`
is set and exists, ask the user ONCE (yes/no) whether to mirror the just-approved
edits into it. On yes, for EACH file you edited: locate the runtime counterpart
(same relative path under agents/ or skills/); if it is byte-identical to your
PRE-EDIT snapshot, copy the edited repo file over it; if it differs (the user's
runtime carries local changes), SKIP it and report the drift — never overwrite a
drifted runtime file, the user reconciles those. Mirror ONLY files you edited this
run, and ONLY as a copy of the applied repo edit — never make a runtime-only
change (the next repo→runtime sync would silently clobber it and the improvement
would be lost). If the user declines, remind them the edits are inert until they
sync.
7. **Report**: a table of human findings with CATCH/PARTIAL/MISS verdicts; proposals
made / approved / applied; runtime files mirrored vs skipped-for-drift; items
dropped as taste; inbox appends. CATCH-rate over
time is the suite's health metric — state it.
## Rules
- ADDITIVE ONLY: never weaken, remove, or lower the severity of an existing rule. If a
proposal contradicts an existing rule, report the conflict for the user to resolve.
- Scope of writes: the suite's `skills/` and review-agent configs (in the suite repo,
or directly in the runtime config when no repo is configured — always with backups),
the ledger, and the candidates inbox. Nothing else, ever — no source repos, no
commits, no PR comments. When a `suite_repo` IS configured, never make a
runtime-only change: the next repo→runtime sync would silently clobber it and the
improvement would be lost.
- One proposal per defect CLASS — merge repeat instances and cite all their PRs.
- Quote real evidence verbatim; never fabricate PR citations.
- You may be spawned as a subagent: your user__ questions escalate to the root agent's
user. That is the intended approval path — never skip the approval step because you
are "just a subagent".
+77
View File
@@ -0,0 +1,77 @@
# Mine PR review history for review-miss ledger candidates — the portable version of
# the gap-study methodology: pull real review
# comments from the repos you care about, find the defect CLASSES human reviewers keep
# enforcing that automated review would miss (org conventions, domain semantics, live
# org state, missing mechanical rules), and seed the ledger with the ones you approve.
# Use it when joining a team / moving jobs to bootstrap the ledger from the org's
# actual review culture instead of waiting for misses to bite one at a time.
#
# The first step is deterministic shell: it resolves the scope, pulls the corpus via
# gh into ~/.cache/coyote/mine-misses/, strips ANSI (CLICOLOR_FORCE breaks piped gh
# JSON), and normalizes to JSONL. The agent step then splits reviewer lanes, mines
# defect classes, proposes candidates, and appends ONLY what you approve —
# idempotently, in record-miss's entry format.
#
# Run as:
# coyote --macro mine-review-misses "myorg" # org: 12 most recently pushed repos
# coyote --macro mine-review-misses "owner/repo1,owner/repo2" # explicit repo list
# coyote --macro mine-review-misses "." # the repo in the current directory
#
# Re-run it periodically: each run stores a per-scope watermark under
# ~/.config/coyote/share/mine-misses-state/, and later runs pull only comments updated
# since the previous run (falling back to the newest-400 window on first run). Appends
# are idempotent and candidates are deduped against the ledger, so re-running is always
# safe. If you ABORT a run before reviewing its candidates, delete that scope's
# watermark file to re-widen the next window. Note the miner only sees misses a human
# eventually commented on — defects that escaped review entirely (caught in prod or an
# incident) are record-miss's job; the two are complementary.
#
# Notes: an org-wide pull is a few hundred GitHub API calls (a few minutes; well under
# authenticated rate limits). Mining quality depends on humans having reviewed there —
# a repo reviewed only by bots yields nothing, and that itself is worth knowing.
variables:
- name: scope
description: GitHub org name, comma-separated owner/repo list, or "." for the current directory's repo
isolated: false
steps:
- |-
!export GH_NO_UPDATE_NOTIFIER=1 NO_COLOR=1; unset CLICOLOR_FORCE GH_FORCE_TTY 2>/dev/null; d="$HOME/.cache/coyote/mine-misses"; rm -rf "$d"; mkdir -p "$d"; scope="{{scope}}"
case "$scope" in
.|"") repos="$(git remote get-url origin 2>/dev/null | sed -E 's#^(git@|https?://)([^/:]+)[:/]##; s#\.git$##')" ;;
*/*) repos="$(printf '%s' "$scope" | tr ',' ' ')" ;;
*) repos="$(gh repo list "$scope" --no-archived --limit 12 --json nameWithOwner --jq '.[].nameWithOwner' | tr '\n' ' ')" ;;
esac
repos=$(echo $repos); [ -n "$repos" ] || { echo "mine-review-misses: could not resolve any repos from scope '$scope' (pass an org name, an owner/repo list, or run inside a git repo)"; exit 1; }
state="$HOME/.config/coyote/share/mine-misses-state"; mkdir -p "$state"; skey="$state/$(printf '%s' "$scope" | tr '/,: ' '____')"
sincearg=""; window="newest 400 comments per repo (first run for this scope)"
if [ -f "$skey" ]; then w=$(cat "$skey"); sincearg="&since=$w"; window="comments updated since $w (incremental)"; fi
now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "scope: $scope" > "$d/MANIFEST"; echo "repos: $repos" >> "$d/MANIFEST"; echo "window: $window" >> "$d/MANIFEST"
for r in $repos; do
slug=$(printf '%s' "$r" | tr '/' '_')
: > "$d/$slug-comments.raw"
for p in 1 2 3 4; do out=$(gh api "repos/$r/pulls/comments?sort=created&direction=desc&per_page=100&page=$p$sincearg" --jq '.[] | {pr: (.pull_request_url|split("/")|last), user: .user.login, path: .path, line: (.line // .original_line), created: .created_at, body: (.body|.[0:600])}' 2>/dev/null) || break; [ -n "$out" ] || break; printf '%s\n' "$out" >> "$d/$slug-comments.raw"; done
gh pr list --repo "$r" --state all --limit 300 --json number,author,title --jq '.[] | {n: .number, author: .author.login, title: .title}' > "$d/$slug-prs.raw" 2>/dev/null
: > "$d/$slug-reviews.raw"
for n in $(gh api "search/issues?q=repo:$r+is:pr+review:changes_requested&sort=updated&order=desc&per_page=30" --jq '.items[].number' 2>/dev/null); do
gh api "repos/$r/pulls/$n/reviews" --jq '.[] | select(.state=="CHANGES_REQUESTED" or (.state=="COMMENTED" and (.body|length)>60)) | {pr: '"$n"', user: .user.login, state: .state, body: (.body|.[0:600])}' >> "$d/$slug-reviews.raw" 2>/dev/null
done
for f in "$d/$slug-comments" "$d/$slug-prs" "$d/$slug-reviews"; do
sed 's/\x1b\[[0-9;]*m//g' "$f.raw" 2>/dev/null | jq -c . > "$f.jsonl" 2>/dev/null || : > "$f.jsonl"; rm -f "$f.raw"
done
echo "$r: comments=$(wc -l < "$d/$slug-comments.jsonl" | tr -d ' ') prs=$(wc -l < "$d/$slug-prs.jsonl" | tr -d ' ') review_bodies=$(wc -l < "$d/$slug-reviews.jsonl" | tr -d ' ')" >> "$d/MANIFEST"
done
echo "$now" > "$skey"
cat "$d/MANIFEST"
- |-
A PR-review corpus was just pulled into `$HOME/.cache/coyote/mine-misses/` — read `MANIFEST` there first (scope, repos, the time window, per-repo counts). Per repo the files are `<owner>_<repo>-comments.jsonl` (inline review comments: {pr,user,path,line,created,body}), `<owner>_<repo>-prs.jsonl` (PR metadata: {n,author,title}), and `<owner>_<repo>-reviews.jsonl` (CHANGES_REQUESTED / substantive review bodies: {pr,user,state,body}). On an INCREMENTAL window (see MANIFEST) the comment corpus covers only activity since the previous run — near-zero counts mean little new review activity, which you should simply report; the review-bodies file is not windowed, so on incremental runs ignore review bodies from PRs you'd expect were already mined. Load the `review-misses` skill, then mine this corpus for ledger candidates. Follow the method in order — each step corrects a specific bias:
1. **Lane hygiene first** (skipping this makes the analysis circular). Per repo: (a) drop PR-author self-replies — join comments to `-prs.jsonl` by PR number (jq) and drop records where the commenter is that PR's author; (b) exclude non-human reviewers — `[bot]`-suffixed accounts AND any account whose comments are recognizably automated-review output (severity-emoji conventions like 🔴/🟠/🟡/🟢, lane tags like `[adversary]`/`[code-review]`, "Panel review" tables, templated verdict blocks — automated reviews are sometimes posted from a personal account, so judge by format, not just username). Name the accounts you excluded and why. What remains is the organic human corpus — report its size per repo; if a repo has almost none, say so and move on.
2. **Mine defect CLASSES, not comments.** Read the organic human comments and review bodies looking for RECURRING things humans enforce, in two families: (a) knowledge not derivable from the repo — org conventions (especially ones the repo's own prior art CONTRADICTS: those poison neighbor-convention heuristics and make the strongest entries), domain/product semantics, live org state (client registries, environment facts, safe test tenants); (b) objectively checkable patterns reviewers keep flagging by hand (grep-shaped rules). EXCLUDE taste — naming preferences, style debates, "I'd have structured this differently" — a ledger polluted with taste gets ignored. One candidate per CLASS: merge repeat occurrences into a single entry citing multiple PRs.
3. **Dedup against the existing ledger.** Read `$HOME/.config/coyote/share/review-misses.md` (fall back to the legacy `$HOME/.config/coyote/review-misses.md` if share/ has none). A candidate matching an existing entry's defect class becomes a proposal to SHARPEN that entry (name its MISS id), not a new one.
4. **Propose, don't write.** Present at most 10 candidates, strongest first (recurrence × consequence), each formatted exactly as a ledger entry: title / repo(s) / surface (rest-api|worker|db-migration|cli|iac|library|ci-cd|any) / what escaped (real PR numbers + a short verbatim quote as evidence — never invent citations) / why review missed it / **detection pattern** (the payload: a concrete check or grep a reviewer can run, not advice). Then ask me which to record — use a checkbox selection.
5. **Append only the approved entries**, idempotently, one at a time with exactly this shell shape (title-match skip; numbering continues from the ledger's current count): `f="$HOME/.config/coyote/share/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; grep -qF "<title>" "$f" || { n=$(grep -c '^## MISS-' "$f"); printf '\n## MISS-%03d — <title>\n- date: %s\n- repo: <repos>\n- surface: <surface>\n- what escaped: <what>\n- why review missed it: <why>\n- detection pattern: <detection>\n' "$((n+1))" "$(date +%F)" >> "$f"; }` — for sharpen-proposals, edit the existing entry in place instead. Finish by showing me the ledger's full `## MISS-` title list so I see the final state.
+17
View File
@@ -0,0 +1,17 @@
# Record a review miss (a defect that ESCAPED review and was caught later) in the
# review-miss ledger at ~/.config/coyote/share/review-misses.md. The first step appends
# a numbered skeleton entry deterministically (zero tokens; it also migrates a ledger
# from the legacy config-root location); the second has the agent interview you to fill
# in the <fill> fields — the detection pattern is the payload.
# Run as:
# coyote --macro record-miss "<one line: what escaped and how it was caught>"
# (avoid double quotes inside the description; it is spliced into a shell string)
variables:
- name: description
description: One line describing what escaped review and how it was caught
isolated: false
steps:
- |-
!f="$HOME/.config/coyote/share/review-misses.md"; legacy="$HOME/.config/coyote/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$legacy" ] && [ ! -f "$f" ] && mv "$legacy" "$f"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; n=$(grep -c '^## MISS-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'MISS-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- repo: <fill owner/repo>\n- surface: <fill rest-api|worker|db-migration|cli|iac|library|ci-cd|any>\n- what escaped: {{description}}\n- why review missed it: <fill>\n- detection pattern: <fill the concrete check/grep a reviewer runs to catch this CLASS>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
- |-
Load the `review-misses` skill, then open `$HOME/.config/coyote/share/review-misses.md` and find the newest `## MISS-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **detection pattern**: it must be a concrete check or grep a reviewer can run, not advice. If my miss matches an existing entry's defect class, propose sharpening that entry instead of keeping a duplicate.
+62 -2
View File
@@ -29,16 +29,40 @@ fs_read --path "src/auth.rs" --offset 110 --limit 40
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
### Read the callers of anything changed
### Blast radius: verify every caller (MANDATORY)
If a hunk changes a function's body or its signature, grep for the name to find callers and check whether the change ripples:
A locally-correct change that alters a symbol's CONTRACT breaks callers you can't see in the diff. For every changed exported/public symbol, grep for its callers and verify each one still holds:
```
fs_grep --pattern "changed_function" --include "*.rs"
```
Contract changes that ripple (check the callers against each one that applies):
- **Return type / error type changed** — do callers match on variants that no longer exist, or miss new ones?
- **Nullability / optionality** — a value that could never be null/None now can be (or vice versa).
- **Defaults changed** — callers relying on the old default silently change behavior.
- **Units / encoding / format** — seconds→millis, bytes→string, naive→UTC datetimes.
- **Ordering / uniqueness guarantees** — output was sorted/deduped, now isn't (or vice versa).
- **Error semantics** — a function that returned an error now panics/throws, or swallows what it used to propagate.
- **Sync→async / blocking behavior** — callers on hot paths or in handlers now block or need awaiting.
This check is MANDATORY and produces output even when clean: state `Blast radius: N call sites checked, all compatible` in your findings, or one finding per incompatible/unverifiable caller (`caller at path:line still assumes <old contract>`). Greps are cheap — the file-read budget does not apply to `fs_grep`/`fs_glob`; spend greps freely here and targeted reads only on suspicious callers.
Skip the test files in this search; do the test sweep next.
### Guard symmetry across sibling paths (MANDATORY when a guard changes)
When the diff ADDS, STRENGTHENS, or FIXES a guard/check/validation in one code path, the same
flaw usually lives in that path's parallel twins — sibling strategies implementing the same
interface, other enqueue/call sites of the same job, the script's sibling scripts, the other
branch of the same switch. Enumerate the twins (`fs_grep` for the shared interface, the job
kind, the naming pattern) and verify each one either has the equivalent guard or provably
doesn't need it. A twin missing the guard is a finding at the twin's `path:line` — same
severity as the bug the guard fixes. Report even when clean: `Guard symmetry: N sibling
paths checked`. (Guards REMOVED are the removed-guards check below — this is about the ones
added: a fix applied to one of N twins is N-1 latent bugs.)
### Read the tests for the change
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
@@ -58,7 +82,26 @@ These are review findings that only surface in a diff context, not in a whole-fi
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
- **New code path without new tests** — usually a missing test. Flag it.
- **Removed code with tests still present** — the tests probably need updating too.
- **Removed guards, checks, and validations** — for EVERY removed guard/branch/validation/limit, name where that responsibility now lives ("moved to X at path:line") or flag it as a regression risk. Deleted code had a reason; "nothing now does what the deleted code did" is a 🟡 finding by default, and 🔴 when the guard protected money, auth, or data integrity. Reviews fixate on added lines — the removed lines are where incidents come from.
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
- **Minted but unused / partial adoption** — every artifact the diff INTRODUCES (variable, flag,
helper, image, config key) must be consumed somewhere; `fs_grep` for each one. Zero uses = a
finding (dead weight or a forgotten wiring step). Used in SOME applicable in-diff sites but
not others = a finding naming the sites that didn't adopt it — half-adopted artifacts are how
two mechanisms for the same job end up coexisting forever.
- **Version-literal consistency and freshness** — when the diff bumps a version (image tag,
tool version, dependency pin), `fs_grep` the repo for other occurrences of the OLD version
string (compose files, workflows, docs, sibling Dockerfiles) — stragglers are findings. For
NEWLY-pinned versions, check the upstream latest when network tools allow it and flag pins
that start life stale; skip silently offline.
- **Silent-failure fix without remediation** — a diff that fixes a silently-failing write path
(bad address, swallowed error, wrong topic) leaves behind whatever state was damaged or
omitted while it was broken. The fix must state how that backlog gets reconciled (backfill,
reconciliation run, "provably no traffic since X") — absent statement is a finding.
- **Normative doc changes** — a diff adding/changing a rules doc, convention, or runbook: grep
the governed tree for PRE-EXISTING violations of the new rule (the doc is wrong or the tree
is — say which). Runbooks/docs hardcoding mutable environment identifiers (personal accounts,
org names, pool IDs) are a maintenance hazard — flag unless marked with an ownership note.
### Scope discipline
@@ -84,6 +127,23 @@ A diff review is a review of THE CHANGE, not the whole file:
- Do they cover the unhappy paths, not just the happy ones?
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
### Test adequacy: the mutation question (MANDATORY)
Presence of tests proves nothing. For EACH behavior change in the diff, ask: **"if this
specific logic were wrong, which test would fail?"** and name it. No nameable test = a
finding (`behavior <X> has no test that would catch its regression`). State the mapping
even when clean: `Tests: <behavior> covered by <test name>`.
Named adequacy anti-patterns — each is a finding even when coverage looks green:
- **Mock-assertion tests** — the test asserts the mock was called, never the real outcome; it verifies wiring, not behavior, and passes when the logic is wrong.
- **Unexercised branch** — the diff adds a branch/condition no test drives down; coverage of the function ≠ coverage of the new path.
- **Implementation-mirroring tests** — the test recomputes the expected value using the same logic as the code under test; both are wrong together, so it can never fail meaningfully. Expected values must be independently derived literals.
- **Missing negative case** — a new validation/guard with tests only proving it ACCEPTS good input, never that it REJECTS bad input. The reject path is the whole point of a guard.
- **Leaky fixture** — an integration test against a shared/persistent database inserts fixture
rows directly with no paired cleanup (deferred delete/teardown/transaction rollback). The
test passes today and leaves a primary-key landmine for the next run.
## 3. Clarity
- Are names accurate? `get_user` that mutates is a lie; rename or split.
+8
View File
@@ -40,6 +40,14 @@ Does the migration mix a data backfill (UPDATE/INSERT over existing rows) into t
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
### 6. 🟡 `[correctness]` Seed/backfill literals not cross-checked against canonical values
Does the migration INSERT or UPDATE literal data values — enum strings, type/kind/status names, display copy, identifiers? Every literal has a canonical source somewhere: the application's enum/constant definitions, the schema's CHECK constraints, existing seed data, or a migration landing in the SAME stack. `fs_grep` for each literal's canonical definition and verify it matches — a seed row whose value drifts from the enum (or that omits a column a same-stack migration is about to make meaningful, or that conflicts on a constraint a sibling migration changes) fails at deploy time or, worse, seeds wrong data silently. Check the same-stack interaction explicitly: seed scripts and schema migrations that land together must be reviewed together.
### 7. 🟢 `[convention]` New table without audit columns
Does a new table carry the repo's standard audit columns — typically `created_at`/`updated_at`, and `version`/row-tracking where the house convention uses them? READ two or three sibling tables' migrations for the actual house convention first (Ground-truth discipline below) and cite one; a new table silently skipping it is the finding. A table that genuinely needs no audit trail (pure join table, ephemeral scratch) can be exempt — the author should say so, not leave it to inference.
## Ground-truth discipline
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
@@ -68,5 +68,13 @@ For inapplicable changes the whole block collapses to: `## Observability` / `Not
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
- **Per-verb metric families.** Sibling metrics differing only by an operation verb
(`thing_create_total`, `thing_update_total`, `thing_delete_total`) fragment dashboards and
alerts; the idiomatic shape is ONE family with an `operation` (and `outcome`) label. Flag the
family split — bounded label values, so cardinality stays sane (the unbounded-label check is
separate and still applies).
- **Telemetry outside its feature flag.** A change that gates behavior behind a flag but
records the behavior's metrics/logs unconditionally reports activity that isn't happening
(or leaks rollout state); gate the telemetry with the behavior.
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
+97
View File
@@ -0,0 +1,97 @@
---
name: review-misses
description: |
The review-miss ledger: escaped defects (bugs that survived review and were caught later)
recorded as reusable detection patterns. Review orchestrators load this skill, read the
ledger, and route matching entries to reviewers as first-class checks — this is the
feedback loop that makes the review stack converge instead of plateau.
Triggers: loaded by review orchestrators (e.g. code-reviewer) at review start; "record a miss", "review miss"
license: MIT
metadata:
author: Alex Clarke
version: 1.0.0
tags: [review, feedback-loop, ledger]
---
# review-misses
A review stack without a feedback loop plateaus: it catches the same classes forever and
misses the same classes forever. This skill closes the loop — every defect that ESCAPED
review (shipped, then caught in prod, by the user, or by a later change) becomes a ledger
entry with a detection pattern, and every future review runs those patterns as
first-class checks.
## The ledger
Location: `$HOME/.config/coyote/share/review-misses.md` (expand `$HOME` at runtime —
never hardcode a user's home path). The `share/` subdirectory is the convention for
durable DATA files that are not configuration — it keeps the config root uncluttered
(the /usr/share analogy). The `record-miss` macro migrates a ledger from the legacy
config-root location automatically; when reading, if `share/review-misses.md` is absent,
also check the legacy `$HOME/.config/coyote/review-misses.md` before concluding "empty".
**Scope: global and permanent.** One ledger per environment, shared across every repo,
review, and reviewer — it is the institutional memory of escaped defects, not per-review
state. Reviews only READ it; nothing about a review is ever written to it. The `repo:`
and `surface:` fields are how one global ledger stays relevant per review (filtering,
not partitioning). Concurrent reviews are fine: reviewers are concurrent readers, and
writes happen only through the `record-miss` macro — a human act, append-only. It
deliberately lives under config rather than a cache directory: caches are disposable by
contract, and losing this file loses the learning loop.
- **Missing or empty ledger**: skip silently — a one-line note in the synthesis
("review-miss ledger: empty") and nothing else. Never invent entries.
- Entries are append-only history. Never rewrite or delete entries during a review.
Entry format:
```markdown
## MISS-NNN — <one-line title>
- date: YYYY-MM-DD
- repo: <owner/repo where it escaped>
- surface: <rest-api | worker | db-migration | cli | iac | library | ci-cd | any>
- what escaped: <the defect, concretely>
- why review missed it: <root cause in review terms — wrong context, unchecked caller, mock-assert test, …>
- detection pattern: <the concrete check/grep/question a reviewer runs to catch this CLASS of defect>
```
## Consuming the ledger (review orchestrators)
1. At review start, read the ledger (if present).
2. Filter entries to the ones RELEVANT to this review: matching `surface` (against the
resolved surfaces), matching `repo`, or `surface: any`. Irrelevant entries stay out of
worker context — the ledger must not bloat every review.
3. Paste each relevant entry's title + detection pattern into the CONTEXT section of the
file-reviewers (or lanes) whose files it could apply to.
4. In the final report footer, note `review-miss patterns applied: MISS-004, MISS-011`
(or `none relevant`).
## Applying a pattern (reviewers)
- A ledger detection pattern is a FIRST-CLASS check — same standing as a loaded skill's
checklist item. Run it against the diff like any other check.
- When a pattern fires, the finding cites the entry: `recurrence of MISS-007` in the
finding title. Recurrences of an escaped defect class default to 🔴 — the class has
already burned us once.
- When a pattern doesn't fire, silence is fine — no per-entry accounting needed.
## Recording a miss
Record via the `record-miss` macro (`coyote --macro record-miss "<what escaped>"`), which
appends a numbered skeleton entry and prompts for the remaining fields. Or append by hand
in the entry format above.
To BOOTSTRAP a ledger from scratch (new team, new org, new environment), the
`mine-review-misses` macro (`coyote --macro mine-review-misses "<org | repo list | .>"`)
pulls real PR review history, mines the defect classes human reviewers keep enforcing,
and appends the candidates you approve — same entry format, same idempotent numbering.
Rules for good entries:
- **The detection pattern is the payload.** "Be more careful with X" is not a pattern;
"grep for `time.Now()` in files that also touch period/interval math; flag any not going
through the clock interface" is. If you can't state the check a reviewer runs, keep
distilling before recording.
- One entry per defect CLASS, not per incident — if a new miss matches an existing
entry's class, sharpen that entry's detection pattern instead of adding a duplicate.
- Never record secrets, customer data, or incident details beyond what the pattern needs.
+4
View File
@@ -40,6 +40,10 @@ Does the new consumer bound how many messages it processes at once — worker-po
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
### 6. 🟡 `[convention]` Job queue used for synchronous request-path work
Does request-path code enqueue a job and then WAIT for its result before responding — polling job status, blocking on a completion channel, sleeping in a loop? That buys the queue's costs (latency, serialization, a second failure domain, at-least-once semantics) without its benefit (durable async execution), and it serializes under load: N concurrent requests queue behind a worker pool sized for background throughput. The synchronous shape is a direct call (with a timeout); the queue is for work that outlives the request. Legitimate exceptions exist — a durable workflow where the caller genuinely resumes later, or intentional admission control — but they should be stated; flag the enqueue-then-wait shape when they aren't. Also the inverse smell: an expensive in-process retry/sleep loop inside a job handler where the queue's own scheduling (snooze/requeue with delay) is the idiomatic tool — the handler holds a worker slot hostage to sleep.
## Ground-truth discipline
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.