feat: created the finding-verifier and whetstone agents for continuous code quality improvements

This commit is contained in:
Dark-Alex-17 committed 2026-09-09 17:22:26 -06:00
1 parent 71c3283b87
commit 2928125b0e
16 files changed
+848 -13

No files matched your search

+62 -2
View File
@@ -29,16 +29,40 @@ fs_read --path "src/auth.rs" --offset 110 --limit 40
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
### Read the callers of anything changed
### Blast radius: verify every caller (MANDATORY)
If a hunk changes a function's body or its signature, grep for the name to find callers and check whether the change ripples:
A locally-correct change that alters a symbol's CONTRACT breaks callers you can't see in the diff. For every changed exported/public symbol, grep for its callers and verify each one still holds:
```
fs_grep --pattern "changed_function" --include "*.rs"
```
Contract changes that ripple (check the callers against each one that applies):
- **Return type / error type changed** — do callers match on variants that no longer exist, or miss new ones?
- **Nullability / optionality** — a value that could never be null/None now can be (or vice versa).
- **Defaults changed** — callers relying on the old default silently change behavior.
- **Units / encoding / format** — seconds→millis, bytes→string, naive→UTC datetimes.
- **Ordering / uniqueness guarantees** — output was sorted/deduped, now isn't (or vice versa).
- **Error semantics** — a function that returned an error now panics/throws, or swallows what it used to propagate.
- **Sync→async / blocking behavior** — callers on hot paths or in handlers now block or need awaiting.
This check is MANDATORY and produces output even when clean: state `Blast radius: N call sites checked, all compatible` in your findings, or one finding per incompatible/unverifiable caller (`caller at path:line still assumes <old contract>`). Greps are cheap — the file-read budget does not apply to `fs_grep`/`fs_glob`; spend greps freely here and targeted reads only on suspicious callers.
Skip the test files in this search; do the test sweep next.
### Guard symmetry across sibling paths (MANDATORY when a guard changes)
When the diff ADDS, STRENGTHENS, or FIXES a guard/check/validation in one code path, the same
flaw usually lives in that path's parallel twins — sibling strategies implementing the same
interface, other enqueue/call sites of the same job, the script's sibling scripts, the other
branch of the same switch. Enumerate the twins (`fs_grep` for the shared interface, the job
kind, the naming pattern) and verify each one either has the equivalent guard or provably
doesn't need it. A twin missing the guard is a finding at the twin's `path:line` — same
severity as the bug the guard fixes. Report even when clean: `Guard symmetry: N sibling
paths checked`. (Guards REMOVED are the removed-guards check below — this is about the ones
added: a fix applied to one of N twins is N-1 latent bugs.)
### Read the tests for the change
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
@@ -58,7 +82,26 @@ These are review findings that only surface in a diff context, not in a whole-fi
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
- **New code path without new tests** — usually a missing test. Flag it.
- **Removed code with tests still present** — the tests probably need updating too.
- **Removed guards, checks, and validations** — for EVERY removed guard/branch/validation/limit, name where that responsibility now lives ("moved to X at path:line") or flag it as a regression risk. Deleted code had a reason; "nothing now does what the deleted code did" is a 🟡 finding by default, and 🔴 when the guard protected money, auth, or data integrity. Reviews fixate on added lines — the removed lines are where incidents come from.
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
- **Minted but unused / partial adoption** — every artifact the diff INTRODUCES (variable, flag,
helper, image, config key) must be consumed somewhere; `fs_grep` for each one. Zero uses = a
finding (dead weight or a forgotten wiring step). Used in SOME applicable in-diff sites but
not others = a finding naming the sites that didn't adopt it — half-adopted artifacts are how
two mechanisms for the same job end up coexisting forever.
- **Version-literal consistency and freshness** — when the diff bumps a version (image tag,
tool version, dependency pin), `fs_grep` the repo for other occurrences of the OLD version
string (compose files, workflows, docs, sibling Dockerfiles) — stragglers are findings. For
NEWLY-pinned versions, check the upstream latest when network tools allow it and flag pins
that start life stale; skip silently offline.
- **Silent-failure fix without remediation** — a diff that fixes a silently-failing write path
(bad address, swallowed error, wrong topic) leaves behind whatever state was damaged or
omitted while it was broken. The fix must state how that backlog gets reconciled (backfill,
reconciliation run, "provably no traffic since X") — absent statement is a finding.
- **Normative doc changes** — a diff adding/changing a rules doc, convention, or runbook: grep
the governed tree for PRE-EXISTING violations of the new rule (the doc is wrong or the tree
is — say which). Runbooks/docs hardcoding mutable environment identifiers (personal accounts,
org names, pool IDs) are a maintenance hazard — flag unless marked with an ownership note.
### Scope discipline
@@ -84,6 +127,23 @@ A diff review is a review of THE CHANGE, not the whole file:
- Do they cover the unhappy paths, not just the happy ones?
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
### Test adequacy: the mutation question (MANDATORY)
Presence of tests proves nothing. For EACH behavior change in the diff, ask: **"if this
specific logic were wrong, which test would fail?"** and name it. No nameable test = a
finding (`behavior <X> has no test that would catch its regression`). State the mapping
even when clean: `Tests: <behavior> covered by <test name>`.
Named adequacy anti-patterns — each is a finding even when coverage looks green:
- **Mock-assertion tests** — the test asserts the mock was called, never the real outcome; it verifies wiring, not behavior, and passes when the logic is wrong.
- **Unexercised branch** — the diff adds a branch/condition no test drives down; coverage of the function ≠ coverage of the new path.
- **Implementation-mirroring tests** — the test recomputes the expected value using the same logic as the code under test; both are wrong together, so it can never fail meaningfully. Expected values must be independently derived literals.
- **Missing negative case** — a new validation/guard with tests only proving it ACCEPTS good input, never that it REJECTS bad input. The reject path is the whole point of a guard.
- **Leaky fixture** — an integration test against a shared/persistent database inserts fixture
rows directly with no paired cleanup (deferred delete/teardown/transaction rollback). The
test passes today and leaves a primary-key landmine for the next run.
## 3. Clarity
- Are names accurate? `get_user` that mutates is a lie; rename or split.
+8
View File
@@ -40,6 +40,14 @@ Does the migration mix a data backfill (UPDATE/INSERT over existing rows) into t
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
### 6. 🟡 `[correctness]` Seed/backfill literals not cross-checked against canonical values
Does the migration INSERT or UPDATE literal data values — enum strings, type/kind/status names, display copy, identifiers? Every literal has a canonical source somewhere: the application's enum/constant definitions, the schema's CHECK constraints, existing seed data, or a migration landing in the SAME stack. `fs_grep` for each literal's canonical definition and verify it matches — a seed row whose value drifts from the enum (or that omits a column a same-stack migration is about to make meaningful, or that conflicts on a constraint a sibling migration changes) fails at deploy time or, worse, seeds wrong data silently. Check the same-stack interaction explicitly: seed scripts and schema migrations that land together must be reviewed together.
### 7. 🟢 `[convention]` New table without audit columns
Does a new table carry the repo's standard audit columns — typically `created_at`/`updated_at`, and `version`/row-tracking where the house convention uses them? READ two or three sibling tables' migrations for the actual house convention first (Ground-truth discipline below) and cite one; a new table silently skipping it is the finding. A table that genuinely needs no audit trail (pure join table, ephemeral scratch) can be exempt — the author should say so, not leave it to inference.
## Ground-truth discipline
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
@@ -68,5 +68,13 @@ For inapplicable changes the whole block collapses to: `## Observability` / `Not
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
- **Per-verb metric families.** Sibling metrics differing only by an operation verb
(`thing_create_total`, `thing_update_total`, `thing_delete_total`) fragment dashboards and
alerts; the idiomatic shape is ONE family with an `operation` (and `outcome`) label. Flag the
family split — bounded label values, so cardinality stays sane (the unbounded-label check is
separate and still applies).
- **Telemetry outside its feature flag.** A change that gates behavior behind a flag but
records the behavior's metrics/logs unconditionally reports activity that isn't happening
(or leaks rollout state); gate the telemetry with the behavior.
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
+97
View File
@@ -0,0 +1,97 @@
---
name: review-misses
description: |
The review-miss ledger: escaped defects (bugs that survived review and were caught later)
recorded as reusable detection patterns. Review orchestrators load this skill, read the
ledger, and route matching entries to reviewers as first-class checks — this is the
feedback loop that makes the review stack converge instead of plateau.
Triggers: loaded by review orchestrators (e.g. code-reviewer) at review start; "record a miss", "review miss"
license: MIT
metadata:
author: Alex Clarke
version: 1.0.0
tags: [review, feedback-loop, ledger]
---
# review-misses
A review stack without a feedback loop plateaus: it catches the same classes forever and
misses the same classes forever. This skill closes the loop — every defect that ESCAPED
review (shipped, then caught in prod, by the user, or by a later change) becomes a ledger
entry with a detection pattern, and every future review runs those patterns as
first-class checks.
## The ledger
Location: `$HOME/.config/coyote/share/review-misses.md` (expand `$HOME` at runtime —
never hardcode a user's home path). The `share/` subdirectory is the convention for
durable DATA files that are not configuration — it keeps the config root uncluttered
(the /usr/share analogy). The `record-miss` macro migrates a ledger from the legacy
config-root location automatically; when reading, if `share/review-misses.md` is absent,
also check the legacy `$HOME/.config/coyote/review-misses.md` before concluding "empty".
**Scope: global and permanent.** One ledger per environment, shared across every repo,
review, and reviewer — it is the institutional memory of escaped defects, not per-review
state. Reviews only READ it; nothing about a review is ever written to it. The `repo:`
and `surface:` fields are how one global ledger stays relevant per review (filtering,
not partitioning). Concurrent reviews are fine: reviewers are concurrent readers, and
writes happen only through the `record-miss` macro — a human act, append-only. It
deliberately lives under config rather than a cache directory: caches are disposable by
contract, and losing this file loses the learning loop.
- **Missing or empty ledger**: skip silently — a one-line note in the synthesis
("review-miss ledger: empty") and nothing else. Never invent entries.
- Entries are append-only history. Never rewrite or delete entries during a review.
Entry format:
```markdown
## MISS-NNN — <one-line title>
- date: YYYY-MM-DD
- repo: <owner/repo where it escaped>
- surface: <rest-api | worker | db-migration | cli | iac | library | ci-cd | any>
- what escaped: <the defect, concretely>
- why review missed it: <root cause in review terms — wrong context, unchecked caller, mock-assert test, …>
- detection pattern: <the concrete check/grep/question a reviewer runs to catch this CLASS of defect>
```
## Consuming the ledger (review orchestrators)
1. At review start, read the ledger (if present).
2. Filter entries to the ones RELEVANT to this review: matching `surface` (against the
resolved surfaces), matching `repo`, or `surface: any`. Irrelevant entries stay out of
worker context — the ledger must not bloat every review.
3. Paste each relevant entry's title + detection pattern into the CONTEXT section of the
file-reviewers (or lanes) whose files it could apply to.
4. In the final report footer, note `review-miss patterns applied: MISS-004, MISS-011`
(or `none relevant`).
## Applying a pattern (reviewers)
- A ledger detection pattern is a FIRST-CLASS check — same standing as a loaded skill's
checklist item. Run it against the diff like any other check.
- When a pattern fires, the finding cites the entry: `recurrence of MISS-007` in the
finding title. Recurrences of an escaped defect class default to 🔴 — the class has
already burned us once.
- When a pattern doesn't fire, silence is fine — no per-entry accounting needed.
## Recording a miss
Record via the `record-miss` macro (`coyote --macro record-miss "<what escaped>"`), which
appends a numbered skeleton entry and prompts for the remaining fields. Or append by hand
in the entry format above.
To BOOTSTRAP a ledger from scratch (new team, new org, new environment), the
`mine-review-misses` macro (`coyote --macro mine-review-misses "<org | repo list | .>"`)
pulls real PR review history, mines the defect classes human reviewers keep enforcing,
and appends the candidates you approve — same entry format, same idempotent numbering.
Rules for good entries:
- **The detection pattern is the payload.** "Be more careful with X" is not a pattern;
"grep for `time.Now()` in files that also touch period/interval math; flag any not going
through the clock interface" is. If you can't state the check a reviewer runs, keep
distilling before recording.
- One entry per defect CLASS, not per incident — if a new miss matches an existing
entry's class, sharpen that entry's detection pattern instead of adding a duplicate.
- Never record secrets, customer data, or incident details beyond what the pattern needs.
+4
View File
@@ -40,6 +40,10 @@ Does the new consumer bound how many messages it processes at once — worker-po
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
### 6. 🟡 `[convention]` Job queue used for synchronous request-path work
Does request-path code enqueue a job and then WAIT for its result before responding — polling job status, blocking on a completion channel, sleeping in a loop? That buys the queue's costs (latency, serialization, a second failure domain, at-least-once semantics) without its benefit (durable async execution), and it serializes under load: N concurrent requests queue behind a worker pool sized for background throughput. The synchronous shape is a direct call (with a timeout); the queue is for work that outlives the request. Legitimate exceptions exist — a durable workflow where the caller genuinely resumes later, or intentional admission control — but they should be stated; flag the enqueue-then-wait shape when they aren't. Also the inverse smell: an expensive in-process retry/sleep loop inside a job handler where the queue's own scheduling (snooze/requeue with delay) is the idiomatic tool — the handler holds a worker slot hostage to sleep.
## Ground-truth discipline
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.