feat: created the finding-verifier and whetstone agents for continuous code quality improvements
This commit is contained in:
16 files changed
+848
-13
No files matched your search
@@ -29,16 +29,40 @@ fs_read --path "src/auth.rs" --offset 110 --limit 40
|
||||
|
||||
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
|
||||
|
||||
### Read the callers of anything changed
|
||||
### Blast radius: verify every caller (MANDATORY)
|
||||
|
||||
If a hunk changes a function's body or its signature, grep for the name to find callers and check whether the change ripples:
|
||||
A locally-correct change that alters a symbol's CONTRACT breaks callers you can't see in the diff. For every changed exported/public symbol, grep for its callers and verify each one still holds:
|
||||
|
||||
```
|
||||
fs_grep --pattern "changed_function" --include "*.rs"
|
||||
```
|
||||
|
||||
Contract changes that ripple (check the callers against each one that applies):
|
||||
|
||||
- **Return type / error type changed** — do callers match on variants that no longer exist, or miss new ones?
|
||||
- **Nullability / optionality** — a value that could never be null/None now can be (or vice versa).
|
||||
- **Defaults changed** — callers relying on the old default silently change behavior.
|
||||
- **Units / encoding / format** — seconds→millis, bytes→string, naive→UTC datetimes.
|
||||
- **Ordering / uniqueness guarantees** — output was sorted/deduped, now isn't (or vice versa).
|
||||
- **Error semantics** — a function that returned an error now panics/throws, or swallows what it used to propagate.
|
||||
- **Sync→async / blocking behavior** — callers on hot paths or in handlers now block or need awaiting.
|
||||
|
||||
This check is MANDATORY and produces output even when clean: state `Blast radius: N call sites checked, all compatible` in your findings, or one finding per incompatible/unverifiable caller (`caller at path:line still assumes <old contract>`). Greps are cheap — the file-read budget does not apply to `fs_grep`/`fs_glob`; spend greps freely here and targeted reads only on suspicious callers.
|
||||
|
||||
Skip the test files in this search; do the test sweep next.
|
||||
|
||||
### Guard symmetry across sibling paths (MANDATORY when a guard changes)
|
||||
|
||||
When the diff ADDS, STRENGTHENS, or FIXES a guard/check/validation in one code path, the same
|
||||
flaw usually lives in that path's parallel twins — sibling strategies implementing the same
|
||||
interface, other enqueue/call sites of the same job, the script's sibling scripts, the other
|
||||
branch of the same switch. Enumerate the twins (`fs_grep` for the shared interface, the job
|
||||
kind, the naming pattern) and verify each one either has the equivalent guard or provably
|
||||
doesn't need it. A twin missing the guard is a finding at the twin's `path:line` — same
|
||||
severity as the bug the guard fixes. Report even when clean: `Guard symmetry: N sibling
|
||||
paths checked`. (Guards REMOVED are the removed-guards check below — this is about the ones
|
||||
added: a fix applied to one of N twins is N-1 latent bugs.)
|
||||
|
||||
### Read the tests for the change
|
||||
|
||||
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
|
||||
@@ -58,7 +82,26 @@ These are review findings that only surface in a diff context, not in a whole-fi
|
||||
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
|
||||
- **New code path without new tests** — usually a missing test. Flag it.
|
||||
- **Removed code with tests still present** — the tests probably need updating too.
|
||||
- **Removed guards, checks, and validations** — for EVERY removed guard/branch/validation/limit, name where that responsibility now lives ("moved to X at path:line") or flag it as a regression risk. Deleted code had a reason; "nothing now does what the deleted code did" is a 🟡 finding by default, and 🔴 when the guard protected money, auth, or data integrity. Reviews fixate on added lines — the removed lines are where incidents come from.
|
||||
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
|
||||
- **Minted but unused / partial adoption** — every artifact the diff INTRODUCES (variable, flag,
|
||||
helper, image, config key) must be consumed somewhere; `fs_grep` for each one. Zero uses = a
|
||||
finding (dead weight or a forgotten wiring step). Used in SOME applicable in-diff sites but
|
||||
not others = a finding naming the sites that didn't adopt it — half-adopted artifacts are how
|
||||
two mechanisms for the same job end up coexisting forever.
|
||||
- **Version-literal consistency and freshness** — when the diff bumps a version (image tag,
|
||||
tool version, dependency pin), `fs_grep` the repo for other occurrences of the OLD version
|
||||
string (compose files, workflows, docs, sibling Dockerfiles) — stragglers are findings. For
|
||||
NEWLY-pinned versions, check the upstream latest when network tools allow it and flag pins
|
||||
that start life stale; skip silently offline.
|
||||
- **Silent-failure fix without remediation** — a diff that fixes a silently-failing write path
|
||||
(bad address, swallowed error, wrong topic) leaves behind whatever state was damaged or
|
||||
omitted while it was broken. The fix must state how that backlog gets reconciled (backfill,
|
||||
reconciliation run, "provably no traffic since X") — absent statement is a finding.
|
||||
- **Normative doc changes** — a diff adding/changing a rules doc, convention, or runbook: grep
|
||||
the governed tree for PRE-EXISTING violations of the new rule (the doc is wrong or the tree
|
||||
is — say which). Runbooks/docs hardcoding mutable environment identifiers (personal accounts,
|
||||
org names, pool IDs) are a maintenance hazard — flag unless marked with an ownership note.
|
||||
|
||||
### Scope discipline
|
||||
|
||||
@@ -84,6 +127,23 @@ A diff review is a review of THE CHANGE, not the whole file:
|
||||
- Do they cover the unhappy paths, not just the happy ones?
|
||||
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
|
||||
|
||||
### Test adequacy: the mutation question (MANDATORY)
|
||||
|
||||
Presence of tests proves nothing. For EACH behavior change in the diff, ask: **"if this
|
||||
specific logic were wrong, which test would fail?"** and name it. No nameable test = a
|
||||
finding (`behavior <X> has no test that would catch its regression`). State the mapping
|
||||
even when clean: `Tests: <behavior> covered by <test name>`.
|
||||
|
||||
Named adequacy anti-patterns — each is a finding even when coverage looks green:
|
||||
|
||||
- **Mock-assertion tests** — the test asserts the mock was called, never the real outcome; it verifies wiring, not behavior, and passes when the logic is wrong.
|
||||
- **Unexercised branch** — the diff adds a branch/condition no test drives down; coverage of the function ≠ coverage of the new path.
|
||||
- **Implementation-mirroring tests** — the test recomputes the expected value using the same logic as the code under test; both are wrong together, so it can never fail meaningfully. Expected values must be independently derived literals.
|
||||
- **Missing negative case** — a new validation/guard with tests only proving it ACCEPTS good input, never that it REJECTS bad input. The reject path is the whole point of a guard.
|
||||
- **Leaky fixture** — an integration test against a shared/persistent database inserts fixture
|
||||
rows directly with no paired cleanup (deferred delete/teardown/transaction rollback). The
|
||||
test passes today and leaves a primary-key landmine for the next run.
|
||||
|
||||
## 3. Clarity
|
||||
|
||||
- Are names accurate? `get_user` that mutates is a lie; rename or split.
|
||||
|
||||
@@ -40,6 +40,14 @@ Does the migration mix a data backfill (UPDATE/INSERT over existing rows) into t
|
||||
|
||||
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
|
||||
|
||||
### 6. 🟡 `[correctness]` Seed/backfill literals not cross-checked against canonical values
|
||||
|
||||
Does the migration INSERT or UPDATE literal data values — enum strings, type/kind/status names, display copy, identifiers? Every literal has a canonical source somewhere: the application's enum/constant definitions, the schema's CHECK constraints, existing seed data, or a migration landing in the SAME stack. `fs_grep` for each literal's canonical definition and verify it matches — a seed row whose value drifts from the enum (or that omits a column a same-stack migration is about to make meaningful, or that conflicts on a constraint a sibling migration changes) fails at deploy time or, worse, seeds wrong data silently. Check the same-stack interaction explicitly: seed scripts and schema migrations that land together must be reviewed together.
|
||||
|
||||
### 7. 🟢 `[convention]` New table without audit columns
|
||||
|
||||
Does a new table carry the repo's standard audit columns — typically `created_at`/`updated_at`, and `version`/row-tracking where the house convention uses them? READ two or three sibling tables' migrations for the actual house convention first (Ground-truth discipline below) and cite one; a new table silently skipping it is the finding. A table that genuinely needs no audit trail (pure join table, ephemeral scratch) can be exempt — the author should say so, not leave it to inference.
|
||||
|
||||
## Ground-truth discipline
|
||||
|
||||
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
|
||||
|
||||
@@ -68,5 +68,13 @@ For inapplicable changes the whole block collapses to: `## Observability` / `Not
|
||||
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
|
||||
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
|
||||
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
|
||||
- **Per-verb metric families.** Sibling metrics differing only by an operation verb
|
||||
(`thing_create_total`, `thing_update_total`, `thing_delete_total`) fragment dashboards and
|
||||
alerts; the idiomatic shape is ONE family with an `operation` (and `outcome`) label. Flag the
|
||||
family split — bounded label values, so cardinality stays sane (the unbounded-label check is
|
||||
separate and still applies).
|
||||
- **Telemetry outside its feature flag.** A change that gates behavior behind a flag but
|
||||
records the behavior's metrics/logs unconditionally reports activity that isn't happening
|
||||
(or leaks rollout state); gate the telemetry with the behavior.
|
||||
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
|
||||
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: review-misses
|
||||
description: |
|
||||
The review-miss ledger: escaped defects (bugs that survived review and were caught later)
|
||||
recorded as reusable detection patterns. Review orchestrators load this skill, read the
|
||||
ledger, and route matching entries to reviewers as first-class checks — this is the
|
||||
feedback loop that makes the review stack converge instead of plateau.
|
||||
Triggers: loaded by review orchestrators (e.g. code-reviewer) at review start; "record a miss", "review miss"
|
||||
license: MIT
|
||||
metadata:
|
||||
author: Alex Clarke
|
||||
version: 1.0.0
|
||||
tags: [review, feedback-loop, ledger]
|
||||
---
|
||||
|
||||
# review-misses
|
||||
|
||||
A review stack without a feedback loop plateaus: it catches the same classes forever and
|
||||
misses the same classes forever. This skill closes the loop — every defect that ESCAPED
|
||||
review (shipped, then caught in prod, by the user, or by a later change) becomes a ledger
|
||||
entry with a detection pattern, and every future review runs those patterns as
|
||||
first-class checks.
|
||||
|
||||
## The ledger
|
||||
|
||||
Location: `$HOME/.config/coyote/share/review-misses.md` (expand `$HOME` at runtime —
|
||||
never hardcode a user's home path). The `share/` subdirectory is the convention for
|
||||
durable DATA files that are not configuration — it keeps the config root uncluttered
|
||||
(the /usr/share analogy). The `record-miss` macro migrates a ledger from the legacy
|
||||
config-root location automatically; when reading, if `share/review-misses.md` is absent,
|
||||
also check the legacy `$HOME/.config/coyote/review-misses.md` before concluding "empty".
|
||||
|
||||
**Scope: global and permanent.** One ledger per environment, shared across every repo,
|
||||
review, and reviewer — it is the institutional memory of escaped defects, not per-review
|
||||
state. Reviews only READ it; nothing about a review is ever written to it. The `repo:`
|
||||
and `surface:` fields are how one global ledger stays relevant per review (filtering,
|
||||
not partitioning). Concurrent reviews are fine: reviewers are concurrent readers, and
|
||||
writes happen only through the `record-miss` macro — a human act, append-only. It
|
||||
deliberately lives under config rather than a cache directory: caches are disposable by
|
||||
contract, and losing this file loses the learning loop.
|
||||
|
||||
- **Missing or empty ledger**: skip silently — a one-line note in the synthesis
|
||||
("review-miss ledger: empty") and nothing else. Never invent entries.
|
||||
- Entries are append-only history. Never rewrite or delete entries during a review.
|
||||
|
||||
Entry format:
|
||||
|
||||
```markdown
|
||||
## MISS-NNN — <one-line title>
|
||||
- date: YYYY-MM-DD
|
||||
- repo: <owner/repo where it escaped>
|
||||
- surface: <rest-api | worker | db-migration | cli | iac | library | ci-cd | any>
|
||||
- what escaped: <the defect, concretely>
|
||||
- why review missed it: <root cause in review terms — wrong context, unchecked caller, mock-assert test, …>
|
||||
- detection pattern: <the concrete check/grep/question a reviewer runs to catch this CLASS of defect>
|
||||
```
|
||||
|
||||
## Consuming the ledger (review orchestrators)
|
||||
|
||||
1. At review start, read the ledger (if present).
|
||||
2. Filter entries to the ones RELEVANT to this review: matching `surface` (against the
|
||||
resolved surfaces), matching `repo`, or `surface: any`. Irrelevant entries stay out of
|
||||
worker context — the ledger must not bloat every review.
|
||||
3. Paste each relevant entry's title + detection pattern into the CONTEXT section of the
|
||||
file-reviewers (or lanes) whose files it could apply to.
|
||||
4. In the final report footer, note `review-miss patterns applied: MISS-004, MISS-011`
|
||||
(or `none relevant`).
|
||||
|
||||
## Applying a pattern (reviewers)
|
||||
|
||||
- A ledger detection pattern is a FIRST-CLASS check — same standing as a loaded skill's
|
||||
checklist item. Run it against the diff like any other check.
|
||||
- When a pattern fires, the finding cites the entry: `recurrence of MISS-007` in the
|
||||
finding title. Recurrences of an escaped defect class default to 🔴 — the class has
|
||||
already burned us once.
|
||||
- When a pattern doesn't fire, silence is fine — no per-entry accounting needed.
|
||||
|
||||
## Recording a miss
|
||||
|
||||
Record via the `record-miss` macro (`coyote --macro record-miss "<what escaped>"`), which
|
||||
appends a numbered skeleton entry and prompts for the remaining fields. Or append by hand
|
||||
in the entry format above.
|
||||
|
||||
To BOOTSTRAP a ledger from scratch (new team, new org, new environment), the
|
||||
`mine-review-misses` macro (`coyote --macro mine-review-misses "<org | repo list | .>"`)
|
||||
pulls real PR review history, mines the defect classes human reviewers keep enforcing,
|
||||
and appends the candidates you approve — same entry format, same idempotent numbering.
|
||||
|
||||
Rules for good entries:
|
||||
|
||||
- **The detection pattern is the payload.** "Be more careful with X" is not a pattern;
|
||||
"grep for `time.Now()` in files that also touch period/interval math; flag any not going
|
||||
through the clock interface" is. If you can't state the check a reviewer runs, keep
|
||||
distilling before recording.
|
||||
- One entry per defect CLASS, not per incident — if a new miss matches an existing
|
||||
entry's class, sharpen that entry's detection pattern instead of adding a duplicate.
|
||||
- Never record secrets, customer data, or incident details beyond what the pattern needs.
|
||||
@@ -40,6 +40,10 @@ Does the new consumer bound how many messages it processes at once — worker-po
|
||||
|
||||
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
|
||||
|
||||
### 6. 🟡 `[convention]` Job queue used for synchronous request-path work
|
||||
|
||||
Does request-path code enqueue a job and then WAIT for its result before responding — polling job status, blocking on a completion channel, sleeping in a loop? That buys the queue's costs (latency, serialization, a second failure domain, at-least-once semantics) without its benefit (durable async execution), and it serializes under load: N concurrent requests queue behind a worker pool sized for background throughput. The synchronous shape is a direct call (with a timeout); the queue is for work that outlives the request. Legitimate exceptions exist — a durable workflow where the caller genuinely resumes later, or intentional admission control — but they should be stated; flag the enqueue-then-wait shape when they aren't. Also the inverse smell: an expensive in-process retry/sleep loop inside a job handler where the queue's own scheduling (snooze/requeue with delay) is the idiomatic tool — the handler holds a worker slot hostage to sleep.
|
||||
|
||||
## Ground-truth discipline
|
||||
|
||||
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.
|
||||
|
||||
Reference in new issue
Block a user