feat: created the finding-verifier and whetstone agents for continuous code quality improvements
This commit is contained in:
16 files changed
+848
-13
No files matched your search
@@ -0,0 +1,77 @@
|
||||
# Mine PR review history for review-miss ledger candidates — the portable version of
|
||||
# the gap-study methodology: pull real review
|
||||
# comments from the repos you care about, find the defect CLASSES human reviewers keep
|
||||
# enforcing that automated review would miss (org conventions, domain semantics, live
|
||||
# org state, missing mechanical rules), and seed the ledger with the ones you approve.
|
||||
# Use it when joining a team / moving jobs to bootstrap the ledger from the org's
|
||||
# actual review culture instead of waiting for misses to bite one at a time.
|
||||
#
|
||||
# The first step is deterministic shell: it resolves the scope, pulls the corpus via
|
||||
# gh into ~/.cache/coyote/mine-misses/, strips ANSI (CLICOLOR_FORCE breaks piped gh
|
||||
# JSON), and normalizes to JSONL. The agent step then splits reviewer lanes, mines
|
||||
# defect classes, proposes candidates, and appends ONLY what you approve —
|
||||
# idempotently, in record-miss's entry format.
|
||||
#
|
||||
# Run as:
|
||||
# coyote --macro mine-review-misses "myorg" # org: 12 most recently pushed repos
|
||||
# coyote --macro mine-review-misses "owner/repo1,owner/repo2" # explicit repo list
|
||||
# coyote --macro mine-review-misses "." # the repo in the current directory
|
||||
#
|
||||
# Re-run it periodically: each run stores a per-scope watermark under
|
||||
# ~/.config/coyote/share/mine-misses-state/, and later runs pull only comments updated
|
||||
# since the previous run (falling back to the newest-400 window on first run). Appends
|
||||
# are idempotent and candidates are deduped against the ledger, so re-running is always
|
||||
# safe. If you ABORT a run before reviewing its candidates, delete that scope's
|
||||
# watermark file to re-widen the next window. Note the miner only sees misses a human
|
||||
# eventually commented on — defects that escaped review entirely (caught in prod or an
|
||||
# incident) are record-miss's job; the two are complementary.
|
||||
#
|
||||
# Notes: an org-wide pull is a few hundred GitHub API calls (a few minutes; well under
|
||||
# authenticated rate limits). Mining quality depends on humans having reviewed there —
|
||||
# a repo reviewed only by bots yields nothing, and that itself is worth knowing.
|
||||
variables:
|
||||
- name: scope
|
||||
description: GitHub org name, comma-separated owner/repo list, or "." for the current directory's repo
|
||||
isolated: false
|
||||
steps:
|
||||
- |-
|
||||
!export GH_NO_UPDATE_NOTIFIER=1 NO_COLOR=1; unset CLICOLOR_FORCE GH_FORCE_TTY 2>/dev/null; d="$HOME/.cache/coyote/mine-misses"; rm -rf "$d"; mkdir -p "$d"; scope="{{scope}}"
|
||||
case "$scope" in
|
||||
.|"") repos="$(git remote get-url origin 2>/dev/null | sed -E 's#^(git@|https?://)([^/:]+)[:/]##; s#\.git$##')" ;;
|
||||
*/*) repos="$(printf '%s' "$scope" | tr ',' ' ')" ;;
|
||||
*) repos="$(gh repo list "$scope" --no-archived --limit 12 --json nameWithOwner --jq '.[].nameWithOwner' | tr '\n' ' ')" ;;
|
||||
esac
|
||||
repos=$(echo $repos); [ -n "$repos" ] || { echo "mine-review-misses: could not resolve any repos from scope '$scope' (pass an org name, an owner/repo list, or run inside a git repo)"; exit 1; }
|
||||
state="$HOME/.config/coyote/share/mine-misses-state"; mkdir -p "$state"; skey="$state/$(printf '%s' "$scope" | tr '/,: ' '____')"
|
||||
sincearg=""; window="newest 400 comments per repo (first run for this scope)"
|
||||
if [ -f "$skey" ]; then w=$(cat "$skey"); sincearg="&since=$w"; window="comments updated since $w (incremental)"; fi
|
||||
now=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
echo "scope: $scope" > "$d/MANIFEST"; echo "repos: $repos" >> "$d/MANIFEST"; echo "window: $window" >> "$d/MANIFEST"
|
||||
for r in $repos; do
|
||||
slug=$(printf '%s' "$r" | tr '/' '_')
|
||||
: > "$d/$slug-comments.raw"
|
||||
for p in 1 2 3 4; do out=$(gh api "repos/$r/pulls/comments?sort=created&direction=desc&per_page=100&page=$p$sincearg" --jq '.[] | {pr: (.pull_request_url|split("/")|last), user: .user.login, path: .path, line: (.line // .original_line), created: .created_at, body: (.body|.[0:600])}' 2>/dev/null) || break; [ -n "$out" ] || break; printf '%s\n' "$out" >> "$d/$slug-comments.raw"; done
|
||||
gh pr list --repo "$r" --state all --limit 300 --json number,author,title --jq '.[] | {n: .number, author: .author.login, title: .title}' > "$d/$slug-prs.raw" 2>/dev/null
|
||||
: > "$d/$slug-reviews.raw"
|
||||
for n in $(gh api "search/issues?q=repo:$r+is:pr+review:changes_requested&sort=updated&order=desc&per_page=30" --jq '.items[].number' 2>/dev/null); do
|
||||
gh api "repos/$r/pulls/$n/reviews" --jq '.[] | select(.state=="CHANGES_REQUESTED" or (.state=="COMMENTED" and (.body|length)>60)) | {pr: '"$n"', user: .user.login, state: .state, body: (.body|.[0:600])}' >> "$d/$slug-reviews.raw" 2>/dev/null
|
||||
done
|
||||
for f in "$d/$slug-comments" "$d/$slug-prs" "$d/$slug-reviews"; do
|
||||
sed 's/\x1b\[[0-9;]*m//g' "$f.raw" 2>/dev/null | jq -c . > "$f.jsonl" 2>/dev/null || : > "$f.jsonl"; rm -f "$f.raw"
|
||||
done
|
||||
echo "$r: comments=$(wc -l < "$d/$slug-comments.jsonl" | tr -d ' ') prs=$(wc -l < "$d/$slug-prs.jsonl" | tr -d ' ') review_bodies=$(wc -l < "$d/$slug-reviews.jsonl" | tr -d ' ')" >> "$d/MANIFEST"
|
||||
done
|
||||
echo "$now" > "$skey"
|
||||
cat "$d/MANIFEST"
|
||||
- |-
|
||||
A PR-review corpus was just pulled into `$HOME/.cache/coyote/mine-misses/` — read `MANIFEST` there first (scope, repos, the time window, per-repo counts). Per repo the files are `<owner>_<repo>-comments.jsonl` (inline review comments: {pr,user,path,line,created,body}), `<owner>_<repo>-prs.jsonl` (PR metadata: {n,author,title}), and `<owner>_<repo>-reviews.jsonl` (CHANGES_REQUESTED / substantive review bodies: {pr,user,state,body}). On an INCREMENTAL window (see MANIFEST) the comment corpus covers only activity since the previous run — near-zero counts mean little new review activity, which you should simply report; the review-bodies file is not windowed, so on incremental runs ignore review bodies from PRs you'd expect were already mined. Load the `review-misses` skill, then mine this corpus for ledger candidates. Follow the method in order — each step corrects a specific bias:
|
||||
|
||||
1. **Lane hygiene first** (skipping this makes the analysis circular). Per repo: (a) drop PR-author self-replies — join comments to `-prs.jsonl` by PR number (jq) and drop records where the commenter is that PR's author; (b) exclude non-human reviewers — `[bot]`-suffixed accounts AND any account whose comments are recognizably automated-review output (severity-emoji conventions like 🔴/🟠/🟡/🟢, lane tags like `[adversary]`/`[code-review]`, "Panel review" tables, templated verdict blocks — automated reviews are sometimes posted from a personal account, so judge by format, not just username). Name the accounts you excluded and why. What remains is the organic human corpus — report its size per repo; if a repo has almost none, say so and move on.
|
||||
|
||||
2. **Mine defect CLASSES, not comments.** Read the organic human comments and review bodies looking for RECURRING things humans enforce, in two families: (a) knowledge not derivable from the repo — org conventions (especially ones the repo's own prior art CONTRADICTS: those poison neighbor-convention heuristics and make the strongest entries), domain/product semantics, live org state (client registries, environment facts, safe test tenants); (b) objectively checkable patterns reviewers keep flagging by hand (grep-shaped rules). EXCLUDE taste — naming preferences, style debates, "I'd have structured this differently" — a ledger polluted with taste gets ignored. One candidate per CLASS: merge repeat occurrences into a single entry citing multiple PRs.
|
||||
|
||||
3. **Dedup against the existing ledger.** Read `$HOME/.config/coyote/share/review-misses.md` (fall back to the legacy `$HOME/.config/coyote/review-misses.md` if share/ has none). A candidate matching an existing entry's defect class becomes a proposal to SHARPEN that entry (name its MISS id), not a new one.
|
||||
|
||||
4. **Propose, don't write.** Present at most 10 candidates, strongest first (recurrence × consequence), each formatted exactly as a ledger entry: title / repo(s) / surface (rest-api|worker|db-migration|cli|iac|library|ci-cd|any) / what escaped (real PR numbers + a short verbatim quote as evidence — never invent citations) / why review missed it / **detection pattern** (the payload: a concrete check or grep a reviewer can run, not advice). Then ask me which to record — use a checkbox selection.
|
||||
|
||||
5. **Append only the approved entries**, idempotently, one at a time with exactly this shell shape (title-match skip; numbering continues from the ledger's current count): `f="$HOME/.config/coyote/share/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; grep -qF "<title>" "$f" || { n=$(grep -c '^## MISS-' "$f"); printf '\n## MISS-%03d — <title>\n- date: %s\n- repo: <repos>\n- surface: <surface>\n- what escaped: <what>\n- why review missed it: <why>\n- detection pattern: <detection>\n' "$((n+1))" "$(date +%F)" >> "$f"; }` — for sharpen-proposals, edit the existing entry in place instead. Finish by showing me the ledger's full `## MISS-` title list so I see the final state.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Record a review miss (a defect that ESCAPED review and was caught later) in the
|
||||
# review-miss ledger at ~/.config/coyote/share/review-misses.md. The first step appends
|
||||
# a numbered skeleton entry deterministically (zero tokens; it also migrates a ledger
|
||||
# from the legacy config-root location); the second has the agent interview you to fill
|
||||
# in the <fill> fields — the detection pattern is the payload.
|
||||
# Run as:
|
||||
# coyote --macro record-miss "<one line: what escaped and how it was caught>"
|
||||
# (avoid double quotes inside the description; it is spliced into a shell string)
|
||||
variables:
|
||||
- name: description
|
||||
description: One line describing what escaped review and how it was caught
|
||||
isolated: false
|
||||
steps:
|
||||
- |-
|
||||
!f="$HOME/.config/coyote/share/review-misses.md"; legacy="$HOME/.config/coyote/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$legacy" ] && [ ! -f "$f" ] && mv "$legacy" "$f"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; n=$(grep -c '^## MISS-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'MISS-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- repo: <fill owner/repo>\n- surface: <fill rest-api|worker|db-migration|cli|iac|library|ci-cd|any>\n- what escaped: {{description}}\n- why review missed it: <fill>\n- detection pattern: <fill the concrete check/grep a reviewer runs to catch this CLASS>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
|
||||
- |-
|
||||
Load the `review-misses` skill, then open `$HOME/.config/coyote/share/review-misses.md` and find the newest `## MISS-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **detection pattern**: it must be a concrete check or grep a reviewer can run, not advice. If my miss matches an existing entry's defect class, propose sharpening that entry instead of keeping a duplicate.
|
||||
Reference in new issue
Block a user