Compare commits
321
Commits
5fe3198df7
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2846d488b6
|
||
|
|
4c3c3e69b2
|
||
|
|
3a017d7fda | ||
|
|
53e0f3b54b | ||
|
|
22809e2a1b | ||
|
|
9ae26f4194 | ||
|
|
a3c4b6a364 | ||
|
|
72eb5a100c | ||
|
|
afab1b6da5 | ||
|
|
c9fcea2849 | ||
|
|
e7a7cd9704 | ||
|
|
2b3839881c | ||
|
|
c32ba9157e | ||
|
|
0442d77059 | ||
|
|
4e33979a92 | ||
|
|
fac1569646 | ||
|
|
6490128f07 | ||
|
|
3e496ec2fd | ||
|
|
57809ddf29 | ||
|
|
184712621e | ||
|
|
c45d73c94a | ||
|
|
4b2ab2f867 | ||
|
|
4cec912f8e | ||
|
|
e0bcf340fe | ||
|
|
9609e333f3 | ||
|
|
a8c0ec64fa | ||
|
|
37ba25adc5 | ||
|
|
a65280754e | ||
|
|
b6022654eb | ||
|
|
0c17076bdf | ||
|
|
a85d933d33 | ||
|
|
13640981c7 | ||
|
|
f106b101ab | ||
|
|
77322698e4 | ||
|
|
e237845827 | ||
|
|
4571ed647e | ||
|
|
0b55f7c735 | ||
|
|
461245a57c | ||
|
|
dcf2d99fa9 | ||
|
|
d524328d86 | ||
|
|
e002bd6727 | ||
|
|
56ed4664cd | ||
|
|
3ee78a46df | ||
|
|
dc6c176b4f | ||
|
|
00de9d40d1 | ||
|
|
08dfbac6f6 | ||
|
|
74e7bc54eb | ||
|
|
dc2f26d0e6 | ||
|
|
4e53e143bb | ||
|
|
166a3807f8 | ||
|
|
04b983459b | ||
|
|
fcdcb303a3 | ||
|
|
5bd1174ddd | ||
|
|
bfa058130a | ||
|
|
1c73b82812 | ||
|
|
9e811c088f | ||
|
|
02b9986807 | ||
|
|
ad9b319149 | ||
|
|
611900057c | ||
|
|
ec66494e8f | ||
|
|
3d37ed237a | ||
|
|
83978617bc | ||
|
|
bf10aeb74d | ||
|
|
c2e30c12a6 | ||
|
|
879cf8e097 | ||
|
|
22078b1ddc | ||
|
|
f7d0f93fa1 | ||
|
|
4b2428b334 | ||
|
|
dee89d4ca3 | ||
|
|
90f7631b35 | ||
|
|
6ae093a482 | ||
|
|
c73d5317a0 | ||
|
|
a3b71a6889 | ||
|
|
739c1be8bb | ||
|
|
1744d2ba52 | ||
|
|
6804f198bc | ||
|
|
7ace1731c6 | ||
|
|
89a9398c28 | ||
|
|
e7bd75947c | ||
|
|
d89d24aff4 | ||
|
|
f59421844a | ||
|
|
3dc84f5aef | ||
|
|
23fe2c1231 | ||
|
|
9da0b91673 | ||
|
|
5495813baf | ||
|
|
717a48bef2 | ||
|
|
6514588e05 | ||
|
|
73cf4a625b | ||
|
|
4087f3b3d0 | ||
|
|
0defc61bec | ||
|
|
7155b1b4fb | ||
|
|
42380d4b82 | ||
|
|
e62502c862 | ||
|
|
fc3bb259cb | ||
|
|
0c6b7c8594 | ||
|
|
8710737964 | ||
|
|
363104fe77 | ||
|
|
8f775dc2d9 | ||
|
|
2731e98ed1 | ||
|
|
0c2a68ecc4 | ||
|
|
bd7cad21e2 | ||
|
|
0c4d0d777c | ||
|
|
b020e1b75e | ||
|
|
9693cdd4ac | ||
|
|
a0cc5f291e | ||
|
|
ce20927713 | ||
|
|
9e7c957a3d | ||
|
|
3b66101361 | ||
|
|
0dc9dd91c3 | ||
|
|
9825ccd31e | ||
|
|
688ababecc | ||
|
|
d4605794e8 | ||
|
|
3a2736b667 | ||
|
|
003d036e31 | ||
|
|
e4dfbe906c | ||
|
|
57c1a86195 | ||
|
|
8be9c8e152 | ||
|
|
fa7f96e9e8 | ||
|
|
5046aebca3 | ||
|
|
1d6823ca0c | ||
|
|
9fb90be851 | ||
|
|
4cc5563ba8 | ||
|
|
fcc0049ab2 | ||
|
|
dca9b1553e | ||
|
|
ea31236d29 | ||
|
|
d16db663df | ||
|
|
94d48fb76e | ||
|
|
6c50c60c22 | ||
|
|
f31885ba07 | ||
|
|
fd26f43b84 | ||
|
|
e240be8bda | ||
|
|
9e2f3c0f12 | ||
|
|
8746e3e29a | ||
|
|
5b9852d80e | ||
|
|
99d4b20ac3 | ||
|
|
17864c8864 | ||
|
|
2a04c4f5e1 | ||
|
|
ca58176ef9 | ||
|
|
cfb83a629e | ||
|
|
8377f8287b | ||
|
|
ffb75cb168 | ||
|
|
58b0dbd812 | ||
|
|
bd5c0a7cdd | ||
|
|
d84dea21b6 | ||
|
|
755b25f9b2 | ||
|
|
23f941b09f | ||
|
|
77da48a40f | ||
|
|
5d59adadbc | ||
|
|
9a6ffcb8ca | ||
|
|
62e6fafaef | ||
|
|
beca133082 | ||
|
|
c7ec4ed124 | ||
|
|
cee5cf0f36 | ||
|
|
85cb7b0fbf | ||
|
|
d89592fee6 | ||
|
|
342f39f30e | ||
|
|
4f0c7e4431 | ||
|
|
fcf67b07a7 | ||
|
|
78e326039d | ||
|
|
f18c436e2c | ||
|
|
786e5542d4 | ||
|
|
18f5ada585 | ||
|
|
b90e04f227 | ||
|
|
3339920377 | ||
|
|
9a505f951a | ||
|
|
b87367bb1b | ||
|
|
ad3a5c6d49 | ||
|
|
68e893d684 | ||
|
|
156f3193a9 | ||
|
|
4555af8299 | ||
|
|
ea76331c49 | ||
|
|
84b5a72eac | ||
|
|
27101909e0 | ||
|
|
300c7aeff0 | ||
|
|
00c3b60c2f | ||
|
|
22421a4e74 | ||
|
|
fa5ee5f4fc | ||
|
|
ba3a904baf | ||
|
|
8e42a59b98 | ||
|
|
081fa2c9d4 | ||
|
|
48753eaa46 | ||
|
|
62ecb5b415 | ||
|
|
7ada3fb47c | ||
|
|
1cf42fffa2 | ||
|
|
66b30bb06a | ||
|
|
3f49db86b2 | ||
|
|
7dac4620a8 | ||
|
|
5b4462135f | ||
|
|
fdae98adff | ||
|
|
f230c28bf4 | ||
|
|
13b8eb87a9 | ||
|
|
0ce2654173 | ||
|
|
c98b6bc9cd | ||
|
|
f2d90ed7a6 | ||
|
|
22976a79f9 | ||
|
|
4263d684e9 | ||
|
|
a8496aa50e | ||
|
|
4e72c6f5e6 | ||
|
|
ebb9e9af62 | ||
|
|
80c20afcdd | ||
|
|
8407729d5a | ||
|
|
1e46743420 | ||
|
|
d50059a56f | ||
|
|
3dcb9541b1 | ||
|
|
7dc950a092 | ||
|
|
80ace77e44 | ||
|
|
18875a4cc9 | ||
|
|
75cc2a4b9c | ||
|
|
be0c5462ed | ||
|
|
1c68331738 | ||
|
|
677bce18b8 | ||
|
|
32625cdc76 | ||
|
|
50d1e78f6d | ||
|
|
2d43745b57 | ||
|
|
b8466e4284 | ||
|
|
d5d6dd9df4 | ||
|
|
fca086ba09 | ||
|
|
3254fe9f3c | ||
|
|
a7528d6e1a | ||
|
|
442953ef7d | ||
|
|
1ec7916816 | ||
|
|
eb7d1421da | ||
|
|
f97a83f165 | ||
|
|
3599b0a861 | ||
|
|
2e7cd6581e | ||
|
|
406a850bdb | ||
|
|
99e592565a | ||
|
|
838335f84d | ||
|
|
b50ec42868 | ||
|
|
e744da8870 | ||
|
|
eb78a0e32b | ||
|
|
e8237bd924 | ||
|
|
2f85462e2a | ||
|
|
3e91813cff | ||
|
|
99fb6e24f1 | ||
|
|
24b5504e21 | ||
|
|
02e2227aa0 | ||
|
|
778aa6449c | ||
|
|
a9f89f0414 | ||
|
|
f072aa1b2c | ||
|
|
32f4adb71f | ||
|
|
b4c01e0f28 | ||
|
|
1a5824ce4f | ||
|
|
324dd3397a | ||
|
|
727cab11ff | ||
|
|
ccec38c5b9 | ||
|
|
cce8381962 | ||
|
|
1ea755e864 | ||
|
|
6aad82c6fd | ||
|
|
aa1287b902 | ||
|
|
d7a16363b8 | ||
|
|
36601d8668 | ||
|
|
8e9a1f1ee8 | ||
|
|
8f9284cb92 | ||
|
|
69a3d928ee | ||
|
|
4c5aa8d895 | ||
|
|
3aa35e1259 | ||
|
|
4dee1a6849 | ||
|
|
f342dd7dd7 | ||
|
|
cd0318f384 | ||
|
|
d630c7e30b | ||
|
|
b666b7e494 | ||
|
|
e26543f21a | ||
|
|
10b8fe5cae | ||
|
|
3393123baf | ||
|
|
0bef0f2c39 | ||
|
|
874ef7f6ec | ||
|
|
e746ca1f87 | ||
|
|
dc5e3201da | ||
|
|
cdbff88b4f | ||
|
|
62122fc698 | ||
|
|
31c33b5a23 | ||
|
|
2f839b332d | ||
|
|
b65aa76d2d | ||
|
|
d8f1778bc7 | ||
|
|
c10965cd71 | ||
|
|
b4a73aae7f | ||
|
|
846bc39263 | ||
|
|
824edf587b | ||
|
|
0083cf0c42 | ||
|
|
f1cf9a6ae7 | ||
|
|
2049316364 | ||
|
|
a7744e7f89 | ||
|
|
24088a48a1 | ||
|
|
1df82b50b6 | ||
|
|
452d9e1b60 | ||
|
|
0f0f5448ef | ||
|
|
6e1683393f | ||
|
|
d71cf75209 | ||
|
|
ddce9e330e | ||
|
|
2928125b0e | ||
|
|
71c3283b87 | ||
|
|
47edaa35ff | ||
|
|
7e84804ed1 | ||
|
|
7f5b75dbdf | ||
|
|
3865750f5a | ||
|
|
dbec2f77d1 | ||
|
|
6a4352d8b4 | ||
|
|
3e8ecdbeab | ||
|
|
13ebe04472 | ||
|
|
695153f88e | ||
|
|
6104a48e56 | ||
|
|
a5984eca80 | ||
|
|
4651658b34 | ||
|
|
fc8114433b | ||
|
|
7c09593c84 | ||
|
|
e0845ff8e9 | ||
|
|
ed41844d6f | ||
|
|
34f60ead6c | ||
|
|
106813263a | ||
|
|
ff6289c89a | ||
|
|
cec3453da8 | ||
|
|
66ced35943 | ||
|
|
d6dc6c96d6 | ||
|
|
cb0c3a9653 | ||
|
|
a0dc4550b1 | ||
|
|
f66036e9d8 | ||
|
|
61c3350ad3 | ||
|
|
9f389ba82b | ||
|
|
2ee4e58933 | ||
|
|
eaab57375e |
No files matched your search
@@ -1 +1,7 @@
|
|||||||
assets/config-template.yaml text eol=lf
|
assets/config-template.yaml text eol=lf
|
||||||
|
assets/hooks/*.sh text eol=lf
|
||||||
|
assets/agents/** text eol=lf
|
||||||
|
assets/functions/tools/* text eol=lf
|
||||||
|
assets/functions/scripts/* text eol=lf
|
||||||
|
assets/functions/utils/*.sh text eol=lf
|
||||||
|
scripts/*.sh text eol=lf
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
### AI assistance (if any):
|
|
||||||
- List tools here and files touched by them
|
|
||||||
|
|
||||||
### Authorship & Understanding
|
|
||||||
|
|
||||||
- [ ] I wrote or heavily modified this code myself
|
|
||||||
- [ ] I understand how it works end-to-end
|
|
||||||
- [ ] I can maintain this code in the future
|
|
||||||
- [ ] No undisclosed AI-generated code was used
|
|
||||||
- [ ] If AI assistance was used, it is documented below
|
|
||||||
|
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
### Description
|
||||||
|
<!-- What problem is this PR solving? -->
|
||||||
|
|
||||||
|
### Alternatives Considered (Optional)
|
||||||
|
<!-- What alternatives did you consider? Why did you choose this approach? -->
|
||||||
|
|
||||||
|
### AI assistance (if any)
|
||||||
|
<!-- Required if AI assistance was used to write or modify code in this PR. -->
|
||||||
|
<!-- List tools here -->
|
||||||
|
|
||||||
|
### Authorship & Understanding
|
||||||
|
|
||||||
|
- [ ] I wrote or heavily modified this code myself. If I didn't, I gave proper attribution to AI assistance above.
|
||||||
|
- [ ] I understand how it works end-to-end
|
||||||
|
- [ ] I can maintain this code in the future
|
||||||
|
- [ ] No undisclosed AI-generated code was used
|
||||||
|
- [ ] If AI assistance was used, it is documented above
|
||||||
|
|
||||||
@@ -3,7 +3,7 @@ name: CI
|
|||||||
on:
|
on:
|
||||||
pull_request:
|
pull_request:
|
||||||
branches:
|
branches:
|
||||||
- '*'
|
- '**'
|
||||||
push:
|
push:
|
||||||
branches:
|
branches:
|
||||||
- main
|
- main
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ on:
|
|||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
contents: write
|
contents: read
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
group: update-models
|
group: update-models
|
||||||
@@ -26,70 +26,65 @@ jobs:
|
|||||||
RUSTFLAGS: --deny warnings
|
RUSTFLAGS: --deny warnings
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
with:
|
with:
|
||||||
persist-credentials: true
|
ssh-key: ${{ secrets.RELEASE_BOT_SSH_KEY }}
|
||||||
|
persist-credentials: true
|
||||||
|
|
||||||
- name: Install Rust Toolchain Components
|
- name: Install Rust Toolchain Components
|
||||||
uses: dtolnay/rust-toolchain@stable
|
uses: dtolnay/rust-toolchain@stable
|
||||||
|
|
||||||
- uses: Swatinem/rust-cache@v2
|
- uses: Swatinem/rust-cache@v2
|
||||||
|
|
||||||
- name: Cache DuckDB Extensions
|
- name: Cache DuckDB Extensions
|
||||||
id: duckdb-extensions
|
id: duckdb-extensions
|
||||||
uses: actions/cache@v4
|
uses: actions/cache@v4
|
||||||
with:
|
with:
|
||||||
path: ~/.duckdb/extensions
|
path: ~/.duckdb/extensions
|
||||||
key: duckdb-ext-ubuntu-latest-${{ hashFiles('Cargo.lock') }}
|
key: duckdb-ext-ubuntu-latest-${{ hashFiles('Cargo.lock') }}
|
||||||
|
|
||||||
- name: Install DuckDB Extensions
|
- name: Install DuckDB Extensions
|
||||||
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
|
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
|
||||||
run: cargo test --all duckdb
|
run: cargo test --all duckdb
|
||||||
|
|
||||||
# Coyote loads its config before handling --generate-models/--list-models, and a
|
- name: Write stub config
|
||||||
# bare non-TTY runner has no config file (and no way to answer the first-run
|
run: |
|
||||||
# prompt), so point COYOTE_CONFIG_DIR at a throwaway dir with a minimal stub.
|
COYOTE_CONFIG_DIR="$(mktemp -d)"
|
||||||
- name: Write stub config
|
printf 'clients:\n - type: openai\n - type: claude\n' > "$COYOTE_CONFIG_DIR/config.yaml"
|
||||||
run: |
|
echo "COYOTE_CONFIG_DIR=$COYOTE_CONFIG_DIR" >> "$GITHUB_ENV"
|
||||||
COYOTE_CONFIG_DIR="$(mktemp -d)"
|
|
||||||
printf 'clients:\n - type: openai\n - type: claude\n' > "$COYOTE_CONFIG_DIR/config.yaml"
|
|
||||||
echo "COYOTE_CONFIG_DIR=$COYOTE_CONFIG_DIR" >> "$GITHUB_ENV"
|
|
||||||
|
|
||||||
- name: Regenerate model catalog
|
- name: Regenerate model catalog
|
||||||
run: cargo run --release -- --generate-models
|
run: cargo run --release -- --generate-models
|
||||||
|
|
||||||
- name: Detect catalog changes
|
- name: Detect catalog changes
|
||||||
id: diff
|
id: diff
|
||||||
run: echo "changed=$(git diff --quiet models.yaml && echo false || echo true)" >> "$GITHUB_OUTPUT"
|
run: echo "changed=$(git diff --quiet models.yaml && echo false || echo true)" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# The binary built by the regeneration step embedded the OLD models.yaml (the file
|
# The binary built by the regeneration step embedded the OLD models.yaml (the file
|
||||||
# was rewritten after that build), so rebuild before smoke-testing: cargo tracks
|
# was rewritten after that build), so rebuild before smoke-testing: cargo tracks
|
||||||
# include_str! inputs, making this a cheap incremental rebuild of the final crate.
|
# include_str! inputs, making this a cheap incremental rebuild of the final crate.
|
||||||
- name: Rebuild with the refreshed catalog
|
- name: Rebuild with the refreshed catalog
|
||||||
if: steps.diff.outputs.changed == 'true'
|
if: steps.diff.outputs.changed == 'true'
|
||||||
run: cargo build --release
|
run: cargo build --release
|
||||||
|
|
||||||
- name: Test
|
- name: Test
|
||||||
if: steps.diff.outputs.changed == 'true'
|
if: steps.diff.outputs.changed == 'true'
|
||||||
run: cargo test --all
|
run: cargo test --all
|
||||||
|
|
||||||
- name: Smoke test the refreshed catalog
|
- name: Smoke test the refreshed catalog
|
||||||
if: steps.diff.outputs.changed == 'true'
|
if: steps.diff.outputs.changed == 'true'
|
||||||
run: |
|
run: |
|
||||||
COUNT=$(./target/release/coyote --list-models | wc -l)
|
COUNT=$(./target/release/coyote --list-models | wc -l)
|
||||||
echo "Listed $COUNT chat models"
|
echo "Listed $COUNT chat models"
|
||||||
test "$COUNT" -ge 40
|
test "$COUNT" -ge 40
|
||||||
|
|
||||||
# Pushes made with the default GITHUB_TOKEN do not trigger other workflows,
|
- name: Commit and push
|
||||||
# so this push cannot start a workflow loop. The pull --rebase covers a human
|
if: steps.diff.outputs.changed == 'true'
|
||||||
# push landing mid-run; if the race is lost anyway, the next cron run retries.
|
run: |
|
||||||
- name: Commit and push
|
git add models.yaml
|
||||||
if: steps.diff.outputs.changed == 'true'
|
git checkout -- .
|
||||||
run: |
|
git config user.name 'github-actions[bot]'
|
||||||
git add models.yaml
|
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
||||||
git checkout -- .
|
git commit -m 'chore: refresh generated model catalog [skip ci]'
|
||||||
git config user.name 'github-actions[bot]'
|
git pull --rebase origin main
|
||||||
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
git push origin main
|
||||||
git commit -m 'chore: refresh generated model catalog'
|
|
||||||
git pull --rebase origin main
|
|
||||||
git push origin main
|
|
||||||
+149
@@ -1,3 +1,152 @@
|
|||||||
|
## v0.10.3 (2026-09-15)
|
||||||
|
|
||||||
|
### Feat
|
||||||
|
|
||||||
|
- create a config alias for default_reasoning_level (to match some wording and make the config flexible)
|
||||||
|
- support custom resource definitions for openai-compatible Oauth customization
|
||||||
|
- Support specifying the compression model to use when compressing a session for greater cost controls
|
||||||
|
- Added full cost tracking to Coyote and added it to the default right prompt
|
||||||
|
- added support for input/output token cache cost tracking
|
||||||
|
- Support persisting paused todo lists to sessions so users can exit and resume paused todo sessions; Also fixed a bug so todo list pausing is only supported when being used in a session since it typically requires multi-turn communication
|
||||||
|
- Support token caching to reduce LLM costs in Anthropic models (not typically handled automatically by APIs)
|
||||||
|
- **TASK-030**: callers escalate on GAUNTLET_REVIEW_INCOMPLETE instead of findings-routing
|
||||||
|
- **TASK-032**: review-gauntlet incomplete-review machine line + ×N attempt details
|
||||||
|
- **TASK-029**: review-gauntlet fault-only lane re-run loop
|
||||||
|
- **TASK-028**: escalation_timeout 0 waits indefinitely and is the default
|
||||||
|
- **graph**: stream llm-node completions so node timeout governs long generations
|
||||||
|
- **TASK-023**: deep-research hardening — fail-safe scripts, fault fallbacks, pipeline notes
|
||||||
|
- **TASK-020**: finding-verifier + step-runner fail-closed hardening
|
||||||
|
- **TASK-019**: code-reviewer fail-closed review pipeline
|
||||||
|
- **TASK-018**: review-gauntlet lane degradation + verification passthrough
|
||||||
|
- **TASK-017**: adversary execution evidence + fail-closed routing
|
||||||
|
- **TASK-016**: engine + test fixes from the Phase-1 re-review
|
||||||
|
- **TASK-011**: traverse agent fallback edges in map peer-discovery BFS
|
||||||
|
- **TASK-010**: add max_attempts to read-only agent lanes in bundled graphs
|
||||||
|
- **TASK-009**: add fallback routing to agent nodes
|
||||||
|
- **TASK-008**: add shared transient matcher and retry support to agent nodes
|
||||||
|
- Fixed graph agents with new agent variable passing functionality
|
||||||
|
- **graph**: warn when a declared variable is shadowed by an initial_state key
|
||||||
|
- **agents**: advertise declared agent variables in agent__list_available
|
||||||
|
- **agents**: allow passing agent variables to agent__spawn and agent__task_create
|
||||||
|
- seed variables into initial graph state
|
||||||
|
- Significant code review suite rewrite with improved graph agent features
|
||||||
|
- let install commands remove files that have been removed from bundles, both internal and external
|
||||||
|
- update the web_search_coyote tool to attempt to use the current model first then fallback to the configured model for web searches
|
||||||
|
- expose the current coyote model to tools
|
||||||
|
- removed temperature specification in deep-research and finding-verifier agents to make them more univerally usable
|
||||||
|
- **graph**: thread the graph abort into llm nodes and tool execution
|
||||||
|
- **graph**: abort a graph run on Ctrl-C from anywhere
|
||||||
|
- **graph**: warn when an llm node's turn cap is disabled
|
||||||
|
- **graph**: treat llm max_iterations: 0 as no turn cap
|
||||||
|
- **graph**: warn when the visit cap is disabled
|
||||||
|
- **graph**: treat max_loop_iterations: 0 as no visit cap
|
||||||
|
- **graph**: warn when a timeout is disabled
|
||||||
|
- **graph**: treat timeout: 0 as no wall-clock bound
|
||||||
|
- **graph**: validate agent node inputs at load time
|
||||||
|
- **graph**: seed a child graph agent's state from an agent node's inputs map
|
||||||
|
- **graph**: validate map branch subgraphs instead of single branch nodes
|
||||||
|
- **graph**: run map branches as per-item subgraph chains
|
||||||
|
- **graph**: accept a `{{template}}` for a map node's `max_concurrency`
|
||||||
|
- created the finding-verifier and whetstone agents for continuous code quality improvements
|
||||||
|
- teammates flag enabling peer messaging between concurrent graph agent nodes
|
||||||
|
- support explicit can_spawn_agents, max_concurrent_agents, and max_agent_depth keys in graph.yaml
|
||||||
|
- support setting the tool_timeout in the main config file instead of only via an environment variable
|
||||||
|
- Added new environment variable to force showing tool call logging when not in a tty
|
||||||
|
|
||||||
|
### Fix
|
||||||
|
|
||||||
|
- empty tool results can't cause errors in bedrock, claude, openai, or vertexai clients anymore
|
||||||
|
- proper support for reasoning/thinking in bedrock models
|
||||||
|
- **TASK-032**: enforce incomplete⇒BLOCKED structurally; anchor adversary degraded detection
|
||||||
|
- **TASK-028**: pin example-doc wording; make not-found escalation hint match queue semantics
|
||||||
|
- **TASK-028**: pin global-template exclusion, tighten timeout boundary test, reword stale hint
|
||||||
|
- **adversary**: red verification runs block CONFORMS; re-derive timeout envelope; robust runner
|
||||||
|
- **review-suite**: detect nested finding-verifier faults; harden render/fault markers
|
||||||
|
- **review-gauntlet**: summary-line critical count, nested-fault hook, fail-closed lane fallback
|
||||||
|
- **graph**: typed ApiStatusError transient classification; warn on bare output_schema
|
||||||
|
- **review-suite**: mark example verification commands as placeholders; pin variable description and prompt note
|
||||||
|
- **graph**: auto-merge only schema-declared keys from llm output
|
||||||
|
- **review-suite**: pass verification_commands as a declared variable, never LLM-extracted from prose
|
||||||
|
- **agents**: scrub GRAPH_STATE* from adversary verification commands; raise review-suite max_agent_depth
|
||||||
|
- **adversary**: narrow died-marker predicate, degraded banner for died criteria, hermetic script tests
|
||||||
|
- **adversary**: fail closed on PIPELINE-FAULT evidence regardless of status; pin bail-chain and validator shape
|
||||||
|
- **adversary**: fail closed per criterion when check_criterion dies
|
||||||
|
- **engine**: carry full error chain in llm node fault text
|
||||||
|
- **TASK-020**: skip POSIX bash script tests on Windows
|
||||||
|
- **TASK-020**: surface step-runner LLM-fault captures where the fallback lands
|
||||||
|
- **TASK-019**: carry PIPELINE-FAULT prefix through verdict.py's crash guard
|
||||||
|
- **TASK-018**: harden verdict_gate fault routing and widen default_lanes fallback
|
||||||
|
- **TASK-017**: bound run_checks runtime and fail-close its node-level death
|
||||||
|
- **tests**: collision-proof parser test temp files to deflake macOS CI
|
||||||
|
- Don't infinitely compress a session once it passes a threshold when the threshold is close to the max token count
|
||||||
|
- fix spinner ctrl-c early exit not clearing message and restoring cursor
|
||||||
|
- **graph**: build peer retirement guards before spawning their tasks
|
||||||
|
- **graph**: reject agents that ship both config.yaml and graph.yaml at load
|
||||||
|
- **graph**: bound concurrency caps by the semaphore permit limit
|
||||||
|
- **graph**: make turn-cap and template checks consistent across validation modes
|
||||||
|
- **graph**: leave output absent after state_updates when it was absent before
|
||||||
|
- **graph**: validate the whole max_concurrency template string
|
||||||
|
- **graph**: reject as == output_key on map nodes
|
||||||
|
- **graph**: inherit the session abort in agent-node children
|
||||||
|
- **graph**: retire the teammate identity before structured extraction
|
||||||
|
- **graph**: reject a literal max_concurrency of 0 instead of clamping it
|
||||||
|
- **graph**: clear peer identity fields after every map chain step
|
||||||
|
- **graph**: pick the map teammate identity name by BFS from the branch entry
|
||||||
|
- **graph**: name the offending node type when a map branch reaches a disallowed node
|
||||||
|
- **function**: build bundled bash tools without writing back to the source tree
|
||||||
|
- **agents**: name the map-branch output key in deep-research and finding-verifier
|
||||||
|
- **graph**: honor an explicit `output` key in state_updates
|
||||||
|
- Fixed the bad formatting in the finding-verifier agent
|
||||||
|
- properly anchored the role frontmatter regex so it doesn't catch in-instruction separators
|
||||||
|
- support non-object tool result outputs for bedrock
|
||||||
|
- exclude null bytes from fs_grep output on binaries
|
||||||
|
- suppress subagent error and warning output to stderr/stdout
|
||||||
|
|
||||||
|
### Refactor
|
||||||
|
|
||||||
|
- **graph**: keep the ordered visit log test-only
|
||||||
|
- **graph**: move the cfg(test) OUTPUT_KEY import into llm's test module
|
||||||
|
- **graph**: drop the duplicated locator prefix from map chain step errors
|
||||||
|
|
||||||
|
## v0.10.2 (2026-09-04)
|
||||||
|
|
||||||
|
### Feat
|
||||||
|
|
||||||
|
- added a clear line separator for the end of LLM turns to make it more clear where they are
|
||||||
|
- sort models by release date
|
||||||
|
- Improved the architect, explore, and sisyphus agents to have a team mode so they can all work in parallel as a team and updated the parallel-research skill to make explicit mention of this functionality
|
||||||
|
- support auto continuation in subagents and prevent them from calling todo__pause
|
||||||
|
- apply provider quirks to models created from a name
|
||||||
|
- refresh the model catalog from live API data at runtime
|
||||||
|
- generate the model catalog from live API data
|
||||||
|
- add model catalog source adapters and merge engine
|
||||||
|
- add hand-owned model quirks overlay
|
||||||
|
- apply provider quirks to models created from a name
|
||||||
|
- refresh the model catalog from live API data at runtime
|
||||||
|
- generate the model catalog from live API data
|
||||||
|
- add model catalog source adapters and merge engine
|
||||||
|
- add hand-owned model quirks overlay
|
||||||
|
- Improved the architect, explore, and sisyphus agents to have a team mode so they can all work in parallel as a team and updated the parallel-research skill to make explicit mention of this functionality
|
||||||
|
- added tab completions to --update-bundle flag
|
||||||
|
- added a dedicated config_dir output to coyote --info/.info
|
||||||
|
|
||||||
|
### Fix
|
||||||
|
|
||||||
|
- children subagents couldn't properly send messages to parent inboxes
|
||||||
|
- --generate-models parsed before vault initialization
|
||||||
|
- drive tool rounds iteratively instead of recursively to prevent stack overflow on long agentic turns to fix latent stack overflow recursion bug
|
||||||
|
- session replays now include mid-turn messages so the full session gets replayed
|
||||||
|
- resolve duplicate model ids by requested type
|
||||||
|
- let ctrl-c interrupt blocking job and agent collects
|
||||||
|
- skip terminal color-scheme queries in test builds
|
||||||
|
- memoize quirk glob patterns and isolate concurrent override writes
|
||||||
|
- memoize quirk glob patterns and isolate concurrent override writes
|
||||||
|
- children subagents couldn't properly send messages to parent inboxes
|
||||||
|
|
||||||
|
### Perf
|
||||||
|
|
||||||
|
- probe models-override staleness in the background refresh task
|
||||||
|
|
||||||
## v0.10.1 (2026-08-31)
|
## v0.10.1 (2026-08-31)
|
||||||
|
|
||||||
### Fix
|
### Fix
|
||||||
|
|||||||
+3
-2
@@ -78,8 +78,9 @@ act -W .github/workflows/release.yml --input_type bump=minor
|
|||||||
|
|
||||||
## Authorship Policy
|
## Authorship Policy
|
||||||
|
|
||||||
All code in this repository is written and reviewed by humans. AI-generated code (e.g., Copilot, ChatGPT,
|
All code in this repository is written and reviewed by humans, or AI-generated where explicitly disclosed.
|
||||||
Claude, etc.) is not permitted unless explicitly disclosed and approved.
|
AI-generated code (e.g., Copilot, ChatGPT, Claude, etc.) is permitted only when explicitly disclosed and
|
||||||
|
approved. For example: disclosing the use of AI-generated code in a PR description.
|
||||||
|
|
||||||
Submissions must certify that the contributor understands and can maintain the code they submit.
|
Submissions must certify that the contributor understands and can maintain the code they submit.
|
||||||
|
|
||||||
|
|||||||
Generated
+332
-238
File diff suppressed because it is too large.
Load diff
+2
-1
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "coyote-ai"
|
name = "coyote-ai"
|
||||||
version = "0.10.1"
|
version = "0.10.3"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
authors = ["Alex Clarke <alex.j.tusa@gmail.com>"]
|
authors = ["Alex Clarke <alex.j.tusa@gmail.com>"]
|
||||||
description = "The batteries-included runtime for LLMs"
|
description = "The batteries-included runtime for LLMs"
|
||||||
@@ -146,6 +146,7 @@ ctor = "1.0.13"
|
|||||||
pretty_assertions = "1.4.0"
|
pretty_assertions = "1.4.0"
|
||||||
rmcp = { version = "3.1.2", features = ["server"] }
|
rmcp = { version = "3.1.2", features = ["server"] }
|
||||||
serial_test = "3"
|
serial_test = "3"
|
||||||
|
tokio = { version = "1.34.0", features = ["test-util"] }
|
||||||
|
|
||||||
[[bin]]
|
[[bin]]
|
||||||
name = "coyote"
|
name = "coyote"
|
||||||
|
|||||||
+6
-1
@@ -82,6 +82,8 @@ RUN set -euo pipefail; \
|
|||||||
chown 1000:1000 /home/agent/.cargo/bin/coyote; \
|
chown 1000:1000 /home/agent/.cargo/bin/coyote; \
|
||||||
rm -rf "$TMPDIR"
|
rm -rf "$TMPDIR"
|
||||||
|
|
||||||
|
COPY --chmod=0755 scripts/docker-entrypoint.sh /usr/local/bin/coyote-entrypoint
|
||||||
|
|
||||||
FROM scratch
|
FROM scratch
|
||||||
|
|
||||||
ARG COYOTE_VERSION
|
ARG COYOTE_VERSION
|
||||||
@@ -107,4 +109,7 @@ WORKDIR /home/agent/workspace
|
|||||||
|
|
||||||
USER 1000
|
USER 1000
|
||||||
|
|
||||||
ENTRYPOINT ["coyote"]
|
# tini as PID 1 (as in the sandbox-templates base) reaps orphans and forwards
|
||||||
|
# signals; coyote-entrypoint runs coyote by default, or the given command
|
||||||
|
# (e.g. the Docker Sandboxes keep-alive) when the first arg is sh/bash/a path.
|
||||||
|
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/coyote-entrypoint"]
|
||||||
@@ -56,6 +56,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g
|
|||||||
* [Agents](https://github.com/Dark-Alex-17/coyote/wiki/Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools.
|
* [Agents](https://github.com/Dark-Alex-17/coyote/wiki/Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools.
|
||||||
* [Graph Agents](https://github.com/Dark-Alex-17/coyote/wiki/Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns).
|
* [Graph Agents](https://github.com/Dark-Alex-17/coyote/wiki/Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns).
|
||||||
* [Background Jobs](https://github.com/Dark-Alex-17/coyote/wiki/Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working, and completion arrives as a push notification.
|
* [Background Jobs](https://github.com/Dark-Alex-17/coyote/wiki/Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working, and completion arrives as a push notification.
|
||||||
|
* [Hooks](https://github.com/Dark-Alex-17/coyote/wiki/Hooks): Run your own fire-and-forget commands when lifecycle events fire. Turns, tools, LLM requests, agents, sub-agent escalations, and more (26 events). Perfect for desktop notifications, activity logging, and downstream automation; hooks never block Coyote or change its behavior.
|
||||||
* [Todo System](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System): Built-in task tracking for improved LLM reliability with smaller models.
|
* [Todo System](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System): Built-in task tracking for improved LLM reliability with smaller models.
|
||||||
* [Environment Variables](https://github.com/Dark-Alex-17/coyote/wiki/Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables.
|
* [Environment Variables](https://github.com/Dark-Alex-17/coyote/wiki/Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables.
|
||||||
* [Client Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Clients): Configuration instructions for various LLM providers.
|
* [Client Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Clients): Configuration instructions for various LLM providers.
|
||||||
@@ -143,6 +144,13 @@ docker run --rm -it \
|
|||||||
darkalex17/coyote
|
darkalex17/coyote
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Arguments are passed to `coyote`. If the first argument is `sh`, `bash`, or an absolute path, that command runs instead,
|
||||||
|
e.g. to get a shell inside the image:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm -it darkalex17/coyote bash
|
||||||
|
```
|
||||||
|
|
||||||
### Scripts
|
### Scripts
|
||||||
#### Linux/MacOS (`bash`)
|
#### Linux/MacOS (`bash`)
|
||||||
You can use the following command to run a bash script that downloads and installs the latest version of `coyote` for your
|
You can use the following command to run a bash script that downloads and installs the latest version of `coyote` for your
|
||||||
|
|||||||
@@ -1,5 +1,16 @@
|
|||||||
# Adversary
|
# Adversary
|
||||||
|
|
||||||
|
> [!IMPORTANT]
|
||||||
|
> **v2: now a GRAPH agent.** The plan's acceptance criteria are extracted into a list and the graph
|
||||||
|
> fans out ONE fresh-context verification branch per criterion — partial coverage ("checked 8 of 12")
|
||||||
|
> is structurally impossible. Each verdict passes a deterministic contract gate (MET requires
|
||||||
|
> evidence including a proving test; no test = at best PARTIAL), a holistic ADDITIVE-only pass hunts
|
||||||
|
> absence/scope-drift/substitution/gamed-tests, and the CONFORMS/DIVERGES sentinel is computed by a
|
||||||
|
> script (missing plan = DIVERGES, fail-closed). The conformance doctrine below is structurally
|
||||||
|
> enforced: CONFORMS requires ALL MET + zero holistic complaints + zero pipeline faults +
|
||||||
|
> zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
|
||||||
|
|
||||||
|
|
||||||
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
|
An **adversarial plan-conformance reviewer**. Where [`code-reviewer`](../code-reviewer/README.md)
|
||||||
asks *"is this code good?"*, `adversary` asks a different, harder question:
|
asks *"is this code good?"*, `adversary` asks a different, harder question:
|
||||||
|
|
||||||
@@ -31,6 +42,9 @@ ADVERSARIAL_REVIEW: CONFORMS
|
|||||||
Criteria: N/N met (all with tests).
|
Criteria: N/N met (all with tests).
|
||||||
```
|
```
|
||||||
|
|
||||||
|
CONFORMS is deterministic: ALL MET + zero holistic complaints + zero pipeline faults +
|
||||||
|
zero red verification runs (exit≠0/timeout/skipped, or a runner error when commands were declared).
|
||||||
|
|
||||||
```
|
```
|
||||||
ADVERSARIAL_REVIEW: DIVERGES
|
ADVERSARIAL_REVIEW: DIVERGES
|
||||||
Criteria: X/N met, Y partial, Z unmet/diverged.
|
Criteria: X/N met, Y partial, Z unmet/diverged.
|
||||||
@@ -39,6 +53,12 @@ Complaints:
|
|||||||
2. ...
|
2. ...
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Every verdict — CONFORMS or DIVERGES — ends with a `Verification runs:` section: the recorded
|
||||||
|
PASS/FAIL results (command, exit code, duration, failure tail) of the caller-declared verification
|
||||||
|
commands, or an explicit marker when none were declared or the run never reached that stage. The
|
||||||
|
one exception is the verdict script's own crash guard, which emits a bare DIVERGES with a
|
||||||
|
"verdict computation error" line — and no section — when verdict computation itself dies.
|
||||||
|
|
||||||
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
|
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
|
||||||
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
|
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
|
||||||
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
|
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
|
||||||
@@ -48,18 +68,43 @@ violation) and cites `file:line`. Vague complaints are not emitted.
|
|||||||
|
|
||||||
## How it reviews
|
## How it reviews
|
||||||
|
|
||||||
Driven by the [`adversarial-review`](../../skills/adversarial-review/SKILL.md) skill:
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["parse (llm): extract acceptance criteria"] --> B["facts (diff_facts.py): resolve + bound the diff"]
|
||||||
|
V["verification_commands (declared graph variable — never parsed from the prompt)"] --> R
|
||||||
|
B --> R["run_checks (script): run the declared verification commands ONCE, record results"]
|
||||||
|
R --> C["map over criteria: one fresh-context branch each (parallel)"]
|
||||||
|
C --> D["check_criterion (llm + fs/ast tools)"]
|
||||||
|
D --> E["crit_gate (script): MET needs change+test evidence"]
|
||||||
|
E -. "reject-retry once; exhausted = PARTIAL" .-> D
|
||||||
|
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
|
||||||
|
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults + zero red runs; no criteria = DIVERGES"]
|
||||||
|
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
|
||||||
|
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
|
||||||
|
B -. "fallback" .-> P
|
||||||
|
R -. "fallback" .-> P
|
||||||
|
D -. "fallback" .-> CF["criterion_fault (script): UNMET + PIPELINE-FAULT evidence for that ONE criterion"]
|
||||||
|
CF -. "map join" .-> F
|
||||||
|
F -. "fallback" .-> G
|
||||||
|
P --> G
|
||||||
|
```
|
||||||
|
|
||||||
1. Map **every** acceptance criterion to specific evidence in the diff → ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. No test proving the behavior ⇒ at best ⚠️ Partial.
|
The conformance doctrine is baked into the graph's own nodes:
|
||||||
|
|
||||||
|
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
|
||||||
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
||||||
3. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL. Independently of the criterion judgments, the verdict script blocks CONFORMS on any red run (exit≠0/timeout/skipped) and on an ENVIRONMENT runner-error marker when commands were declared — the declared commands never ran, so the run is unproven. The commands arrive through the declared `verification_commands` graph variable (see [Declaring verification commands](#declaring-verification-commands)) — `parse` never extracts them from the prompt, so the prompt is no longer an extraction source for the commands.
|
||||||
|
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||||
|
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
|
||||||
|
6. **Fail closed per criterion**: a criterion check that dies (max_iterations exhausted, API failure) falls back to `criterion_fault`, which records a schema-shaped UNMET verdict carrying a `PIPELINE-FAULT: criterion ` evidence marker for that one criterion. The other criteria still get verdicts, the report says the check DIED rather than judged, and the run is DIVERGES. That exact marker prefix (the one `criterion_fault` emits) forces the criterion into unmet regardless of the status it carries, so a verdict that echoes it with `MET` can never count toward CONFORMS; `check_criterion` is told never to begin its own evidence with `PIPELINE-FAULT:`.
|
||||||
|
|
||||||
It is **read-only** — it produces a verdict, never a fix.
|
It is **read-only** — it produces a verdict, never a fix.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
Typically spawned by `sisyphus` (or `architect`) alongside `code-reviewer`. The spawn prompt IS its
|
Typically spawned by `sisyphus` (or `architect`) alongside `code-reviewer`. The spawn prompt IS its
|
||||||
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria:
|
entire context, so it must include the diff (or a base ref to fetch) **and** the acceptance criteria.
|
||||||
|
Verification commands travel separately, as a declared variable:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
agent__spawn --agent adversary --prompt "
|
agent__spawn --agent adversary --prompt "
|
||||||
@@ -71,24 +116,44 @@ Run get_diff (or --base main), or: <paste diff>
|
|||||||
|
|
||||||
## PLAN — acceptance criteria to check against
|
## PLAN — acceptance criteria to check against
|
||||||
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
|
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
|
||||||
"
|
" --variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
```
|
```
|
||||||
|
|
||||||
Direct invocation for ad-hoc use:
|
Direct invocation for ad-hoc use:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
coyote -a adversary --agent-variable project_dir /path/to/repo \
|
coyote -a adversary --agent-variable project_dir /path/to/repo \
|
||||||
|
--agent-variable verification_commands '["cargo test --all"]' \
|
||||||
"Review staged changes against these criteria: <paste criteria>"
|
"Review staged changes against these criteria: <paste criteria>"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Declaring verification commands
|
||||||
|
|
||||||
|
`verification_commands` is a declared graph variable (default `'[]'`): a JSON array of shell
|
||||||
|
commands, passed as a string because agent variables always arrive as strings. `run_checks` runs
|
||||||
|
exactly those commands with a shell — so the declaration is a **trust boundary**. The prompt is the
|
||||||
|
wrong side of it: it also carries plan and diff text pasted from the repo under review, and an LLM
|
||||||
|
`parse` node extracting commands from that prose would let anything in the repo's text run under
|
||||||
|
the reviewer's shell. Hence the commands are never in the prompt and never in `parse`'s output
|
||||||
|
schema; `review-gauntlet` forwards its own `verification_commands` variable to the adversary lane
|
||||||
|
as a structured `inputs:` passthrough, not as prompt text.
|
||||||
|
|
||||||
|
- `'[]'` (or omitted) ⇒ the "none declared" marker; execution criteria are at best PARTIAL.
|
||||||
|
- Anything that is not a JSON array of strings (unparsable JSON, an object, non-string items) is
|
||||||
|
an invalid declaration: `run_checks` executes **nothing** and records a
|
||||||
|
`PIPELINE-FAULT: verification_commands declaration invalid — …` marker, which forces DIVERGES:
|
||||||
|
it fails closed as a PIPELINE-FAULT (complaint #1) — it never degrades to an ENVIRONMENT marker.
|
||||||
|
- The engine merges only the keys declared under `parse`'s `output_schema.properties` into state,
|
||||||
|
so an injected instruction that makes the parse LLM emit an extra `verification_commands` key
|
||||||
|
cannot reach state: the undeclared key is dropped and the declared variable stands.
|
||||||
|
|
||||||
### Tools
|
### Tools
|
||||||
|
|
||||||
- `get_diff [--base <ref>]` — staged → unstaged → `HEAD~1` fallback (or an explicit base/PR branch).
|
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
|
||||||
- `get_changed_files [--base <ref>]` — quick changed-file map.
|
- The `run_checks` script executes the commands declared via the `verification_commands` variable and records the results — criterion branches cite the record and never run anything themselves.
|
||||||
- Plus read-only `fs_*` and `ast_grep` for ground-truth checks.
|
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [`adversarial-review`](../../skills/adversarial-review/SKILL.md) — the conformance methodology it runs on.
|
|
||||||
- [`code-reviewer`](../code-reviewer/README.md) — the quality reviewer it runs alongside.
|
- [`code-reviewer`](../code-reviewer/README.md) — the quality reviewer it runs alongside.
|
||||||
- [`plan-review`](../../skills/plan-review/SKILL.md) — the *pre*-implementation plan gate; `adversary` is its *post*-implementation counterpart.
|
- [`plan-review`](../../skills/plan-review/SKILL.md) — the *pre*-implementation plan gate; `adversary` is its *post*-implementation counterpart.
|
||||||
@@ -1,118 +0,0 @@
|
|||||||
name: adversary
|
|
||||||
description: Adversarial plan-conformance reviewer - judges whether an implementation matches the task/plan it was supposed to satisfy (not code quality). Returns a blocking CONFORMS/DIVERGES verdict. Complements code-reviewer. Designed to be delegated to by sisyphus.
|
|
||||||
version: 1.0.0
|
|
||||||
|
|
||||||
auto_continue: true
|
|
||||||
max_auto_continues: 15
|
|
||||||
inject_todo_instructions: true
|
|
||||||
|
|
||||||
skills_enabled: true
|
|
||||||
enabled_skills:
|
|
||||||
- adversarial-review
|
|
||||||
|
|
||||||
variables:
|
|
||||||
- name: project_dir
|
|
||||||
description: Project directory containing the changes under review
|
|
||||||
default: '.'
|
|
||||||
- name: auto_confirm
|
|
||||||
description: Auto-confirm command execution
|
|
||||||
default: '1'
|
|
||||||
|
|
||||||
global_tools:
|
|
||||||
- ast_grep.sh
|
|
||||||
- fs_read.sh
|
|
||||||
- fs_cat.sh
|
|
||||||
- fs_grep.sh
|
|
||||||
- fs_glob.sh
|
|
||||||
- fs_ls.sh
|
|
||||||
- execute_command.sh
|
|
||||||
|
|
||||||
instructions: |
|
|
||||||
You are an adversarial plan-conformance reviewer. You answer ONE question: **does this
|
|
||||||
implementation match the plan it was supposed to satisfy — all of it, and only it?** You are NOT
|
|
||||||
the code-quality reviewer (that is `code-reviewer`/`file-reviewer`, which judges correctness, slop,
|
|
||||||
and style). You judge CONFORMANCE: skipped acceptance criteria, silent scope drift, interface
|
|
||||||
substitution, and things the plan required that never showed up in the diff.
|
|
||||||
|
|
||||||
Your value is independence and suspicion. Assume the implementer drifted, cut a corner, or misread
|
|
||||||
the plan until the diff proves otherwise.
|
|
||||||
|
|
||||||
## Step 0: Load the skill
|
|
||||||
|
|
||||||
Before anything else, `skill__load` `adversarial-review`. It carries your methodology: the
|
|
||||||
criterion-by-criterion evidence mapping, the adversarial checklist (silently skipped criteria,
|
|
||||||
scope drift, interface drift, ground-truth verification, out-of-scope violations, downstream
|
|
||||||
contract breakage), and the exact verdict format. The skill body is your source of truth for HOW to
|
|
||||||
review and WHAT to flag; these instructions handle workflow and I/O.
|
|
||||||
|
|
||||||
## Input (the spawn prompt IS your entire context)
|
|
||||||
|
|
||||||
You are given:
|
|
||||||
1. **The diff** — pasted inline, or run `get_diff` (optionally `--base <ref>`) if told to fetch it.
|
|
||||||
2. **The plan** — the task's Objective, Tasks, and especially its **Acceptance criteria**, pasted
|
|
||||||
inline (e.g. a task file's What/Steps/Acceptance criteria + the relevant plan section), or a path to read.
|
|
||||||
|
|
||||||
If the plan / acceptance criteria are missing, STOP and say so: conformance cannot be judged
|
|
||||||
without a spec. Do not invent criteria or guess intent.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Load `adversarial-review`.
|
|
||||||
2. Get the diff (inline or via `get_diff`) and identify the changed files.
|
|
||||||
3. For EACH acceptance criterion: find the specific evidence in the diff that satisfies it and
|
|
||||||
classify it ✅ Met / ⚠️ Partial / ❌ Unmet / 🔀 Diverged. A criterion with no test proving its
|
|
||||||
behavior is at best ⚠️ Partial.
|
|
||||||
4. Ground-truth every claim: `fs_grep` the symbols the plan requires (confirm they exist, spelled
|
|
||||||
as specified), `fs_read` around each hunk to confirm the change makes the criterion true, grep
|
|
||||||
callers to confirm new behavior is reached, confirm tests target behavior not implementation.
|
|
||||||
Use `ast_grep` for structural checks (e.g. "was this function signature actually changed?").
|
|
||||||
5. Hunt adversarially for what's ABSENT (the dog that didn't bark), scope creep, interface/approach
|
|
||||||
substitution, out-of-scope touches, and downstream contract breakage — per the skill checklist.
|
|
||||||
6. Emit the verdict in the skill's exact format.
|
|
||||||
|
|
||||||
## Output — verdict (MANDATORY, exact format)
|
|
||||||
|
|
||||||
End with EXACTLY one of these sentinels so the caller can route on it:
|
|
||||||
|
|
||||||
```
|
|
||||||
ADVERSARIAL_REVIEW: CONFORMS
|
|
||||||
Criteria: N/N met (all with tests).
|
|
||||||
<optional: 1-3 non-blocking observations>
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
ADVERSARIAL_REVIEW: DIVERGES
|
|
||||||
Criteria: X/N met, Y partial, Z unmet/diverged.
|
|
||||||
Complaints:
|
|
||||||
1. Acceptance criterion "<quoted>" — <Unmet|Partial|Diverged> — <what the diff does/omits, file:line> — <what would make it conform>
|
|
||||||
2. Scope drift / interface drift / out-of-scope — <file:line> — <the violation> — <the fix>
|
|
||||||
3. ...
|
|
||||||
```
|
|
||||||
|
|
||||||
Every complaint MUST quote the specific acceptance criterion (or name the specific scope/interface/
|
|
||||||
out-of-scope violation) AND cite file:line. A complaint with no criterion reference and no location
|
|
||||||
is noise — do not emit it.
|
|
||||||
|
|
||||||
## Rules
|
|
||||||
|
|
||||||
1. **You are read-only.** Never modify files. You produce a verdict; the implementer owns the fix.
|
|
||||||
2. **Conformance, not quality.** Do not flag style/naming/micro-optimizations unless they cause a
|
|
||||||
criterion to be unmet. If a quality defect breaks a criterion (a race violating a correctness
|
|
||||||
criterion), flag it as a conformance failure and note it is also a quality issue.
|
|
||||||
3. **No test ⇒ not met.** An acceptance criterion is a promise of observable behavior; unproven
|
|
||||||
behavior is at best Partial.
|
|
||||||
4. **Absence is a finding.** Review what SHOULD be in the diff per the plan, not only what IS.
|
|
||||||
5. **Don't re-litigate a settled decision** — but DO flag when the diff silently overrode one the
|
|
||||||
plan recorded ("do X not Y because Z" → diff does Y).
|
|
||||||
6. **The plan can be the culprit.** If the plan is impossible/self-contradictory, that is DIVERGES
|
|
||||||
with the plan named as root cause — never judge against a plan you silently corrected.
|
|
||||||
7. Be terse and decisive. Three real divergences beat fifteen weak ones. If everything is a nitpick,
|
|
||||||
it CONFORMS — say so.
|
|
||||||
|
|
||||||
## Context
|
|
||||||
- Project: {{project_dir}}
|
|
||||||
- CWD: {{__cwd__}}
|
|
||||||
- Shell: {{__shell__}}
|
|
||||||
|
|
||||||
## Available Tools
|
|
||||||
{{__tools__}}
|
|
||||||
@@ -0,0 +1,405 @@
|
|||||||
|
name: adversary
|
||||||
|
description: |
|
||||||
|
Graph-based adversarial plan-conformance reviewer. The plan's acceptance
|
||||||
|
criteria are extracted into a list and the graph mechanically fans out ONE
|
||||||
|
fresh-context verification branch per criterion — "checked 8 of 12 and waved
|
||||||
|
the rest through" becomes structurally impossible. Each criterion verdict
|
||||||
|
(MET / PARTIAL / UNMET / DIVERGED) passes a deterministic contract gate
|
||||||
|
(MET requires evidence including a test proving the behavior — no test means
|
||||||
|
at best PARTIAL). A holistic pass then hunts what per-criterion checks cannot
|
||||||
|
see: absence (the dog that didn't bark), scope drift, interface substitution,
|
||||||
|
silently-overridden recorded decisions, out-of-scope touches, and gamed
|
||||||
|
tests — ADDITIVE-only: it can add complaints, never flip a criterion verdict.
|
||||||
|
The final CONFORMS/DIVERGES sentinel is computed by a script: CONFORMS
|
||||||
|
requires every criterion MET, zero holistic complaints, zero pipeline faults,
|
||||||
|
and zero red verification runs (exit≠0/timeout/skipped, or a runner error
|
||||||
|
when commands were declared). Missing plan = DIVERGES (fail-closed) naming
|
||||||
|
the missing spec. Caller-declared verification
|
||||||
|
commands are executed ONCE by a deterministic stage and recorded — criterion
|
||||||
|
branches cite the record and never run anything themselves. Any pipeline
|
||||||
|
fault (parse, diff resolution, verification runner, or holistic failure)
|
||||||
|
lands a PIPELINE-FAULT marker and fails closed into DIVERGES: degraded is
|
||||||
|
never passing.
|
||||||
|
|
||||||
|
version: '1.0'
|
||||||
|
|
||||||
|
global_tools:
|
||||||
|
- ast_grep.sh
|
||||||
|
- fs_read.sh
|
||||||
|
- fs_cat.sh
|
||||||
|
- fs_grep.sh
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: project_dir
|
||||||
|
description: Project directory containing the changes under review
|
||||||
|
default: '.'
|
||||||
|
- name: verification_commands
|
||||||
|
description: |
|
||||||
|
JSON array of shell commands the CALLER declares for the deterministic
|
||||||
|
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
||||||
|
prompt; empty array = none declared. Must not boot shared stacks or
|
||||||
|
services (compose projects, fixed ports) — those collide with the probe
|
||||||
|
lane running in parallel.
|
||||||
|
default: '[]'
|
||||||
|
|
||||||
|
settings:
|
||||||
|
max_loop_iterations: 10
|
||||||
|
log_state_snapshots: false
|
||||||
|
# 3600 (run_checks node timeout, worst case) + 5400 (the previous
|
||||||
|
# whole-graph budget, kept as the envelope for every stage other than
|
||||||
|
# run_checks: parse/facts before it, criteria map/holistic/verdict after)
|
||||||
|
# = 9000. A graph timeout kills from outside — no fallback fires and the
|
||||||
|
# sentinel is lost.
|
||||||
|
timeout: 9000
|
||||||
|
|
||||||
|
initial_state:
|
||||||
|
project_dir_in: ''
|
||||||
|
parse_retries: 0
|
||||||
|
criteria: []
|
||||||
|
diff_spec: auto
|
||||||
|
plan_notes: ''
|
||||||
|
scope_notes: ''
|
||||||
|
extra_context: ''
|
||||||
|
changed_files: []
|
||||||
|
diff_text: ''
|
||||||
|
diff_note: ''
|
||||||
|
exec_results: ''
|
||||||
|
crit_verdicts: []
|
||||||
|
extra_complaints: []
|
||||||
|
observations: ''
|
||||||
|
adv_report: ''
|
||||||
|
gate_feedback: ''
|
||||||
|
gate_attempts: 0
|
||||||
|
parse_failure: ''
|
||||||
|
holistic_failure: ''
|
||||||
|
pipeline_faults: []
|
||||||
|
|
||||||
|
start: parse
|
||||||
|
|
||||||
|
nodes:
|
||||||
|
parse:
|
||||||
|
id: parse
|
||||||
|
type: llm
|
||||||
|
description: Extract the acceptance criteria and plan facts from the spawn prompt. Extraction only — no judging, no criteria invented.
|
||||||
|
instructions: |
|
||||||
|
You extract a plan-conformance review request into structured fields.
|
||||||
|
EXTRACTION ONLY — never invent criteria, never guess intent.
|
||||||
|
|
||||||
|
- `criteria`: every acceptance criterion from the plan/task, one entry each,
|
||||||
|
with a short id (c1, c2, …) and SELF-CONTAINED text (what observable
|
||||||
|
behavior is promised, including any test expectations the plan states).
|
||||||
|
Split compound criteria ("A and B") into separate entries when they are
|
||||||
|
independently verifiable. When the caller supplied NO plan/criteria at
|
||||||
|
all, emit an empty array — downstream fails closed.
|
||||||
|
An empty array is ONLY correct when the prompt genuinely contains no
|
||||||
|
plan/criteria section — when the prompt carries one (a "Plan /
|
||||||
|
acceptance criteria:" or "PLAN —" block), returning zero criteria is
|
||||||
|
an extraction FAILURE, never a valid result.
|
||||||
|
The caller's explicit enumeration is AUTHORITATIVE: when the caller
|
||||||
|
declares "these N are the (only) criteria" or supplies a numbered
|
||||||
|
criteria list, extract EXACTLY those N — never add entries they did
|
||||||
|
not enumerate. NEVER mine criteria from sections the caller labels as
|
||||||
|
context ("FACTS", "Quality bar", "context, not criteria", background
|
||||||
|
material) — route those into plan_notes / scope_notes / extra_context
|
||||||
|
instead. Splitting applies only WITHIN a declared criterion; when in
|
||||||
|
doubt, FEWER criteria, verbatim.
|
||||||
|
- `diff_spec`: the git diff basis the caller named — a range, a ref/SHA,
|
||||||
|
"staged", "worktree", or "auto" when unnamed (a caller phrase like
|
||||||
|
"get_diff --base X" means diff_spec = "X").
|
||||||
|
- `project_dir_in`: the directory/worktree path the caller named as
|
||||||
|
holding the code under review; empty string when the caller named none.
|
||||||
|
- `plan_notes`: recorded decisions the plan states ("do X not Y because Z"),
|
||||||
|
verbatim — the holistic pass checks the diff didn't silently override them.
|
||||||
|
- `scope_notes`: what the plan says NOT to touch / explicit boundaries.
|
||||||
|
- `extra_context`: anything else conformance-relevant, condensed.
|
||||||
|
prompt: |
|
||||||
|
Extract the conformance-review facts from this request:
|
||||||
|
|
||||||
|
{{initial_prompt}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
criteria:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
id:
|
||||||
|
type: string
|
||||||
|
text:
|
||||||
|
type: string
|
||||||
|
required: [id, text]
|
||||||
|
diff_spec:
|
||||||
|
type: string
|
||||||
|
plan_notes:
|
||||||
|
type: string
|
||||||
|
scope_notes:
|
||||||
|
type: string
|
||||||
|
extra_context:
|
||||||
|
type: string
|
||||||
|
project_dir_in:
|
||||||
|
type: string
|
||||||
|
required: [criteria, diff_spec, plan_notes, scope_notes, extra_context, project_dir_in]
|
||||||
|
max_iterations: 3
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
parse_failure: '{{output}}'
|
||||||
|
fallback: pipeline_fault
|
||||||
|
next: facts
|
||||||
|
|
||||||
|
facts:
|
||||||
|
id: facts
|
||||||
|
type: script
|
||||||
|
description: Deterministic diff resolution — changed files and the (bounded) diff text every criterion branch verifies against.
|
||||||
|
script: scripts/diff_facts.py
|
||||||
|
timeout: 120
|
||||||
|
fallback: pipeline_fault
|
||||||
|
next: run_checks
|
||||||
|
|
||||||
|
run_checks:
|
||||||
|
id: run_checks
|
||||||
|
type: script
|
||||||
|
description: Deterministic execution evidence — runs ONLY the commands declared via the verification_commands variable in the project dir and records exec_results; criterion branches cite the record and never execute anything. Undeclared commands degrade to a "none declared" marker, runner errors to an ENVIRONMENT marker (criterion branches judge it PARTIAL/unproven; the verdict additionally blocks CONFORMS when commands were declared); a declaration that is not a JSON array of strings executes nothing and records a PIPELINE-FAULT (fail-closed); a 3300s total deadline keeps the script under the node timeout (commands past it are recorded as skipped), and if the node is killed anyway the fallback records a PIPELINE-FAULT.
|
||||||
|
script: scripts/run_checks.py
|
||||||
|
timeout: 3600
|
||||||
|
fallback: pipeline_fault
|
||||||
|
next: map_criteria
|
||||||
|
|
||||||
|
map_criteria:
|
||||||
|
id: map_criteria
|
||||||
|
type: map
|
||||||
|
over: '{{criteria}}'
|
||||||
|
as: criterion
|
||||||
|
branch: check_criterion
|
||||||
|
output_key: crit_verdict
|
||||||
|
collect_into: crit_verdicts
|
||||||
|
max_concurrency: 6
|
||||||
|
next: holistic
|
||||||
|
|
||||||
|
check_criterion:
|
||||||
|
id: check_criterion
|
||||||
|
type: llm
|
||||||
|
description: Fresh-context verification of ONE acceptance criterion against the diff and the working tree (ENVIRONMENT/SKIPPED runs judge PARTIAL here; the deterministic verdict blocks CONFORMS when declared commands did not run).
|
||||||
|
instructions: |
|
||||||
|
You verify ONE acceptance criterion against a code change. Fresh context,
|
||||||
|
one criterion, full suspicion: assume the implementer drifted, cut a
|
||||||
|
corner, or misread the plan until the diff proves otherwise.
|
||||||
|
|
||||||
|
Classification (exactly one):
|
||||||
|
- `MET` — the diff satisfies the criterion AND a test proves the promised
|
||||||
|
behavior. Evidence MUST cite both: the satisfying change (file:line) and
|
||||||
|
the test targeting the behavior (file:line). No test ⇒ NOT MET — at best
|
||||||
|
PARTIAL (a criterion is a promise of observable behavior; unproven
|
||||||
|
behavior is unverified behavior).
|
||||||
|
- `PARTIAL` — partially satisfied, or satisfied but unproven (missing/weak
|
||||||
|
test), or you could not establish it either way (say why — doubt lands
|
||||||
|
here, never in MET).
|
||||||
|
- `UNMET` — the plan requires it and the diff does not do it. Absence is a
|
||||||
|
finding: review what SHOULD be there, not only what IS.
|
||||||
|
- `DIVERGED` — the diff does something ELSE instead (interface substituted,
|
||||||
|
approach swapped, criterion reinterpreted).
|
||||||
|
|
||||||
|
Ground-truth discipline: fs_grep the symbols the criterion names (confirm
|
||||||
|
they exist spelled as specified), fs_read around the relevant hunks to
|
||||||
|
confirm the change makes the criterion true, confirm the test targets
|
||||||
|
BEHAVIOR (not implementation echo), use ast_grep for structural claims.
|
||||||
|
Watch for gamed evidence: a test weakened/deleted so it passes, hardcoded
|
||||||
|
values shaped to the criterion's example, TODO/stub presented as done, a
|
||||||
|
mock expectation whose count is relaxed (`AnyTimes`, `MinTimes`, ignored
|
||||||
|
request body) AND whose matcher is permissive (`gomock.Any()`,
|
||||||
|
`mock.Anything`) — it proves a call happened, never that it carried the
|
||||||
|
criterion's values, so it is not a proving test —
|
||||||
|
those are PARTIAL or DIVERGED with the gaming named in the complaint.
|
||||||
|
|
||||||
|
Retry / redelivery / idempotency demands: before you mark a criterion PARTIAL
|
||||||
|
or UNMET for lacking a retry, re-run, or "second attempt emits nothing" spec,
|
||||||
|
classify the handler. STATEFUL — it carries something across attempts (emits
|
||||||
|
events, calls an external system, performs an unconditional write, keeps an
|
||||||
|
in-memory or ledger record) — a retry spec is a legitimate demand. STATELESS —
|
||||||
|
each attempt is a pure function of the request plus a conditional store
|
||||||
|
predicate (`WHERE status = 'pending'`, `ended_at IS NULL`, `INSERT … ON
|
||||||
|
CONFLICT DO NOTHING`) and nothing is emitted or ledgered outside that
|
||||||
|
predicate — then "retry produces no event" is a tautology of the mechanism,
|
||||||
|
not a missing behavior: demand at most one test of the predicate itself,
|
||||||
|
never a retry spec. Name the classification and its file:line evidence in
|
||||||
|
the complaint either way.
|
||||||
|
|
||||||
|
Execution criteria ("tests pass", "lint clean", "build succeeds") are
|
||||||
|
judged ONLY against the recorded verification runs in your prompt — you
|
||||||
|
have no execution tools BY DESIGN and must never claim to have run
|
||||||
|
anything yourself:
|
||||||
|
- a recorded green run (exit 0) of a command covering the criterion is
|
||||||
|
the proving evidence — MET, citing that run (the command + exit 0).
|
||||||
|
- a recorded failing run covering the criterion ⇒ UNMET, citing the
|
||||||
|
recorded exit code and output tail.
|
||||||
|
- "none declared" or an ENVIRONMENT marker ⇒ at best PARTIAL, with an
|
||||||
|
ENVIRONMENT note in the complaint naming the exact command the caller
|
||||||
|
should declare via the `verification_commands` variable (structured
|
||||||
|
input, e.g. "cargo test --all") — commands are never declared in the
|
||||||
|
prompt.
|
||||||
|
- a SKIPPED record (the runner's total deadline expired before that
|
||||||
|
command ran) ⇒ at best PARTIAL for the criteria it covers, same
|
||||||
|
treatment as an ENVIRONMENT marker: unproven at the criterion level.
|
||||||
|
The deterministic verdict additionally blocks CONFORMS when declared
|
||||||
|
commands did not run — that is its job, not yours.
|
||||||
|
|
||||||
|
Output format — respond with ONLY this JSON object, no prose around it:
|
||||||
|
{"id": "<the criterion's id, echoed>",
|
||||||
|
"status": "MET" | "PARTIAL" | "UNMET" | "DIVERGED",
|
||||||
|
"evidence": "<for MET: the satisfying change file:line AND the proving test
|
||||||
|
file:line, quoted decisively; otherwise the closest relevant citations or empty>",
|
||||||
|
"complaint": "<required for non-MET: what the diff does/omits (file:line) and
|
||||||
|
what would make it conform; empty string for MET>"}
|
||||||
|
|
||||||
|
Your response is machine-validated: a malformed response is rejected back
|
||||||
|
to you exactly once with the reason, then recorded as PARTIAL (unproven).
|
||||||
|
The id is stamped from the criterion itself — your echo is informative only.
|
||||||
|
You have a bounded tool budget — prefer fs_grep and line-ranged fs_read over
|
||||||
|
whole-file reads, and conclude as soon as the evidence is decisive.
|
||||||
|
Never begin `evidence` with the literal `PIPELINE-FAULT:` — that prefix is
|
||||||
|
reserved for the graph's own fault markers.
|
||||||
|
prompt: |
|
||||||
|
Repository: {{project_dir}}
|
||||||
|
Changed files: {{changed_files}}
|
||||||
|
|
||||||
|
Criterion to verify (JSON):
|
||||||
|
{{criterion}}
|
||||||
|
|
||||||
|
Plan-recorded decisions (context): {{plan_notes}}
|
||||||
|
{{extra_context}}
|
||||||
|
|
||||||
|
The diff{{diff_note}}:
|
||||||
|
{{diff_text}}
|
||||||
|
|
||||||
|
Verification runs (recorded by the deterministic run_checks stage):
|
||||||
|
{{exec_results}}
|
||||||
|
{{gate_feedback}}
|
||||||
|
tools:
|
||||||
|
- fs_read
|
||||||
|
- fs_cat
|
||||||
|
- fs_grep
|
||||||
|
- ast_grep
|
||||||
|
max_iterations: 30
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
crit_verdict: '{{output}}'
|
||||||
|
fallback: criterion_fault
|
||||||
|
next: crit_gate
|
||||||
|
|
||||||
|
crit_gate:
|
||||||
|
id: crit_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic contract gate per criterion verdict — status enum, MET requires evidence, non-MET requires a complaint; reject-and-retry once, then record PARTIAL (unproven). Never sinks the map.
|
||||||
|
script: scripts/crit_gate.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
criterion_fault:
|
||||||
|
id: criterion_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed per-criterion fault marker (reached only via check_criterion's fallback) — converts the captured "LLM node failed" text into a schema-shaped UNMET verdict carrying a PIPELINE-FAULT evidence marker, so one dead criterion check can never sink the map or read as met.
|
||||||
|
script: scripts/criterion_fault.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
holistic:
|
||||||
|
id: holistic
|
||||||
|
type: llm
|
||||||
|
description: Cross-criterion adversarial pass — hunts what per-criterion checks cannot see. ADDITIVE-only; cannot flip criterion verdicts.
|
||||||
|
instructions: |
|
||||||
|
You run the holistic adversarial pass over a whole change AFTER each
|
||||||
|
acceptance criterion was independently verified. You hunt what
|
||||||
|
per-criterion checks cannot see. You are ADDITIVE-ONLY: you may add
|
||||||
|
complaints; you may NOT flip, soften, or re-judge any criterion verdict.
|
||||||
|
|
||||||
|
The checklist (each hit becomes one complaint with kind, file:line
|
||||||
|
location, the violation, and the fix):
|
||||||
|
- **Absence** — something the PLAN requires that maps to no criterion and
|
||||||
|
never showed up in the diff (the dog that didn't bark). A missing
|
||||||
|
retry/redelivery spec is an absence ONLY for a handler that carries state
|
||||||
|
or side effects across attempts; a handler that is a pure function of the
|
||||||
|
request plus a conditional store predicate owes none.
|
||||||
|
- **Scope drift / out-of-scope** — the diff touches what the plan's scope
|
||||||
|
notes say not to touch, or quietly does extra unplanned work.
|
||||||
|
- **Interface/approach substitution** — the plan named an interface,
|
||||||
|
library, or approach; the diff substituted another without the plan
|
||||||
|
recording that decision.
|
||||||
|
- **Silently overridden recorded decision** — plan says "do X not Y
|
||||||
|
because Z"; diff does Y.
|
||||||
|
- **Downstream contract breakage** — the diff changes a contract the plan
|
||||||
|
promised to preserve.
|
||||||
|
- **Gamed tests** — tests weakened, deleted, or shaped (hardcoded
|
||||||
|
expectations, skipped assertions) so the criteria APPEAR met.
|
||||||
|
- **The plan itself is the culprit** — impossible or self-contradictory
|
||||||
|
requirements: that is a complaint naming the PLAN as root cause; never
|
||||||
|
silently judge against a corrected version of it.
|
||||||
|
|
||||||
|
Be terse and decisive: three real complaints beat fifteen weak ones. A
|
||||||
|
complaint with no location and no criterion/scope reference is noise — do
|
||||||
|
not emit it. When everything is a nitpick, emit zero complaints and put
|
||||||
|
1-3 non-blocking notes in `observations`.
|
||||||
|
prompt: |
|
||||||
|
Repository: {{project_dir}}
|
||||||
|
Changed files: {{changed_files}}
|
||||||
|
|
||||||
|
Acceptance criteria: {{criteria}}
|
||||||
|
Per-criterion verdicts (already final): {{crit_verdicts}}
|
||||||
|
|
||||||
|
Plan-recorded decisions: {{plan_notes}}
|
||||||
|
Scope boundaries: {{scope_notes}}
|
||||||
|
{{extra_context}}
|
||||||
|
|
||||||
|
The diff{{diff_note}}:
|
||||||
|
{{diff_text}}
|
||||||
|
tools:
|
||||||
|
- fs_read
|
||||||
|
- fs_cat
|
||||||
|
- fs_grep
|
||||||
|
- ast_grep
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
extra_complaints:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
kind:
|
||||||
|
type: string
|
||||||
|
location:
|
||||||
|
type: string
|
||||||
|
violation:
|
||||||
|
type: string
|
||||||
|
fix:
|
||||||
|
type: string
|
||||||
|
required: [kind, location, violation, fix]
|
||||||
|
observations:
|
||||||
|
type: string
|
||||||
|
required: [extra_complaints, observations]
|
||||||
|
max_iterations: 45
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
holistic_failure: '{{output}}'
|
||||||
|
fallback: verdict
|
||||||
|
next: verdict
|
||||||
|
|
||||||
|
pipeline_fault:
|
||||||
|
id: pipeline_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed fault marker (reached only via fallback routes from parse/facts/run_checks) — records a PIPELINE-FAULT in state and continues to verdict, which turns any fault into DIVERGES. The graph never dies without its sentinel.
|
||||||
|
script: scripts/pipeline_fault.py
|
||||||
|
timeout: 30
|
||||||
|
next: verdict
|
||||||
|
|
||||||
|
verdict:
|
||||||
|
id: verdict
|
||||||
|
type: script
|
||||||
|
description: Deterministic verdict — CONFORMS requires every criterion MET, zero holistic complaints, zero pipeline faults, and zero recorded red verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error marker when commands were declared); no criteria at all fails closed and any recorded PIPELINE-FAULT forces DIVERGES (degraded is never passing). Assembles the exact sentinel format.
|
||||||
|
script: scripts/verdict.py
|
||||||
|
timeout: 30
|
||||||
|
next: done
|
||||||
|
|
||||||
|
done:
|
||||||
|
id: done
|
||||||
|
type: end
|
||||||
|
output: |
|
||||||
|
{{adv_report}}
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Contract gate for the adversary's per-criterion verify chain.
|
||||||
|
|
||||||
|
Validates each criterion verdict against the contract:
|
||||||
|
- a single JSON object with status in {MET, PARTIAL, UNMET, DIVERGED}
|
||||||
|
- MET requires non-empty evidence (the change AND the proving test)
|
||||||
|
- non-MET requires a non-empty complaint
|
||||||
|
|
||||||
|
The authoritative id is stamped from the criterion item — the model's echo is
|
||||||
|
never trusted. Malformed output is rejected back to check_criterion exactly
|
||||||
|
once with the reason; a second failure records PARTIAL (unproven) so a
|
||||||
|
malformed item can never sink the map or masquerade as MET.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
STATUSES = {"MET", "PARTIAL", "UNMET", "DIVERGED"}
|
||||||
|
MAX_RETRIES = 1
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def criterion_fields(state):
|
||||||
|
c = state.get("criterion")
|
||||||
|
if isinstance(c, dict):
|
||||||
|
cid = c.get("id")
|
||||||
|
text = c.get("text")
|
||||||
|
return (
|
||||||
|
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
|
||||||
|
text.strip() if isinstance(text, str) else "",
|
||||||
|
)
|
||||||
|
return "unknown", ""
|
||||||
|
|
||||||
|
|
||||||
|
def extract_json(text):
|
||||||
|
if not isinstance(text, str) or not text.strip():
|
||||||
|
raise ValueError("criterion-check output is empty or not text")
|
||||||
|
text = re.sub(r"```[a-zA-Z]*", "", text).replace("```", "")
|
||||||
|
start, end = text.find("{"), text.rfind("}")
|
||||||
|
if start == -1 or end <= start:
|
||||||
|
raise ValueError("no JSON object found in criterion-check output")
|
||||||
|
return json.loads(text[start : end + 1])
|
||||||
|
|
||||||
|
|
||||||
|
def validate(obj):
|
||||||
|
if not isinstance(obj, dict):
|
||||||
|
raise ValueError("criterion-check output is not a JSON object")
|
||||||
|
status = obj.get("status")
|
||||||
|
if status not in STATUSES:
|
||||||
|
raise ValueError(f"status must be one of {sorted(STATUSES)}, got {status!r}")
|
||||||
|
evidence = obj.get("evidence") if isinstance(obj.get("evidence"), str) else ""
|
||||||
|
complaint = obj.get("complaint") if isinstance(obj.get("complaint"), str) else ""
|
||||||
|
if status == "MET" and not evidence.strip():
|
||||||
|
raise ValueError(
|
||||||
|
"MET requires non-empty evidence citing the satisfying change AND the "
|
||||||
|
"test proving the behavior (file:line)"
|
||||||
|
)
|
||||||
|
if status != "MET" and not complaint.strip():
|
||||||
|
raise ValueError(
|
||||||
|
f"{status} requires a non-empty complaint (what the diff does/omits at "
|
||||||
|
"file:line, and what would make it conform)"
|
||||||
|
)
|
||||||
|
return status, evidence, complaint
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
cid, ctext = criterion_fields(state)
|
||||||
|
attempts = state.get("gate_attempts")
|
||||||
|
if not isinstance(attempts, int):
|
||||||
|
attempts = 0
|
||||||
|
|
||||||
|
try:
|
||||||
|
status, evidence, complaint = validate(extract_json(state.get("crit_verdict")))
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"crit_verdict": {
|
||||||
|
"id": cid,
|
||||||
|
"text": ctext,
|
||||||
|
"status": status,
|
||||||
|
"evidence": evidence,
|
||||||
|
"complaint": complaint,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
reason = str(e)
|
||||||
|
|
||||||
|
if attempts < MAX_RETRIES:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"_next": "check_criterion",
|
||||||
|
"gate_attempts": attempts + 1,
|
||||||
|
"gate_feedback": (
|
||||||
|
"PREVIOUS RESPONSE REJECTED by the verdict gate: " + reason +
|
||||||
|
". Respond with ONLY the JSON object described in your instructions."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"crit_verdict": {
|
||||||
|
"id": cid,
|
||||||
|
"text": ctext,
|
||||||
|
"status": "PARTIAL",
|
||||||
|
"evidence": "",
|
||||||
|
"complaint": (
|
||||||
|
"criterion check output failed machine validation after retry: "
|
||||||
|
+ reason
|
||||||
|
+ " — treat this criterion as unproven"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the gate must never crash the chain
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"crit_verdict": {
|
||||||
|
"id": "unknown",
|
||||||
|
"text": "",
|
||||||
|
"status": "PARTIAL",
|
||||||
|
"evidence": "",
|
||||||
|
"complaint": f"criterion gate error: {e} — treat as unproven",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed per-criterion fault marker for the adversary graph.
|
||||||
|
|
||||||
|
Reached ONLY via check_criterion's `fallback:` route, after the engine has
|
||||||
|
written its "LLM node …failed: <chain>" text into `crit_verdict` through
|
||||||
|
state_updates. Converts that text into a schema-shaped UNMET verdict whose
|
||||||
|
evidence carries a "PIPELINE-FAULT:" marker, so one criterion check that dies
|
||||||
|
(max_iterations exhausted, API failure) is recorded as DIED rather than
|
||||||
|
judged, the other criteria still get verdicts, and the run is DIVERGES. The
|
||||||
|
map collects it like any other verdict — the branch never sinks the map. The
|
||||||
|
engine applies state_updates before routing to the fallback (src/graph/llm.rs:
|
||||||
|
`apply_state_updates_with_output`, then `outcome_from`), so the "no failure
|
||||||
|
text captured" branch is belt-and-braces rather than an expected path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def criterion_fields(state):
|
||||||
|
c = state.get("criterion")
|
||||||
|
if isinstance(c, dict):
|
||||||
|
cid = c.get("id")
|
||||||
|
text = c.get("text")
|
||||||
|
return (
|
||||||
|
cid.strip() if isinstance(cid, str) and cid.strip() else "unknown",
|
||||||
|
text.strip() if isinstance(text, str) else "",
|
||||||
|
)
|
||||||
|
return "unknown", ""
|
||||||
|
|
||||||
|
|
||||||
|
def failure_detail(state):
|
||||||
|
captured = state.get("crit_verdict")
|
||||||
|
detail = captured.strip() if isinstance(captured, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
return "no failure text captured"
|
||||||
|
detail = detail.replace("\n", " ")
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
return detail
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
cid, ctext = criterion_fields(state)
|
||||||
|
detail = failure_detail(state)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"crit_verdict": {
|
||||||
|
"id": cid,
|
||||||
|
"text": ctext,
|
||||||
|
"status": "UNMET",
|
||||||
|
"evidence": f"PIPELINE-FAULT: criterion check failed — {detail}",
|
||||||
|
"complaint": (
|
||||||
|
f"criterion check DIED (pipeline fault) — {detail}; the "
|
||||||
|
"criterion was NOT verified and is treated as unmet "
|
||||||
|
"(fail-closed)"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"crit_verdict": {
|
||||||
|
"id": "unknown",
|
||||||
|
"text": "",
|
||||||
|
"status": "UNMET",
|
||||||
|
"evidence": f"PIPELINE-FAULT: criterion fault-marker script error: {e}",
|
||||||
|
"complaint": (
|
||||||
|
f"criterion check DIED (pipeline fault) — fault-marker script "
|
||||||
|
f"error: {e}; the criterion was NOT verified and is treated as "
|
||||||
|
"unmet (fail-closed)"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic diff resolution for the adversary graph.
|
||||||
|
|
||||||
|
Resolves the diff spec (get_diff semantics: staged → unstaged → HEAD~1 for
|
||||||
|
"auto"), lists changed files, and captures the bounded diff text every
|
||||||
|
criterion branch verifies against. Failures degrade into diff_note — the
|
||||||
|
review proceeds fail-closed rather than crashing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
MAX_DIFF_CHARS = 60000
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
state = load_state()
|
||||||
|
|
||||||
|
# Zero criteria from a prompt that visibly carries a plan is an extraction
|
||||||
|
# failure (observed non-deterministically live), not a fail-closed no-plan
|
||||||
|
# run — re-run the parse ONCE before the verdict fails closed on it.
|
||||||
|
_crit = state.get("criteria") or []
|
||||||
|
_prompt = state.get("initial_prompt") or ""
|
||||||
|
if not _crit and isinstance(_prompt, str) and re.search(
|
||||||
|
r"acceptance criteria|PLAN\s*—|Plan / acceptance", _prompt, re.I
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
_pr = int(state.get("parse_retries") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
_pr = 0
|
||||||
|
if _pr < 1:
|
||||||
|
print(json.dumps({"_next": "parse", "parse_retries": _pr + 1}))
|
||||||
|
raise SystemExit(0)
|
||||||
|
proj = os.path.expanduser(
|
||||||
|
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
|
||||||
|
)
|
||||||
|
spec = (state.get("diff_spec") or "auto").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
r = subprocess.run(["git", "-C", proj, *args], capture_output=True, text=True, timeout=90)
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise RuntimeError((r.stderr or r.stdout).strip()[:300])
|
||||||
|
return r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
out = {"diff_note": "", "project_dir": proj}
|
||||||
|
try:
|
||||||
|
if spec in ("auto", ""):
|
||||||
|
text = git("diff", "--cached")
|
||||||
|
if not text.strip():
|
||||||
|
text = git("diff")
|
||||||
|
if not text.strip():
|
||||||
|
text = git("diff", "HEAD~1")
|
||||||
|
elif spec == "staged":
|
||||||
|
text = git("diff", "--cached")
|
||||||
|
elif spec == "worktree":
|
||||||
|
text = git("diff", "HEAD")
|
||||||
|
else:
|
||||||
|
text = git("diff", spec)
|
||||||
|
|
||||||
|
files = sorted({
|
||||||
|
line[6:].strip()
|
||||||
|
for line in text.splitlines()
|
||||||
|
if line.startswith("+++ b/")
|
||||||
|
})
|
||||||
|
out["changed_files"] = files[:300]
|
||||||
|
if len(files) > 300:
|
||||||
|
out["diff_note"] += (
|
||||||
|
f" (file list capped at 300 of {len(files)} changed files — use the fs "
|
||||||
|
"tools for the rest)"
|
||||||
|
)
|
||||||
|
if len(text) > MAX_DIFF_CHARS:
|
||||||
|
out["diff_note"] += (
|
||||||
|
f" (truncated to {MAX_DIFF_CHARS} chars of {len(text)} — use the fs tools "
|
||||||
|
"to inspect files beyond the cut)"
|
||||||
|
)
|
||||||
|
text = text[:MAX_DIFF_CHARS]
|
||||||
|
out["diff_text"] = text if text.strip() else "(empty diff — nothing changed)"
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
out["changed_files"] = []
|
||||||
|
out["diff_text"] = "(diff unavailable)"
|
||||||
|
out["diff_note"] = f" (diff resolution failed for spec {spec!r}: {e})"
|
||||||
|
|
||||||
|
print(json.dumps(out))
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed fault marker for the adversary graph.
|
||||||
|
|
||||||
|
Reached ONLY via `fallback:` routes (parse, facts, run_checks). Appends a
|
||||||
|
machine-readable "PIPELINE-FAULT:" marker to `pipeline_faults` and continues
|
||||||
|
to `verdict`, which turns any recorded fault into DIVERGES — the graph never
|
||||||
|
dies without its sentinel, and a degraded run can never read as CONFORMS.
|
||||||
|
|
||||||
|
Stage attribution — grounded in engine behavior: a SCRIPT node's fallback
|
||||||
|
route writes NOTHING into state (src/graph/executor.rs script arm routes to
|
||||||
|
the fallback without recording the error), so the two script stages are
|
||||||
|
discriminated by which stage outputs are present:
|
||||||
|
- parse (llm): its `state_updates` wrote an "LLM node …failed: <chain>"
|
||||||
|
string into `parse_failure` — attribute to parse.
|
||||||
|
- run_checks vs facts (scripts): diff_facts.py unconditionally writes a
|
||||||
|
NONEMPTY `diff_text` (even its internal-failure path writes
|
||||||
|
"(diff unavailable)"), while the graph's initial value is ''. A nonempty
|
||||||
|
`diff_text` proves facts completed, so the fault is the verification
|
||||||
|
runner (run_checks killed at the node level, e.g. by the node timeout);
|
||||||
|
an empty `diff_text` means facts itself died.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
parse_failure = state.get("parse_failure")
|
||||||
|
detail = parse_failure.strip() if isinstance(parse_failure, str) else ""
|
||||||
|
diff_text = state.get("diff_text")
|
||||||
|
facts_completed = isinstance(diff_text, str) and bool(diff_text.strip())
|
||||||
|
if detail.startswith("LLM node"):
|
||||||
|
detail = detail.replace("\n", " ")
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
faults.append(f"PIPELINE-FAULT: parse failed — cannot review: {detail}")
|
||||||
|
elif facts_completed:
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: verification runner killed — the run_checks stage "
|
||||||
|
"died at the node level before recording exec_results; execution "
|
||||||
|
"criteria are unproven"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: diff resolution failed — changed files and diff text "
|
||||||
|
"could not be resolved; criterion verification never ran"
|
||||||
|
)
|
||||||
|
print(json.dumps({"pipeline_faults": faults}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"pipeline_faults": [
|
||||||
|
f"PIPELINE-FAULT: fault-marker script error: {e} — treat the "
|
||||||
|
"run as degraded"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,252 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic execution-evidence recorder for the adversary graph.
|
||||||
|
|
||||||
|
Runs ONLY the caller-declared verification commands (verification_commands)
|
||||||
|
in the project dir and records the outcomes as `exec_results` — a list of
|
||||||
|
{cmd, exit, duration_s, tail} records. Criterion branches cite this record;
|
||||||
|
they never execute anything themselves.
|
||||||
|
|
||||||
|
NO auto-detection: undeclared commands mean a "none declared" marker
|
||||||
|
(fail-visible, not fail-guessed). Any runner error degrades into an
|
||||||
|
ENVIRONMENT marker — this script never fails the node.
|
||||||
|
|
||||||
|
Trust boundary: `verification_commands` is a DECLARED graph variable (or the
|
||||||
|
review-gauntlet's structured `inputs:` passthrough), never a field an LLM
|
||||||
|
extracts from the spawn prompt — that prose also carries plan/diff text pasted
|
||||||
|
from the repo under review, and whatever is declared here runs with
|
||||||
|
shell=True. Variables land in state as strings, so a JSON-encoded list is
|
||||||
|
accepted alongside a real list. A declaration that is not a JSON array of
|
||||||
|
strings is a caller contract violation, not a runner hiccup: it executes
|
||||||
|
NOTHING and fails closed as a PIPELINE-FAULT (complaint #1) — it never
|
||||||
|
degrades to an ENVIRONMENT marker. The engine merges only the keys an
|
||||||
|
llm node's `output_schema.properties` declares, so a parse LLM coaxed into
|
||||||
|
emitting an extra `verification_commands` key cannot overwrite the declared
|
||||||
|
value: the undeclared key is dropped before it reaches state.
|
||||||
|
|
||||||
|
Budget: commands run sequentially, so TOTAL runtime is bounded by a deadline
|
||||||
|
(TOTAL_DEADLINE_SECS = 3300s, the run_checks node's 3600s timeout minus
|
||||||
|
margin). Without it, a handful of hanging commands (4 x 900s) would blow the
|
||||||
|
NODE timeout and the executor would kill this script from OUTSIDE — the
|
||||||
|
in-script degradation would never fire. Each command gets
|
||||||
|
min(PER_COMMAND_TIMEOUT_SECS, remaining budget); commands the deadline
|
||||||
|
leaves no budget for are recorded as {"cmd", "skipped": "deadline"} entries,
|
||||||
|
so the in-script handling stays authoritative. The env var
|
||||||
|
ADVERSARY_RUN_CHECKS_DEADLINE_SECS overrides the deadline (test seam), and
|
||||||
|
ADVERSARY_STACK_LOCK_PATH overrides the review-stack lock file location
|
||||||
|
(isolation seam — see acquire_stack_lock).
|
||||||
|
|
||||||
|
Process hygiene: commands run through the shell. On POSIX each command gets
|
||||||
|
its own session (start_new_session) and a timeout SIGKILLs the whole process
|
||||||
|
group, so grandchildren of a hung command cannot linger. The post-kill
|
||||||
|
reap/drain is itself bounded by DRAIN_TIMEOUT_SECS: a grandchild that escaped
|
||||||
|
the session (its own start_new_session) but still holds the stdout pipe
|
||||||
|
would otherwise block the drain until it exits. On non-POSIX platforms only
|
||||||
|
the direct shell child is killed — grandchildren of a timed-out command may
|
||||||
|
survive (documented limitation).
|
||||||
|
|
||||||
|
Verification commands run with GRAPH_STATE* (GRAPH_STATE, GRAPH_STATE_FILE)
|
||||||
|
scrubbed from their environment: the reviewed repo's own tests may exec these very
|
||||||
|
scripts with their own GRAPH_STATE, and load_state() prefers
|
||||||
|
GRAPH_STATE_FILE, so an inherited live state file would silently win.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
try:
|
||||||
|
import fcntl
|
||||||
|
except ImportError: # non-POSIX (Windows): advisory flock unavailable
|
||||||
|
fcntl = None
|
||||||
|
|
||||||
|
PER_COMMAND_TIMEOUT_SECS = 900
|
||||||
|
TOTAL_DEADLINE_SECS = float(os.environ.get("ADVERSARY_RUN_CHECKS_DEADLINE_SECS") or 3300)
|
||||||
|
STACK_LOCK_PATH = os.environ.get("ADVERSARY_STACK_LOCK_PATH") or os.path.expanduser(
|
||||||
|
"~/.cache/coyote/review-stack.lock"
|
||||||
|
)
|
||||||
|
STACK_LOCK_WAIT_SECS = float(os.environ.get("ADVERSARY_STACK_LOCK_WAIT_SECS") or 1200)
|
||||||
|
|
||||||
|
|
||||||
|
def acquire_stack_lock():
|
||||||
|
"""Serialize with the probe lane's shared-stack work (advisory flock on a
|
||||||
|
well-known path; the probe is instructed to flock the same file around
|
||||||
|
stack boot/teardown). Bounded wait so a hogged lock can never turn into a
|
||||||
|
false red verification: on timeout we proceed WITHOUT the lock and note it
|
||||||
|
on stderr. The fd is held for the process lifetime (released on exit).
|
||||||
|
On non-POSIX platforms (no fcntl) the lock is skipped entirely."""
|
||||||
|
if fcntl is None:
|
||||||
|
sys.stderr.write("WARN: review-stack lock unsupported on this platform; running unlocked\n")
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
os.makedirs(os.path.dirname(STACK_LOCK_PATH), exist_ok=True)
|
||||||
|
fd = os.open(STACK_LOCK_PATH, os.O_CREAT | os.O_RDWR)
|
||||||
|
except OSError as e:
|
||||||
|
sys.stderr.write(f"WARN: review-stack lock unavailable ({e}); running unlocked\n")
|
||||||
|
return None
|
||||||
|
limit = time.monotonic() + STACK_LOCK_WAIT_SECS
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
return fd
|
||||||
|
except OSError:
|
||||||
|
if time.monotonic() >= limit:
|
||||||
|
os.close(fd)
|
||||||
|
sys.stderr.write(
|
||||||
|
"WARN: review-stack lock still held after bounded wait — proceeding "
|
||||||
|
"WITHOUT it; shared-stack collisions with the probe lane are possible\n"
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
time.sleep(5)
|
||||||
|
DRAIN_TIMEOUT_SECS = 10
|
||||||
|
TAIL_LINES = 50
|
||||||
|
MAX_TAIL_CHARS = 8000
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidDeclaration(ValueError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def declared_commands(declared):
|
||||||
|
"""Normalize the verification_commands declaration to a list of commands.
|
||||||
|
|
||||||
|
Accepts a JSON list or a JSON-encoded string of one; missing/None means
|
||||||
|
none declared. Anything else raises InvalidDeclaration.
|
||||||
|
"""
|
||||||
|
if declared is None:
|
||||||
|
return []
|
||||||
|
if isinstance(declared, str):
|
||||||
|
if not declared.strip():
|
||||||
|
raise InvalidDeclaration("empty string (declare '[]' for none)")
|
||||||
|
try:
|
||||||
|
declared = json.loads(declared)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
raise InvalidDeclaration(f"not valid JSON: {e}") from e
|
||||||
|
if not isinstance(declared, list):
|
||||||
|
raise InvalidDeclaration(
|
||||||
|
f"expected a JSON array of strings, got {type(declared).__name__}"
|
||||||
|
)
|
||||||
|
bad = [c for c in declared if not isinstance(c, str)]
|
||||||
|
if bad:
|
||||||
|
raise InvalidDeclaration(f"non-string item(s): {json.dumps(bad)}")
|
||||||
|
return [c.strip() for c in declared if c.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def tail_of(text):
|
||||||
|
tail = "\n".join(text.splitlines()[-TAIL_LINES:])
|
||||||
|
if len(tail) > MAX_TAIL_CHARS:
|
||||||
|
tail = "…" + tail[-MAX_TAIL_CHARS:]
|
||||||
|
return tail
|
||||||
|
|
||||||
|
|
||||||
|
def run_one(cmd, proj, budget):
|
||||||
|
"""Run one command, hard-bounded by `budget` seconds; return its record."""
|
||||||
|
start = time.monotonic()
|
||||||
|
popen_kwargs = {"start_new_session": True} if os.name == "posix" else {}
|
||||||
|
env = {k: v for k, v in os.environ.items() if not k.startswith("GRAPH_STATE")}
|
||||||
|
p = subprocess.Popen(
|
||||||
|
cmd,
|
||||||
|
shell=True,
|
||||||
|
cwd=proj,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
text=True,
|
||||||
|
encoding="utf-8",
|
||||||
|
errors="replace",
|
||||||
|
env=env,
|
||||||
|
**popen_kwargs,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
out_s, err_s = p.communicate(timeout=budget)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
if os.name == "posix":
|
||||||
|
try:
|
||||||
|
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
|
||||||
|
except (ProcessLookupError, PermissionError):
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
p.kill()
|
||||||
|
try:
|
||||||
|
p.communicate(timeout=DRAIN_TIMEOUT_SECS)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
pass
|
||||||
|
return {
|
||||||
|
"cmd": cmd,
|
||||||
|
"exit": -1,
|
||||||
|
"duration_s": round(time.monotonic() - start, 1),
|
||||||
|
"tail": f"TIMEOUT: no exit within {budget:.0f}s — recorded as a failing run",
|
||||||
|
}
|
||||||
|
combined = (out_s or "") + (("\n" + err_s) if err_s else "")
|
||||||
|
return {
|
||||||
|
"cmd": cmd,
|
||||||
|
"exit": p.returncode,
|
||||||
|
"duration_s": round(time.monotonic() - start, 1),
|
||||||
|
"tail": tail_of(combined),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
try:
|
||||||
|
cmds = declared_commands(state.get("verification_commands"))
|
||||||
|
except InvalidDeclaration as e:
|
||||||
|
msg = f"PIPELINE-FAULT: verification_commands declaration invalid — {e}"
|
||||||
|
prior = state.get("pipeline_faults")
|
||||||
|
faults = [f for f in prior if isinstance(f, str)] if isinstance(prior, list) else []
|
||||||
|
faults.append(msg)
|
||||||
|
print(json.dumps({"exec_results": msg, "pipeline_faults": faults}))
|
||||||
|
return
|
||||||
|
if not cmds:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"exec_results": (
|
||||||
|
"none declared — the caller supplied no verification_commands; "
|
||||||
|
"execution criteria (build/test/lint) cannot be independently "
|
||||||
|
"confirmed. Declare the exact commands via the "
|
||||||
|
"verification_commands variable to get recorded runs."
|
||||||
|
)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
proj = os.path.expanduser(
|
||||||
|
(state.get("project_dir") or "").strip() or "."
|
||||||
|
)
|
||||||
|
deadline = time.monotonic() + TOTAL_DEADLINE_SECS
|
||||||
|
results = []
|
||||||
|
_stack_lock = acquire_stack_lock() if cmds else None # noqa: F841 — held until exit
|
||||||
|
for cmd in cmds:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
results.append({"cmd": cmd, "skipped": "deadline"})
|
||||||
|
continue
|
||||||
|
results.append(run_one(cmd, proj, min(PER_COMMAND_TIMEOUT_SECS, remaining)))
|
||||||
|
print(json.dumps({"exec_results": results}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — never fail the node; degrade visibly
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"exec_results": (
|
||||||
|
f"ENVIRONMENT: verification runner error: {e} — the declared "
|
||||||
|
"commands could not be executed; execution criteria are unproven "
|
||||||
|
"(PARTIAL at the criterion level; the verdict blocks CONFORMS when "
|
||||||
|
"commands were declared)"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic conformance verdict for the adversary graph.
|
||||||
|
|
||||||
|
CONFORMS requires: at least one criterion, every criterion MET, zero
|
||||||
|
holistic complaints, zero pipeline faults, and zero recorded red
|
||||||
|
verification runs (exit≠0/timeout/skipped, or an ENVIRONMENT runner-error
|
||||||
|
marker when commands were declared — the declared commands never ran, so the
|
||||||
|
run is unproven). Everything else — including a missing plan or a degraded
|
||||||
|
(faulted) pipeline — is DIVERGES (fail-closed).
|
||||||
|
Any "PIPELINE-FAULT:" marker recorded in `pipeline_faults` (or a holistic
|
||||||
|
llm-node failure captured in `holistic_failure`) becomes complaint #1;
|
||||||
|
per-criterion results are still reported. A criterion whose check DIED
|
||||||
|
(criterion_fault's UNMET verdict, evidence prefixed DIED_EVIDENCE_PREFIX) is
|
||||||
|
rendered as died rather than judged and counted as unmet regardless of the
|
||||||
|
status it carries. Assembles the exact sentinel format the callers route on.
|
||||||
|
Never crashes into a silent verdict.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_FAULT_DETAIL_CHARS = 300
|
||||||
|
MAX_MARKER_CHARS = 200
|
||||||
|
DIED_MARKER = "criterion check DIED (pipeline fault)"
|
||||||
|
DIED_EVIDENCE_PREFIX = "PIPELINE-FAULT: criterion "
|
||||||
|
|
||||||
|
|
||||||
|
def died(v):
|
||||||
|
evidence = v.get("evidence")
|
||||||
|
return isinstance(evidence, str) and evidence.startswith(DIED_EVIDENCE_PREFIX)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def collect_pipeline_faults(state):
|
||||||
|
"""Recorded fault markers, plus a holistic llm-node failure if one occurred.
|
||||||
|
|
||||||
|
`holistic_failure` holds the "LLM node …failed: <chain>" string when the
|
||||||
|
holistic pass fell back to verdict; on a successful pass it holds the
|
||||||
|
rendered schema output (never that prefix).
|
||||||
|
"""
|
||||||
|
faults = [
|
||||||
|
f.strip()
|
||||||
|
for f in (state.get("pipeline_faults") or [])
|
||||||
|
if isinstance(f, str) and f.strip()
|
||||||
|
]
|
||||||
|
hf = state.get("holistic_failure")
|
||||||
|
if isinstance(hf, str) and hf.strip().startswith("LLM node"):
|
||||||
|
detail = hf.strip().replace("\n", " ")
|
||||||
|
if len(detail) > MAX_FAULT_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_FAULT_DETAIL_CHARS - 1] + "…"
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: holistic pass failed — criterion verdicts stand "
|
||||||
|
f"but cross-cutting hunt did not run ({detail})"
|
||||||
|
)
|
||||||
|
return faults
|
||||||
|
|
||||||
|
|
||||||
|
def append_criterion_complaints(lines, verdicts, extra, start):
|
||||||
|
"""Numbered complaints for non-MET verdicts and holistic extras."""
|
||||||
|
i = start
|
||||||
|
for v in verdicts:
|
||||||
|
if v.get("status") == "MET" and not died(v):
|
||||||
|
continue
|
||||||
|
i += 1
|
||||||
|
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||||
|
if len(text) > 140:
|
||||||
|
text = text[:137] + "…"
|
||||||
|
complaint = (v.get("complaint") or "").strip()
|
||||||
|
if died(v):
|
||||||
|
if not complaint.startswith(DIED_MARKER):
|
||||||
|
complaint = f"{DIED_MARKER} — {complaint}" if complaint else DIED_MARKER
|
||||||
|
lines.append(f'{i}. Acceptance criterion "{text}" — {complaint}')
|
||||||
|
else:
|
||||||
|
lines.append(
|
||||||
|
f'{i}. Acceptance criterion "{text}" — {v.get("status", "?").title()} — '
|
||||||
|
f"{complaint}"
|
||||||
|
)
|
||||||
|
for c in extra:
|
||||||
|
i += 1
|
||||||
|
lines.append(
|
||||||
|
f"{i}. {c.get('kind', 'violation')} — {c.get('location', '?')} — "
|
||||||
|
f"{c.get('violation', '')} — {c.get('fix', '')}"
|
||||||
|
)
|
||||||
|
return i
|
||||||
|
|
||||||
|
|
||||||
|
def declared_nonempty(declared):
|
||||||
|
"""True when verification_commands names at least one command.
|
||||||
|
|
||||||
|
Accepts a list or a JSON-encoded string of one. None, unparseable, or
|
||||||
|
empty declarations count as not declared — run_checks already records
|
||||||
|
a PIPELINE-FAULT for the invalid ones.
|
||||||
|
"""
|
||||||
|
if isinstance(declared, str):
|
||||||
|
try:
|
||||||
|
declared = json.loads(declared)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return False
|
||||||
|
if not isinstance(declared, list):
|
||||||
|
return False
|
||||||
|
return any(isinstance(c, str) and c.strip() for c in declared)
|
||||||
|
|
||||||
|
|
||||||
|
def exec_run_complaints(exec_results, declared):
|
||||||
|
"""One complaint per red verification run recorded by run_checks.py.
|
||||||
|
|
||||||
|
A red run is a record that was skipped (deadline) or exited nonzero
|
||||||
|
(including the -1 TIMEOUT sentinel). An "ENVIRONMENT: …" runner-error
|
||||||
|
marker with a non-empty declaration is one complaint: the declared
|
||||||
|
commands never ran. Other string markers ("none declared",
|
||||||
|
"PIPELINE-FAULT…") are not runs and yield nothing.
|
||||||
|
"""
|
||||||
|
if isinstance(exec_results, str):
|
||||||
|
marker = exec_results.strip().replace("\n", " ")
|
||||||
|
if not marker.startswith("ENVIRONMENT:") or not declared_nonempty(declared):
|
||||||
|
return []
|
||||||
|
if len(marker) > MAX_MARKER_CHARS:
|
||||||
|
marker = marker[: MAX_MARKER_CHARS - 1] + "…"
|
||||||
|
return [
|
||||||
|
f"Verification runner error — the declared command(s) never ran ({marker}); "
|
||||||
|
"unproven runs block CONFORMS"
|
||||||
|
]
|
||||||
|
if not isinstance(exec_results, list):
|
||||||
|
return []
|
||||||
|
out = []
|
||||||
|
for r in exec_results:
|
||||||
|
if not isinstance(r, dict):
|
||||||
|
continue
|
||||||
|
cmd = r.get("cmd", "?")
|
||||||
|
if r.get("skipped"):
|
||||||
|
out.append(f"Verification run `{cmd}` — never ran: deadline expired (unproven)")
|
||||||
|
continue
|
||||||
|
code = r.get("exit")
|
||||||
|
if code == 0:
|
||||||
|
continue
|
||||||
|
outcome = "TIMEOUT (exit -1)" if code == -1 else f"exit {code}"
|
||||||
|
out.append(
|
||||||
|
f"Verification run `{cmd}` — {outcome} (recorded FAIL; a red declared "
|
||||||
|
"command blocks CONFORMS regardless of criterion judgments)"
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def render_exec_results(exec_results):
|
||||||
|
"""Lines for the "Verification runs:" report section.
|
||||||
|
|
||||||
|
exec_results is a list of {cmd, exit, duration_s, tail} records (or
|
||||||
|
{cmd, skipped} for commands the runner's total deadline left no budget
|
||||||
|
for) from run_checks.py, or a string marker ("none declared —…" /
|
||||||
|
"ENVIRONMENT: …"), or empty when the run never reached the stage.
|
||||||
|
"""
|
||||||
|
if isinstance(exec_results, list):
|
||||||
|
out = []
|
||||||
|
for r in exec_results:
|
||||||
|
if not isinstance(r, dict):
|
||||||
|
continue
|
||||||
|
if r.get("skipped"):
|
||||||
|
out.append(
|
||||||
|
f"- [SKIPPED] `{r.get('cmd', '?')}` — never ran: the runner's "
|
||||||
|
"total deadline expired first; covered criteria are unproven"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
status = "PASS" if r.get("exit") == 0 else "FAIL"
|
||||||
|
out.append(
|
||||||
|
f"- [{status}] `{r.get('cmd', '?')}` — exit {r.get('exit', '?')}, "
|
||||||
|
f"{r.get('duration_s', '?')}s"
|
||||||
|
)
|
||||||
|
if status == "FAIL":
|
||||||
|
tail = (r.get("tail") or "").strip()
|
||||||
|
out.extend(" " + t for t in tail.splitlines()[-15:])
|
||||||
|
return out or ["- (no verification commands were run)"]
|
||||||
|
if isinstance(exec_results, str) and exec_results.strip():
|
||||||
|
return [f"- {exec_results.strip()}"]
|
||||||
|
return ["- none recorded (the run did not reach the verification stage)"]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
verdicts = [v for v in (state.get("crit_verdicts") or []) if isinstance(v, dict)]
|
||||||
|
extra = [c for c in (state.get("extra_complaints") or []) if isinstance(c, dict)]
|
||||||
|
observations = (state.get("observations") or "").strip()
|
||||||
|
faults = collect_pipeline_faults(state)
|
||||||
|
reds = exec_run_complaints(
|
||||||
|
state.get("exec_results"), state.get("verification_commands")
|
||||||
|
)
|
||||||
|
n = len(verdicts)
|
||||||
|
|
||||||
|
met = [v for v in verdicts if v.get("status") == "MET" and not died(v)]
|
||||||
|
partial = [v for v in verdicts if v.get("status") == "PARTIAL" and not died(v)]
|
||||||
|
bad = [
|
||||||
|
v for v in verdicts if died(v) or v.get("status") in ("UNMET", "DIVERGED")
|
||||||
|
]
|
||||||
|
|
||||||
|
lines = []
|
||||||
|
if faults:
|
||||||
|
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||||
|
if n:
|
||||||
|
lines.append(
|
||||||
|
f"Criteria: {len(met)}/{n} met, {len(partial)} partial, "
|
||||||
|
f"{len(bad)} unmet/diverged — degraded run: pipeline fault recorded "
|
||||||
|
"(fail-closed)."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
lines.append(
|
||||||
|
"Criteria: none verified — degraded run: pipeline fault recorded "
|
||||||
|
"(fail-closed)."
|
||||||
|
)
|
||||||
|
lines.append("Complaints:")
|
||||||
|
i = 0
|
||||||
|
for f in faults:
|
||||||
|
i += 1
|
||||||
|
lines.append(f"{i}. {f}")
|
||||||
|
i = append_criterion_complaints(lines, verdicts, extra, i)
|
||||||
|
for c in reds:
|
||||||
|
i += 1
|
||||||
|
lines.append(f"{i}. {c}")
|
||||||
|
if observations:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Non-blocking observations:")
|
||||||
|
lines.append(observations)
|
||||||
|
elif n == 0:
|
||||||
|
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||||
|
lines.append("Criteria: none provided.")
|
||||||
|
lines.append("Complaints:")
|
||||||
|
lines.append(
|
||||||
|
"1. No acceptance criteria were provided or extractable — conformance cannot "
|
||||||
|
"be judged without a spec (fail-closed). Supply the plan's Objective/Tasks/"
|
||||||
|
"Acceptance criteria and re-run."
|
||||||
|
)
|
||||||
|
for i, c in enumerate(reds, start=2):
|
||||||
|
lines.append(f"{i}. {c}")
|
||||||
|
elif not partial and not bad and not extra and not reds:
|
||||||
|
lines.append("ADVERSARIAL_REVIEW: CONFORMS")
|
||||||
|
lines.append(f"Criteria: {n}/{n} met (all with tests).")
|
||||||
|
if observations:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Non-blocking observations:")
|
||||||
|
lines.append(observations)
|
||||||
|
else:
|
||||||
|
lines.append("ADVERSARIAL_REVIEW: DIVERGES")
|
||||||
|
header = f"Criteria: {len(met)}/{n} met, {len(partial)} partial, {len(bad)} unmet/diverged"
|
||||||
|
notes = []
|
||||||
|
if died_count := sum(1 for v in verdicts if died(v)):
|
||||||
|
notes.append(f"degraded run: {died_count} criterion check(s) died (fail-closed)")
|
||||||
|
if reds:
|
||||||
|
notes.append(f"{len(reds)} red verification run(s) (fail-closed)")
|
||||||
|
lines.append(header + "".join(f" — {note}" for note in notes) + ".")
|
||||||
|
lines.append("Complaints:")
|
||||||
|
i = append_criterion_complaints(lines, verdicts, extra, 0)
|
||||||
|
for c in reds:
|
||||||
|
i += 1
|
||||||
|
lines.append(f"{i}. {c}")
|
||||||
|
if observations:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Non-blocking observations:")
|
||||||
|
lines.append(observations)
|
||||||
|
|
||||||
|
# Per-criterion evidence appendix (MET entries, for auditability).
|
||||||
|
if met:
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Met criteria (evidence):")
|
||||||
|
for v in met:
|
||||||
|
text = (v.get("text") or "").strip().replace("\n", " ")
|
||||||
|
if len(text) > 100:
|
||||||
|
text = text[:97] + "…"
|
||||||
|
lines.append(f'- "{text}" — {(v.get("evidence") or "").strip()}')
|
||||||
|
|
||||||
|
lines.append("")
|
||||||
|
lines.append("Verification runs:")
|
||||||
|
lines.extend(render_exec_results(state.get("exec_results")))
|
||||||
|
|
||||||
|
print(json.dumps({"adv_report": "\n".join(lines)}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — never a silent verdict
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"adv_report": (
|
||||||
|
"ADVERSARIAL_REVIEW: DIVERGES\n"
|
||||||
|
f"Criteria: verdict computation error: {e} — treat as failed, not as conforming."
|
||||||
|
)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -1,78 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -eo pipefail
|
|
||||||
|
|
||||||
# @env LLM_OUTPUT=/dev/stdout
|
|
||||||
# @env LLM_AGENT_VAR_PROJECT_DIR=.
|
|
||||||
# @describe Adversarial plan-conformance reviewer tools
|
|
||||||
|
|
||||||
_project_dir() {
|
|
||||||
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
|
|
||||||
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# @cmd Get the git diff to review for plan conformance. Returns staged changes, or unstaged if nothing is staged, or the HEAD~1 diff if the working tree is clean.
|
|
||||||
# @option --base Optional base ref to diff against (e.g., "main", "HEAD~3", a commit SHA, or a PR base branch)
|
|
||||||
get_diff() {
|
|
||||||
local project_dir
|
|
||||||
project_dir=$(_project_dir)
|
|
||||||
# shellcheck disable=SC2154
|
|
||||||
local base="${argc_base:-}"
|
|
||||||
|
|
||||||
local diff_output=""
|
|
||||||
if [[ -n "${base}" ]]; then
|
|
||||||
diff_output=$(cd "${project_dir}" && git diff "${base}" 2>&1) || true
|
|
||||||
else
|
|
||||||
diff_output=$(cd "${project_dir}" && git diff --cached 2>&1) || true
|
|
||||||
if [[ -z "${diff_output}" ]]; then
|
|
||||||
diff_output=$(cd "${project_dir}" && git diff 2>&1) || true
|
|
||||||
fi
|
|
||||||
if [[ -z "${diff_output}" ]]; then
|
|
||||||
diff_output=$(cd "${project_dir}" && git diff HEAD~1 2>&1) || true
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -z "${diff_output}" ]]; then
|
|
||||||
echo "No changes found to review in ${project_dir}." >> "$LLM_OUTPUT"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
local file_count
|
|
||||||
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
|
|
||||||
{
|
|
||||||
echo "Diff contains changes to ${file_count} file(s):"
|
|
||||||
echo ""
|
|
||||||
echo "${diff_output}"
|
|
||||||
} >> "$LLM_OUTPUT"
|
|
||||||
}
|
|
||||||
|
|
||||||
# @cmd Get the list of changed files with stats (a quick map of what to check against the plan).
|
|
||||||
# @option --base Optional base ref to diff against
|
|
||||||
get_changed_files() {
|
|
||||||
local project_dir
|
|
||||||
project_dir=$(_project_dir)
|
|
||||||
local base="${argc_base:-}"
|
|
||||||
|
|
||||||
local stat_output=""
|
|
||||||
if [[ -n "${base}" ]]; then
|
|
||||||
stat_output=$(cd "${project_dir}" && git diff --stat "${base}" 2>&1) || true
|
|
||||||
else
|
|
||||||
stat_output=$(cd "${project_dir}" && git diff --cached --stat 2>&1) || true
|
|
||||||
if [[ -z "${stat_output}" ]]; then
|
|
||||||
stat_output=$(cd "${project_dir}" && git diff --stat 2>&1) || true
|
|
||||||
fi
|
|
||||||
if [[ -z "${stat_output}" ]]; then
|
|
||||||
stat_output=$(cd "${project_dir}" && git diff --stat HEAD~1 2>&1) || true
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -z "${stat_output}" ]]; then
|
|
||||||
echo "No changes found in ${project_dir}." >> "$LLM_OUTPUT"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
{
|
|
||||||
echo "Changed files:"
|
|
||||||
echo ""
|
|
||||||
echo "${stat_output}"
|
|
||||||
} >> "$LLM_OUTPUT"
|
|
||||||
}
|
|
||||||
@@ -137,6 +137,7 @@ coyote -a architect \
|
|||||||
| `design_doc` | *(empty)* | Path to the design doc; asked for if unset. |
|
| `design_doc` | *(empty)* | Path to the design doc; asked for if unset. |
|
||||||
| `base_branch` | `main` | Branch the run branch forks from and the PR targets. |
|
| `base_branch` | `main` | Branch the run branch forks from and the PR targets. |
|
||||||
| `autonomy` | `full` | `full` (no gates) · `plan-gate` (approve breakdown once) · `phase-gate` (approve each task). |
|
| `autonomy` | `full` | `full` (no gates) · `plan-gate` (approve breakdown once) · `phase-gate` (approve each task). |
|
||||||
|
| `escalation_policy` | `interactive` | `interactive` (pause and ask) · `autonomous` (self-resolve decisions covered by a standing ruling in `~/.config/coyote/share/standing-rulings.md` or a clearly recommended option; everything self-resolved lands in a “Decisions taken autonomously” digest in the Phase F report and PR body; secrets/money/destructive actions and novel conflicts still escalate). |
|
||||||
| `parallel_tasks` | `0` | `0` = sequential (default) · `1` = opt-in worktree-parallel execution for eligible tasks. |
|
| `parallel_tasks` | `0` | `0` = sequential (default) · `1` = opt-in worktree-parallel execution for eligible tasks. |
|
||||||
| `auto_confirm` | `1` | Skip the shell confirm guard (needed for non-interactive autonomous runs). |
|
| `auto_confirm` | `1` | Skip the shell confirm guard (needed for non-interactive autonomous runs). |
|
||||||
|
|
||||||
@@ -159,6 +160,9 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
|||||||
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
|
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
|
||||||
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
|
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
|
||||||
rebase and re-verify.
|
rebase and re-verify.
|
||||||
|
- Optionally, the `agent__task_*` queue may mirror the on-disk `blocked_by` graph so completing a
|
||||||
|
task auto-dispatches its unblocked dependents. The queue is in-memory convenience only — disk task
|
||||||
|
files remain the durable source of truth.
|
||||||
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
|
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
|
||||||
CI-check watch) is unchanged in both modes.
|
CI-check watch) is unchanged in both modes.
|
||||||
|
|
||||||
@@ -172,6 +176,8 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
|||||||
| [`oracle`](../oracle/README.md) | Plan review (`plan-review`); diagnosis when a task fails after Sisyphus recovery. |
|
| [`oracle`](../oracle/README.md) | Plan review (`plan-review`); diagnosis when a task fails after Sisyphus recovery. |
|
||||||
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
|
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
|
||||||
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
|
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
|
||||||
|
| [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. |
|
||||||
|
| [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; a `GAUNTLET_REVIEW_INCOMPLETE:` line is an infrastructure fault escalated to the user (retry / accept NEEDS-HUMAN / abort), never a divergence; direct adversary/probe spawns remain the fallback. |
|
||||||
|
|
||||||
## Related skills
|
## Related skills
|
||||||
|
|
||||||
@@ -179,4 +185,3 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
|||||||
- [`task-tracking`](../../skills/task-tracking/SKILL.md) — the task-file schema, lifecycle, and consistency checks.
|
- [`task-tracking`](../../skills/task-tracking/SKILL.md) — the task-file schema, lifecycle, and consistency checks.
|
||||||
- [`plan-gatekeeping`](../../skills/plan-gatekeeping/SKILL.md) — the gatekeeper's self-containedness manifest.
|
- [`plan-gatekeeping`](../../skills/plan-gatekeeping/SKILL.md) — the gatekeeper's self-containedness manifest.
|
||||||
- [`plan-authoring`](../../skills/plan-authoring/SKILL.md) / [`plan-review`](../../skills/plan-review/SKILL.md) — plan schema + oracle's executability review.
|
- [`plan-authoring`](../../skills/plan-authoring/SKILL.md) / [`plan-review`](../../skills/plan-review/SKILL.md) — plan schema + oracle's executability review.
|
||||||
- [`adversarial-review`](../../skills/adversarial-review/SKILL.md) — the adversary's conformance methodology.
|
|
||||||
@@ -6,7 +6,7 @@ description: |
|
|||||||
plan-conformance check (plus a black-box usage-pattern probe for consumer-facing surface), and
|
plan-conformance check (plus a black-box usage-pattern probe for consumer-facing surface), and
|
||||||
finishes with ONE draft PR (CI checks watched to green) plus tracked follow-up tasks. Task state
|
finishes with ONE draft PR (CI checks watched to green) plus tracked follow-up tasks. Task state
|
||||||
lives on disk in a plans directory, so runs survive context compression.
|
lives on disk in a plans directory, so runs survive context compression.
|
||||||
version: 2.2.0
|
version: 2.8.0
|
||||||
agent_session: temp
|
agent_session: temp
|
||||||
auto_continue: true
|
auto_continue: true
|
||||||
max_auto_continues: 100
|
max_auto_continues: 100
|
||||||
@@ -18,9 +18,12 @@ spawnable_agents:
|
|||||||
- oracle
|
- oracle
|
||||||
- explore
|
- explore
|
||||||
- librarian
|
- librarian
|
||||||
|
- deep-research
|
||||||
- adversary
|
- adversary
|
||||||
- probe
|
- probe
|
||||||
|
- review-gauntlet
|
||||||
- gatekeeper
|
- gatekeeper
|
||||||
|
- whetstone
|
||||||
max_concurrent_agents: 10
|
max_concurrent_agents: 10
|
||||||
max_agent_depth: 10
|
max_agent_depth: 10
|
||||||
inject_spawn_instructions: true
|
inject_spawn_instructions: true
|
||||||
@@ -52,6 +55,9 @@ variables:
|
|||||||
- name: autonomy
|
- name: autonomy
|
||||||
description: 'How autonomous the run is: full (no gates), plan-gate (approve breakdown once, then autonomous), phase-gate (approve each task)'
|
description: 'How autonomous the run is: full (no gates), plan-gate (approve breakdown once, then autonomous), phase-gate (approve each task)'
|
||||||
default: full
|
default: full
|
||||||
|
- name: escalation_policy
|
||||||
|
description: "How escalation-worthy decisions (yours and relayed child escalations) are handled: 'interactive' pauses and asks the user; 'autonomous' self-resolves decisions covered by a standing ruling or a clearly recommended option, reports them in the Phase F digest, and only pings on hard triggers or novel conflicts"
|
||||||
|
default: 'interactive'
|
||||||
- name: auto_confirm
|
- name: auto_confirm
|
||||||
description: Auto-confirm command execution (1 = skip the shell guard_operation TTY prompt, needed for non-interactive autonomous runs)
|
description: Auto-confirm command execution (1 = skip the shell guard_operation TTY prompt, needed for non-interactive autonomous runs)
|
||||||
default: '1'
|
default: '1'
|
||||||
@@ -113,6 +119,37 @@ instructions: |
|
|||||||
AFTER its merge landed and integration tests passed is routine documented cleanup, not a
|
AFTER its merge landed and integration tests passed is routine documented cleanup, not a
|
||||||
destructive action.
|
destructive action.
|
||||||
|
|
||||||
|
## Escalation policy: {{escalation_policy}}
|
||||||
|
|
||||||
|
Orthogonal to autonomy mode: autonomy decides which GATES pause the run; escalation policy decides
|
||||||
|
what happens when a genuine decision point fires — yours, or a child agent's escalation.
|
||||||
|
|
||||||
|
- **interactive** (default) — escalate to the user exactly as written throughout this config.
|
||||||
|
- **autonomous** — the user pre-delegated decision-making. Before pinging the user (and before
|
||||||
|
relaying any child `pending_escalations` to them), work this ladder IN ORDER:
|
||||||
|
1. **Hard triggers — ALWAYS the user's; no ruling can delegate them:** secrets/credentials,
|
||||||
|
money movement, destructive or irreversible actions, anything the Hard blocks / Anti-patterns
|
||||||
|
sections forbid — and any decision that already failed once after an autonomous resolution.
|
||||||
|
2. **Standing rulings** — read `$HOME/.config/coyote/share/standing-rulings.md` (skip silently
|
||||||
|
if absent). A ruling whose trigger GENUINELY matches resolves the decision: apply it, or
|
||||||
|
answer the blocked child via `agent__reply_escalation`. Match on substance, not resemblance;
|
||||||
|
unsure means no match.
|
||||||
|
3. **Clearly recommended option** — if the decision carries options with one clearly recommended
|
||||||
|
on concrete evidence (a reviewer's proposed fix, a repo convention), take it.
|
||||||
|
4. **Otherwise — escalate** as interactive mode would. Autonomy never converts "I don't know"
|
||||||
|
into a guess.
|
||||||
|
|
||||||
|
When a resolution amends a plan/task's acceptance criteria, the criterion TEXT must be rewritten
|
||||||
|
before any conformance re-run — when the decision was a child's, your `agent__reply_escalation`
|
||||||
|
reply must instruct it to amend its plan context (the adversary verifies the plan-as-amended,
|
||||||
|
not the stale wording).
|
||||||
|
|
||||||
|
Every autonomous resolution — yours and the ones you replied to children — gets one line in
|
||||||
|
Phase F's `## Decisions taken autonomously` rollup: what was decided, the basis (`ruling:
|
||||||
|
<id/title>` or `recommended option`), and what changed (including plan amendments). When a novel
|
||||||
|
decision DID reach the user, suggest `coyote --macro record-ruling` in the final report so that
|
||||||
|
class never asks again.
|
||||||
|
|
||||||
## The pipeline (drive this to completion)
|
## The pipeline (drive this to completion)
|
||||||
|
|
||||||
### Phase A — Orient (once, at startup)
|
### Phase A — Orient (once, at startup)
|
||||||
@@ -144,7 +181,9 @@ instructions: |
|
|||||||
<label>; authoritative sources preferred; return a distilled, deduplicated checklist"), put the
|
<label>; authoritative sources preferred; return a distilled, deduplicated checklist"), put the
|
||||||
returned checklist through the same accept/drop round with the user, and write the ACCEPTED
|
returned checklist through the same accept/drop round with the user, and write the ACCEPTED
|
||||||
items into the plan — as measurable acceptance criteria on the relevant tasks where possible,
|
items into the plan — as measurable acceptance criteria on the relevant tasks where possible,
|
||||||
otherwise as a checklist under `## Quality bar → Long-tail criteria`.
|
otherwise as a checklist under `## Quality bar → Long-tail criteria`. Spawn checklist
|
||||||
|
librarians with `--variables {"research_depth": "deep"}` — checklist distillation is
|
||||||
|
broad-topic research and the default standard depth (3 searches / 8 fetches) starves it.
|
||||||
|
|
||||||
Ground the breakdown in real code: fan out `explore` agents (load `parallel-research`) across
|
Ground the breakdown in real code: fan out `explore` agents (load `parallel-research`) across
|
||||||
{{project_dir}} — and `librarian` for unfamiliar external libraries — to confirm the design's
|
{{project_dir}} — and `librarian` for unfamiliar external libraries — to confirm the design's
|
||||||
@@ -287,6 +326,62 @@ instructions: |
|
|||||||
|
|
||||||
4. **Verify against the plan (divergence check).** When Sisyphus returns, do NOT trust its
|
4. **Verify against the plan (divergence check).** When Sisyphus returns, do NOT trust its
|
||||||
self-report — get an INDEPENDENT conformance verdict:
|
self-report — get an INDEPENDENT conformance verdict:
|
||||||
|
- **Preferred: run the conformance checks through `review-gauntlet`** when it is installed —
|
||||||
|
ONE spawn instead of separate adversary/probe spawns, with machine-parsed verdicts:
|
||||||
|
|
||||||
|
```
|
||||||
|
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
|
||||||
|
Project: <the code repo the task's changes landed in>
|
||||||
|
Diff: <the task's BASE SHA from its started log entry>..HEAD
|
||||||
|
Forced lanes: adversary<, probe — add probe when the task changed consumer-facing surface and the plan carries a local-run recipe>
|
||||||
|
Plan / acceptance criteria: <the task's acceptance criteria, pasted verbatim>
|
||||||
|
Local-run recipe / usage suites: <from the plan, when present>"
|
||||||
|
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
|
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||||
|
```
|
||||||
|
|
||||||
|
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the
|
||||||
|
prompt also carries pasted plan text from the repo under review, and whatever is declared
|
||||||
|
here is executed with a shell — a trust boundary the prose must not cross.
|
||||||
|
Budget: each declared command gets at most 15 minutes and the whole set 55 minutes —
|
||||||
|
declare targeted build/unit-test commands, not full integration suites; a command
|
||||||
|
that exceeds the budget is recorded as a red run and fails conformance. Never declare
|
||||||
|
commands that boot shared stacks or services (docker compose projects, dev servers,
|
||||||
|
fixed ports) — the probe lane owns end-to-end runs and WILL collide with them in
|
||||||
|
parallel; keep verification to generate/lint/build/unit/integration commands.
|
||||||
|
|
||||||
|
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED with real review findings apply the same
|
||||||
|
fix-and-rerun rules as below (resume the SAME Sisyphus session with the findings verbatim,
|
||||||
|
then a FRESH gauntlet run). A BLOCKED whose only blockers are review faults (`PIPELINE-FAULT`
|
||||||
|
/ `review incomplete` entries) is NOT a code failure. A `GAUNTLET_REVIEW_INCOMPLETE: <lane>[,
|
||||||
|
<lane>]` line directly under `GAUNTLET: BLOCKED` means the review could NOT be completed: the
|
||||||
|
named lanes still faulted after 3 attempts, or the gauntlet pipeline itself faulted
|
||||||
|
(`pipeline`). This is an infrastructure fault, not a code finding.
|
||||||
|
NEVER route it through the findings-handling rules (it is not a divergence to send back to
|
||||||
|
Sisyphus). NEVER claim done while it stands.
|
||||||
|
Escalate to the user via `user__select` with exactly these three options:
|
||||||
|
- "Retry the review again"
|
||||||
|
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
|
||||||
|
- "Abort this task"
|
||||||
|
Retry → a FRESH gauntlet run. Accept → close the task and record the skipped review (lane
|
||||||
|
names or `pipeline`) in the task's `log.md`, the Phase F digest, and the PR body. Abort →
|
||||||
|
leave the task `in-progress`, log the abort in its `log.md`, and stop to report. Findings AND
|
||||||
|
incomplete on the same output → fix the findings first, then re-run the gauntlet (the re-run
|
||||||
|
retries the lane) — escalate only if the line persists. Under `escalation_policy: autonomous`
|
||||||
|
the standard ladder applies (a standing ruling may pre-answer this decision).
|
||||||
|
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed;
|
||||||
|
on that path an adversary/probe result that is ONLY a `PIPELINE-FAULT` (no verdict on the
|
||||||
|
code) is likewise an infrastructure fault to retry, never a divergence.
|
||||||
|
Each gauntlet round's outcome is recorded on the task's PR as a NEW comment titled
|
||||||
|
`## Review round N @ <full head SHA>` (lane table, verdicts, what changed since the
|
||||||
|
prior round) — never by rewriting the PR description, which carries only the stable
|
||||||
|
summary and the required digest sections; comments are the append-only review log.
|
||||||
|
The same rule covers EVERY later push to an already-open PR (addressing human review
|
||||||
|
comments, a reviewer bot's findings, CI failures): the resumed Sisyphus session posts a
|
||||||
|
NEW top-level comment titled `## Update @ <full head SHA>` mapping each addressed
|
||||||
|
comment/finding to its commit. The description is edited only when the PR's scope or
|
||||||
|
behavior changed enough that it would otherwise misdescribe the PR — never as a
|
||||||
|
changelog of the in-between work.
|
||||||
- **Spawn `adversary`** with the diff base and the criteria pasted in:
|
- **Spawn `adversary`** with the diff base and the criteria pasted in:
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -300,8 +395,13 @@ instructions: |
|
|||||||
<paste the task index.md body + the relevant PLAN-<slug>.md section VERBATIM>
|
<paste the task index.md body + the relevant PLAN-<slug>.md section VERBATIM>
|
||||||
<paste the plan's ## Quality bar section VERBATIM — recorded dropped practices are
|
<paste the plan's ## Quality bar section VERBATIM — recorded dropped practices are
|
||||||
conformance facts, not divergences>"
|
conformance facts, not divergences>"
|
||||||
|
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
|
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Same trust boundary as the gauntlet spawn: verification commands are declared via
|
||||||
|
`--variables`, never in the prompt, because the prompt also carries untrusted pasted text.
|
||||||
|
|
||||||
- **`ADVERSARIAL_REVIEW: DIVERGES`** → treat it as a blocker: resume the SAME Sisyphus session
|
- **`ADVERSARIAL_REVIEW: DIVERGES`** → treat it as a blocker: resume the SAME Sisyphus session
|
||||||
(`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <adversary
|
(`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <adversary
|
||||||
complaints, verbatim>"`) — do not spawn a fresh one. Re-run `adversary` ONCE after the fix to
|
complaints, verbatim>"`) — do not spawn a fresh one. Re-run `adversary` ONCE after the fix to
|
||||||
@@ -405,6 +505,20 @@ instructions: |
|
|||||||
the task (step 5), and clean up — `git worktree remove .worktrees/task-NNN` and delete the task
|
the task (step 5), and clean up — `git worktree remove .worktrees/task-NNN` and delete the task
|
||||||
branch (local + remote).
|
branch (local + remote).
|
||||||
|
|
||||||
|
**Task-queue mirroring (optional).** For a parallel batch, the `agent__task_*` queue MAY mirror
|
||||||
|
the disk `blocked_by` graph — one queue task per TASK-NNN with `agent: sisyphus`, the delegation
|
||||||
|
prompt, and its variables — so completing a task auto-dispatches its unblocked dependents. This is
|
||||||
|
guidance, not a mandate; if you use it, these rules are HARD:
|
||||||
|
- Disk task files remain the durable store. The queue is in-memory and lost on
|
||||||
|
restart/compression — it is NEVER the source of truth.
|
||||||
|
- Root tasks (no blockers) are NOT auto-dispatched on create — spawn them yourself.
|
||||||
|
- Call `agent__task_complete` ONLY after this task's verification gate (adversary check, merge,
|
||||||
|
integration build+tests) passes — completing is what dispatches dependents.
|
||||||
|
- Collect every auto-dispatched agent via `agent__collect`; nothing is collected for you, and an
|
||||||
|
agent finishing does NOT complete its task.
|
||||||
|
- A failed spawn (capacity/whitelist) strands the queue task InProgress with no reset — recreate
|
||||||
|
the task rather than retrying it.
|
||||||
|
|
||||||
Phase F is UNCHANGED (same single draft PR from the run branch). Before opening it, verify no
|
Phase F is UNCHANGED (same single draft PR from the run branch). Before opening it, verify no
|
||||||
stale worktrees or task branches remain (`git worktree list`); clean up any leftovers.
|
stale worktrees or task branches remain (`git worktree list`); clean up any leftovers.
|
||||||
|
|
||||||
@@ -427,6 +541,9 @@ instructions: |
|
|||||||
- a checklist of every TASK-NNN (title + commit SHAs),
|
- a checklist of every TASK-NNN (title + commit SHAs),
|
||||||
- a **`## Review decisions`** section: one line per collected `rejected-finding:` entry;
|
- a **`## Review decisions`** section: one line per collected `rejected-finding:` entry;
|
||||||
omit the section entirely when there are none,
|
omit the section entirely when there are none,
|
||||||
|
- a **`## Decisions taken autonomously`** section (only when `escalation_policy: autonomous`
|
||||||
|
resolved anything): one line per self-resolved decision with its basis and what changed —
|
||||||
|
rolled up from your own resolutions and each Sisyphus report's section; omit otherwise,
|
||||||
- a **`## Follow-up / manual actions`** section: one checkbox line per follow-up (WHAT, WHERE,
|
- a **`## Follow-up / manual actions`** section: one checkbox line per follow-up (WHAT, WHERE,
|
||||||
WHY, WHEN — pre-merge items FIRST and clearly marked), or "None." if there are none. This
|
WHY, WHEN — pre-merge items FIRST and clearly marked), or "None." if there are none. This
|
||||||
section is the reviewer's contract for what the code does NOT do by itself.
|
section is the reviewer's contract for what the code does NOT do by itself.
|
||||||
@@ -452,7 +569,16 @@ instructions: |
|
|||||||
the run branch (`chore(plan): ...`) — they become part of the PR.
|
the run branch (`chore(plan): ...`) — they become part of the PR.
|
||||||
6. Run the `task-tracking` consistency checks; fix anything you introduced.
|
6. Run the `task-tracking` consistency checks; fix anything you introduced.
|
||||||
7. Report: the PLAN, every TASK-NNN with its commits, the single draft PR URL with checks green,
|
7. Report: the PLAN, every TASK-NNN with its commits, the single draft PR URL with checks green,
|
||||||
the follow-up TASKs created (with their WHEN), and anything deferred/escalated. STOP.
|
the follow-up TASKs created (with their WHEN), and anything deferred/escalated — plus the
|
||||||
|
`## Decisions taken autonomously` rollup when the escalation policy resolved anything. STOP.
|
||||||
|
|
||||||
|
Continuous improvement: if any part of this run addressed PR review feedback that did not come
|
||||||
|
from the run's own reviewer lanes — humans, a teammate's review agent, another team's bot — spawn
|
||||||
|
`whetstone` ONCE before the final report: pass the PR ref, the human feedback verbatim, and
|
||||||
|
what the run's own review passes flagged. Collect it; its approval questions escalate to the
|
||||||
|
user directly. Never block the run's completion on it; skip silently when all feedback came from
|
||||||
|
the run's own lanes. (Per-task feedback handled inside a Sisyphus session is covered by Sisyphus's
|
||||||
|
own whetstone hook — don't double-spawn for the same feedback.)
|
||||||
|
|
||||||
## Durable state (survive context compression)
|
## Durable state (survive context compression)
|
||||||
|
|
||||||
@@ -470,11 +596,13 @@ instructions: |
|
|||||||
|-------|---------|
|
|-------|---------|
|
||||||
| `sisyphus` | Implement ONE task's code in {{project_dir}} (its own explore/coder/verify/review loop). One per task. |
|
| `sisyphus` | Implement ONE task's code in {{project_dir}} (its own explore/coder/verify/review loop). One per task. |
|
||||||
| `explore` | Ground the design/plan in real code in {{project_dir}}; read other local repos for library usage/legacy patterns/call sites. Fan out in parallel. |
|
| `explore` | Ground the design/plan in real code in {{project_dir}}; read other local repos for library usage/legacy patterns/call sites. Fan out in parallel. |
|
||||||
| `librarian` | External docs/OSS examples for unfamiliar libraries the design touches. |
|
| `librarian` | External docs/OSS examples for unfamiliar libraries the design touches. Design-time/broad topics: pass `--variables {"research_depth": "deep"}` (~3× budgets); default depth is for narrow lookups. One librarian per question beats one broad librarian — deep is for inherently broad questions. |
|
||||||
|
| `deep-research` | Open-ended/exhaustive research that should churn until its critique loop passes (plans sub-questions, deep librarian per question, credibility vetting). Spawn with `--variables {"autonomy": "autonomous"}` — MANDATORY, the supervised default blocks on a human approval gate; raise `reflexion_budget` (default 2) for longer churn. |
|
||||||
| `oracle` | Plan review (`plan-review`), and diagnosis when a task fails after Sisyphus recovery. |
|
| `oracle` | Plan review (`plan-review`), and diagnosis when a task fails after Sisyphus recovery. |
|
||||||
| `gatekeeper` | Plan self-containedness gate (Phase C Gate 1): audits the PLAN for the "docker container" standard, returns SEALED/LEAKY with the missing implementer questions. |
|
| `gatekeeper` | Plan self-containedness gate (Phase C Gate 1): audits the PLAN for the "docker container" standard, returns SEALED/LEAKY with the missing implementer questions. |
|
||||||
| `adversary` | Post-implementation plan-conformance verdict per task (CONFORMS/DIVERGES). |
|
| `adversary` | Post-implementation plan-conformance verdict per task (CONFORMS/DIVERGES). |
|
||||||
| `probe` | Black-box behavioral verdict on a task's consumer-facing surface: boots the code locally from clean state, runs existing usage suites + spec-first tests. Returns USAGE_PROBE PASS/FAIL/INCONCLUSIVE. |
|
| `probe` | Black-box behavioral verdict on a task's consumer-facing surface: boots the code locally from clean state, runs existing usage suites + spec-first tests. Returns USAGE_PROBE PASS/FAIL/INCONCLUSIVE. |
|
||||||
|
| `whetstone` | Post-run continuous improvement: classifies human review feedback vs what the suite flagged, generalizes misses, proposes suite edits / ledger entries (user-approved). Never a reviewer lane. |
|
||||||
|
|
||||||
## Escalation handling
|
## Escalation handling
|
||||||
|
|
||||||
@@ -482,6 +610,11 @@ instructions: |
|
|||||||
Answer from context if you can, else prompt the user, then `agent__reply_escalation` to unblock the
|
Answer from context if you can, else prompt the user, then `agent__reply_escalation` to unblock the
|
||||||
child. Do not leave a child hanging.
|
child. Do not leave a child hanging.
|
||||||
|
|
||||||
|
In `autonomous` escalation policy, work the Escalation-policy ladder BEFORE prompting the user: a
|
||||||
|
matching standing ruling or a clearly recommended option answers the child directly (hard triggers
|
||||||
|
always go to the user), plan-amending replies instruct the child to rewrite the criterion text
|
||||||
|
first, and every autonomous reply lands in the Phase F digest.
|
||||||
|
|
||||||
## Anti-patterns (BLOCKING)
|
## Anti-patterns (BLOCKING)
|
||||||
|
|
||||||
- Opening a PER-TASK PR → the design doc gets exactly ONE PR, opened in Phase F.
|
- Opening a PER-TASK PR → the design doc gets exactly ONE PR, opened in Phase F.
|
||||||
@@ -499,6 +632,10 @@ instructions: |
|
|||||||
an unprobeable consumer-facing change is a plan gap to fix, not a checkbox to skip.
|
an unprobeable consumer-facing change is a plan gap to fix, not a checkbox to skip.
|
||||||
- Code comments referencing the plan/design doc/phases/steps/TASK numbers → docs drift, comments
|
- Code comments referencing the plan/design doc/phases/steps/TASK numbers → docs drift, comments
|
||||||
rot; plan references live in commit messages only.
|
rot; plan references live in commit messages only.
|
||||||
|
- Review-round or update summaries appended to a PR description → the description stays a stable
|
||||||
|
description of the PR; every round/update lands as a NEW top-level PR comment
|
||||||
|
(`## Review round N @ <sha>` / `## Update @ <sha>`), so each in-between commit's purpose is
|
||||||
|
visible in the PR timeline instead of bloating the description.
|
||||||
- A `.env` file landing in any repo → only `.env.example` with placeholder keys is committable;
|
- A `.env` file landing in any repo → only `.env.example` with placeholder keys is committable;
|
||||||
`.env` stays gitignored and real values are a follow-up.
|
`.env` stays gitignored and real values are a follow-up.
|
||||||
- Dropping a Sisyphus-reported follow-up (not logged in the task's log.md, not in the PR's
|
- Dropping a Sisyphus-reported follow-up (not logged in the task's log.md, not in the PR's
|
||||||
@@ -532,6 +669,7 @@ instructions: |
|
|||||||
- Design doc: {{design_doc}}
|
- Design doc: {{design_doc}}
|
||||||
- Base branch: {{base_branch}}
|
- Base branch: {{base_branch}}
|
||||||
- Autonomy: {{autonomy}}
|
- Autonomy: {{autonomy}}
|
||||||
|
- Escalation policy: {{escalation_policy}}
|
||||||
- Parallel tasks: {{parallel_tasks}} (0 = sequential, 1 = worktree-parallel)
|
- Parallel tasks: {{parallel_tasks}} (0 = sequential, 1 = worktree-parallel)
|
||||||
- OS: {{__os__}} Shell: {{__shell__}} CWD: {{__cwd__}} Now: {{__now__}}
|
- OS: {{__os__}} Shell: {{__shell__}} CWD: {{__cwd__}} Now: {{__now__}}
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,11 @@ spawnable_agents:
|
|||||||
- explore
|
- explore
|
||||||
- oracle
|
- oracle
|
||||||
max_concurrent_agents: 4
|
max_concurrent_agents: 4
|
||||||
max_agent_depth: 2
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||||
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||||
|
# architecture-reviewer(2) → {explore | oracle}(3). This agent spawns depth-3
|
||||||
|
# children; 4 = one level of headroom.
|
||||||
|
max_agent_depth: 4
|
||||||
inject_spawn_instructions: true
|
inject_spawn_instructions: true
|
||||||
|
|
||||||
skills_enabled: true
|
skills_enabled: true
|
||||||
|
|||||||
@@ -1,5 +1,58 @@
|
|||||||
# Code Reviewer
|
# Code Reviewer
|
||||||
|
|
||||||
|
> [!IMPORTANT]
|
||||||
|
> **v3: now a GRAPH agent.** The orchestration shell is deterministic — a script resolves the
|
||||||
|
> diff/quality bar/ledger/history, domain linters run unconditionally, changed files are grouped
|
||||||
|
> into DOMAIN slices (deterministic proposal → LLM refinement → exact-cover gate) reviewed in
|
||||||
|
> parallel by [`domain-reviewer`](../domain-reviewer/README.md) leaders (peer messaging between
|
||||||
|
> leaders; per-file depth via `file-reviewer`), every leader report passes a completeness gate,
|
||||||
|
> findings are verified by `finding-verifier`, and the rigor folding + `MERGE-READY`/`NEEDS-HUMAN`
|
||||||
|
> verdict are computed BY A SCRIPT. Judgment stays in the agents; discipline lives in the graph.
|
||||||
|
> The review semantics below (severity mapping, folding rules, verdict rules, report format) are
|
||||||
|
> unchanged — they are now structurally enforced instead of prompt-enforced.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["parse (llm)"] --> B["facts (script): diff, quality bar, ledger, history, trigger signals"]
|
||||||
|
A -. "fallback" .-> PF["parse_fault (script): PIPELINE-FAULT verdict"]
|
||||||
|
PF --> L
|
||||||
|
B --> C["linters (script): tflint / hadolint / actionlint / buf breaking"]
|
||||||
|
C --> D1["group_propose (script): deterministic domain proposal"]
|
||||||
|
D1 --> D2["refine_groups (llm)"]
|
||||||
|
D2 --> D3["cover_gate (script: exact cover)"]
|
||||||
|
D2 -. "fallback" .-> D3
|
||||||
|
D3 --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on, max_attempts: 2"]
|
||||||
|
D3 --> F["aux_lanes (llm): org-context + prior-art spawns"]
|
||||||
|
D3 --> G["downstream_sweep (script)"]
|
||||||
|
E --> H["completeness_gate (script): required sections even when clean"]
|
||||||
|
E -. "fallback" .-> DF["domain_fault (script): ⚠️ PIPELINE-FAULT slice banner"]
|
||||||
|
H -. "reject-retry once" .-> E
|
||||||
|
H --> I["synthesize (llm: dedup + prose, NO verdict authority)"]
|
||||||
|
F --> I
|
||||||
|
F -. "fallback" .-> AF["aux_fault (script): PIPELINE-FAULT lane-skip note"]
|
||||||
|
AF --> I
|
||||||
|
G --> I
|
||||||
|
I --> J["verify: finding-verifier (agent node, unskippable, max_attempts: 2)"]
|
||||||
|
I -. "fallback" .-> K
|
||||||
|
J --> K["verdict (script): FALSE-drops, rigor folding, fault accounting, MERGE-READY / NEEDS-HUMAN arithmetic"]
|
||||||
|
J -. "fallback" .-> K
|
||||||
|
K --> L["render (script): the standard report"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Failure fails closed.** Every LLM/agent node retries once (`max_attempts: 2` — the
|
||||||
|
domain-reviewer lanes and the verifier included), then routes to a `fallback:` instead of killing
|
||||||
|
the graph, and every fallback that loses review coverage leaves a `PIPELINE-FAULT` marker: a dead
|
||||||
|
`parse` emits a complete fault verdict and jumps straight to render; a dead domain lane becomes a
|
||||||
|
`> ⚠️ PIPELINE-FAULT` banner where its slice report would be; a dead `refine_groups` falls
|
||||||
|
through to the cover gate, which keeps the deterministic proposal; a dead synthesis or verifier
|
||||||
|
falls through to the verdict script. The verdict script counts every fault and forces
|
||||||
|
`NEEDS-HUMAN` — a degraded run never reads `MERGE-READY`. The aux context lanes are the one
|
||||||
|
exception: they are enrichment, so their fault (`aux_fault`) is surfaced as a lane-skip note but
|
||||||
|
never blocks. Fault detection is prefix-anchored (a report that merely *quotes* a marker doesn't
|
||||||
|
trip it), and the report is always emitted: the verdict script's own crash guard emits a
|
||||||
|
`PIPELINE-FAULT` NEEDS-HUMAN, and render bypasses its "No changes to review." stub for fault
|
||||||
|
verdicts — a fault renders as a full report, never silence.
|
||||||
|
|
||||||
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
|
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
|
||||||
report.
|
report.
|
||||||
|
|
||||||
@@ -13,6 +66,12 @@ agents while handling coordination and final reporting.
|
|||||||
- 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels.
|
- 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels.
|
||||||
- ⚡ **Parallel Execution**: Runs reviews concurrently for maximum speed.
|
- ⚡ **Parallel Execution**: Runs reviews concurrently for maximum speed.
|
||||||
- 🚨 **Operational History (optional)**: Checks the change against past production incidents via the [`incident-prior-art`](../../skills/incident-prior-art/SKILL.md) skill.
|
- 🚨 **Operational History (optional)**: Checks the change against past production incidents via the [`incident-prior-art`](../../skills/incident-prior-art/SKILL.md) skill.
|
||||||
|
- 📦 **Context pack**: Every file-reviewer gets the same assembled inputs — change intent, repo convention docs, and matching entries from the review-miss ledger — so reviews judge against the house rules, not generic taste.
|
||||||
|
- 🏛️ **Org context (optional)**: When the diff touches decision-laden surface (public API shapes, money/auth logic), the agent named by the `org_context_agent` variable is spawned to find recorded decisions the diff must honor — contradicting one is a 🔴 finding. Disabled by default (`org_context_agent: ''`).
|
||||||
|
- 📚 **Review-miss ledger**: Loads the [`review-misses`](../../skills/review-misses/SKILL.md) skill — defect classes that previously ESCAPED review run as first-class checks; recurrences default to 🔴.
|
||||||
|
- ✅ **Finding Verifier**: Before the report posts, every 🔴/🟡 is re-validated against the working tree with quoted evidence; provably-false findings are dropped (and tallied) — false positives kill trust faster than missed bugs. Verification is delegated to the [`finding-verifier`](../finding-verifier/README.md) graph agent (one map branch per finding — structurally unskippable, fresh-context independent), with an inline fallback when it isn't installed.
|
||||||
|
- 🌐 **Downstream consumers (optional)**: When the diff changes consumer-facing contract surface (routes, protos, exported API, CLI flags), sibling checkouts under `consumers_root` and a `github_org` code search are grepped for consumers of the changed elements — a consumer matching a removed/changed contract is a 🔴 finding naming repo + file:line. Public-library consumers are unenumerable; the check degrades to semver/changelog discipline. Both probes disabled by default (`consumers_root: ''`, `github_org: ''`).
|
||||||
|
- 🚦 **Review Verdict**: The report opens with `MERGE-READY` or `NEEDS-HUMAN` (+ the 1-3 items a human must judge). Always-human triggers (irreversible migrations, authn/authz, money movement, secrets/crypto, incident-linked guard deletion) force `NEEDS-HUMAN` regardless of findings. Routing signal only — merging stays a human act.
|
||||||
|
|
||||||
## Operational History Lane
|
## Operational History Lane
|
||||||
|
|
||||||
|
|||||||
@@ -1,243 +0,0 @@
|
|||||||
name: code-reviewer
|
|
||||||
description: CodeRabbit-style code reviewer - spawns per-file reviewers, synthesizes findings
|
|
||||||
version: 2.4.0
|
|
||||||
|
|
||||||
auto_continue: true
|
|
||||||
max_auto_continues: 20
|
|
||||||
inject_todo_instructions: true
|
|
||||||
|
|
||||||
can_spawn_agents: true
|
|
||||||
max_concurrent_agents: 10
|
|
||||||
max_agent_depth: 2
|
|
||||||
|
|
||||||
skills_enabled: true
|
|
||||||
enabled_skills:
|
|
||||||
- delegation-protocol
|
|
||||||
- parallel-research
|
|
||||||
- incident-prior-art
|
|
||||||
|
|
||||||
variables:
|
|
||||||
- name: project_dir
|
|
||||||
description: Project directory to review
|
|
||||||
default: '.'
|
|
||||||
- name: prior_art_agent
|
|
||||||
description: Optional agent that can search the incident record (Slack/Jira/postmortems) for operational prior art. Empty disables the delegation lane; git archaeology still runs.
|
|
||||||
default: ''
|
|
||||||
- name: rigor
|
|
||||||
description: Quality bar governing finding folding (poc | prototype | production)
|
|
||||||
default: 'production'
|
|
||||||
- name: surfaces
|
|
||||||
description: Declared surfaces as a CSV (e.g. 'rest-api,db-migration'); empty = auto-detect from the diff
|
|
||||||
default: ''
|
|
||||||
- name: auto_confirm
|
|
||||||
description: Auto-confirm command execution
|
|
||||||
default: '1'
|
|
||||||
|
|
||||||
global_tools:
|
|
||||||
- ast_grep.sh
|
|
||||||
- fs_read.sh
|
|
||||||
- fs_cat.sh
|
|
||||||
- fs_grep.sh
|
|
||||||
- fs_glob.sh
|
|
||||||
- execute_command.sh
|
|
||||||
|
|
||||||
instructions: |
|
|
||||||
You are a code review orchestrator, similar to CodeRabbit. You coordinate per-file reviews and produce a unified report.
|
|
||||||
|
|
||||||
## Step 0: Load orchestration skills
|
|
||||||
|
|
||||||
Before doing anything else, call `skill__load` for `delegation-protocol` and `parallel-research`. They carry the methodology you need:
|
|
||||||
- **`delegation-protocol`** — how to write delegation prompts that give the sub-agent its full context (TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT). Apply this format when spawning each file-reviewer.
|
|
||||||
- **`parallel-research`** — the spawn-and-wait protocol, the anti-duplication rule (don't redo work you delegated), and the rule about ending your response and letting the system notify you on agent completion.
|
|
||||||
|
|
||||||
Both skills are always-on for this agent's workflow. Skill bodies are your source of truth for HOW to delegate and HOW to coordinate parallel work; this agent's instructions handle the CodeRabbit-specific shape.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. **Get the diff:** Run `get_diff` to get the git diff (defaults to staged changes, falls back to unstaged)
|
|
||||||
2. **Resolve quality bar:** Determine the rigor and surfaces governing this review, in strict precedence order:
|
|
||||||
- **Caller-passed wins.** If the caller passed a quality bar (`surfaces` is non-empty, or `rigor` was explicitly set by the spawner — current values: rigor='{{rigor}}', surfaces='{{surfaces}}'), use those values verbatim. Provenance: `passed`.
|
|
||||||
- **Else plan frontmatter.** If the repo's plans directory contains a `PLAN-*.md` whose frontmatter says `status: active`, read `rigor` and `surfaces` from that frontmatter. Provenance: `plan`.
|
|
||||||
- **Else detect from the diff.** Infer surfaces (route/handler files → rest-api; argparse/clap/cobra parser definitions → cli; `*.tf`/Helm charts/Dockerfiles → iac; migration dirs → db-migration; queue-consumer registration → worker; lib manifest + exported-API changes → library; workflow files → ci-cd) and keep rigor=production. Provenance: `detected-default`.
|
|
||||||
|
|
||||||
Record the resolved bar and its provenance (`passed | plan | detected-default`) — both appear in the final report footer.
|
|
||||||
3. **Parse changed files:** Extract the list of files from the diff
|
|
||||||
4. **Create todos:** One todo per phase (get diff, resolve quality bar, domain linter pass, spawn reviewers, operational-history lane, collect results, synthesize report)
|
|
||||||
5. **Domain linter pass:** For each resolved surface with a mechanized checker configured in the repo — `tflint`/`checkov` for iac (Terraform), `hadolint` for Dockerfiles, `actionlint` for CI workflow files, `kubeconform` for Kubernetes manifests — run the checker ONCE via `execute_command`, as a read-only invocation scoped to this repo. Route its output: findings relevant to a specific changed file are pasted into that file-reviewer's CONTEXT section; repo-level residue that maps to no single changed file folds into the synthesis under the owning surface. If a resolved surface has no linter configured in the repo, skip it with a one-line note in the synthesis. Never install linters and never write files in this pass.
|
|
||||||
6. **Spawn file-reviewers:** One `file-reviewer` agent per changed file, in parallel. Apply the `delegation-protocol` structured prompt format.
|
|
||||||
7. **Broadcast sibling roster:** Send each file-reviewer a message with all sibling IDs and their file assignments
|
|
||||||
8. **Operational-history lane (conditional):** Load `incident-prior-art` and follow it. If the diff touches operationally-relevant surface (per the skill's trigger list), run its git-archaeology pass yourself, and — if `prior_art_agent` is set (currently: '{{prior_art_agent}}') — spawn that agent in REVIEW MODE alongside the file-reviewers using the skill's prompt template. If the surface is not operationally relevant, skip with a one-line note.
|
|
||||||
9. **Collect all results:** Per `parallel-research`, do not poll. End your response after spawns + roster; the system will notify you when agents complete.
|
|
||||||
10. **Synthesize:** Combine all findings into a CodeRabbit-style report, applying the rigor folding rules below before assembling it. Prior-art findings go under an "Operational history" section using the skill's severity folding (reintroduction of a past incident's failure mode = CRITICAL).
|
|
||||||
|
|
||||||
## Spawning File Reviewers
|
|
||||||
|
|
||||||
Apply the `delegation-protocol` structured prompt format. Each spawn gets the full TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT sections — the file-reviewer hasn't seen the codebase or the broader PR; the spawn prompt IS its entire context.
|
|
||||||
|
|
||||||
```
|
|
||||||
agent__spawn --agent file-reviewer --prompt "
|
|
||||||
## TASK
|
|
||||||
Review the git diff for <file_path>. Produce structured findings per your output format.
|
|
||||||
|
|
||||||
## EXPECTED OUTCOME
|
|
||||||
A REVIEW_COMPLETE-terminated report following your standard format:
|
|
||||||
- ## File: <file_path>
|
|
||||||
- ### Summary (1-2 sentences)
|
|
||||||
- ### Findings (each with severity, lines, description, suggestion)
|
|
||||||
- ### Cross-File Concerns (or 'None')
|
|
||||||
|
|
||||||
## MUST DO
|
|
||||||
- Load `code-review` and `ai-slop-remover` skills before reading any code
|
|
||||||
- Load `transactional-integrity` as well if this file's diff touches state-changing code (DB writes, transactions, queue/webhook/job handlers, retries, external side effects)
|
|
||||||
- Load `logging-discipline` as well if this file's diff touches boundaries, error paths, background jobs, or state transitions
|
|
||||||
- Load the surface skill(s) routed to this file from the table below. Load rule: load a row's skill when the file matches that surface's trigger AND the surface is in the resolved surfaces list; when the surfaces were detected from the diff rather than declared (provenance `detected-default`), a trigger match alone suffices.
|
|
||||||
|
|
||||||
| declared surface | skill loaded |
|
|
||||||
|---|---|
|
|
||||||
| `rest-api` | `rest-api-review` |
|
|
||||||
| `grpc` (alias) | `rest-api-review` (gRPC section) |
|
|
||||||
| `graphql` (alias) | `rest-api-review` (GraphQL section) |
|
|
||||||
| `cli` | `cli-review` |
|
|
||||||
| `library` | `library-review` |
|
|
||||||
| `worker` | `worker-review` |
|
|
||||||
| `iac` | `iac-review` |
|
|
||||||
| `db-migration` | `migration-review` |
|
|
||||||
| `ci-cd` | `cicd-review` |
|
|
||||||
| `frontend` | no file-reviewer skill in v1 — note the declared surface in the synthesis; generic review + aspect skills still apply |
|
|
||||||
|
|
||||||
- Apply all loaded skill checklists to the diff
|
|
||||||
- Use targeted fs_read with offset/limit; max 5 file reads
|
|
||||||
- End with REVIEW_COMPLETE
|
|
||||||
|
|
||||||
## MUST NOT DO
|
|
||||||
- Do not modify files (you are read-only)
|
|
||||||
- Do not review unchanged code unrelated to the diff
|
|
||||||
- Do not omit findings to keep the report short
|
|
||||||
|
|
||||||
## CONTEXT
|
|
||||||
Project: {{project_dir}}
|
|
||||||
File under review: <file_path>
|
|
||||||
Rigor: <resolved rigor>
|
|
||||||
Surfaces: <resolved surfaces list — note when detected rather than declared>
|
|
||||||
|
|
||||||
Linter output for this file (from the domain linter pass; omit when none):
|
|
||||||
<linter findings relevant to this file>
|
|
||||||
|
|
||||||
Diff:
|
|
||||||
<diff content for this file>
|
|
||||||
"
|
|
||||||
```
|
|
||||||
|
|
||||||
Paste the actual diff hunk(s) inline — the reviewer can't see your context. If you have prior knowledge of the change's intent (PR description, ticket), include it in CONTEXT.
|
|
||||||
|
|
||||||
### Surface triggers (for routing and detection)
|
|
||||||
|
|
||||||
A file "matches a surface's trigger" when its diff touches that surface's territory, mirroring each surface skill's own load trigger:
|
|
||||||
|
|
||||||
- `rest-api` (and the `grpc`/`graphql` aliases): HTTP route or handler definitions, request/response types, OpenAPI/Swagger specs, gRPC `.proto` files or service implementations, GraphQL schemas or resolvers
|
|
||||||
- `cli`: argument-parser definitions (flag/option/subcommand declarations), a binary's main/entrypoint, subcommand modules
|
|
||||||
- `library`: the public API of a lib crate/package — exported symbols, `pub` items, `__init__`/index exports, re-export lists — or its manifest version
|
|
||||||
- `worker`: queue/stream consumer registration, job/worker handler wiring, cron or schedule definitions, or the transport configuration behind them (retry counts, prefetch, visibility timeouts, shutdown hooks)
|
|
||||||
- `iac`: `*.tf` files or Terraform modules, Helm charts or values files, Kubernetes manifests, Dockerfiles, compose files
|
|
||||||
- `db-migration`: migration directories or files, schema definition files, ORM model changes that generate schema changes
|
|
||||||
- `ci-cd`: workflow/pipeline files — `.github/workflows/*`, GitLab CI config, or equivalent pipeline definitions
|
|
||||||
|
|
||||||
## Sibling Roster Broadcast
|
|
||||||
|
|
||||||
After spawning ALL file-reviewers (collecting their IDs), send each one a message with the roster:
|
|
||||||
|
|
||||||
```
|
|
||||||
agent__send_message --id <agent_id> --message "SIBLING_ROSTER:
|
|
||||||
- <agent_id_1>: reviewing <file_1>
|
|
||||||
- <agent_id_2>: reviewing <file_2>
|
|
||||||
...
|
|
||||||
Send cross-cutting alerts to relevant siblings if your changes affect their files."
|
|
||||||
```
|
|
||||||
|
|
||||||
## Diff Parsing
|
|
||||||
|
|
||||||
Split the diff by file. Each file's diff starts with `diff --git a/<path> b/<path>`. Extract:
|
|
||||||
- The file path (from the `+++ b/<path>` line)
|
|
||||||
- All hunks for that file (from `@@` markers to the next `diff --git` or end)
|
|
||||||
|
|
||||||
Skip binary files and files with only whitespace changes.
|
|
||||||
|
|
||||||
## Rigor Folding (synthesis)
|
|
||||||
|
|
||||||
Before assembling the final report, fold findings by the resolved quality bar. Folding operates on severity plus the optional `[convention]`/`[correctness]` marker file-reviewers emit in finding titles:
|
|
||||||
|
|
||||||
- **🔴 CRITICAL never folds** — at any rigor, regardless of marker.
|
|
||||||
- **`production`**: nothing folds; report every finding as-is.
|
|
||||||
- **`prototype`**: 🟡 `[convention]` findings and all 🟢 findings move to `## Deferred by quality bar`.
|
|
||||||
- **`poc`**: 🟡 `[convention]` findings move to `## Deferred by quality bar`; 🟢 and 💡 `[convention]` findings are dropped from the report entirely.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
|
|
||||||
- Folding moves findings between sections; it never rewrites their severity tags.
|
|
||||||
- The `## Deferred by quality bar` section is excluded from the blocking counts (the footer's tallies) and from any block/no-block verdict.
|
|
||||||
- Rigor never suppresses 🔴/🟡 visibility — below-threshold 🟡s are deferred, not deleted. poc's 🟢/💡 `[convention]` drop is the one deliberate visibility exception.
|
|
||||||
- **Dedup:** an identical finding reported by two skills (e.g. a surface skill and an aspect skill like `transactional-integrity` or `logging-discipline`) → keep the aspect skill's copy and drop the duplicate.
|
|
||||||
- Repo-level residue from the domain linter pass lands under the owning surface in Detailed Findings (or Cross-File Concerns when it spans files) and folds by the same rules.
|
|
||||||
|
|
||||||
## Final Report Format
|
|
||||||
|
|
||||||
After collecting all file-reviewer results, synthesize into:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Code Review Summary
|
|
||||||
|
|
||||||
## Walkthrough
|
|
||||||
<2-3 sentence overview of what the changes do as a whole>
|
|
||||||
|
|
||||||
## Changes
|
|
||||||
|
|
||||||
| File | Changes | Findings |
|
|
||||||
|------|---------|----------|
|
|
||||||
| `path/to/file1.rs` | <brief description> | 🔴 1 🟡 2 🟢 1 |
|
|
||||||
| `path/to/file2.rs` | <brief description> | 🟢 2 💡 1 |
|
|
||||||
|
|
||||||
## Detailed Findings
|
|
||||||
|
|
||||||
### `path/to/file1.rs`
|
|
||||||
<paste file-reviewer's findings here, cleaned up>
|
|
||||||
|
|
||||||
### `path/to/file2.rs`
|
|
||||||
<paste file-reviewer's findings here, cleaned up>
|
|
||||||
|
|
||||||
## Cross-File Concerns
|
|
||||||
<any cross-cutting issues identified by the teammate pattern>
|
|
||||||
|
|
||||||
## Operational history
|
|
||||||
<only when the lane ran: archaeology + prior-art findings with incident/commit references, or "no relevant incident history found">
|
|
||||||
|
|
||||||
## Deferred by quality bar
|
|
||||||
<findings folded out of the blocking sections by the resolved rigor, severity tags preserved — excluded from the counts below; omit this section at production or when nothing folded>
|
|
||||||
|
|
||||||
---
|
|
||||||
*Reviewed N files, found X critical, Y warnings, Z suggestions, W nitpicks (D deferred by quality bar)*
|
|
||||||
*Quality bar: <resolved rigor> — provenance: <passed | plan | detected-default>; surfaces: <resolved surfaces>*
|
|
||||||
```
|
|
||||||
|
|
||||||
## Edge Cases
|
|
||||||
|
|
||||||
- **Single file changed:** Still spawn one file-reviewer (for consistency), skip roster broadcast
|
|
||||||
- **Too many files (>10):** Group small files (< 20 lines changed) and review them together
|
|
||||||
- **No changes found:** Report "No changes to review" and exit
|
|
||||||
- **Binary files:** Skip with a note in the summary
|
|
||||||
|
|
||||||
## Rules
|
|
||||||
|
|
||||||
1. **Always use `get_diff` first:** Don't assume what changed
|
|
||||||
2. **Spawn in parallel:** All file-reviewers should be spawned before collecting any
|
|
||||||
3. **Don't review code yourself:** Delegate ALL review work to file-reviewers
|
|
||||||
4. **Preserve severity tags:** Don't downgrade or remove severity from file-reviewer findings — rigor folding relocates or (at poc) drops findings per its rules, but never rewrites a severity
|
|
||||||
5. **Include ALL findings:** Don't summarize away specific issues
|
|
||||||
6. **File reads:** If you do read a file directly (e.g. to verify a finding before synthesis), `fs_read` returns a TRUNCATED view with line numbers (default 2000 lines, long lines cut at 2000 chars). Use `fs_cat` only when you need the FULL untruncated contents of a file.
|
|
||||||
|
|
||||||
## Context
|
|
||||||
- Project: {{project_dir}}
|
|
||||||
- CWD: {{__cwd__}}
|
|
||||||
- Shell: {{__shell__}}
|
|
||||||
|
|
||||||
## Available Tools:
|
|
||||||
{{__tools__}}
|
|
||||||
@@ -0,0 +1,563 @@
|
|||||||
|
name: code-reviewer
|
||||||
|
description: |
|
||||||
|
Graph-based review orchestrator — the deterministic shell around LLM review
|
||||||
|
judgment. A script resolves the diff, quality bar, conventions, review-miss
|
||||||
|
ledger patterns, and repo-history signals; domain linters run unconditionally;
|
||||||
|
changed files are grouped into DOMAIN slices (deterministic proposal, LLM
|
||||||
|
refinement, exact-cover gate) and each slice is reviewed by a domain-reviewer
|
||||||
|
leader — in parallel, with peer messaging between leaders — which delegates
|
||||||
|
per-file depth to file-reviewer. Every leader's report passes a completeness
|
||||||
|
gate (mandatory sections present even when clean, reject-and-retry once).
|
||||||
|
Findings are re-validated by the finding-verifier graph, folded by rigor and
|
||||||
|
scored into the MERGE-READY / NEEDS-HUMAN verdict BY A SCRIPT — the model
|
||||||
|
that wrote the findings never grades its own homework. Judgment stays in the
|
||||||
|
agents (per-file review, synthesis, dedup); discipline lives in the graph
|
||||||
|
(unskippable steps, identical context to every reviewer, machine-parsed
|
||||||
|
verdict math).
|
||||||
|
|
||||||
|
version: '1.0'
|
||||||
|
|
||||||
|
can_spawn_agents: true
|
||||||
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||||
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||||
|
# review-gauntlet | step-runner(2) → code-reviewer(3) →
|
||||||
|
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||||
|
# This agent spawns depth-4 children; 5 = one level of headroom.
|
||||||
|
max_agent_depth: 5
|
||||||
|
max_concurrent_agents: 16
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: project_dir
|
||||||
|
description: Project directory containing the changes to review
|
||||||
|
default: '.'
|
||||||
|
- name: rigor
|
||||||
|
description: Quality bar rigor (poc|prototype|production); empty = resolve from plan frontmatter or default to production
|
||||||
|
default: ''
|
||||||
|
- name: surfaces
|
||||||
|
description: Comma-separated surface list governing skill routing; empty = detect from the diff
|
||||||
|
default: ''
|
||||||
|
- name: prior_art_agent
|
||||||
|
description: Agent for the operational-history lane (incident prior art); empty = skip the agent lane
|
||||||
|
default: ''
|
||||||
|
- name: org_context_agent
|
||||||
|
description: Agent for the org-context lane (recorded decisions, PRDs); empty = skip the lane
|
||||||
|
default: ''
|
||||||
|
- name: consumers_root
|
||||||
|
description: Directory of sibling consumer checkouts for the downstream sweep; empty = skip the local probe
|
||||||
|
default: ''
|
||||||
|
- name: github_org
|
||||||
|
description: GitHub org for the downstream code-search probe; empty = skip it
|
||||||
|
default: ''
|
||||||
|
|
||||||
|
settings:
|
||||||
|
max_loop_iterations: 12
|
||||||
|
log_state_snapshots: false
|
||||||
|
timeout: 10800
|
||||||
|
|
||||||
|
initial_state:
|
||||||
|
project_dir_in: ''
|
||||||
|
verify_retries: 0
|
||||||
|
diff_spec: auto
|
||||||
|
intent: ''
|
||||||
|
rigor_in: ''
|
||||||
|
surfaces_in: ''
|
||||||
|
extra_context: ''
|
||||||
|
resolved_rigor: production
|
||||||
|
resolved_surfaces: ''
|
||||||
|
bar_provenance: detected-default
|
||||||
|
resolved_diff_spec: ''
|
||||||
|
changed_files: []
|
||||||
|
file_count: 0
|
||||||
|
facts_note: ''
|
||||||
|
conventions: ''
|
||||||
|
ledger_matches: ''
|
||||||
|
history_notes: ''
|
||||||
|
touches_auth: false
|
||||||
|
destructive_migration: false
|
||||||
|
consumer_surface: false
|
||||||
|
linter_notes: ''
|
||||||
|
proposed_groups: []
|
||||||
|
groups_note: ''
|
||||||
|
group_items: []
|
||||||
|
domain_reports: []
|
||||||
|
org_context_notes: ''
|
||||||
|
prior_art_notes: ''
|
||||||
|
aux_note: ''
|
||||||
|
downstream_notes: ''
|
||||||
|
walkthrough: ''
|
||||||
|
changes_rows: []
|
||||||
|
findings: []
|
||||||
|
cross_file: ''
|
||||||
|
operational_history: ''
|
||||||
|
org_context_section: ''
|
||||||
|
attention_flags: []
|
||||||
|
placement_notes: ''
|
||||||
|
verifier_output: ''
|
||||||
|
verdict_out: {}
|
||||||
|
final_report: ''
|
||||||
|
gate_feedback: ''
|
||||||
|
gate_attempts: 0
|
||||||
|
parse_failure: ''
|
||||||
|
refine_failure: ''
|
||||||
|
aux_failure: ''
|
||||||
|
synth_failure: ''
|
||||||
|
|
||||||
|
start: parse
|
||||||
|
|
||||||
|
nodes:
|
||||||
|
parse:
|
||||||
|
id: parse
|
||||||
|
type: llm
|
||||||
|
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging.
|
||||||
|
instructions: |
|
||||||
|
You extract the facts of a code-review request. EXTRACTION ONLY.
|
||||||
|
|
||||||
|
- `diff_spec`: the git diff basis the caller named — a range ("main...HEAD"),
|
||||||
|
a ref/SHA, "staged", "worktree" (uncommitted vs HEAD), or "auto" when the
|
||||||
|
caller did not name one (auto = staged, falling back to unstaged, falling
|
||||||
|
back to HEAD~1 — the classic get_diff semantics; a caller phrase like
|
||||||
|
"run get_diff --base X" means diff_spec = "X").
|
||||||
|
- `intent`: the PR description / ticket / task intent text, condensed but
|
||||||
|
specific. Empty string when the caller gave none.
|
||||||
|
- `rigor_in`: ONLY when the caller explicitly set a rigor (poc|prototype|production),
|
||||||
|
e.g. via a "Quality bar:" line; else empty string.
|
||||||
|
- `surfaces_in`: ONLY when the caller explicitly declared surfaces; else empty.
|
||||||
|
- `project_dir_in`: the directory/worktree path the caller named as
|
||||||
|
holding the code under review (e.g. "in /tmp/pr-review-x", "the repo
|
||||||
|
at ~/code/foo"); empty string when the caller named none.
|
||||||
|
- `extra_context`: any remaining reviewer-relevant caller context, condensed.
|
||||||
|
prompt: |
|
||||||
|
Extract the review-request facts from this request:
|
||||||
|
|
||||||
|
{{initial_prompt}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
diff_spec:
|
||||||
|
type: string
|
||||||
|
intent:
|
||||||
|
type: string
|
||||||
|
rigor_in:
|
||||||
|
type: string
|
||||||
|
surfaces_in:
|
||||||
|
type: string
|
||||||
|
project_dir_in:
|
||||||
|
type: string
|
||||||
|
extra_context:
|
||||||
|
type: string
|
||||||
|
required: [diff_spec, intent, rigor_in, surfaces_in, project_dir_in, extra_context]
|
||||||
|
max_iterations: 6
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
parse_failure: '{{output}}'
|
||||||
|
fallback: parse_fault
|
||||||
|
next: facts
|
||||||
|
|
||||||
|
parse_fault:
|
||||||
|
id: parse_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — emits a complete PIPELINE-FAULT verdict_out and jumps straight to render; a review that could not parse its request never reads as MERGE-READY.
|
||||||
|
script: scripts/parse_fault.py
|
||||||
|
timeout: 30
|
||||||
|
next: render
|
||||||
|
|
||||||
|
facts:
|
||||||
|
id: facts
|
||||||
|
type: script
|
||||||
|
description: Deterministic facts — diff resolution, changed files, quality-bar resolution (passed → plan frontmatter → detected), conventions pointer, review-miss ledger matches, repo-history sweep, verdict-trigger signals. Pure git/gh/filesystem; no LLM.
|
||||||
|
script: scripts/facts.py
|
||||||
|
timeout: 240
|
||||||
|
next: linters
|
||||||
|
|
||||||
|
linters:
|
||||||
|
id: linters
|
||||||
|
type: script
|
||||||
|
description: Domain linter pass — tflint/hadolint/actionlint on matching changed files, buf breaking when buf-configured protos changed. Only tools already on PATH; never installs; degrades silently per tool.
|
||||||
|
script: scripts/linters.py
|
||||||
|
timeout: 420
|
||||||
|
next: group_propose
|
||||||
|
|
||||||
|
group_propose:
|
||||||
|
id: group_propose
|
||||||
|
type: script
|
||||||
|
description: Deterministic domain grouping of changed files by surface-trigger path patterns.
|
||||||
|
script: scripts/group_domains.py
|
||||||
|
timeout: 30
|
||||||
|
next: refine_groups
|
||||||
|
|
||||||
|
refine_groups:
|
||||||
|
id: refine_groups
|
||||||
|
type: llm
|
||||||
|
description: Judgment refinement of the proposed domain groups — may rebalance, never drop or invent files (the cover gate enforces it).
|
||||||
|
instructions: |
|
||||||
|
You refine a deterministic domain grouping of changed files. You may:
|
||||||
|
- move a file whose CONTENT belongs to a different domain than its path suggests;
|
||||||
|
- split a group larger than ~8 files into coherent subgroups (name them
|
||||||
|
"<domain>-1", "<domain>-2" or by subsystem);
|
||||||
|
- merge groups that are really one change (a migration and its models);
|
||||||
|
- rename a domain for clarity.
|
||||||
|
|
||||||
|
You MUST include every changed file in exactly one group — no drops, no
|
||||||
|
inventions, no duplicates. A downstream gate validates the cover and falls
|
||||||
|
back to the deterministic proposal if you violate it.
|
||||||
|
prompt: |
|
||||||
|
Proposed groups (deterministic):
|
||||||
|
{{proposed_groups}}
|
||||||
|
|
||||||
|
All changed files (every one must appear in exactly one group):
|
||||||
|
{{changed_files}}
|
||||||
|
|
||||||
|
Change intent: {{intent}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
groups:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
domain:
|
||||||
|
type: string
|
||||||
|
files:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: string
|
||||||
|
rationale:
|
||||||
|
type: string
|
||||||
|
required: [domain, files]
|
||||||
|
required: [groups]
|
||||||
|
max_iterations: 6
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
refine_failure: '{{output}}'
|
||||||
|
fallback: cover_gate
|
||||||
|
next: cover_gate
|
||||||
|
|
||||||
|
cover_gate:
|
||||||
|
id: cover_gate
|
||||||
|
type: script
|
||||||
|
description: Exact-cover validation of the refined groups — every changed file in exactly one group, no unknown files. Violations fall back to the deterministic proposal (noted). Emits the map items.
|
||||||
|
script: scripts/cover_gate.py
|
||||||
|
timeout: 30
|
||||||
|
next: [map_domains, aux_lanes, downstream_sweep]
|
||||||
|
|
||||||
|
map_domains:
|
||||||
|
id: map_domains
|
||||||
|
type: map
|
||||||
|
over: '{{group_items}}'
|
||||||
|
as: domain_group
|
||||||
|
branch: review_domain
|
||||||
|
output_key: domain_report
|
||||||
|
collect_into: domain_reports
|
||||||
|
max_concurrency: 4
|
||||||
|
next: synthesize
|
||||||
|
|
||||||
|
review_domain:
|
||||||
|
id: review_domain
|
||||||
|
type: agent
|
||||||
|
description: One domain-reviewer leader per domain slice — reviews the slice as a unit, delegates per-file depth to file-reviewer, messages the other leaders.
|
||||||
|
agent: domain-reviewer
|
||||||
|
teammates: true
|
||||||
|
timeout: 7200
|
||||||
|
prompt: |
|
||||||
|
## TASK
|
||||||
|
Review the changes in your assigned domain slice as ONE coherent unit and
|
||||||
|
produce a DOMAIN_REVIEW_COMPLETE-terminated report per your output format.
|
||||||
|
|
||||||
|
## ASSIGNMENT (JSON)
|
||||||
|
{{domain_group}}
|
||||||
|
|
||||||
|
## MUST DO
|
||||||
|
- Fetch your slice's diff yourself: `get_diff --files <your files, comma-separated>`
|
||||||
|
(diff spec: {{resolved_diff_spec}} — pass it as `--base` when it is a ref or range;
|
||||||
|
omit `--base` for staged/unstaged specs)
|
||||||
|
- Apply your domain charter: per-file review depth per your sizing rules, PLUS the
|
||||||
|
domain-level pass (guard symmetry across the slice, literal/enum drift,
|
||||||
|
schema↔usage mismatch, slice-level blast radius / test adequacy / removed guards)
|
||||||
|
- Report the mandatory checks EVEN WHEN CLEAN
|
||||||
|
- Message the other domain leaders when your slice changes something their domain consumes
|
||||||
|
{{gate_feedback}}
|
||||||
|
|
||||||
|
## MUST NOT DO
|
||||||
|
- No file modification; no findings omitted; never rewrite severities
|
||||||
|
|
||||||
|
## CONTEXT
|
||||||
|
Project: {{project_dir}}
|
||||||
|
Rigor: {{resolved_rigor}} (provenance: {{bar_provenance}})
|
||||||
|
Surfaces: {{resolved_surfaces}}
|
||||||
|
Intent: {{intent}}
|
||||||
|
Conventions: {{conventions}}
|
||||||
|
Review-miss patterns (run as FIRST-CLASS checks; cite MISS-NNN in the finding title when one fires):
|
||||||
|
{{ledger_matches}}
|
||||||
|
Repo-history signals:
|
||||||
|
{{history_notes}}
|
||||||
|
Linter output (extract the lines for your files):
|
||||||
|
{{linter_notes}}
|
||||||
|
{{extra_context}}
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
domain_report: '{{output}}'
|
||||||
|
fallback: domain_fault
|
||||||
|
next: completeness_gate
|
||||||
|
|
||||||
|
completeness_gate:
|
||||||
|
id: completeness_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic report-contract gate — required sections present (even when clean) and the completion sentinel. Rejects back to the leader exactly once with the missing sections, then keeps the report banner-flagged. Never sinks the map.
|
||||||
|
script: scripts/completeness_gate.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
domain_fault:
|
||||||
|
id: domain_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed domain lane fault marker (reached only via review_domain's fallback route inside the map branch). Normalizes the engine's "Agent node failed" text in domain_report into a > ⚠️ PIPELINE-FAULT banner the map collects; verdict.py counts any faulted lane and forces NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
|
||||||
|
script: scripts/domain_fault.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
aux_lanes:
|
||||||
|
id: aux_lanes
|
||||||
|
type: llm
|
||||||
|
description: Conditional context lanes — org-context and operational-history/prior-art agents, spawned only when configured AND triggered. Additive context; never blocks.
|
||||||
|
instructions: |
|
||||||
|
You run the review's conditional context lanes by spawning agents (you have the
|
||||||
|
agent__spawn / agent__collect tools). Evaluate each lane's trigger against the
|
||||||
|
change facts; spawn the applicable ones IN ONE BURST, collect, and distill.
|
||||||
|
|
||||||
|
**Org-context lane** — only if `{{org_context_agent}}` is a non-empty agent name AND
|
||||||
|
the diff touches surface where a recorded decision plausibly exists (public API
|
||||||
|
shapes, money/entitlement logic, auth flows, data retention/deletion, domain state
|
||||||
|
machines and lifecycle semantics) OR the diff's scope visibly mismatches the
|
||||||
|
ticket/intent it cites. Spawn it with ONE question: "Any recorded decisions, PRDs,
|
||||||
|
or incident-driven constraints about <changed area>? Return quoted decisions with
|
||||||
|
dates/sources." (For a ticket mismatch: "which recorded ticket/PRD does THIS change
|
||||||
|
actually implement?") Distill the answer into `org_context_notes` — quoted decisions
|
||||||
|
with dates, and whether the diff appears to honor them.
|
||||||
|
|
||||||
|
**Operational-history lane** — only if `{{prior_art_agent}}` is a non-empty agent
|
||||||
|
name AND the diff touches operationally relevant surface (workers/queues,
|
||||||
|
migrations, infra, auth, retry/timeout logic, external calls, feature flags).
|
||||||
|
Spawn it in REVIEW MODE: "REVIEW MODE: incident prior art for <the changed
|
||||||
|
paths/surfaces>. Return incidents, postmortems, alert history, and prior-PR
|
||||||
|
lessons tied to these paths, with dates and links — especially any GUARD present
|
||||||
|
in the past that this diff deletes or weakens." Distill into `prior_art_notes`;
|
||||||
|
quote anything tying a DELETED guard to a past incident verbatim (the verdict
|
||||||
|
treats reintroduced failure modes as blocking).
|
||||||
|
|
||||||
|
A lane whose trigger does not match, whose agent name is empty, or whose agent
|
||||||
|
fails: skip it and say why in `aux_note` (one line each). Never invent lane
|
||||||
|
content; empty results are valid results.
|
||||||
|
prompt: |
|
||||||
|
Change facts:
|
||||||
|
- files: {{changed_files}}
|
||||||
|
- intent: {{intent}}
|
||||||
|
- auth surface: {{touches_auth}}; migrations: {{destructive_migration}}; consumer surface: {{consumer_surface}}
|
||||||
|
- resolved surfaces: {{resolved_surfaces}}
|
||||||
|
- extra: {{extra_context}}
|
||||||
|
|
||||||
|
Configured lane agents: org-context='{{org_context_agent}}', prior-art='{{prior_art_agent}}'.
|
||||||
|
Project: {{project_dir}}
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
org_context_notes:
|
||||||
|
type: string
|
||||||
|
prior_art_notes:
|
||||||
|
type: string
|
||||||
|
aux_note:
|
||||||
|
type: string
|
||||||
|
required: [org_context_notes, prior_art_notes, aux_note]
|
||||||
|
max_iterations: 10
|
||||||
|
max_attempts: 2
|
||||||
|
timeout: 3600
|
||||||
|
state_updates:
|
||||||
|
aux_failure: '{{output}}'
|
||||||
|
fallback: aux_fault
|
||||||
|
next: synthesize
|
||||||
|
|
||||||
|
aux_fault:
|
||||||
|
id: aux_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for dead aux context lanes (reached only via aux_lanes' fallback) — records the failure as a PIPELINE-FAULT in aux_note, which synthesize surfaces as a lane skip. Aux lanes are enrichment; the degradation is visible but never blocks.
|
||||||
|
script: scripts/aux_fault.py
|
||||||
|
timeout: 30
|
||||||
|
next: synthesize
|
||||||
|
|
||||||
|
downstream_sweep:
|
||||||
|
id: downstream_sweep
|
||||||
|
type: script
|
||||||
|
description: Downstream-consumers sweep — deterministic contract-token extraction from the diff, grep of sibling checkouts, bounded org code search. Runs only when consumer-facing surface changed; degrades silently.
|
||||||
|
script: scripts/downstream_sweep.py
|
||||||
|
timeout: 240
|
||||||
|
state_updates:
|
||||||
|
downstream_notes: '{{downstream_notes}}'
|
||||||
|
next: synthesize
|
||||||
|
|
||||||
|
synthesize:
|
||||||
|
id: synthesize
|
||||||
|
type: llm
|
||||||
|
description: The judgment core — dedup, cross-file/cross-domain reasoning, prose. Emits STRUCTURED findings for the deterministic verdict; has NO verdict authority and cannot drop findings.
|
||||||
|
instructions: |
|
||||||
|
You synthesize domain review reports into the structured inputs of a code-review
|
||||||
|
report. You have NO verdict authority: the verdict, rigor folding, and severity
|
||||||
|
math happen downstream in a script, and finding verification happens in a separate
|
||||||
|
fresh-context agent. Your job is judgment, not grading.
|
||||||
|
|
||||||
|
Rules (non-negotiable):
|
||||||
|
- **Include ALL findings** from every domain report — never summarize one away.
|
||||||
|
Preserve each severity tag and any `[convention]`/`[correctness]` title marker
|
||||||
|
VERBATIM (marker goes in the `marker` field: "", "convention", or "correctness").
|
||||||
|
- **Dedup**: an identical finding reported twice (e.g. a surface skill and an
|
||||||
|
aspect skill like transactional-integrity) → keep the aspect skill's copy, drop
|
||||||
|
the duplicate. Dedup is the ONLY permitted removal.
|
||||||
|
- **Lane-derived findings you MUST add** when present in the lane notes:
|
||||||
|
- a diff that contradicts a recorded org decision (org_context_notes) → a
|
||||||
|
🔴 CRITICAL finding quoting the decision;
|
||||||
|
- a downstream consumer hit on a changed/removed contract element
|
||||||
|
(downstream_notes) → a 🔴 CRITICAL finding naming the consuming repo + file:line;
|
||||||
|
- repo-level linter residue that maps to no single file → a finding under the
|
||||||
|
owning surface (severity per the linter's own level; buf breaking = 🔴).
|
||||||
|
- **Every finding needs**: id (f1, f2, …), severity (icon + word verbatim, e.g.
|
||||||
|
"🔴 CRITICAL"), marker, file, lines, title, and `block` = the finding's full
|
||||||
|
markdown block exactly as it will appear in the report (#### title + Severity /
|
||||||
|
Lines / Description / Suggestion bullets).
|
||||||
|
- **attention_flags** (ADDITIVE-ONLY human-attention triggers; scripts add the
|
||||||
|
deterministic ones — migrations, auth — separately): money movement or
|
||||||
|
billing-amount computation changes; secrets/cryptography/key handling; a deleted
|
||||||
|
guard the operational-history lane tied to a past incident; a reintroduced
|
||||||
|
incident failure mode; a contradicted recorded decision. One short line each.
|
||||||
|
- **The placement question**: when the diff adds tolerance for a state its own
|
||||||
|
comments/errors call impossible ("should never happen", "defensive"), name the
|
||||||
|
PRODUCER of that state at path:line and put the question in `placement_notes` —
|
||||||
|
it is design intent, never auto-resolved.
|
||||||
|
- Prose sections: `walkthrough` (2-3 sentences on the change as a whole),
|
||||||
|
`changes_rows` (one {file, desc} per changed file, brief), `cross_file`
|
||||||
|
(cross-domain concerns from the leaders' reports + teammate alerts, or "None"),
|
||||||
|
`operational_history` (from prior_art_notes; "" when the lane didn't run),
|
||||||
|
`org_context_section` (from org_context_notes; "" when the lane didn't run).
|
||||||
|
Prior review decisions supplied in the caller context are SETTLED: never
|
||||||
|
invert or re-litigate one on the same line without NEW evidence (the line
|
||||||
|
changed, a new failure appeared). If you believe a settled decision is
|
||||||
|
wrong, do not re-flag it as a finding — surface it in `cross_file` as an
|
||||||
|
explicit conflict citing the prior decision, so the human sees a
|
||||||
|
disagreement, not a flip-flop.
|
||||||
|
|
||||||
|
prompt: |
|
||||||
|
Domain reports ({{file_count}} changed files, grouped into the domain slices below):
|
||||||
|
{{domain_reports}}
|
||||||
|
|
||||||
|
Lane notes:
|
||||||
|
- org context: {{org_context_notes}}
|
||||||
|
- operational history / prior art: {{prior_art_notes}}
|
||||||
|
- downstream consumers: {{downstream_notes}}
|
||||||
|
- lane skips: {{aux_note}}
|
||||||
|
- linter summary: {{linter_notes}}
|
||||||
|
- facts notes: {{facts_note}} {{groups_note}}
|
||||||
|
|
||||||
|
Intent: {{intent}}
|
||||||
|
Rigor: {{resolved_rigor}}; surfaces: {{resolved_surfaces}}
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
walkthrough:
|
||||||
|
type: string
|
||||||
|
changes_rows:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
file:
|
||||||
|
type: string
|
||||||
|
desc:
|
||||||
|
type: string
|
||||||
|
required: [file, desc]
|
||||||
|
findings:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
id:
|
||||||
|
type: string
|
||||||
|
severity:
|
||||||
|
type: string
|
||||||
|
marker:
|
||||||
|
type: string
|
||||||
|
file:
|
||||||
|
type: string
|
||||||
|
lines:
|
||||||
|
type: string
|
||||||
|
title:
|
||||||
|
type: string
|
||||||
|
block:
|
||||||
|
type: string
|
||||||
|
required: [id, severity, marker, file, lines, title, block]
|
||||||
|
cross_file:
|
||||||
|
type: string
|
||||||
|
operational_history:
|
||||||
|
type: string
|
||||||
|
org_context_section:
|
||||||
|
type: string
|
||||||
|
attention_flags:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: string
|
||||||
|
placement_notes:
|
||||||
|
type: string
|
||||||
|
required: [walkthrough, changes_rows, findings, cross_file, operational_history, org_context_section, attention_flags, placement_notes]
|
||||||
|
max_iterations: 6
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
synth_failure: '{{output}}'
|
||||||
|
fallback: verdict
|
||||||
|
next: verify
|
||||||
|
|
||||||
|
verify:
|
||||||
|
id: verify
|
||||||
|
type: agent
|
||||||
|
description: Independent fresh-context verification of every 🔴/🟡 finding — structurally on the path to the report; cannot be skipped.
|
||||||
|
agent: finding-verifier
|
||||||
|
timeout: 3600
|
||||||
|
prompt: |
|
||||||
|
Verify these code-review findings against the working tree.
|
||||||
|
|
||||||
|
Project: {{project_dir}}
|
||||||
|
|
||||||
|
Findings (JSON; verify every 🔴 and 🟡 entry — id, severity, path, lines, claim
|
||||||
|
are the fields; `title` + `block` together are the claim):
|
||||||
|
{{findings}}
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
verifier_output: '{{output}}'
|
||||||
|
fallback: verdict
|
||||||
|
next: verify_gate
|
||||||
|
|
||||||
|
verify_gate:
|
||||||
|
id: verify_gate
|
||||||
|
type: script
|
||||||
|
description: Local verifier-fault retry — when the verifier completed but its output carries PIPELINE-FAULT markers (or the verifier died), re-run the cheap finding-verifier ONCE instead of letting the fault escalate into a full lane re-run at the gauntlet. Falls through to verdict when clean or out of retries; a gate crash falls forward on the declared edge.
|
||||||
|
script: scripts/verify_gate.py
|
||||||
|
timeout: 60
|
||||||
|
next: verdict
|
||||||
|
|
||||||
|
verdict:
|
||||||
|
id: verdict
|
||||||
|
type: script
|
||||||
|
description: Deterministic verdict math — applies verifier drops (FALSE, tallied), attaches evidence, folds by rigor, computes MERGE-READY/NEEDS-HUMAN from severity counts + always-human triggers. The model never grades its own homework.
|
||||||
|
script: scripts/verdict.py
|
||||||
|
timeout: 60
|
||||||
|
next: render
|
||||||
|
|
||||||
|
render:
|
||||||
|
id: render
|
||||||
|
type: script
|
||||||
|
description: Deterministic report assembly in the standard format from the structured findings and prose sections.
|
||||||
|
script: scripts/render.py
|
||||||
|
timeout: 30
|
||||||
|
next: done
|
||||||
|
|
||||||
|
done:
|
||||||
|
id: done
|
||||||
|
type: end
|
||||||
|
output: |
|
||||||
|
{{final_report}}
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for dead aux context lanes of the code-reviewer graph.
|
||||||
|
|
||||||
|
Reached ONLY via aux_lanes' `fallback:` route. aux_lanes' state_updates wrote
|
||||||
|
an "LLM node …failed: <chain>" string into `aux_failure` before routing here;
|
||||||
|
this node records the failure as a PIPELINE-FAULT in `aux_note`, which flows
|
||||||
|
into synthesize's "lane skips" prompt line. Aux lanes are ENRICHMENT, not
|
||||||
|
review coverage: the degradation is made VISIBLE in the report's inputs but
|
||||||
|
never blocks — verdict.py deliberately does NOT scan aux_note for faults.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("aux_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{"aux_note": f"PIPELINE-FAULT: aux context lanes failed after retries — {detail}"}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"aux_note": "PIPELINE-FAULT: aux context lanes failed after retries — "
|
||||||
|
f"fault-marker script error: {e}"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Report-contract gate for each domain-reviewer's output.
|
||||||
|
|
||||||
|
"Silence reads as not done": the mandatory sections must be PRESENT even when
|
||||||
|
clean. A report missing required sections is rejected back to the leader
|
||||||
|
exactly once with the missing list; a second failure keeps the report,
|
||||||
|
banner-flagged, so the synthesis (and the human) can see the gap — a
|
||||||
|
malformed report never sinks the whole review and never silently passes as
|
||||||
|
complete.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
REQUIRED = [
|
||||||
|
("## Domain:", "the `## Domain:` header"),
|
||||||
|
("### Findings", "the `### Findings` section"),
|
||||||
|
("### Mandatory checks", "the `### Mandatory checks` section (report blast radius / test adequacy / guard symmetry even when clean)"),
|
||||||
|
("DOMAIN_REVIEW_COMPLETE", "the `DOMAIN_REVIEW_COMPLETE` sentinel"),
|
||||||
|
]
|
||||||
|
MAX_RETRIES = 1
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
report = state.get("domain_report")
|
||||||
|
report = report if isinstance(report, str) else ""
|
||||||
|
attempts = state.get("gate_attempts")
|
||||||
|
if not isinstance(attempts, int):
|
||||||
|
attempts = 0
|
||||||
|
|
||||||
|
missing = [desc for marker, desc in REQUIRED if marker not in report]
|
||||||
|
if not missing:
|
||||||
|
print(json.dumps({}))
|
||||||
|
return
|
||||||
|
|
||||||
|
if attempts < MAX_RETRIES:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"_next": "review_domain",
|
||||||
|
"gate_attempts": attempts + 1,
|
||||||
|
"gate_feedback": (
|
||||||
|
"- COMPLETENESS GATE REJECTED your previous report — missing: "
|
||||||
|
+ "; ".join(missing)
|
||||||
|
+ ". Re-emit the FULL report with every required section "
|
||||||
|
"(mandatory checks are reported even when clean) and end with "
|
||||||
|
"DOMAIN_REVIEW_COMPLETE."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
group = state.get("domain_group")
|
||||||
|
domain = group.get("domain", "?") if isinstance(group, dict) else "?"
|
||||||
|
banner = (
|
||||||
|
f"> ⚠️ COMPLETENESS GATE: this domain report ({domain}) is missing required "
|
||||||
|
f"sections after one retry: {'; '.join(missing)}. Treat the missing checks as "
|
||||||
|
"NOT DONE, not as clean.\n\n"
|
||||||
|
)
|
||||||
|
print(json.dumps({"domain_report": banner + report}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a gate crash must never sink the map
|
||||||
|
print(json.dumps({"domain_report": f"> ⚠️ COMPLETENESS GATE ERROR: {e}\n\n(report unavailable)"}))
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Exact-cover gate for the refined domain groups.
|
||||||
|
|
||||||
|
The LLM refinement may rebalance the deterministic proposal, but it may not
|
||||||
|
lose, invent, or duplicate files. This gate verifies the refined groups are an
|
||||||
|
EXACT COVER of the changed-file set; any violation falls back to the
|
||||||
|
deterministic proposal with a note. The LLM can shape the grouping, never
|
||||||
|
shrink it. Emits `group_items` — the map node's item list.
|
||||||
|
|
||||||
|
Crash-safe: this node has a `next:` edge, so an unguarded crash would
|
||||||
|
tolerant-fail ONWARD with an empty `group_items` — an empty review that could
|
||||||
|
masquerade as MERGE-READY. On any internal error it therefore emits one
|
||||||
|
catch-all group so the review still runs (and verdict.py's pipeline tripwire
|
||||||
|
catches the empty-report case independently).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
changed = state.get("changed_files") or []
|
||||||
|
proposed = state.get("proposed_groups") or []
|
||||||
|
refined = state.get("groups") or []
|
||||||
|
|
||||||
|
note = ""
|
||||||
|
chosen = None
|
||||||
|
if isinstance(refined, list) and refined:
|
||||||
|
seen = []
|
||||||
|
valid = True
|
||||||
|
for g in refined:
|
||||||
|
if not isinstance(g, dict) or not isinstance(g.get("files"), list) or not g.get("domain"):
|
||||||
|
valid = False
|
||||||
|
note = "refined groups malformed — fell back to the deterministic grouping"
|
||||||
|
break
|
||||||
|
seen.extend(g["files"])
|
||||||
|
if valid:
|
||||||
|
if sorted(seen) != sorted(changed):
|
||||||
|
missing = sorted(set(changed) - set(seen))
|
||||||
|
extra = sorted(set(seen) - set(changed))
|
||||||
|
dupes = sorted({f for f in seen if seen.count(f) > 1})
|
||||||
|
note = (
|
||||||
|
"refined groups failed the exact-cover check "
|
||||||
|
f"(missing={missing[:5]} extra={extra[:5]} duplicated={dupes[:5]}) — "
|
||||||
|
"fell back to the deterministic grouping"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
chosen = [{"domain": g["domain"], "files": g["files"]} for g in refined]
|
||||||
|
if chosen is None:
|
||||||
|
# refine_groups' fallback lands here with `groups` never set; make the
|
||||||
|
# degradation visible in the synthesis prompt via groups_note.
|
||||||
|
refine_failure = state.get("refine_failure")
|
||||||
|
if not refined and isinstance(refine_failure, str) and refine_failure.startswith("LLM node"):
|
||||||
|
note = "refinement lane failed after retries — fell back to the deterministic grouping"
|
||||||
|
chosen = [{"domain": g["domain"], "files": g["files"]} for g in proposed if isinstance(g, dict)]
|
||||||
|
|
||||||
|
print(json.dumps({"group_items": chosen, "groups_note": note}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crash must never tolerant-fail into an empty review
|
||||||
|
try:
|
||||||
|
files = load_state().get("changed_files") or []
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
files = []
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"group_items": [{"domain": "all-changes", "files": files}],
|
||||||
|
"groups_note": f"cover gate error: {e} — fell back to a single catch-all group",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed domain lane fault marker for the code-reviewer graph.
|
||||||
|
|
||||||
|
Reached ONLY via review_domain's `fallback:` route inside the map branch.
|
||||||
|
Grounded in engine behavior: on agent-node failure the engine writes
|
||||||
|
"Agent node failed: {e:#}" into `domain_report` via the node's state_updates
|
||||||
|
BEFORE routing to the fallback, so the raw failure text is already in state.
|
||||||
|
This node normalizes it into a "> ⚠️ PIPELINE-FAULT: domain review lane
|
||||||
|
failed after retries — …" banner the map collects as the lane's report;
|
||||||
|
verdict.py counts any banner-prefixed report as a fault and forces
|
||||||
|
NEEDS-HUMAN — a dead domain lane never reads as a clean slice.
|
||||||
|
|
||||||
|
Slice identity comes from `domain_group` (the map item cover_gate emitted,
|
||||||
|
which carries `domain` and `files`); a missing/malformed item degrades to
|
||||||
|
the failure detail alone.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
FAULT_PREFIX = "> ⚠️ PIPELINE-FAULT: domain review lane failed after retries — "
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
slice_desc = ""
|
||||||
|
group = state.get("domain_group")
|
||||||
|
if isinstance(group, dict):
|
||||||
|
domain = group.get("domain")
|
||||||
|
files = [f for f in (group.get("files") or []) if isinstance(f, str)]
|
||||||
|
if isinstance(domain, str) and domain.strip():
|
||||||
|
slice_desc = f"domain '{domain.strip()}' (files: {', '.join(files)}): "
|
||||||
|
raw = state.get("domain_report")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(json.dumps({"domain_report": f"{FAULT_PREFIX}{slice_desc}{detail}"}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"domain_report": "> ⚠️ PIPELINE-FAULT: domain review lane failed "
|
||||||
|
f"after retries — fault-marker script error: {e}"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Downstream-consumers sweep for the code-reviewer graph.
|
||||||
|
|
||||||
|
Runs only when consumer-facing contract surface changed. Extracts the most
|
||||||
|
specific changed contract tokens from the diff deterministically (route path
|
||||||
|
literals, proto message/service/rpc names, exported symbols), greps sibling
|
||||||
|
checkouts under `consumers_root`, and runs a bounded org code search via gh.
|
||||||
|
Hits are reported for the synthesis to raise as 🔴 findings naming the
|
||||||
|
consuming repo + file:line. Degrades silently when unconfigured/offline;
|
||||||
|
public-library caveats are noted, not guessed at.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
state = load_state()
|
||||||
|
proj = os.path.abspath(state.get("project_dir") or ".")
|
||||||
|
spec = state.get("resolved_diff_spec") or "worktree"
|
||||||
|
consumers_root = os.path.expanduser((state.get("consumers_root") or "").strip())
|
||||||
|
github_org = (state.get("github_org") or "").strip()
|
||||||
|
|
||||||
|
if not state.get("consumer_surface"):
|
||||||
|
print(json.dumps({"downstream_notes": "(skipped — no consumer-facing contract surface in the diff)"}))
|
||||||
|
raise SystemExit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def diff_text():
|
||||||
|
args = {"staged": ["--cached"], "worktree": ["HEAD"], "HEAD~1": ["HEAD~1"]}.get(spec, [spec])
|
||||||
|
r = subprocess.run(["git", "-C", proj, "diff", *args], capture_output=True, text=True, timeout=90)
|
||||||
|
return r.stdout if r.returncode == 0 else ""
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
changed = [
|
||||||
|
l[1:] for l in diff_text().splitlines()
|
||||||
|
if (l.startswith("+") or l.startswith("-")) and not l.startswith(("+++", "---"))
|
||||||
|
]
|
||||||
|
text = "\n".join(changed)
|
||||||
|
tokens = set()
|
||||||
|
tokens.update(re.findall(r'["\'](/[a-z0-9][a-z0-9/_{}.:-]{5,})["\']', text)) # route paths
|
||||||
|
tokens.update(re.findall(r"^\s*(?:message|service|rpc)\s+([A-Z][A-Za-z0-9_]{5,})", text, re.M)) # proto names
|
||||||
|
tokens.update(re.findall(r"^\s*(?:pub\s+fn|pub\s+struct|func|export\s+(?:function|const|class))\s+([A-Z][A-Za-z0-9_]{5,})", text, re.M))
|
||||||
|
tokens = sorted(tokens, key=len, reverse=True)[:5]
|
||||||
|
|
||||||
|
notes = []
|
||||||
|
if not tokens:
|
||||||
|
notes.append("(no specific contract tokens extractable from the diff — sweep skipped)")
|
||||||
|
else:
|
||||||
|
notes.append(f"contract tokens probed: {tokens}")
|
||||||
|
if consumers_root and os.path.isdir(consumers_root):
|
||||||
|
self_name = os.path.basename(proj)
|
||||||
|
for tok in tokens:
|
||||||
|
r = subprocess.run(
|
||||||
|
["grep", "-rIl", "--exclude-dir=.git", "--exclude-dir=node_modules",
|
||||||
|
"--exclude-dir=target", "--exclude-dir=vendor", tok, consumers_root],
|
||||||
|
capture_output=True, text=True, timeout=60,
|
||||||
|
)
|
||||||
|
hits = [
|
||||||
|
h for h in r.stdout.splitlines()
|
||||||
|
if h.strip() and f"/{self_name}/" not in h and not h.startswith(proj)
|
||||||
|
][:4]
|
||||||
|
for h in hits:
|
||||||
|
notes.append(f"CONSUMER HIT (sibling checkout): {h} references '{tok}' — "
|
||||||
|
"a changed/removed contract element with a live consumer is a 🔴 finding")
|
||||||
|
elif consumers_root:
|
||||||
|
notes.append(f"(consumers_root '{consumers_root}' not found — local probe skipped)")
|
||||||
|
else:
|
||||||
|
notes.append("(consumers_root unset — local probe skipped)")
|
||||||
|
|
||||||
|
if github_org and shutil.which("gh"):
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("CLICOLOR_FORCE", None)
|
||||||
|
env.pop("FORCE_COLOR", None)
|
||||||
|
env["GH_NO_UPDATE_NOTIFIER"] = "1"
|
||||||
|
for tok in tokens:
|
||||||
|
r = subprocess.run(
|
||||||
|
["gh", "api", "search/code", "-f", f'q=org:{github_org} "{tok}"',
|
||||||
|
"--jq", r'.items[:4][] | "\(.repository.full_name): \(.path)"'],
|
||||||
|
capture_output=True, text=True, timeout=30, env=env,
|
||||||
|
)
|
||||||
|
if r.returncode == 0 and r.stdout.strip():
|
||||||
|
for line in r.stdout.strip().splitlines():
|
||||||
|
notes.append(f"CONSUMER HIT (org search): {line} references '{tok}'")
|
||||||
|
elif github_org:
|
||||||
|
notes.append("(gh unavailable — org code search skipped)")
|
||||||
|
else:
|
||||||
|
notes.append("(github_org unset — org code search skipped)")
|
||||||
|
notes.append("Public-library caveat: consumers of a PUBLIC library are unenumerable — "
|
||||||
|
"for exported-API changes the check degrades to semver/changelog discipline.")
|
||||||
|
print(json.dumps({"downstream_notes": "\n".join(notes)}))
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
print(json.dumps({"downstream_notes": f"(sweep degraded: {e})"}))
|
||||||
@@ -0,0 +1,288 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic review facts for the code-reviewer graph.
|
||||||
|
|
||||||
|
Resolves the diff, the changed-file list, the quality bar (caller-passed →
|
||||||
|
plan frontmatter → detected default), the conventions pointer, review-miss
|
||||||
|
ledger matches, the repo-history sweep (fix/revert clusters, bounded gh
|
||||||
|
PR-comment pull), and the deterministic verdict-trigger signals. Pure
|
||||||
|
git/gh/filesystem — no LLM. Failures degrade into `facts_note`; they never
|
||||||
|
sink the review.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import glob
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
state = load_state()
|
||||||
|
proj = os.path.expanduser(
|
||||||
|
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
|
||||||
|
)
|
||||||
|
spec = (state.get("diff_spec") or "auto").strip()
|
||||||
|
notes = []
|
||||||
|
out = {"project_dir": proj}
|
||||||
|
|
||||||
|
|
||||||
|
def git(*args, timeout=60):
|
||||||
|
r = subprocess.run(["git", "-C", proj, *args], capture_output=True, text=True, timeout=timeout)
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise RuntimeError((r.stderr or r.stdout).strip()[:300])
|
||||||
|
return r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def scrubbed_env():
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("CLICOLOR_FORCE", None)
|
||||||
|
env.pop("FORCE_COLOR", None)
|
||||||
|
env["GH_NO_UPDATE_NOTIFIER"] = "1"
|
||||||
|
env["NO_COLOR"] = "1"
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
# --- diff resolution (get_diff semantics) -----------------------------------
|
||||||
|
resolved = spec
|
||||||
|
files = []
|
||||||
|
try:
|
||||||
|
if spec in ("auto", ""):
|
||||||
|
names = git("diff", "--cached", "--numstat")
|
||||||
|
resolved = "staged"
|
||||||
|
if not names.strip():
|
||||||
|
names = git("diff", "--numstat")
|
||||||
|
resolved = "worktree"
|
||||||
|
if not names.strip():
|
||||||
|
names = git("diff", "HEAD~1", "--numstat")
|
||||||
|
resolved = "HEAD~1"
|
||||||
|
elif spec == "staged":
|
||||||
|
names = git("diff", "--cached", "--numstat")
|
||||||
|
elif spec == "worktree":
|
||||||
|
names = git("diff", "HEAD", "--numstat")
|
||||||
|
else:
|
||||||
|
names = git("diff", spec, "--numstat")
|
||||||
|
for line in names.splitlines():
|
||||||
|
parts = line.split("\t")
|
||||||
|
if len(parts) == 3:
|
||||||
|
added, deleted, path = parts
|
||||||
|
if added == "-" and deleted == "-":
|
||||||
|
notes.append(f"binary file skipped: {path}")
|
||||||
|
continue
|
||||||
|
files.append(path.strip())
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
notes.append(f"diff resolution failed for spec {spec!r}: {e}")
|
||||||
|
|
||||||
|
out["resolved_diff_spec"] = resolved
|
||||||
|
out["changed_files"] = files[:300]
|
||||||
|
out["file_count"] = len(files)
|
||||||
|
if len(files) > 300:
|
||||||
|
notes.append(
|
||||||
|
f"file list truncated: {len(files)} changed files, only the first 300 are "
|
||||||
|
"grouped and reviewed"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def diff_text():
|
||||||
|
try:
|
||||||
|
if resolved == "staged":
|
||||||
|
return git("diff", "--cached", timeout=90)
|
||||||
|
if resolved == "worktree":
|
||||||
|
return git("diff", "HEAD", timeout=90) if spec == "worktree" else git("diff", timeout=90)
|
||||||
|
if resolved == "HEAD~1":
|
||||||
|
return git("diff", "HEAD~1", timeout=90)
|
||||||
|
return git("diff", resolved, timeout=90)
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
dtext = diff_text()
|
||||||
|
added_lines = "\n".join(
|
||||||
|
l for l in dtext.splitlines() if l.startswith("+") and not l.startswith("+++")
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- quality bar resolution --------------------------------------------------
|
||||||
|
rigor_in = (state.get("rigor_in") or state.get("rigor") or "").strip()
|
||||||
|
surfaces_in = (state.get("surfaces_in") or state.get("surfaces") or "").strip()
|
||||||
|
resolved_rigor, resolved_surfaces, provenance = "production", "", "detected-default"
|
||||||
|
if rigor_in or surfaces_in:
|
||||||
|
resolved_rigor = rigor_in or "production"
|
||||||
|
resolved_surfaces = surfaces_in
|
||||||
|
provenance = "passed"
|
||||||
|
else:
|
||||||
|
for plan in sorted(glob.glob(os.path.join(proj, "plans", "PLAN-*.md"))):
|
||||||
|
try:
|
||||||
|
with open(plan) as f:
|
||||||
|
head = f.read(4000)
|
||||||
|
m = re.match(r"\A---\n(.*?)\n---", head, re.S)
|
||||||
|
if m and re.search(r"^status:\s*active\s*$", m.group(1), re.M):
|
||||||
|
rig = re.search(r"^rigor:\s*(\S+)", m.group(1), re.M)
|
||||||
|
sur = re.search(r"^surfaces:\s*(.+)$", m.group(1), re.M)
|
||||||
|
if rig:
|
||||||
|
resolved_rigor = rig.group(1).strip()
|
||||||
|
if sur:
|
||||||
|
resolved_surfaces = sur.group(1).strip("[] \n")
|
||||||
|
provenance = "plan"
|
||||||
|
break
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
|
||||||
|
SURFACE_PATTERNS = [
|
||||||
|
("db-migration", r"migrations?/|\.sql$|schema\.(rb|prisma|sql)"),
|
||||||
|
("iac", r"\.tf$|helm|charts?/|Dockerfile|docker-compose|k8s/|manifests?/"),
|
||||||
|
("ci-cd", r"\.github/workflows/|\.gitlab-ci|Jenkinsfile|\.circleci/"),
|
||||||
|
("worker", r"workers?/|jobs?/|consumers?/|cron|celery|sidekiq|river"),
|
||||||
|
("rest-api", r"routes?|handlers?|controllers?|endpoints?|\bapi\b|\.proto$|openapi|swagger|graphql|resolvers?"),
|
||||||
|
("cli", r"\bcmd/|\bcli\b"),
|
||||||
|
("library", r"go\.mod$|package\.json$|Cargo\.toml$|pyproject\.toml$|setup\.py$|__init__\.py$"),
|
||||||
|
]
|
||||||
|
if provenance == "detected-default" and files:
|
||||||
|
detected = []
|
||||||
|
for surface, pat in SURFACE_PATTERNS:
|
||||||
|
if any(re.search(pat, f, re.I) for f in files):
|
||||||
|
detected.append(surface)
|
||||||
|
resolved_surfaces = ", ".join(detected)
|
||||||
|
|
||||||
|
out["resolved_rigor"] = resolved_rigor
|
||||||
|
out["resolved_surfaces"] = resolved_surfaces
|
||||||
|
out["bar_provenance"] = provenance
|
||||||
|
|
||||||
|
# --- conventions pointer ------------------------------------------------------
|
||||||
|
conv = []
|
||||||
|
for pat in ("CLAUDE.md", "COYOTE.md", "CONTRIBUTING.md", ".golangci*", ".eslintrc*",
|
||||||
|
"rustfmt.toml", ".rubocop.yml", "ruff.toml", ".editorconfig", "Makefile"):
|
||||||
|
conv.extend(sorted(glob.glob(os.path.join(proj, pat)))[:2])
|
||||||
|
out["conventions"] = ", ".join(os.path.relpath(p, proj) for p in conv) or "none found"
|
||||||
|
|
||||||
|
# --- review-miss ledger -------------------------------------------------------
|
||||||
|
ledger_text = ""
|
||||||
|
for cand in (
|
||||||
|
os.path.join(os.path.expanduser("~"), ".config", "coyote", "share", "review-misses.md"),
|
||||||
|
os.path.join(os.path.expanduser("~"), ".config", "coyote", "review-misses.md"),
|
||||||
|
):
|
||||||
|
if os.path.isfile(cand):
|
||||||
|
try:
|
||||||
|
with open(cand) as f:
|
||||||
|
ledger_text = f.read()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
break
|
||||||
|
|
||||||
|
matches = []
|
||||||
|
if ledger_text:
|
||||||
|
try:
|
||||||
|
repo_name = os.path.basename(git("rev-parse", "--show-toplevel").strip())
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
repo_name = os.path.basename(os.path.abspath(proj))
|
||||||
|
keywords = {repo_name.lower()}
|
||||||
|
keywords.update(s.strip().lower() for s in resolved_surfaces.split(",") if s.strip())
|
||||||
|
for f in files:
|
||||||
|
keywords.update(p.lower() for p in re.split(r"[/_.-]", f) if len(p) >= 4)
|
||||||
|
entries = re.split(r"(?=^## MISS-)", ledger_text, flags=re.M)
|
||||||
|
for entry in entries:
|
||||||
|
if not entry.startswith("## MISS-"):
|
||||||
|
continue
|
||||||
|
body = entry.lower()
|
||||||
|
if any(k in body for k in keywords):
|
||||||
|
lines = entry.strip().splitlines()
|
||||||
|
matches.append("\n".join(lines[:8]))
|
||||||
|
if len(matches) >= 8:
|
||||||
|
break
|
||||||
|
out["ledger_matches"] = "\n\n".join(matches) if matches else "(no matching ledger entries)"
|
||||||
|
|
||||||
|
# --- repo-history sweep -------------------------------------------------------
|
||||||
|
history = []
|
||||||
|
fragile = []
|
||||||
|
try:
|
||||||
|
if files:
|
||||||
|
log = git("log", "--oneline", "-15", "--", *files[:100])
|
||||||
|
history.append(log.strip()[:2000])
|
||||||
|
per_file = {}
|
||||||
|
for f in files[:50]:
|
||||||
|
try:
|
||||||
|
fl = git("log", "--oneline", "-10", "--", f)
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
continue
|
||||||
|
hits = [l for l in fl.splitlines() if re.search(r"\b(fix|revert|hotfix)\b", l, re.I)]
|
||||||
|
if len(hits) >= 2:
|
||||||
|
fragile.append(f)
|
||||||
|
per_file[f] = hits[:2]
|
||||||
|
for f, hits in per_file.items():
|
||||||
|
history.append(f"FRAGILE: {f} — {len(hits)}+ fix/revert commits recently — raise scrutiny")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
notes.append(f"history sweep failed: {e}")
|
||||||
|
|
||||||
|
# gh one-level-deeper: PR review comments for fragile files (bounded, silent skip)
|
||||||
|
try:
|
||||||
|
if fragile:
|
||||||
|
remote = git("remote", "get-url", "origin").strip()
|
||||||
|
m = re.search(r"github\.com[:/]([^/]+/[^/.]+)", remote)
|
||||||
|
if m:
|
||||||
|
repo_slug = m.group(1)
|
||||||
|
env = scrubbed_env()
|
||||||
|
shas = []
|
||||||
|
for f in fragile[:2]:
|
||||||
|
lg = git("log", "--format=%H", "-2", "--", f)
|
||||||
|
shas.extend(lg.split())
|
||||||
|
pr_nums = []
|
||||||
|
for sha in shas[:3]:
|
||||||
|
r = subprocess.run(
|
||||||
|
["gh", "api", f"repos/{repo_slug}/commits/{sha}/pulls", "--jq", ".[].number"],
|
||||||
|
capture_output=True, text=True, timeout=30, env=env,
|
||||||
|
)
|
||||||
|
if r.returncode == 0:
|
||||||
|
pr_nums.extend(n for n in r.stdout.split() if n.isdigit())
|
||||||
|
seen = []
|
||||||
|
for n in pr_nums:
|
||||||
|
if n not in seen:
|
||||||
|
seen.append(n)
|
||||||
|
comments = []
|
||||||
|
for n in seen[:3]:
|
||||||
|
r = subprocess.run(
|
||||||
|
["gh", "api", f"repos/{repo_slug}/pulls/{n}/comments",
|
||||||
|
"--jq", r'.[] | "\(.path): \(.body)"'],
|
||||||
|
capture_output=True, text=True, timeout=30, env=env,
|
||||||
|
)
|
||||||
|
if r.returncode == 0:
|
||||||
|
for line in r.stdout.splitlines():
|
||||||
|
path = line.split(":", 1)[0]
|
||||||
|
if any(path in f or f in path for f in files):
|
||||||
|
comments.append(line[:200])
|
||||||
|
if len(comments) >= 6:
|
||||||
|
break
|
||||||
|
if comments:
|
||||||
|
history.append("Prior PR review comments on fragile paths:\n" + "\n".join(comments[:6]))
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass # offline / not GitHub / no gh — silent by design
|
||||||
|
|
||||||
|
out["history_notes"] = "\n".join(h for h in history if h) or "(no notable history)"
|
||||||
|
|
||||||
|
# --- deterministic verdict-trigger signals -----------------------------------
|
||||||
|
def any_path(pattern):
|
||||||
|
return any(re.search(pattern, f, re.I) for f in files)
|
||||||
|
|
||||||
|
|
||||||
|
migration_files = [f for f in files if re.search(r"migrations?/|\.sql$", f, re.I)]
|
||||||
|
mig_added = "\n".join(
|
||||||
|
l for l in dtext.splitlines()
|
||||||
|
if l.startswith("+") and not l.startswith("+++")
|
||||||
|
) if migration_files else ""
|
||||||
|
out["destructive_migration"] = bool(
|
||||||
|
migration_files
|
||||||
|
and re.search(r"(?i)drop\s+(table|column)|rename\s+(table|column|to)|alter\s+\S+\s+drop", mig_added)
|
||||||
|
)
|
||||||
|
out["touches_auth"] = bool(
|
||||||
|
any_path(r"auth(?!or)|login|logout|token|session|permission|rbac|\bacl\b|credential")
|
||||||
|
or re.search(r"auth(?!or)|password|jwt|session|credential", added_lines, re.I)
|
||||||
|
)
|
||||||
|
out["consumer_surface"] = any_path(
|
||||||
|
r"routes?|handlers?|\bapi\b|endpoints?|\bcmd/|\bcli\b|\.proto$|openapi|swagger|controllers?|graphql"
|
||||||
|
)
|
||||||
|
|
||||||
|
if notes:
|
||||||
|
out["facts_note"] = "; ".join(notes)[:800]
|
||||||
|
print(json.dumps(out))
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic domain grouping of changed files for the code-reviewer graph.
|
||||||
|
|
||||||
|
Each file lands in the FIRST matching domain (patterns ordered most-specific
|
||||||
|
first); unmatched files land in `general`. When the grouping is very wide,
|
||||||
|
single-file groups merge into `general` so the fan-out stays sane. The LLM
|
||||||
|
refinement node may rebalance this proposal; the cover gate guarantees no
|
||||||
|
file is lost either way. Crash-safe: an error emits one catch-all proposal
|
||||||
|
rather than tolerant-failing into an empty grouping.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
DOMAINS = [
|
||||||
|
("db", r"migrations?/|\.sql$|schema\.(rb|prisma|sql)|models?/"),
|
||||||
|
("iac", r"\.tf$|helm|charts?/|(^|/)Dockerfile|docker-compose|k8s/|manifests?/"),
|
||||||
|
("ci-cd", r"\.github/workflows/|\.gitlab-ci|Jenkinsfile|\.circleci/"),
|
||||||
|
("workers", r"workers?/|jobs?/|consumers?/|cron|tasks?/"),
|
||||||
|
("api", r"routes?|handlers?|controllers?|endpoints?|\bapi\b|\.proto$|openapi|swagger|graphql|resolvers?"),
|
||||||
|
("cli", r"\bcmd/|\bcli\b"),
|
||||||
|
("deps", r"go\.(mod|sum)$|package(-lock)?\.json$|yarn\.lock$|Cargo\.(toml|lock)$|requirements[^/]*\.txt$|pyproject\.toml$|Gemfile|pom\.xml$|composer\.json$"),
|
||||||
|
("tests", r"(^|/)tests?/|_test\.|\.test\.|spec\."),
|
||||||
|
("docs", r"\.(md|rst|adoc|txt)$"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
files = state.get("changed_files") or []
|
||||||
|
|
||||||
|
groups = {}
|
||||||
|
for f in files:
|
||||||
|
domain = "general"
|
||||||
|
for name, pat in DOMAINS:
|
||||||
|
if re.search(pat, f, re.I):
|
||||||
|
domain = name
|
||||||
|
break
|
||||||
|
groups.setdefault(domain, []).append(f)
|
||||||
|
|
||||||
|
# Wide fan-out: merge single-file groups into general.
|
||||||
|
if len(groups) > 6:
|
||||||
|
for name in list(groups):
|
||||||
|
if name != "general" and len(groups[name]) == 1:
|
||||||
|
groups.setdefault("general", []).extend(groups.pop(name))
|
||||||
|
|
||||||
|
proposed = [{"domain": k, "files": v} for k, v in sorted(groups.items())]
|
||||||
|
print(json.dumps({"proposed_groups": proposed}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — never tolerant-fail into an empty grouping
|
||||||
|
try:
|
||||||
|
files = load_state().get("changed_files") or []
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
files = []
|
||||||
|
print(json.dumps({"proposed_groups": [{"domain": "all-changes", "files": files}]}))
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Domain linter pass for the code-reviewer graph.
|
||||||
|
|
||||||
|
Runs whichever mechanized checkers exist on PATH against the matching changed
|
||||||
|
files: tflint (Terraform), hadolint (Dockerfiles), actionlint (workflow
|
||||||
|
files), and `buf breaking` when buf-configured protos changed. Read-only,
|
||||||
|
never installs anything, and each tool degrades silently to a skip note.
|
||||||
|
Output is routed per-file where possible so the domain leaders can extract
|
||||||
|
their slice's lines.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
state = load_state()
|
||||||
|
proj = state.get("project_dir") or "."
|
||||||
|
files = state.get("changed_files") or []
|
||||||
|
spec = state.get("resolved_diff_spec") or "worktree"
|
||||||
|
notes = []
|
||||||
|
|
||||||
|
|
||||||
|
def run(cmd, timeout=60):
|
||||||
|
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, cwd=proj)
|
||||||
|
return (r.stdout + r.stderr).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def capped(text, n=40):
|
||||||
|
lines = text.splitlines()
|
||||||
|
return "\n".join(lines[:n]) + (f"\n… ({len(lines) - n} more lines)" if len(lines) > n else "")
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
tf = [f for f in files if f.endswith(".tf")]
|
||||||
|
if tf and shutil.which("tflint"):
|
||||||
|
notes.append("[tflint]\n" + capped(run(["tflint", "--no-color"], 120) or "clean"))
|
||||||
|
elif tf:
|
||||||
|
notes.append("[tflint] skipped — not on PATH")
|
||||||
|
|
||||||
|
dockers = [f for f in files if re.search(r"(^|/)Dockerfile", f)]
|
||||||
|
if dockers and shutil.which("hadolint"):
|
||||||
|
notes.append("[hadolint]\n" + capped(run(["hadolint", "--no-color", *dockers[:10]], 60) or "clean"))
|
||||||
|
elif dockers:
|
||||||
|
notes.append("[hadolint] skipped — not on PATH")
|
||||||
|
|
||||||
|
workflows = [f for f in files if ".github/workflows/" in f]
|
||||||
|
if workflows and shutil.which("actionlint"):
|
||||||
|
notes.append("[actionlint]\n" + capped(run(["actionlint", "-no-color", *workflows[:10]], 60) or "clean"))
|
||||||
|
elif workflows:
|
||||||
|
notes.append("[actionlint] skipped — not on PATH")
|
||||||
|
|
||||||
|
protos = [f for f in files if f.endswith(".proto")]
|
||||||
|
if protos and os.path.isfile(os.path.join(proj, "buf.yaml")) and shutil.which("buf"):
|
||||||
|
base = None
|
||||||
|
if "..." in spec:
|
||||||
|
base = spec.split("...")[0]
|
||||||
|
elif ".." in spec:
|
||||||
|
base = spec.split("..")[0]
|
||||||
|
elif spec not in ("staged", "worktree", "HEAD~1"):
|
||||||
|
base = spec
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
r = subprocess.run(["git", "-C", proj, "rev-parse", "HEAD"],
|
||||||
|
capture_output=True, text=True, timeout=15)
|
||||||
|
base = r.stdout.strip() if r.returncode == 0 else None
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
base = None
|
||||||
|
if base:
|
||||||
|
out = run(["buf", "breaking", "--against", f".git#ref={base}"], 120)
|
||||||
|
notes.append("[buf breaking] (wire-compat breaks are 🔴 by default)\n" + capped(out or "clean"))
|
||||||
|
else:
|
||||||
|
notes.append("[buf breaking] skipped — no base ref resolvable")
|
||||||
|
elif protos:
|
||||||
|
notes.append("[buf breaking] skipped — no buf.yaml or buf not on PATH")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
notes.append(f"[linters] pass degraded: {e}")
|
||||||
|
|
||||||
|
print(json.dumps({"linter_notes": "\n\n".join(notes) if notes else "(no applicable linters)"}))
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead parse stage of the code-reviewer graph.
|
||||||
|
|
||||||
|
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||||
|
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||||
|
this node emits a complete PIPELINE-FAULT `verdict_out` and the graph jumps
|
||||||
|
straight to render — a review that could not even parse its request never
|
||||||
|
reads as MERGE-READY. render.py renders a PIPELINE-FAULT verdict despite the
|
||||||
|
empty changed-file list (its "No changes to review." early return skips
|
||||||
|
fault-reason verdicts).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
ATTENTION_LINE = (
|
||||||
|
"PIPELINE-FAULT: parse failed — the review request could not be parsed; "
|
||||||
|
"the review did not run"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def fault_verdict(detail):
|
||||||
|
return {
|
||||||
|
"verdict": "NEEDS-HUMAN",
|
||||||
|
"reason": f"PIPELINE-FAULT: parse failed — {detail}",
|
||||||
|
"counts": {},
|
||||||
|
"deferred_count": 0,
|
||||||
|
"dropped_count": 0,
|
||||||
|
"dropped_titles": [],
|
||||||
|
"attention": [ATTENTION_LINE],
|
||||||
|
"findings_final": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("parse_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the parse stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(json.dumps({"verdict_out": fault_verdict(detail)}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(json.dumps({"verdict_out": fault_verdict(f"fault-marker script error: {e}")}))
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic report assembly for the code-reviewer graph.
|
||||||
|
|
||||||
|
Builds the standard report format from the structured findings and prose
|
||||||
|
sections. Pure templating — no judgment, no re-summarization, no finding
|
||||||
|
ever added or removed here.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
# LLM-influenced values rendered after the summary line: a newline could
|
||||||
|
# forge a second summary line that review-gauntlet/verdict_gate.py parses.
|
||||||
|
def one_line(x):
|
||||||
|
return str(x).replace("\n", " ")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
v = state.get("verdict_out") or {}
|
||||||
|
findings = v.get("findings_final") or []
|
||||||
|
counts = v.get("counts") or {}
|
||||||
|
changed = state.get("changed_files") or []
|
||||||
|
rigor = one_line(state.get("resolved_rigor") or "production")
|
||||||
|
|
||||||
|
# A fault can leave changed_files empty (a parse fault dies before the
|
||||||
|
# diff is resolved; a verifier fault fires even on an empty diff) — a
|
||||||
|
# fault-degraded verdict must still render as a full report. parse_fault
|
||||||
|
# puts the marker in the reason; verdict.py prepends it to attention.
|
||||||
|
faulted = str(v.get("reason") or "").startswith("PIPELINE-FAULT:") or any(
|
||||||
|
isinstance(a, str) and a.startswith("PIPELINE-FAULT:") for a in v.get("attention") or []
|
||||||
|
)
|
||||||
|
if not changed and not faulted:
|
||||||
|
print(json.dumps({"final_report": "# Code Review Summary\n\nNo changes to review."}))
|
||||||
|
return
|
||||||
|
|
||||||
|
verdict = v.get("verdict", "NEEDS-HUMAN")
|
||||||
|
lines = [
|
||||||
|
"# Code Review Summary",
|
||||||
|
"",
|
||||||
|
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
|
||||||
|
f"**Verdict: {verdict}** — {one_line(v.get('reason', ''))}",
|
||||||
|
"",
|
||||||
|
"## Walkthrough",
|
||||||
|
state.get("walkthrough") or "(no walkthrough provided)",
|
||||||
|
"",
|
||||||
|
"## Changes",
|
||||||
|
"",
|
||||||
|
"| File | Changes | Findings |",
|
||||||
|
"|------|---------|----------|",
|
||||||
|
]
|
||||||
|
|
||||||
|
desc_by_file = {}
|
||||||
|
for row in state.get("changes_rows") or []:
|
||||||
|
if isinstance(row, dict) and row.get("file"):
|
||||||
|
desc_by_file[row["file"]] = row.get("desc", "")
|
||||||
|
blocking = [f for f in findings if f.get("section") == "blocking"]
|
||||||
|
deferred = [f for f in findings if f.get("section") == "deferred"]
|
||||||
|
for f in changed:
|
||||||
|
per = [x for x in blocking if x.get("file") == f]
|
||||||
|
tally = " ".join(
|
||||||
|
f"{icon} {n}" for icon, n in
|
||||||
|
((i, sum(1 for x in per if x.get("icon") == i)) for i in ("🔴", "🟡", "🟢", "💡"))
|
||||||
|
if n
|
||||||
|
) or "—"
|
||||||
|
lines.append(f"| `{f}` | {desc_by_file.get(f, '')} | {tally} |")
|
||||||
|
|
||||||
|
lines += ["", "## Detailed Findings", ""]
|
||||||
|
by_file = {}
|
||||||
|
for x in blocking:
|
||||||
|
by_file.setdefault(x.get("file", "(unattributed)"), []).append(x)
|
||||||
|
if by_file:
|
||||||
|
for f in sorted(by_file):
|
||||||
|
lines += [f"### `{f}`", ""]
|
||||||
|
for x in by_file[f]:
|
||||||
|
lines += [x.get("block", ""), ""]
|
||||||
|
else:
|
||||||
|
lines += ["No blocking findings.", ""]
|
||||||
|
|
||||||
|
lines += ["## Cross-File Concerns", state.get("cross_file") or "None", ""]
|
||||||
|
if (state.get("operational_history") or "").strip():
|
||||||
|
lines += ["## Operational history", state["operational_history"], ""]
|
||||||
|
if (state.get("org_context_section") or "").strip():
|
||||||
|
lines += ["## Org context", state["org_context_section"], ""]
|
||||||
|
if deferred and rigor != "production":
|
||||||
|
lines += ["## Deferred by quality bar", ""]
|
||||||
|
for x in deferred:
|
||||||
|
lines += [x.get("block", ""), ""]
|
||||||
|
attention = v.get("attention") or []
|
||||||
|
if verdict == "NEEDS-HUMAN" and attention:
|
||||||
|
lines += ["## Human attention required", ""]
|
||||||
|
lines += [f"- {a}" for a in attention]
|
||||||
|
lines += [""]
|
||||||
|
|
||||||
|
miss_ids = sorted({
|
||||||
|
m for x in findings for m in
|
||||||
|
__import__("re").findall(r"MISS-\d+", (x.get("title") or "") + (x.get("block") or ""))
|
||||||
|
})
|
||||||
|
dropped = ", ".join(one_line(t) for t in v.get("dropped_titles", [])[:5])
|
||||||
|
lines += [
|
||||||
|
"---",
|
||||||
|
# CONTRACT: review-gauntlet/verdict_gate.py parses this line — keep the format in sync
|
||||||
|
f"*Reviewed {len(changed)} files, found {counts.get('🔴', 0)} critical, "
|
||||||
|
f"{counts.get('🟡', 0)} warnings, {counts.get('🟢', 0)} suggestions, "
|
||||||
|
f"{counts.get('💡', 0)} nitpicks ({v.get('deferred_count', 0)} deferred by quality bar)*",
|
||||||
|
f"*Quality bar: {rigor} — provenance: {one_line(state.get('bar_provenance', '?'))}; "
|
||||||
|
f"surfaces: {one_line(state.get('resolved_surfaces') or 'none')}*",
|
||||||
|
f"*Verdict: {verdict}; verifier: dropped {v.get('dropped_count', 0)} provably-false"
|
||||||
|
+ (f" ({dropped})" if v.get("dropped_count") else "")
|
||||||
|
+ f"; review-miss patterns applied: {', '.join(miss_ids) if miss_ids else 'none relevant'}*",
|
||||||
|
]
|
||||||
|
print(json.dumps({"final_report": "\n".join(lines)}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
print(json.dumps({"final_report": f"# Code Review Summary\n\n**Verdict: NEEDS-HUMAN** — report rendering error: {e}"}))
|
||||||
@@ -0,0 +1,283 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic verdict math for the code-reviewer graph.
|
||||||
|
|
||||||
|
The model that wrote the findings never grades its own homework:
|
||||||
|
- verifier verdicts are applied mechanically (FALSE → drop, tallied;
|
||||||
|
VERIFIED → evidence attached; UNVERIFIABLE → kept, marked);
|
||||||
|
- rigor folding is arithmetic over severity + [convention]/[correctness]
|
||||||
|
markers, exactly per the published rules;
|
||||||
|
- MERGE-READY/NEEDS-HUMAN is computed from counts + always-human triggers
|
||||||
|
(deterministic signals ∪ the synthesis's ADDITIVE-only flags).
|
||||||
|
Fail-closed: any PIPELINE-FAULT recorded upstream (dead domain lane, dead
|
||||||
|
verifier, verifier verdicts carrying a PIPELINE-FAULT note or the
|
||||||
|
`pipeline-fault` sentinel id, dead synthesis, missing domain reports)
|
||||||
|
forces NEEDS-HUMAN — a degraded run can never read MERGE-READY. Faults
|
||||||
|
always lead the attention list; the display cap applies only to the
|
||||||
|
non-fault entries so a degraded run can never hide a fault.
|
||||||
|
The gate never crashes into a silent verdict: any internal error emits
|
||||||
|
NEEDS-HUMAN naming the error.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def sev_icon(severity):
|
||||||
|
for icon in ("🔴", "🟡", "🟢", "💡"):
|
||||||
|
if icon in (severity or ""):
|
||||||
|
return icon
|
||||||
|
return "🟢"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_verifier_entries(text):
|
||||||
|
"""Extract the raw list of {id, verdict, evidence, note} dicts from
|
||||||
|
finding-verifier output. Order and duplicates are preserved so fault
|
||||||
|
accounting can count every entry (crash-guard entries all carry the
|
||||||
|
same "unknown" id)."""
|
||||||
|
if not isinstance(text, str) or not text.strip():
|
||||||
|
return []
|
||||||
|
start, end = text.find("["), text.rfind("]")
|
||||||
|
if start != -1 and end > start:
|
||||||
|
try:
|
||||||
|
return [
|
||||||
|
v for v in json.loads(text[start : end + 1]) if isinstance(v, dict) and v.get("id")
|
||||||
|
]
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
entries = []
|
||||||
|
for m in re.finditer(r'\{[^{}]*"id"[^{}]*\}', text):
|
||||||
|
try:
|
||||||
|
v = json.loads(m.group(0))
|
||||||
|
if v.get("id"):
|
||||||
|
entries.append(v)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
continue
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def parse_verifier(entries):
|
||||||
|
"""Index verifier entries by finding id; the last entry per id wins."""
|
||||||
|
return {str(v["id"]): v for v in entries}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
findings = [f for f in (state.get("findings") or []) if isinstance(f, dict)]
|
||||||
|
rigor = (state.get("resolved_rigor") or "production").lower()
|
||||||
|
verifier_entries = parse_verifier_entries(state.get("verifier_output"))
|
||||||
|
verdicts = parse_verifier(verifier_entries)
|
||||||
|
|
||||||
|
kept, dropped_titles = [], []
|
||||||
|
for f in findings:
|
||||||
|
icon = sev_icon(f.get("severity"))
|
||||||
|
v = verdicts.get(str(f.get("id")))
|
||||||
|
block = f.get("block") or f"#### {f.get('title', '(untitled)')}"
|
||||||
|
if v and icon in ("🔴", "🟡"):
|
||||||
|
if v.get("verdict") == "FALSE":
|
||||||
|
dropped_titles.append(f.get("title", f.get("id", "?")))
|
||||||
|
continue
|
||||||
|
if v.get("verdict") == "VERIFIED" and v.get("evidence"):
|
||||||
|
block += f"\n- **Evidence** (verifier): {v['evidence']}"
|
||||||
|
elif v.get("verdict") == "UNVERIFIABLE":
|
||||||
|
block += "\n- *(unverified: " + (v.get("note") or "could not be established") + ")*"
|
||||||
|
elif icon in ("🔴", "🟡") and not v:
|
||||||
|
block += "\n- *(unverified: no verifier verdict returned for this finding)*"
|
||||||
|
kept.append({**f, "icon": icon, "block": block})
|
||||||
|
|
||||||
|
# --- rigor folding (arithmetic, never rewrites severities) --------------
|
||||||
|
for f in kept:
|
||||||
|
marker = (f.get("marker") or "").lower()
|
||||||
|
icon = f["icon"]
|
||||||
|
section = "blocking"
|
||||||
|
if icon == "🔴":
|
||||||
|
section = "blocking"
|
||||||
|
elif rigor == "prototype" and ((icon == "🟡" and marker == "convention") or icon == "🟢"):
|
||||||
|
section = "deferred"
|
||||||
|
elif rigor == "poc":
|
||||||
|
if icon == "🟡" and marker == "convention":
|
||||||
|
section = "deferred"
|
||||||
|
elif icon in ("🟢", "💡") and marker == "convention":
|
||||||
|
section = "dropped-by-poc"
|
||||||
|
f["section"] = section
|
||||||
|
final = [f for f in kept if f["section"] != "dropped-by-poc"]
|
||||||
|
blocking = [f for f in final if f["section"] == "blocking"]
|
||||||
|
deferred = [f for f in final if f["section"] == "deferred"]
|
||||||
|
|
||||||
|
counts = {i: sum(1 for f in blocking if f["icon"] == i) for i in ("🔴", "🟡", "🟢", "💡")}
|
||||||
|
yellow_correctness = sum(
|
||||||
|
1 for f in blocking if f["icon"] == "🟡" and (f.get("marker") or "").lower() == "correctness"
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- always-human triggers ----------------------------------------------
|
||||||
|
attention = []
|
||||||
|
if state.get("destructive_migration"):
|
||||||
|
attention.append("Destructive/irreversible migration in the diff (DROP/RENAME) — needs a human signature.")
|
||||||
|
if state.get("touches_auth"):
|
||||||
|
attention.append("Authentication/authorization surface changed — needs a human signature.")
|
||||||
|
for flag in state.get("attention_flags") or []:
|
||||||
|
if isinstance(flag, str) and flag.strip():
|
||||||
|
attention.append(flag.strip())
|
||||||
|
placement = (state.get("placement_notes") or "").strip()
|
||||||
|
if placement:
|
||||||
|
attention.append("Placement question (design intent, never auto-resolved): " + placement)
|
||||||
|
# dedup, preserve order
|
||||||
|
seen = set()
|
||||||
|
attention = [a for a in attention if not (a.lower() in seen or seen.add(a.lower()))]
|
||||||
|
|
||||||
|
# --- pipeline-fault accounting (fail closed) -----------------------------
|
||||||
|
# PREFIX-anchored: a report that merely QUOTES a marker mid-text must not
|
||||||
|
# trip it. Any fault forces NEEDS-HUMAN.
|
||||||
|
faults = []
|
||||||
|
changed = state.get("changed_files") or []
|
||||||
|
reports = state.get("domain_reports") or []
|
||||||
|
faulted = sum(
|
||||||
|
1 for r in reports if isinstance(r, str) and r.lstrip().startswith("> ⚠️ PIPELINE-FAULT:")
|
||||||
|
)
|
||||||
|
if faulted:
|
||||||
|
faults.append(
|
||||||
|
f"PIPELINE-FAULT: {faulted} of {len(reports)} domain review lane(s) "
|
||||||
|
"failed after retries — their slices were not reviewed"
|
||||||
|
)
|
||||||
|
verifier_raw = state.get("verifier_output")
|
||||||
|
if isinstance(verifier_raw, str) and verifier_raw.lstrip().startswith("Agent node failed:"):
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: finding verification failed — findings render as unverified"
|
||||||
|
)
|
||||||
|
# finding-verifier completes "successfully" with faults INSIDE the payload:
|
||||||
|
# parse_fault's `pipeline-fault` sentinel entry, or a per-finding
|
||||||
|
# PIPELINE-FAULT note from verify_fault or verdict_gate's crash guard.
|
||||||
|
# Anchored on the sentinel id / note prefix only. Counted over the raw
|
||||||
|
# entries, not the id-indexed dict, so N crash-guard entries sharing the
|
||||||
|
# "unknown" id count as N.
|
||||||
|
fv_faults = [
|
||||||
|
v
|
||||||
|
for v in verifier_entries
|
||||||
|
if v.get("id") == "pipeline-fault"
|
||||||
|
or (isinstance(v.get("note"), str) and v["note"].lstrip().startswith("PIPELINE-FAULT:"))
|
||||||
|
]
|
||||||
|
# parse_fault's sentinel = the verifier verified NOTHING (hard fault, lane-
|
||||||
|
# degrading). Per-finding verify_fault / crash-guard notes = those findings
|
||||||
|
# simply stand UNVERIFIED (soft: proportionate NEEDS-HUMAN attention — a
|
||||||
|
# ~1 h lane re-run to re-check a few findings is the wrong trade, and the
|
||||||
|
# verify_gate node already retried the verifier locally once).
|
||||||
|
hard_fv = [v for v in fv_faults if v.get("id") == "pipeline-fault"]
|
||||||
|
soft_fv = [v for v in fv_faults if v.get("id") != "pipeline-fault"]
|
||||||
|
|
||||||
|
def _fv_detail(entries):
|
||||||
|
first = entries[0].get("note")
|
||||||
|
detail = first.strip().replace("\n", " ") if isinstance(first, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
return detail
|
||||||
|
|
||||||
|
soft_notes = []
|
||||||
|
if hard_fv:
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: finding verification degraded — the verifier verified "
|
||||||
|
f"nothing ({_fv_detail(hard_fv)})"
|
||||||
|
)
|
||||||
|
if soft_fv:
|
||||||
|
soft_notes.append(
|
||||||
|
f"Verifier: {len(soft_fv)} finding(s) could not be verified (verifier "
|
||||||
|
f"branch fault: {_fv_detail(soft_fv)}) — they stand above UNVERIFIED; "
|
||||||
|
"verify them manually"
|
||||||
|
)
|
||||||
|
# Files beyond facts.py's 300-file cap were never grouped or reviewed —
|
||||||
|
# disclose the gap and force NEEDS-HUMAN with wording that does NOT match
|
||||||
|
# the gauntlet's degraded-run anchors (a lane re-run cannot fix a cap).
|
||||||
|
coverage_notes = []
|
||||||
|
try:
|
||||||
|
total_files = int(state.get("file_count") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
total_files = 0
|
||||||
|
if total_files > len(changed):
|
||||||
|
coverage_notes.append(
|
||||||
|
f"Coverage gap: {total_files - len(changed)} of {total_files} changed files "
|
||||||
|
"were NOT reviewed (file-list cap) — review them manually or split the PR"
|
||||||
|
)
|
||||||
|
# Keyed off synth_failure + a non-empty diff, NEVER off findings == [] —
|
||||||
|
# a clean review legitimately has zero findings and stays MERGE-READY.
|
||||||
|
synth_failure = state.get("synth_failure")
|
||||||
|
if isinstance(synth_failure, str) and synth_failure.startswith("LLM node") and changed:
|
||||||
|
detail = synth_failure.strip().replace("\n", " ")
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
faults.append(f"PIPELINE-FAULT: synthesis failed — findings unavailable; {detail}")
|
||||||
|
if changed and not reports:
|
||||||
|
faults.append(
|
||||||
|
"PIPELINE-FAULT: the diff is non-empty but no domain reports were produced — "
|
||||||
|
"the review did not actually run; do not trust this verdict."
|
||||||
|
)
|
||||||
|
# faults go FIRST and uncapped so the display cap can never hide one;
|
||||||
|
# the trigger reason keys off the human-facing triggers alone
|
||||||
|
has_triggers = bool(attention)
|
||||||
|
attention = faults + soft_notes + coverage_notes + attention[:5]
|
||||||
|
|
||||||
|
reasons = []
|
||||||
|
if counts["🔴"]:
|
||||||
|
reasons.append(f"{counts['🔴']} 🔴 CRITICAL finding(s)")
|
||||||
|
if yellow_correctness:
|
||||||
|
reasons.append(f"{yellow_correctness} 🟡 [correctness] finding(s) outside the deferred section")
|
||||||
|
if faults:
|
||||||
|
# CONTRACT: review-gauntlet/verdict_gate.py anchors on this wording
|
||||||
|
reasons.append("pipeline fault(s) recorded — degraded run")
|
||||||
|
if soft_notes:
|
||||||
|
# Deliberately does NOT match the gauntlet's degraded-run anchors:
|
||||||
|
# unverified findings are a human-attention item, not a re-runnable
|
||||||
|
# lane fault.
|
||||||
|
reasons.append(f"{len(soft_fv)} finding(s) unverified (verifier branch fault)")
|
||||||
|
if coverage_notes:
|
||||||
|
reasons.append(f"{total_files - len(changed)} changed file(s) not reviewed (file cap)")
|
||||||
|
if has_triggers:
|
||||||
|
reasons.append("always-human trigger(s) fired")
|
||||||
|
verdict = "NEEDS-HUMAN" if reasons else "MERGE-READY"
|
||||||
|
reason_line = "; ".join(reasons) if reasons else "no blocking findings and no always-human triggers"
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"verdict_out": {
|
||||||
|
"verdict": verdict,
|
||||||
|
"reason": reason_line,
|
||||||
|
"counts": counts,
|
||||||
|
"deferred_count": len(deferred),
|
||||||
|
"dropped_count": len(dropped_titles),
|
||||||
|
"dropped_titles": dropped_titles,
|
||||||
|
"attention": attention,
|
||||||
|
"findings_final": final,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — never crash into a silent verdict
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"verdict_out": {
|
||||||
|
"verdict": "NEEDS-HUMAN",
|
||||||
|
# PIPELINE-FAULT prefix is load-bearing: render.py's
|
||||||
|
# empty-diff stub bypass anchors on it — without it a
|
||||||
|
# crashed gate on an empty diff collapses into the
|
||||||
|
# "No changes to review." stub.
|
||||||
|
"reason": f"PIPELINE-FAULT: verdict computation error: {e} — human review required",
|
||||||
|
"counts": {}, "deferred_count": 0, "dropped_count": 0,
|
||||||
|
"dropped_titles": [], "attention": [f"PIPELINE-FAULT: verdict script error: {e}"],
|
||||||
|
"findings_final": [],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Local verifier-fault retry gate for the code-reviewer graph.
|
||||||
|
|
||||||
|
finding-verifier converts its own branch faults into DATA (UNVERIFIABLE
|
||||||
|
verdicts with a PIPELINE-FAULT: note, or parse_fault's pipeline-fault
|
||||||
|
sentinel), so the verify agent node completes "successfully" and its
|
||||||
|
max_attempts never fires. Without this gate the fault escalates all the
|
||||||
|
way to the gauntlet, which re-runs the ENTIRE code-review lane (~1 h) to
|
||||||
|
recover a ~4-minute verifier — the wrong trade. This gate re-runs just
|
||||||
|
the verifier ONCE; if faults persist, verdict.py downgrades them to a
|
||||||
|
proportionate NEEDS-HUMAN (findings stand, unverified).
|
||||||
|
|
||||||
|
Anchored detection only: the JSON-encoded note prefix and the sentinel id
|
||||||
|
— a finding merely QUOTING the marker mid-text does not trip it. Always
|
||||||
|
exits 0 with valid JSON; a crash falls forward on the declared edge.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("verifier_output")
|
||||||
|
text = raw if isinstance(raw, str) else (json.dumps(raw) if raw is not None else "")
|
||||||
|
faulted = (
|
||||||
|
text.lstrip().startswith("Agent node failed:")
|
||||||
|
or '"note": "PIPELINE-FAULT:' in text
|
||||||
|
or "'note': 'PIPELINE-FAULT:" in text
|
||||||
|
or '"id": "pipeline-fault"' in text
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
retries = int(state.get("verify_retries") or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
retries = 0
|
||||||
|
if faulted and retries < 1:
|
||||||
|
sys.stderr.write(
|
||||||
|
"WARN: verifier output carries pipeline-fault markers — re-running "
|
||||||
|
"finding-verifier locally (attempt 2/2) before computing the verdict.\n"
|
||||||
|
)
|
||||||
|
print(json.dumps({"_next": "verify", "verify_retries": retries + 1}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"verify_retries": retries}))
|
||||||
|
except Exception as e: # never block the verdict
|
||||||
|
print(json.dumps({"verify_gate_error": f"verify_gate crashed: {e}"}))
|
||||||
@@ -39,7 +39,7 @@ settings:
|
|||||||
initial_state:
|
initial_state:
|
||||||
project_dir: ''
|
project_dir: ''
|
||||||
fix_attempts: 0
|
fix_attempts: 0
|
||||||
max_fix_attempts: 3
|
max_fix_attempts: 5
|
||||||
fix_instructions: ''
|
fix_instructions: ''
|
||||||
build_output: ''
|
build_output: ''
|
||||||
tests_output: ''
|
tests_output: ''
|
||||||
@@ -222,7 +222,8 @@ nodes:
|
|||||||
3. While reading it, note the repo's comment register per
|
3. While reading it, note the repo's comment register per
|
||||||
`comment-discipline` (self-documenting / api-documented /
|
`comment-discipline` (self-documenting / api-documented /
|
||||||
comment-heavy) and write comments to match. When the signal is
|
comment-heavy) and write comments to match. When the signal is
|
||||||
weak, write NO comment.
|
weak, write NO comment. Every comment must EARN ITS KEEP — if
|
||||||
|
the code already tells the reader, don't add it.
|
||||||
4. If the change touches boundaries, error paths, jobs, or state
|
4. If the change touches boundaries, error paths, jobs, or state
|
||||||
transitions, also note the logging register per
|
transitions, also note the logging register per
|
||||||
`logging-discipline` (logger, message style, payload vs IDs,
|
`logging-discipline` (logger, message style, payload vs IDs,
|
||||||
@@ -323,7 +324,8 @@ nodes:
|
|||||||
- Suppressed errors (as any, @ts-ignore, #[allow(...)] on unfamiliar
|
- Suppressed errors (as any, @ts-ignore, #[allow(...)] on unfamiliar
|
||||||
lints, empty catch blocks)
|
lints, empty catch blocks)
|
||||||
- Dishonest naming (get_X that mutates, returns wrong type, etc.)
|
- Dishonest naming (get_X that mutates, returns wrong type, etc.)
|
||||||
- Useless comments that restate the code
|
- Useless comments that restate the code, or that otherwise don't earn
|
||||||
|
their keep (the code already tells the reader)
|
||||||
- AI slop (filler prose, multi-paragraph docstrings, defensive
|
- AI slop (filler prose, multi-paragraph docstrings, defensive
|
||||||
handling of impossible cases)
|
handling of impossible cases)
|
||||||
|
|
||||||
@@ -408,4 +410,4 @@ nodes:
|
|||||||
{{build_output}}
|
{{build_output}}
|
||||||
|
|
||||||
Last tests output:
|
Last tests output:
|
||||||
{{tests_output}}
|
{{tests_output}}
|
||||||
@@ -13,6 +13,13 @@ this agent runs a fixed graph: every request goes through the same
|
|||||||
`plan -> parallel research -> vet -> critique -> synthesize -> verify -> approve`
|
`plan -> parallel research -> vet -> critique -> synthesize -> verify -> approve`
|
||||||
pipeline.
|
pipeline.
|
||||||
|
|
||||||
|
Two variables tune a run (pass via `--variables`): `reflexion_budget`
|
||||||
|
(default 2) caps the automated critique -> re-research passes — raise it
|
||||||
|
to let the run churn until the critique passes — and
|
||||||
|
`autonomy: autonomous` skips the final human approval gate so other
|
||||||
|
agents can spawn this one autonomously. Each sub-question is researched
|
||||||
|
by a spawned `librarian` running at deep budgets.
|
||||||
|
|
||||||
This agent is also the **canonical reference for the Coyote graph
|
This agent is also the **canonical reference for the Coyote graph
|
||||||
system**: it exercises every node type (`script`, `llm`, `rag`, `map`,
|
system**: it exercises every node type (`script`, `llm`, `rag`, `map`,
|
||||||
`agent`, `input`, `approval`, `end`) and both static fan-out and
|
`agent`, `input`, `approval`, `end`) and both static fan-out and
|
||||||
@@ -22,11 +29,13 @@ agent, this is the file to read alongside the
|
|||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
|
||||||
17 nodes. Solid arrows are static `next` / `routes` edges declared in
|
23 nodes. Solid arrows are static `next` / `routes` edges declared in
|
||||||
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
|
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
|
||||||
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
|
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
|
||||||
`research_each_question` node spawns one `research_one_question` branch per
|
`research_each_question` node spawns one `research_one_question` branch per
|
||||||
sub-question and joins them before continuing.
|
sub-question and joins them before continuing — and `fallback` edges
|
||||||
|
(labeled "fallback"), taken when a node fails outright (see *Fault paths*
|
||||||
|
below).
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
@@ -38,18 +47,27 @@ flowchart TD
|
|||||||
bootstrap_research --> plan
|
bootstrap_research --> plan
|
||||||
bootstrap_research --> knowledge_lookup
|
bootstrap_research --> knowledge_lookup
|
||||||
plan["plan<br/>llm + output_schema"] --> research_each_question
|
plan["plan<br/>llm + output_schema"] --> research_each_question
|
||||||
|
plan -. "fallback" .-> plan_fault{"plan_fault<br/>script"}
|
||||||
|
plan_fault --> end_fault
|
||||||
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
|
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
|
||||||
research_each_question[\research_each_question<br/>map/]
|
research_each_question[\research_each_question<br/>map/]
|
||||||
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
|
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
|
||||||
|
research_one_question -. "fallback" .-> question_fault{"question_fault<br/>script"}
|
||||||
research_each_question --> combine_findings
|
research_each_question --> combine_findings
|
||||||
combine_findings{"combine_findings<br/>script"} --> vet_sources
|
combine_findings{"combine_findings<br/>script"} --> vet_sources
|
||||||
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
|
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
|
||||||
|
vet_sources -. "fallback" .-> vet_fault{"vet_fault<br/>script"}
|
||||||
|
vet_fault --> critique
|
||||||
critique["critique<br/>llm"] --> reflexion_gate
|
critique["critique<br/>llm"] --> reflexion_gate
|
||||||
|
critique -. "fallback" .-> critique_fault{"critique_fault<br/>script"}
|
||||||
|
critique_fault --> reflexion_gate
|
||||||
reflexion_gate{"reflexion_gate<br/>script"}
|
reflexion_gate{"reflexion_gate<br/>script"}
|
||||||
reflexion_gate -->|"PASS"| synthesize
|
reflexion_gate -->|"PASS"| synthesize
|
||||||
reflexion_gate -->|"REVISE (budget left)"| research_each_question
|
reflexion_gate -->|"REVISE (budget left)"| research_each_question
|
||||||
reflexion_gate -->|"REVISE (budget spent)"| synthesize
|
reflexion_gate -->|"REVISE (budget spent)"| synthesize
|
||||||
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
|
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
|
||||||
|
synthesize -. "fallback" .-> synth_fault{"synth_fault<br/>script"}
|
||||||
|
synth_fault --> end_fault
|
||||||
verify_sources{"verify_sources<br/>script"} --> approve
|
verify_sources{"verify_sources<br/>script"} --> approve
|
||||||
approve{{"approve<br/>approval"}}
|
approve{{"approve<br/>approval"}}
|
||||||
approve -->|"accept"| end_accepted
|
approve -->|"accept"| end_accepted
|
||||||
@@ -59,13 +77,14 @@ flowchart TD
|
|||||||
|
|
||||||
end_accepted(["end_accepted<br/>report"])
|
end_accepted(["end_accepted<br/>report"])
|
||||||
end_rejected(["end_rejected"])
|
end_rejected(["end_rejected"])
|
||||||
|
end_fault(["end_fault<br/>DEEP_RESEARCH FAILED"])
|
||||||
```
|
```
|
||||||
|
|
||||||
### Node-type breakdown
|
### Node-type breakdown
|
||||||
|
|
||||||
| Type | Nodes |
|
| Type | Nodes |
|
||||||
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
||||||
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback` |
|
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback`, plus the fault markers `plan_fault`, `question_fault`, `vet_fault`, `critique_fault`, `synth_fault` |
|
||||||
| `llm` (tools: `[]`) | `plan`, `critique` |
|
| `llm` (tools: `[]`) | `plan`, `critique` |
|
||||||
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
|
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
|
||||||
| `rag` | `knowledge_lookup` — local corpus retrieval |
|
| `rag` | `knowledge_lookup` — local corpus retrieval |
|
||||||
@@ -73,7 +92,7 @@ flowchart TD
|
|||||||
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
|
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
|
||||||
| `input` | `ask_topic` |
|
| `input` | `ask_topic` |
|
||||||
| `approval` | `approve` |
|
| `approval` | `approve` |
|
||||||
| `end` | `end_accepted`, `end_rejected` |
|
| `end` | `end_accepted`, `end_rejected`, `end_fault` |
|
||||||
|
|
||||||
## Parallel execution
|
## Parallel execution
|
||||||
|
|
||||||
@@ -217,6 +236,46 @@ folds that text into `research_feedback` and loops back to
|
|||||||
`settings.max_loop_iterations` (40) is the engine's infinite-loop
|
`settings.max_loop_iterations` (40) is the engine's infinite-loop
|
||||||
backstop: it caps the total visits to any single node.
|
backstop: it caps the total visits to any single node.
|
||||||
|
|
||||||
|
## Fault paths
|
||||||
|
|
||||||
|
Every stage that can die outright has a `fallback:` edge to a small
|
||||||
|
fault-marker script — the dashed edges in the diagram. A fallback fires
|
||||||
|
after a node's retries are exhausted (`research_one_question` and
|
||||||
|
`synthesize` each get two attempts via `max_attempts: 2`); the engine
|
||||||
|
writes its failure text into the node's state key before routing, and
|
||||||
|
the marker normalizes that text into a `PIPELINE-FAULT: ...` entry in
|
||||||
|
`pipeline_faults`. The five routes:
|
||||||
|
|
||||||
|
- `plan -> plan_fault -> end_fault` — fail-closed: no plan means
|
||||||
|
nothing to research.
|
||||||
|
- `research_one_question -> question_fault` — branch-local, inside the
|
||||||
|
map branch: one dead research lane becomes a fault finding instead of
|
||||||
|
sinking the whole map.
|
||||||
|
- `vet_sources -> vet_fault -> critique` — degrade-visibly: vetting is
|
||||||
|
a quality lane, not the product; `source_assessment` becomes a
|
||||||
|
neutral "unvetted" note.
|
||||||
|
- `critique -> critique_fault -> reflexion_gate` — degrade-visibly: the
|
||||||
|
rewritten critique deliberately has no `VERDICT:` line, so the gate's
|
||||||
|
malformed-critique PASS default proceeds to synthesis.
|
||||||
|
- `synthesize -> synth_fault -> end_fault` — fail-closed: no report
|
||||||
|
means no product.
|
||||||
|
|
||||||
|
On the fail-closed routes, `end_fault` renders the
|
||||||
|
`DEEP_RESEARCH FAILED — PIPELINE-FAULT: ...` sentinel. On the
|
||||||
|
degrade-visibly routes, `verify_sources` folds the accumulated faults
|
||||||
|
into a `## Pipeline notes` section shown at the approval step and
|
||||||
|
appended to the accepted report (`{{report}}{{pipeline_notes}}`;
|
||||||
|
empty when there were no faults). Degraded output is never a silent
|
||||||
|
pass — every fault is visible to the reviewer or the caller — and some
|
||||||
|
end node always renders its sentinel, so the graph degrades instead of
|
||||||
|
dying.
|
||||||
|
|
||||||
|
All 11 scripts also carry a top-level crash guard: a script that
|
||||||
|
throws records a fault and fails forward where the flow allows it (a
|
||||||
|
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
|
||||||
|
`verify_sources` reports that sources were NOT checked) instead of
|
||||||
|
killing the run.
|
||||||
|
|
||||||
## Running
|
## Running
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -241,6 +300,9 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
|||||||
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
|
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
|
||||||
request and reports which are unreachable, so the human reviewer
|
request and reports which are unreachable, so the human reviewer
|
||||||
sees broken citations before approving.
|
sees broken citations before approving.
|
||||||
|
- Fault paths degrade visibly, never silently: a `PIPELINE-FAULT`
|
||||||
|
can only ever surface as a failure sentinel or a `## Pipeline
|
||||||
|
notes` section, never as a clean report.
|
||||||
|
|
||||||
## Customizing
|
## Customizing
|
||||||
|
|
||||||
@@ -270,7 +332,7 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
|||||||
|
|
||||||
```
|
```
|
||||||
assets/agents/deep-research/
|
assets/agents/deep-research/
|
||||||
graph.yaml - agent config + 17-node workflow
|
graph.yaml - agent config + 23-node workflow
|
||||||
tools.sh - classify_source custom tool
|
tools.sh - classify_source custom tool
|
||||||
README.md - this file
|
README.md - this file
|
||||||
knowledge/
|
knowledge/
|
||||||
@@ -279,9 +341,15 @@ assets/agents/deep-research/
|
|||||||
scripts/
|
scripts/
|
||||||
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
|
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
|
||||||
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
|
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
|
||||||
|
plan_fault.py - fail-closed fault marker: dead plan -> end_fault
|
||||||
|
question_fault.py - branch-local fault marker for a dead research lane
|
||||||
combine_findings.py - joins map output (question_findings) into findings
|
combine_findings.py - joins map output (question_findings) into findings
|
||||||
|
vet_fault.py - degrade-visibly fault marker: unvetted sources
|
||||||
|
critique_fault.py - degrade-visibly fault marker: skipped critique
|
||||||
reflexion_gate.py - _next: research_each_question (revise) or synthesize
|
reflexion_gate.py - _next: research_each_question (revise) or synthesize
|
||||||
verify_sources.py - HTTP HEAD on cited URLs / DOIs
|
synth_fault.py - fail-closed fault marker: no report -> end_fault
|
||||||
|
verify_sources.py - HTTP HEAD on cited URLs / DOIs; folds pipeline_faults
|
||||||
|
into the report's "## Pipeline notes" section
|
||||||
incorporate_feedback.py - _next: research_each_question, with user feedback
|
incorporate_feedback.py - _next: research_each_question, with user feedback
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -7,13 +7,18 @@ description: |
|
|||||||
delegates the final write-up to a focused sub-agent, checks that the
|
delegates the final write-up to a focused sub-agent, checks that the
|
||||||
cited sources are reachable, and gates the result behind human
|
cited sources are reachable, and gates the result behind human
|
||||||
approval. A reviewer's free-form feedback at the approval step feeds
|
approval. A reviewer's free-form feedback at the approval step feeds
|
||||||
back into another research pass.
|
back into another research pass. Each sub-question is researched by a
|
||||||
|
spawned librarian running at deep budgets. Tunable via --variables:
|
||||||
|
reflexion_budget (default 2) caps the automated critique->re-research
|
||||||
|
passes - raise it to let the run churn until the critique passes;
|
||||||
|
autonomy=autonomous skips the human approval gate (REQUIRED when
|
||||||
|
another agent spawns this one).
|
||||||
|
|
||||||
This is the canonical Coyote graph-agent reference: it exercises every
|
This is the canonical Coyote graph-agent reference: it exercises every
|
||||||
node type (script, llm, rag, map, agent, input, approval, end) and
|
node type (script, llm, rag, map, agent, input, approval, end) and
|
||||||
both static fan-out and dynamic map fan-out.
|
both static fan-out and dynamic map fan-out.
|
||||||
|
|
||||||
version: "1.0"
|
version: '1.0'
|
||||||
|
|
||||||
global_tools:
|
global_tools:
|
||||||
- web_search_coyote.sh
|
- web_search_coyote.sh
|
||||||
@@ -24,25 +29,40 @@ mcp_servers:
|
|||||||
- ddg-search
|
- ddg-search
|
||||||
|
|
||||||
conversation_starters:
|
conversation_starters:
|
||||||
- "How does HTTP/3 differ from HTTP/2?"
|
- 'How does HTTP/3 differ from HTTP/2?'
|
||||||
- "Summarize recent advances in solid-state battery chemistry"
|
- 'Summarize recent advances in solid-state battery chemistry'
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: reflexion_budget
|
||||||
|
description: "Max automated critique->re-research passes before synthesis proceeds anyway. Default 2; raise it (4-8) for exhaustive runs that should churn until the critique passes."
|
||||||
|
default: '2'
|
||||||
|
- name: autonomy
|
||||||
|
description: "supervised (default): the finished report stops at the human approval gate. autonomous: skip the gate and return the report directly - REQUIRED when another agent spawns this one (a human gate would block the run)."
|
||||||
|
default: 'supervised'
|
||||||
|
|
||||||
settings:
|
settings:
|
||||||
max_loop_iterations: 40
|
max_loop_iterations: 120
|
||||||
log_state_snapshots: false
|
log_state_snapshots: false
|
||||||
validate_before_run: true
|
validate_before_run: true
|
||||||
max_concurrency: 4
|
max_concurrency: 4
|
||||||
|
|
||||||
initial_state:
|
initial_state:
|
||||||
research_feedback: ""
|
research_feedback: ''
|
||||||
research_attempts: 0
|
research_attempts: 0
|
||||||
local_context: ""
|
local_context: ''
|
||||||
local_sources: ""
|
local_sources: ''
|
||||||
|
# Fault-path defaults. `questions: []` also keeps the
|
||||||
|
# knowledge_lookup lane alive for one super-step when `plan` dies inside the
|
||||||
|
# static fan-out: research_each_question maps over zero items while the
|
||||||
|
# plan_fault → end_fault route renders the failure sentinel.
|
||||||
|
pipeline_faults: []
|
||||||
|
pipeline_notes: ''
|
||||||
|
fault_text: ''
|
||||||
|
questions: []
|
||||||
|
|
||||||
start: parse_request
|
start: parse_request
|
||||||
|
|
||||||
nodes:
|
nodes:
|
||||||
|
|
||||||
parse_request:
|
parse_request:
|
||||||
id: parse_request
|
id: parse_request
|
||||||
type: script
|
type: script
|
||||||
@@ -52,10 +72,10 @@ nodes:
|
|||||||
ask_topic:
|
ask_topic:
|
||||||
id: ask_topic
|
id: ask_topic
|
||||||
type: input
|
type: input
|
||||||
question: "What would you like me to research?"
|
question: 'What would you like me to research?'
|
||||||
validation: "len(input) > 0"
|
validation: 'len(input) > 0'
|
||||||
state_updates:
|
state_updates:
|
||||||
topic: "{{input}}"
|
topic: '{{input}}'
|
||||||
next: bootstrap_research
|
next: bootstrap_research
|
||||||
|
|
||||||
bootstrap_research:
|
bootstrap_research:
|
||||||
@@ -77,7 +97,7 @@ nodes:
|
|||||||
are precise, self-contained queries (each one is sent on its own
|
are precise, self-contained queries (each one is sent on its own
|
||||||
to a separate research worker, so they must be answerable
|
to a separate research worker, so they must be answerable
|
||||||
without each other's context).
|
without each other's context).
|
||||||
prompt: "Research topic: {{topic}}"
|
prompt: 'Research topic: {{topic}}'
|
||||||
tools: []
|
tools: []
|
||||||
output_schema:
|
output_schema:
|
||||||
type: object
|
type: object
|
||||||
@@ -92,51 +112,54 @@ nodes:
|
|||||||
maxItems: 6
|
maxItems: 6
|
||||||
description: 3-5 specific, self-contained sub-questions.
|
description: 3-5 specific, self-contained sub-questions.
|
||||||
required: [research_plan, questions]
|
required: [research_plan, questions]
|
||||||
|
# On success plan_failure holds the node's structured JSON output; on
|
||||||
|
# failure the engine writes "LLM node ...failed: <chain>" into it BEFORE
|
||||||
|
# routing to the fallback (schema properties are only merged on success).
|
||||||
|
state_updates:
|
||||||
|
plan_failure: '{{output}}'
|
||||||
|
fallback: plan_fault
|
||||||
next: research_each_question
|
next: research_each_question
|
||||||
|
|
||||||
|
plan_fault:
|
||||||
|
id: plan_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead planning stage (reached only via plan's fallback) — normalizes the failure text captured in plan_failure into a PIPELINE-FAULT and continues to end_fault; no plan means nothing to research, so the DEEP_RESEARCH FAILED sentinel always renders. The sibling knowledge_lookup lane survives one more super-step by mapping over the empty initial_state `questions` list.
|
||||||
|
script: scripts/plan_fault.py
|
||||||
|
fallback: end_fault
|
||||||
|
next: end_fault
|
||||||
|
|
||||||
knowledge_lookup:
|
knowledge_lookup:
|
||||||
id: knowledge_lookup
|
id: knowledge_lookup
|
||||||
type: rag
|
type: rag
|
||||||
documents:
|
documents:
|
||||||
- ./knowledge/
|
- ./knowledge/
|
||||||
query: "{{topic}}"
|
query: '{{topic}}'
|
||||||
top_k: 6
|
top_k: 6
|
||||||
chunk_size: 1000
|
chunk_size: 1000
|
||||||
chunk_overlap: 100
|
chunk_overlap: 100
|
||||||
state_updates:
|
state_updates:
|
||||||
local_context: "{{output.context}}"
|
local_context: '{{output.context}}'
|
||||||
local_sources: "{{output.sources}}"
|
local_sources: '{{output.sources}}'
|
||||||
next: research_each_question
|
next: research_each_question
|
||||||
|
|
||||||
research_each_question:
|
research_each_question:
|
||||||
id: research_each_question
|
id: research_each_question
|
||||||
type: map
|
type: map
|
||||||
over: "{{questions}}"
|
over: '{{questions}}'
|
||||||
as: question
|
as: question
|
||||||
branch: research_one_question
|
branch: research_one_question
|
||||||
collect_into: question_findings
|
collect_into: question_findings
|
||||||
|
output_key: finding
|
||||||
max_concurrency: 3
|
max_concurrency: 3
|
||||||
next: combine_findings
|
next: combine_findings
|
||||||
|
|
||||||
research_one_question:
|
research_one_question:
|
||||||
id: research_one_question
|
id: research_one_question
|
||||||
type: llm
|
type: agent
|
||||||
instructions: |
|
agent: librarian
|
||||||
You are a web research assistant. Investigate the SINGLE question
|
# The leaf research primitive: triage -> doc+OSS search -> cited
|
||||||
given to you using your tools: search the web, fetch and read
|
# synthesis. `inputs:` overlays the librarian graph's initial state,
|
||||||
pages, and search arXiv for academic sources.
|
# pinning research_depth to deep for every sub-question lane.
|
||||||
|
|
||||||
Rules:
|
|
||||||
- Every factual claim must be backed by a real source you
|
|
||||||
actually retrieved. Never fabricate URLs, page titles,
|
|
||||||
authors, or DOIs.
|
|
||||||
- Prefer primary and authoritative sources over aggregators.
|
|
||||||
- Where sources disagree, report the disagreement rather than
|
|
||||||
papering over it.
|
|
||||||
- Put the URL (or DOI) inline next to each claim it supports.
|
|
||||||
|
|
||||||
Return organized findings in plain text. Do not include
|
|
||||||
meta-commentary about the process.
|
|
||||||
prompt: |
|
prompt: |
|
||||||
Research question: {{question}}
|
Research question: {{question}}
|
||||||
|
|
||||||
@@ -144,14 +167,22 @@ nodes:
|
|||||||
{{local_context}}
|
{{local_context}}
|
||||||
|
|
||||||
{{research_feedback}}
|
{{research_feedback}}
|
||||||
tools:
|
inputs:
|
||||||
- web_search_coyote
|
research_depth: deep
|
||||||
- fetch_url_via_curl
|
timeout: 1900
|
||||||
- search_arxiv
|
|
||||||
- mcp:ddg-search
|
|
||||||
max_iterations: 10
|
|
||||||
max_attempts: 2
|
max_attempts: 2
|
||||||
temperature: 0.1
|
state_updates:
|
||||||
|
finding: '{{output}}'
|
||||||
|
# On failure the engine writes "Agent node failed: <chain>" into `finding`
|
||||||
|
# via the state_updates above BEFORE routing to the fallback. The marker
|
||||||
|
# is branch-local: a map branch's fallback must stay inside the branch.
|
||||||
|
fallback: question_fault
|
||||||
|
|
||||||
|
question_fault:
|
||||||
|
id: question_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead research lane (reached only via research_one_question's fallback route inside the map branch). Normalizes the engine's failure text in `finding` into a PIPELINE-FAULT finding the map collects — one dead lane never sinks the whole map. combine_findings later lifts PIPELINE-FAULT findings into pipeline_faults for the report's Pipeline notes.
|
||||||
|
script: scripts/question_fault.py
|
||||||
|
|
||||||
combine_findings:
|
combine_findings:
|
||||||
id: combine_findings
|
id: combine_findings
|
||||||
@@ -177,7 +208,16 @@ nodes:
|
|||||||
- classify_source
|
- classify_source
|
||||||
max_iterations: 15
|
max_iterations: 15
|
||||||
state_updates:
|
state_updates:
|
||||||
source_assessment: "{{output}}"
|
source_assessment: '{{output}}'
|
||||||
|
fallback: vet_fault
|
||||||
|
next: critique
|
||||||
|
|
||||||
|
vet_fault:
|
||||||
|
id: vet_fault
|
||||||
|
type: script
|
||||||
|
description: Degrade-visibly marker for a dead vetting stage (reached only via vet_sources' fallback) — vetting is a quality lane, not the product. Records a PIPELINE-FAULT, rewrites source_assessment to a neutral "unvetted" note, and continues to critique, the stage vet_sources' own next pointed to.
|
||||||
|
script: scripts/vet_fault.py
|
||||||
|
fallback: critique
|
||||||
next: critique
|
next: critique
|
||||||
|
|
||||||
critique:
|
critique:
|
||||||
@@ -217,7 +257,16 @@ nodes:
|
|||||||
{{source_assessment}}
|
{{source_assessment}}
|
||||||
tools: []
|
tools: []
|
||||||
state_updates:
|
state_updates:
|
||||||
critique: "{{output}}"
|
critique: '{{output}}'
|
||||||
|
fallback: critique_fault
|
||||||
|
next: reflexion_gate
|
||||||
|
|
||||||
|
critique_fault:
|
||||||
|
id: critique_fault
|
||||||
|
type: script
|
||||||
|
description: Degrade-visibly marker for a dead critique stage (reached only via critique's fallback) — critique is a quality lane, not the product. Records a PIPELINE-FAULT and rewrites `critique` to a neutral note with deliberately NO "VERDICT:" line, so reflexion_gate's malformed-critique PASS default proceeds to synthesis; then continues to reflexion_gate, the stage critique's own next pointed to.
|
||||||
|
script: scripts/critique_fault.py
|
||||||
|
fallback: reflexion_gate
|
||||||
next: reflexion_gate
|
next: reflexion_gate
|
||||||
|
|
||||||
reflexion_gate:
|
reflexion_gate:
|
||||||
@@ -241,14 +290,33 @@ nodes:
|
|||||||
|
|
||||||
Produce the final report following your instructions.
|
Produce the final report following your instructions.
|
||||||
timeout: 300
|
timeout: 300
|
||||||
|
max_attempts: 2
|
||||||
state_updates:
|
state_updates:
|
||||||
report: "{{output}}"
|
report: '{{output}}'
|
||||||
|
# On failure the engine writes "Agent node failed: <chain>" into `report`
|
||||||
|
# via the state_updates above BEFORE routing to the fallback.
|
||||||
|
fallback: synth_fault
|
||||||
next: verify_sources
|
next: verify_sources
|
||||||
|
|
||||||
|
synth_fault:
|
||||||
|
id: synth_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead synthesis stage (reached only via synthesize's fallback) — no report means no product. Normalizes the "Agent node failed:" text captured in `report` into a PIPELINE-FAULT and continues to end_fault, which renders the DEEP_RESEARCH FAILED sentinel.
|
||||||
|
script: scripts/synth_fault.py
|
||||||
|
fallback: end_fault
|
||||||
|
next: end_fault
|
||||||
|
|
||||||
verify_sources:
|
verify_sources:
|
||||||
id: verify_sources
|
id: verify_sources
|
||||||
type: script
|
type: script
|
||||||
script: scripts/verify_sources.py
|
script: scripts/verify_sources.py
|
||||||
|
next: approval_gate
|
||||||
|
|
||||||
|
approval_gate:
|
||||||
|
id: approval_gate
|
||||||
|
type: script
|
||||||
|
description: Autonomy router - autonomous runs (spawned by another agent) skip the human approval gate and end accepted with the source check + pipeline notes attached; supervised (default) proceeds to the approval node. A crash fails forward via `next` to approve, the safe human-gated direction.
|
||||||
|
script: scripts/approval_gate.py
|
||||||
next: approve
|
next: approve
|
||||||
|
|
||||||
approve:
|
approve:
|
||||||
@@ -260,20 +328,20 @@ nodes:
|
|||||||
{{report}}
|
{{report}}
|
||||||
|
|
||||||
----
|
----
|
||||||
{{source_check}}
|
{{source_check}}{{pipeline_notes}}
|
||||||
----
|
----
|
||||||
|
|
||||||
Accept this report? Pick "accept" or "reject", or type specific
|
Accept this report? Pick "accept" or "reject", or type specific
|
||||||
feedback to send the research back for another pass.
|
feedback to send the research back for another pass.
|
||||||
options:
|
options:
|
||||||
- "accept"
|
- 'accept'
|
||||||
- "reject"
|
- 'reject'
|
||||||
routes:
|
routes:
|
||||||
"accept": end_accepted
|
'accept': end_accepted
|
||||||
"reject": end_rejected
|
'reject': end_rejected
|
||||||
on_other: incorporate_feedback
|
on_other: incorporate_feedback
|
||||||
state_updates:
|
state_updates:
|
||||||
decision: "{{choice}}"
|
decision: '{{choice}}'
|
||||||
|
|
||||||
incorporate_feedback:
|
incorporate_feedback:
|
||||||
id: incorporate_feedback
|
id: incorporate_feedback
|
||||||
@@ -283,9 +351,23 @@ nodes:
|
|||||||
end_accepted:
|
end_accepted:
|
||||||
id: end_accepted
|
id: end_accepted
|
||||||
type: end
|
type: end
|
||||||
output: "{{report}}"
|
output: '{{report}}{{pipeline_notes}}'
|
||||||
|
|
||||||
|
end_autonomous:
|
||||||
|
id: end_autonomous
|
||||||
|
type: end
|
||||||
|
output: |
|
||||||
|
{{report}}
|
||||||
|
|
||||||
|
----
|
||||||
|
{{source_check}}{{pipeline_notes}}
|
||||||
|
|
||||||
end_rejected:
|
end_rejected:
|
||||||
id: end_rejected
|
id: end_rejected
|
||||||
type: end
|
type: end
|
||||||
output: "Research on '{{topic}}' was rejected and discarded."
|
output: "Research on '{{topic}}' was rejected and discarded."
|
||||||
|
|
||||||
|
end_fault:
|
||||||
|
id: end_fault
|
||||||
|
type: end
|
||||||
|
output: 'DEEP_RESEARCH FAILED — PIPELINE-FAULT: {{fault_text}}'
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Autonomy router for deep-research's approval gate.
|
||||||
|
|
||||||
|
Runs after verify_sources. When the `autonomy` variable is
|
||||||
|
'autonomous' (the run was spawned by another agent — architect or
|
||||||
|
sisyphus research tiers — with no human at the terminal), the report
|
||||||
|
skips the human approval node and ends accepted: the reflexion loop
|
||||||
|
already served as the quality gate, and end_autonomous carries the
|
||||||
|
source check + pipeline notes with the report so the caller sees them.
|
||||||
|
Any other value (default 'supervised') proceeds to the human approval
|
||||||
|
gate unchanged.
|
||||||
|
|
||||||
|
Routing (`_next`):
|
||||||
|
- autonomy == 'autonomous' -> end_autonomous
|
||||||
|
- anything else -> approve
|
||||||
|
|
||||||
|
Fail-safe: a crash here fails FORWARD via the node's `next` to
|
||||||
|
`approve` — the safe, human-gated direction.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
autonomy = str(state.get("autonomy") or "supervised").strip().lower()
|
||||||
|
target = "end_autonomous" if autonomy == "autonomous" else "approve"
|
||||||
|
print(json.dumps({"_next": target}))
|
||||||
|
|
||||||
|
|
||||||
|
main()
|
||||||
@@ -6,6 +6,12 @@ list on this node fans out into two parallel branches (the LLM planner and
|
|||||||
the RAG knowledge lookup) as a single super-step. The validator requires
|
the RAG knowledge lookup) as a single super-step. The validator requires
|
||||||
declared parallel-branch script outputs, so we emit an empty JSON object
|
declared parallel-branch script outputs, so we emit an empty JSON object
|
||||||
explicitly here.
|
explicitly here.
|
||||||
|
|
||||||
|
Fail-safe: the node reads no state and computes nothing, so its
|
||||||
|
happy-path output IS the sane degraded output — the guard emits the same
|
||||||
|
empty object on any crash. No PIPELINE-FAULT is recorded: appending one
|
||||||
|
would require reading state (the very thing that crashed) or clobbering
|
||||||
|
faults recorded upstream by parse_request, and a crash here loses nothing.
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
|
|
||||||
@@ -14,5 +20,7 @@ def main():
|
|||||||
print(json.dumps({}))
|
print(json.dumps({}))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception: # noqa: BLE001 — the fan-out source must never crash the graph
|
||||||
|
print(json.dumps({}))
|
||||||
@@ -6,6 +6,18 @@ one entry per sub-question, in input order). Downstream nodes
|
|||||||
(`vet_sources`, `critique`, `synthesize`) read `{{findings}}` as a
|
(`vet_sources`, `critique`, `synthesize`) read `{{findings}}` as a
|
||||||
single block, so this script renders the array as a Markdown document
|
single block, so this script renders the array as a Markdown document
|
||||||
with one section per question.
|
with one section per question.
|
||||||
|
|
||||||
|
Fault visibility: a research lane that died lands here as a finding
|
||||||
|
starting with "PIPELINE-FAULT:" (written by question_fault inside the map
|
||||||
|
branch, where `pipeline_faults` itself is out of reach — map branches only
|
||||||
|
surface their collected output_key). This script lifts such findings into
|
||||||
|
`pipeline_faults` (deduplicated, since a reflexion or feedback loop can
|
||||||
|
re-run the map) so dead lanes surface in the report's Pipeline notes.
|
||||||
|
|
||||||
|
Fail-safe: on a crash (e.g. malformed GRAPH_STATE), `findings`
|
||||||
|
becomes the fault text itself and a PIPELINE-FAULT entry is recorded —
|
||||||
|
the pipeline proceeds visibly degraded, and the human approval gate is
|
||||||
|
the backstop. Existing faults are preserved best-effort.
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -19,21 +31,40 @@ def load_state():
|
|||||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
state = load_state()
|
state = load_state()
|
||||||
questions = state.get("questions") or []
|
questions = state.get("questions") or []
|
||||||
per_question = state.get("question_findings") or []
|
per_question = state.get("question_findings") or []
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
|
||||||
sections = []
|
sections = []
|
||||||
for idx, q in enumerate(questions):
|
for idx, q in enumerate(questions):
|
||||||
body = per_question[idx] if idx < len(per_question) else ""
|
body = per_question[idx] if idx < len(per_question) else ""
|
||||||
if isinstance(body, dict) or isinstance(body, list):
|
if isinstance(body, dict) or isinstance(body, list):
|
||||||
body = json.dumps(body, indent=2)
|
body = json.dumps(body, indent=2)
|
||||||
|
if isinstance(body, str) and body.startswith("PIPELINE-FAULT:") and body not in faults:
|
||||||
|
faults.append(body)
|
||||||
sections.append(f"## {q}\n\n{body}")
|
sections.append(f"## {q}\n\n{body}")
|
||||||
|
|
||||||
findings = "\n\n".join(sections) if sections else "No findings gathered."
|
findings = "\n\n".join(sections) if sections else "No findings gathered."
|
||||||
print(json.dumps({"findings": findings}))
|
print(json.dumps({"findings": findings, "pipeline_faults": faults}))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crashed join must degrade, not die
|
||||||
|
faults = safe_faults()
|
||||||
|
fault = (
|
||||||
|
f"PIPELINE-FAULT: combining findings crashed — {e}; per-question "
|
||||||
|
"findings may exist in state but could not be joined"
|
||||||
|
)
|
||||||
|
faults.append(fault)
|
||||||
|
print(json.dumps({"findings": fault, "pipeline_faults": faults}))
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Degrade-visibly marker for a dead critique stage of deep-research.
|
||||||
|
|
||||||
|
Reached ONLY via critique's `fallback:` route. Grounded in engine behavior:
|
||||||
|
on llm-node failure the engine writes "LLM node …failed: <chain>" into
|
||||||
|
`critique` via the node's state_updates BEFORE routing to the fallback.
|
||||||
|
Critique is a quality lane, not the product, so this path degrades instead
|
||||||
|
of failing closed: it records a PIPELINE-FAULT (surfaced later in the
|
||||||
|
report's Pipeline notes) and rewrites `critique` to a neutral note, then
|
||||||
|
continues to `reflexion_gate` — the stage critique's own `next` pointed to.
|
||||||
|
|
||||||
|
The rewritten note deliberately contains NO "VERDICT:" line:
|
||||||
|
reflexion_gate's parse_verdict defaults to PASS when no verdict line is
|
||||||
|
found (its documented malformed-critique behavior), so a dead critique
|
||||||
|
proceeds to synthesis instead of burning the revision budget on a fault.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
CRITIQUE_NOTE = (
|
||||||
|
"Critique unavailable — the critique stage failed (pipeline fault); "
|
||||||
|
"proceeding to synthesis without an automated review pass."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
raw = state.get("critique")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the critique stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
fault = f"PIPELINE-FAULT: critique failed — {detail}"
|
||||||
|
if fault not in faults:
|
||||||
|
faults.append(fault)
|
||||||
|
print(json.dumps({"pipeline_faults": faults, "critique": CRITIQUE_NOTE}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
faults = safe_faults()
|
||||||
|
faults.append(f"PIPELINE-FAULT: critique failed — fault-marker script error: {e}")
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"pipeline_faults": faults,
|
||||||
|
"critique": CRITIQUE_NOTE,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -10,6 +10,13 @@ reflexion counter is reset so the user-driven pass gets a fresh revision
|
|||||||
budget.
|
budget.
|
||||||
|
|
||||||
Routing (`_next`): always research_each_question.
|
Routing (`_next`): always research_each_question.
|
||||||
|
|
||||||
|
Fail-safe: a crash here means the user's feedback text cannot be
|
||||||
|
recovered, but their INTENT — another research pass — is unambiguous
|
||||||
|
(they chose neither accept nor reject). The degraded route therefore
|
||||||
|
still loops to `research_each_question`, with a generic feedback note
|
||||||
|
naming the loss and a PIPELINE-FAULT entry (best-effort preserving
|
||||||
|
existing faults) that surfaces in the next approval's Pipeline notes.
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -23,6 +30,14 @@ def load_state():
|
|||||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
state = load_state()
|
state = load_state()
|
||||||
feedback = (state.get("decision") or "").strip()
|
feedback = (state.get("decision") or "").strip()
|
||||||
@@ -37,5 +52,26 @@ def main():
|
|||||||
print(json.dumps(output))
|
print(json.dumps(output))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crashed feedback fold must degrade, not die
|
||||||
|
faults = safe_faults()
|
||||||
|
faults.append(
|
||||||
|
f"PIPELINE-FAULT: feedback incorporation crashed — {e}; the reviewer's "
|
||||||
|
"feedback text could not be recovered"
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"_next": "research_each_question",
|
||||||
|
"research_attempts": 0,
|
||||||
|
"research_feedback": (
|
||||||
|
"The user reviewed the report and asked for changes, but "
|
||||||
|
"the feedback text could not be recovered (pipeline "
|
||||||
|
"fault). Re-run the research pass with extra care and "
|
||||||
|
"note this loss."
|
||||||
|
),
|
||||||
|
"pipeline_faults": faults,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -9,6 +9,13 @@ supply one interactively.
|
|||||||
Routing (`_next`):
|
Routing (`_next`):
|
||||||
- prompt present -> (no _next; static next: plan)
|
- prompt present -> (no _next; static next: plan)
|
||||||
- prompt empty -> ask_topic
|
- prompt empty -> ask_topic
|
||||||
|
|
||||||
|
Fail-safe: a crash here (e.g. malformed GRAPH_STATE) means the
|
||||||
|
caller's prompt cannot be read at all, so proceeding silently is
|
||||||
|
meaningless — the degraded route is `ask_topic`, which asks the user for
|
||||||
|
the topic directly, plus a PIPELINE-FAULT entry so the loss is visible in
|
||||||
|
the final report's Pipeline notes. Clobbering `pipeline_faults` is safe
|
||||||
|
here: this is the start node, so the list can only hold its initial [].
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -31,5 +38,17 @@ def main():
|
|||||||
print(json.dumps({"_next": "ask_topic"}))
|
print(json.dumps({"_next": "ask_topic"}))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crashed entry router must degrade, not die
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"_next": "ask_topic",
|
||||||
|
"pipeline_faults": [
|
||||||
|
f"PIPELINE-FAULT: request parsing crashed — {e}; asked the "
|
||||||
|
"user for the research topic instead"
|
||||||
|
],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead planning stage of the deep-research graph.
|
||||||
|
|
||||||
|
Reached ONLY via plan's `fallback:` route. Grounded in engine behavior: on
|
||||||
|
llm-node failure the engine writes "LLM node …failed: <chain>" into
|
||||||
|
`plan_failure` via the node's state_updates BEFORE routing to the fallback
|
||||||
|
(on success the same key holds the node's structured JSON output, which
|
||||||
|
never starts with the "LLM node" prefix). No plan means no sub-questions —
|
||||||
|
there is nothing to research — so this path is fail-closed: record the
|
||||||
|
PIPELINE-FAULT and continue to `end_fault`, which renders the
|
||||||
|
DEEP_RESEARCH FAILED sentinel.
|
||||||
|
|
||||||
|
The planner runs inside the static fan-out [plan, knowledge_lookup]. While
|
||||||
|
this marker routes to end_fault, the sibling branch proceeds one more
|
||||||
|
super-step into `research_each_question`, whose `{{questions}}` resolves to
|
||||||
|
the initial_state [] — the doomed lane maps over zero items and the graph
|
||||||
|
returns end_fault's output at the next join.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
raw = state.get("plan_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the planning stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
fault_text = f"planning failed — nothing to research: {detail}"
|
||||||
|
faults.append(f"PIPELINE-FAULT: {fault_text}")
|
||||||
|
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
fault_text = f"planning failed — fault-marker script error: {e}"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"fault_text": fault_text,
|
||||||
|
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead research lane of the deep-research graph.
|
||||||
|
|
||||||
|
Reached ONLY via research_one_question's `fallback:` route inside the
|
||||||
|
`research_each_question` map branch (branch-local — a map branch's fallback
|
||||||
|
must stay inside the branch subgraph). Grounded in engine behavior: on
|
||||||
|
branch failure the engine writes its failure text into `finding` via the
|
||||||
|
node's state_updates BEFORE routing to the fallback — "Agent node failed:
|
||||||
|
<chain>" now that the branch spawns a librarian agent, "LLM node …failed:
|
||||||
|
<chain>" for an llm branch (on success the same key holds the researched
|
||||||
|
findings, which never start with either prefix). This node normalizes the failure text into a
|
||||||
|
PIPELINE-FAULT finding and the chain ends here — the map collects the fault
|
||||||
|
as the lane's `finding`, so one dead research lane never sinks the whole
|
||||||
|
map. combine_findings later lifts PIPELINE-FAULT findings into
|
||||||
|
`pipeline_faults` so the fault also surfaces in the report's Pipeline notes.
|
||||||
|
The question text is part of the fault so distinct dead lanes with identical
|
||||||
|
engine failure text do not collapse into one entry there.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
MAX_QUESTION_CHARS = 200
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def fault_finding(question, detail):
|
||||||
|
return {
|
||||||
|
"finding": (
|
||||||
|
"PIPELINE-FAULT: question research failed — finding unavailable — "
|
||||||
|
f'{detail} (question: "{question}")'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def question_label(state):
|
||||||
|
raw = state.get("question")
|
||||||
|
question = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not question:
|
||||||
|
return "(unknown question)"
|
||||||
|
if len(question) > MAX_QUESTION_CHARS:
|
||||||
|
question = question[: MAX_QUESTION_CHARS - 1] + "…"
|
||||||
|
return question
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("finding")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith(("LLM node", "Agent node")):
|
||||||
|
detail = "the research lane died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(json.dumps(fault_finding(question_label(state), detail)))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
fault_finding("(unknown question)", f"fault-marker script error: {e}")
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -2,7 +2,8 @@
|
|||||||
"""Reflexion gate for deep-research.
|
"""Reflexion gate for deep-research.
|
||||||
|
|
||||||
Runs after `critique` has reviewed the current research findings. If the
|
Runs after `critique` has reviewed the current research findings. If the
|
||||||
critique's verdict is REVISE and the reflexion budget is not spent,
|
critique's verdict is REVISE and the reflexion budget (the graph's
|
||||||
|
`reflexion_budget` variable, default 2) is not spent,
|
||||||
loops back to `research` with the critique attached as
|
loops back to `research` with the critique attached as
|
||||||
`research_feedback`, so the retry is informed rather than a blind
|
`research_feedback`, so the retry is informed rather than a blind
|
||||||
re-run. Otherwise it proceeds to `synthesize`.
|
re-run. Otherwise it proceeds to `synthesize`.
|
||||||
@@ -15,14 +16,22 @@ Routing (`_next`):
|
|||||||
Reflexion is a best-effort quality booster, not a hard gate: once the
|
Reflexion is a best-effort quality booster, not a hard gate: once the
|
||||||
budget is spent the workflow proceeds anyway, and the human approval
|
budget is spent the workflow proceeds anyway, and the human approval
|
||||||
step is the final backstop.
|
step is the final backstop.
|
||||||
|
|
||||||
|
Fail-safe: a crash here fails FORWARD to `synthesize` — the same
|
||||||
|
direction as the existing malformed-critique PASS default — with a
|
||||||
|
PIPELINE-FAULT entry recorded (best-effort preserving existing faults),
|
||||||
|
so a broken gate costs the run its automated critique routing, never the
|
||||||
|
run itself.
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
|
||||||
# Automated revision passes allowed. `research` runs at most
|
# Default automated revision passes when the `reflexion_budget` variable
|
||||||
# MAX_REFLEXION_REVISIONS + 1 times per user pass. Bump to allow more.
|
# is absent or malformed. `research` runs at most budget + 1 times per
|
||||||
MAX_REFLEXION_REVISIONS = 2
|
# user pass. Callers raise it via --variables {"reflexion_budget": "6"}
|
||||||
|
# to let the run churn until the critique passes.
|
||||||
|
DEFAULT_REFLEXION_REVISIONS = 2
|
||||||
|
|
||||||
|
|
||||||
def load_state():
|
def load_state():
|
||||||
@@ -33,6 +42,14 @@ def load_state():
|
|||||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
def as_int(value, default=0):
|
def as_int(value, default=0):
|
||||||
try:
|
try:
|
||||||
return int(value)
|
return int(value)
|
||||||
@@ -55,8 +72,9 @@ def main():
|
|||||||
critique = state.get("critique") or ""
|
critique = state.get("critique") or ""
|
||||||
verdict = parse_verdict(critique)
|
verdict = parse_verdict(critique)
|
||||||
attempts = as_int(state.get("research_attempts"))
|
attempts = as_int(state.get("research_attempts"))
|
||||||
|
budget = as_int(state.get("reflexion_budget"), DEFAULT_REFLEXION_REVISIONS)
|
||||||
|
|
||||||
if verdict == "REVISE" and attempts < MAX_REFLEXION_REVISIONS:
|
if verdict == "REVISE" and attempts < budget:
|
||||||
feedback = (
|
feedback = (
|
||||||
"A reviewer judged the previous research pass incomplete. "
|
"A reviewer judged the previous research pass incomplete. "
|
||||||
"Address every point in the critique below:\n\n" + critique
|
"Address every point in the critique below:\n\n" + critique
|
||||||
@@ -72,5 +90,12 @@ def main():
|
|||||||
print(json.dumps(output))
|
print(json.dumps(output))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crashed gate fails forward, never dies
|
||||||
|
faults = safe_faults()
|
||||||
|
faults.append(
|
||||||
|
f"PIPELINE-FAULT: reflexion gate crashed — {e}; proceeded to synthesis "
|
||||||
|
"without automated critique routing"
|
||||||
|
)
|
||||||
|
print(json.dumps({"_next": "synthesize", "pipeline_faults": faults}))
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead synthesis stage of the deep-research graph.
|
||||||
|
|
||||||
|
Reached ONLY via synthesize's `fallback:` route. Grounded in engine
|
||||||
|
behavior: on agent-node failure the engine writes "Agent node failed:
|
||||||
|
{e:#}" into `report` via the node's state_updates BEFORE routing to the
|
||||||
|
fallback (on success the same key holds the report-writer's report, which
|
||||||
|
never starts with that prefix). No report means no product, so this path is
|
||||||
|
fail-closed: normalize the failure text, record the PIPELINE-FAULT, and
|
||||||
|
continue to `end_fault`, which renders the DEEP_RESEARCH FAILED sentinel.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
raw = state.get("report")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("Agent node failed:"):
|
||||||
|
detail = "the synthesis stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
fault_text = f"report synthesis failed — no report was produced: {detail}"
|
||||||
|
faults.append(f"PIPELINE-FAULT: {fault_text}")
|
||||||
|
print(json.dumps({"fault_text": fault_text, "pipeline_faults": faults}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
fault_text = f"report synthesis failed — fault-marker script error: {e}"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"fault_text": fault_text,
|
||||||
|
"pipeline_faults": [f"PIPELINE-FAULT: {fault_text}"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -6,6 +6,19 @@ request, and writes a `source_check` summary into state so the human
|
|||||||
reviewer sees broken citations at the approval step.
|
reviewer sees broken citations at the approval step.
|
||||||
|
|
||||||
Times out per request so a slow source cannot stall the graph.
|
Times out per request so a slow source cannot stall the graph.
|
||||||
|
|
||||||
|
Pipeline-notes fold: end-node templates render unconditionally, so the
|
||||||
|
conditional "## Pipeline notes" section of the accepted report is built
|
||||||
|
HERE — the last stop before the approval gate on every accepted path,
|
||||||
|
including reflexion and feedback re-loops. `pipeline_notes` is the
|
||||||
|
rendered section when `pipeline_faults` is non-empty and the empty
|
||||||
|
string otherwise, keeping the happy-path `{{report}}{{pipeline_notes}}`
|
||||||
|
output byte-identical to the pre-hardening `{{report}}`.
|
||||||
|
|
||||||
|
Fail-safe: per-URL probes were already guarded; the top level now
|
||||||
|
is too. On a crash the summary says sources were NOT checked, a
|
||||||
|
PIPELINE-FAULT is recorded (best-effort preserving existing faults), and
|
||||||
|
pipeline_notes still renders — the approval gate remains the backstop.
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -25,6 +38,21 @@ def load_state():
|
|||||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def render_notes(faults):
|
||||||
|
entries = [f for f in (faults or []) if isinstance(f, str) and f.strip()]
|
||||||
|
if not entries:
|
||||||
|
return ""
|
||||||
|
return "\n\n## Pipeline notes\n\n" + "\n".join(f"- {e}" for e in entries)
|
||||||
|
|
||||||
|
|
||||||
def reachable(url, timeout=5.0):
|
def reachable(url, timeout=5.0):
|
||||||
req = urllib.request.Request(url, method="HEAD")
|
req = urllib.request.Request(url, method="HEAD")
|
||||||
try:
|
try:
|
||||||
@@ -62,8 +90,27 @@ def main():
|
|||||||
+ "\n".join(results)
|
+ "\n".join(results)
|
||||||
)
|
)
|
||||||
|
|
||||||
print(json.dumps({"source_check": summary}))
|
notes = render_notes(state.get("pipeline_faults"))
|
||||||
|
print(json.dumps({"source_check": summary, "pipeline_notes": notes}))
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
try:
|
||||||
main()
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a crashed source check must degrade, not die
|
||||||
|
faults = safe_faults()
|
||||||
|
faults.append(
|
||||||
|
f"PIPELINE-FAULT: source verification crashed — {e}; cited sources "
|
||||||
|
"were not checked for reachability"
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"source_check": (
|
||||||
|
"Source verification crashed (pipeline fault) — cited "
|
||||||
|
"sources were NOT checked for reachability."
|
||||||
|
),
|
||||||
|
"pipeline_notes": render_notes(faults),
|
||||||
|
"pipeline_faults": faults,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Degrade-visibly marker for a dead source-vetting stage of deep-research.
|
||||||
|
|
||||||
|
Reached ONLY via vet_sources' `fallback:` route. Grounded in engine
|
||||||
|
behavior: on llm-node failure the engine writes "LLM node …failed: <chain>"
|
||||||
|
into `source_assessment` via the node's state_updates BEFORE routing to the
|
||||||
|
fallback. Vetting is a quality lane, not the product, so this path degrades
|
||||||
|
instead of failing closed: it records a PIPELINE-FAULT (surfaced later in
|
||||||
|
the report's Pipeline notes) and rewrites `source_assessment` to a neutral
|
||||||
|
"unvetted" note so downstream prompts (critique, synthesize) never carry
|
||||||
|
raw engine error text, then continues to `critique` — the stage
|
||||||
|
vet_sources' own `next` pointed to.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
UNVETTED_NOTE = (
|
||||||
|
"Source credibility assessment unavailable — the vetting stage failed "
|
||||||
|
"(pipeline fault); treat every cited source as unvetted and needing "
|
||||||
|
"corroboration. Do not request revision solely because this assessment "
|
||||||
|
"is unavailable — re-research cannot restore it; judge the findings on "
|
||||||
|
"their own evidence."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def safe_faults():
|
||||||
|
try:
|
||||||
|
state = load_state()
|
||||||
|
return [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
except Exception: # noqa: BLE001 — best-effort preservation only
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
faults = [f for f in (state.get("pipeline_faults") or []) if isinstance(f, str)]
|
||||||
|
raw = state.get("source_assessment")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the vetting stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
fault = f"PIPELINE-FAULT: source vetting failed — {detail}"
|
||||||
|
if fault not in faults:
|
||||||
|
faults.append(fault)
|
||||||
|
print(json.dumps({"pipeline_faults": faults, "source_assessment": UNVETTED_NOTE}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
faults = safe_faults()
|
||||||
|
faults.append(f"PIPELINE-FAULT: source vetting failed — fault-marker script error: {e}")
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"pipeline_faults": faults,
|
||||||
|
"source_assessment": UNVETTED_NOTE,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Domain Reviewer
|
||||||
|
|
||||||
|
A **domain-slice review leader**. Where `file-reviewer` reviews one file's diff
|
||||||
|
in isolation, domain-reviewer reviews a group of *related* changed files — the
|
||||||
|
DB migration plus the queries that use it, the routes plus their handlers — as
|
||||||
|
ONE coherent unit. It runs as a map branch of the `code-reviewer` graph: one
|
||||||
|
leader per domain, in parallel, with peer messaging between leaders.
|
||||||
|
|
||||||
|
## Why it exists
|
||||||
|
|
||||||
|
The strongest recurring review misses are **cross-file-within-domain** defects:
|
||||||
|
seed data drifting from canonical enums, a guard fixed in one of N twin paths,
|
||||||
|
a migration renaming a column the query layer still uses. Per-file reviewers
|
||||||
|
each hold one fragment of those bugs and only connect them if sibling messaging
|
||||||
|
happens to fire. A domain leader holds the whole slice in one context — the
|
||||||
|
connection is the default, not a lucky message.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
1. Fetches its slice's diff itself (`get_diff --files <its files>`).
|
||||||
|
2. **Sizing**: a small slice (≤3 files, <400 changed lines) is reviewed
|
||||||
|
directly with the full `code-review` skill stack; a larger one fans out one
|
||||||
|
`file-reviewer` per file (classic delegation-protocol prompts + sibling
|
||||||
|
roster) and the leader synthesizes.
|
||||||
|
3. **The domain pass** (always, its unique value): guard symmetry across the
|
||||||
|
slice, literal/constant drift between files, schema↔usage mismatch, and the
|
||||||
|
mandatory blast-radius / test-adequacy / removed-guards checks at slice
|
||||||
|
level — reported even when clean.
|
||||||
|
4. **Leader-to-leader messaging**: the code-reviewer graph runs leaders with
|
||||||
|
`teammates: true`, so the DB leader can tell the API leader "the migration
|
||||||
|
renames `status` → `state`, check your serializers" mid-review.
|
||||||
|
5. Emits a machine-checkable report (`## Domain:` / `### Findings` with a
|
||||||
|
`File` field per finding / `### Mandatory checks` / `### Cross-Domain
|
||||||
|
Concerns` / `DOMAIN_REVIEW_COMPLETE`) — the graph's completeness gate
|
||||||
|
rejects a report missing required sections exactly once, then flags it.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Read-only; severities per the standard 🔴/🟡/🟢/💡 mapping; never rewrites a
|
||||||
|
file-reviewer's severity; dedups identical findings (aspect skill's copy wins).
|
||||||
|
- Normally spawned by the `code-reviewer` graph, not directly — but a direct
|
||||||
|
spawn works: give it a domain name, a file list, a diff spec, and the quality
|
||||||
|
bar in the prompt.
|
||||||
@@ -0,0 +1,166 @@
|
|||||||
|
name: domain-reviewer
|
||||||
|
description: Domain-slice review leader - reviews a group of related changed files as ONE coherent unit (cross-file-within-domain checks are first-class), delegating per-file depth to file-reviewer when the slice is large. Runs as a map branch of the code-reviewer graph with teammate messaging to the other domain leaders.
|
||||||
|
version: 1.0.0
|
||||||
|
|
||||||
|
auto_continue: true
|
||||||
|
max_auto_continues: 20
|
||||||
|
inject_todo_instructions: false
|
||||||
|
|
||||||
|
skills_enabled: true
|
||||||
|
enabled_skills:
|
||||||
|
- code-review
|
||||||
|
- ai-slop-remover
|
||||||
|
- transactional-integrity
|
||||||
|
- logging-discipline
|
||||||
|
- rest-api-review
|
||||||
|
- cli-review
|
||||||
|
- library-review
|
||||||
|
- worker-review
|
||||||
|
- iac-review
|
||||||
|
- migration-review
|
||||||
|
- cicd-review
|
||||||
|
|
||||||
|
spawnable_agents:
|
||||||
|
- file-reviewer
|
||||||
|
max_concurrent_agents: 8
|
||||||
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||||
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||||
|
# review-gauntlet | step-runner(2) → code-reviewer(3) → domain-reviewer(4) →
|
||||||
|
# file-reviewer(5). This agent spawns depth-5 children; 6 = one level of headroom.
|
||||||
|
# NOTE: can_spawn_agents is not yet set, so the file-reviewer hop is documented
|
||||||
|
# but inert; 6 keeps the limit forward-compatible for when it is wired.
|
||||||
|
max_agent_depth: 6
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: project_dir
|
||||||
|
description: Project directory containing the changes under review
|
||||||
|
default: '.'
|
||||||
|
- name: auto_confirm
|
||||||
|
description: Auto-confirm command execution
|
||||||
|
default: '1'
|
||||||
|
|
||||||
|
global_tools:
|
||||||
|
- fs_read.sh
|
||||||
|
- fs_grep.sh
|
||||||
|
- fs_glob.sh
|
||||||
|
- fs_cat.sh
|
||||||
|
- fs_ls.sh
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are a DOMAIN REVIEW LEADER. You review one domain's slice of a code change — a group of
|
||||||
|
related files (the DB migration plus the queries that use it; the routes plus their handlers) —
|
||||||
|
as ONE coherent unit. Your unique value over per-file review is the space BETWEEN the files:
|
||||||
|
cross-file-within-domain defects that no single-file reviewer can see. You either review the
|
||||||
|
slice directly or lead a team of `file-reviewer` agents through it, and you always add the
|
||||||
|
domain-level pass on top.
|
||||||
|
|
||||||
|
Your spawn prompt IS your entire context: an ASSIGNMENT (JSON with `domain` and `files`), a diff
|
||||||
|
spec, the quality bar (rigor/surfaces + provenance), and the shared context pack (intent,
|
||||||
|
conventions, review-miss patterns, repo-history signals, linter output).
|
||||||
|
|
||||||
|
## Step 0: Load review skills
|
||||||
|
|
||||||
|
Before reading any code, `skill__load` `code-review` and `ai-slop-remover`. Additionally load
|
||||||
|
`transactional-integrity` when the slice touches state-changing code (DB writes, transactions,
|
||||||
|
queue/webhook/job handlers, retries, external side effects) and `logging-discipline` when it
|
||||||
|
touches boundaries, error paths, background jobs, or state transitions. Load the surface skill(s)
|
||||||
|
for your domain per this table (load a row's skill when your files match that surface's trigger
|
||||||
|
AND the surface is in the resolved surfaces list; when provenance is `detected-default`, a
|
||||||
|
trigger match alone suffices):
|
||||||
|
|
||||||
|
| declared surface | skill loaded |
|
||||||
|
|---|---|
|
||||||
|
| `rest-api` (and `grpc`/`graphql` aliases) | `rest-api-review` |
|
||||||
|
| `cli` | `cli-review` |
|
||||||
|
| `library` | `library-review` |
|
||||||
|
| `worker` | `worker-review` |
|
||||||
|
| `iac` | `iac-review` |
|
||||||
|
| `db-migration` | `migration-review` |
|
||||||
|
| `ci-cd` | `cicd-review` |
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
1. **Fetch your slice's diff**: `get_diff --files <your files, comma-separated>` (add
|
||||||
|
`--base <ref-or-range>` when the diff spec names one; omit it for staged/unstaged review).
|
||||||
|
2. **Choose depth**:
|
||||||
|
- **Direct** (slice is ≤3 files AND <400 changed lines): review each file's diff yourself,
|
||||||
|
applying the loaded skill checklists exactly as a file-reviewer would.
|
||||||
|
- **Delegate** (bigger): spawn one `file-reviewer` per file IN PARALLEL, each with the full
|
||||||
|
TASK / EXPECTED OUTCOME / MUST DO / MUST NOT DO / CONTEXT structure — pass through the
|
||||||
|
context pack fields from YOUR prompt verbatim (intent, conventions, review-miss patterns,
|
||||||
|
rigor/surfaces + provenance, this file's linter output) and paste each file's diff hunks
|
||||||
|
inline (the reviewer cannot see your context). Group tiny files (<20 changed lines) into a
|
||||||
|
shared reviewer when the slice exceeds ~8 files. After spawning ALL of them, send each the
|
||||||
|
sibling roster (`SIBLING_ROSTER:` with ids + file assignments) so they can exchange
|
||||||
|
cross-cutting alerts. Then collect results — do not poll.
|
||||||
|
3. **The domain pass** (ALWAYS yours, direct or delegated — this is why you exist). Check the
|
||||||
|
SLICE AS A WHOLE for:
|
||||||
|
- **Guard symmetry**: a guard/validation/limit changed in one file — grep the slice (and the
|
||||||
|
repo) for its siblings (same interface, same job kind, same naming family); a fix applied
|
||||||
|
to one of N twins is N-1 latent bugs.
|
||||||
|
- **Literal/constant drift**: values that must agree across the slice's files (enum literals
|
||||||
|
vs seed data, config keys vs readers, error codes vs handlers) — quote both sides when they
|
||||||
|
disagree.
|
||||||
|
- **Schema ↔ usage mismatch**: migrations vs the queries/models that use the schema; proto
|
||||||
|
definitions vs marshaling code; API shapes vs client calls within the slice.
|
||||||
|
- **Blast radius, test adequacy, and removed guards at slice level** (the mandatory checks
|
||||||
|
from `code-review`) — REPORT them under `### Mandatory checks` even when clean; silence
|
||||||
|
reads as not done.
|
||||||
|
4. **Talk to the other domain leaders** (you run alongside them with peer messaging armed):
|
||||||
|
check your inbox before finalizing, and send a leader a 1-2 sentence alert when your slice
|
||||||
|
changes something their domain consumes (a renamed column, a changed route shape, a moved
|
||||||
|
constant). Messages move facts sideways; they never replace your report.
|
||||||
|
5. **Synthesize the domain report** in the exact output format below.
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
Structure your final response EXACTLY as:
|
||||||
|
|
||||||
|
```
|
||||||
|
## Domain: <domain name>
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
<2-3 sentences: what this slice changes as a unit>
|
||||||
|
|
||||||
|
### Findings
|
||||||
|
|
||||||
|
#### <finding title (optional [convention]/[correctness] marker per the loaded skills)>
|
||||||
|
- **Severity**: 🔴 CRITICAL | 🟡 WARNING | 🟢 SUGGESTION | 💡 NITPICK
|
||||||
|
- **File**: <path>
|
||||||
|
- **Lines**: <start>-<end>
|
||||||
|
- **Description**: <clear explanation of the issue>
|
||||||
|
- **Suggestion**: <how to fix it>
|
||||||
|
|
||||||
|
#### <next finding>
|
||||||
|
...
|
||||||
|
|
||||||
|
### Mandatory checks
|
||||||
|
- Blast radius: <result, or "clean — callers of the changed symbols checked: <list>">
|
||||||
|
- Test adequacy: <result, or "clean — behavior changes covered by: <tests>">
|
||||||
|
- Guard symmetry: <result, or "clean — no guard changes in this slice" / "twins checked: <list>">
|
||||||
|
|
||||||
|
### Cross-Domain Concerns
|
||||||
|
<alerts you sent or received and concerns affecting other domains, or "None">
|
||||||
|
|
||||||
|
DOMAIN_REVIEW_COMPLETE
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **You are read-only.** Never modify files.
|
||||||
|
2. **Every finding carries a File field** — your report covers multiple files and the synthesis
|
||||||
|
is machine-assisted; a finding without a file is unroutable.
|
||||||
|
3. **Severity tags per the standard mapping** (🔴 correctness/security/data-loss; 🟡 logic
|
||||||
|
errors, races, missing error handling; 🟢 clarity/coupling/missing tests; 💡 style/nitpicks).
|
||||||
|
Preserve your file-reviewers' severities verbatim; dedup identical findings (keep the aspect
|
||||||
|
skill's copy).
|
||||||
|
4. **Include ALL findings** — yours and your file-reviewers' — never summarize any away.
|
||||||
|
5. **Mandatory checks report even when clean.**
|
||||||
|
6. **End with DOMAIN_REVIEW_COMPLETE** on its own line.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
- Project: {{project_dir}}
|
||||||
|
- CWD: {{__cwd__}}
|
||||||
|
|
||||||
|
## Available Tools
|
||||||
|
{{__tools__}}
|
||||||
Executable
+63
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -eo pipefail
|
||||||
|
|
||||||
|
# @env LLM_OUTPUT=/dev/stdout
|
||||||
|
# @env LLM_AGENT_VAR_PROJECT_DIR=.
|
||||||
|
# @describe Domain-slice reviewer tools
|
||||||
|
|
||||||
|
_project_dir() {
|
||||||
|
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
|
||||||
|
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# @cmd Get the git diff for this domain's file slice. Defaults to staged changes, falls back to unstaged, then HEAD~1. Pass --base for a ref or range (e.g. "main...HEAD").
|
||||||
|
# @option --base Optional base ref or range to diff against (e.g. "main...HEAD", "HEAD~3", a SHA)
|
||||||
|
# @option --files Comma-separated list of file paths to limit the diff to (your assignment's files)
|
||||||
|
get_diff() {
|
||||||
|
local project_dir
|
||||||
|
project_dir=$(_project_dir)
|
||||||
|
# shellcheck disable=SC2154
|
||||||
|
local base="${argc_base:-}"
|
||||||
|
# shellcheck disable=SC2154
|
||||||
|
local files_csv="${argc_files:-}"
|
||||||
|
|
||||||
|
local -a path_args=()
|
||||||
|
if [[ -n "${files_csv}" ]]; then
|
||||||
|
local IFS=','
|
||||||
|
local f
|
||||||
|
for f in ${files_csv}; do
|
||||||
|
f="$(echo "${f}" | sed -e 's/^ *//' -e 's/ *$//')"
|
||||||
|
[[ -n "${f}" ]] && path_args+=("${f}")
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
_diff() {
|
||||||
|
if ((${#path_args[@]})); then
|
||||||
|
(cd "${project_dir}" && git diff "$@" -- "${path_args[@]}" 2>&1) || true
|
||||||
|
else
|
||||||
|
(cd "${project_dir}" && git diff "$@" 2>&1) || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
local diff_output=""
|
||||||
|
if [[ -n "${base}" ]]; then
|
||||||
|
diff_output=$(_diff "${base}")
|
||||||
|
else
|
||||||
|
diff_output=$(_diff --cached)
|
||||||
|
[[ -z "${diff_output}" ]] && diff_output=$(_diff)
|
||||||
|
[[ -z "${diff_output}" ]] && diff_output=$(_diff HEAD~1)
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "${diff_output}" ]]; then
|
||||||
|
echo "No changes found for the requested files in ${project_dir}." >> "$LLM_OUTPUT"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
local file_count
|
||||||
|
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
|
||||||
|
{
|
||||||
|
echo "Diff contains changes to ${file_count} file(s):"
|
||||||
|
echo ""
|
||||||
|
echo "${diff_output}"
|
||||||
|
} >> "$LLM_OUTPUT"
|
||||||
|
}
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
name: file-reviewer
|
name: file-reviewer
|
||||||
description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns
|
description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns
|
||||||
version: 2.3.0
|
version: 2.4.0
|
||||||
|
|
||||||
skills_enabled: true
|
skills_enabled: true
|
||||||
enabled_skills:
|
enabled_skills:
|
||||||
@@ -85,6 +85,7 @@ instructions: |
|
|||||||
|
|
||||||
The `code-review` skill teaches the investigation workflow. Apply these per-review caps on top:
|
The `code-review` skill teaches the investigation workflow. Apply these per-review caps on top:
|
||||||
- **Max 5 fs_read calls per review.** Be deliberate about which files you read.
|
- **Max 5 fs_read calls per review.** Be deliberate about which files you read.
|
||||||
|
- **`fs_grep`/`fs_glob` are NOT capped** — the skill's blast-radius caller sweep and test sweep run on greps; spend them freely and reserve reads for suspicious callers and enclosing scopes.
|
||||||
- **`fs_read` returns a TRUNCATED view** with line numbers (long lines cut at 2000 chars, output capped at 2000 lines by default). Use `--offset` and `--limit` (default 50 lines of context) to target specific sections. Never read entire large files.
|
- **`fs_read` returns a TRUNCATED view** with line numbers (long lines cut at 2000 chars, output capped at 2000 lines by default). Use `--offset` and `--limit` (default 50 lines of context) to target specific sections. Never read entire large files.
|
||||||
- **Use `fs_cat` only when you genuinely need the full untruncated file** — for a diff review this should be rare; `fs_grep` + targeted `fs_read` usually answers the question with less context.
|
- **Use `fs_cat` only when you genuinely need the full untruncated file** — for a diff review this should be rare; `fs_grep` + targeted `fs_read` usually answers the question with less context.
|
||||||
- **Focus on the diff.** Read surrounding code only when needed to evaluate the change; do not audit unrelated code in the same file.
|
- **Focus on the diff.** Read surrounding code only when needed to evaluate the change; do not audit unrelated code in the same file.
|
||||||
@@ -134,7 +135,8 @@ instructions: |
|
|||||||
1. **Be specific.** Reference exact line numbers and code.
|
1. **Be specific.** Reference exact line numbers and code.
|
||||||
2. **Be actionable.** Every finding must have a suggestion.
|
2. **Be actionable.** Every finding must have a suggestion.
|
||||||
3. **Never modify files.** You are read-only.
|
3. **Never modify files.** You are read-only.
|
||||||
4. **Always end with REVIEW_COMPLETE.**
|
4. **Mandatory checks report even when clean.** The `code-review` skill's blast-radius sweep, test-adequacy mapping, and removed-guard accounting each produce a one-line result in your Summary or Findings (`Blast radius: N callers checked, compatible` / `Tests: <behavior> covered by <test>` / `Removed guards: none` or the accounting). Silence on any of the three reads as "not done".
|
||||||
|
5. **Always end with REVIEW_COMPLETE.**
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
- Project: {{project_dir}}
|
- Project: {{project_dir}}
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# finding-verifier
|
||||||
|
|
||||||
|
A **graph** agent that independently verifies code-review findings against the working
|
||||||
|
tree. It exists because false positives kill hands-off trust faster than missed bugs do —
|
||||||
|
and because an orchestrator verifying its own workers' findings inline can skip, fudge,
|
||||||
|
or confirmation-bias the check. A graph can't: the structure IS the discipline.
|
||||||
|
|
||||||
|
## How it works
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["parse (llm, max_attempts 2)"] --> B["verify_each (map, max_concurrency 4): one branch per finding"]
|
||||||
|
A -. "fallback" .-> PF["parse_fault (script): verdicts = one UNVERIFIABLE PIPELINE-FAULT entry"]
|
||||||
|
PF --> D
|
||||||
|
B -. "spawns × N" .-> C["verify_one (llm + fs tools, max_attempts 2)"]
|
||||||
|
C --> G["verdict_gate (script): machine-validates the JSON contract"]
|
||||||
|
G -. "reject-retry once" .-> C
|
||||||
|
C -. "fallback" .-> VF["verify_fault (script): branch-local UNVERIFIABLE marker"]
|
||||||
|
B --> D(["done (end): FINDING_VERIFIER_RESULTS"])
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
|
||||||
|
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
|
||||||
|
Gets two attempts (`max_attempts: 2`); if both fail, its `fallback` routes to
|
||||||
|
`parse_fault`, which rewrites `verdicts` to a single UNVERIFIABLE entry naming the fault
|
||||||
|
and jumps straight to `done`.
|
||||||
|
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
|
||||||
|
finding is structurally impossible; results collect in input order.
|
||||||
|
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
|
||||||
|
and answers ONE narrow question: does the cited code exist, and does it behave as the
|
||||||
|
claim says? Verdicts:
|
||||||
|
- `VERIFIED` — with the decisive line(s) quoted verbatim (`path:line`).
|
||||||
|
- `FALSE` — the code doesn't exist anywhere or provably doesn't do what the claim says.
|
||||||
|
Requires PROOF, not disagreement — doubt never lands here.
|
||||||
|
- `UNVERIFIABLE` — couldn't establish either way (runtime behavior, external systems).
|
||||||
|
|
||||||
|
Each branch is a chain: `verify_one → verdict_gate`. The gate machine-validates the JSON
|
||||||
|
contract (verdict enum; evidence required for VERIFIED/FALSE; note for UNVERIFIABLE),
|
||||||
|
stamps the authoritative id FROM the finding item (the model's echo is never trusted),
|
||||||
|
rejects back to `verify_one` exactly once with the reason, and on a second failure records
|
||||||
|
UNVERIFIABLE — a malformed branch never sinks the map. If `verify_one` itself dies (both
|
||||||
|
`max_attempts: 2` attempts fail), its `fallback` routes to `verify_fault`, which
|
||||||
|
normalizes the engine's failure text into a branch-local UNVERIFIABLE verdict the map
|
||||||
|
still collects.
|
||||||
|
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
|
||||||
|
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
|
||||||
|
marked). No final LLM step that could editorialize.
|
||||||
|
|
||||||
|
The dashed edges are fault paths. All three fault emitters (`parse_fault`, `verify_fault`,
|
||||||
|
and `verdict_gate`'s own crash guard) surface as `PIPELINE-FAULT:` text in a verdict's
|
||||||
|
`note`, and a fault can only ever produce UNVERIFIABLE — degraded output is never a
|
||||||
|
passing verdict. Every path ends at `done`, so the `FINDING_VERIFIER_RESULTS`
|
||||||
|
sentinel is always emitted and the caller's verify lane degrades instead of dying.
|
||||||
|
|
||||||
|
## What it is NOT
|
||||||
|
|
||||||
|
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
|
||||||
|
"matters" — that authority stays with the reviewers and the orchestrator's rigor rules.
|
||||||
|
- **Not a reviewer.** It reads only what findings cite (plus greps to locate moved code).
|
||||||
|
|
||||||
|
## Caller
|
||||||
|
|
||||||
|
Spawned by [`code-reviewer`](../code-reviewer/README.md) during its Finding Verifier
|
||||||
|
pass, ONCE per review, with the full 🔴/🟡 list and `project_dir`. If this agent isn't
|
||||||
|
installed, code-reviewer runs the same contract inline as a fallback.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
|
||||||
|
per node).
|
||||||
|
- `scripts/verdict_gate.py` — the deterministic contract gate.
|
||||||
|
- `scripts/parse_fault.py`, `scripts/verify_fault.py` — the fault markers behind the
|
||||||
|
dashed fallback edges.
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
name: finding-verifier
|
||||||
|
description: |
|
||||||
|
Graph-based independent verifier for code-review findings. Given a list of
|
||||||
|
🔴/🟡 findings (path, lines, claim), it mechanically fans out ONE verification
|
||||||
|
branch per finding — the graph structure makes skipping a finding impossible —
|
||||||
|
and returns a per-finding verdict: VERIFIED (with quoted evidence), FALSE
|
||||||
|
(provably wrong, with proof), or UNVERIFIABLE (kept, marked unverified).
|
||||||
|
Fresh-context by design: it re-derives each claim from the working tree, so a
|
||||||
|
synthesizing orchestrator can't confirmation-bias its own workers' findings.
|
||||||
|
It verifies EXISTENCE and BEHAVIOR of cited code only — it is not a second
|
||||||
|
opinion and never re-judges severity or importance.
|
||||||
|
|
||||||
|
version: '1.0'
|
||||||
|
|
||||||
|
global_tools:
|
||||||
|
- fs_read.sh
|
||||||
|
- fs_cat.sh
|
||||||
|
- fs_grep.sh
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: project_dir
|
||||||
|
description: Repository the findings cite; relative paths in findings resolve against it.
|
||||||
|
default: '.'
|
||||||
|
|
||||||
|
settings:
|
||||||
|
max_loop_iterations: 8
|
||||||
|
log_state_snapshots: true
|
||||||
|
timeout: 3000
|
||||||
|
|
||||||
|
initial_state:
|
||||||
|
findings_list: []
|
||||||
|
verdicts: []
|
||||||
|
gate_feedback: ''
|
||||||
|
gate_attempts: 0
|
||||||
|
parse_failure: ''
|
||||||
|
|
||||||
|
start: parse
|
||||||
|
|
||||||
|
nodes:
|
||||||
|
parse:
|
||||||
|
id: parse
|
||||||
|
type: llm
|
||||||
|
description: Extract the 🔴/🟡 findings from the spawn prompt into a structured list. Extraction only — no judging.
|
||||||
|
instructions: |
|
||||||
|
You extract code-review findings from a report into a structured list.
|
||||||
|
EXTRACTION ONLY — do not judge, filter, merge, or rephrase-away any
|
||||||
|
finding. Rules:
|
||||||
|
|
||||||
|
- Include every 🔴 CRITICAL and 🟡 WARNING finding. Ignore 🟢/💡
|
||||||
|
findings (they post unverified by design).
|
||||||
|
- Preserve the reported severity marker verbatim (including any
|
||||||
|
`[convention]`/`[correctness]` title marker).
|
||||||
|
- `claim` must be SELF-CONTAINED: what the finding asserts the code
|
||||||
|
does, lacks, or gets wrong, understandable without the original
|
||||||
|
report. Include the suggested-fix context when it clarifies the claim.
|
||||||
|
- `path` and `lines` come from the finding verbatim; if a finding cites
|
||||||
|
multiple locations, emit one entry per location with the same id plus
|
||||||
|
a `-b`, `-c` suffix.
|
||||||
|
- `project_dir`: the repository directory named in the request (a
|
||||||
|
"Repository:" or "Project:" line). When none is named, echo the
|
||||||
|
default shown in the prompt.
|
||||||
|
prompt: |
|
||||||
|
Extract the findings from this verification request (default repository
|
||||||
|
directory when the request names none: {{project_dir}}):
|
||||||
|
|
||||||
|
{{initial_prompt}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
findings_list:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
id:
|
||||||
|
type: string
|
||||||
|
description: Short stable slug for mapping the verdict back (e.g. "f1", "f2-b").
|
||||||
|
severity:
|
||||||
|
type: string
|
||||||
|
description: The severity marker verbatim (e.g. "🔴 CRITICAL", "🟡 WARNING [correctness]").
|
||||||
|
path:
|
||||||
|
type: string
|
||||||
|
description: File path the finding cites, as reported.
|
||||||
|
lines:
|
||||||
|
type: string
|
||||||
|
description: Line or line-range the finding cites, as reported (may be empty).
|
||||||
|
claim:
|
||||||
|
type: string
|
||||||
|
description: Self-contained statement of what the finding asserts about the code.
|
||||||
|
required: [id, severity, path, lines, claim]
|
||||||
|
project_dir:
|
||||||
|
type: string
|
||||||
|
description: Repository directory named in the request (echo the prompt default when none).
|
||||||
|
required: [findings_list, project_dir]
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
parse_failure: '{{output}}'
|
||||||
|
fallback: parse_fault
|
||||||
|
next: verify_each
|
||||||
|
|
||||||
|
parse_fault:
|
||||||
|
id: parse_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — rewrites `verdicts` to a single UNVERIFIABLE PIPELINE-FAULT entry and jumps straight to done, so the FINDING_VERIFIER_RESULTS sentinel still renders and the caller's verify lane degrades instead of dying.
|
||||||
|
script: scripts/parse_fault.py
|
||||||
|
timeout: 30
|
||||||
|
next: done
|
||||||
|
|
||||||
|
verify_each:
|
||||||
|
id: verify_each
|
||||||
|
type: map
|
||||||
|
over: '{{findings_list}}'
|
||||||
|
as: finding
|
||||||
|
branch: verify_one
|
||||||
|
collect_into: verdicts
|
||||||
|
output_key: verdict
|
||||||
|
max_concurrency: 4
|
||||||
|
next: done
|
||||||
|
|
||||||
|
verify_one:
|
||||||
|
id: verify_one
|
||||||
|
type: llm
|
||||||
|
description: Verify ONE finding against the working tree — existence and behavior of the cited code, nothing else.
|
||||||
|
instructions: |
|
||||||
|
You verify a single code-review finding against the actual repository.
|
||||||
|
Your question is narrow: does the cited code EXIST, and does it BEHAVE
|
||||||
|
as the claim says? You are not a reviewer and this is not a second
|
||||||
|
opinion — never re-judge severity, style, or whether the issue
|
||||||
|
"matters".
|
||||||
|
|
||||||
|
Method:
|
||||||
|
1. `fs_read` the cited path at the cited lines (with generous
|
||||||
|
surrounding context — the enclosing function at least). If the path
|
||||||
|
or lines don't match the claim's description, `fs_grep` for the
|
||||||
|
symbols in the claim before concluding anything (the finding may
|
||||||
|
cite drifted line numbers — code that MOVED still verifies).
|
||||||
|
2. Compare what the code actually does against the claim.
|
||||||
|
|
||||||
|
Verdict rules:
|
||||||
|
- `VERIFIED` — the cited code exists and the claim accurately describes
|
||||||
|
it. `evidence` MUST be the decisive line(s) quoted verbatim with
|
||||||
|
path:line.
|
||||||
|
- `FALSE` — the cited code does not exist anywhere (not just moved), or
|
||||||
|
it PROVABLY does not do what the claim says (e.g. the claim says a
|
||||||
|
nil-check is missing but it is right there). `evidence` MUST quote
|
||||||
|
the proof. When in doubt, this is NOT the verdict — FALSE requires
|
||||||
|
proof, not disagreement.
|
||||||
|
- `UNVERIFIABLE` — you could not establish it either way (claim depends
|
||||||
|
on runtime behavior, external systems, or code you cannot locate with
|
||||||
|
confidence). Explain why in `note`. Doubt lands here, never in FALSE.
|
||||||
|
|
||||||
|
Output format — respond with ONLY this JSON object, no prose before or
|
||||||
|
after it:
|
||||||
|
{"id": "<the finding's id, echoed back verbatim>",
|
||||||
|
"verdict": "VERIFIED" | "FALSE" | "UNVERIFIABLE",
|
||||||
|
"evidence": "<verbatim quoted line(s) with path:line — required for
|
||||||
|
VERIFIED and FALSE, empty string for UNVERIFIABLE>",
|
||||||
|
"note": "<one line — for FALSE the disproof in words, for UNVERIFIABLE
|
||||||
|
why it could not be established; empty string otherwise>"}
|
||||||
|
|
||||||
|
Your response is machine-validated by a deterministic gate: a malformed
|
||||||
|
response (wrong shape, unknown verdict value, missing required
|
||||||
|
evidence/note) is rejected back to you exactly once with the reason,
|
||||||
|
then recorded as UNVERIFIABLE. The `id` you echo is informative only —
|
||||||
|
the gate stamps the authoritative id from the finding itself.
|
||||||
|
prompt: |
|
||||||
|
Repository: {{project_dir}}
|
||||||
|
|
||||||
|
Verify this single finding:
|
||||||
|
{{finding}}
|
||||||
|
{{gate_feedback}}
|
||||||
|
tools:
|
||||||
|
- fs_read
|
||||||
|
- fs_cat
|
||||||
|
- fs_grep
|
||||||
|
max_iterations: 28
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
verdict: '{{output}}'
|
||||||
|
fallback: verify_fault
|
||||||
|
next: verdict_gate
|
||||||
|
|
||||||
|
verdict_gate:
|
||||||
|
id: verdict_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic contract gate for each verdict — validates shape, verdict enum, and evidence/note rules; rejects back to verify_one exactly once with the reason, then records UNVERIFIABLE. No LLM judgment.
|
||||||
|
script: scripts/verdict_gate.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
verify_fault:
|
||||||
|
id: verify_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead verify branch (reached only via verify_one's fallback; branch-local). Normalizes the engine's failure text in `verdict` into a schema-conformant UNVERIFIABLE verdict the map collects — a dead verifier lane never sinks the whole map or masquerades as verified.
|
||||||
|
script: scripts/verify_fault.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
done:
|
||||||
|
id: done
|
||||||
|
type: end
|
||||||
|
output: |
|
||||||
|
FINDING_VERIFIER_RESULTS
|
||||||
|
(verdict semantics for the caller: VERIFIED = keep the finding and paste its evidence
|
||||||
|
quote into the Description; FALSE = drop the finding and tally it in the footer;
|
||||||
|
UNVERIFIABLE = keep the finding, marked unverified. Severities are never rewritten.)
|
||||||
|
|
||||||
|
{{verdicts}}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead parse stage of the finding-verifier graph.
|
||||||
|
|
||||||
|
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||||
|
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||||
|
this node rewrites `verdicts` to a single schema-conformant UNVERIFIABLE
|
||||||
|
entry naming the fault and the graph jumps straight to `done` — the
|
||||||
|
FINDING_VERIFIER_RESULTS sentinel still renders, so the caller (e.g.
|
||||||
|
code-reviewer's verify lane) degrades instead of dying. Findings that were
|
||||||
|
never extracted cannot be verified; the caller's "UNVERIFIABLE = keep the
|
||||||
|
finding, marked unverified" semantics do the right thing with the entry.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def fault_verdicts(detail):
|
||||||
|
return {
|
||||||
|
"verdicts": [
|
||||||
|
{
|
||||||
|
"id": "pipeline-fault",
|
||||||
|
"verdict": "UNVERIFIABLE",
|
||||||
|
"evidence": "",
|
||||||
|
"note": (
|
||||||
|
"PIPELINE-FAULT: parse failed — findings could not be extracted "
|
||||||
|
f"from the verification request; no finding was verified: {detail}"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("parse_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the parse stage died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(json.dumps(fault_verdicts(detail)))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(json.dumps(fault_verdicts(f"fault-marker script error: {e}")))
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Contract gate for finding-verifier's per-finding verify chain.
|
||||||
|
|
||||||
|
Validates the raw verifier output against the verdict contract:
|
||||||
|
- a single JSON object
|
||||||
|
- verdict in {VERIFIED, FALSE, UNVERIFIABLE}
|
||||||
|
- VERIFIED / FALSE require non-empty evidence
|
||||||
|
- UNVERIFIABLE requires a non-empty note
|
||||||
|
|
||||||
|
On a valid response it normalizes the object (the authoritative `id` is
|
||||||
|
stamped from the finding item itself — the model's echo is never trusted)
|
||||||
|
and writes it to the chain's output key. On an invalid response it routes
|
||||||
|
back to `verify_one` exactly once with the rejection reason; if the retry
|
||||||
|
is also invalid, it records an UNVERIFIABLE verdict naming the failure so
|
||||||
|
a malformed item can never sink the whole map or masquerade as a pass.
|
||||||
|
|
||||||
|
Deterministic by design: no LLM judgment happens here, and the script
|
||||||
|
always exits 0 with a valid JSON object on stdout.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
VERDICTS = {"VERIFIED", "FALSE", "UNVERIFIABLE"}
|
||||||
|
MAX_RETRIES = 1
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def finding_id(state):
|
||||||
|
finding = state.get("finding")
|
||||||
|
if isinstance(finding, dict):
|
||||||
|
fid = finding.get("id")
|
||||||
|
if isinstance(fid, str) and fid.strip():
|
||||||
|
return fid.strip()
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_json(text):
|
||||||
|
if not isinstance(text, str) or not text.strip():
|
||||||
|
raise ValueError("verifier output is empty or not text")
|
||||||
|
text = re.sub(r"```[a-zA-Z]*", "", text).replace("```", "")
|
||||||
|
start, end = text.find("{"), text.rfind("}")
|
||||||
|
if start == -1 or end <= start:
|
||||||
|
raise ValueError("no JSON object found in verifier output")
|
||||||
|
return json.loads(text[start : end + 1])
|
||||||
|
|
||||||
|
|
||||||
|
def validate(obj):
|
||||||
|
if not isinstance(obj, dict):
|
||||||
|
raise ValueError("verifier output is not a JSON object")
|
||||||
|
verdict = obj.get("verdict")
|
||||||
|
if verdict not in VERDICTS:
|
||||||
|
raise ValueError(f"verdict must be one of {sorted(VERDICTS)}, got {verdict!r}")
|
||||||
|
evidence = obj.get("evidence") if isinstance(obj.get("evidence"), str) else ""
|
||||||
|
note = obj.get("note") if isinstance(obj.get("note"), str) else ""
|
||||||
|
if verdict in ("VERIFIED", "FALSE") and not evidence.strip():
|
||||||
|
raise ValueError(
|
||||||
|
f"{verdict} requires non-empty evidence quoting the decisive line(s) with path:line"
|
||||||
|
)
|
||||||
|
if verdict == "UNVERIFIABLE" and not note.strip():
|
||||||
|
raise ValueError("UNVERIFIABLE requires a note explaining why it could not be established")
|
||||||
|
return verdict, evidence, note
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
fid = finding_id(state)
|
||||||
|
attempts = state.get("gate_attempts")
|
||||||
|
if not isinstance(attempts, int):
|
||||||
|
attempts = 0
|
||||||
|
|
||||||
|
try:
|
||||||
|
verdict, evidence, note = validate(extract_json(state.get("verdict")))
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{"verdict": {"id": fid, "verdict": verdict, "evidence": evidence, "note": note}}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
except Exception as e: # noqa: BLE001 — every failure is a rejection reason
|
||||||
|
reason = str(e)
|
||||||
|
|
||||||
|
if attempts < MAX_RETRIES:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"_next": "verify_one",
|
||||||
|
"gate_attempts": attempts + 1,
|
||||||
|
"gate_feedback": (
|
||||||
|
"PREVIOUS RESPONSE REJECTED by the verdict gate: "
|
||||||
|
+ reason
|
||||||
|
+ ". Respond with ONLY the JSON object described under"
|
||||||
|
" 'Output format' — no prose, no code fences."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
# Deliberately not PIPELINE-FAULT-prefixed: model misbehavior, not infrastructure.
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"verdict": {
|
||||||
|
"id": fid,
|
||||||
|
"verdict": "UNVERIFIABLE",
|
||||||
|
"evidence": "",
|
||||||
|
"note": f"verifier output failed machine validation after retry: {reason}",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the gate must never crash the chain
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"verdict": {
|
||||||
|
"id": "unknown",
|
||||||
|
"verdict": "UNVERIFIABLE",
|
||||||
|
"evidence": "",
|
||||||
|
"note": f"PIPELINE-FAULT: verdict gate error: {e}",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead verify branch of the finding-verifier graph.
|
||||||
|
|
||||||
|
Reached ONLY via verify_one's `fallback:` route inside the map branch.
|
||||||
|
Grounded in engine behavior: on llm-node failure the engine writes
|
||||||
|
"LLM node failed: <chain>" into `verdict` via the node's state_updates
|
||||||
|
BEFORE routing to the fallback, so the raw failure text is already in
|
||||||
|
state. This node normalizes it into a schema-conformant UNVERIFIABLE
|
||||||
|
verdict (authoritative id stamped from the finding item, mirroring
|
||||||
|
verdict_gate.py) and the chain ends there — the map collects a valid
|
||||||
|
verdict, so a dead verifier lane never sinks the whole map or masquerades
|
||||||
|
as verified.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def finding_id(state):
|
||||||
|
finding = state.get("finding")
|
||||||
|
if isinstance(finding, dict):
|
||||||
|
fid = finding.get("id")
|
||||||
|
if isinstance(fid, str) and fid.strip():
|
||||||
|
return fid.strip()
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def fault_verdict(fid, detail):
|
||||||
|
return {
|
||||||
|
"verdict": {
|
||||||
|
"id": fid,
|
||||||
|
"verdict": "UNVERIFIABLE",
|
||||||
|
"evidence": "",
|
||||||
|
"note": f"PIPELINE-FAULT: verifier lane failed after retries — {detail}",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
fid = finding_id(state)
|
||||||
|
raw = state.get("verdict")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if not detail.startswith("LLM node"):
|
||||||
|
detail = "the verifier lane died without recording a failure detail"
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(json.dumps(fault_verdict(fid, detail)))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(json.dumps(fault_verdict("unknown", f"fault-marker script error: {e}")))
|
||||||
@@ -6,13 +6,16 @@ description: |
|
|||||||
preamble. The "external grep" sibling of explore (which handles
|
preamble. The "external grep" sibling of explore (which handles
|
||||||
internal/codebase grep). Designed to be fanned out 1-3 in parallel by
|
internal/codebase grep). Designed to be fanned out 1-3 in parallel by
|
||||||
sisyphus alongside explore when unfamiliar libraries/APIs/frameworks are
|
sisyphus alongside explore when unfamiliar libraries/APIs/frameworks are
|
||||||
involved.
|
involved. Supports research_depth=deep (via --variables) for broad
|
||||||
|
design-time topics: ~3x the search/OSS/fetch budgets of the standard
|
||||||
|
quick-lookup mode.
|
||||||
|
|
||||||
version: '1.0'
|
version: '1.0'
|
||||||
|
|
||||||
global_tools:
|
global_tools:
|
||||||
- web_search_coyote.sh
|
- web_search_coyote.sh
|
||||||
- fetch_url_via_curl.sh
|
- fetch_url_via_curl.sh
|
||||||
|
- search_arxiv.sh
|
||||||
|
|
||||||
mcp_servers:
|
mcp_servers:
|
||||||
- ddg-search
|
- ddg-search
|
||||||
@@ -26,14 +29,14 @@ variables:
|
|||||||
- name: project_dir
|
- name: project_dir
|
||||||
description: Project directory for context (unused in MVP but reserved for future iterations).
|
description: Project directory for context (unused in MVP but reserved for future iterations).
|
||||||
default: '.'
|
default: '.'
|
||||||
|
- name: research_depth
|
||||||
|
description: "standard (default) = quick targeted lookup budgets. deep = design-time research: ~3x search/OSS/fetch budgets for broad topics (best-practice checklists, architecture surveys)."
|
||||||
|
default: 'standard'
|
||||||
|
|
||||||
settings:
|
settings:
|
||||||
max_loop_iterations: 12
|
max_loop_iterations: 12
|
||||||
log_state_snapshots: true
|
log_state_snapshots: true
|
||||||
timeout: 600
|
timeout: 1800
|
||||||
|
|
||||||
reducers:
|
|
||||||
output: overwrite
|
|
||||||
|
|
||||||
initial_state:
|
initial_state:
|
||||||
language_ecosystem: 'general'
|
language_ecosystem: 'general'
|
||||||
@@ -120,7 +123,7 @@ nodes:
|
|||||||
search:
|
search:
|
||||||
id: search
|
id: search
|
||||||
type: llm
|
type: llm
|
||||||
description: Identify 3-5 authoritative documentation sources via ddg-search.
|
description: Identify authoritative documentation sources via ddg-search (3-5 standard depth, 6-12 deep).
|
||||||
skills_enabled: true
|
skills_enabled: true
|
||||||
enabled_skills:
|
enabled_skills:
|
||||||
- ai-slop-remover
|
- ai-slop-remover
|
||||||
@@ -132,7 +135,10 @@ nodes:
|
|||||||
Priority order:
|
Priority order:
|
||||||
1. Official documentation - PRIORITIZE the hinted doc domains when
|
1. Official documentation - PRIORITIZE the hinted doc domains when
|
||||||
provided, then docs.X.org / readthedocs.io / MDN / vendor docs
|
provided, then docs.X.org / readthedocs.io / MDN / vendor docs
|
||||||
2. Specifications (RFCs, W3C, ECMA, IEEE)
|
2. Specifications (RFCs, W3C, ECMA, IEEE) and academic sources -
|
||||||
|
the search_arxiv tool is available; use it when the topic is
|
||||||
|
academic/research-flavored (arXiv/DOI links count as
|
||||||
|
authoritative)
|
||||||
3. Credible secondary references (PEPs, official blog posts) - only
|
3. Credible secondary references (PEPs, official blog posts) - only
|
||||||
if 1-2 are sparse
|
if 1-2 are sparse
|
||||||
|
|
||||||
@@ -145,14 +151,19 @@ nodes:
|
|||||||
|
|
||||||
## Search budget and fail-fast rules
|
## Search budget and fail-fast rules
|
||||||
|
|
||||||
You have a HARD BUDGET of 3 search calls total. After 3 calls, stop
|
Your budget is set by the `Research depth` line in the prompt:
|
||||||
calling tools and produce your final answer with whatever you have.
|
- standard -> HARD BUDGET of 3 search calls total.
|
||||||
|
- deep -> HARD BUDGET of 10 search calls total; spend the extra
|
||||||
|
calls covering DISTINCT subtopics of the research topic, never
|
||||||
|
re-running near-duplicate queries.
|
||||||
|
After the budget is spent, stop calling tools and produce your final
|
||||||
|
answer with whatever you have.
|
||||||
|
|
||||||
If a search returns "HTTP 202 Accepted", empty results, error messages,
|
If a search returns "HTTP 202 Accepted", empty results, error messages,
|
||||||
or rate-limit warnings: that counts as a used call. Do not retry the
|
or rate-limit warnings: that counts as a used call. Do not retry the
|
||||||
same query - either rephrase OR give up.
|
same query - either rephrase OR give up.
|
||||||
|
|
||||||
If after 3 calls you have NO usable URLs, output exactly:
|
If after spending the budget you have NO usable URLs, output exactly:
|
||||||
|
|
||||||
NO_AUTHORITATIVE_SOURCES_FOUND
|
NO_AUTHORITATIVE_SOURCES_FOUND
|
||||||
Reason: <one line>
|
Reason: <one line>
|
||||||
@@ -171,9 +182,11 @@ nodes:
|
|||||||
URL: <full url>
|
URL: <full url>
|
||||||
...
|
...
|
||||||
|
|
||||||
Output 3-5 source blocks. No prose intro, no closing summary.
|
Output 3-5 source blocks (standard depth) or 6-12 (deep). No prose
|
||||||
|
intro, no closing summary.
|
||||||
prompt: |
|
prompt: |
|
||||||
Research topic: {{initial_prompt}}
|
Research topic: {{initial_prompt}}
|
||||||
|
Research depth: {{research_depth}}
|
||||||
|
|
||||||
Triage hints:
|
Triage hints:
|
||||||
- Language/ecosystem: {{language_ecosystem}}
|
- Language/ecosystem: {{language_ecosystem}}
|
||||||
@@ -186,8 +199,9 @@ nodes:
|
|||||||
pathlib` style queries).
|
pathlib` style queries).
|
||||||
tools:
|
tools:
|
||||||
- mcp:ddg-search
|
- mcp:ddg-search
|
||||||
|
- search_arxiv
|
||||||
- web_search_coyote
|
- web_search_coyote
|
||||||
max_iterations: 15
|
max_iterations: 30
|
||||||
state_updates:
|
state_updates:
|
||||||
search_output: '{{output}}'
|
search_output: '{{output}}'
|
||||||
fallback: synthesize
|
fallback: synthesize
|
||||||
@@ -196,7 +210,7 @@ nodes:
|
|||||||
search_oss:
|
search_oss:
|
||||||
id: search_oss
|
id: search_oss
|
||||||
type: llm
|
type: llm
|
||||||
description: Find 2-3 production OSS examples relevant to the topic via the personal-github MCP.
|
description: Find production OSS examples relevant to the topic via the personal-github MCP (2-3 standard depth, 4-8 deep).
|
||||||
skills_enabled: true
|
skills_enabled: true
|
||||||
enabled_skills:
|
enabled_skills:
|
||||||
- ai-slop-remover
|
- ai-slop-remover
|
||||||
@@ -222,8 +236,12 @@ nodes:
|
|||||||
|
|
||||||
## Search budget and fail-fast rules
|
## Search budget and fail-fast rules
|
||||||
|
|
||||||
HARD BUDGET: 8 tool calls total. After 8 calls, stop and output what
|
Your budget is set by the `Research depth` line in the prompt:
|
||||||
you have - even one or two examples is fine.
|
- standard -> HARD BUDGET of 8 tool calls total; output 2-3 blocks.
|
||||||
|
- deep -> HARD BUDGET of 20 tool calls total; output 4-8 blocks
|
||||||
|
across DIFFERENT repos/approaches, not near-duplicates.
|
||||||
|
After the budget is spent, stop and output what you have - even one or
|
||||||
|
two examples is fine.
|
||||||
|
|
||||||
If you find no production examples, output exactly:
|
If you find no production examples, output exactly:
|
||||||
|
|
||||||
@@ -243,10 +261,12 @@ nodes:
|
|||||||
|
|
||||||
REPO: ...
|
REPO: ...
|
||||||
|
|
||||||
Output 2-3 blocks. The URL should point to a specific file that
|
Output 2-3 blocks (standard depth) or 4-8 (deep). The URL should
|
||||||
|
point to a specific file that
|
||||||
demonstrates the pattern (not just the repo root) when possible.
|
demonstrates the pattern (not just the repo root) when possible.
|
||||||
prompt: |
|
prompt: |
|
||||||
Research topic: {{initial_prompt}}
|
Research topic: {{initial_prompt}}
|
||||||
|
Research depth: {{research_depth}}
|
||||||
|
|
||||||
Triage hints:
|
Triage hints:
|
||||||
- Language/ecosystem: {{language_ecosystem}}
|
- Language/ecosystem: {{language_ecosystem}}
|
||||||
@@ -257,7 +277,7 @@ nodes:
|
|||||||
Filter to {{language_ecosystem}} repositories when the API allows.
|
Filter to {{language_ecosystem}} repositories when the API allows.
|
||||||
tools:
|
tools:
|
||||||
- mcp:personal-github
|
- mcp:personal-github
|
||||||
max_iterations: 15
|
max_iterations: 30
|
||||||
state_updates:
|
state_updates:
|
||||||
oss_output: '{{output}}'
|
oss_output: '{{output}}'
|
||||||
fallback: synthesize
|
fallback: synthesize
|
||||||
@@ -311,8 +331,10 @@ nodes:
|
|||||||
5. If a URL is dead, returns garbage, or is off-topic, note it
|
5. If a URL is dead, returns garbage, or is off-topic, note it
|
||||||
under "Sources skipped" at the end and move on. Do not retry.
|
under "Sources skipped" at the end and move on. Do not retry.
|
||||||
|
|
||||||
Budget: max 8 fetches total (across both source lists). Skip
|
Fetch budget, set by the `Research depth` line in the prompt:
|
||||||
aggressively.
|
- standard -> max 8 fetches total (across both source lists).
|
||||||
|
- deep -> max 25 fetches total. Depth buys MORE SOURCES, never
|
||||||
|
re-fetching dead or off-topic URLs - skip aggressively either way.
|
||||||
|
|
||||||
## Output format
|
## Output format
|
||||||
|
|
||||||
@@ -335,6 +357,7 @@ nodes:
|
|||||||
No flattery, no preamble. Start with `## Findings`.
|
No flattery, no preamble. Start with `## Findings`.
|
||||||
prompt: |
|
prompt: |
|
||||||
Research topic: {{initial_prompt}}
|
Research topic: {{initial_prompt}}
|
||||||
|
Research depth: {{research_depth}}
|
||||||
|
|
||||||
Documentation sources (from doc search branch):
|
Documentation sources (from doc search branch):
|
||||||
{{search_output}}
|
{{search_output}}
|
||||||
@@ -343,7 +366,7 @@ nodes:
|
|||||||
{{oss_output}}
|
{{oss_output}}
|
||||||
tools:
|
tools:
|
||||||
- fetch_url_via_curl
|
- fetch_url_via_curl
|
||||||
max_iterations: 20
|
max_iterations: 45
|
||||||
state_updates:
|
state_updates:
|
||||||
findings: '{{output}}'
|
findings: '{{output}}'
|
||||||
fallback: final_format
|
fallback: final_format
|
||||||
|
|||||||
@@ -101,6 +101,14 @@ instructions: |
|
|||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
|
0. **Shared-stack discipline (parallel review lanes).** Wrap every stack boot/teardown
|
||||||
|
(docker compose, dev servers, fixed-port services) in
|
||||||
|
`flock "$HOME/.cache/coyote/review-stack.lock" <command>` — the adversary's
|
||||||
|
verification stage holds the same lock. A held port NEVER justifies removing or
|
||||||
|
stopping a container you did not create THIS RUN: wait on the lock, pick another
|
||||||
|
port, or return INCONCLUSIVE naming the conflict. Killing another lane's container
|
||||||
|
is a defect, not a remedy.
|
||||||
|
|
||||||
1. **Never modify implementation code.** Your only writes are new/updated TEST files (in the
|
1. **Never modify implementation code.** Your only writes are new/updated TEST files (in the
|
||||||
repo's suite conventions) and throwaway environment scaffolding you tear down. The
|
repo's suite conventions) and throwaway environment scaffolding you tear down. The
|
||||||
implementer owns all fixes.
|
implementer owns all fixes.
|
||||||
@@ -112,7 +120,15 @@ instructions: |
|
|||||||
update — never delete or silence it).
|
update — never delete or silence it).
|
||||||
4. **Clean, local, isolated.** Fresh ephemeral state, mocked externals, no dependence on
|
4. **Clean, local, isolated.** Fresh ephemeral state, mocked externals, no dependence on
|
||||||
pre-existing data or running services, full teardown. Bounded retries for startup only —
|
pre-existing data or running services, full teardown. Bounded retries for startup only —
|
||||||
never to mask a flaky assertion.
|
never to mask a flaky assertion. Teardown MUST NOT depend on a shell trap firing (a
|
||||||
|
killed or timed-out lane leaks containers — it has happened): pick one unique run
|
||||||
|
prefix up front (e.g. `probe-$$`), give EVERY container you create `--name <prefix>-…`
|
||||||
|
and `--label coyote-probe=<prefix>` (compose: `-p <prefix>`), prefer `docker run --rm`,
|
||||||
|
and make the LAST action before emitting your verdict an explicit sweep:
|
||||||
|
`docker ps -aq --filter label=coyote-probe=<prefix> | xargs -r docker rm -f` (compose:
|
||||||
|
`docker compose -p <prefix> down -v`). The label filter keeps the sweep scoped to YOUR
|
||||||
|
resources — rule 0 still forbids touching anyone else's. A leftover you cannot remove
|
||||||
|
is a finding to REPORT in the verdict, never to silence.
|
||||||
5. **Classify every failure** as BUG / EXPECTED-CHANGE / ENV per the skill table. The verdict
|
5. **Classify every failure** as BUG / EXPECTED-CHANGE / ENV per the skill table. The verdict
|
||||||
depends on the classification being honest.
|
depends on the classification being honest.
|
||||||
6. **Committed tests are the deliverable** alongside the verdict: write them where the repo's
|
6. **Committed tests are the deliverable** alongside the verdict: write them where the repo's
|
||||||
|
|||||||
@@ -0,0 +1,219 @@
|
|||||||
|
# Review Gauntlet
|
||||||
|
|
||||||
|
A graph-based post-implementation review gate — **"the done gate"**. An
|
||||||
|
orchestrator that must not claim *done* without green reviews spawns this agent
|
||||||
|
once instead of judging trigger rules and parsing verdict sentinels itself. The
|
||||||
|
graph structure makes skipping a lane or misreading a verdict impossible:
|
||||||
|
trigger signals are computed by a script, lanes run as isolated parallel
|
||||||
|
sub-agents, and the final verdict is a deterministic regex gate — an LLM never
|
||||||
|
gets the chance to rationalize past it.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
|
||||||
|
parse -. "fallback" .-> pfault["parse_fault<br/>(script: PIPELINE-FAULT marker)"]
|
||||||
|
pfault --> gate
|
||||||
|
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
|
||||||
|
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes add)"]
|
||||||
|
select -. "fallback" .-> dlanes["default_lanes<br/>(script: deterministic degraded<br/>selection — wider, never narrower)"]
|
||||||
|
dlanes --> build
|
||||||
|
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
|
||||||
|
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer<br/>(max_attempts: 2)"]]
|
||||||
|
madv --> radv[["adversary<br/>(max_attempts: 2)"]]
|
||||||
|
msec --> rsec[["security-reviewer<br/>(max_attempts: 2)"]]
|
||||||
|
mpb --> rpb[["probe<br/>(max_attempts: 2)"]]
|
||||||
|
rcr -. "fallback" .-> lfault["lane_fault<br/>(script: PIPELINE-FAULT lane result,<br/>collected by the map)"]
|
||||||
|
radv -. "fallback" .-> lfault
|
||||||
|
rsec -. "fallback" .-> lfault
|
||||||
|
rpb -. "fallback" .-> lfault
|
||||||
|
mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
|
||||||
|
rgate -. "_next: retry faulted lanes" .-> build
|
||||||
|
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||||
|
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
|
||||||
|
```
|
||||||
|
|
||||||
|
## Why a graph
|
||||||
|
|
||||||
|
The four independent review lanes existed before this agent — as **prompt
|
||||||
|
discipline** inside the calling orchestrator: "spawn code-reviewer when the
|
||||||
|
change is broad", "🔴 always blocks", "INCONCLUSIVE is never PASS". Prompt
|
||||||
|
discipline degrades under context pressure: a lane gets skipped, a verdict
|
||||||
|
line gets paraphrased, a missing report reads as "nothing to report". Here
|
||||||
|
those rules are *structure*:
|
||||||
|
|
||||||
|
- **Lane selection floor is deterministic** (`build_items.py`): code-review
|
||||||
|
always runs; adversary runs when a spec/plan is present; security runs on
|
||||||
|
attack-surface signals (auth/deps/exec paths or diff content) or hardened
|
||||||
|
posture; probe runs when consumer-facing surface changed *and* a local-run
|
||||||
|
recipe exists. The `select_lanes` LLM can only **widen** the set — a false
|
||||||
|
flag never removes a rule-selected lane. `forced_lanes` from the caller
|
||||||
|
override everything exactly.
|
||||||
|
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
||||||
|
over a 0-or-1-item list — an empty list means the map runs zero branches
|
||||||
|
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
||||||
|
the retry gate on equal-length paths; it falls through to the verdict
|
||||||
|
gate, or routes back to `build_items` for faulted lanes only.
|
||||||
|
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
||||||
|
sentinel is a lane FAILURE, never a pass; the critical count from the
|
||||||
|
code-review report's summary line (raw 🔴 count if absent) blocks
|
||||||
|
regardless of its verdict line; a code-review lane whose own verdict line
|
||||||
|
records a pipeline fault blocks (a degraded review is never a pass); the
|
||||||
|
adversary's own degraded-run header (the `Criteria: …` line directly under
|
||||||
|
its sentinel recording a pipeline fault, died criterion checks, or its
|
||||||
|
verdict script's crash stub) blocks as a pipeline fault,
|
||||||
|
not as a DIVERGES finding;
|
||||||
|
`NEEDS-HUMAN` without 🔴 passes but
|
||||||
|
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
|
||||||
|
with an environment note — it is never treated as PASS or FAIL.
|
||||||
|
- **Verdict independence is structural**: lanes run as isolated sub-agents
|
||||||
|
with no `teammates:` flag — no cross-lane messaging, by construction.
|
||||||
|
- **Failure fails closed, structurally** (`fallback:` routes + `PIPELINE-FAULT`
|
||||||
|
markers): every LLM node and lane retries once (`max_attempts: 2`), then a
|
||||||
|
fallback fires instead of killing the graph — and every fallback degrades
|
||||||
|
toward BLOCKED, never toward a pass. A dead lane becomes a `PIPELINE-FAULT:`
|
||||||
|
lane result (`lane_fault.py`) that the gate reports as BLOCKED *naming the
|
||||||
|
lane* — distinct from SKIPPED (not selected). A dead `parse` records a fault
|
||||||
|
and jumps straight to the gate (`parse_fault.py`). A dead `select_lanes`
|
||||||
|
degrades to a deterministic selection (`default_lanes.py`): code-review +
|
||||||
|
adversary, plus probe on consumer surface *with a local-run recipe*, plus
|
||||||
|
security on auth/deps/exec
|
||||||
|
signals or hardened posture, unioned with caller-forced lanes — wider, never
|
||||||
|
narrower. A crashed `build_items` records a fault so an all-SKIPPED gate
|
||||||
|
still blocks. Fault detection is prefix-anchored — a real review that merely
|
||||||
|
*quotes* "PIPELINE-FAULT:" flows to the normal sentinel rules. And the
|
||||||
|
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
||||||
|
`GAUNTLET: BLOCKED`, never silence.
|
||||||
|
|
||||||
|
## Fault-only re-runs
|
||||||
|
|
||||||
|
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
|
||||||
|
only lanes that **faulted** — a missing report (stood in for by a synthetic
|
||||||
|
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
|
||||||
|
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
|
||||||
|
report, or a nested graph's own degraded-run wording (the code-review verdict
|
||||||
|
line / the adversary's header line directly under its sentinel). A real verdict
|
||||||
|
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
|
||||||
|
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
|
||||||
|
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
|
||||||
|
|
||||||
|
- **Stash + restore**: every lane report that ran this pass is stashed in
|
||||||
|
`kept_results`; a re-run pass overwrites the untouched lanes' collected
|
||||||
|
results with `[]`, so their stashed reports are restored before the gate.
|
||||||
|
The verdict gate always sees each lane's *last* report.
|
||||||
|
- **Retry never narrows selection**: lane selection is not re-run. A re-run
|
||||||
|
pass rebuilds items only for the retried lanes, and the stashed results
|
||||||
|
stand in for the rest.
|
||||||
|
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
|
||||||
|
attempts or the budget lands in `review_incomplete`, and its fault flows to
|
||||||
|
the verdict gate as-is.
|
||||||
|
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
|
||||||
|
guards append to an existing `signals_error` rather than overwrite it, and
|
||||||
|
`build_items.py` preserves the prior `lanes_summary`.
|
||||||
|
|
||||||
|
A non-empty `GAUNTLET_REVIEW_INCOMPLETE:` line structurally implies
|
||||||
|
`GAUNTLET: BLOCKED`: the verdict gate forces every lane named in
|
||||||
|
`review_incomplete` to a BLOCKED row (`review incomplete after N attempt(s)`)
|
||||||
|
and a matching blocker, so a restored earlier report can never turn an
|
||||||
|
incomplete review into a pass — and malformed `review_incomplete` bookkeeping
|
||||||
|
is itself recorded as a `pipeline` fault, never dropped.
|
||||||
|
|
||||||
|
Whenever at least one lane (or the pipeline) could not complete, the first
|
||||||
|
line reads `GAUNTLET: BLOCKED` followed immediately by
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
|
||||||
|
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
|
||||||
|
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
|
||||||
|
builder, the retry gate, the default-lanes fallback, or the verdict gate
|
||||||
|
itself (the default-lanes fallback's crash still runs the widened lanes, so
|
||||||
|
`pipeline` appears alongside real lane rows). The report gains a
|
||||||
|
`## Review incomplete` section naming each exhausted lane with its
|
||||||
|
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
|
||||||
|
re-run (N > 1). The *named* entries are faults to re-run, never code
|
||||||
|
findings; the line can accompany real findings from the other lanes, which
|
||||||
|
still appear under `## Blockers` and must be fixed as usual.
|
||||||
|
|
||||||
|
## Sentinels parsed
|
||||||
|
|
||||||
|
| Lane | Sentinel | Blocks on |
|
||||||
|
|------|----------|-----------|
|
||||||
|
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
|
||||||
|
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; own degraded-run header (pipeline fault, died criterion checks, or its verdict script's crash stub); missing sentinel |
|
||||||
|
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
|
||||||
|
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
|
||||||
|
|
||||||
|
## Spawning it
|
||||||
|
|
||||||
|
```
|
||||||
|
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
|
||||||
|
|
||||||
|
Project: /abs/path/to/repo
|
||||||
|
Diff: main...HEAD (a git range, a ref, or 'worktree' for uncommitted changes)
|
||||||
|
Rigor: production Security posture: standard
|
||||||
|
|
||||||
|
Plan / acceptance criteria:
|
||||||
|
<paste the criteria the implementation must satisfy — enables the adversary lane>
|
||||||
|
|
||||||
|
Local-run recipe / usage suites:
|
||||||
|
<how to boot the service locally + where existing suites live — enables the probe lane>
|
||||||
|
|
||||||
|
Forced lanes: (optional, ADDITIVE — e.g. 'adversary, probe' run on top of the computed selection; forcing never drops the always-on floor)" \
|
||||||
|
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
|
```
|
||||||
|
|
||||||
|
Verification commands are NOT part of the prompt. They are the declared
|
||||||
|
`verification_commands` graph variable (default `'[]'`; a JSON array of shell
|
||||||
|
commands as a string, since agent variables arrive as strings), also settable
|
||||||
|
from the CLI:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
|
||||||
|
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
||||||
|
```
|
||||||
|
|
||||||
|
The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
|
||||||
|
table, the lane-selection reasons, blockers, attention items, and each lane's
|
||||||
|
full report in a collapsible section. On BLOCKED with a
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — this is an
|
||||||
|
infrastructure fault, never a code finding. Fix any non-fault blockers listed
|
||||||
|
under `## Blockers` first (the re-run retries the lane); if the line persists,
|
||||||
|
escalate to the user with three options — retry the review again (a **fresh**
|
||||||
|
gauntlet run), accept NEEDS-HUMAN and proceed without that lane (recorded in the
|
||||||
|
report / PR body), or abort — never route it through your findings-handling
|
||||||
|
rules. On BLOCKED without it: fix the findings per your own findings-handling
|
||||||
|
rules, then spawn a **fresh** gauntlet run. On PASS with *Human attention
|
||||||
|
required*: complete the work but carry those items into your final report
|
||||||
|
verbatim.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||||
|
non-trivial work — that's why code-review is unconditionally on); the
|
||||||
|
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||||
|
- Lane sub-agents get generous finite per-attempt timeouts (see the
|
||||||
|
`timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
|
||||||
|
the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
|
||||||
|
the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
|
||||||
|
is derived from that envelope in graph.yaml's comment. A lane that still
|
||||||
|
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
|
||||||
|
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
|
||||||
|
- The `verification_commands` variable is forwarded to the adversary lane as a
|
||||||
|
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
||||||
|
lands the raw value in the child's state) — never through the lane prompt
|
||||||
|
and never extracted by `parse`. The adversary's `run_checks` stage executes
|
||||||
|
those commands with a shell, which makes the declaration a trust boundary:
|
||||||
|
the prompt also carries plan text pasted from the repo under review, so
|
||||||
|
commands must never be inferred from it, and the engine merges only the keys
|
||||||
|
`parse`'s `output_schema.properties` declares, so an extra
|
||||||
|
`verification_commands` key emitted by the LLM is dropped before it reaches
|
||||||
|
state.
|
||||||
|
Undeclared stays undeclared
|
||||||
|
(`'[]'` ⇒ "none declared"); a declaration that is not a JSON array of
|
||||||
|
strings executes nothing and records a `PIPELINE-FAULT`, which the
|
||||||
|
adversary lane reports as DIVERGES and the gate as BLOCKED.
|
||||||
|
- Signals failures degrade WIDER: if git can't compute the diff, the surface
|
||||||
|
is unknown — the security lane is added (and probe, when a local-run recipe
|
||||||
|
exists) on top of caller context + forced lanes, and the report notes it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
||||||
|
auth/authn/authz/oauth but not "author") — the safe direction, since signals
|
||||||
|
only ever ADD the security lane.
|
||||||
|
- Requires the `code-reviewer`, `adversary`, `security-reviewer`, and `probe`
|
||||||
|
agents to be installed (they ship with Coyote).
|
||||||
@@ -0,0 +1,457 @@
|
|||||||
|
name: review-gauntlet
|
||||||
|
description: |
|
||||||
|
Graph-based post-implementation review gate — "the done gate". Given a diff
|
||||||
|
spec and the work's context (plan/acceptance criteria, rigor, security
|
||||||
|
posture), it deterministically computes diff signals, selects the applicable
|
||||||
|
independent review lanes — code-reviewer (always), adversary (when a
|
||||||
|
spec/plan exists), security-reviewer (attack-surface signals), probe
|
||||||
|
(consumer-facing surface with a local-run recipe) — runs the selected lanes
|
||||||
|
IN PARALLEL as isolated sub-agents, and machine-parses their verdict
|
||||||
|
sentinels into one deterministic verdict: GAUNTLET: PASS or
|
||||||
|
GAUNTLET: BLOCKED. An orchestrator that must not claim "done" without green
|
||||||
|
reviews spawns this once instead of judging trigger rules and parsing
|
||||||
|
verdicts itself — the graph structure makes skipping a lane or misreading a
|
||||||
|
verdict impossible. Invariants: lane selection can only be WIDENED by LLM
|
||||||
|
judgment, never narrowed (deterministic rules are the floor); a missing
|
||||||
|
verdict sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks
|
||||||
|
with an environment note (it is never PASS or FAIL); reviewer lanes run
|
||||||
|
verdict-independent — no cross-lane messaging.
|
||||||
|
|
||||||
|
version: '1.0'
|
||||||
|
|
||||||
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||||
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||||
|
# review-gauntlet(2) → code-reviewer(3) →
|
||||||
|
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||||
|
# This agent spawns depth-3 children; 4 = one level of headroom.
|
||||||
|
max_agent_depth: 4
|
||||||
|
max_concurrent_agents: 12
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: project_dir
|
||||||
|
description: Repository the change lives in; signals and every review lane run against it.
|
||||||
|
default: '.'
|
||||||
|
- name: verification_commands
|
||||||
|
description: |
|
||||||
|
JSON array of shell commands the CALLER declares for the deterministic
|
||||||
|
run_checks stage — e.g. '["cargo test --all"]'. Never inferred from the
|
||||||
|
prompt; empty array = none declared.
|
||||||
|
default: '[]'
|
||||||
|
|
||||||
|
settings:
|
||||||
|
max_loop_iterations: 10
|
||||||
|
log_state_snapshots: true
|
||||||
|
# A faulted lane can be re-run by retry_gate for up to 3 passes, and each
|
||||||
|
# pass's agent node retries up to max_attempts = 2 with a fresh per-attempt
|
||||||
|
# budget (src/graph/agent.rs retry_transient/bounded_attempt):
|
||||||
|
# 3 passes × 2 attempts × 11400 (max lane timeout) = 68400, + 1200 margin
|
||||||
|
# for the script stages (signals/select/build/retry gate/verdict gate)
|
||||||
|
# = 69600; the parse and select_lanes llm nodes carry no `timeout:` and
|
||||||
|
# are bounded by max_iterations.
|
||||||
|
timeout: 69600
|
||||||
|
|
||||||
|
initial_state:
|
||||||
|
retry_notes: []
|
||||||
|
diff_spec: worktree
|
||||||
|
plan_context: ''
|
||||||
|
rigor: production
|
||||||
|
security_posture: standard
|
||||||
|
probe_context: ''
|
||||||
|
extra_context: ''
|
||||||
|
forced_lanes: []
|
||||||
|
changed_files: []
|
||||||
|
file_count: 0
|
||||||
|
signals_summary: ''
|
||||||
|
signals_error: ''
|
||||||
|
touches_auth: false
|
||||||
|
touches_deps: false
|
||||||
|
touches_exec: false
|
||||||
|
touches_proto: false
|
||||||
|
touches_migrations: false
|
||||||
|
consumer_surface: false
|
||||||
|
docs_only: false
|
||||||
|
add_code_review: false
|
||||||
|
add_adversary: false
|
||||||
|
add_security: false
|
||||||
|
add_probe: false
|
||||||
|
lane_reasons: ''
|
||||||
|
parse_failure: ''
|
||||||
|
select_lanes_failure: ''
|
||||||
|
lanes_degraded: ''
|
||||||
|
code_review_items: []
|
||||||
|
adversary_items: []
|
||||||
|
security_items: []
|
||||||
|
probe_items: []
|
||||||
|
code_review_results: []
|
||||||
|
adversary_results: []
|
||||||
|
security_results: []
|
||||||
|
probe_results: []
|
||||||
|
lane_attempts: {}
|
||||||
|
kept_results: {}
|
||||||
|
retry_lanes: []
|
||||||
|
review_incomplete: []
|
||||||
|
gauntlet_started_at: 0
|
||||||
|
lanes_summary: ''
|
||||||
|
gauntlet_verdict: ''
|
||||||
|
gauntlet_report: ''
|
||||||
|
gauntlet_incomplete_line: ''
|
||||||
|
|
||||||
|
start: parse
|
||||||
|
|
||||||
|
nodes:
|
||||||
|
parse:
|
||||||
|
id: parse
|
||||||
|
type: llm
|
||||||
|
description: Extract the review request's facts from the spawn prompt. Extraction only — no judging, no lane selection.
|
||||||
|
instructions: |
|
||||||
|
You extract the facts of a review request into structured fields.
|
||||||
|
EXTRACTION ONLY — do not decide which review lanes should run.
|
||||||
|
|
||||||
|
- `diff_spec`: the git diff specification the caller named, as a BARE
|
||||||
|
revision token usable verbatim in `git diff <spec>` — a ref range
|
||||||
|
("main...HEAD", "HEAD~3..HEAD"), a single ref, or the literal
|
||||||
|
"worktree" for uncommitted changes against HEAD. NEVER prose, a
|
||||||
|
command, or a path: from "run get_diff --base origin/main" extract
|
||||||
|
"origin/main...HEAD"; from "diff origin/main...HEAD in ~/code/x"
|
||||||
|
extract "origin/main...HEAD". When the caller did not name one, use
|
||||||
|
"worktree".
|
||||||
|
- `project_dir_in`: the repository directory the caller named as holding
|
||||||
|
the change (absolute path preferred); empty string when the caller
|
||||||
|
named none.
|
||||||
|
- `plan_context`: the plan / spec / acceptance-criteria text the
|
||||||
|
implementation must satisfy, copied through VERBATIM — never
|
||||||
|
summarize it and never replace pasted text with file paths or a
|
||||||
|
sentence saying it was included; the adversary lane fails closed on
|
||||||
|
whatever you drop. Empty string when the caller supplied none.
|
||||||
|
- `rigor`: poc | prototype | production. Default "production" when
|
||||||
|
not stated.
|
||||||
|
- `security_posture`: prototype | standard | hardened. Default
|
||||||
|
"standard" when not stated.
|
||||||
|
- `probe_context`: the local-run recipe and/or existing usage-suite
|
||||||
|
pointers for black-box probing (how to boot the service, where the
|
||||||
|
suites live). Empty string when the caller supplied none.
|
||||||
|
- `forced_lanes`: lanes the caller explicitly demanded (verbatim
|
||||||
|
names). Empty array = automatic selection. Forcing is ADDITIVE
|
||||||
|
downstream: it widens the computed selection, never replaces it.
|
||||||
|
- `extra_context`: any remaining caller context reviewers should see,
|
||||||
|
condensed.
|
||||||
|
prompt: |
|
||||||
|
Extract the review-request facts from this request:
|
||||||
|
|
||||||
|
{{initial_prompt}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
diff_spec:
|
||||||
|
type: string
|
||||||
|
plan_context:
|
||||||
|
type: string
|
||||||
|
rigor:
|
||||||
|
type: string
|
||||||
|
security_posture:
|
||||||
|
type: string
|
||||||
|
probe_context:
|
||||||
|
type: string
|
||||||
|
forced_lanes:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: string
|
||||||
|
extra_context:
|
||||||
|
type: string
|
||||||
|
project_dir_in:
|
||||||
|
type: string
|
||||||
|
required: [diff_spec, plan_context, rigor, security_posture, probe_context, forced_lanes, extra_context, project_dir_in]
|
||||||
|
max_iterations: 3
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
parse_failure: '{{output}}'
|
||||||
|
fallback: parse_fault
|
||||||
|
next: signals
|
||||||
|
|
||||||
|
parse_fault:
|
||||||
|
id: parse_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — records a PIPELINE-FAULT in signals_error and jumps straight to the verdict gate, which blocks on it. Lane item lists stay their initial [] so every lane records SKIPPED.
|
||||||
|
script: scripts/parse_fault.py
|
||||||
|
timeout: 30
|
||||||
|
next: verdict_gate
|
||||||
|
|
||||||
|
signals:
|
||||||
|
id: signals
|
||||||
|
type: script
|
||||||
|
description: Deterministic diff signals — file list, attack-surface and consumer-surface indicators. Pure git; no LLM judgment.
|
||||||
|
script: scripts/signals.py
|
||||||
|
timeout: 120
|
||||||
|
next: select_lanes
|
||||||
|
|
||||||
|
select_lanes:
|
||||||
|
id: select_lanes
|
||||||
|
type: llm
|
||||||
|
description: Additive-only lane judgment on top of the deterministic rules. It can WIDEN the gauntlet, never narrow it.
|
||||||
|
instructions: |
|
||||||
|
You look at a change's deterministic diff signals and decide whether any
|
||||||
|
review lane the hard rules might have missed should ALSO run. You can
|
||||||
|
only ADD lanes — the deterministic floor (enforced downstream, not by
|
||||||
|
you) is: code-review always runs; adversary runs when a plan/spec is
|
||||||
|
present; security runs on attack-surface signals or hardened posture;
|
||||||
|
probe runs when consumer-facing surface changed and a local-run recipe
|
||||||
|
exists. Setting a flag to false NEVER removes a lane the rules selected.
|
||||||
|
|
||||||
|
Reasons to add:
|
||||||
|
- `add_security`: the diff handles external/untrusted input, touches
|
||||||
|
crypto/secrets handling, or changes trust boundaries in ways the
|
||||||
|
path-based signals missed.
|
||||||
|
- `add_adversary`: the request references behavior it must preserve or
|
||||||
|
criteria scattered in prose that amount to a spec, even without a
|
||||||
|
formal plan.
|
||||||
|
- `add_probe`: the change alters observable API/CLI behavior even
|
||||||
|
though the path signals missed it (only useful when a local-run
|
||||||
|
recipe exists — the builder skips it otherwise, noting why).
|
||||||
|
- `add_code_review`: always selected by the rules; flag exists only
|
||||||
|
for forced-lane configurations.
|
||||||
|
|
||||||
|
Put one line per added lane into `lane_reasons`. When nothing needs
|
||||||
|
adding, all flags false and empty reasons is the correct answer.
|
||||||
|
prompt: |
|
||||||
|
Change signals (deterministic):
|
||||||
|
- diff spec: {{diff_spec}}
|
||||||
|
- files changed: {{file_count}}
|
||||||
|
- changed files (first 200): {{changed_files}}
|
||||||
|
- touches auth paths/content: {{touches_auth}}
|
||||||
|
- touches dependency manifests: {{touches_deps}}
|
||||||
|
- adds exec/shell/subprocess calls: {{touches_exec}}
|
||||||
|
- touches protobuf/contracts: {{touches_proto}}
|
||||||
|
- touches migrations: {{touches_migrations}}
|
||||||
|
- consumer-facing surface paths: {{consumer_surface}}
|
||||||
|
- docs-only diff: {{docs_only}}
|
||||||
|
- {{signals_summary}}
|
||||||
|
{{signals_error}}
|
||||||
|
|
||||||
|
Caller context:
|
||||||
|
- rigor: {{rigor}}; security posture: {{security_posture}}
|
||||||
|
- plan/spec present: {{plan_context}}
|
||||||
|
- probe recipe present: {{probe_context}}
|
||||||
|
- extra: {{extra_context}}
|
||||||
|
tools: []
|
||||||
|
output_schema:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
add_code_review:
|
||||||
|
type: boolean
|
||||||
|
add_adversary:
|
||||||
|
type: boolean
|
||||||
|
add_security:
|
||||||
|
type: boolean
|
||||||
|
add_probe:
|
||||||
|
type: boolean
|
||||||
|
lane_reasons:
|
||||||
|
type: string
|
||||||
|
required: [add_code_review, add_adversary, add_security, add_probe, lane_reasons]
|
||||||
|
max_iterations: 3
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
select_lanes_failure: '{{output}}'
|
||||||
|
fallback: default_lanes
|
||||||
|
next: build_items
|
||||||
|
|
||||||
|
default_lanes:
|
||||||
|
id: default_lanes
|
||||||
|
type: script
|
||||||
|
description: Deterministic lane-selection fallback (reached only via select_lanes' fallback) — when the additive LLM judgment dies, forces code-review + adversary, plus probe when consumer_surface AND a probe_context (local-run recipe) is present, plus security on attack-surface signals (auth/deps/exec) or hardened posture, unioned with any caller-forced lanes. Selection degrades WIDER, never narrower; the degradation note flows into lanes_summary via build_items.
|
||||||
|
script: scripts/default_lanes.py
|
||||||
|
timeout: 30
|
||||||
|
next: build_items
|
||||||
|
|
||||||
|
build_items:
|
||||||
|
id: build_items
|
||||||
|
type: script
|
||||||
|
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes are ADDITIVE (selection only ever widens; unavailable signals widen it too). Emits one 0-or-1-item list per lane; an empty list skips that lane's map. On a re-run pass (retry_lanes non-empty) it skips selection and emits items only for those lanes, each carrying its attempt number.
|
||||||
|
script: scripts/build_items.py
|
||||||
|
timeout: 30
|
||||||
|
next: [map_code_review, map_adversary, map_security, map_probe]
|
||||||
|
|
||||||
|
map_code_review:
|
||||||
|
id: map_code_review
|
||||||
|
type: map
|
||||||
|
over: '{{code_review_items}}'
|
||||||
|
as: lane_ctx
|
||||||
|
branch: run_code_review
|
||||||
|
output_key: lane_out
|
||||||
|
collect_into: code_review_results
|
||||||
|
max_concurrency: 1
|
||||||
|
next: retry_gate
|
||||||
|
|
||||||
|
map_adversary:
|
||||||
|
id: map_adversary
|
||||||
|
type: map
|
||||||
|
over: '{{adversary_items}}'
|
||||||
|
as: lane_ctx
|
||||||
|
branch: run_adversary
|
||||||
|
output_key: lane_out
|
||||||
|
collect_into: adversary_results
|
||||||
|
max_concurrency: 1
|
||||||
|
next: retry_gate
|
||||||
|
|
||||||
|
map_security:
|
||||||
|
id: map_security
|
||||||
|
type: map
|
||||||
|
over: '{{security_items}}'
|
||||||
|
as: lane_ctx
|
||||||
|
branch: run_security
|
||||||
|
output_key: lane_out
|
||||||
|
collect_into: security_results
|
||||||
|
max_concurrency: 1
|
||||||
|
next: retry_gate
|
||||||
|
|
||||||
|
map_probe:
|
||||||
|
id: map_probe
|
||||||
|
type: map
|
||||||
|
over: '{{probe_items}}'
|
||||||
|
as: lane_ctx
|
||||||
|
branch: run_probe
|
||||||
|
output_key: lane_out
|
||||||
|
collect_into: probe_results
|
||||||
|
max_concurrency: 1
|
||||||
|
next: retry_gate
|
||||||
|
|
||||||
|
run_code_review:
|
||||||
|
id: run_code_review
|
||||||
|
type: agent
|
||||||
|
description: Independent code-quality review lane.
|
||||||
|
agent: code-reviewer
|
||||||
|
# Exceeds the code-reviewer graph's own settings.timeout (10800) by 600 so
|
||||||
|
# the child's graph timeout, not this lane timeout, is the binding bound.
|
||||||
|
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
|
||||||
|
# timeouts) is not derived here — deliberately out of scope for this
|
||||||
|
# change; tracked as a follow-up.
|
||||||
|
timeout: 11400
|
||||||
|
prompt: |
|
||||||
|
Review the code changes in {{project_dir}} (diff: {{diff_spec}}) as an
|
||||||
|
independent post-implementation review.
|
||||||
|
|
||||||
|
Quality bar: rigor={{rigor}}.
|
||||||
|
|
||||||
|
Plan / acceptance-criteria context (may be empty):
|
||||||
|
{{plan_context}}
|
||||||
|
|
||||||
|
{{extra_context}}
|
||||||
|
|
||||||
|
Close your report with your Review Verdict line.
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
lane_out: '{{output}}'
|
||||||
|
fallback: lane_fault
|
||||||
|
|
||||||
|
run_adversary:
|
||||||
|
id: run_adversary
|
||||||
|
type: agent
|
||||||
|
description: Plan-conformance review lane.
|
||||||
|
agent: adversary
|
||||||
|
# Exceeds the adversary graph's own settings.timeout (9000) by 600 so the
|
||||||
|
# child's graph timeout, not this lane timeout, is the binding bound.
|
||||||
|
# NOTE: the child's own internal retry envelope (its nodes' max_attempts ×
|
||||||
|
# timeouts) is not derived here — deliberately out of scope for this
|
||||||
|
# change; tracked as a follow-up.
|
||||||
|
timeout: 9600
|
||||||
|
inputs:
|
||||||
|
verification_commands: '{{verification_commands}}'
|
||||||
|
prompt: |
|
||||||
|
Adversarial plan-conformance review in {{project_dir}}.
|
||||||
|
|
||||||
|
The implementation to check: diff {{diff_spec}}.
|
||||||
|
|
||||||
|
The spec / plan / acceptance criteria it must satisfy:
|
||||||
|
{{plan_context}}
|
||||||
|
|
||||||
|
Verification commands arrive as a structured input (the verification_commands variable), not in this prompt.
|
||||||
|
|
||||||
|
{{extra_context}}
|
||||||
|
|
||||||
|
Return your ADVERSARIAL_REVIEW verdict.
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
lane_out: '{{output}}'
|
||||||
|
fallback: lane_fault
|
||||||
|
|
||||||
|
run_security:
|
||||||
|
id: run_security
|
||||||
|
type: agent
|
||||||
|
description: Security review lane.
|
||||||
|
agent: security-reviewer
|
||||||
|
timeout: 7200
|
||||||
|
prompt: |
|
||||||
|
Security review of the code changes in {{project_dir}} (diff: {{diff_spec}}).
|
||||||
|
|
||||||
|
Security posture: {{security_posture}}.
|
||||||
|
|
||||||
|
{{extra_context}}
|
||||||
|
|
||||||
|
Return your SECURITY_REVIEW verdict.
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
lane_out: '{{output}}'
|
||||||
|
fallback: lane_fault
|
||||||
|
|
||||||
|
run_probe:
|
||||||
|
id: run_probe
|
||||||
|
type: agent
|
||||||
|
description: Black-box usage-pattern verification lane.
|
||||||
|
agent: probe
|
||||||
|
timeout: 7200
|
||||||
|
prompt: |
|
||||||
|
Usage-pattern verification (black-box probe) of the changes in
|
||||||
|
{{project_dir}} (diff: {{diff_spec}}).
|
||||||
|
|
||||||
|
Local-run recipe / existing usage suites:
|
||||||
|
{{probe_context}}
|
||||||
|
|
||||||
|
Plan / acceptance-criteria context:
|
||||||
|
{{plan_context}}
|
||||||
|
|
||||||
|
The workspace may contain test files from an aborted earlier attempt of
|
||||||
|
this lane; reconcile rather than duplicate.
|
||||||
|
|
||||||
|
Shared-stack rule: wrap any stack boot/teardown in
|
||||||
|
`flock "$HOME/.cache/coyote/review-stack.lock" <command>` (the adversary's
|
||||||
|
verification stage holds the same lock), and NEVER remove or stop containers
|
||||||
|
you did not create this run — a held port means wait, re-port, or INCONCLUSIVE.
|
||||||
|
|
||||||
|
Derive expected behaviors from the plan/contract BEFORE reading
|
||||||
|
implementation code, run existing suites first, then author spec-first
|
||||||
|
tests for the uncovered delta. Return your USAGE_PROBE verdict.
|
||||||
|
max_attempts: 2
|
||||||
|
state_updates:
|
||||||
|
lane_out: '{{output}}'
|
||||||
|
fallback: lane_fault
|
||||||
|
|
||||||
|
lane_fault:
|
||||||
|
id: lane_fault
|
||||||
|
type: script
|
||||||
|
description: Fail-closed lane fault marker (reached only via the run_* lanes' fallback routes; shared across the four map branches). Normalizes the engine's "Agent node failed" text in lane_out into a PIPELINE-FAULT lane result the map collects; verdict_gate turns it into BLOCKED — a degraded lane is never a pass.
|
||||||
|
script: scripts/lane_fault.py
|
||||||
|
timeout: 30
|
||||||
|
|
||||||
|
retry_gate:
|
||||||
|
id: retry_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic fault-only re-run gate. Stashes every lane report that ran this pass, restores reports a re-run pass overwrote with [] (a lane not being re-run maps over nothing), classifies each lane that ran as completed / faulted / skipped, counts attempts, and routes back into build_items for faulted lanes that still have attempts left inside the wall-clock budget; otherwise it records them in review_incomplete and falls through to verdict_gate. A real verdict is never re-run, and a gate crash never retries — it records a PIPELINE-FAULT and falls through.
|
||||||
|
script: scripts/retry_gate.py
|
||||||
|
timeout: 30
|
||||||
|
next: verdict_gate
|
||||||
|
|
||||||
|
verdict_gate:
|
||||||
|
id: verdict_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
|
||||||
|
script: scripts/verdict_gate.py
|
||||||
|
timeout: 60
|
||||||
|
next: done
|
||||||
|
|
||||||
|
done:
|
||||||
|
id: done
|
||||||
|
type: end
|
||||||
|
output: |
|
||||||
|
GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
|
||||||
|
|
||||||
|
{{gauntlet_report}}
|
||||||
@@ -0,0 +1,236 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic lane builder for the review gauntlet.
|
||||||
|
|
||||||
|
Selection = hard rules (the floor) ∪ LLM additions ∪ caller-forced lanes.
|
||||||
|
Forcing is ADDITIVE: it can only widen the computed selection — honoring a
|
||||||
|
forced list as the exact set once dropped the always-on code-review lane in
|
||||||
|
a live run. Emits one 0-or-1-item list per lane: an empty list makes that
|
||||||
|
lane's map node run zero branches (map-over-nothing), which IS the skip
|
||||||
|
mechanism — no dynamic routing anywhere.
|
||||||
|
|
||||||
|
The LLM's add_* flags can only ADD lanes. A flag that is false never
|
||||||
|
removes a lane the hard rules selected.
|
||||||
|
|
||||||
|
Unavailable diff signals (malformed spec, git failure) mean the surface is
|
||||||
|
UNKNOWN — selection degrades WIDER (security on; probe when a recipe
|
||||||
|
exists), never narrower.
|
||||||
|
|
||||||
|
A re-run pass (retry_gate emitted `retry_lanes`) skips selection: only the
|
||||||
|
named lanes get an item, stamped with their attempt number, and every other
|
||||||
|
lane maps over nothing — retry_gate restores their earlier reports.
|
||||||
|
|
||||||
|
Fail-closed: a builder crash emits all four lists empty plus a
|
||||||
|
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
|
||||||
|
never kills the graph.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
|
||||||
|
# canonicalize the same way on the degraded path as it does here.
|
||||||
|
ALIASES = {
|
||||||
|
"code-review": "code-review",
|
||||||
|
"code_review": "code-review",
|
||||||
|
"code-reviewer": "code-review",
|
||||||
|
"adversary": "adversary",
|
||||||
|
"adversarial": "adversary",
|
||||||
|
"adversarial-review": "adversary",
|
||||||
|
"security": "security",
|
||||||
|
"security-review": "security",
|
||||||
|
"security-reviewer": "security",
|
||||||
|
"probe": "probe",
|
||||||
|
"usage": "probe",
|
||||||
|
"usage-pattern": "probe",
|
||||||
|
"usage-pattern-testing": "probe",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Keep in sync with retry_gate.LANES
|
||||||
|
LANE_KEYS = {
|
||||||
|
"code-review": "code_review_items",
|
||||||
|
"adversary": "adversary_items",
|
||||||
|
"security": "security_items",
|
||||||
|
"probe": "probe_items",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Filled by main() so the crash guard can append to an upstream fault
|
||||||
|
# instead of overwriting it, and keep the first pass's lanes_summary.
|
||||||
|
crash_context = {"signals_error": "", "lanes_summary": ""}
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
prior = state.get("signals_error")
|
||||||
|
crash_context["signals_error"] = prior if isinstance(prior, str) else ""
|
||||||
|
# retry_gate's wall-clock budget is measured from this stamp; signals.py
|
||||||
|
# is the normal stamper, this is the defensive fallback.
|
||||||
|
stamp = {} if state.get("gauntlet_started_at") else {"gauntlet_started_at": time.time()}
|
||||||
|
|
||||||
|
def text(key):
|
||||||
|
v = state.get(key)
|
||||||
|
return v.strip() if isinstance(v, str) else ""
|
||||||
|
|
||||||
|
crash_context["lanes_summary"] = text("lanes_summary")
|
||||||
|
retry_lanes = [lane for lane in (state.get("retry_lanes") or []) if lane in LANE_KEYS]
|
||||||
|
if retry_lanes:
|
||||||
|
attempts = state.get("lane_attempts") or {}
|
||||||
|
retried = [(lane, int(attempts.get(lane) or 0) + 1) for lane in retry_lanes]
|
||||||
|
note = "- retried: [" + ", ".join(f"{lane} (attempt {n})" for lane, n in retried) + "]"
|
||||||
|
summary = text("lanes_summary")
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
**{key: [] for key in LANE_KEYS.values()},
|
||||||
|
**{
|
||||||
|
LANE_KEYS[lane]: [{"lane": lane, "attempt": n}] for lane, n in retried
|
||||||
|
},
|
||||||
|
"lanes_summary": f"{summary}\n{note}" if summary else note,
|
||||||
|
"retry_lanes": [],
|
||||||
|
**stamp,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
forced_raw = state.get("forced_lanes") or []
|
||||||
|
forced, unknown = [], []
|
||||||
|
for lane in forced_raw:
|
||||||
|
canon = ALIASES.get(str(lane).strip().lower())
|
||||||
|
(forced.append(canon) if canon else unknown.append(str(lane)))
|
||||||
|
|
||||||
|
reasons = []
|
||||||
|
lanes = {"code-review"}
|
||||||
|
reasons.append(
|
||||||
|
"code-review: always on — the gauntlet is only spawned for non-trivial work"
|
||||||
|
)
|
||||||
|
if text("plan_context"):
|
||||||
|
lanes.add("adversary")
|
||||||
|
reasons.append("adversary: a spec/plan/acceptance-criteria context is present")
|
||||||
|
if (
|
||||||
|
state.get("touches_auth")
|
||||||
|
or state.get("touches_deps")
|
||||||
|
or state.get("touches_exec")
|
||||||
|
or state.get("security_posture") == "hardened"
|
||||||
|
):
|
||||||
|
lanes.add("security")
|
||||||
|
reasons.append(
|
||||||
|
"security: attack-surface signals (auth/deps/exec) or hardened posture"
|
||||||
|
)
|
||||||
|
if state.get("consumer_surface") and text("probe_context"):
|
||||||
|
lanes.add("probe")
|
||||||
|
reasons.append(
|
||||||
|
"probe: consumer-facing surface changed and a local-run recipe/suite pointer exists"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Unavailable signals (malformed diff spec, git failure) = surface UNKNOWN.
|
||||||
|
# Degrade WIDER, never narrower — a run without signals once silently
|
||||||
|
# dropped the security and probe lanes.
|
||||||
|
if text("signals_summary") == "signals unavailable":
|
||||||
|
if "security" not in lanes:
|
||||||
|
lanes.add("security")
|
||||||
|
reasons.append(
|
||||||
|
"security: diff signals unavailable — surface unknown, degrading wider"
|
||||||
|
)
|
||||||
|
if "probe" not in lanes and text("probe_context"):
|
||||||
|
lanes.add("probe")
|
||||||
|
reasons.append(
|
||||||
|
"probe: diff signals unavailable and a local-run recipe/suite pointer "
|
||||||
|
"exists — degrading wider"
|
||||||
|
)
|
||||||
|
|
||||||
|
# LLM judgment is additive-only.
|
||||||
|
llm_why = text("lane_reasons")[:300]
|
||||||
|
for flag, lane in (
|
||||||
|
("add_code_review", "code-review"),
|
||||||
|
("add_adversary", "adversary"),
|
||||||
|
("add_security", "security"),
|
||||||
|
("add_probe", "probe"),
|
||||||
|
):
|
||||||
|
if state.get(flag) and lane not in lanes:
|
||||||
|
if lane == "probe" and not text("probe_context"):
|
||||||
|
reasons.append(
|
||||||
|
"probe: requested by lane judgment but skipped — no local-run "
|
||||||
|
"recipe/suite pointer (a probe without one is guaranteed INCONCLUSIVE)"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if lane == "adversary" and not text("plan_context"):
|
||||||
|
reasons.append(
|
||||||
|
"adversary: requested by lane judgment but skipped — no spec/plan "
|
||||||
|
"to check conformance against"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
lanes.add(lane)
|
||||||
|
reasons.append(f"{lane}: added by lane-selection judgment — {llm_why}")
|
||||||
|
|
||||||
|
# Caller-forced lanes are ADDITIVE: they widen the computed selection,
|
||||||
|
# never replace it. (Honoring them as the exact set dropped code-review —
|
||||||
|
# documented as always-on — and adversary in a live run.) Forcing also
|
||||||
|
# bypasses the context guards above: the caller demanded the lane, so it
|
||||||
|
# runs, with the expectation named in the summary.
|
||||||
|
if forced:
|
||||||
|
newly = sorted(set(forced) - lanes)
|
||||||
|
lanes |= set(forced)
|
||||||
|
reasons.append(
|
||||||
|
f"forced lanes (additive): {sorted(set(forced))}"
|
||||||
|
+ (f" — newly added: {newly}" if newly else " — all already selected")
|
||||||
|
)
|
||||||
|
if "probe" in forced and not text("probe_context"):
|
||||||
|
reasons.append(
|
||||||
|
"probe: forced without a local-run recipe/suite pointer — expect INCONCLUSIVE"
|
||||||
|
)
|
||||||
|
if "adversary" in forced and not text("plan_context"):
|
||||||
|
reasons.append(
|
||||||
|
"adversary: forced without spec/plan context — it fails closed "
|
||||||
|
"(DIVERGES) without extractable criteria"
|
||||||
|
)
|
||||||
|
|
||||||
|
if unknown:
|
||||||
|
reasons.append(f"ignored unknown forced lane name(s): {unknown}")
|
||||||
|
|
||||||
|
# A degraded lane selection (default_lanes fallback) must survive into the
|
||||||
|
# final report; lanes_summary is the only channel this script doesn't
|
||||||
|
# overwrite silently.
|
||||||
|
degraded = text("lanes_degraded")
|
||||||
|
if degraded:
|
||||||
|
reasons.insert(0, degraded)
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"code_review_items": [{"lane": "code-review"}] if "code-review" in lanes else [],
|
||||||
|
"adversary_items": [{"lane": "adversary"}] if "adversary" in lanes else [],
|
||||||
|
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
||||||
|
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
||||||
|
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
||||||
|
**stamp,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
|
||||||
|
fault = f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run"
|
||||||
|
if crash_context["signals_error"]:
|
||||||
|
fault = f"{crash_context['signals_error']}; {fault}"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"code_review_items": [],
|
||||||
|
"adversary_items": [],
|
||||||
|
"security_items": [],
|
||||||
|
"probe_items": [],
|
||||||
|
"lanes_summary": crash_context["lanes_summary"],
|
||||||
|
"signals_error": fault,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic lane-selection fallback for the review gauntlet.
|
||||||
|
|
||||||
|
Reached ONLY via select_lanes' `fallback:` route, when the additive LLM
|
||||||
|
judgment dies after retries. Emits `forced_lanes` — code-review + adversary,
|
||||||
|
plus probe when the deterministic signals flagged a consumer-facing surface
|
||||||
|
AND a local-run recipe exists (build_items gates probe the same way — a
|
||||||
|
probe without a recipe is INCONCLUSIVE by construction),
|
||||||
|
plus security when the attack-surface signals (auth/deps/exec) or a hardened
|
||||||
|
posture demand it, unioned with any lanes the caller forced (canonicalized
|
||||||
|
with the same aliases build_items uses) — which build_items honors
|
||||||
|
ADDITIVELY, so selection degrades WIDER, never narrower. Unavailable diff
|
||||||
|
signals also widen: unknown surface adds security (and probe when a
|
||||||
|
local-run recipe exists). The degradation note goes into
|
||||||
|
`lanes_degraded`; build_items folds it into lanes_summary so the final
|
||||||
|
report names the degradation.
|
||||||
|
|
||||||
|
Fail-closed: if the fallback itself crashes it emits the widest safe
|
||||||
|
selection (defaults ∪ security) plus a PIPELINE-FAULT in signals_error so
|
||||||
|
verdict_gate blocks — a crashed fallback never narrows selection silently.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
# Keep in sync with build_items.ALIASES — a caller-forced lane must
|
||||||
|
# canonicalize the same way here as it would on the non-degraded path.
|
||||||
|
ALIASES = {
|
||||||
|
"code-review": "code-review",
|
||||||
|
"code_review": "code-review",
|
||||||
|
"code-reviewer": "code-review",
|
||||||
|
"adversary": "adversary",
|
||||||
|
"adversarial": "adversary",
|
||||||
|
"adversarial-review": "adversary",
|
||||||
|
"security": "security",
|
||||||
|
"security-review": "security",
|
||||||
|
"security-reviewer": "security",
|
||||||
|
"probe": "probe",
|
||||||
|
"usage": "probe",
|
||||||
|
"usage-pattern": "probe",
|
||||||
|
"usage-pattern-testing": "probe",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stable emission order keeps the fallback deterministic.
|
||||||
|
LANE_ORDER = ["code-review", "adversary", "security", "probe"]
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
lanes = {"code-review", "adversary"}
|
||||||
|
probe_context = state.get("probe_context")
|
||||||
|
if (
|
||||||
|
state.get("consumer_surface")
|
||||||
|
and isinstance(probe_context, str)
|
||||||
|
and probe_context.strip()
|
||||||
|
):
|
||||||
|
lanes.add("probe")
|
||||||
|
# The deterministic security signals are already in state when this
|
||||||
|
# fallback runs; a degraded selection must never drop the security lane
|
||||||
|
# build_items' hard rules would have selected.
|
||||||
|
if (
|
||||||
|
state.get("touches_auth")
|
||||||
|
or state.get("touches_deps")
|
||||||
|
or state.get("touches_exec")
|
||||||
|
or state.get("security_posture") == "hardened"
|
||||||
|
):
|
||||||
|
lanes.add("security")
|
||||||
|
# Unavailable signals = surface UNKNOWN → widen (mirrors build_items).
|
||||||
|
if (state.get("signals_summary") or "") == "signals unavailable":
|
||||||
|
lanes.add("security")
|
||||||
|
if isinstance(probe_context, str) and probe_context.strip():
|
||||||
|
lanes.add("probe")
|
||||||
|
# Caller-forced lanes survive degradation — union, never overwrite.
|
||||||
|
for lane in state.get("forced_lanes") or []:
|
||||||
|
if canon := ALIASES.get(str(lane).strip().lower()):
|
||||||
|
lanes.add(canon)
|
||||||
|
forced = [lane for lane in LANE_ORDER if lane in lanes]
|
||||||
|
raw = state.get("select_lanes_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
note = "lane selection degraded to deterministic defaults"
|
||||||
|
if detail:
|
||||||
|
note += f" — {detail}"
|
||||||
|
print(json.dumps({"forced_lanes": forced, "lanes_degraded": note}))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fallback itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"forced_lanes": ["code-review", "adversary", "security"],
|
||||||
|
"lanes_degraded": "lane selection degraded to deterministic "
|
||||||
|
f"defaults (fallback script error: {e})",
|
||||||
|
"signals_error": (
|
||||||
|
f"PIPELINE-FAULT: lane-selection fallback crashed — {e}; "
|
||||||
|
"lanes widened to defaults ∪ security"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed lane fault marker for the review gauntlet.
|
||||||
|
|
||||||
|
Reached ONLY via the run_* lane nodes' `fallback:` routes inside their map
|
||||||
|
branches. Grounded in engine behavior: on agent-node failure the engine
|
||||||
|
writes "Agent node failed: {e:#}" into `lane_out` via the node's
|
||||||
|
state_updates BEFORE routing to the fallback, so the raw failure text is
|
||||||
|
already in state. This node normalizes it into a
|
||||||
|
"PIPELINE-FAULT: <lane> lane failed after retries — …" marker that the map
|
||||||
|
collects as the lane result; verdict_gate turns any PIPELINE-FAULT lane
|
||||||
|
result into BLOCKED — a degraded lane can never read as a pass.
|
||||||
|
|
||||||
|
Lane identity comes from `lane_ctx` (the map item build_items emitted, which
|
||||||
|
carries a `lane` field); a missing/malformed item degrades to "unknown".
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
lane = "unknown"
|
||||||
|
lane_ctx = state.get("lane_ctx")
|
||||||
|
if isinstance(lane_ctx, dict):
|
||||||
|
name = lane_ctx.get("lane")
|
||||||
|
if isinstance(name, str) and name.strip():
|
||||||
|
lane = name.strip()
|
||||||
|
raw = state.get("lane_out")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{"lane_out": f"PIPELINE-FAULT: {lane} lane failed after retries — {detail}"}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"lane_out": "PIPELINE-FAULT: lane failed after retries — "
|
||||||
|
f"fault-marker script error: {e}"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail-closed marker for a dead parse stage of the review gauntlet.
|
||||||
|
|
||||||
|
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||||
|
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||||
|
this node records the failure as a PIPELINE-FAULT in `signals_error` and the
|
||||||
|
graph jumps straight to the verdict gate. Every lane item list is still its
|
||||||
|
initial [] so every lane records SKIPPED, and verdict_gate's signals_error
|
||||||
|
rule makes the verdict BLOCKED — a gauntlet that could not even parse its
|
||||||
|
request never reads as a pass.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
MAX_DETAIL_CHARS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
raw = state.get("parse_failure")
|
||||||
|
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||||
|
if len(detail) > MAX_DETAIL_CHARS:
|
||||||
|
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
|
||||||
|
f"gauntlet: {detail}"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"signals_error": "PIPELINE-FAULT: parse failed — cannot run the "
|
||||||
|
f"gauntlet: fault-marker script error: {e}"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic fault-only re-run gate for the review gauntlet.
|
||||||
|
|
||||||
|
Sits between the four lane maps and the verdict gate. Every lane map runs
|
||||||
|
on every pass — a lane that is not being re-run maps over [] — so a re-run
|
||||||
|
pass overwrites the untouched lanes' collected results with []. The gate's
|
||||||
|
hard rules:
|
||||||
|
|
||||||
|
- every lane that ran this pass is stashed in kept_results, completed
|
||||||
|
and faulted alike, and a lane whose map ran nothing this pass gets
|
||||||
|
its stashed report restored — the verdict gate always sees each
|
||||||
|
lane's LAST report;
|
||||||
|
- a lane is FAULTED only when its report is missing (the map ran but
|
||||||
|
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
|
||||||
|
the verdict gate never reads it as SKIPPED), non-string, the
|
||||||
|
lane_fault.py marker, sentinel-less, or carries the nested graph's own
|
||||||
|
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
|
||||||
|
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
|
||||||
|
- a faulted lane is re-run only while it has attempts left AND the
|
||||||
|
gauntlet is inside its wall-clock budget; otherwise it is recorded in
|
||||||
|
review_incomplete and its fault flows to the verdict gate as-is;
|
||||||
|
- malformed review_incomplete bookkeeping (not a list, or non-str
|
||||||
|
entries) keeps its str entries and records a PIPELINE-FAULT in
|
||||||
|
signals_error — never silently dropped, never split into lane names;
|
||||||
|
- the gate itself must never retry by accident: any internal error
|
||||||
|
records a PIPELINE-FAULT in signals_error and falls through to the
|
||||||
|
verdict gate, which blocks on it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
|
||||||
|
# = 45600; 600 s of slack below that.
|
||||||
|
MAX_RETRY_ELAPSED_SECS = 45000
|
||||||
|
MAX_ATTEMPTS = 3
|
||||||
|
|
||||||
|
# lane name -> (items key, results key)
|
||||||
|
LANES = {
|
||||||
|
"code-review": ("code_review_items", "code_review_results"),
|
||||||
|
"adversary": ("adversary_items", "adversary_results"),
|
||||||
|
"security": ("security_items", "security_results"),
|
||||||
|
"probe": ("probe_items", "probe_results"),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Must stay identical to verdict_gate.py's sentinel regexes: a lane this
|
||||||
|
# gate calls completed must be one the verdict gate can parse, or a drift
|
||||||
|
# would re-run lanes the verdict gate accepts (or accept lanes it cannot
|
||||||
|
# read). A cross-pin test compares the literals in both files.
|
||||||
|
SENTINELS = {
|
||||||
|
"code-review": r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)",
|
||||||
|
"adversary": r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)",
|
||||||
|
"security": r"SECURITY_REVIEW:\s*(PASS|FAIL)",
|
||||||
|
"probe": r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Filled by main() so the crash guard can append to an upstream fault
|
||||||
|
# instead of overwriting it, and report the lanes it had already declined.
|
||||||
|
crash_context = {"signals_error": "", "review_incomplete": []}
|
||||||
|
|
||||||
|
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
|
||||||
|
ADVERSARY_DEGRADED_HEADER = re.compile(
|
||||||
|
r"Criteria: .*("
|
||||||
|
r"degraded run: pipeline fault recorded"
|
||||||
|
r"|degraded run: \d+ criterion check\(s\) died"
|
||||||
|
r"|verdict computation error:"
|
||||||
|
r")"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def adversary_degraded(report):
|
||||||
|
# The adversary's verdict.py emits its degraded-run header on the line
|
||||||
|
# directly under the FIRST sentinel; anchoring there keeps a report that
|
||||||
|
# merely quotes the wording later (observations, complaints) a real verdict.
|
||||||
|
lines = report.splitlines()
|
||||||
|
for i, ln in enumerate(lines):
|
||||||
|
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
|
||||||
|
nxt = lines[i + 1] if i + 1 < len(lines) else ""
|
||||||
|
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def nested_degraded(lane, report):
|
||||||
|
# The nested graphs complete with a sentinel even when a stage inside
|
||||||
|
# them died; their own degraded-run wording marks a partial review,
|
||||||
|
# which is not a completed one. Same anchoring as verdict_gate.py.
|
||||||
|
if lane == "code-review":
|
||||||
|
verdict_line = re.search(r"^\*\*Verdict: .*$", report, re.MULTILINE)
|
||||||
|
return bool(
|
||||||
|
verdict_line
|
||||||
|
and re.search(
|
||||||
|
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
|
||||||
|
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
|
||||||
|
verdict_line.group(0),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if lane == "adversary":
|
||||||
|
return adversary_degraded(report)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def is_faulted(lane, report):
|
||||||
|
if not isinstance(report, str):
|
||||||
|
return True
|
||||||
|
if report.strip().startswith("PIPELINE-FAULT:"):
|
||||||
|
return True
|
||||||
|
if not re.search(SENTINELS[lane], report):
|
||||||
|
return True
|
||||||
|
return nested_degraded(lane, report)
|
||||||
|
|
||||||
|
|
||||||
|
def attempt_count(attempts, lane):
|
||||||
|
try:
|
||||||
|
return int(attempts.get(lane) or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
prior = state.get("signals_error")
|
||||||
|
signals_error = prior if isinstance(prior, str) else ""
|
||||||
|
crash_context["signals_error"] = signals_error
|
||||||
|
out = {}
|
||||||
|
declared = state.get("review_incomplete")
|
||||||
|
if isinstance(declared, list):
|
||||||
|
incomplete = [name for name in declared if isinstance(name, str)]
|
||||||
|
malformed = len(incomplete) != len(declared)
|
||||||
|
else:
|
||||||
|
incomplete = []
|
||||||
|
malformed = declared is not None
|
||||||
|
if malformed:
|
||||||
|
fault = (
|
||||||
|
"PIPELINE-FAULT: malformed review_incomplete bookkeeping "
|
||||||
|
f"({repr(declared)[:80]})"
|
||||||
|
)
|
||||||
|
signals_error = f"{signals_error}; {fault}" if signals_error else fault
|
||||||
|
crash_context["signals_error"] = signals_error
|
||||||
|
out["signals_error"] = signals_error
|
||||||
|
crash_context["review_incomplete"] = incomplete
|
||||||
|
raw_attempts = state.get("lane_attempts")
|
||||||
|
attempts = dict(raw_attempts) if isinstance(raw_attempts, dict) else {}
|
||||||
|
kept = dict(state.get("kept_results") or {})
|
||||||
|
faulted = []
|
||||||
|
|
||||||
|
for lane, (items_key, results_key) in LANES.items():
|
||||||
|
results = state.get(results_key) or []
|
||||||
|
if not state.get(items_key):
|
||||||
|
if not results and lane in kept:
|
||||||
|
out[results_key] = [kept[lane]]
|
||||||
|
continue
|
||||||
|
report = results[0] if results else None
|
||||||
|
if report is None:
|
||||||
|
report = f"PIPELINE-FAULT: {lane} lane produced no report"
|
||||||
|
out[results_key] = [report]
|
||||||
|
kept[lane] = report
|
||||||
|
if is_faulted(lane, report):
|
||||||
|
faulted.append(lane)
|
||||||
|
attempts[lane] = attempt_count(attempts, lane) + 1
|
||||||
|
|
||||||
|
# A missing/zero stamp reads as an ancient start and declines every
|
||||||
|
# retry — the safe direction when the clock's origin is unknown.
|
||||||
|
started = state.get("gauntlet_started_at") or 0
|
||||||
|
within_budget = (time.time() - float(started)) <= MAX_RETRY_ELAPSED_SECS
|
||||||
|
retry = [lane for lane in faulted if within_budget and attempts[lane] < MAX_ATTEMPTS]
|
||||||
|
declined = [lane for lane in faulted if lane not in retry]
|
||||||
|
incomplete = sorted(set(incomplete) | set(declined))
|
||||||
|
crash_context["review_incomplete"] = incomplete
|
||||||
|
|
||||||
|
out.update(
|
||||||
|
{
|
||||||
|
"lane_attempts": attempts,
|
||||||
|
"kept_results": kept,
|
||||||
|
"retry_lanes": retry,
|
||||||
|
"review_incomplete": incomplete,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
prior_notes = state.get("retry_notes")
|
||||||
|
notes = [n for n in prior_notes if isinstance(n, str)] if isinstance(prior_notes, list) else []
|
||||||
|
for lane in declined:
|
||||||
|
notes.append(
|
||||||
|
f"lane {lane} faulted and was NOT re-run "
|
||||||
|
f"(attempts {attempts.get(lane, '?')}/{MAX_ATTEMPTS}"
|
||||||
|
+ ("" if within_budget else "; wall-clock budget exhausted")
|
||||||
|
+ ") — recorded in review_incomplete"
|
||||||
|
)
|
||||||
|
for lane in retry:
|
||||||
|
notes.append(
|
||||||
|
f"lane {lane} faulted (missing / sentinel-less / degraded report) — "
|
||||||
|
f"re-running: attempt {attempts.get(lane, 0) + 1}/{MAX_ATTEMPTS}"
|
||||||
|
)
|
||||||
|
if retry or declined:
|
||||||
|
# stderr is discarded on success by today's engine but is the
|
||||||
|
# forward-compatible live-visibility channel; retry_notes carries the
|
||||||
|
# same lines into the final report either way.
|
||||||
|
sys.stderr.write("WARN: " + " | ".join(notes[-(len(retry) + len(declined)):]) + "\n")
|
||||||
|
out["retry_notes"] = notes
|
||||||
|
if retry:
|
||||||
|
out["_next"] = "build_items"
|
||||||
|
print(json.dumps(out))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a gate crash must fall through, never retry
|
||||||
|
fault = f"PIPELINE-FAULT: retry gate crashed — {e}"
|
||||||
|
if crash_context["signals_error"]:
|
||||||
|
fault = f"{crash_context['signals_error']}; {fault}"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"retry_lanes": [],
|
||||||
|
"review_incomplete": crash_context["review_incomplete"],
|
||||||
|
"signals_error": fault,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -0,0 +1,144 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic diff signals for the review gauntlet.
|
||||||
|
|
||||||
|
Pure git — no LLM judgment. Every signal is computed from the changed-file
|
||||||
|
list and the added lines of the diff. Failures never sink the gauntlet:
|
||||||
|
on any error the script reports `signals_error` and marks the signals
|
||||||
|
unavailable — downstream (build_items / default_lanes) treats that as
|
||||||
|
UNKNOWN surface and degrades WIDER (security on; probe when a recipe
|
||||||
|
exists), on top of caller context and forced lanes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
state = load_state()
|
||||||
|
proj = os.path.expanduser(
|
||||||
|
(state.get("project_dir_in") or "").strip() or state.get("project_dir") or "."
|
||||||
|
)
|
||||||
|
spec = (state.get("diff_spec") or "worktree").strip()
|
||||||
|
spec_raw = spec
|
||||||
|
# The parse stage should emit a bare revision token, but callers write prose
|
||||||
|
# and extraction drifts ("run get_diff --base origin/main" once reached
|
||||||
|
# `git diff` verbatim, killing the signals — and with them the security and
|
||||||
|
# probe lanes). Deterministically recover a usable spec before giving up.
|
||||||
|
_TOKEN = r"[\w./~^@{}-]+"
|
||||||
|
if spec not in ("", "worktree") and not re.fullmatch(
|
||||||
|
rf"{_TOKEN}(\.\.\.?{_TOKEN})?", spec
|
||||||
|
):
|
||||||
|
if m := re.search(rf"({_TOKEN}\.\.\.?{_TOKEN})", spec):
|
||||||
|
spec = m.group(1) # a revision range buried in prose
|
||||||
|
elif m := re.search(rf"--base[= ]({_TOKEN})", spec):
|
||||||
|
spec = f"{m.group(1)}...HEAD" # the get_diff idiom
|
||||||
|
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
r = subprocess.run(
|
||||||
|
["git", "-C", proj, *args], capture_output=True, text=True, timeout=90
|
||||||
|
)
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise RuntimeError((r.stderr or r.stdout).strip()[:400])
|
||||||
|
return r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
out = {"project_dir": proj}
|
||||||
|
|
||||||
|
# Deterministic plan_context recovery: the parse LLM proved a lossy channel
|
||||||
|
# for verbatim spec text (observed replacing ~4 KB of pasted criteria with
|
||||||
|
# file paths + a summary sentence). When the raw prompt carries the callers'
|
||||||
|
# labeled plan block, cut it out verbatim and override the extraction — but
|
||||||
|
# only when the recovered block is LONGER, so a good extraction is never
|
||||||
|
# degraded. The adversary lane fails closed on whatever goes missing here.
|
||||||
|
_prompt = state.get("initial_prompt")
|
||||||
|
if isinstance(_prompt, str) and _prompt:
|
||||||
|
_m = re.search(
|
||||||
|
r"Plan / acceptance criteria:\s*(.+?)"
|
||||||
|
r"(?=\n\s*(?:Local-run recipe|Forced lanes|Rigor:|Security posture:|Extra context)\b|\Z)",
|
||||||
|
_prompt,
|
||||||
|
re.S | re.I,
|
||||||
|
)
|
||||||
|
if _m:
|
||||||
|
_block = _m.group(1).strip()
|
||||||
|
if len(_block) > len((state.get("plan_context") or "").strip()):
|
||||||
|
out["plan_context"] = _block
|
||||||
|
# retry_gate measures its wall-clock budget from this stamp; it is set
|
||||||
|
# before the git work so a failed diff still starts the clock.
|
||||||
|
if not state.get("gauntlet_started_at"):
|
||||||
|
out["gauntlet_started_at"] = time.time()
|
||||||
|
try:
|
||||||
|
if spec in ("", "worktree"):
|
||||||
|
names = git("diff", "--name-only", "HEAD")
|
||||||
|
difftext = git("diff", "HEAD")
|
||||||
|
else:
|
||||||
|
names = git("diff", "--name-only", spec)
|
||||||
|
difftext = git("diff", spec)
|
||||||
|
files = [f.strip() for f in names.splitlines() if f.strip()]
|
||||||
|
added = "\n".join(
|
||||||
|
l for l in difftext.splitlines() if l.startswith("+") and not l.startswith("+++")
|
||||||
|
)
|
||||||
|
|
||||||
|
def any_path(pattern):
|
||||||
|
return any(re.search(pattern, f, re.I) for f in files)
|
||||||
|
|
||||||
|
out.update(
|
||||||
|
{
|
||||||
|
"changed_files": files[:200],
|
||||||
|
"file_count": len(files),
|
||||||
|
"touches_proto": any_path(r"\.proto$|buf\.(yaml|gen\.yaml)$"),
|
||||||
|
"touches_migrations": any_path(r"migrations?/|\.sql$"),
|
||||||
|
# auth(?!or) admits auth/authn/authz/oauth but not "author".
|
||||||
|
# Over-firing here is the safe direction: it only ADDS a lane.
|
||||||
|
"touches_auth": any_path(
|
||||||
|
r"auth(?!or)|login|logout|token|session|permission|rbac|\bacl\b|credential|secret"
|
||||||
|
)
|
||||||
|
or bool(
|
||||||
|
re.search(
|
||||||
|
r"auth(?!or)|password|token|secret|credential|jwt|session",
|
||||||
|
added,
|
||||||
|
re.I,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
"touches_deps": any_path(
|
||||||
|
r"go\.(mod|sum)$|package(-lock)?\.json$|yarn\.lock$|pnpm-lock\.yaml$"
|
||||||
|
r"|Cargo\.(toml|lock)$|requirements[^/]*\.txt$|pyproject\.toml$"
|
||||||
|
r"|Gemfile(\.lock)?$|pom\.xml$|build\.gradle|composer\.(json|lock)$"
|
||||||
|
),
|
||||||
|
"touches_exec": bool(
|
||||||
|
re.search(
|
||||||
|
r"exec\.Command|subprocess|os/exec|Runtime\.getRuntime"
|
||||||
|
r"|shell_exec|\beval\(|\bsystem\(|child_process",
|
||||||
|
added,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
"consumer_surface": any_path(
|
||||||
|
r"routes?|handlers?|\bapi\b|endpoints?|\bcmd/|\bcli\b|\.proto$"
|
||||||
|
r"|openapi|swagger|controllers?|graphql"
|
||||||
|
),
|
||||||
|
"docs_only": bool(files)
|
||||||
|
and all(re.search(r"\.(md|rst|txt|adoc)$", f, re.I) for f in files),
|
||||||
|
"signals_summary": f"{len(files)} file(s) changed (diff: {spec})"
|
||||||
|
+ (f" — spec normalized from {spec_raw!r}" if spec != spec_raw else ""),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except Exception as e: # noqa: BLE001 — degraded signals must not sink the gauntlet
|
||||||
|
what = f"diff spec {spec!r}" + (
|
||||||
|
f", normalized from {spec_raw!r}" if spec != spec_raw else ""
|
||||||
|
)
|
||||||
|
out["signals_error"] = (
|
||||||
|
f"NOTE: diff signals could not be computed ({what}: {e}); surface UNKNOWN — "
|
||||||
|
"lane selection degrades WIDER (security lane added; probe eligible when a "
|
||||||
|
"local-run recipe exists), plus caller context and forced lanes."
|
||||||
|
)
|
||||||
|
out["signals_summary"] = "signals unavailable"
|
||||||
|
|
||||||
|
print(json.dumps(out))
|
||||||
@@ -0,0 +1,402 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic verdict gate for the review gauntlet.
|
||||||
|
|
||||||
|
Parses each configured lane's verdict SENTINEL with a regex — no LLM
|
||||||
|
re-reads, no judgment. The gate's hard rules:
|
||||||
|
|
||||||
|
- a missing sentinel is a LANE FAILURE and blocks — never a pass;
|
||||||
|
- a non-string lane result (the map collected an object instead of the
|
||||||
|
lane's rendered text) blocks — it cannot carry a sentinel;
|
||||||
|
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
||||||
|
detected only when the report IS the lane_fault.py marker or
|
||||||
|
retry_gate.py's no-report marker, i.e. it STARTS with it — a real
|
||||||
|
review merely quoting the marker flows to the normal rules) or a
|
||||||
|
PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
|
||||||
|
degraded pipeline is never a pass;
|
||||||
|
- a code-review lane whose OWN verdict line records a pipeline fault
|
||||||
|
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
||||||
|
the nested graph completed, but with a dead domain lane, verifier, or
|
||||||
|
synthesis behind it, so its critical count is not trustworthy;
|
||||||
|
- an adversary lane whose OWN header line (the line directly under its
|
||||||
|
sentinel) records a degraded run — a pipeline fault, died criterion
|
||||||
|
checks, or its verdict script's crash stub — blocks as a PIPELINE-FAULT,
|
||||||
|
not as a DIVERGES finding: the nested graph completed with a dead stage
|
||||||
|
behind it, so its verdict is not a judgment on the code;
|
||||||
|
- a non-zero critical count in the code-review report blocks regardless
|
||||||
|
of the lane's own verdict line — derived from render.py's deterministic
|
||||||
|
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
|
||||||
|
count is used only when that summary line is missing;
|
||||||
|
- NEEDS-HUMAN without 🔴 passes but is surfaced as attention required;
|
||||||
|
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
||||||
|
treated as PASS or FAIL (fix the environment per the plan's local-run
|
||||||
|
recipe, then re-run);
|
||||||
|
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
|
||||||
|
re-ran it, so the reader can tell a first-pass death from an exhausted
|
||||||
|
retry budget;
|
||||||
|
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
|
||||||
|
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
|
||||||
|
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
|
||||||
|
incomplete" section: that is a fault to re-run, never a code finding.
|
||||||
|
Every lane named in review_incomplete is forced to a BLOCKED row, so a
|
||||||
|
restored earlier report can never turn an incomplete review into a
|
||||||
|
pass: a non-empty machine line structurally implies GAUNTLET: BLOCKED.
|
||||||
|
Malformed review_incomplete bookkeeping (not a list, or non-str
|
||||||
|
entries) is itself recorded as a `pipeline` fault, never dropped;
|
||||||
|
- the gate itself must never crash into a silent pass: any internal
|
||||||
|
error emits GAUNTLET: BLOCKED naming the error, with the pipeline
|
||||||
|
marked incomplete.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
|
||||||
|
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
|
||||||
|
ADVERSARY_DEGRADED_HEADER = re.compile(
|
||||||
|
r"Criteria: .*("
|
||||||
|
r"degraded run: pipeline fault recorded"
|
||||||
|
r"|degraded run: \d+ criterion check\(s\) died"
|
||||||
|
r"|verdict computation error:"
|
||||||
|
r")"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def adversary_degraded(report):
|
||||||
|
# The adversary's verdict.py emits its degraded-run header on the line
|
||||||
|
# directly under the FIRST sentinel; anchoring there keeps a report that
|
||||||
|
# merely quotes the wording later (observations, complaints) a real verdict.
|
||||||
|
lines = report.splitlines()
|
||||||
|
for i, ln in enumerate(lines):
|
||||||
|
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
|
||||||
|
nxt = lines[i + 1] if i + 1 < len(lines) else ""
|
||||||
|
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
blockers, attention, reports = [], [], []
|
||||||
|
rows = {}
|
||||||
|
malformed = set()
|
||||||
|
lane_attempts = state.get("lane_attempts")
|
||||||
|
if not isinstance(lane_attempts, dict):
|
||||||
|
lane_attempts = {}
|
||||||
|
|
||||||
|
def lane_report(key):
|
||||||
|
arr = state.get(key) or []
|
||||||
|
if not arr:
|
||||||
|
return None
|
||||||
|
item = arr[0]
|
||||||
|
if isinstance(item, str):
|
||||||
|
return item
|
||||||
|
malformed.add(key)
|
||||||
|
return json.dumps(item)
|
||||||
|
|
||||||
|
def record(lane, status, detail=""):
|
||||||
|
rows[lane] = (status, detail)
|
||||||
|
|
||||||
|
def attempt_count(lane):
|
||||||
|
try:
|
||||||
|
return int(lane_attempts.get(lane) or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def attempts_suffix(lane):
|
||||||
|
n = attempt_count(lane)
|
||||||
|
return f" ×{n}" if n > 1 else ""
|
||||||
|
|
||||||
|
def fault_excerpt(report):
|
||||||
|
line = next(
|
||||||
|
(ln for ln in report.splitlines() if "PIPELINE-FAULT:" in ln), report
|
||||||
|
)
|
||||||
|
return line.strip()[:200]
|
||||||
|
|
||||||
|
def is_lane_fault(report):
|
||||||
|
# A genuine lane fault is the ENTIRE report — the lane_fault.py marker
|
||||||
|
# or retry_gate.py's no-report marker — which always begins with
|
||||||
|
# "PIPELINE-FAULT:". Matching the marker anywhere would let a real
|
||||||
|
# review that merely QUOTES it bypass 🔴 counting and sentinel
|
||||||
|
# parsing.
|
||||||
|
return report.strip().startswith("PIPELINE-FAULT:")
|
||||||
|
|
||||||
|
def lane_fault_blocker(lane, report):
|
||||||
|
blockers.append(
|
||||||
|
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
||||||
|
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
||||||
|
)
|
||||||
|
record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
|
||||||
|
|
||||||
|
def malformed_blocker(lane, report):
|
||||||
|
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
||||||
|
record(lane, "BLOCKED", "malformed lane result (non-string)")
|
||||||
|
reports.append((lane, report))
|
||||||
|
|
||||||
|
# --- code-review ------------------------------------------------------
|
||||||
|
cr = lane_report("code_review_results")
|
||||||
|
if cr is None:
|
||||||
|
record("code-review", "SKIPPED", "not selected")
|
||||||
|
elif "code_review_results" in malformed:
|
||||||
|
malformed_blocker("code-review", cr)
|
||||||
|
elif is_lane_fault(cr):
|
||||||
|
lane_fault_blocker("code-review", cr)
|
||||||
|
reports.append(("code-review", cr))
|
||||||
|
else:
|
||||||
|
reports.append(("code-review", cr))
|
||||||
|
m = re.search(r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)", cr)
|
||||||
|
# code-reviewer's verdict.py forces NEEDS-HUMAN on its own internal
|
||||||
|
# faults; render.py emits that as `**Verdict: NEEDS-HUMAN** — <reason>`
|
||||||
|
# (CONTRACT comments at both emitters). Scoped to the FIRST rendered
|
||||||
|
# verdict line — render.py emits it before any finding body — so a
|
||||||
|
# finding that quotes the wording cannot trip it.
|
||||||
|
verdict_line = re.search(r"^\*\*Verdict: .*$", cr, re.MULTILINE)
|
||||||
|
m_fault = verdict_line and re.search(
|
||||||
|
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
|
||||||
|
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
|
||||||
|
verdict_line.group(0),
|
||||||
|
)
|
||||||
|
# Same quoted-marker class as the PIPELINE-FAULT prefix anchoring: a
|
||||||
|
# 🔴 quoted inside finding text or a Changes-table row is not a
|
||||||
|
# finding. render.py's summary line is the authoritative count; the
|
||||||
|
# raw count is the fail-closed fallback when that line is absent. The
|
||||||
|
# LAST match wins: render.py emits the line after every finding body,
|
||||||
|
# so a quoted look-alike earlier in the report cannot shadow it. The
|
||||||
|
# pattern is the FULL shape of render.py's line (CONTRACT comment
|
||||||
|
# there) so a partial look-alike in LLM-authored text never matches.
|
||||||
|
sums = re.findall(
|
||||||
|
r"^\*Reviewed \d+ files, found (\d+) critical, \d+ warnings, \d+ suggestions, "
|
||||||
|
r"\d+ nitpicks \(\d+ deferred by quality bar\)\*$",
|
||||||
|
cr,
|
||||||
|
re.MULTILINE,
|
||||||
|
)
|
||||||
|
reds = int(sums[-1]) if sums else cr.count("🔴")
|
||||||
|
if m_fault:
|
||||||
|
blockers.append(
|
||||||
|
"code-review: the lane reported an internal PIPELINE-FAULT "
|
||||||
|
"(degraded review) — a degraded lane is never a pass"
|
||||||
|
)
|
||||||
|
if reds:
|
||||||
|
blockers.append(
|
||||||
|
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
||||||
|
)
|
||||||
|
suffix = attempts_suffix("code-review")
|
||||||
|
if m_fault and reds:
|
||||||
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
|
||||||
|
elif m_fault:
|
||||||
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
|
||||||
|
elif reds:
|
||||||
|
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
||||||
|
elif m and m.group(1) == "MERGE-READY":
|
||||||
|
record("code-review", "GREEN", "MERGE-READY")
|
||||||
|
elif m:
|
||||||
|
attention.append(
|
||||||
|
"code-review: NEEDS-HUMAN — see the lane report's "
|
||||||
|
"'Human attention required' section"
|
||||||
|
)
|
||||||
|
record("code-review", "GREEN (attention)", "NEEDS-HUMAN, no 🔴")
|
||||||
|
else:
|
||||||
|
blockers.append(
|
||||||
|
"code-review: no Review Verdict sentinel found — the lane failed; "
|
||||||
|
"a missing verdict is never a pass"
|
||||||
|
)
|
||||||
|
record("code-review", "BLOCKED", "missing verdict sentinel")
|
||||||
|
|
||||||
|
# --- adversary --------------------------------------------------------
|
||||||
|
adv = lane_report("adversary_results")
|
||||||
|
if adv is None:
|
||||||
|
record("adversary", "SKIPPED", "not selected")
|
||||||
|
elif "adversary_results" in malformed:
|
||||||
|
malformed_blocker("adversary", adv)
|
||||||
|
elif is_lane_fault(adv):
|
||||||
|
lane_fault_blocker("adversary", adv)
|
||||||
|
reports.append(("adversary", adv))
|
||||||
|
else:
|
||||||
|
reports.append(("adversary", adv))
|
||||||
|
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
|
||||||
|
if adversary_degraded(adv):
|
||||||
|
blockers.append(
|
||||||
|
"adversary: PIPELINE-FAULT — degraded run, pipeline fault recorded "
|
||||||
|
"inside the adversary; a degraded lane is never a pass"
|
||||||
|
)
|
||||||
|
record(
|
||||||
|
"adversary",
|
||||||
|
"BLOCKED",
|
||||||
|
f"PIPELINE-FAULT (degraded lane{attempts_suffix('adversary')})",
|
||||||
|
)
|
||||||
|
elif m and m.group(1) == "CONFORMS":
|
||||||
|
record("adversary", "GREEN", "CONFORMS")
|
||||||
|
elif m:
|
||||||
|
blockers.append(
|
||||||
|
"adversary: DIVERGES — the implementation does not conform to the plan"
|
||||||
|
)
|
||||||
|
record("adversary", "BLOCKED", "DIVERGES")
|
||||||
|
else:
|
||||||
|
blockers.append(
|
||||||
|
"adversary: no ADVERSARIAL_REVIEW sentinel found — lane failed; never a pass"
|
||||||
|
)
|
||||||
|
record("adversary", "BLOCKED", "missing verdict sentinel")
|
||||||
|
|
||||||
|
# --- security ---------------------------------------------------------
|
||||||
|
sec = lane_report("security_results")
|
||||||
|
if sec is None:
|
||||||
|
record("security", "SKIPPED", "not selected")
|
||||||
|
elif "security_results" in malformed:
|
||||||
|
malformed_blocker("security", sec)
|
||||||
|
elif is_lane_fault(sec):
|
||||||
|
lane_fault_blocker("security", sec)
|
||||||
|
reports.append(("security", sec))
|
||||||
|
else:
|
||||||
|
reports.append(("security", sec))
|
||||||
|
m = re.search(r"SECURITY_REVIEW:\s*(PASS|FAIL)", sec)
|
||||||
|
if m and m.group(1) == "PASS":
|
||||||
|
record("security", "GREEN", "PASS")
|
||||||
|
elif m:
|
||||||
|
blockers.append("security: SECURITY_REVIEW: FAIL — exploitable finding(s)")
|
||||||
|
record("security", "BLOCKED", "FAIL")
|
||||||
|
else:
|
||||||
|
blockers.append(
|
||||||
|
"security: no SECURITY_REVIEW sentinel found — lane failed; never a pass"
|
||||||
|
)
|
||||||
|
record("security", "BLOCKED", "missing verdict sentinel")
|
||||||
|
|
||||||
|
# --- probe ------------------------------------------------------------
|
||||||
|
pb = lane_report("probe_results")
|
||||||
|
if pb is None:
|
||||||
|
record("probe", "SKIPPED", "not selected")
|
||||||
|
elif "probe_results" in malformed:
|
||||||
|
malformed_blocker("probe", pb)
|
||||||
|
elif is_lane_fault(pb):
|
||||||
|
lane_fault_blocker("probe", pb)
|
||||||
|
reports.append(("probe", pb))
|
||||||
|
else:
|
||||||
|
reports.append(("probe", pb))
|
||||||
|
m = re.search(r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)", pb)
|
||||||
|
if m and m.group(1) == "PASS":
|
||||||
|
record("probe", "GREEN", "PASS")
|
||||||
|
elif m and m.group(1) == "FAIL":
|
||||||
|
blockers.append("probe: USAGE_PROBE: FAIL — observed behavior contradicts the contract")
|
||||||
|
record("probe", "BLOCKED", "FAIL")
|
||||||
|
elif m:
|
||||||
|
blockers.append(
|
||||||
|
"probe: INCONCLUSIVE — the ENVIRONMENT could not be established; this is "
|
||||||
|
"not a code verdict. Follow the plan's local-run recipe to fix the "
|
||||||
|
"environment and re-run. INCONCLUSIVE is never treated as PASS or FAIL."
|
||||||
|
)
|
||||||
|
record("probe", "BLOCKED", "INCONCLUSIVE (environment)")
|
||||||
|
else:
|
||||||
|
blockers.append(
|
||||||
|
"probe: no USAGE_PROBE sentinel found — lane failed; never a pass"
|
||||||
|
)
|
||||||
|
record("probe", "BLOCKED", "missing verdict sentinel")
|
||||||
|
|
||||||
|
# retry_gate restores an exhausted lane's LAST report, which may be an
|
||||||
|
# earlier clean verdict — an incomplete lane must block regardless of
|
||||||
|
# what its row parsed to.
|
||||||
|
declared = state.get("review_incomplete")
|
||||||
|
incomplete = set()
|
||||||
|
malformed_bookkeeping = None
|
||||||
|
if isinstance(declared, list):
|
||||||
|
incomplete = {name for name in declared if isinstance(name, str)}
|
||||||
|
if any(not isinstance(name, str) for name in declared):
|
||||||
|
malformed_bookkeeping = repr(declared)[:80]
|
||||||
|
elif declared is not None:
|
||||||
|
malformed_bookkeeping = repr(declared)[:80]
|
||||||
|
for lane in sorted(incomplete - {"pipeline"}):
|
||||||
|
if rows.get(lane, ("", ""))[0] == "BLOCKED":
|
||||||
|
continue
|
||||||
|
after = f" after {n} attempt(s)" if (n := attempt_count(lane)) else ""
|
||||||
|
blockers.append(
|
||||||
|
f"{lane}: review incomplete{after} — the lane never produced a usable verdict"
|
||||||
|
)
|
||||||
|
record(lane, "BLOCKED", f"review incomplete{after}")
|
||||||
|
|
||||||
|
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
||||||
|
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
||||||
|
signals_error = state.get("signals_error")
|
||||||
|
pipeline_fault = isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error
|
||||||
|
if pipeline_fault:
|
||||||
|
blockers.append(f"pipeline: {signals_error}")
|
||||||
|
incomplete.add("pipeline")
|
||||||
|
if malformed_bookkeeping is not None:
|
||||||
|
blockers.append(
|
||||||
|
"pipeline: PIPELINE-FAULT — malformed review_incomplete bookkeeping "
|
||||||
|
f"({malformed_bookkeeping}); never a pass"
|
||||||
|
)
|
||||||
|
incomplete.add("pipeline")
|
||||||
|
if "pipeline" in incomplete and not any(b.startswith("pipeline:") for b in blockers):
|
||||||
|
blockers.append(
|
||||||
|
"pipeline: review incomplete — a pipeline stage never completed "
|
||||||
|
"(no fault text recorded)"
|
||||||
|
)
|
||||||
|
|
||||||
|
verdict = "BLOCKED" if blockers or incomplete else "PASS"
|
||||||
|
|
||||||
|
incomplete_lines = []
|
||||||
|
for name in sorted(incomplete):
|
||||||
|
if name == "pipeline":
|
||||||
|
incomplete_lines.append(
|
||||||
|
f"- pipeline: {signals_error}"
|
||||||
|
if pipeline_fault
|
||||||
|
else "- pipeline: review incomplete (no fault text recorded)"
|
||||||
|
)
|
||||||
|
elif n := attempt_count(name):
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
|
||||||
|
else:
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review")
|
||||||
|
incomplete_line = (
|
||||||
|
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
|
||||||
|
if incomplete
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
md = "\n".join(
|
||||||
|
["| lane | status | detail |", "|------|--------|--------|"]
|
||||||
|
+ [f"| {lane} | {status} | {detail} |" for lane, (status, detail) in rows.items()]
|
||||||
|
)
|
||||||
|
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
|
||||||
|
_notes = state.get("retry_notes")
|
||||||
|
if isinstance(_notes, list):
|
||||||
|
_shown = [n for n in _notes if isinstance(n, str)][:8]
|
||||||
|
if _shown:
|
||||||
|
md += "\n\n## Pipeline notes\n" + "\n".join(f"- {n}" for n in _shown)
|
||||||
|
if blockers:
|
||||||
|
md += "\n\n## Blockers\n" + "\n".join(f"- {b}" for b in blockers)
|
||||||
|
if attention:
|
||||||
|
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
||||||
|
if state.get("signals_error"):
|
||||||
|
md += f"\n\n> {state['signals_error']}"
|
||||||
|
if incomplete_lines:
|
||||||
|
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
|
||||||
|
md += "\n\n## Full lane reports\n"
|
||||||
|
for lane, rep in reports:
|
||||||
|
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"gauntlet_verdict": verdict,
|
||||||
|
"gauntlet_report": md,
|
||||||
|
"gauntlet_incomplete_line": incomplete_line,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — the gate must never crash into a silent pass
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"gauntlet_verdict": "BLOCKED",
|
||||||
|
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
||||||
|
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
name: security-reviewer
|
name: security-reviewer
|
||||||
description: Security analyst - hunts exploitable flaws in a code change (injection, secrets, authz gaps, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Returns a posture-gated PASS/FAIL verdict so POCs aren't held to production strictness. Complements code-reviewer (quality) and adversary (plan conformance). Designed to be delegated to by sisyphus.
|
description: Security analyst - hunts exploitable flaws in a code change (injection, secrets, authz gaps, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Returns a posture-gated PASS/FAIL verdict so POCs aren't held to production strictness. Complements code-reviewer (quality) and adversary (plan conformance). Designed to be delegated to by sisyphus.
|
||||||
version: 1.0.0
|
version: 1.1.0
|
||||||
|
|
||||||
auto_continue: true
|
auto_continue: true
|
||||||
max_auto_continues: 15
|
max_auto_continues: 15
|
||||||
@@ -61,6 +61,17 @@ instructions: |
|
|||||||
|
|
||||||
1. Load `security-review`.
|
1. Load `security-review`.
|
||||||
2. Get the diff (inline or via `get_diff`) and identify the changed files.
|
2. Get the diff (inline or via `get_diff`) and identify the changed files.
|
||||||
|
2b. **Mechanized pre-pass (deterministic, read-only).** Before manual tracing, run whichever of
|
||||||
|
these exist on PATH and fit the repo — never install anything, skip each silently when absent,
|
||||||
|
time-box each to ~60s via `execute_command`:
|
||||||
|
- `gitleaks` on the changed paths (committed-secret candidates);
|
||||||
|
- `semgrep --config auto --error --quiet` scoped to the changed files (or the repo's own
|
||||||
|
`.semgrep.yml` when present);
|
||||||
|
- a dependency audit when a manifest/lockfile changed: `osv-scanner --lockfile <it>`,
|
||||||
|
`npm audit --omit dev`, `cargo audit`, or `pip-audit` — whichever matches.
|
||||||
|
Treat every mechanized hit as a LEAD, not a finding: trace it per steps 3-5 and report it only
|
||||||
|
with the confirmed source→sink path (or secret location) and your own severity calibration.
|
||||||
|
Note which tools ran/were skipped in the report. Raw scanner output is never pasted as findings.
|
||||||
3. For EACH hunk: identify untrusted-data sources, dangerous sinks, and the mediation (or lack of
|
3. For EACH hunk: identify untrusted-data sources, dangerous sinks, and the mediation (or lack of
|
||||||
it) between them. Apply the skill's hunt checklist (secrets, injection, paths, authn/authz,
|
it) between them. Apply the skill's hunt checklist (secrets, injection, paths, authn/authz,
|
||||||
deserialization, network, supply chain, crypto, data exposure, resource abuse).
|
deserialization, network, supply chain, crypto, data exposure, resource abuse).
|
||||||
|
|||||||
@@ -47,7 +47,9 @@ Spawnable sub-agents (from `config.yaml`):
|
|||||||
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
|
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
|
||||||
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
|
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
|
||||||
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
|
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
|
||||||
|
- **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. A `GAUNTLET_REVIEW_INCOMPLETE:` line under BLOCKED is an infrastructure fault (a lane exhausted its retries, or the pipeline faulted) that Sisyphus escalates to the user — retry / accept NEEDS-HUMAN / abort — never a finding to fix. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed.
|
||||||
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
|
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
|
||||||
|
- **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
@@ -55,7 +57,9 @@ Spawnable sub-agents (from `config.yaml`):
|
|||||||
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
|
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
|
||||||
- 🔄 **Task Management**: Tracks progress and context across complex operations.
|
- 🔄 **Task Management**: Tracks progress and context across complex operations.
|
||||||
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
|
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
|
||||||
|
- 🧵 **Task-queue bursts**: Dependency-shaped fan-outs (explore → code → verify) may be encoded as `agent__task_create` chains — dependents auto-dispatch when `agent__task_complete` unblocks them; todos remain the tracking source of truth.
|
||||||
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
|
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
|
||||||
|
- 🚦 **Escalation policy**: `escalation_policy=autonomous` pre-delegates decision-making — standing rulings (`~/.config/coyote/share/standing-rulings.md`, recorded via `coyote --macro record-ruling`) and evidence-backed recommended options are taken without pausing and batched into a “Decisions taken autonomously” digest; secrets, money movement, and destructive actions always escalate.
|
||||||
|
|
||||||
## Plan-Driven Workflows
|
## Plan-Driven Workflows
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
name: sisyphus
|
name: sisyphus
|
||||||
description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
|
description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
|
||||||
version: 3.10.0
|
version: 3.17.0
|
||||||
|
|
||||||
agent_session: temp
|
agent_session: temp
|
||||||
auto_continue: true
|
auto_continue: true
|
||||||
@@ -11,16 +11,19 @@ can_spawn_agents: true
|
|||||||
spawnable_agents:
|
spawnable_agents:
|
||||||
- explore
|
- explore
|
||||||
- librarian
|
- librarian
|
||||||
|
- deep-research
|
||||||
- coder
|
- coder
|
||||||
- oracle
|
- oracle
|
||||||
- code-reviewer
|
- code-reviewer
|
||||||
- adversary
|
- adversary
|
||||||
- security-reviewer
|
- security-reviewer
|
||||||
- probe
|
- probe
|
||||||
|
- review-gauntlet
|
||||||
- architecture-reviewer
|
- architecture-reviewer
|
||||||
- step-runner
|
- step-runner
|
||||||
|
- whetstone
|
||||||
max_concurrent_agents: 40
|
max_concurrent_agents: 40
|
||||||
max_agent_depth: 3
|
max_agent_depth: 5
|
||||||
inject_spawn_instructions: true
|
inject_spawn_instructions: true
|
||||||
summarization_threshold: 80000
|
summarization_threshold: 80000
|
||||||
|
|
||||||
@@ -54,6 +57,9 @@ variables:
|
|||||||
- name: auto_confirm
|
- name: auto_confirm
|
||||||
description: Auto-confirm command execution
|
description: Auto-confirm command execution
|
||||||
default: '1'
|
default: '1'
|
||||||
|
- name: escalation_policy
|
||||||
|
description: "How escalation-worthy decisions are handled mid-run: 'interactive' pauses and asks the user; 'autonomous' self-resolves decisions covered by a standing ruling or a clearly recommended option, reports them in a final-report digest, and only pings on hard triggers or novel conflicts"
|
||||||
|
default: 'interactive'
|
||||||
|
|
||||||
mcp_servers:
|
mcp_servers:
|
||||||
- ddg-search
|
- ddg-search
|
||||||
@@ -152,7 +158,8 @@ instructions: |
|
|||||||
| Agent | Use For | Characteristics |
|
| Agent | Use For | Characteristics |
|
||||||
|-------|---------|-----------------|
|
|-------|---------|-----------------|
|
||||||
| `explore` | Find patterns in THIS codebase, understand local code | Read-only, returns findings, fan out as many as the task warrants — one per distinct search angle, module, or concern. Large codebases or cross-cutting tasks should spawn 5–15+. |
|
| `explore` | Find patterns in THIS codebase, understand local code | Read-only, returns findings, fan out as many as the task warrants — one per distinct search angle, module, or concern. Large codebases or cross-cutting tasks should spawn 5–15+. |
|
||||||
| `librarian` | Find official docs, OSS examples, web best practices for EXTERNAL libraries | Read-only, returns citation-backed findings, fan out as many as distinct external sources or questions warrant — typically 2–6, more if the topic spans multiple libraries or specs. |
|
| `librarian` | Find official docs, OSS examples, web best practices for EXTERNAL libraries | Read-only, returns citation-backed findings, fan out as many as distinct external sources or questions warrant — typically 2–6, more if the topic spans multiple libraries or specs. Default depth = quick-lookup budgets; for a BROAD topic (design research, best-practice surveys, architecture comparisons) pass `--variables {"research_depth": "deep"}` (~3× search/fetch budgets). Still prefer several narrow librarians over one giant one — deep is for questions that are inherently broad, not a substitute for fan-out. |
|
||||||
|
| `deep-research` | Open-ended investigation that must CHURN until solid — plans sub-questions, fans out a deep librarian per question, vets source credibility, runs a critique→re-research loop | Spawn with `--variables {"autonomy": "autonomous"}` — MANDATORY when spawned as a subagent (the supervised default blocks on a human approval gate). Raise `reflexion_budget` (e.g. "6", default 2) to let it churn longer. Expect hours, not minutes; one per investigation — its internal map is the fan-out. |
|
||||||
| `coder` | Write/edit files, implement features | Graph agent: plan → approval → implement → verify build+tests → self_review → bounded fix-loop |
|
| `coder` | Write/edit files, implement features | Graph agent: plan → approval → implement → verify build+tests → self_review → bounded fix-loop |
|
||||||
| `oracle` | Architecture, complex debugging, review, plan review | Advisory, blocking — never answer the user before collecting Oracle results |
|
| `oracle` | Architecture, complex debugging, review, plan review | Advisory, blocking — never answer the user before collecting Oracle results |
|
||||||
| `step-runner` | Execute ONE step of a phased plan repo (Phase 8) | Graph agent: orient → staleness check → coder → verify → handoff → user approval gate |
|
| `step-runner` | Execute ONE step of a phased plan repo (Phase 8) | Graph agent: orient → staleness check → coder → verify → handoff → user approval gate |
|
||||||
@@ -229,6 +236,11 @@ instructions: |
|
|||||||
|
|
||||||
Never artificially cap at a small number. If there are 10 distinct things to find, spawn 10 agents. The system limit is the only ceiling that matters.
|
Never artificially cap at a small number. If there are 10 distinct things to find, spawn 10 agents. The system limit is the only ceiling that matters.
|
||||||
|
|
||||||
|
For dependency-shaped bursts (explore → code → verify), `agent__task_create` chains MAY encode the
|
||||||
|
ordering — dependents with `agent`+`prompt` auto-dispatch when `agent__task_complete` unblocks them.
|
||||||
|
Same rules as direct spawning: collect every auto-dispatched agent via `agent__collect`, and an agent
|
||||||
|
finishing does NOT complete its task — call `agent__task_complete` yourself. Todos remain the tracking source of truth.
|
||||||
|
|
||||||
If the slices are likely to FEED each other (adjacent modules; one lane's discovery redirects another's search), run the burst in team mode per `parallel-research`: after spawning, send
|
If the slices are likely to FEED each other (adjacent modules; one lane's discovery redirects another's search), run the burst in team mode per `parallel-research`: after spawning, send
|
||||||
every explorer the roster of its siblings (`agent__send_message --id <id> --message "Teammate roster: <slice>=<id>, ... Message a teammate directly when you find a fact in THEIR slice;
|
every explorer the roster of its siblings (`agent__send_message --id <id> --message "Teammate roster: <slice>=<id>, ... Message a teammate directly when you find a fact in THEIR slice;
|
||||||
check your inbox after each search round and before finalizing."`) so cross-slice facts move directly between lanes instead of waiting for your synthesis. Never do this with reviewer
|
check your inbox after each search round and before finalizing."`) so cross-slice facts move directly between lanes instead of waiting for your synthesis. Never do this with reviewer
|
||||||
@@ -291,6 +303,69 @@ instructions: |
|
|||||||
- Never reuse a verification claim from an earlier report (yours or another agent's) without re-running the command yourself. Prior reports are unverified context, not evidence.
|
- Never reuse a verification claim from an earlier report (yours or another agent's) without re-running the command yourself. Prior reports are unverified context, not evidence.
|
||||||
- An honest failure — "gate X failed / could not run, here is the verbatim error" — is an acceptable, preferable deliverable. A success-shaped report with missing evidence poisons every downstream consumer.
|
- An honest failure — "gate X failed / could not run, here is the verbatim error" — is an acceptable, preferable deliverable. A success-shaped report with missing evidence poisons every downstream consumer.
|
||||||
|
|
||||||
|
### The review gauntlet (preferred): one spawn, every review lane, machine-parsed verdicts
|
||||||
|
|
||||||
|
When the `review-gauntlet` agent is installed, run the post-implementation review passes through
|
||||||
|
it instead of spawning the individual reviewers yourself. Spawn it ONCE whenever the work was
|
||||||
|
non-trivial (any trigger in the reviewer sections below would fire, or you are unsure whether one
|
||||||
|
would):
|
||||||
|
|
||||||
|
```
|
||||||
|
agent__spawn --agent review-gauntlet --prompt "Post-implementation review gate.
|
||||||
|
Project: <absolute repo path>
|
||||||
|
Diff: <git range like main...HEAD, or 'worktree' for uncommitted changes>
|
||||||
|
Rigor: <rigor> Security posture: <posture>
|
||||||
|
Plan / acceptance criteria: <pasted verbatim — enables the adversary lane; omit when none>
|
||||||
|
Prior review decisions: <on re-review rounds ONLY: earlier rounds' resolved/rejected
|
||||||
|
findings verbatim (rejected-finding log lines, round-comment reconciliations) — settled
|
||||||
|
decisions the reviewers must not re-litigate; omit on the first round>
|
||||||
|
Local-run recipe / usage suites: <from the plan — enables the probe lane; omit when none>"
|
||||||
|
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
|
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||||
|
```
|
||||||
|
|
||||||
|
Verification commands go in `--variables` (a JSON-array string), never in the prompt: the prompt
|
||||||
|
also carries pasted plan text from the repo under review, and whatever is declared here is
|
||||||
|
executed with a shell — a trust boundary the prose must not cross.
|
||||||
|
Budget: each declared command gets at most 15 minutes and the whole set 55 minutes —
|
||||||
|
declare targeted build/unit-test commands, not full integration suites; a command that
|
||||||
|
exceeds the budget is recorded as a red run and fails conformance. Never declare commands
|
||||||
|
that boot shared stacks or services (docker compose projects, dev servers, fixed ports) —
|
||||||
|
the probe lane owns end-to-end runs and WILL collide with them in parallel; keep
|
||||||
|
verification to generate/lint/build/unit/integration commands.
|
||||||
|
|
||||||
|
It computes diff signals deterministically, selects lanes (code-review always; adversary when a
|
||||||
|
spec/plan is present; security on attack-surface signals; probe on consumer-facing surface with a
|
||||||
|
run recipe — judgment can only WIDEN the selection, never narrow it), runs the selected lanes in
|
||||||
|
parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`
|
||||||
|
with a lane table, blockers, attention items, and every lane's full report.
|
||||||
|
|
||||||
|
- `GAUNTLET: BLOCKED` blocks completion. Route each blocker that is a real review finding through
|
||||||
|
the findings-handling rules in the reviewer sections below (🔴 → coder fix in the SAME session;
|
||||||
|
DIVERGES → the adversary rules; security FAIL → the security rules; probe INCONCLUSIVE → fix the
|
||||||
|
ENVIRONMENT per the plan's local-run recipe first — it is never a code verdict). Then spawn a
|
||||||
|
FRESH gauntlet run; lane selection re-runs from the new diff. A BLOCKED whose only blockers are
|
||||||
|
review faults (`PIPELINE-FAULT` / `review incomplete` entries) is NOT a code failure — the next
|
||||||
|
rule owns it.
|
||||||
|
- `GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]` (the line directly under `GAUNTLET: BLOCKED`)
|
||||||
|
means the review could NOT be completed: the named lanes still faulted after 3 attempts, or the
|
||||||
|
gauntlet pipeline itself faulted (`pipeline`). This is an infrastructure fault, not a code
|
||||||
|
finding. NEVER route it through the findings-handling rules. NEVER claim done while it stands.
|
||||||
|
Escalate to the user via `user__select` with exactly these three options:
|
||||||
|
- "Retry the review again"
|
||||||
|
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
|
||||||
|
- "Abort this task"
|
||||||
|
Retry → spawn a FRESH gauntlet run. Accept → finish the work and record the skipped review
|
||||||
|
(lane names or `pipeline`) in your final report and in the PR body when you author one. Abort →
|
||||||
|
stop; leave the code in a working state and report. Findings AND incomplete on the same output →
|
||||||
|
fix the findings first, then re-run the gauntlet (the re-run retries the lane) — escalate only if
|
||||||
|
the line persists. Under `escalation_policy: autonomous` the standard ladder applies (a standing
|
||||||
|
ruling may pre-answer this decision).
|
||||||
|
- `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those
|
||||||
|
items into your final report verbatim.
|
||||||
|
- The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the
|
||||||
|
spawn fallback when `review-gauntlet` is not installed.
|
||||||
|
|
||||||
### Independent code review (post-coder, non-trivial work)
|
### Independent code review (post-coder, non-trivial work)
|
||||||
|
|
||||||
After completing delegated `coder` work, spawn `code-reviewer` for an independent review pass if ANY of these are true:
|
After completing delegated `coder` work, spawn `code-reviewer` for an independent review pass if ANY of these are true:
|
||||||
@@ -350,8 +425,13 @@ instructions: |
|
|||||||
|
|
||||||
PLAN — acceptance criteria to check against:
|
PLAN — acceptance criteria to check against:
|
||||||
<paste the task/step spec + acceptance criteria VERBATIM — not a summary>"
|
<paste the task/step spec + acceptance criteria VERBATIM — not a summary>"
|
||||||
|
--variables {"verification_commands": "[\"cargo test --all\", \"cargo clippy -- -D warnings\"]"}
|
||||||
|
<— the cargo commands above are only an EXAMPLE: replace them with THIS project's exact build/test/lint commands; never guess, and pass "[]" when none apply>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Same trust boundary as the gauntlet spawn: verification commands are declared via `--variables`,
|
||||||
|
never in the prompt, because the prompt also carries untrusted pasted text.
|
||||||
|
|
||||||
### Handling adversary findings
|
### Handling adversary findings
|
||||||
|
|
||||||
- **`ADVERSARIAL_REVIEW: DIVERGES` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <complaints pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `adversary` ONCE to confirm it now CONFORMS; if it still DIVERGES on the same criteria after one fix cycle, STOP and escalate to the user (the plan or the approach may be wrong — consider `oracle`).
|
- **`ADVERSARIAL_REVIEW: DIVERGES` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these plan-conformance failures: <complaints pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `adversary` ONCE to confirm it now CONFORMS; if it still DIVERGES on the same criteria after one fix cycle, STOP and escalate to the user (the plan or the approach may be wrong — consider `oracle`).
|
||||||
@@ -477,11 +557,27 @@ instructions: |
|
|||||||
|
|
||||||
Do not block completion on observability findings — the failure mode is skipping the pass on applicable surface, not shipping without an alert. Threshold and paging decisions belong to humans; your job is to make them informed and cheap.
|
Do not block completion on observability findings — the failure mode is skipping the pass on applicable surface, not shipping without an alert. Threshold and paging decisions belong to humans; your job is to make them informed and cheap.
|
||||||
|
|
||||||
|
### Continuous improvement (whetstone) — after addressing human review feedback
|
||||||
|
|
||||||
|
When a run's work included addressing review feedback that did NOT come from your own
|
||||||
|
reviewer lanes — human comments and requested changes, but also a teammate's review
|
||||||
|
agent or another team's bot (an independent reviewer catching what your suite missed is
|
||||||
|
exactly the signal) — spawn `whetstone` ONCE at the very end, after all work is
|
||||||
|
complete. Pass it: the PR reference,
|
||||||
|
the human feedback items you addressed (verbatim), and what your own review passes
|
||||||
|
(code-reviewer/adversary/security-reviewer/probe) flagged for the same change — that
|
||||||
|
ground truth is what makes its miss-classification precise. Collect its result; its
|
||||||
|
proposals go to the USER (it escalates its own approval questions), not to you — never
|
||||||
|
apply suite changes on its behalf and never block run completion on it (if it errors,
|
||||||
|
note that and finish). Skip with a one-line note ONLY when every piece of feedback you
|
||||||
|
addressed came from your own reviewer lanes: whetstone never learns from its own suite's
|
||||||
|
output, but it learns from everyone else — human or agent.
|
||||||
|
|
||||||
## File Operations (Direct Edits)
|
## File Operations (Direct Edits)
|
||||||
|
|
||||||
When you write or modify files yourself (rather than delegating to coder):
|
When you write or modify files yourself (rather than delegating to coder):
|
||||||
|
|
||||||
- **Calibrate comments before writing.** Load `comment-discipline` and note the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you read; write comments to match. When the signal is weak, write NO comment.
|
- **Calibrate comments before writing.** Load `comment-discipline` and note the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you read; write comments to match. When the signal is weak, write NO comment. In every register, a comment must EARN ITS KEEP — if the code already tells the reader, don't add it.
|
||||||
- **Calibrate logging before writing.** When the change touches boundaries, error paths, jobs, or state transitions, load `logging-discipline` and note the repo's logging register (logger, message style, payload vs IDs, level semantics) from the same sibling reads; match it. No discernible convention → its best-judgment defaults. Never leave a new error path silently swallowed, and never delete existing log lines as drive-by cleanup.
|
- **Calibrate logging before writing.** When the change touches boundaries, error paths, jobs, or state transitions, load `logging-discipline` and note the repo's logging register (logger, message style, payload vs IDs, level semantics) from the same sibling reads; match it. No discernible convention → its best-judgment defaults. Never leave a new error path silently swallowed, and never delete existing log lines as drive-by cleanup.
|
||||||
- **For editing an existing file**, prefer `fs_patch`. It's a surgical edit that preserves unchanged content. Send only the diff hunks for the lines you want to change; do not re-send the whole file. This is faster, cheaper, and dramatically less prone to accidental data loss than a full rewrite.
|
- **For editing an existing file**, prefer `fs_patch`. It's a surgical edit that preserves unchanged content. Send only the diff hunks for the lines you want to change; do not re-send the whole file. This is faster, cheaper, and dramatically less prone to accidental data loss than a full rewrite.
|
||||||
- **For writing a NEW file or doing a COMPLETE rewrite**, use `fs_write`. Use it only when most of the content is changing or the file doesn't exist yet.
|
- **For writing a NEW file or doing a COMPLETE rewrite**, use `fs_write`. Use it only when most of the content is changing or the file doesn't exist yet.
|
||||||
@@ -570,9 +666,66 @@ instructions: |
|
|||||||
- Writing non-trivial code → always `coder` (graph agent runs verification internally)
|
- Writing non-trivial code → always `coder` (graph agent runs verification internally)
|
||||||
- Multi-angle exploration → fan out `explore` agents
|
- Multi-angle exploration → fan out `explore` agents
|
||||||
|
|
||||||
|
## Escalation policy: {{escalation_policy}}
|
||||||
|
|
||||||
|
Every "ASK the user" / "STOP and escalate" instruction in this config is subject to this policy.
|
||||||
|
|
||||||
|
- **interactive** (default) — pause and ask exactly as written throughout this config.
|
||||||
|
- **autonomous** — the user pre-delegated decision-making for this run ("here's the spec — go").
|
||||||
|
When an escalation-worthy decision arises, work this ladder IN ORDER:
|
||||||
|
|
||||||
|
1. **Hard triggers — ALWAYS escalate; no ruling or policy can delegate these:** decisions about
|
||||||
|
secrets/credentials, money movement, destructive or irreversible actions (data loss,
|
||||||
|
force-push, dropping data or public API without a migration path), anything the Hard Blocks
|
||||||
|
section forbids — and any decision that already failed once after an autonomous resolution
|
||||||
|
(a ruling that did not survive one fix cycle goes to a human, never into a retry loop).
|
||||||
|
2. **Standing rulings** — read `$HOME/.config/coyote/share/standing-rulings.md` (skip silently
|
||||||
|
if absent). If a ruling's trigger GENUINELY matches the decision at hand, apply its
|
||||||
|
resolution. Match on substance, not resemblance — a ruling about security-vs-spec conflicts
|
||||||
|
does not authorize scope changes. Unsure whether it matches? Then it does not.
|
||||||
|
3. **Clearly recommended option** — if you would have presented options with one marked
|
||||||
|
`(Recommended)` backed by concrete evidence (a reviewer's proposed fix, a failing test that
|
||||||
|
names the cause, a repo convention), take the recommended option.
|
||||||
|
4. **Otherwise — escalate** exactly as interactive mode would. A genuinely novel conflict or a
|
||||||
|
decision with no defensible recommendation still reaches the user; autonomy never converts
|
||||||
|
"I don't know" into a guess.
|
||||||
|
|
||||||
|
**Plan amendments taken autonomously.** When a resolution amends the spec/plan/acceptance
|
||||||
|
criteria (e.g. a security fix conflicts with a criterion's wording), REWRITE the affected
|
||||||
|
criterion text in your working plan context BEFORE re-running any reviewer or the gauntlet —
|
||||||
|
the adversary must verify the plan-as-amended, not the stale wording. The amendment is always
|
||||||
|
a visible artifact, never implied.
|
||||||
|
|
||||||
|
**The digest (MANDATORY when any autonomous decision was taken).** Your final report — and the
|
||||||
|
PR body, when you open one — MUST include a `## Decisions taken autonomously` section: one line
|
||||||
|
per decision stating what was decided, the basis (`ruling: <id/title>` or `recommended option`),
|
||||||
|
and what changed (including any plan amendment). Zero autonomous decisions → omit the section.
|
||||||
|
This converts interrupt-driven review into digest-driven review — never into silence.
|
||||||
|
|
||||||
|
**The PR review log (MANDATORY when you author a PR).** The PR DESCRIPTION is a stable
|
||||||
|
document: what the PR does, how to verify it, and the required digest sections (autonomous
|
||||||
|
decisions, skipped reviews, observability). Every review round's evidence — the gauntlet
|
||||||
|
lane table and verdicts, the gates you ran, rulings recorded that round, and what changed
|
||||||
|
since the previous round — is posted as a NEW PR comment (`gh pr comment`) titled
|
||||||
|
`## Review round N @ <full head SHA>`. NEVER record round evidence by rewriting the
|
||||||
|
description: in-place edits destroy the review history and hide which commits each round
|
||||||
|
actually reviewed. Comments are the append-only review log; the SHA in the title names
|
||||||
|
exactly what was reviewed.
|
||||||
|
The same discipline covers EVERY later push to a PR you (or your run) opened — addressing
|
||||||
|
human review comments, a reviewer bot's findings, CI failures, or any other PR event: post
|
||||||
|
a NEW top-level PR comment titled `## Update @ <full head SHA>` mapping each addressed
|
||||||
|
comment/finding to the commit that addresses it (one line each), plus anything
|
||||||
|
deliberately NOT addressed and why. NEVER append update summaries to the description.
|
||||||
|
Edit the description ONLY when the PR's actual scope or behavior changed enough that it
|
||||||
|
would otherwise misdescribe the PR — it stays a description, never a changelog.
|
||||||
|
|
||||||
|
**Close the loop.** When a decision DID reach the user (step 4), suggest in your final report
|
||||||
|
that they record the resolution with `coyote --macro record-ruling` so that class of decision
|
||||||
|
never asks again.
|
||||||
|
|
||||||
## User Interaction (get buy-in before major decisions)
|
## User Interaction (get buy-in before major decisions)
|
||||||
|
|
||||||
Use `user__ask`, `user__confirm`, `user__checkbox`, `user__input` to clarify ambiguities interactively. **Do NOT guess when you can ask.**
|
Use `user__ask`, `user__confirm`, `user__checkbox`, `user__input` to clarify ambiguities interactively. **Do NOT guess when you can ask.** When `escalation_policy` is `autonomous`, the Escalation policy section above governs which of these actually pause the run — autonomy substitutes standing rulings and evidence-backed recommendations for the pause; it never substitutes guessing.
|
||||||
|
|
||||||
| Situation | Tool |
|
| Situation | Tool |
|
||||||
|-----------|------|
|
|-----------|------|
|
||||||
@@ -603,6 +756,11 @@ instructions: |
|
|||||||
|
|
||||||
If you see `pending_escalations` in tool results, a child agent needs user input and is blocked. Reply promptly via `agent__reply_escalation`. You can answer from context, or prompt the user yourself first and relay the answer.
|
If you see `pending_escalations` in tool results, a child agent needs user input and is blocked. Reply promptly via `agent__reply_escalation`. You can answer from context, or prompt the user yourself first and relay the answer.
|
||||||
|
|
||||||
|
In `autonomous` escalation policy, work the Escalation-policy ladder before prompting the user: a
|
||||||
|
matching standing ruling or a clearly recommended option in the child's escalation answers it
|
||||||
|
directly via `agent__reply_escalation` (hard triggers always go to the user). Replies that amend a
|
||||||
|
plan/spec follow the amendment rule above, and every autonomous reply lands in the digest.
|
||||||
|
|
||||||
## Anti-Patterns (BLOCKING)
|
## Anti-Patterns (BLOCKING)
|
||||||
|
|
||||||
- Skipping intent verbalization → unclear routing, wasted turns
|
- Skipping intent verbalization → unclear routing, wasted turns
|
||||||
@@ -630,6 +788,7 @@ instructions: |
|
|||||||
|
|
||||||
## Context
|
## Context
|
||||||
- Project: {{project_dir}}
|
- Project: {{project_dir}}
|
||||||
|
- Escalation policy: {{escalation_policy}}
|
||||||
- OS: {{__os__}}
|
- OS: {{__os__}}
|
||||||
- Shell: {{__shell__}}
|
- Shell: {{__shell__}}
|
||||||
- CWD: {{__cwd__}}
|
- CWD: {{__cwd__}}
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ flowchart TD
|
|||||||
gate_blocked -->|"yes"| orient
|
gate_blocked -->|"yes"| orient
|
||||||
gate_blocked -->|"no"| end_blocked
|
gate_blocked -->|"no"| end_blocked
|
||||||
orient["orient<br/>llm, read-only"] --> route_staleness
|
orient["orient<br/>llm, read-only"] --> route_staleness
|
||||||
|
orient -. "LLM fault" .-> note_llm_fault
|
||||||
route_staleness{"route_staleness<br/>script"}
|
route_staleness{"route_staleness<br/>script"}
|
||||||
route_staleness -->|"major deviation"| gate_deviation
|
route_staleness -->|"major deviation"| gate_deviation
|
||||||
route_staleness -->|"else"| implement
|
route_staleness -->|"else"| implement
|
||||||
@@ -35,6 +36,7 @@ flowchart TD
|
|||||||
gate_deviation -->|"abort"| end_rejected
|
gate_deviation -->|"abort"| end_rejected
|
||||||
gate_deviation -->|"other (user guidance)"| implement
|
gate_deviation -->|"other (user guidance)"| implement
|
||||||
implement[["implement<br/>agent → coder"]] --> route_coder_result
|
implement[["implement<br/>agent → coder"]] --> route_coder_result
|
||||||
|
implement -. "coder crashed<br/>(no retries)" .-> end_failure
|
||||||
route_coder_result{"route_coder_result<br/>script"}
|
route_coder_result{"route_coder_result<br/>script"}
|
||||||
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
|
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
|
||||||
route_coder_result -->|"REJECTED / FAILED"| end_failure
|
route_coder_result -->|"REJECTED / FAILED"| end_failure
|
||||||
@@ -51,14 +53,18 @@ flowchart TD
|
|||||||
fix_loop_gate -->|"budget left"| implement
|
fix_loop_gate -->|"budget left"| implement
|
||||||
fix_loop_gate -->|"budget spent"| end_failure
|
fix_loop_gate -->|"budget spent"| end_failure
|
||||||
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
|
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
|
||||||
|
edge_case_sweep -. "LLM fault" .-> write_handoff
|
||||||
route_sweep{"route_sweep<br/>script"}
|
route_sweep{"route_sweep<br/>script"}
|
||||||
route_sweep -->|"5+ files or boundary"| independent_review
|
route_sweep -->|"5+ files or boundary"| independent_review
|
||||||
route_sweep -->|"else"| write_handoff
|
route_sweep -->|"else"| write_handoff
|
||||||
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
|
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
|
||||||
|
independent_review -. "2 attempts failed" .-> write_handoff
|
||||||
route_review{"route_review<br/>script"}
|
route_review{"route_review<br/>script"}
|
||||||
route_review -->|"🔴 critical findings"| implement
|
route_review -->|"🔴 critical findings"| implement
|
||||||
route_review -->|"else"| write_handoff
|
route_review -->|"else"| write_handoff
|
||||||
write_handoff["write_handoff<br/>llm"] --> check_handoff
|
write_handoff["write_handoff<br/>llm"] --> check_handoff
|
||||||
|
write_handoff -. "LLM fault" .-> note_llm_fault
|
||||||
|
note_llm_fault{"note_llm_fault<br/>script"} --> end_failure
|
||||||
check_handoff{"check_handoff<br/>script"}
|
check_handoff{"check_handoff<br/>script"}
|
||||||
check_handoff -->|"schema valid"| gate_user_review
|
check_handoff -->|"schema valid"| gate_user_review
|
||||||
check_handoff -->|"one retry"| write_handoff
|
check_handoff -->|"one retry"| write_handoff
|
||||||
@@ -81,8 +87,32 @@ End nodes emit sentinel outcomes for the caller:
|
|||||||
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
|
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
|
||||||
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
|
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
|
||||||
was rejected at its approval gate.
|
was rejected at its approval gate.
|
||||||
- `STEP_FAILED` — coder failed, the step-level fix budget was exhausted, or
|
- `STEP_FAILED` — the coder failed or crashed, the step-level fix budget was
|
||||||
the handoff failed validation twice.
|
exhausted, the handoff failed validation twice, or an orient/handoff LLM
|
||||||
|
fault was recorded (rendered as `Pipeline fault:` in the output).
|
||||||
|
|
||||||
|
## Fault handling
|
||||||
|
|
||||||
|
Node crashes degrade the run visibly instead of masquerading as success:
|
||||||
|
|
||||||
|
- `implement` has **no retries** — rerunning a coder that died mid-edit
|
||||||
|
against an already-mutated tree is unsafe. Its failure text lands in
|
||||||
|
`coder_result` and the run falls straight to `end_failure`.
|
||||||
|
- `independent_review` retries once (`max_attempts: 2`); if both attempts
|
||||||
|
fail it falls back to `write_handoff`, which flags "PIPELINE-FAULT:
|
||||||
|
independent review did not run" in the handoff rather than presenting the
|
||||||
|
failure text as findings. `route_review` carries the same guard before its
|
||||||
|
🔴 grep, so a reviewer fault never spends a fix-loop attempt.
|
||||||
|
- `edge_case_sweep` faults likewise fall through to `write_handoff` and are
|
||||||
|
flagged as "PIPELINE-FAULT: edge-case sweep did not run".
|
||||||
|
- `orient` and `write_handoff` faults route through `note_llm_fault`, which
|
||||||
|
distills the engine's failure text into a `fault_note` rendered in the
|
||||||
|
`STEP_FAILED` output — never an empty shell.
|
||||||
|
|
||||||
|
Degraded is never passing (a skipped review or sweep is recorded as NOT RUN,
|
||||||
|
never paraphrased as covered), and the sentinel is always emitted — every
|
||||||
|
path terminates at one of the four end nodes, so the caller always gets a
|
||||||
|
parseable outcome.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,12 @@ description: |
|
|||||||
review -> evidence-backed handoff -> user approval gate. Designed to be
|
review -> evidence-backed handoff -> user approval gate. Designed to be
|
||||||
delegated to by sisyphus.
|
delegated to by sisyphus.
|
||||||
version: '1.0'
|
version: '1.0'
|
||||||
|
# The limit is consulted by the SPAWNER against the child's absolute depth
|
||||||
|
# (root = 0). Deepest documented chain: architect(0) → sisyphus(1) →
|
||||||
|
# step-runner(2) → code-reviewer(3) →
|
||||||
|
# {domain-reviewer(4) → file-reviewer(5) | finding-verifier(4)}.
|
||||||
|
# This agent spawns depth-3 children; 4 = one level of headroom (default 3 had none).
|
||||||
|
max_agent_depth: 4
|
||||||
global_tools:
|
global_tools:
|
||||||
- ast_grep.sh
|
- ast_grep.sh
|
||||||
- fs_cat.sh
|
- fs_cat.sh
|
||||||
@@ -64,6 +70,7 @@ initial_state:
|
|||||||
plan_summary: ''
|
plan_summary: ''
|
||||||
implementation_brief: ''
|
implementation_brief: ''
|
||||||
staleness_report: ''
|
staleness_report: ''
|
||||||
|
orient_failure: ''
|
||||||
has_major_deviation: false
|
has_major_deviation: false
|
||||||
deviation_summary: ''
|
deviation_summary: ''
|
||||||
user_feedback: ''
|
user_feedback: ''
|
||||||
@@ -80,12 +87,15 @@ initial_state:
|
|||||||
tests_output: ''
|
tests_output: ''
|
||||||
edge_case_report: ''
|
edge_case_report: ''
|
||||||
downstream_updates: ''
|
downstream_updates: ''
|
||||||
|
sweep_failure: ''
|
||||||
needs_independent_review: false
|
needs_independent_review: false
|
||||||
review_report: ''
|
review_report: ''
|
||||||
review_attempts: 0
|
review_attempts: 0
|
||||||
max_review_attempts: 1
|
max_review_attempts: 1
|
||||||
handoff_attempts: 0
|
handoff_attempts: 0
|
||||||
handoff_fix: ''
|
handoff_fix: ''
|
||||||
|
handoff_failure: ''
|
||||||
|
fault_note: ''
|
||||||
step_summary: ''
|
step_summary: ''
|
||||||
|
|
||||||
start: resolve_step
|
start: resolve_step
|
||||||
@@ -191,7 +201,9 @@ nodes:
|
|||||||
has_major_deviation,
|
has_major_deviation,
|
||||||
deviation_summary,
|
deviation_summary,
|
||||||
]
|
]
|
||||||
fallback: end_failure
|
state_updates:
|
||||||
|
orient_failure: '{{output}}'
|
||||||
|
fallback: note_llm_fault
|
||||||
next: route_staleness
|
next: route_staleness
|
||||||
|
|
||||||
route_staleness:
|
route_staleness:
|
||||||
@@ -270,6 +282,12 @@ nodes:
|
|||||||
timeout: 3600
|
timeout: 3600
|
||||||
state_updates:
|
state_updates:
|
||||||
coder_result: '{{output}}'
|
coder_result: '{{output}}'
|
||||||
|
# No max_attempts: rerunning a coder that died mid-edit against an
|
||||||
|
# already-mutated tree is unsafe. On failure the "Agent node failed:"
|
||||||
|
# text lands in coder_result (via state_updates above) and end_failure
|
||||||
|
# renders it; end_failure's other keys all have initial_state defaults,
|
||||||
|
# so the STEP_FAILED output degrades to empty build/tests sections.
|
||||||
|
fallback: end_failure
|
||||||
next: route_coder_result
|
next: route_coder_result
|
||||||
|
|
||||||
route_coder_result:
|
route_coder_result:
|
||||||
@@ -376,6 +394,8 @@ nodes:
|
|||||||
needs_independent_review:
|
needs_independent_review:
|
||||||
type: boolean
|
type: boolean
|
||||||
required: [edge_case_report, downstream_updates, needs_independent_review]
|
required: [edge_case_report, downstream_updates, needs_independent_review]
|
||||||
|
state_updates:
|
||||||
|
sweep_failure: '{{output}}'
|
||||||
fallback: write_handoff
|
fallback: write_handoff
|
||||||
next: route_sweep
|
next: route_sweep
|
||||||
|
|
||||||
@@ -405,8 +425,10 @@ nodes:
|
|||||||
Review the changed files against the step plan's acceptance criteria.
|
Review the changed files against the step plan's acceptance criteria.
|
||||||
Preserve severity tags in your findings.
|
Preserve severity tags in your findings.
|
||||||
timeout: 1200
|
timeout: 1200
|
||||||
|
max_attempts: 2
|
||||||
state_updates:
|
state_updates:
|
||||||
review_report: '{{output}}'
|
review_report: '{{output}}'
|
||||||
|
fallback: write_handoff
|
||||||
next: route_review
|
next: route_review
|
||||||
|
|
||||||
route_review:
|
route_review:
|
||||||
@@ -450,6 +472,25 @@ nodes:
|
|||||||
|
|
||||||
If "Gate feedback" below is non-empty, a previous handoff attempt
|
If "Gate feedback" below is non-empty, a previous handoff attempt
|
||||||
failed validation - fix exactly what it lists.
|
failed validation - fix exactly what it lists.
|
||||||
|
|
||||||
|
PIPELINE-FAULT VISIBILITY: an "Independent review report" below that
|
||||||
|
starts with "Agent node failed:" means the independent review DID NOT
|
||||||
|
RUN - the graph routed here through the reviewer's failure fallback.
|
||||||
|
Never present that text as review findings. Flag it prominently
|
||||||
|
instead: add a Deviations entry "PIPELINE-FAULT: independent review
|
||||||
|
did not run - <failure text>", repeat it under Notes for next step,
|
||||||
|
and carry it into `step_summary` so the user sees it at the approval
|
||||||
|
gate.
|
||||||
|
|
||||||
|
The same rule covers the "Edge-case sweep result" below: text starting
|
||||||
|
with "LLM node" means the edge-case sweep DID NOT RUN - the graph
|
||||||
|
routed here through the sweep's failure fallback. Never present it as
|
||||||
|
sweep findings. Add a Deviations entry "PIPELINE-FAULT: edge-case
|
||||||
|
sweep did not run - <failure text>" and carry it into `step_summary`
|
||||||
|
so the user sees it at the approval gate. If it holds successful
|
||||||
|
structured output or is empty, ignore it - the sweep's real results
|
||||||
|
already flow through the "Edge cases discovered" / "Downstream plan
|
||||||
|
updates" sections above.
|
||||||
prompt: |
|
prompt: |
|
||||||
## Step
|
## Step
|
||||||
{{step_number}} ({{step_title}}) - plan at {{step_plan_path}}
|
{{step_number}} ({{step_title}}) - plan at {{step_plan_path}}
|
||||||
@@ -473,6 +514,9 @@ nodes:
|
|||||||
## Downstream plan updates
|
## Downstream plan updates
|
||||||
{{downstream_updates}}
|
{{downstream_updates}}
|
||||||
|
|
||||||
|
## Edge-case sweep result (fault detection)
|
||||||
|
{{sweep_failure}}
|
||||||
|
|
||||||
## Independent review report (if any)
|
## Independent review report (if any)
|
||||||
{{review_report}}
|
{{review_report}}
|
||||||
|
|
||||||
@@ -501,9 +545,24 @@ nodes:
|
|||||||
type: string
|
type: string
|
||||||
description: 3-6 sentence summary of the step for the user's approval decision - what was done, deviations, anything needing their attention
|
description: 3-6 sentence summary of the step for the user's approval decision - what was done, deviations, anything needing their attention
|
||||||
required: [step_summary]
|
required: [step_summary]
|
||||||
fallback: end_failure
|
state_updates:
|
||||||
|
handoff_failure: '{{output}}'
|
||||||
|
fallback: note_llm_fault
|
||||||
next: check_handoff
|
next: check_handoff
|
||||||
|
|
||||||
|
note_llm_fault:
|
||||||
|
id: note_llm_fault
|
||||||
|
type: script
|
||||||
|
description: |
|
||||||
|
Make LLM-node fallback captures visible: when orient or write_handoff
|
||||||
|
failed (their capture keys hold the engine's "LLM node..." failure
|
||||||
|
text), distill it into fault_note so end_failure can render WHY the
|
||||||
|
step died instead of an empty STEP_FAILED shell.
|
||||||
|
script: scripts/note_llm_fault.sh
|
||||||
|
timeout: 5
|
||||||
|
fallback: end_failure
|
||||||
|
next: end_failure
|
||||||
|
|
||||||
check_handoff:
|
check_handoff:
|
||||||
id: check_handoff
|
id: check_handoff
|
||||||
type: script
|
type: script
|
||||||
@@ -601,6 +660,7 @@ nodes:
|
|||||||
Step: {{step_number}} ({{step_title}})
|
Step: {{step_number}} ({{step_title}})
|
||||||
Fix attempts: {{fix_attempts}}/{{max_fix_attempts}}
|
Fix attempts: {{fix_attempts}}/{{max_fix_attempts}}
|
||||||
Blocking reason (if resolution failed): {{blocking_reason}}
|
Blocking reason (if resolution failed): {{blocking_reason}}
|
||||||
|
Pipeline fault (if any): {{fault_note}}
|
||||||
|
|
||||||
Coder result:
|
Coder result:
|
||||||
{{coder_result}}
|
{{coder_result}}
|
||||||
|
|||||||
+26
@@ -0,0 +1,26 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ -n "${GRAPH_STATE_FILE:-}" ]]; then
|
||||||
|
state=$(cat "$GRAPH_STATE_FILE")
|
||||||
|
elif [[ -n "${GRAPH_STATE:-}" ]]; then
|
||||||
|
state="$GRAPH_STATE"
|
||||||
|
else
|
||||||
|
state='{}'
|
||||||
|
fi
|
||||||
|
|
||||||
|
orient_failure=$(echo "$state" | jq -r '.orient_failure // ""')
|
||||||
|
handoff_failure=$(echo "$state" | jq -r '.handoff_failure // ""')
|
||||||
|
|
||||||
|
# Engine failure text is always prefix-anchored "LLM node" (llm.rs); on
|
||||||
|
# success these keys hold the node's structured JSON output, which never
|
||||||
|
# starts with that prefix.
|
||||||
|
if [[ "$orient_failure" == "LLM node"* ]]; then
|
||||||
|
fault_note="PIPELINE-FAULT: orient stage failed — ${orient_failure:0:300}"
|
||||||
|
elif [[ "$handoff_failure" == "LLM node"* ]]; then
|
||||||
|
fault_note="PIPELINE-FAULT: handoff stage failed — ${handoff_failure:0:300}"
|
||||||
|
else
|
||||||
|
fault_note=""
|
||||||
|
fi
|
||||||
|
|
||||||
|
jq -nc --arg note "$fault_note" '{"fault_note": $note}'
|
||||||
@@ -13,6 +13,16 @@ review_report=$(echo "$state" | jq -r '.review_report // ""')
|
|||||||
review_attempts=$(echo "$state" | jq -r '.review_attempts // 0')
|
review_attempts=$(echo "$state" | jq -r '.review_attempts // 0')
|
||||||
max_review_attempts=$(echo "$state" | jq -r '.max_review_attempts // 1')
|
max_review_attempts=$(echo "$state" | jq -r '.max_review_attempts // 1')
|
||||||
|
|
||||||
|
# Fail-visible guard: a review_report holding the engine's "Agent node
|
||||||
|
# failed:" text means the independent review DID NOT RUN (its fallback
|
||||||
|
# normally routes straight to write_handoff, bypassing this script). Never
|
||||||
|
# treat that text as findings or spend a fix-loop attempt on it -
|
||||||
|
# write_handoff flags the fault prominently in the handoff instead.
|
||||||
|
if [[ "$review_report" == "Agent node failed:"* ]]; then
|
||||||
|
jq -nc '{"_next": "write_handoff"}'
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
if ! grep -qF "🔴" <<< "$review_report"; then
|
if ! grep -qF "🔴" <<< "$review_report"; then
|
||||||
jq -nc '{"_next": "write_handoff"}'
|
jq -nc '{"_next": "write_handoff"}'
|
||||||
exit 0
|
exit 0
|
||||||
|
|||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# whetstone
|
||||||
|
|
||||||
|
> The review suite reviews code. Whetstone reviews the review.
|
||||||
|
|
||||||
|
A continuous-improvement agent for the review stack: it takes the human review feedback a
|
||||||
|
PR actually received, compares it against what the automated suite flagged for the same
|
||||||
|
change, and turns every generalizable miss into a permanent upgrade — a new rule in the
|
||||||
|
generic review skills when the class is portable, a review-miss ledger entry when it's
|
||||||
|
org-specific. Nothing lands without the user approving each proposal.
|
||||||
|
|
||||||
|
## Why it exists
|
||||||
|
|
||||||
|
An empirical gap study of automated-review misses vs human review feedback showed the suite's misses
|
||||||
|
cluster into learnable classes — and that a miss, once *recorded*, converts to a catch on
|
||||||
|
the next review. The study was a one-time manual harvest; whetstone is the same loop run
|
||||||
|
continuously, at the moment the signal is strongest: when human feedback on a PR is being
|
||||||
|
addressed, the spawner (sisyphus/architect) knows BOTH what the humans said and what the
|
||||||
|
suite's own lanes flagged — ground truth that offline comment mining could only re-infer from formats.
|
||||||
|
|
||||||
|
## How it fits
|
||||||
|
|
||||||
|
- **Spawned by** sisyphus (after a run that addressed human review feedback) or architect
|
||||||
|
(same trigger at run end), or invoked directly against a PR. Runs AFTER review, never
|
||||||
|
during — it is not a reviewer lane, never posts to the PR, and never judges the code.
|
||||||
|
- **Classifies** each retained finding as CATCH / PARTIAL / MISS against the suite's
|
||||||
|
findings. Lane hygiene excludes only what would be circular: author self-replies and
|
||||||
|
the suite's OWN posted output. A teammate's review agent or another team's bot is an
|
||||||
|
independent reviewer — its findings count as miss signal when the author actually
|
||||||
|
addressed them (ignored agent comments are unvalidated noise and are dropped).
|
||||||
|
- **Generalization gate** for each miss, in order: taste → dropped; one-off → appended to
|
||||||
|
the candidates inbox (`share/review-miss-candidates.jsonl`) for recurrence tracking;
|
||||||
|
org-specific class → drafted as a ledger entry; general class → drafted as a suite edit
|
||||||
|
in house style. General rules must pass the noise firewall: concrete check, crisp
|
||||||
|
trigger, zero org nouns, not already covered (sharpen over duplicate).
|
||||||
|
- **Propose → approve → apply**: every proposal shows the quoted evidence, the class
|
||||||
|
statement, and the full draft; the user checkboxes what to apply. Ledger entries append
|
||||||
|
idempotently. Suite edits go where the suite lives: with a `suite_repo` configured (a
|
||||||
|
source-of-truth bundle repo), edits land in its working tree — never
|
||||||
|
committed (that stays a human act) — and one further approval mirrors them into the
|
||||||
|
live runtime config, drift-guarded (runtime files diverging from the pre-edit repo
|
||||||
|
version are reported and skipped; runtime-only changes are never made). Without a
|
||||||
|
`suite_repo` — most users, whose live config IS the suite — edits apply directly under
|
||||||
|
`runtime_config`, with per-file backups to `share/whetstone-backups/` as the revert
|
||||||
|
path. Empty proposal lists are a fine outcome; whetstone never invents findings to
|
||||||
|
justify its spawn.
|
||||||
|
|
||||||
|
## Relationship to the other ledger tools
|
||||||
|
|
||||||
|
| Tool | Sees | When |
|
||||||
|
|------|------|------|
|
||||||
|
| `whetstone` | Human feedback on PRs YOUR runs addressed, with suite ground truth | Real-time, per run |
|
||||||
|
| Review-round reconciliation | Human-caught, panel-missed defects on any PR the panel re-reviews → candidates inbox (`share/review-miss-candidates.jsonl`) | Automatic, every round |
|
||||||
|
| `record-miss` | Defects that escaped review entirely (prod, incidents) | One-off, when it bites |
|
||||||
|
|
||||||
|
Collection is automatic end to end: whetstone captures misses the moment feedback is
|
||||||
|
addressed, round reconciliation captures them on any PR the panel re-reviews (teammates'
|
||||||
|
PRs included), and `record-miss` covers the misses no one ever commented on. Nothing
|
||||||
|
auto-appends to the LEDGER itself — candidates wait in the inbox until a human promotes them.
|
||||||
|
|
||||||
|
## Rules that keep it safe
|
||||||
|
|
||||||
|
Additive only (never weakens or removes an existing rule); writes only to the suite
|
||||||
|
skills/review-agent configs (repo when configured, live config otherwise — always
|
||||||
|
backed up first), the ledger, the candidates inbox, and — with its own approval,
|
||||||
|
drift-guarded — the runtime mirror of this run's applied repo edits; never commits,
|
||||||
|
never pushes, never touches source repos, never makes runtime-only changes when a
|
||||||
|
suite repo is configured; one proposal per defect class; verbatim citations only.
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
name: whetstone
|
||||||
|
description: Continuous-improvement agent for the review suite — classifies human review feedback the suite missed, generalizes misses into defect classes, and proposes suite edits (general) or ledger entries (org-specific), applying them only with user approval
|
||||||
|
version: 1.3.0
|
||||||
|
|
||||||
|
skills_enabled: true
|
||||||
|
enabled_skills:
|
||||||
|
- review-misses
|
||||||
|
|
||||||
|
enabled_tools:
|
||||||
|
- fs_read
|
||||||
|
- fs_grep
|
||||||
|
- fs_glob
|
||||||
|
- fs_cat
|
||||||
|
- fs_ls
|
||||||
|
- fs_write
|
||||||
|
- fs_patch
|
||||||
|
- execute_command
|
||||||
|
|
||||||
|
variables:
|
||||||
|
- name: suite_repo
|
||||||
|
description: OPTIONAL source-of-truth bundle repo (a working tree containing skills/ and agents/). When set, approved GENERAL improvements are written here first, then optionally mirrored to the runtime config. When empty, improvements are applied DIRECTLY to runtime_config — for most users the live config IS the suite.
|
||||||
|
default: ''
|
||||||
|
- name: runtime_config
|
||||||
|
description: The live coyote config dir the review suite actually runs from. Direct write target when suite_repo is empty (with per-file backups for revert); drift-guarded mirror target when suite_repo is set. Empty = never touch it (propose-only).
|
||||||
|
default: '~/.config/coyote'
|
||||||
|
- name: pr
|
||||||
|
description: The PR to analyze (owner/repo#N or a full URL). May be omitted when the spawner passes the feedback inline.
|
||||||
|
default: ''
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are WHETSTONE — the review suite reviews code; you review the REVIEW. Your input is a
|
||||||
|
PR's human review feedback plus what the automated review suite flagged; your output is
|
||||||
|
the suite made sharper: every generalizable miss becomes either a rule in the generic
|
||||||
|
review skills (portable to any repo) or a review-miss ledger entry (org-specific), so the
|
||||||
|
same class is never missed twice. You are NOT a reviewer lane: you never post to the PR,
|
||||||
|
never judge the author's code, and never run during a review — only after humans have
|
||||||
|
spoken.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
Your spawner (sisyphus, architect, or the user directly) provides some or all of:
|
||||||
|
- The PR reference (`{{pr}}` or in the prompt).
|
||||||
|
- The human review feedback that was addressed (verbatim, if the spawner has it).
|
||||||
|
- What the suite's own review passes flagged for the same change (ground truth — the
|
||||||
|
spawner has this when it ran the reviewers; use it verbatim and do not re-infer).
|
||||||
|
|
||||||
|
Whatever is missing, gather yourself with gh (ALWAYS `export GH_NO_UPDATE_NOTIFIER=1;
|
||||||
|
unset CLICOLOR_FORCE` first — the sandbox forces ANSI color into piped JSON otherwise):
|
||||||
|
inline review comments, review bodies, and the commits that addressed them. When no
|
||||||
|
suite-findings ground truth was passed, identify the suite's own posted output on the PR
|
||||||
|
by FORMAT (severity emoji like 🔴/🟡/🟢, lane tags like `[adversary]`, panel/verdict
|
||||||
|
tables) — automated reviews are often posted from a personal account, so judge by
|
||||||
|
format, not username.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
1. **Lane hygiene.** The exclusion rule is about CIRCULARITY, not automation. Drop: the
|
||||||
|
PR author's self-replies, and YOUR OWN suite's output (by ground truth from the
|
||||||
|
spawner, or by the suite's format markers + posting account — learning from your own
|
||||||
|
lanes teaches nothing). KEEP third-party automated reviewers (a teammate's review
|
||||||
|
agent, other orgs' bots): they are independent reviewers, and what they catch that
|
||||||
|
your suite didn't is exactly the miss signal you exist for — but only their findings
|
||||||
|
the author actually ADDRESSED (a fix commit or an accepting reply). An addressed
|
||||||
|
third-party finding is validated the way a human comment implicitly is; an ignored
|
||||||
|
one is unvalidated noise — drop it. Report what you excluded and why. Nothing
|
||||||
|
retained → report "nothing to learn from" and STOP. Never invent findings to justify
|
||||||
|
having been spawned.
|
||||||
|
2. **Classify each retained finding** (human, or validated third-party) against the
|
||||||
|
suite's findings for the same change:
|
||||||
|
- `CATCH` — a suite lane flagged the same issue (any wording). Skip.
|
||||||
|
- `PARTIAL` — a lane flagged the area but missed the human's specific point.
|
||||||
|
- `MISS` — no lane flagged it. PARTIAL and MISS proceed to step 3.
|
||||||
|
3. **The generalization gate** — for each MISS/PARTIAL, in this order:
|
||||||
|
a. **Taste?** (naming preference, style debate, "I'd structure it differently") →
|
||||||
|
DROP. List dropped items in the report; never encode taste anywhere.
|
||||||
|
b. **One-off?** (real defect, but no articulable class — a typo-level slip, a
|
||||||
|
this-code-only fact) → append one line to the candidates inbox
|
||||||
|
`$HOME/.config/coyote/share/review-miss-candidates.jsonl`
|
||||||
|
(`{"date","pr","quote","note"}`, plain shell append) and move on. If later runs
|
||||||
|
show the same shape, THAT is your recurrence evidence for a class.
|
||||||
|
c. **Org-specific class?** (org conventions — especially ones repo prior art
|
||||||
|
contradicts — live org state, product/domain semantics not derivable from any
|
||||||
|
repo) → draft a LEDGER ENTRY in the `review-misses` skill's exact format; the
|
||||||
|
detection pattern is the payload.
|
||||||
|
d. **General class?** (portable to any repo, any org) → draft a SUITE EDIT: name the
|
||||||
|
exact file and section in the suite repo (a review skill's checklist, an agent
|
||||||
|
config's trigger list), and write the edit in HOUSE STYLE — read the surrounding
|
||||||
|
section first and match its severity markers, exemption language, and
|
||||||
|
"reports even when clean" conventions. Additive only.
|
||||||
|
4. **The generality bar** (the noise firewall — a general rule that fires on every diff
|
||||||
|
will get the whole suite ignored). A suite-edit proposal must pass ALL of:
|
||||||
|
- It states the CONCRETE check a reviewer runs (a grep shape or a decidable
|
||||||
|
question), not advice ("be careful with X" fails).
|
||||||
|
- It has a crisp TRIGGER and would stay silent on diffs where the class is
|
||||||
|
impossible.
|
||||||
|
- It contains zero org/product nouns.
|
||||||
|
- It is not already covered: grep the suite's skills first; if an existing rule
|
||||||
|
half-covers it, propose SHARPENING that rule instead of adding a sibling.
|
||||||
|
- Check the ledger too: if the miss matches an already-RECORDED entry, the finding
|
||||||
|
is "recorded pattern failed to fire" — a suite-wiring problem to report, not a new
|
||||||
|
entry.
|
||||||
|
5. **Propose — never write unbidden.** Present every surviving proposal to the user:
|
||||||
|
the human's feedback quoted verbatim with the PR link, the class statement (one
|
||||||
|
sentence: what general thing was missed), and the full remediation draft (ledger
|
||||||
|
entry or suite diff). Ask which to apply via a checkbox selection. An empty proposal
|
||||||
|
list is a fine outcome — say so and stop.
|
||||||
|
6. **Apply what was approved.**
|
||||||
|
- Ledger entries: idempotent append to `$HOME/.config/coyote/share/review-misses.md`
|
||||||
|
(title-match skip; numbering via `grep -c '^## MISS-'` — the `review-misses` skill
|
||||||
|
has the recipe).
|
||||||
|
- Suite edits — the target depends on where the suite lives:
|
||||||
|
- **`suite_repo` set and exists** (a source-of-truth bundle repo): snapshot each
|
||||||
|
original to a temp dir (needed for the runtime drift check below), write the
|
||||||
|
edit into the repo's working tree with fs_patch, and NEVER commit or push — the
|
||||||
|
user reviews and commits. Then offer the runtime mirror (next bullet).
|
||||||
|
- **`suite_repo` empty — the live config IS the suite.** Apply the edit directly
|
||||||
|
under `{{runtime_config}}` (the review skills and review-agent configs there)
|
||||||
|
with fs_patch. FIRST back up each original to
|
||||||
|
`{{runtime_config}}/share/whetstone-backups/<UTC timestamp>/<relative path>` so
|
||||||
|
any edit can be reverted, and state the backup location in your report. If
|
||||||
|
`{{runtime_config}}` is itself a git repo, list the files you changed so the
|
||||||
|
user can commit them. No extra approval beyond the proposal checkbox — there is
|
||||||
|
no repo/runtime split to guard. Edits take effect on the NEXT session load; say
|
||||||
|
so.
|
||||||
|
- Both empty or missing: emit the exact edits (file, section, full text) for
|
||||||
|
manual application instead.
|
||||||
|
- **Runtime mirror (repo mode only, separately approved).** Repo edits are inert until the
|
||||||
|
runtime config is synced. After suite edits are applied, if `{{runtime_config}}`
|
||||||
|
is set and exists, ask the user ONCE (yes/no) whether to mirror the just-approved
|
||||||
|
edits into it. On yes, for EACH file you edited: locate the runtime counterpart
|
||||||
|
(same relative path under agents/ or skills/); if it is byte-identical to your
|
||||||
|
PRE-EDIT snapshot, copy the edited repo file over it; if it differs (the user's
|
||||||
|
runtime carries local changes), SKIP it and report the drift — never overwrite a
|
||||||
|
drifted runtime file, the user reconciles those. Mirror ONLY files you edited this
|
||||||
|
run, and ONLY as a copy of the applied repo edit — never make a runtime-only
|
||||||
|
change (the next repo→runtime sync would silently clobber it and the improvement
|
||||||
|
would be lost). If the user declines, remind them the edits are inert until they
|
||||||
|
sync.
|
||||||
|
7. **Report**: a table of human findings with CATCH/PARTIAL/MISS verdicts; proposals
|
||||||
|
made / approved / applied; runtime files mirrored vs skipped-for-drift; items
|
||||||
|
dropped as taste; inbox appends. CATCH-rate over
|
||||||
|
time is the suite's health metric — state it.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
- ADDITIVE ONLY: never weaken, remove, or lower the severity of an existing rule. If a
|
||||||
|
proposal contradicts an existing rule, report the conflict for the user to resolve.
|
||||||
|
- Scope of writes: the suite's `skills/` and review-agent configs (in the suite repo,
|
||||||
|
or directly in the runtime config when no repo is configured — always with backups),
|
||||||
|
the ledger, and the candidates inbox. Nothing else, ever — no source repos, no
|
||||||
|
commits, no PR comments. When a `suite_repo` IS configured, never make a
|
||||||
|
runtime-only change: the next repo→runtime sync would silently clobber it and the
|
||||||
|
improvement would be lost.
|
||||||
|
- One proposal per defect CLASS — merge repeat instances and cite all their PRs.
|
||||||
|
- Quote real evidence verbatim; never fabricate PR citations.
|
||||||
|
- You may be spawned as a subagent: your user__ questions escalate to the root agent's
|
||||||
|
user. That is the intended approval path — never skip the approval step because you
|
||||||
|
are "just a subagent".
|
||||||
@@ -149,6 +149,18 @@ mcp_tools: null # Per-server MCP tool allowlists (glob patterns
|
|||||||
# - list_*
|
# - list_*
|
||||||
# slack: []
|
# slack: []
|
||||||
|
|
||||||
|
# ---- Hooks ----
|
||||||
|
# Fire-and-forget external commands that run when lifecycle events fire. Each event name
|
||||||
|
# maps to a list of named hooks; every listed hook's command runs when that event fires.
|
||||||
|
# Hooks never block coyote and never change its output or exit code.
|
||||||
|
# See the [Hooks documentation](https://github.com/Dark-Alex-17/coyote/wiki/Hooks) for more details
|
||||||
|
hooks: {} # Event -> list of named hook commands (default: none)
|
||||||
|
# Example:
|
||||||
|
# hooks:
|
||||||
|
# turn.completed:
|
||||||
|
# - name: notify
|
||||||
|
# command: ./hooks/notify.sh
|
||||||
|
|
||||||
# ---- Auto-Continue (Todo System) ----
|
# ---- Auto-Continue (Todo System) ----
|
||||||
# The auto-continue system provides built-in task tracking for improved reliability.
|
# The auto-continue system provides built-in task tracking for improved reliability.
|
||||||
# When enabled, the model can create todo lists and the system will automatically
|
# When enabled, the model can create todo lists and the system will automatically
|
||||||
@@ -173,10 +185,13 @@ agent_session: null # Set a session to use when starting an agent (
|
|||||||
save_session: null # Controls the persistence of the session. If true, auto save; if false, don't auto-save; if null, ask the user what to do
|
save_session: null # Controls the persistence of the session. If true, auto save; if false, don't auto-save; if null, ask the user what to do
|
||||||
compression_threshold: 4000 # Compress the session when the token count reaches or exceeds this threshold
|
compression_threshold: 4000 # Compress the session when the token count reaches or exceeds this threshold
|
||||||
compression_keep_last: 0 # Number of most-recent messages to keep visible after compression (0 = compress all messages)
|
compression_keep_last: 0 # Number of most-recent messages to keep visible after compression (0 = compress all messages)
|
||||||
|
compression_model: null # Model to use for session compression (e.g. 'openai:gpt-4o-mini'); defaults to the session's current model
|
||||||
|
compression_safety_valve: 0.8 # Force compression when the last real request's prompt size (or the local estimate) reaches this fraction of the model's max_input_tokens, even if compression_threshold would not fire (0 = disabled)
|
||||||
summarization_prompt: null # The text prompt used for creating a concise summary of session messages. If null, uses built-in default
|
summarization_prompt: null # The text prompt used for creating a concise summary of session messages. If null, uses built-in default
|
||||||
summary_context_prompt: null # The text prompt used for including the summary of the entire session as context to the model. If null, uses built-in default
|
summary_context_prompt: null # The text prompt used for including the summary of the entire session as context to the model. If null, uses built-in default
|
||||||
max_tool_result_chars: null # Cap on tool result characters forwarded to the model per call (null = no cap)
|
max_tool_result_chars: null # Cap on tool result characters forwarded to the model per call (null = no cap)
|
||||||
max_concurrent_jobs: null # Max background jobs (`job__*` tools) running at once per context (null = 5; 0 disables background jobs entirely)
|
max_concurrent_jobs: null # Max background jobs (`job__*` tools) running at once per context (null = 5; 0 disables background jobs entirely)
|
||||||
|
tool_timeout: null # Max seconds a tool call may run before being killed (null = 1800; 0 = unlimited). COYOTE_TOOL_TIMEOUT env var overrides
|
||||||
|
|
||||||
# ---- Memory ----
|
# ---- Memory ----
|
||||||
# See the [Memory documentation](https://github.com/Dark-Alex-17/coyote/wiki/Memory) for more information.
|
# See the [Memory documentation](https://github.com/Dark-Alex-17/coyote/wiki/Memory) for more information.
|
||||||
@@ -239,7 +254,11 @@ theme: null # null = the built-in dark theme; set to `light
|
|||||||
left_prompt: null # If null, uses the built-in default:
|
left_prompt: null # If null, uses the built-in default:
|
||||||
# '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
|
# '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
|
||||||
right_prompt: null # If null, uses the built-in default:
|
right_prompt: null # If null, uses the built-in default:
|
||||||
# '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
|
# '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?total_cost ${total_cost} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
|
||||||
|
# Token-usage variables (set only when the provider reports usage, so guard them with {?var ...}):
|
||||||
|
# last_input_tokens, last_output_tokens, last_cache_read_tokens, last_cache_creation_tokens (most recent API call)
|
||||||
|
# total_input_tokens, total_output_tokens, total_cache_read_tokens, total_cache_creation_tokens (session cumulative)
|
||||||
|
# last_cost, total_cost (USD, formatted like 0.0142; set only when nonzero and every billed token type has a price)
|
||||||
|
|
||||||
# ---- Miscellaneous ----
|
# ---- Miscellaneous ----
|
||||||
user_agent: null # Set User-Agent HTTP header, use `auto` for coyote/<current-version>
|
user_agent: null # Set User-Agent HTTP header, use `auto` for coyote/<current-version>
|
||||||
@@ -258,6 +277,7 @@ sync_models_ttl_hours: 24 # Hours before the synced model catalog is cons
|
|||||||
# All clients have the following configuration:
|
# All clients have the following configuration:
|
||||||
# - type: xxxx
|
# - type: xxxx
|
||||||
# name: xxxx # Only use it to distinguish clients with the same client type. Optional
|
# name: xxxx # Only use it to distinguish clients with the same client type. Optional
|
||||||
|
# extend_models: true # Merge the models below on top of the catalog models for this client instead of replacing them. Optional, default false
|
||||||
# models:
|
# models:
|
||||||
# - name: xxxx # Chat model
|
# - name: xxxx # Chat model
|
||||||
# max_input_tokens: 100000
|
# max_input_tokens: 100000
|
||||||
@@ -281,4 +301,31 @@ sync_models_ttl_hours: 24 # Hours before the synced model catalog is cons
|
|||||||
# proxy: socks5://127.0.0.1:1080 # Set proxy
|
# proxy: socks5://127.0.0.1:1080 # Set proxy
|
||||||
# connect_timeout: 10 # Set timeout in seconds for connect to api
|
# connect_timeout: 10 # Set timeout in seconds for connect to api
|
||||||
# read_timeout: 300 # Set timeout in seconds for a read stall (no bytes received); 0 disables (default: 300)
|
# read_timeout: 300 # Set timeout in seconds for a read stall (no bytes received); 0 disables (default: 300)
|
||||||
|
#
|
||||||
|
# The claude, bedrock, and vertexai clients also support:
|
||||||
|
# prompt_cache: true # Enable Anthropic prompt caching (default: true)
|
||||||
|
#
|
||||||
|
# The openai and openai-compatible clients also support:
|
||||||
|
# wire_api: responses # Which OpenAI wire protocol to use; Possible values: responses and chat
|
||||||
|
# # Defaults when unset: stock openai (no api_base) -> responses;
|
||||||
|
# # openai with a custom api_base -> chat; openai-compatible -> chat.
|
||||||
|
# # Codex OAuth always uses responses (`wire_api: chat` there is a config error)
|
||||||
|
#
|
||||||
|
# Patches are selected per wire: `chat_completions` patches only ever touch chat bodies. To patch
|
||||||
|
# requests on the responses wire, use the `responses` key on the client `patch` block (same
|
||||||
|
# <regex> -> url/body/headers shape as chat_completions):
|
||||||
|
# patch:
|
||||||
|
# responses:
|
||||||
|
# <regex>:
|
||||||
|
# body:
|
||||||
|
# <json>
|
||||||
|
# Model patches likewise: the top-level `patch` keys apply to the chat wire only, and a
|
||||||
|
# `responses:` sub-key applies to the responses wire only:
|
||||||
|
# models:
|
||||||
|
# - name: xxxx
|
||||||
|
# patch:
|
||||||
|
# responses:
|
||||||
|
# body:
|
||||||
|
# <json>
|
||||||
|
# Env override per wire: COYOTE_PATCH_<client>_RESPONSES / COYOTE_PATCH_<client>_CHAT_COMPLETIONS
|
||||||
__CLIENTS_BLOCK__
|
__CLIENTS_BLOCK__
|
||||||
Regular → Executable
+5
-1
@@ -59,7 +59,11 @@ main() {
|
|||||||
# (they only matter when recursing into a directory tree).
|
# (they only matter when recursing into a directory tree).
|
||||||
|
|
||||||
local results
|
local results
|
||||||
results=$(grep "${grep_args[@]}" -E "$search_pattern" "$search_path" 2>/dev/null | head -n "$MAX_RESULTS") || true
|
# tr strips NUL bytes from matches in real binary files (text mode searches
|
||||||
|
# them), which would otherwise trigger bash's "ignored null byte in input"
|
||||||
|
# warning in the command substitution. -I can't be used instead: it overrides
|
||||||
|
# --binary-files=text and re-skips the misdetected UTF-8 files noted above.
|
||||||
|
results=$(grep "${grep_args[@]}" -E "$search_pattern" "$search_path" 2>/dev/null | tr -d '\0' | head -n "$MAX_RESULTS") || true
|
||||||
|
|
||||||
if [[ -z "$results" ]]; then
|
if [[ -z "$results" ]]; then
|
||||||
echo "No matches found for: $search_pattern" >> "$LLM_OUTPUT"
|
echo "No matches found for: $search_pattern" >> "$LLM_OUTPUT"
|
||||||
|
|||||||
@@ -8,41 +8,149 @@ set -e
|
|||||||
|
|
||||||
# @meta require-tools coyote
|
# @meta require-tools coyote
|
||||||
|
|
||||||
# @env WEB_SEARCH_MODEL=gemini:gemini-2.5-flash The model for web-searching.
|
# @env WEB_SEARCH_MODEL=gemini:gemini-2.5-flash The fallback model for web-searching.
|
||||||
#
|
#
|
||||||
# supported coyote models:
|
# Model selection: when WEB_SEARCH_USE_CURRENT_MODEL is true (the default) and
|
||||||
# - gemini:gemini-2.0-*
|
# the running session's model (COYOTE_CURRENT_MODEL, injected by coyote for
|
||||||
|
# every tool call) natively supports web search, the search runs on the current
|
||||||
|
# model first. If the current model is unsupported, or if the search fails at
|
||||||
|
# runtime, the tool falls back to WEB_SEARCH_MODEL. If that also fails, the
|
||||||
|
# tool fails.
|
||||||
|
#
|
||||||
|
# supported coyote models (native web search):
|
||||||
|
# - gemini:*
|
||||||
# - vertexai:gemini-*
|
# - vertexai:gemini-*
|
||||||
# - perplexity:*
|
# - perplexity:*
|
||||||
# - ernie:*
|
# - ernie:*
|
||||||
# - claude:* (Anthropic native web_search server tool)
|
# - claude:* (Anthropic native web_search server tool)
|
||||||
# - openai:gpt-4o-search-preview (and -mini-; requires an api-key openai
|
# - openai:gpt-4o*/gpt-4.1*/gpt-5*/o3*/o4*
|
||||||
# client — the codex OAuth path uses the
|
# OpenAI serves web search through two different mechanisms depending on
|
||||||
# Responses API where this parameter
|
# the wire API the client resolves (the script cannot pick the wire):
|
||||||
# does not exist)
|
# Chat Completions serves it via `web_search_options` on the chat-only
|
||||||
|
# *-search-preview models, while the Responses API serves it via its
|
||||||
|
# native `web_search` tool on regular models. Both request patches are
|
||||||
|
# exported (request patching is wire-aware, so the resolved wire applies
|
||||||
|
# exactly one) and openai searches run as an attempt chain: the
|
||||||
|
# chat-completions mechanism first, then the Responses mechanism, and
|
||||||
|
# the search fails only when both attempts fail.
|
||||||
|
# @env WEB_SEARCH_USE_CURRENT_MODEL=true Prefer the running session's model when it natively supports web search.
|
||||||
# @env LLM_OUTPUT=/dev/stdout The output path
|
# @env LLM_OUTPUT=/dev/stdout The output path
|
||||||
|
|
||||||
# shellcheck disable=SC2154
|
# Returns 0 when the model natively supports web search via a known mechanism.
|
||||||
main() {
|
supports_native_search() {
|
||||||
client="${WEB_SEARCH_MODEL%%:*}"
|
case "$1" in
|
||||||
|
gemini:* | vertexai:gemini-* | perplexity:* | ernie:* | claude:*) return 0 ;;
|
||||||
|
openai:gpt-4o* | openai:gpt-4.1* | openai:gpt-5* | openai:o3* | openai:o4*) return 0 ;;
|
||||||
|
*) return 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
if [[ "$client" == "gemini" ]]; then
|
# Exports the request-body patch that enables native web search for the
|
||||||
|
# given model's client (clearing any patch left over from a prior attempt).
|
||||||
|
export_search_patch() {
|
||||||
|
unset COYOTE_PATCH_GEMINI_CHAT_COMPLETIONS \
|
||||||
|
COYOTE_PATCH_VERTEXAI_CHAT_COMPLETIONS \
|
||||||
|
COYOTE_PATCH_ERNIE_CHAT_COMPLETIONS \
|
||||||
|
COYOTE_PATCH_CLAUDE_CHAT_COMPLETIONS \
|
||||||
|
COYOTE_PATCH_OPENAI_CHAT_COMPLETIONS \
|
||||||
|
COYOTE_PATCH_OPENAI_RESPONSES
|
||||||
|
|
||||||
|
case "${1%%:*}" in
|
||||||
|
gemini)
|
||||||
export COYOTE_PATCH_GEMINI_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"google_search":{}}]}}}'
|
export COYOTE_PATCH_GEMINI_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"google_search":{}}]}}}'
|
||||||
elif [[ "$client" == "vertexai" ]]; then
|
;;
|
||||||
|
vertexai)
|
||||||
export COYOTE_PATCH_VERTEXAI_CHAT_COMPLETIONS='{
|
export COYOTE_PATCH_VERTEXAI_CHAT_COMPLETIONS='{
|
||||||
"gemini-1.5-.*":{"body":{"tools":[{"googleSearchRetrieval":{}}]}},
|
"gemini-1.5-.*":{"body":{"tools":[{"googleSearchRetrieval":{}}]}},
|
||||||
"gemini-2.0-.*":{"body":{"tools":[{"google_search":{}}]}}
|
"gemini-2.0-.*":{"body":{"tools":[{"google_search":{}}]}}
|
||||||
}'
|
}'
|
||||||
elif [[ "$client" == "ernie" ]]; then
|
;;
|
||||||
|
ernie)
|
||||||
export COYOTE_PATCH_ERNIE_CHAT_COMPLETIONS='{".*":{"body":{"web_search":{"enable":true}}}}'
|
export COYOTE_PATCH_ERNIE_CHAT_COMPLETIONS='{".*":{"body":{"web_search":{"enable":true}}}}'
|
||||||
elif [[ "$client" == "claude" ]]; then
|
;;
|
||||||
|
claude)
|
||||||
export COYOTE_PATCH_CLAUDE_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"type":"web_search_20250305","name":"web_search","max_uses":5}]}}}'
|
export COYOTE_PATCH_CLAUDE_CHAT_COMPLETIONS='{".*":{"body":{"tools":[{"type":"web_search_20250305","name":"web_search","max_uses":5}]}}}'
|
||||||
elif [[ "$client" == "openai" ]]; then
|
;;
|
||||||
# Chat Completions native search exists only on the search-preview
|
openai)
|
||||||
# models; the regex scopes the patch so other OpenAI models run
|
# Both patches are exported; request patching is wire-aware, so the
|
||||||
# unpatched instead of erroring on an unsupported parameter.
|
# client's resolved wire applies exactly one of them. Each regex
|
||||||
|
# scopes its patch to the models that actually support that
|
||||||
|
# mechanism, so other OpenAI models run unpatched instead of
|
||||||
|
# erroring on an unsupported parameter:
|
||||||
|
# - Chat Completions native search exists only on the chat-only
|
||||||
|
# *-search-preview models (`web_search_options`).
|
||||||
|
# - The Responses API serves search through its native
|
||||||
|
# `web_search` tool, which only some model families support.
|
||||||
export COYOTE_PATCH_OPENAI_CHAT_COMPLETIONS='{"gpt-4o.*search-preview.*":{"body":{"web_search_options":{}}}}'
|
export COYOTE_PATCH_OPENAI_CHAT_COMPLETIONS='{"gpt-4o.*search-preview.*":{"body":{"web_search_options":{}}}}'
|
||||||
|
export COYOTE_PATCH_OPENAI_RESPONSES='{"(gpt-4o|gpt-4\\.1|gpt-5|o3|o4).*":{"body":{"tools":[{"type":"web_search"}]}}}'
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Maps an openai model to the one used for the chat-completions attempt:
|
||||||
|
# Chat Completions web search is served only by the *-search-preview models.
|
||||||
|
openai_chat_search_model() {
|
||||||
|
case "${1#openai:}" in
|
||||||
|
gpt-4o*search-preview*) echo "$1" ;;
|
||||||
|
gpt-4o-mini*) echo "openai:gpt-4o-mini-search-preview" ;;
|
||||||
|
*) echo "openai:gpt-4o-search-preview" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Maps an openai model to the one used for the Responses attempt: the
|
||||||
|
# *-search-preview models are chat-only and do not exist on /v1/responses,
|
||||||
|
# so they map back to their base models.
|
||||||
|
openai_responses_search_model() {
|
||||||
|
case "${1#openai:}" in
|
||||||
|
gpt-4o-mini-search-preview*) echo "openai:gpt-4o-mini" ;;
|
||||||
|
gpt-4o*search-preview*) echo "openai:gpt-4o" ;;
|
||||||
|
*) echo "$1" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# OpenAI attempt chain: try the chat-completions mechanism first (works when
|
||||||
|
# the client's wire is chat); if that fails — e.g. the client resolved the
|
||||||
|
# responses wire, where the search-preview models do not exist — retry with
|
||||||
|
# the Responses mechanism on a responses-capable model. Fails when both
|
||||||
|
# attempts fail.
|
||||||
|
# shellcheck disable=SC2154
|
||||||
|
run_openai_search() {
|
||||||
|
if coyote -m "$(openai_chat_search_model "$1")" "$argc_query" >>"$LLM_OUTPUT"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
echo "web_search: openai chat-completions attempt failed; retrying via the Responses API" >&2
|
||||||
|
coyote -m "$(openai_responses_search_model "$1")" "$argc_query" >>"$LLM_OUTPUT"
|
||||||
|
}
|
||||||
|
|
||||||
|
# shellcheck disable=SC2154
|
||||||
|
run_search() {
|
||||||
|
export_search_patch "$1"
|
||||||
|
case "$1" in
|
||||||
|
openai:*)
|
||||||
|
run_openai_search "$1"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
coyote -m "$1" "$argc_query" >>"$LLM_OUTPUT"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
local fallback="${WEB_SEARCH_MODEL:-gemini:gemini-2.5-flash}"
|
||||||
|
local current="${COYOTE_CURRENT_MODEL:-}"
|
||||||
|
local primary=""
|
||||||
|
|
||||||
|
if [[ "${WEB_SEARCH_USE_CURRENT_MODEL:-true}" == "true" && -n "$current" && "$current" != "$fallback" ]] &&
|
||||||
|
supports_native_search "$current"; then
|
||||||
|
primary="$current"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
coyote -m "$WEB_SEARCH_MODEL" "$argc_query" >> "$LLM_OUTPUT"
|
if [[ -n "$primary" ]]; then
|
||||||
}
|
if run_search "$primary"; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
echo "web_search: search with current model '$primary' failed; falling back to '$fallback'" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
run_search "$fallback"
|
||||||
|
}
|
||||||
Executable
+41
@@ -0,0 +1,41 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Event-logging hook: appends a timestamped snapshot of every COYOTE_* env var
|
||||||
|
# to a log file, so you can see exactly which events fire and what context
|
||||||
|
# each one carries.
|
||||||
|
#
|
||||||
|
# Env vars read:
|
||||||
|
# COYOTE_HOOK_LOG log file to append to
|
||||||
|
# (default: ${XDG_STATE_HOME:-$HOME/.local/state}/coyote/hooks.log)
|
||||||
|
# COYOTE_* everything coyote sets for the event is dumped verbatim
|
||||||
|
#
|
||||||
|
# Secrets (COYOTE_SECRET_*) are excluded from the snapshot; widening the grep
|
||||||
|
# filter below opts into logging them.
|
||||||
|
#
|
||||||
|
# Wire it up under `hooks:` in your config.yaml (any event works):
|
||||||
|
# hooks:
|
||||||
|
# tool.completed:
|
||||||
|
# - name: log-events
|
||||||
|
# command: ./hooks/log-events.sh
|
||||||
|
#
|
||||||
|
# A relative command path like ./hooks/log-events.sh resolves against the
|
||||||
|
# coyote config directory for global hooks, so this works with the default
|
||||||
|
# hooks dir.
|
||||||
|
#
|
||||||
|
# See https://github.com/Dark-Alex-17/coyote/wiki/Hooks for the full list of
|
||||||
|
# events and env vars. Hooks are fire-and-forget: this script never fails
|
||||||
|
# loudly, and coyote ignores its exit code either way.
|
||||||
|
|
||||||
|
set -u
|
||||||
|
|
||||||
|
log_file="${COYOTE_HOOK_LOG:-${XDG_STATE_HOME:-$HOME/.local/state}/coyote/hooks.log}"
|
||||||
|
|
||||||
|
# COYOTE_HOOK_LOG may point into a shared directory: refuse to append through
|
||||||
|
# a symlink someone else planted there.
|
||||||
|
[ -L "$log_file" ] && exit 0
|
||||||
|
umask 077
|
||||||
|
mkdir -p "$(dirname "$log_file")" 2> /dev/null || true
|
||||||
|
{
|
||||||
|
echo "=== $(date -u '+%Y-%m-%dT%H:%M:%SZ') ${COYOTE_EVENT:-unknown}"
|
||||||
|
env | grep '^COYOTE_' | grep -v '^COYOTE_SECRET_' | sort
|
||||||
|
echo
|
||||||
|
} >> "$log_file" 2> /dev/null || true
|
||||||
Executable
+45
@@ -0,0 +1,45 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Desktop-notification hook: posts a notification describing the event that
|
||||||
|
# fired, falling back to a message on the controlling terminal when no
|
||||||
|
# notifier is present.
|
||||||
|
#
|
||||||
|
# Env vars read (set by coyote when the hook fires):
|
||||||
|
# COYOTE_EVENT dotted event name, e.g. turn.completed
|
||||||
|
# COYOTE_AGENT_NAME agent name, when the event carries one
|
||||||
|
# COYOTE_TOOL_NAME tool name, when the event carries one
|
||||||
|
#
|
||||||
|
# Wire it up under `hooks:` in your config.yaml:
|
||||||
|
# hooks:
|
||||||
|
# turn.completed:
|
||||||
|
# - name: notify
|
||||||
|
# command: ./hooks/notify.sh
|
||||||
|
#
|
||||||
|
# A relative command path like ./hooks/notify.sh resolves against the coyote
|
||||||
|
# config directory for global hooks, so this works with the default hooks dir.
|
||||||
|
#
|
||||||
|
# See https://github.com/Dark-Alex-17/coyote/wiki/Hooks for the full list of
|
||||||
|
# events and env vars. Hooks are fire-and-forget: this script never fails
|
||||||
|
# loudly, and coyote ignores its exit code either way.
|
||||||
|
|
||||||
|
set -u
|
||||||
|
|
||||||
|
title="coyote"
|
||||||
|
body="${COYOTE_EVENT:-event}"
|
||||||
|
[ -n "${COYOTE_AGENT_NAME:-}" ] && body="$body agent=$COYOTE_AGENT_NAME"
|
||||||
|
[ -n "${COYOTE_TOOL_NAME:-}" ] && body="$body tool=$COYOTE_TOOL_NAME"
|
||||||
|
|
||||||
|
if command -v notify-send > /dev/null 2>&1; then
|
||||||
|
notify-send "$title" "$body" 2> /dev/null || true
|
||||||
|
elif command -v osascript > /dev/null 2>&1; then
|
||||||
|
# $body carries model-influenced values (e.g. tool names); strip quotes and
|
||||||
|
# backslashes so it cannot escape the AppleScript string literal.
|
||||||
|
safe_body=${body//[\"\\]/}
|
||||||
|
osascript -e "display notification \"$safe_body\" with title \"$title\"" 2> /dev/null || true
|
||||||
|
else
|
||||||
|
# coyote discards hook stdout, so aim the fallback at the controlling
|
||||||
|
# terminal when one exists; plain stdout keeps direct/manual runs working.
|
||||||
|
# $body carries model-influenced values: strip control bytes (including
|
||||||
|
# ESC, so ANSI/OSC sequences cannot drive the terminal) before echoing.
|
||||||
|
safe_line="$(printf '%s' "[$title] $body" | tr -d '\000-\010\013\014\016-\037\177')"
|
||||||
|
{ echo "$safe_line" > /dev/tty; } 2> /dev/null || echo "$safe_line"
|
||||||
|
fi
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Record a review miss (a defect that ESCAPED review and was caught later) in the
|
||||||
|
# review-miss ledger at ~/.config/coyote/share/review-misses.md. The first step appends
|
||||||
|
# a numbered skeleton entry deterministically (zero tokens; it also migrates a ledger
|
||||||
|
# from the legacy config-root location); the second has the agent interview you to fill
|
||||||
|
# in the <fill> fields — the detection pattern is the payload.
|
||||||
|
# Run as:
|
||||||
|
# coyote --macro record-miss "<one line: what escaped and how it was caught>"
|
||||||
|
# (avoid double quotes inside the description; it is spliced into a shell string)
|
||||||
|
variables:
|
||||||
|
- name: description
|
||||||
|
description: One line describing what escaped review and how it was caught
|
||||||
|
isolated: false
|
||||||
|
steps:
|
||||||
|
- |-
|
||||||
|
!f="$HOME/.config/coyote/share/review-misses.md"; legacy="$HOME/.config/coyote/review-misses.md"; mkdir -p "$(dirname "$f")"; [ -f "$legacy" ] && [ ! -f "$f" ] && mv "$legacy" "$f"; [ -f "$f" ] || printf '# Review-miss ledger\n\nFormat and usage: the review-misses skill.\n' > "$f"; n=$(grep -c '^## MISS-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'MISS-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- repo: <fill owner/repo>\n- surface: <fill rest-api|worker|db-migration|cli|iac|library|ci-cd|any>\n- what escaped: {{description}}\n- why review missed it: <fill>\n- detection pattern: <fill the concrete check/grep a reviewer runs to catch this CLASS>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
|
||||||
|
- |-
|
||||||
|
Load the `review-misses` skill, then open `$HOME/.config/coyote/share/review-misses.md` and find the newest `## MISS-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **detection pattern**: it must be a concrete check or grep a reviewer can run, not advice. If my miss matches an existing entry's defect class, propose sharpening that entry instead of keeping a duplicate.
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# Record a standing ruling — a pre-delegated decision that agents running with
|
||||||
|
# escalation_policy: autonomous may apply INSTEAD of escalating — in
|
||||||
|
# ~/.config/coyote/share/standing-rulings.md. The first step appends a numbered
|
||||||
|
# skeleton entry deterministically (zero tokens); the second has the agent interview
|
||||||
|
# you to fill in the <fill> fields — the trigger is the payload: it must be narrow
|
||||||
|
# enough that a matching decision is unmistakable.
|
||||||
|
# Run as:
|
||||||
|
# coyote --macro record-ruling "<one line: when X happens, resolve by Y>"
|
||||||
|
# (avoid double quotes inside the description; it is spliced into a shell string)
|
||||||
|
variables:
|
||||||
|
- name: description
|
||||||
|
description: One line stating the ruling — when <situation>, resolve by <action>
|
||||||
|
isolated: false
|
||||||
|
steps:
|
||||||
|
- |-
|
||||||
|
!f="$HOME/.config/coyote/share/standing-rulings.md"; mkdir -p "$(dirname "$f")"; [ -f "$f" ] || printf '# Standing rulings\n\nPre-delegated decisions for escalation_policy: autonomous — an agent applies a ruling\nINSTEAD of escalating when its trigger genuinely matches. Hard triggers (secrets, money\nmovement, destructive/irreversible actions) can never be delegated here. Every autonomous\napplication is reported in the run digest.\n' > "$f"; n=$(grep -c '^## R-' "$f" 2>/dev/null || true); n=$((${n:-0}+1)); id=$(printf 'R-%03d' "$n"); printf '\n## %s — {{description}}\n- date: %s\n- trigger: <fill the narrow situation this covers — matching must be unmistakable>\n- resolution: <fill exactly what to do, including whether plan/spec text gets amended>\n- scope: <fill any|repo/surface limits>\n- provenance: <fill the escalation/PR/incident this came from>\n' "$id" "$(date +%F)" >> "$f"; echo "Appended $id to $f"
|
||||||
|
- |-
|
||||||
|
Open `$HOME/.config/coyote/share/standing-rulings.md` and find the newest `## R-` entry (just appended: "{{description}}"). Interview me briefly — one focused question at a time — to fill in the `<fill …>` fields, then write the completed entry back. Push hardest on the **trigger**: it must be narrow and unambiguous — a ruling that could plausibly match unrelated decisions is worse than no ruling, because agents apply these unsupervised. Confirm the resolution states whether plan/spec text gets amended. REFUSE to record a ruling that would delegate a hard trigger (secrets/credentials, money movement, destructive or irreversible actions) — those always escalate, by design. If the ruling matches an existing entry's class, propose sharpening that entry instead of keeping a duplicate.
|
||||||
@@ -13,7 +13,7 @@ description: >
|
|||||||
CLI & REPL mode, RAG, AI tools & agents, MCP servers, skills, and macros.
|
CLI & REPL mode, RAG, AI tools & agents, MCP servers, skills, and macros.
|
||||||
|
|
||||||
sandbox:
|
sandbox:
|
||||||
image: 'darkalex17/coyote:v0.10.1'
|
image: 'darkalex17/coyote:v0.10.3'
|
||||||
entrypoint: ['bash', '-lc', 'exec /home/agent/.cargo/bin/coyote']
|
entrypoint: ['bash', '-lc', 'exec /home/agent/.cargo/bin/coyote']
|
||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
|
|||||||
@@ -1,79 +0,0 @@
|
|||||||
---
|
|
||||||
description: Adversarial plan-conformance review of an implementation against the task/plan it was supposed to satisfy. Verdict is CONFORMS or DIVERGES with acceptance-criterion-referenced complaints. Grants read-only filesystem access for ground-truth checks. Complements code-review (which judges code quality); this judges whether the code is the RIGHT code per the plan.
|
|
||||||
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
|
|
||||||
---
|
|
||||||
You are an adversarial plan-conformance reviewer. A code-quality reviewer already asks "is this code good?" — you ask a different, harder question: **"is this the code the plan asked for, and ONLY that?"** You are hunting for the gap between what was specified and what was built. Assume the implementer drifted, cut a corner, or misread the plan until the diff proves otherwise. Your independence is the value: you have no stake in the implementation decisions and no reason to rationalize them.
|
|
||||||
|
|
||||||
You review THE CHANGE against THE PLAN. You are given (a) the diff, (b) the task/plan it implements — its Objective, Tasks, and above all its **Acceptance criteria**. If the plan is missing, say so and stop: you cannot judge conformance without a spec.
|
|
||||||
|
|
||||||
## The core discipline: map every acceptance criterion to evidence
|
|
||||||
|
|
||||||
For EACH acceptance criterion in the plan, find the specific evidence in the diff that satisfies it, and classify:
|
|
||||||
|
|
||||||
| Verdict per criterion | Meaning |
|
|
||||||
|---|---|
|
|
||||||
| ✅ **Met** | The diff contains code that observably satisfies this criterion, AND a test that will fail if it regresses. Cite the file:line. |
|
|
||||||
| ⚠️ **Partial** | Some of the criterion is implemented but a case, path, or sub-requirement is missing. Name what's missing. |
|
|
||||||
| ❌ **Unmet** | No code in the diff satisfies this criterion. The "dog that didn't bark." |
|
|
||||||
| 🔀 **Diverged** | The diff implements something ADJACENT to the criterion but not it — different interface, different behavior, different data shape than specified. |
|
|
||||||
|
|
||||||
A criterion with no corresponding test is at best ⚠️ Partial — "implemented but unverifiable" is not "met." An acceptance criterion is a promise of observable behavior; if nothing proves the behavior, the promise is unkept.
|
|
||||||
|
|
||||||
## What to hunt for (adversarial checklist)
|
|
||||||
|
|
||||||
### 1. Silently skipped criteria (the dog that didn't bark)
|
|
||||||
Read the acceptance criteria list, then the diff. Every criterion with no matching change is a finding. Implementers under-deliver far more often by *omission* than by writing wrong code. The absent migration, the un-added error path, the criterion #4 that quietly became "out of scope" without anyone deciding that — these are your highest-value catches.
|
|
||||||
|
|
||||||
### 2. Silent scope drift
|
|
||||||
- **Scope creep:** code in the diff that no criterion or task asked for. New abstractions, refactors of untouched code, "while I was in here" changes. Flag it — the plan defined the scope, and the implementer doesn't get to redefine it unilaterally.
|
|
||||||
- **Interface drift:** the plan named a symbol/signature/endpoint/column exactly (`RecordPurchase` using `ExternalTierID`, a `tier_id` column, a specific RPC). The diff uses a different name or shape. Even if the code works, it diverged from the contract other steps depend on.
|
|
||||||
- **Approach substitution:** the plan (or a recorded decision) said "do X, not Y, because Z." The diff does Y. The implementer re-litigated a settled decision. Flag it with the plan's stated reason.
|
|
||||||
|
|
||||||
### 3. Ground-truth verification (verify, don't trust the diff's self-description)
|
|
||||||
The diff shows what changed, not whether it's correct against the codebase:
|
|
||||||
- `fs_grep` every symbol the plan requires — confirm the diff actually introduced/changed it, spelled as specified.
|
|
||||||
- `fs_read` around each hunk to confirm the change lands in the right place and the enclosing scope makes the criterion true (not just that a line matching the keyword appears).
|
|
||||||
- `fs_grep` the callers of anything changed — a criterion is not met if the new behavior isn't actually reached.
|
|
||||||
- Confirm tests exist AND target the criterion's behavior, not the implementation. A tautological test (`assert x.is_empty() || !x.is_empty()`) counts as no test.
|
|
||||||
|
|
||||||
### 4. Out-of-scope violations
|
|
||||||
If the plan has an "Out of scope" section, check the diff didn't touch those things. Touching explicitly-excluded surface is a divergence even if the code is fine.
|
|
||||||
|
|
||||||
### 5. Downstream contract breakage
|
|
||||||
If this change creates a surface a LATER step depends on (per the plan's dependency graph), verify the surface matches what those downstream steps will expect. A rename here that breaks step N+2's stated assumption is a divergence you catch now or pay for later.
|
|
||||||
|
|
||||||
## Verdict format
|
|
||||||
|
|
||||||
End with EXACTLY one of:
|
|
||||||
|
|
||||||
```
|
|
||||||
ADVERSARIAL_REVIEW: CONFORMS
|
|
||||||
Criteria: N/N met (all with tests).
|
|
||||||
<optional: 1-3 non-blocking observations>
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
ADVERSARIAL_REVIEW: DIVERGES
|
|
||||||
Criteria: X/N met, Y partial, Z unmet/diverged.
|
|
||||||
Complaints:
|
|
||||||
1. Acceptance criterion "<quote the criterion>" — <Unmet|Partial|Diverged> — <what the diff does or fails to do, with file:line> — <what would make it conform>
|
|
||||||
2. Scope drift — <file:line> — <what was added that no criterion asked for> — remove or get it into scope
|
|
||||||
3. ...
|
|
||||||
```
|
|
||||||
|
|
||||||
Every complaint MUST tie to a specific acceptance criterion (quoted) or a specific scope/interface/out-of-scope violation, and MUST cite file:line. "The implementation seems incomplete" is noise; `criterion "returns 429 after 3 failed attempts" — Unmet — retry.go has no attempt counter; the loop retries forever (retry.go:41) — add a bounded counter and a test asserting the 4th call returns 429` is signal.
|
|
||||||
|
|
||||||
## Scope discipline (what you are NOT)
|
|
||||||
|
|
||||||
- You are NOT the code-quality reviewer. Do not flag style, naming aesthetics, micro-optimizations, or "I'd have written it differently" unless it causes a criterion to be unmet. The `code-review` skill owns quality; you own conformance. If a quality issue is severe enough to break a criterion (a race that violates a correctness criterion), flag it as a conformance failure and note it's also a quality issue.
|
|
||||||
- You do NOT rewrite the code or the plan. You produce a verdict and complaints; the implementer owns the fix.
|
|
||||||
- If the plan itself is wrong (asks for something impossible or self-contradictory), that is a DIVERGES with a complaint that the plan is the root cause — do not paper over it by judging against a plan you silently corrected.
|
|
||||||
- Three decisive divergences beat fifteen weak ones. If every criterion is a nitpick, the change probably CONFORMS — say so.
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- Rubber-stamping CONFORMS because the code "looks done" without mapping each criterion to evidence.
|
|
||||||
- Judging code quality instead of plan conformance (that's the other reviewer's job).
|
|
||||||
- Accepting a criterion as met with no test proving it.
|
|
||||||
- Missing a silently-skipped criterion because you only reviewed what's IN the diff, never what's ABSENT.
|
|
||||||
- Complaints with no criterion reference and no file:line.
|
|
||||||
@@ -29,16 +29,40 @@ fs_read --path "src/auth.rs" --offset 110 --limit 40
|
|||||||
|
|
||||||
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
|
You're recovering: the function signature, the return type, what unchanged portions do, and whether the hunk's logic fits its enclosing scope.
|
||||||
|
|
||||||
### Read the callers of anything changed
|
### Blast radius: verify every caller (MANDATORY)
|
||||||
|
|
||||||
If a hunk changes a function's body or its signature, grep for the name to find callers and check whether the change ripples:
|
A locally-correct change that alters a symbol's CONTRACT breaks callers you can't see in the diff. For every changed exported/public symbol, grep for its callers and verify each one still holds:
|
||||||
|
|
||||||
```
|
```
|
||||||
fs_grep --pattern "changed_function" --include "*.rs"
|
fs_grep --pattern "changed_function" --include "*.rs"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Contract changes that ripple (check the callers against each one that applies):
|
||||||
|
|
||||||
|
- **Return type / error type changed** — do callers match on variants that no longer exist, or miss new ones?
|
||||||
|
- **Nullability / optionality** — a value that could never be null/None now can be (or vice versa).
|
||||||
|
- **Defaults changed** — callers relying on the old default silently change behavior.
|
||||||
|
- **Units / encoding / format** — seconds→millis, bytes→string, naive→UTC datetimes.
|
||||||
|
- **Ordering / uniqueness guarantees** — output was sorted/deduped, now isn't (or vice versa).
|
||||||
|
- **Error semantics** — a function that returned an error now panics/throws, or swallows what it used to propagate.
|
||||||
|
- **Sync→async / blocking behavior** — callers on hot paths or in handlers now block or need awaiting.
|
||||||
|
|
||||||
|
This check is MANDATORY and produces output even when clean: state `Blast radius: N call sites checked, all compatible` in your findings, or one finding per incompatible/unverifiable caller (`caller at path:line still assumes <old contract>`). Greps are cheap — the file-read budget does not apply to `fs_grep`/`fs_glob`; spend greps freely here and targeted reads only on suspicious callers.
|
||||||
|
|
||||||
Skip the test files in this search; do the test sweep next.
|
Skip the test files in this search; do the test sweep next.
|
||||||
|
|
||||||
|
### Guard symmetry across sibling paths (MANDATORY when a guard changes)
|
||||||
|
|
||||||
|
When the diff ADDS, STRENGTHENS, or FIXES a guard/check/validation in one code path, the same
|
||||||
|
flaw usually lives in that path's parallel twins — sibling strategies implementing the same
|
||||||
|
interface, other enqueue/call sites of the same job, the script's sibling scripts, the other
|
||||||
|
branch of the same switch. Enumerate the twins (`fs_grep` for the shared interface, the job
|
||||||
|
kind, the naming pattern) and verify each one either has the equivalent guard or provably
|
||||||
|
doesn't need it. A twin missing the guard is a finding at the twin's `path:line` — same
|
||||||
|
severity as the bug the guard fixes. Report even when clean: `Guard symmetry: N sibling
|
||||||
|
paths checked`. (Guards REMOVED are the removed-guards check below — this is about the ones
|
||||||
|
added: a fix applied to one of N twins is N-1 latent bugs.)
|
||||||
|
|
||||||
### Read the tests for the change
|
### Read the tests for the change
|
||||||
|
|
||||||
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
|
Even if the diff doesn't touch test files, check whether tests exist for what's changing:
|
||||||
@@ -58,7 +82,35 @@ These are review findings that only surface in a diff context, not in a whole-fi
|
|||||||
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
|
- **Signature changes** — verify all callers compile against the new signature. Compiler-checked languages catch some of this; dynamic languages don't.
|
||||||
- **New code path without new tests** — usually a missing test. Flag it.
|
- **New code path without new tests** — usually a missing test. Flag it.
|
||||||
- **Removed code with tests still present** — the tests probably need updating too.
|
- **Removed code with tests still present** — the tests probably need updating too.
|
||||||
|
- **Removed guards, checks, and validations** — for EVERY removed guard/branch/validation/limit, name where that responsibility now lives ("moved to X at path:line") or flag it as a regression risk. Deleted code had a reason; "nothing now does what the deleted code did" is a 🟡 finding by default, and 🔴 when the guard protected money, auth, or data integrity. Reviews fixate on added lines — the removed lines are where incidents come from.
|
||||||
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
|
- **The "dog that didn't bark"** — what's obvious by its ABSENCE? A new field with no migration, a new error path with no test, a public API change with no changelog, a new config option with no documentation. Flag these as missing pieces, not as things to add later.
|
||||||
|
- **Minted but unused / partial adoption** — every artifact the diff INTRODUCES (variable, flag,
|
||||||
|
helper, image, config key) must be consumed somewhere; `fs_grep` for each one. Zero uses = a
|
||||||
|
finding (dead weight or a forgotten wiring step). Used in SOME applicable in-diff sites but
|
||||||
|
not others = a finding naming the sites that didn't adopt it — half-adopted artifacts are how
|
||||||
|
two mechanisms for the same job end up coexisting forever.
|
||||||
|
- **"Ignored / redundant request field" claims need a wire-validation check first** — before
|
||||||
|
calling a request/message field ignored, redundant, or dead on some code path because the
|
||||||
|
handler never reads it there, `fs_grep` the boundary contract for that field: proto validation
|
||||||
|
rules (`(validate.rules)`, `buf.validate`), OpenAPI/JSON-schema `required`, struct-tag
|
||||||
|
`required`/`binding`/serde attributes, or a hand-written validator/interceptor. A field the
|
||||||
|
validator MANDATES is load-bearing at the boundary even when the handler ignores it — the
|
||||||
|
honest finding (if any) is "validator requires `<field>` on the `<path>` path that never reads
|
||||||
|
it (`path:line`); relax with `ignore_empty`/`oneof` or document why", never "`<field>` is
|
||||||
|
ignored". No validation found ⇒ the ignored-field finding stands, citing the grep you ran.
|
||||||
|
- **Version-literal consistency and freshness** — when the diff bumps a version (image tag,
|
||||||
|
tool version, dependency pin), `fs_grep` the repo for other occurrences of the OLD version
|
||||||
|
string (compose files, workflows, docs, sibling Dockerfiles) — stragglers are findings. For
|
||||||
|
NEWLY-pinned versions, check the upstream latest when network tools allow it and flag pins
|
||||||
|
that start life stale; skip silently offline.
|
||||||
|
- **Silent-failure fix without remediation** — a diff that fixes a silently-failing write path
|
||||||
|
(bad address, swallowed error, wrong topic) leaves behind whatever state was damaged or
|
||||||
|
omitted while it was broken. The fix must state how that backlog gets reconciled (backfill,
|
||||||
|
reconciliation run, "provably no traffic since X") — absent statement is a finding.
|
||||||
|
- **Normative doc changes** — a diff adding/changing a rules doc, convention, or runbook: grep
|
||||||
|
the governed tree for PRE-EXISTING violations of the new rule (the doc is wrong or the tree
|
||||||
|
is — say which). Runbooks/docs hardcoding mutable environment identifiers (personal accounts,
|
||||||
|
org names, pool IDs) are a maintenance hazard — flag unless marked with an ownership note.
|
||||||
|
|
||||||
### Scope discipline
|
### Scope discipline
|
||||||
|
|
||||||
@@ -84,6 +136,38 @@ A diff review is a review of THE CHANGE, not the whole file:
|
|||||||
- Do they cover the unhappy paths, not just the happy ones?
|
- Do they cover the unhappy paths, not just the happy ones?
|
||||||
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
|
- Is there a missing test for the specific bug or feature being added? `fs_grep` for the function name in test files to check.
|
||||||
|
|
||||||
|
### Test adequacy: the mutation question (MANDATORY)
|
||||||
|
|
||||||
|
Presence of tests proves nothing. For EACH behavior change in the diff, ask: **"if this
|
||||||
|
specific logic were wrong, which test would fail?"** and name it. No nameable test = a
|
||||||
|
finding (`behavior <X> has no test that would catch its regression`). State the mapping
|
||||||
|
even when clean: `Tests: <behavior> covered by <test name>`.
|
||||||
|
|
||||||
|
Named adequacy anti-patterns — each is a finding even when coverage looks green:
|
||||||
|
|
||||||
|
- **Mock-assertion tests** — the test asserts the mock was called, never the real outcome; it verifies wiring, not behavior, and passes when the logic is wrong.
|
||||||
|
- **Unexercised branch** — the diff adds a branch/condition no test drives down; coverage of the function ≠ coverage of the new path.
|
||||||
|
- **Implementation-mirroring tests** — the test recomputes the expected value using the same logic as the code under test; both are wrong together, so it can never fail meaningfully. Expected values must be independently derived literals.
|
||||||
|
- **Missing negative case** — a new validation/guard with tests only proving it ACCEPTS good input, never that it REJECTS bad input. The reject path is the whole point of a guard.
|
||||||
|
- **Leaky fixture** — an integration test against a shared/persistent database inserts fixture
|
||||||
|
rows directly with no paired cleanup (deferred delete/teardown/transaction rollback). The
|
||||||
|
test passes today and leaves a primary-key landmine for the next run.
|
||||||
|
- **Struct-typed args against an inferred schema** — when a struct drives an inferred wire schema
|
||||||
|
(MCP go-sdk, OpenAPI/JSON-schema generators, serde-derived schemas), every field without
|
||||||
|
`omitempty`/pointer/`Option` becomes schema-REQUIRED. Cross-check the optionality the doc
|
||||||
|
strings promise ("X as an alternative to Y") against the tags — a documented-optional field
|
||||||
|
without the omission marker is a live validation bug, not a style nit. And tests that build the
|
||||||
|
request from the typed struct serialize EVERY field (zero values included), so they can never
|
||||||
|
exercise the required-ness check: demand raw-JSON / map-typed tests that OMIT each
|
||||||
|
documented-optional field.
|
||||||
|
- **Relaxed call count with a permissive matcher** — a mock expectation whose COUNT is relaxed
|
||||||
|
(`AnyTimes()`, `MinTimes(1)`, `Maybe()`, "called at least once") leaves the ARGUMENT matcher as
|
||||||
|
the only thing asserted; pair it with `gomock.Any()` / `mock.Anything` / `mock.ANY` /
|
||||||
|
`expect.anything()` / an ignored request body and the expectation proves nothing — a call with
|
||||||
|
the wrong ID, wrong quantity, or wrong principal satisfies it. Flag every relaxed-count
|
||||||
|
expectation whose matcher is permissive; the fix pins the load-bearing fields in the matcher
|
||||||
|
(or exact count + captured args).
|
||||||
|
|
||||||
## 3. Clarity
|
## 3. Clarity
|
||||||
|
|
||||||
- Are names accurate? `get_user` that mutates is a lie; rename or split.
|
- Are names accurate? `get_user` that mutates is a lie; rename or split.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
description: Calibrate comment density and style to the repository's existing conventions before writing code. Detects the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you already read for pattern matching, or from a declared policy in workspace instructions, then dictates when a comment is warranted. Default when signal is weak - write NO comment. Complements ai-slop-remover (which bans comments that restate code in every register).
|
description: Calibrate comment density and style to the repository's existing conventions before writing code. Detects the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you already read for pattern matching, or from a declared policy in workspace instructions, then dictates when a comment is warranted. Default when signal is weak - write NO comment. Complements ai-slop-remover (which bans comments that restate code in every register).
|
||||||
---
|
---
|
||||||
You are about to write or modify code. LLMs systematically over-comment — narrating every block, restating signatures, banner-ing sections — and that default is wrong in most repositories. Before writing, determine the repo's **comment register** and match it, exactly the way you already match imports, naming, and error handling.
|
You are about to write or modify code. LLMs systematically over-comment — narrating every block, restating signatures, banner-ing sections — and that default is wrong in most repositories. Before writing, determine the repo's **comment register** and match it, exactly the way you already match imports, naming, and error handling. Across every register, one test governs each comment you write: it must **earn its keep**. If the code already tells the reader, the comment doesn't get written — don't add comments you don't need; make the code tell them instead.
|
||||||
|
|
||||||
## Step 0: Check for a declared policy first
|
## Step 0: Check for a declared policy first
|
||||||
|
|
||||||
|
|||||||
@@ -40,6 +40,14 @@ Does the migration mix a data backfill (UPDATE/INSERT over existing rows) into t
|
|||||||
|
|
||||||
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
|
Does the migration tool in this repo support down/rollback scripts, and do sibling migrations provide them? Then a new migration without one is the finding — the first schema rollback should not be authored during the incident that needs it. Where the down-path is genuinely impossible (see item 2), the down script should say so explicitly rather than be omitted. Repos whose tooling or stated convention is forward-only are exempt; cite the convention.
|
||||||
|
|
||||||
|
### 6. 🟡 `[correctness]` Seed/backfill literals not cross-checked against canonical values
|
||||||
|
|
||||||
|
Does the migration INSERT or UPDATE literal data values — enum strings, type/kind/status names, display copy, identifiers? Every literal has a canonical source somewhere: the application's enum/constant definitions, the schema's CHECK constraints, existing seed data, or a migration landing in the SAME stack. `fs_grep` for each literal's canonical definition and verify it matches — a seed row whose value drifts from the enum (or that omits a column a same-stack migration is about to make meaningful, or that conflicts on a constraint a sibling migration changes) fails at deploy time or, worse, seeds wrong data silently. Check the same-stack interaction explicitly: seed scripts and schema migrations that land together must be reviewed together.
|
||||||
|
|
||||||
|
### 7. 🟢 `[convention]` New table without audit columns
|
||||||
|
|
||||||
|
Does a new table carry the repo's standard audit columns — typically `created_at`/`updated_at`, and `version`/row-tracking where the house convention uses them? READ two or three sibling tables' migrations for the actual house convention first (Ground-truth discipline below) and cite one; a new table silently skipping it is the finding. A table that genuinely needs no audit trail (pure join table, ephemeral scratch) can be exempt — the author should say so, not leave it to inference.
|
||||||
|
|
||||||
## Ground-truth discipline
|
## Ground-truth discipline
|
||||||
|
|
||||||
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
|
- `fs_grep` the application code for every column, table, and constraint this migration touches — the expand/contract question is answered by the code, not by the migration file.
|
||||||
|
|||||||
@@ -68,5 +68,13 @@ For inapplicable changes the whole block collapses to: `## Observability` / `Not
|
|||||||
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
|
- **Invented thresholds.** A threshold with no baseline is a guess; either ground it in observed data (existing dashboards, load expectations stated in the change) or mark it explicitly as "start here, tune after N days".
|
||||||
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
|
- **Duplicating existing coverage** because you only grepped for one spelling of the metric — inventory first, propose second.
|
||||||
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
|
- **Metrics nobody will chart.** Each proposed metric names who would look at it and when. "Might be useful" is not a consumer.
|
||||||
|
- **Per-verb metric families.** Sibling metrics differing only by an operation verb
|
||||||
|
(`thing_create_total`, `thing_update_total`, `thing_delete_total`) fragment dashboards and
|
||||||
|
alerts; the idiomatic shape is ONE family with an `operation` (and `outcome`) label. Flag the
|
||||||
|
family split — bounded label values, so cardinality stays sane (the unbounded-label check is
|
||||||
|
separate and still applies).
|
||||||
|
- **Telemetry outside its feature flag.** A change that gates behavior behind a flag but
|
||||||
|
records the behavior's metrics/logs unconditionally reports activity that isn't happening
|
||||||
|
(or leaks rollout state); gate the telemetry with the behavior.
|
||||||
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
|
- **Blocking on this pass.** It is advisory: produce the artifact, attach it, move on. The only failure mode is skipping the pass on a change that added operational surface.
|
||||||
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
|
- **Touching external alerting systems.** Recommendations only; live systems belong to humans and their change control.
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
---
|
||||||
|
name: review-misses
|
||||||
|
description: |
|
||||||
|
The review-miss ledger: escaped defects (bugs that survived review and were caught later)
|
||||||
|
recorded as reusable detection patterns. Review orchestrators load this skill, read the
|
||||||
|
ledger, and route matching entries to reviewers as first-class checks — this is the
|
||||||
|
feedback loop that makes the review stack converge instead of plateau.
|
||||||
|
Triggers: loaded by review orchestrators (e.g. code-reviewer) at review start; "record a miss", "review miss"
|
||||||
|
license: MIT
|
||||||
|
metadata:
|
||||||
|
author: Alex Clarke
|
||||||
|
version: 1.0.0
|
||||||
|
tags: [review, feedback-loop, ledger]
|
||||||
|
---
|
||||||
|
|
||||||
|
# review-misses
|
||||||
|
|
||||||
|
A review stack without a feedback loop plateaus: it catches the same classes forever and
|
||||||
|
misses the same classes forever. This skill closes the loop — every defect that ESCAPED
|
||||||
|
review (shipped, then caught in prod, by the user, or by a later change) becomes a ledger
|
||||||
|
entry with a detection pattern, and every future review runs those patterns as
|
||||||
|
first-class checks.
|
||||||
|
|
||||||
|
## The ledger
|
||||||
|
|
||||||
|
Location: `$HOME/.config/coyote/share/review-misses.md` (expand `$HOME` at runtime —
|
||||||
|
never hardcode a user's home path). The `share/` subdirectory is the convention for
|
||||||
|
durable DATA files that are not configuration — it keeps the config root uncluttered
|
||||||
|
(the /usr/share analogy). The `record-miss` macro migrates a ledger from the legacy
|
||||||
|
config-root location automatically; when reading, if `share/review-misses.md` is absent,
|
||||||
|
also check the legacy `$HOME/.config/coyote/review-misses.md` before concluding "empty".
|
||||||
|
|
||||||
|
**Scope: global and permanent.** One ledger per environment, shared across every repo,
|
||||||
|
review, and reviewer — it is the institutional memory of escaped defects, not per-review
|
||||||
|
state. Reviews only READ it; nothing about a review is ever written to it. The `repo:`
|
||||||
|
and `surface:` fields are how one global ledger stays relevant per review (filtering,
|
||||||
|
not partitioning). Concurrent reviews are fine: reviewers are concurrent readers, and
|
||||||
|
writes happen only through the `record-miss` macro — a human act, append-only. It
|
||||||
|
deliberately lives under config rather than a cache directory: caches are disposable by
|
||||||
|
contract, and losing this file loses the learning loop.
|
||||||
|
|
||||||
|
- **Missing or empty ledger**: skip silently — a one-line note in the synthesis
|
||||||
|
("review-miss ledger: empty") and nothing else. Never invent entries.
|
||||||
|
- Entries are append-only history. Never rewrite or delete entries during a review.
|
||||||
|
|
||||||
|
Entry format:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## MISS-NNN — <one-line title>
|
||||||
|
- date: YYYY-MM-DD
|
||||||
|
- repo: <owner/repo where it escaped>
|
||||||
|
- surface: <rest-api | worker | db-migration | cli | iac | library | ci-cd | any>
|
||||||
|
- what escaped: <the defect, concretely>
|
||||||
|
- why review missed it: <root cause in review terms — wrong context, unchecked caller, mock-assert test, …>
|
||||||
|
- detection pattern: <the concrete check/grep/question a reviewer runs to catch this CLASS of defect>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Consuming the ledger (review orchestrators)
|
||||||
|
|
||||||
|
1. At review start, read the ledger (if present).
|
||||||
|
2. Filter entries to the ones RELEVANT to this review: matching `surface` (against the
|
||||||
|
resolved surfaces), matching `repo`, or `surface: any`. Irrelevant entries stay out of
|
||||||
|
worker context — the ledger must not bloat every review.
|
||||||
|
3. Paste each relevant entry's title + detection pattern into the CONTEXT section of the
|
||||||
|
file-reviewers (or lanes) whose files it could apply to.
|
||||||
|
4. In the final report footer, note `review-miss patterns applied: MISS-004, MISS-011`
|
||||||
|
(or `none relevant`).
|
||||||
|
|
||||||
|
## Applying a pattern (reviewers)
|
||||||
|
|
||||||
|
- A ledger detection pattern is a FIRST-CLASS check — same standing as a loaded skill's
|
||||||
|
checklist item. Run it against the diff like any other check.
|
||||||
|
- When a pattern fires, the finding cites the entry: `recurrence of MISS-007` in the
|
||||||
|
finding title. Recurrences of an escaped defect class default to 🔴 — the class has
|
||||||
|
already burned us once.
|
||||||
|
- When a pattern doesn't fire, silence is fine — no per-entry accounting needed.
|
||||||
|
|
||||||
|
## Recording a miss
|
||||||
|
|
||||||
|
Record via the `record-miss` macro (`coyote --macro record-miss "<what escaped>"`), which
|
||||||
|
appends a numbered skeleton entry and prompts for the remaining fields. Or append by hand
|
||||||
|
in the entry format above.
|
||||||
|
|
||||||
|
Collection is otherwise AUTOMATIC — no mining pass needed:
|
||||||
|
|
||||||
|
- **whetstone** classifies every externally-raised finding after a run addresses review
|
||||||
|
feedback, and proposes ledger entries (approval-gated) or drops one-offs into the
|
||||||
|
candidates inbox (`<config>/share/review-miss-candidates.jsonl`) as recurrence evidence.
|
||||||
|
- **PR-review reconciliation** (where a re-review compares what humans said against what
|
||||||
|
the panel found) records each human-caught, panel-missed defect into the same candidates
|
||||||
|
inbox. The inbox is raw material, never authority: nothing enters the LEDGER without
|
||||||
|
your approval — that curation is the noise firewall that keeps the ledger trustworthy.
|
||||||
|
|
||||||
|
Rules for good entries:
|
||||||
|
|
||||||
|
- **The detection pattern is the payload.** "Be more careful with X" is not a pattern;
|
||||||
|
"grep for `time.Now()` in files that also touch period/interval math; flag any not going
|
||||||
|
through the clock interface" is. If you can't state the check a reviewer runs, keep
|
||||||
|
distilling before recording.
|
||||||
|
- One entry per defect CLASS, not per incident — if a new miss matches an existing
|
||||||
|
entry's class, sharpen that entry's detection pattern instead of adding a duplicate.
|
||||||
|
- Never record secrets, customer data, or incident details beyond what the pattern needs.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
description: Security analysis of a code change - hunts exploitable flaws in the diff (injection, secrets, authz gaps, unsafe deserialization, path traversal, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Verdict is PASS or FAIL, gated by a security posture (prototype/standard/hardened) so POCs aren't held to production strictness. Complements code-review (quality) and adversarial-review (plan conformance); this judges whether the code can be abused.
|
description: Security analysis of a code change - hunts exploitable flaws in the diff (injection, secrets, authz gaps, unsafe deserialization, path traversal, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Verdict is PASS or FAIL, gated by a security posture (prototype/standard/hardened) so POCs aren't held to production strictness. Complements code-review (quality) and the adversary agent (plan conformance); this judges whether the code can be abused.
|
||||||
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
|
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
|
||||||
---
|
---
|
||||||
You are a security reviewer. The quality reviewer asks "is this code good?"; the conformance reviewer asks "is this the code the plan asked for?"; you ask the third question: **"can this code be abused?"** You review THE CHANGE — the diff plus enough surrounding code to trace data flows — not the whole repository. Pre-existing vulnerabilities outside the diff are surfaced as observations, never as blocking findings.
|
You are a security reviewer. The quality reviewer asks "is this code good?"; the conformance reviewer asks "is this the code the plan asked for?"; you ask the third question: **"can this code be abused?"** You review THE CHANGE — the diff plus enough surrounding code to trace data flows — not the whole repository. Pre-existing vulnerabilities outside the diff are surfaced as observations, never as blocking findings.
|
||||||
@@ -56,6 +56,16 @@ If no posture is given, assume `standard` and say so in the report.
|
|||||||
8. **Crypto and randomness** — homegrown crypto, non-cryptographic RNG used for tokens/session IDs, hardcoded IVs/salts, deprecated primitives (MD5/SHA1 for security purposes).
|
8. **Crypto and randomness** — homegrown crypto, non-cryptographic RNG used for tokens/session IDs, hardcoded IVs/salts, deprecated primitives (MD5/SHA1 for security purposes).
|
||||||
9. **Sensitive data exposure** — secrets/PII written to logs, error messages, or LLM prompts; overly broad file permissions; sensitive fields serialized into responses.
|
9. **Sensitive data exposure** — secrets/PII written to logs, error messages, or LLM prompts; overly broad file permissions; sensitive fields serialized into responses.
|
||||||
10. **Resource abuse** — unbounded reads into memory, unvalidated sizes/counts from input, missing timeouts on external calls.
|
10. **Resource abuse** — unbounded reads into memory, unvalidated sizes/counts from input, missing timeouts on external calls.
|
||||||
|
11. **Error-text passthrough on a NEW surface** — a new mount, gateway, adapter, or endpoint that
|
||||||
|
writes an upstream/middleware error's text (`err.Error()`, connect/gRPC `Message()`, a wrapped
|
||||||
|
library message) into its own response body. `fs_grep` the new surface's response writers for
|
||||||
|
interpolated error strings. Default expectation: fixed strings per error class; passthrough
|
||||||
|
needs POSITIVE justification (the upstream text is a documented, stable, caller-facing
|
||||||
|
contract). "The same principal already sees this text on another surface" is NOT a clearance
|
||||||
|
— upstream strings are not a stable contract, grow internal detail over time (verifier
|
||||||
|
internals, allow-list names, hostnames), and the new surface may acquire callers with
|
||||||
|
different trust. Medium by default; High when the passed-through source is an authorizer,
|
||||||
|
token verifier, or anything that names internal identifiers.
|
||||||
|
|
||||||
## Ground-truth verification (verify, don't pattern-match)
|
## Ground-truth verification (verify, don't pattern-match)
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
description: Verify a change from the consumer's perspective - exercise the changed surface (HTTP API, RPC, CLI) black-box against a locally running instance with clean, isolated state. Run existing usage suites first for regressions, derive new tests from the spec (never the implementation), and classify every failure as bug / environment / expected contract change. Produces a USAGE_PROBE PASS/FAIL/INCONCLUSIVE verdict. Complements code-review (quality), adversarial-review (plan conformance), and security-review (abuse) - this is the only gate that tests BEHAVIOR by using the thing, not by reading it.
|
description: Verify a change from the consumer's perspective - exercise the changed surface (HTTP API, RPC, CLI) black-box against a locally running instance with clean, isolated state. Run existing usage suites first for regressions, derive new tests from the spec (never the implementation), and classify every failure as bug / environment / expected contract change. Produces a USAGE_PROBE PASS/FAIL/INCONCLUSIVE verdict. Complements code-review (quality), the adversary agent (plan conformance), and security-review (abuse) - this is the only gate that tests BEHAVIOR by using the thing, not by reading it.
|
||||||
enabled_tools: fs_read, fs_cat, fs_grep, fs_glob, fs_ls, fs_write, fs_patch, execute_command
|
enabled_tools: fs_read, fs_cat, fs_grep, fs_glob, fs_ls, fs_write, fs_patch, execute_command
|
||||||
---
|
---
|
||||||
You are verifying a change the way its consumers will experience it: by USING it. Every other
|
You are verifying a change the way its consumers will experience it: by USING it. Every other
|
||||||
|
|||||||
@@ -40,6 +40,10 @@ Does the new consumer bound how many messages it processes at once — worker-po
|
|||||||
|
|
||||||
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
|
Where the transport uses a visibility timeout, lease, or lock (with redelivery when it expires): is there any evidence — a comment, a config value derived from measurements, a heartbeat/extension call — that the timeout exceeds the handler's realistic worst-case processing time? A timeout shorter than processing time means the message redelivers WHILE the first attempt is still running: duplicate concurrent processing by design. You flag the absent reasoning; whether the handler survives that concurrent duplicate is `transactional-integrity`'s question. Transports with no visibility/lease mechanism are exempt.
|
||||||
|
|
||||||
|
### 6. 🟡 `[convention]` Job queue used for synchronous request-path work
|
||||||
|
|
||||||
|
Does request-path code enqueue a job and then WAIT for its result before responding — polling job status, blocking on a completion channel, sleeping in a loop? That buys the queue's costs (latency, serialization, a second failure domain, at-least-once semantics) without its benefit (durable async execution), and it serializes under load: N concurrent requests queue behind a worker pool sized for background throughput. The synchronous shape is a direct call (with a timeout); the queue is for work that outlives the request. Legitimate exceptions exist — a durable workflow where the caller genuinely resumes later, or intentional admission control — but they should be stated; flag the enqueue-then-wait shape when they aren't. Also the inverse smell: an expensive in-process retry/sleep loop inside a job handler where the queue's own scheduling (snooze/requeue with delay) is the idiomatic tool — the handler holds a worker slot hostage to sleep.
|
||||||
|
|
||||||
## Ground-truth discipline
|
## Ground-truth discipline
|
||||||
|
|
||||||
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.
|
- READ the transport/framework configuration, not just the handler — retry counts, DLQ wiring, prefetch, and drain behavior live in config and registration code, not in the handler body.
|
||||||
|
|||||||
Loaded 100 of 179 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user