docs: sync agent READMEs + diagrams with resilience hardening
This commit is contained in:
8 files changed
+216
-30
No files matched your search
@@ -49,6 +49,12 @@ Complaints:
|
|||||||
2. ...
|
2. ...
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Every verdict — CONFORMS or DIVERGES — ends with a `Verification runs:` section: the recorded
|
||||||
|
PASS/FAIL results (command, exit code, duration, failure tail) of the caller-declared verification
|
||||||
|
commands, or an explicit marker when none were declared or the run never reached that stage. The
|
||||||
|
one exception is the verdict script's own crash guard, which emits a bare DIVERGES with a
|
||||||
|
"verdict computation error" line — and no section — when verdict computation itself dies.
|
||||||
|
|
||||||
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
|
A `DIVERGES` verdict **blocks** completion. The caller (sisyphus/architect) must reconcile it —
|
||||||
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
|
resume the SAME coder/sisyphus session with the complaints pasted verbatim — or escalate. It mirrors
|
||||||
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
|
the `oracle` + `plan-review` gate used before implementation, but applied *after* implementation.
|
||||||
@@ -60,21 +66,29 @@ violation) and cites `file:line`. Vague complaints are not emitted.
|
|||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
A["parse (llm): extract acceptance criteria"] --> B["diff_facts (script): resolve + bound the diff"]
|
A["parse (llm): extract acceptance criteria + declared verification commands"] --> B["diff_facts (script): resolve + bound the diff"]
|
||||||
B --> C["map over criteria: one fresh-context branch each (parallel)"]
|
B --> R["run_checks (script): run the declared verification commands ONCE, record results"]
|
||||||
|
R --> C["map over criteria: one fresh-context branch each (parallel)"]
|
||||||
C --> D["check_criterion (llm + fs/ast tools)"]
|
C --> D["check_criterion (llm + fs/ast tools)"]
|
||||||
D --> E["crit_gate (script): MET needs change+test evidence"]
|
D --> E["crit_gate (script): MET needs change+test evidence"]
|
||||||
E -. "reject-retry once; exhausted = PARTIAL" .-> D
|
E -. "reject-retry once; exhausted = PARTIAL" .-> D
|
||||||
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
|
E --> F["holistic (llm): ADDITIVE-only hunt - absence, scope drift, substitution, gamed tests"]
|
||||||
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras; no criteria = DIVERGES"]
|
F --> G["verdict (script): CONFORMS iff ALL MET + zero extras + zero faults; no criteria = DIVERGES"]
|
||||||
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
|
G --> H(["ADVERSARIAL_REVIEW: CONFORMS | DIVERGES"])
|
||||||
|
A -. "fallback" .-> P["pipeline_fault (script): record PIPELINE-FAULT marker"]
|
||||||
|
B -. "fallback" .-> P
|
||||||
|
R -. "fallback" .-> P
|
||||||
|
F -. "fallback" .-> G
|
||||||
|
P --> G
|
||||||
```
|
```
|
||||||
|
|
||||||
The conformance doctrine is baked into the graph's own nodes:
|
The conformance doctrine is baked into the graph's own nodes:
|
||||||
|
|
||||||
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
|
1. **Every** acceptance criterion gets its own fresh-context verification branch — MET / PARTIAL / UNMET / DIVERGED, machine-gated (MET requires the satisfying change AND the proving test, cited). No test ⇒ at best PARTIAL. Partial coverage of the criteria list is structurally impossible.
|
||||||
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
2. Ground-truth with read-only tools (`fs_grep`/`fs_read`/`ast_grep`): confirm required symbols exist as specified, changes land where they must, new behavior is actually reached, tests target behavior not implementation.
|
||||||
3. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
3. Execution evidence is recorded **once, deterministically**: `run_checks` runs ONLY the caller-declared verification commands (never invented or auto-detected), sequentially under a 3300s total deadline — commands the deadline leaves no budget for are recorded as SKIPPED. Criterion branches have no execution tools by design: "tests pass"-style criteria are judged against the record — a recorded green run is MET citing it; no declared command ⇒ at best PARTIAL.
|
||||||
|
4. Hunt adversarially for the **absent**: skipped criteria, scope creep, interface/approach substitution, out-of-scope touches, downstream contract breakage.
|
||||||
|
5. **Fail closed on pipeline faults**: if parse, diff resolution, or the verification runner dies, its fallback routes to `pipeline_fault`, which records a `PIPELINE-FAULT:` marker and continues to the verdict (a failed holistic pass falls back straight to the verdict, which synthesizes the same marker). Any recorded fault forces DIVERGES with the fault as complaint #1 — degraded is never passing, and the sentinel is always emitted: the graph never dies without one.
|
||||||
|
|
||||||
It is **read-only** — it produces a verdict, never a fix.
|
It is **read-only** — it produces a verdict, never a fix.
|
||||||
|
|
||||||
@@ -93,6 +107,9 @@ Run get_diff (or --base main), or: <paste diff>
|
|||||||
|
|
||||||
## PLAN — acceptance criteria to check against
|
## PLAN — acceptance criteria to check against
|
||||||
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
|
<paste the task index.md body + the relevant PLAN-*.md section, verbatim>
|
||||||
|
|
||||||
|
## VERIFICATION (optional)
|
||||||
|
<exact build/test/lint commands to run, one per line — undeclared commands are never guessed>
|
||||||
"
|
"
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -106,6 +123,7 @@ coyote -a adversary --agent-variable project_dir /path/to/repo \
|
|||||||
### Tools
|
### Tools
|
||||||
|
|
||||||
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
|
- The graph's `diff_facts` script resolves the diff itself (staged → unstaged → `HEAD~1`, or an explicit ref/range named in the prompt) — there is no `get_diff` tool anymore.
|
||||||
|
- The `run_checks` script executes the caller-declared verification commands and records the results — criterion branches cite the record and never run anything themselves.
|
||||||
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
|
- Criterion branches carry read-only `fs_read`/`fs_cat`/`fs_grep`/`ast_grep` for ground-truth checks.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|||||||
@@ -160,6 +160,9 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
|||||||
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
|
- Architect integrates: completed task branches merge into the run branch **one at a time**, with a
|
||||||
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
|
full build + test run after every merge. Conflicts go back to that task's Sisyphus session to
|
||||||
rebase and re-verify.
|
rebase and re-verify.
|
||||||
|
- Optionally, the `agent__task_*` queue may mirror the on-disk `blocked_by` graph so completing a
|
||||||
|
task auto-dispatches its unblocked dependents. The queue is in-memory convenience only — disk task
|
||||||
|
files remain the durable source of truth.
|
||||||
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
|
- Worktrees and task branches are cleaned up after each clean merge. Phase F (single draft PR +
|
||||||
CI-check watch) is unchanged in both modes.
|
CI-check watch) is unchanged in both modes.
|
||||||
|
|
||||||
|
|||||||
@@ -14,21 +14,44 @@
|
|||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
A["parse (llm)"] --> B["facts (script): diff, quality bar, ledger, history, trigger signals"]
|
A["parse (llm)"] --> B["facts (script): diff, quality bar, ledger, history, trigger signals"]
|
||||||
|
A -. "fallback" .-> PF["parse_fault (script): PIPELINE-FAULT verdict"]
|
||||||
|
PF --> L
|
||||||
B --> C["linters (script): tflint / hadolint / actionlint / buf breaking"]
|
B --> C["linters (script): tflint / hadolint / actionlint / buf breaking"]
|
||||||
C --> D["group_domains (script) --> refine_groups (llm) --> cover_gate (script: exact cover)"]
|
C --> D1["group_propose (script): deterministic domain proposal"]
|
||||||
D --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on"]
|
D1 --> D2["refine_groups (llm)"]
|
||||||
D --> F["aux_lanes (llm): org-context + prior-art spawns"]
|
D2 --> D3["cover_gate (script: exact cover)"]
|
||||||
D --> G["downstream_sweep (script)"]
|
D2 -. "fallback" .-> D3
|
||||||
|
D3 --> E["map over domain slices (parallel): domain-reviewer leaders, teammates on, max_attempts: 2"]
|
||||||
|
D3 --> F["aux_lanes (llm): org-context + prior-art spawns"]
|
||||||
|
D3 --> G["downstream_sweep (script)"]
|
||||||
E --> H["completeness_gate (script): required sections even when clean"]
|
E --> H["completeness_gate (script): required sections even when clean"]
|
||||||
|
E -. "fallback" .-> DF["domain_fault (script): ⚠️ PIPELINE-FAULT slice banner"]
|
||||||
H -. "reject-retry once" .-> E
|
H -. "reject-retry once" .-> E
|
||||||
H --> I["synthesize (llm: dedup + prose, NO verdict authority)"]
|
H --> I["synthesize (llm: dedup + prose, NO verdict authority)"]
|
||||||
F --> I
|
F --> I
|
||||||
|
F -. "fallback" .-> AF["aux_fault (script): PIPELINE-FAULT lane-skip note"]
|
||||||
|
AF --> I
|
||||||
G --> I
|
G --> I
|
||||||
I --> J["verify: finding-verifier (agent node, unskippable)"]
|
I --> J["verify: finding-verifier (agent node, unskippable, max_attempts: 2)"]
|
||||||
J --> K["verdict (script): FALSE-drops, rigor folding, MERGE-READY / NEEDS-HUMAN arithmetic"]
|
I -. "fallback" .-> K
|
||||||
|
J --> K["verdict (script): FALSE-drops, rigor folding, fault accounting, MERGE-READY / NEEDS-HUMAN arithmetic"]
|
||||||
|
J -. "fallback" .-> K
|
||||||
K --> L["render (script): the standard report"]
|
K --> L["render (script): the standard report"]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Failure fails closed.** Every LLM/agent node retries once (`max_attempts: 2` — the
|
||||||
|
domain-reviewer lanes and the verifier included), then routes to a `fallback:` instead of killing
|
||||||
|
the graph, and every fallback that loses review coverage leaves a `PIPELINE-FAULT` marker: a dead
|
||||||
|
`parse` emits a complete fault verdict and jumps straight to render; a dead domain lane becomes a
|
||||||
|
`> ⚠️ PIPELINE-FAULT` banner where its slice report would be; a dead `refine_groups` falls
|
||||||
|
through to the cover gate, which keeps the deterministic proposal; a dead synthesis or verifier
|
||||||
|
falls through to the verdict script. The verdict script counts every fault and forces
|
||||||
|
`NEEDS-HUMAN` — a degraded run never reads `MERGE-READY`. The aux context lanes are the one
|
||||||
|
exception: they are enrichment, so their fault (`aux_fault`) is surfaced as a lane-skip note but
|
||||||
|
never blocks. Fault detection is prefix-anchored (a report that merely *quotes* a marker doesn't
|
||||||
|
trip it), and the report is always emitted: the verdict script's own crash guard emits a
|
||||||
|
`PIPELINE-FAULT` NEEDS-HUMAN, and render bypasses its "No changes to review." stub for fault
|
||||||
|
verdicts — a fault renders as a full report, never silence.
|
||||||
|
|
||||||
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
|
A CodeRabbit-style code review orchestrator that coordinates per-file reviews and synthesizes findings into a unified
|
||||||
report.
|
report.
|
||||||
|
|||||||
@@ -22,11 +22,13 @@ agent, this is the file to read alongside the
|
|||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
|
||||||
17 nodes. Solid arrows are static `next` / `routes` edges declared in
|
23 nodes. Solid arrows are static `next` / `routes` edges declared in
|
||||||
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
|
`graph.yaml`; script nodes can also route dynamically via `_next` (shown as
|
||||||
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
|
labeled branches out of the diamond). Dotted arrows show `map` fan-out — the
|
||||||
`research_each_question` node spawns one `research_one_question` branch per
|
`research_each_question` node spawns one `research_one_question` branch per
|
||||||
sub-question and joins them before continuing.
|
sub-question and joins them before continuing — and `fallback` edges
|
||||||
|
(labeled "fallback"), taken when a node fails outright (see *Fault paths*
|
||||||
|
below).
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
@@ -38,18 +40,27 @@ flowchart TD
|
|||||||
bootstrap_research --> plan
|
bootstrap_research --> plan
|
||||||
bootstrap_research --> knowledge_lookup
|
bootstrap_research --> knowledge_lookup
|
||||||
plan["plan<br/>llm + output_schema"] --> research_each_question
|
plan["plan<br/>llm + output_schema"] --> research_each_question
|
||||||
|
plan -. "fallback" .-> plan_fault{"plan_fault<br/>script"}
|
||||||
|
plan_fault --> end_fault
|
||||||
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
|
knowledge_lookup[("knowledge_lookup<br/>rag")] --> research_each_question
|
||||||
research_each_question[\research_each_question<br/>map/]
|
research_each_question[\research_each_question<br/>map/]
|
||||||
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
|
research_each_question -. "spawns × N" .-> research_one_question["research_one_question<br/>llm + web tools"]
|
||||||
|
research_one_question -. "fallback" .-> question_fault{"question_fault<br/>script"}
|
||||||
research_each_question --> combine_findings
|
research_each_question --> combine_findings
|
||||||
combine_findings{"combine_findings<br/>script"} --> vet_sources
|
combine_findings{"combine_findings<br/>script"} --> vet_sources
|
||||||
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
|
vet_sources["vet_sources<br/>llm + classify_source"] --> critique
|
||||||
|
vet_sources -. "fallback" .-> vet_fault{"vet_fault<br/>script"}
|
||||||
|
vet_fault --> critique
|
||||||
critique["critique<br/>llm"] --> reflexion_gate
|
critique["critique<br/>llm"] --> reflexion_gate
|
||||||
|
critique -. "fallback" .-> critique_fault{"critique_fault<br/>script"}
|
||||||
|
critique_fault --> reflexion_gate
|
||||||
reflexion_gate{"reflexion_gate<br/>script"}
|
reflexion_gate{"reflexion_gate<br/>script"}
|
||||||
reflexion_gate -->|"PASS"| synthesize
|
reflexion_gate -->|"PASS"| synthesize
|
||||||
reflexion_gate -->|"REVISE (budget left)"| research_each_question
|
reflexion_gate -->|"REVISE (budget left)"| research_each_question
|
||||||
reflexion_gate -->|"REVISE (budget spent)"| synthesize
|
reflexion_gate -->|"REVISE (budget spent)"| synthesize
|
||||||
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
|
synthesize[["synthesize<br/>agent → report-writer"]] --> verify_sources
|
||||||
|
synthesize -. "fallback" .-> synth_fault{"synth_fault<br/>script"}
|
||||||
|
synth_fault --> end_fault
|
||||||
verify_sources{"verify_sources<br/>script"} --> approve
|
verify_sources{"verify_sources<br/>script"} --> approve
|
||||||
approve{{"approve<br/>approval"}}
|
approve{{"approve<br/>approval"}}
|
||||||
approve -->|"accept"| end_accepted
|
approve -->|"accept"| end_accepted
|
||||||
@@ -59,13 +70,14 @@ flowchart TD
|
|||||||
|
|
||||||
end_accepted(["end_accepted<br/>report"])
|
end_accepted(["end_accepted<br/>report"])
|
||||||
end_rejected(["end_rejected"])
|
end_rejected(["end_rejected"])
|
||||||
|
end_fault(["end_fault<br/>DEEP_RESEARCH FAILED"])
|
||||||
```
|
```
|
||||||
|
|
||||||
### Node-type breakdown
|
### Node-type breakdown
|
||||||
|
|
||||||
| Type | Nodes |
|
| Type | Nodes |
|
||||||
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
|
||||||
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback` |
|
| `script` (Python) | `parse_request`, `bootstrap_research`, `combine_findings`, `reflexion_gate`, `verify_sources`, `incorporate_feedback`, plus the fault markers `plan_fault`, `question_fault`, `vet_fault`, `critique_fault`, `synth_fault` |
|
||||||
| `llm` (tools: `[]`) | `plan`, `critique` |
|
| `llm` (tools: `[]`) | `plan`, `critique` |
|
||||||
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
|
| `llm` (with tool whitelist) | `research_one_question`, `vet_sources` |
|
||||||
| `rag` | `knowledge_lookup` — local corpus retrieval |
|
| `rag` | `knowledge_lookup` — local corpus retrieval |
|
||||||
@@ -73,7 +85,7 @@ flowchart TD
|
|||||||
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
|
| `agent` | `synthesize` — spawns the `report-writer` sub-agent |
|
||||||
| `input` | `ask_topic` |
|
| `input` | `ask_topic` |
|
||||||
| `approval` | `approve` |
|
| `approval` | `approve` |
|
||||||
| `end` | `end_accepted`, `end_rejected` |
|
| `end` | `end_accepted`, `end_rejected`, `end_fault` |
|
||||||
|
|
||||||
## Parallel execution
|
## Parallel execution
|
||||||
|
|
||||||
@@ -217,6 +229,46 @@ folds that text into `research_feedback` and loops back to
|
|||||||
`settings.max_loop_iterations` (40) is the engine's infinite-loop
|
`settings.max_loop_iterations` (40) is the engine's infinite-loop
|
||||||
backstop: it caps the total visits to any single node.
|
backstop: it caps the total visits to any single node.
|
||||||
|
|
||||||
|
## Fault paths
|
||||||
|
|
||||||
|
Every stage that can die outright has a `fallback:` edge to a small
|
||||||
|
fault-marker script — the dashed edges in the diagram. A fallback fires
|
||||||
|
after a node's retries are exhausted (`research_one_question` and
|
||||||
|
`synthesize` each get two attempts via `max_attempts: 2`); the engine
|
||||||
|
writes its failure text into the node's state key before routing, and
|
||||||
|
the marker normalizes that text into a `PIPELINE-FAULT: ...` entry in
|
||||||
|
`pipeline_faults`. The five routes:
|
||||||
|
|
||||||
|
- `plan -> plan_fault -> end_fault` — fail-closed: no plan means
|
||||||
|
nothing to research.
|
||||||
|
- `research_one_question -> question_fault` — branch-local, inside the
|
||||||
|
map branch: one dead research lane becomes a fault finding instead of
|
||||||
|
sinking the whole map.
|
||||||
|
- `vet_sources -> vet_fault -> critique` — degrade-visibly: vetting is
|
||||||
|
a quality lane, not the product; `source_assessment` becomes a
|
||||||
|
neutral "unvetted" note.
|
||||||
|
- `critique -> critique_fault -> reflexion_gate` — degrade-visibly: the
|
||||||
|
rewritten critique deliberately has no `VERDICT:` line, so the gate's
|
||||||
|
malformed-critique PASS default proceeds to synthesis.
|
||||||
|
- `synthesize -> synth_fault -> end_fault` — fail-closed: no report
|
||||||
|
means no product.
|
||||||
|
|
||||||
|
On the fail-closed routes, `end_fault` renders the
|
||||||
|
`DEEP_RESEARCH FAILED — PIPELINE-FAULT: ...` sentinel. On the
|
||||||
|
degrade-visibly routes, `verify_sources` folds the accumulated faults
|
||||||
|
into a `## Pipeline notes` section shown at the approval step and
|
||||||
|
appended to the accepted report (`{{report}}{{pipeline_notes}}`;
|
||||||
|
empty when there were no faults). Degraded output is never a silent
|
||||||
|
pass — every fault is visible to the reviewer or the caller — and some
|
||||||
|
end node always renders its sentinel, so the graph degrades instead of
|
||||||
|
dying.
|
||||||
|
|
||||||
|
All 11 scripts also carry a top-level crash guard (R3): a script that
|
||||||
|
throws records a fault and fails forward where the flow allows it (a
|
||||||
|
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
|
||||||
|
`verify_sources` reports that sources were NOT checked) instead of
|
||||||
|
killing the run.
|
||||||
|
|
||||||
## Running
|
## Running
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -241,6 +293,9 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
|||||||
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
|
- `verify_sources` probes every cited URL / DOI with an HTTP HEAD
|
||||||
request and reports which are unreachable, so the human reviewer
|
request and reports which are unreachable, so the human reviewer
|
||||||
sees broken citations before approving.
|
sees broken citations before approving.
|
||||||
|
- Fault paths degrade visibly, never silently: a `PIPELINE-FAULT`
|
||||||
|
can only ever surface as a failure sentinel or a `## Pipeline
|
||||||
|
notes` section, never as a clean report.
|
||||||
|
|
||||||
## Customizing
|
## Customizing
|
||||||
|
|
||||||
@@ -270,7 +325,7 @@ coyote -a deep-research # no prompt -> triggers ask_topic
|
|||||||
|
|
||||||
```
|
```
|
||||||
assets/agents/deep-research/
|
assets/agents/deep-research/
|
||||||
graph.yaml - agent config + 17-node workflow
|
graph.yaml - agent config + 23-node workflow
|
||||||
tools.sh - classify_source custom tool
|
tools.sh - classify_source custom tool
|
||||||
README.md - this file
|
README.md - this file
|
||||||
knowledge/
|
knowledge/
|
||||||
@@ -279,9 +334,15 @@ assets/agents/deep-research/
|
|||||||
scripts/
|
scripts/
|
||||||
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
|
parse_request.py - _next: bootstrap_research, or ask_topic if no topic
|
||||||
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
|
bootstrap_research.py - fan-out source: next [plan, knowledge_lookup]
|
||||||
|
plan_fault.py - fail-closed fault marker: dead plan -> end_fault
|
||||||
|
question_fault.py - branch-local fault marker for a dead research lane
|
||||||
combine_findings.py - joins map output (question_findings) into findings
|
combine_findings.py - joins map output (question_findings) into findings
|
||||||
|
vet_fault.py - degrade-visibly fault marker: unvetted sources
|
||||||
|
critique_fault.py - degrade-visibly fault marker: skipped critique
|
||||||
reflexion_gate.py - _next: research_each_question (revise) or synthesize
|
reflexion_gate.py - _next: research_each_question (revise) or synthesize
|
||||||
verify_sources.py - HTTP HEAD on cited URLs / DOIs
|
synth_fault.py - fail-closed fault marker: no report -> end_fault
|
||||||
|
verify_sources.py - HTTP HEAD on cited URLs / DOIs; folds pipeline_faults
|
||||||
|
into the report's "## Pipeline notes" section
|
||||||
incorporate_feedback.py - _next: research_each_question, with user feedback
|
incorporate_feedback.py - _next: research_each_question, with user feedback
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -7,13 +7,23 @@ or confirmation-bias the check. A graph can't: the structure IS the discipline.
|
|||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
```
|
```mermaid
|
||||||
parse (llm) ──▶ verify_each (map, max_concurrency 4) ──▶ done (end)
|
flowchart TD
|
||||||
└─▶ verify_one (llm + fs tools) ──▶ verdict_gate (script), one CHAIN per finding
|
A["parse (llm, max_attempts 2)"] --> B["verify_each (map, max_concurrency 4): one branch per finding"]
|
||||||
|
A -. "fallback" .-> PF["parse_fault (script): verdicts = one UNVERIFIABLE PIPELINE-FAULT entry"]
|
||||||
|
PF --> D
|
||||||
|
B -. "spawns × N" .-> C["verify_one (llm + fs tools, max_attempts 2)"]
|
||||||
|
C --> G["verdict_gate (script): machine-validates the JSON contract"]
|
||||||
|
G -. "reject-retry once" .-> C
|
||||||
|
C -. "fallback" .-> VF["verify_fault (script): branch-local UNVERIFIABLE marker"]
|
||||||
|
B --> D(["done (end): FINDING_VERIFIER_RESULTS"])
|
||||||
```
|
```
|
||||||
|
|
||||||
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
|
1. **parse** — extracts every 🔴/🟡 finding from the spawn prompt into a structured list
|
||||||
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
|
(id, severity verbatim, path, lines, self-contained claim). Extraction only — no judging.
|
||||||
|
Gets two attempts (`max_attempts: 2`); if both fail, its `fallback` routes to
|
||||||
|
`parse_fault`, which rewrites `verdicts` to a single UNVERIFIABLE entry naming the fault
|
||||||
|
and jumps straight to `done`.
|
||||||
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
|
2. **verify_each** — a `map` node fans out one `verify_one` branch per finding. Skipping a
|
||||||
finding is structurally impossible; results collect in input order.
|
finding is structurally impossible; results collect in input order.
|
||||||
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
|
3. **verify_one** — reads the cited code (greps for moved code before concluding anything)
|
||||||
@@ -28,12 +38,19 @@ parse (llm) ──▶ verify_each (map, max_concurrency 4) ──▶ done (end)
|
|||||||
contract (verdict enum; evidence required for VERIFIED/FALSE; note for UNVERIFIABLE),
|
contract (verdict enum; evidence required for VERIFIED/FALSE; note for UNVERIFIABLE),
|
||||||
stamps the authoritative id FROM the finding item (the model's echo is never trusted),
|
stamps the authoritative id FROM the finding item (the model's echo is never trusted),
|
||||||
rejects back to `verify_one` exactly once with the reason, and on a second failure records
|
rejects back to `verify_one` exactly once with the reason, and on a second failure records
|
||||||
UNVERIFIABLE — a malformed branch never sinks the map.
|
UNVERIFIABLE — a malformed branch never sinks the map. If `verify_one` itself dies (both
|
||||||
Kept, marked unverified.
|
`max_attempts: 2` attempts fail), its `fallback` routes to `verify_fault`, which
|
||||||
|
normalizes the engine's failure text into a branch-local UNVERIFIABLE verdict the map
|
||||||
|
still collects.
|
||||||
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
|
4. **done** — a deterministic `end` node emits the raw verdict list plus the caller
|
||||||
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
|
contract (VERIFIED = keep + paste evidence; FALSE = drop + tally; UNVERIFIABLE = keep,
|
||||||
marked). No final LLM step that could editorialize.
|
marked). No final LLM step that could editorialize.
|
||||||
|
|
||||||
|
The dashed edges are fault paths. Both fault markers surface as `PIPELINE-FAULT:` text in
|
||||||
|
a verdict's `note`, and a fault can only ever produce UNVERIFIABLE — degraded output is
|
||||||
|
never a passing verdict. Every path ends at `done`, so the `FINDING_VERIFIER_RESULTS`
|
||||||
|
sentinel is always emitted and the caller's verify lane degrades instead of dying.
|
||||||
|
|
||||||
## What it is NOT
|
## What it is NOT
|
||||||
|
|
||||||
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
|
- **Not a second opinion.** It never re-judges severity, style, or whether a finding
|
||||||
@@ -50,3 +67,6 @@ installed, code-reviewer runs the same contract inline as a fallback.
|
|||||||
|
|
||||||
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
|
- `graph.yaml` — the agent (no tools.sh; the graph grants `fs_read`/`fs_cat`/`fs_grep`
|
||||||
per node).
|
per node).
|
||||||
|
- `scripts/verdict_gate.py` — the deterministic contract gate.
|
||||||
|
- `scripts/parse_fault.py`, `scripts/verify_fault.py` — the fault markers behind the
|
||||||
|
dashed fallback edges.
|
||||||
@@ -11,13 +11,21 @@ gets the chance to rationalize past it.
|
|||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
|
parse["parse<br/>(llm: extract request facts)"] --> signals["signals<br/>(script: deterministic diff facts)"]
|
||||||
|
parse -. "fallback" .-> pfault["parse_fault<br/>(script: PIPELINE-FAULT marker)"]
|
||||||
|
pfault --> gate
|
||||||
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
|
signals --> select["select_lanes<br/>(llm: ADDITIVE-only judgment)"]
|
||||||
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes override exactly)"]
|
select --> build["build_items<br/>(script: rules floor ∪ additions;<br/>forced lanes override exactly)"]
|
||||||
|
select -. "fallback" .-> dlanes["default_lanes<br/>(script: deterministic degraded<br/>selection — wider, never narrower)"]
|
||||||
|
dlanes --> build
|
||||||
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
|
build --> mcr["map: code-review"] & madv["map: adversary"] & msec["map: security"] & mpb["map: probe"]
|
||||||
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer"]]
|
mcr -->|"over: 0-or-1 items"| rcr[["code-reviewer<br/>(max_attempts: 2)"]]
|
||||||
madv --> radv[["adversary"]]
|
madv --> radv[["adversary<br/>(max_attempts: 2)"]]
|
||||||
msec --> rsec[["security-reviewer"]]
|
msec --> rsec[["security-reviewer<br/>(max_attempts: 2)"]]
|
||||||
mpb --> rpb[["probe"]]
|
mpb --> rpb[["probe<br/>(max_attempts: 2)"]]
|
||||||
|
rcr -. "fallback" .-> lfault["lane_fault<br/>(script: PIPELINE-FAULT lane result,<br/>collected by the map)"]
|
||||||
|
radv -. "fallback" .-> lfault
|
||||||
|
rsec -. "fallback" .-> lfault
|
||||||
|
rpb -. "fallback" .-> lfault
|
||||||
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||||
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
||||||
```
|
```
|
||||||
@@ -49,6 +57,21 @@ those rules are *structure*:
|
|||||||
with an environment note — it is never treated as PASS or FAIL.
|
with an environment note — it is never treated as PASS or FAIL.
|
||||||
- **Verdict independence is structural**: lanes run as isolated sub-agents
|
- **Verdict independence is structural**: lanes run as isolated sub-agents
|
||||||
with no `teammates:` flag — no cross-lane messaging, by construction.
|
with no `teammates:` flag — no cross-lane messaging, by construction.
|
||||||
|
- **Failure fails closed, structurally** (`fallback:` routes + `PIPELINE-FAULT`
|
||||||
|
markers): every LLM node and lane retries once (`max_attempts: 2`), then a
|
||||||
|
fallback fires instead of killing the graph — and every fallback degrades
|
||||||
|
toward BLOCKED, never toward a pass. A dead lane becomes a `PIPELINE-FAULT:`
|
||||||
|
lane result (`lane_fault.py`) that the gate reports as BLOCKED *naming the
|
||||||
|
lane* — distinct from SKIPPED (not selected). A dead `parse` records a fault
|
||||||
|
and jumps straight to the gate (`parse_fault.py`). A dead `select_lanes`
|
||||||
|
degrades to a deterministic selection (`default_lanes.py`): code-review +
|
||||||
|
adversary, plus probe on consumer surface, plus security on auth/deps/exec
|
||||||
|
signals or hardened posture, unioned with caller-forced lanes — wider, never
|
||||||
|
narrower. A crashed `build_items` records a fault so an all-SKIPPED gate
|
||||||
|
still blocks. Fault detection is prefix-anchored — a real review that merely
|
||||||
|
*quotes* "PIPELINE-FAULT:" flows to the normal sentinel rules. And the
|
||||||
|
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
||||||
|
`GAUNTLET: BLOCKED`, never silence.
|
||||||
|
|
||||||
## Sentinels parsed
|
## Sentinels parsed
|
||||||
|
|
||||||
@@ -74,6 +97,9 @@ Plan / acceptance criteria:
|
|||||||
Local-run recipe / usage suites:
|
Local-run recipe / usage suites:
|
||||||
<how to boot the service locally + where existing suites live — enables the probe lane>
|
<how to boot the service locally + where existing suites live — enables the probe lane>
|
||||||
|
|
||||||
|
Verification commands: (optional — exact build/test/lint commands, passed
|
||||||
|
verbatim to the adversary lane; omitted = none declared, never invented)
|
||||||
|
|
||||||
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)"
|
Forced lanes: (optional — e.g. 'adversary, probe' to run exactly those)"
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -89,9 +115,13 @@ final report verbatim.
|
|||||||
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||||
non-trivial work — that's why code-review is unconditionally on); the
|
non-trivial work — that's why code-review is unconditionally on); the
|
||||||
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet).
|
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet)
|
||||||
A lane that dies or times out produces no sentinel — which is a BLOCKED,
|
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
|
||||||
never a silent pass.
|
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
|
||||||
|
pass.
|
||||||
|
- Caller-declared verification commands are extracted verbatim by `parse` and
|
||||||
|
passed through to the adversary lane's prompt — never invented or
|
||||||
|
auto-detected; undeclared stays undeclared.
|
||||||
- Signals failures degrade gracefully: if git can't compute the diff, lane
|
- Signals failures degrade gracefully: if git can't compute the diff, lane
|
||||||
selection falls back to caller context + forced lanes, and the report notes
|
selection falls back to caller context + forced lanes, and the report notes
|
||||||
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
it. The auth-path regex deliberately over-fires (`auth(?!or)` matches
|
||||||
|
|||||||
@@ -57,6 +57,7 @@ Spawnable sub-agents (from `config.yaml`):
|
|||||||
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
|
- 💻 **CLI Coding**: Provides a natural language interface for writing and editing code.
|
||||||
- 🔄 **Task Management**: Tracks progress and context across complex operations.
|
- 🔄 **Task Management**: Tracks progress and context across complex operations.
|
||||||
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
|
- 🛠️ **Tool Integration**: Seamlessly uses system tools for building, testing, and file manipulation.
|
||||||
|
- 🧵 **Task-queue bursts**: Dependency-shaped fan-outs (explore → code → verify) may be encoded as `agent__task_create` chains — dependents auto-dispatch when `agent__task_complete` unblocks them; todos remain the tracking source of truth.
|
||||||
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
|
- 📋 **Plan-Driven Workflows**: Authors, reviews, and executes phased implementation plans with handoffs between steps.
|
||||||
- 🚦 **Escalation policy**: `escalation_policy=autonomous` pre-delegates decision-making — standing rulings (`~/.config/coyote/share/standing-rulings.md`, recorded via `coyote --macro record-ruling`) and evidence-backed recommended options are taken without pausing and batched into a “Decisions taken autonomously” digest; secrets, money movement, and destructive actions always escalate.
|
- 🚦 **Escalation policy**: `escalation_policy=autonomous` pre-delegates decision-making — standing rulings (`~/.config/coyote/share/standing-rulings.md`, recorded via `coyote --macro record-ruling`) and evidence-backed recommended options are taken without pausing and batched into a “Decisions taken autonomously” digest; secrets, money movement, and destructive actions always escalate.
|
||||||
|
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ flowchart TD
|
|||||||
gate_blocked -->|"yes"| orient
|
gate_blocked -->|"yes"| orient
|
||||||
gate_blocked -->|"no"| end_blocked
|
gate_blocked -->|"no"| end_blocked
|
||||||
orient["orient<br/>llm, read-only"] --> route_staleness
|
orient["orient<br/>llm, read-only"] --> route_staleness
|
||||||
|
orient -. "LLM fault" .-> note_llm_fault
|
||||||
route_staleness{"route_staleness<br/>script"}
|
route_staleness{"route_staleness<br/>script"}
|
||||||
route_staleness -->|"major deviation"| gate_deviation
|
route_staleness -->|"major deviation"| gate_deviation
|
||||||
route_staleness -->|"else"| implement
|
route_staleness -->|"else"| implement
|
||||||
@@ -35,6 +36,7 @@ flowchart TD
|
|||||||
gate_deviation -->|"abort"| end_rejected
|
gate_deviation -->|"abort"| end_rejected
|
||||||
gate_deviation -->|"other (user guidance)"| implement
|
gate_deviation -->|"other (user guidance)"| implement
|
||||||
implement[["implement<br/>agent → coder"]] --> route_coder_result
|
implement[["implement<br/>agent → coder"]] --> route_coder_result
|
||||||
|
implement -. "coder crashed<br/>(no retries)" .-> end_failure
|
||||||
route_coder_result{"route_coder_result<br/>script"}
|
route_coder_result{"route_coder_result<br/>script"}
|
||||||
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
|
route_coder_result -->|"CODER_COMPLETE"| verify_format_lint
|
||||||
route_coder_result -->|"REJECTED / FAILED"| end_failure
|
route_coder_result -->|"REJECTED / FAILED"| end_failure
|
||||||
@@ -51,14 +53,18 @@ flowchart TD
|
|||||||
fix_loop_gate -->|"budget left"| implement
|
fix_loop_gate -->|"budget left"| implement
|
||||||
fix_loop_gate -->|"budget spent"| end_failure
|
fix_loop_gate -->|"budget spent"| end_failure
|
||||||
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
|
edge_case_sweep["edge_case_sweep<br/>llm"] --> route_sweep
|
||||||
|
edge_case_sweep -. "LLM fault" .-> write_handoff
|
||||||
route_sweep{"route_sweep<br/>script"}
|
route_sweep{"route_sweep<br/>script"}
|
||||||
route_sweep -->|"5+ files or boundary"| independent_review
|
route_sweep -->|"5+ files or boundary"| independent_review
|
||||||
route_sweep -->|"else"| write_handoff
|
route_sweep -->|"else"| write_handoff
|
||||||
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
|
independent_review[["independent_review<br/>agent → code-reviewer"]] --> route_review
|
||||||
|
independent_review -. "2 attempts failed" .-> write_handoff
|
||||||
route_review{"route_review<br/>script"}
|
route_review{"route_review<br/>script"}
|
||||||
route_review -->|"🔴 critical findings"| implement
|
route_review -->|"🔴 critical findings"| implement
|
||||||
route_review -->|"else"| write_handoff
|
route_review -->|"else"| write_handoff
|
||||||
write_handoff["write_handoff<br/>llm"] --> check_handoff
|
write_handoff["write_handoff<br/>llm"] --> check_handoff
|
||||||
|
write_handoff -. "LLM fault" .-> note_llm_fault
|
||||||
|
note_llm_fault{"note_llm_fault<br/>script"} --> end_failure
|
||||||
check_handoff{"check_handoff<br/>script"}
|
check_handoff{"check_handoff<br/>script"}
|
||||||
check_handoff -->|"schema valid"| gate_user_review
|
check_handoff -->|"schema valid"| gate_user_review
|
||||||
check_handoff -->|"one retry"| write_handoff
|
check_handoff -->|"one retry"| write_handoff
|
||||||
@@ -81,8 +87,32 @@ End nodes emit sentinel outcomes for the caller:
|
|||||||
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
|
- `STEP_BLOCKED` — `depends_on` unsatisfied and the user declined to proceed.
|
||||||
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
|
- `STEP_REJECTED` — user aborted at the deviation gate, or the coder's plan
|
||||||
was rejected at its approval gate.
|
was rejected at its approval gate.
|
||||||
- `STEP_FAILED` — coder failed, the step-level fix budget was exhausted, or
|
- `STEP_FAILED` — the coder failed or crashed, the step-level fix budget was
|
||||||
the handoff failed validation twice.
|
exhausted, the handoff failed validation twice, or an orient/handoff LLM
|
||||||
|
fault was recorded (rendered as `Pipeline fault:` in the output).
|
||||||
|
|
||||||
|
## Fault handling
|
||||||
|
|
||||||
|
Node crashes degrade the run visibly instead of masquerading as success:
|
||||||
|
|
||||||
|
- `implement` has **no retries** — rerunning a coder that died mid-edit
|
||||||
|
against an already-mutated tree is unsafe. Its failure text lands in
|
||||||
|
`coder_result` and the run falls straight to `end_failure`.
|
||||||
|
- `independent_review` retries once (`max_attempts: 2`); if both attempts
|
||||||
|
fail it falls back to `write_handoff`, which flags "PIPELINE-FAULT:
|
||||||
|
independent review did not run" in the handoff rather than presenting the
|
||||||
|
failure text as findings. `route_review` carries the same guard before its
|
||||||
|
🔴 grep, so a reviewer fault never spends a fix-loop attempt.
|
||||||
|
- `edge_case_sweep` faults likewise fall through to `write_handoff` and are
|
||||||
|
flagged as "PIPELINE-FAULT: edge-case sweep did not run".
|
||||||
|
- `orient` and `write_handoff` faults route through `note_llm_fault`, which
|
||||||
|
distills the engine's failure text into a `fault_note` rendered in the
|
||||||
|
`STEP_FAILED` output — never an empty shell.
|
||||||
|
|
||||||
|
Degraded is never passing (a skipped review or sweep is recorded as NOT RUN,
|
||||||
|
never paraphrased as covered), and the sentinel is always emitted — every
|
||||||
|
path terminates at one of the four end nodes, so the caller always gets a
|
||||||
|
parseable outcome.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user