Merge pull request #28 from Dark-Alex-17/feat/PLAN-review-retry
PLAN-review-retry: self-healing review gate (fault-only lane retries) + untimed approvals
This commit is contained in:
20 files changed
+2862
-147
No files matched your search
@@ -3,7 +3,7 @@ name: CI
|
|||||||
on:
|
on:
|
||||||
pull_request:
|
pull_request:
|
||||||
branches:
|
branches:
|
||||||
- '*'
|
- '**'
|
||||||
push:
|
push:
|
||||||
branches:
|
branches:
|
||||||
- main
|
- main
|
||||||
|
|||||||
@@ -177,7 +177,7 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
|
|||||||
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
|
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
|
||||||
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
|
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
|
||||||
| [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. |
|
| [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. |
|
||||||
| [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; direct adversary/probe spawns remain the fallback. |
|
| [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; a `GAUNTLET_REVIEW_INCOMPLETE:` line is an infrastructure fault escalated to the user (retry / accept NEEDS-HUMAN / abort), never a divergence; direct adversary/probe spawns remain the fallback. |
|
||||||
|
|
||||||
## Related skills
|
## Related skills
|
||||||
|
|
||||||
|
|||||||
@@ -341,9 +341,28 @@ instructions: |
|
|||||||
prompt also carries pasted plan text from the repo under review, and whatever is declared
|
prompt also carries pasted plan text from the repo under review, and whatever is declared
|
||||||
here is executed with a shell — a trust boundary the prose must not cross.
|
here is executed with a shell — a trust boundary the prose must not cross.
|
||||||
|
|
||||||
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED apply the same fix-and-rerun rules as below
|
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED with real review findings apply the same
|
||||||
(resume the SAME Sisyphus session with the findings verbatim, then a FRESH gauntlet run).
|
fix-and-rerun rules as below (resume the SAME Sisyphus session with the findings verbatim,
|
||||||
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed.
|
then a FRESH gauntlet run). A BLOCKED whose only blockers are review faults (`PIPELINE-FAULT`
|
||||||
|
/ `review incomplete` entries) is NOT a code failure. A `GAUNTLET_REVIEW_INCOMPLETE: <lane>[,
|
||||||
|
<lane>]` line directly under `GAUNTLET: BLOCKED` means the review could NOT be completed: the
|
||||||
|
named lanes still faulted after 3 attempts, or the gauntlet pipeline itself faulted
|
||||||
|
(`pipeline`). This is an infrastructure fault, not a code finding.
|
||||||
|
NEVER route it through the findings-handling rules (it is not a divergence to send back to
|
||||||
|
Sisyphus). NEVER claim done while it stands.
|
||||||
|
Escalate to the user via `user__select` with exactly these three options:
|
||||||
|
- "Retry the review again"
|
||||||
|
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
|
||||||
|
- "Abort this task"
|
||||||
|
Retry → a FRESH gauntlet run. Accept → close the task and record the skipped review (lane
|
||||||
|
names or `pipeline`) in the task's `log.md`, the Phase F digest, and the PR body. Abort →
|
||||||
|
leave the task `in-progress`, log the abort in its `log.md`, and stop to report. Findings AND
|
||||||
|
incomplete on the same output → fix the findings first, then re-run the gauntlet (the re-run
|
||||||
|
retries the lane) — escalate only if the line persists. Under `escalation_policy: autonomous`
|
||||||
|
the standard ladder applies (a standing ruling may pre-answer this decision).
|
||||||
|
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed;
|
||||||
|
on that path an adversary/probe result that is ONLY a `PIPELINE-FAULT` (no verdict on the
|
||||||
|
code) is likewise an infrastructure fault to retry, never a divergence.
|
||||||
- **Spawn `adversary`** with the diff base and the criteria pasted in:
|
- **Spawn `adversary`** with the diff base and the criteria pasted in:
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -26,8 +26,10 @@ flowchart TD
|
|||||||
radv -. "fallback" .-> lfault
|
radv -. "fallback" .-> lfault
|
||||||
rsec -. "fallback" .-> lfault
|
rsec -. "fallback" .-> lfault
|
||||||
rpb -. "fallback" .-> lfault
|
rpb -. "fallback" .-> lfault
|
||||||
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
|
||||||
gate --> done(["GAUNTLET: PASS | BLOCKED"])
|
rgate -. "_next: retry faulted lanes" .-> build
|
||||||
|
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
|
||||||
|
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
|
||||||
```
|
```
|
||||||
|
|
||||||
## Why a graph
|
## Why a graph
|
||||||
@@ -49,12 +51,17 @@ those rules are *structure*:
|
|||||||
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
- **Conditional lanes without dynamic routing**: each lane is a `map` node
|
||||||
over a 0-or-1-item list — an empty list means the map runs zero branches
|
over a 0-or-1-item list — an empty list means the map runs zero branches
|
||||||
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
(map-over-nothing), which *is* the skip mechanism. All four maps join at
|
||||||
the verdict gate on equal-length paths.
|
the retry gate on equal-length paths; it falls through to the verdict
|
||||||
|
gate, or routes back to `build_items` for faulted lanes only.
|
||||||
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
|
||||||
sentinel is a lane FAILURE, never a pass; the critical count from the
|
sentinel is a lane FAILURE, never a pass; the critical count from the
|
||||||
code-review report's summary line (raw 🔴 count if absent) blocks
|
code-review report's summary line (raw 🔴 count if absent) blocks
|
||||||
regardless of its verdict line; a code-review lane whose own verdict line
|
regardless of its verdict line; a code-review lane whose own verdict line
|
||||||
records a pipeline fault blocks (a degraded review is never a pass);
|
records a pipeline fault blocks (a degraded review is never a pass); the
|
||||||
|
adversary's own degraded-run header (the `Criteria: …` line directly under
|
||||||
|
its sentinel recording a pipeline fault, died criterion checks, or its
|
||||||
|
verdict script's crash stub) blocks as a pipeline fault,
|
||||||
|
not as a DIVERGES finding;
|
||||||
`NEEDS-HUMAN` without 🔴 passes but
|
`NEEDS-HUMAN` without 🔴 passes but
|
||||||
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
|
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
|
||||||
with an environment note — it is never treated as PASS or FAIL.
|
with an environment note — it is never treated as PASS or FAIL.
|
||||||
@@ -77,12 +84,59 @@ those rules are *structure*:
|
|||||||
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints
|
||||||
`GAUNTLET: BLOCKED`, never silence.
|
`GAUNTLET: BLOCKED`, never silence.
|
||||||
|
|
||||||
|
## Fault-only re-runs
|
||||||
|
|
||||||
|
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
|
||||||
|
only lanes that **faulted** — a missing report (stood in for by a synthetic
|
||||||
|
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
|
||||||
|
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
|
||||||
|
report, or a nested graph's own degraded-run wording (the code-review verdict
|
||||||
|
line / the adversary's header line directly under its sentinel). A real verdict
|
||||||
|
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
|
||||||
|
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
|
||||||
|
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
|
||||||
|
|
||||||
|
- **Stash + restore**: every lane report that ran this pass is stashed in
|
||||||
|
`kept_results`; a re-run pass overwrites the untouched lanes' collected
|
||||||
|
results with `[]`, so their stashed reports are restored before the gate.
|
||||||
|
The verdict gate always sees each lane's *last* report.
|
||||||
|
- **Retry never narrows selection**: lane selection is not re-run. A re-run
|
||||||
|
pass rebuilds items only for the retried lanes, and the stashed results
|
||||||
|
stand in for the rest.
|
||||||
|
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
|
||||||
|
attempts or the budget lands in `review_incomplete`, and its fault flows to
|
||||||
|
the verdict gate as-is.
|
||||||
|
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
|
||||||
|
guards append to an existing `signals_error` rather than overwrite it, and
|
||||||
|
`build_items.py` preserves the prior `lanes_summary`.
|
||||||
|
|
||||||
|
A non-empty `GAUNTLET_REVIEW_INCOMPLETE:` line structurally implies
|
||||||
|
`GAUNTLET: BLOCKED`: the verdict gate forces every lane named in
|
||||||
|
`review_incomplete` to a BLOCKED row (`review incomplete after N attempt(s)`)
|
||||||
|
and a matching blocker, so a restored earlier report can never turn an
|
||||||
|
incomplete review into a pass — and malformed `review_incomplete` bookkeeping
|
||||||
|
is itself recorded as a `pipeline` fault, never dropped.
|
||||||
|
|
||||||
|
Whenever at least one lane (or the pipeline) could not complete, the first
|
||||||
|
line reads `GAUNTLET: BLOCKED` followed immediately by
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
|
||||||
|
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
|
||||||
|
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
|
||||||
|
builder, the retry gate, the default-lanes fallback, or the verdict gate
|
||||||
|
itself (the default-lanes fallback's crash still runs the widened lanes, so
|
||||||
|
`pipeline` appears alongside real lane rows). The report gains a
|
||||||
|
`## Review incomplete` section naming each exhausted lane with its
|
||||||
|
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
|
||||||
|
re-run (N > 1). The *named* entries are faults to re-run, never code
|
||||||
|
findings; the line can accompany real findings from the other lanes, which
|
||||||
|
still appear under `## Blockers` and must be fixed as usual.
|
||||||
|
|
||||||
## Sentinels parsed
|
## Sentinels parsed
|
||||||
|
|
||||||
| Lane | Sentinel | Blocks on |
|
| Lane | Sentinel | Blocks on |
|
||||||
|------|----------|-----------|
|
|------|----------|-----------|
|
||||||
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
|
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
|
||||||
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; missing sentinel |
|
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; own degraded-run header (pipeline fault, died criterion checks, or its verdict script's crash stub); missing sentinel |
|
||||||
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
|
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
|
||||||
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
|
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
|
||||||
|
|
||||||
@@ -115,22 +169,33 @@ coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
|
|||||||
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
--agent-variable verification_commands '["cargo test --all"]' "<prompt>"
|
||||||
```
|
```
|
||||||
|
|
||||||
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status
|
The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
|
||||||
|
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
|
||||||
table, the lane-selection reasons, blockers, attention items, and each lane's
|
table, the lane-selection reasons, blockers, attention items, and each lane's
|
||||||
full report in a collapsible section. On BLOCKED: fix the findings per your
|
full report in a collapsible section. On BLOCKED with a
|
||||||
own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with
|
`GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — this is an
|
||||||
*Human attention required*: complete the work but carry those items into your
|
infrastructure fault, never a code finding. Fix any non-fault blockers listed
|
||||||
final report verbatim.
|
under `## Blockers` first (the re-run retries the lane); if the line persists,
|
||||||
|
escalate to the user with three options — retry the review again (a **fresh**
|
||||||
|
gauntlet run), accept NEEDS-HUMAN and proceed without that lane (recorded in the
|
||||||
|
report / PR body), or abort — never route it through your findings-handling
|
||||||
|
rules. On BLOCKED without it: fix the findings per your own findings-handling
|
||||||
|
rules, then spawn a **fresh** gauntlet run. On PASS with *Human attention
|
||||||
|
required*: complete the work but carry those items into your final report
|
||||||
|
verbatim.
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
- The caller decides *whether* review is warranted at all (the gauntlet is for
|
||||||
non-trivial work — that's why code-review is unconditionally on); the
|
non-trivial work — that's why code-review is unconditionally on); the
|
||||||
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
gauntlet decides *which* of the other lanes apply and what the verdicts mean.
|
||||||
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet)
|
- Lane sub-agents get generous finite per-attempt timeouts (see the
|
||||||
and one retry (`max_attempts: 2`). A lane that still dies or times out falls
|
`timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
|
||||||
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent
|
the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
|
||||||
pass.
|
the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
|
||||||
|
is derived from that envelope in graph.yaml's comment. A lane that still
|
||||||
|
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
|
||||||
|
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
|
||||||
- The `verification_commands` variable is forwarded to the adversary lane as a
|
- The `verification_commands` variable is forwarded to the adversary lane as a
|
||||||
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
|
||||||
lands the raw value in the child's state) — never through the lane prompt
|
lands the raw value in the child's state) — never through the lane prompt
|
||||||
|
|||||||
@@ -41,13 +41,14 @@ variables:
|
|||||||
settings:
|
settings:
|
||||||
max_loop_iterations: 10
|
max_loop_iterations: 10
|
||||||
log_state_snapshots: true
|
log_state_snapshots: true
|
||||||
# max over agent lanes of (max_attempts × lane timeout) = 2 × 11400 = 22800
|
# A faulted lane can be re-run by retry_gate for up to 3 passes, and each
|
||||||
# (agent-node retries each get a fresh per-attempt budget,
|
# pass's agent node retries up to max_attempts = 2 with a fresh per-attempt
|
||||||
# src/graph/agent.rs retry_transient/bounded_attempt) + 1200 margin for
|
# budget (src/graph/agent.rs retry_transient/bounded_attempt):
|
||||||
# the script stages (signals/select/build/gate) = 24000; the parse and
|
# 3 passes × 2 attempts × 11400 (max lane timeout) = 68400, + 1200 margin
|
||||||
# select_lanes llm nodes carry no `timeout:` and are bounded by
|
# for the script stages (signals/select/build/retry gate/verdict gate)
|
||||||
# max_iterations.
|
# = 69600; the parse and select_lanes llm nodes carry no `timeout:` and
|
||||||
timeout: 24000
|
# are bounded by max_iterations.
|
||||||
|
timeout: 69600
|
||||||
|
|
||||||
initial_state:
|
initial_state:
|
||||||
diff_spec: worktree
|
diff_spec: worktree
|
||||||
@@ -84,9 +85,15 @@ initial_state:
|
|||||||
adversary_results: []
|
adversary_results: []
|
||||||
security_results: []
|
security_results: []
|
||||||
probe_results: []
|
probe_results: []
|
||||||
|
lane_attempts: {}
|
||||||
|
kept_results: {}
|
||||||
|
retry_lanes: []
|
||||||
|
review_incomplete: []
|
||||||
|
gauntlet_started_at: 0
|
||||||
lanes_summary: ''
|
lanes_summary: ''
|
||||||
gauntlet_verdict: ''
|
gauntlet_verdict: ''
|
||||||
gauntlet_report: ''
|
gauntlet_report: ''
|
||||||
|
gauntlet_incomplete_line: ''
|
||||||
|
|
||||||
start: parse
|
start: parse
|
||||||
|
|
||||||
@@ -251,7 +258,7 @@ nodes:
|
|||||||
build_items:
|
build_items:
|
||||||
id: build_items
|
id: build_items
|
||||||
type: script
|
type: script
|
||||||
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map.
|
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map. On a re-run pass (retry_lanes non-empty) it skips selection and emits items only for those lanes, each carrying its attempt number.
|
||||||
script: scripts/build_items.py
|
script: scripts/build_items.py
|
||||||
timeout: 30
|
timeout: 30
|
||||||
next: [map_code_review, map_adversary, map_security, map_probe]
|
next: [map_code_review, map_adversary, map_security, map_probe]
|
||||||
@@ -265,7 +272,7 @@ nodes:
|
|||||||
output_key: lane_out
|
output_key: lane_out
|
||||||
collect_into: code_review_results
|
collect_into: code_review_results
|
||||||
max_concurrency: 1
|
max_concurrency: 1
|
||||||
next: verdict_gate
|
next: retry_gate
|
||||||
|
|
||||||
map_adversary:
|
map_adversary:
|
||||||
id: map_adversary
|
id: map_adversary
|
||||||
@@ -276,7 +283,7 @@ nodes:
|
|||||||
output_key: lane_out
|
output_key: lane_out
|
||||||
collect_into: adversary_results
|
collect_into: adversary_results
|
||||||
max_concurrency: 1
|
max_concurrency: 1
|
||||||
next: verdict_gate
|
next: retry_gate
|
||||||
|
|
||||||
map_security:
|
map_security:
|
||||||
id: map_security
|
id: map_security
|
||||||
@@ -287,7 +294,7 @@ nodes:
|
|||||||
output_key: lane_out
|
output_key: lane_out
|
||||||
collect_into: security_results
|
collect_into: security_results
|
||||||
max_concurrency: 1
|
max_concurrency: 1
|
||||||
next: verdict_gate
|
next: retry_gate
|
||||||
|
|
||||||
map_probe:
|
map_probe:
|
||||||
id: map_probe
|
id: map_probe
|
||||||
@@ -298,7 +305,7 @@ nodes:
|
|||||||
output_key: lane_out
|
output_key: lane_out
|
||||||
collect_into: probe_results
|
collect_into: probe_results
|
||||||
max_concurrency: 1
|
max_concurrency: 1
|
||||||
next: verdict_gate
|
next: retry_gate
|
||||||
|
|
||||||
run_code_review:
|
run_code_review:
|
||||||
id: run_code_review
|
id: run_code_review
|
||||||
@@ -412,10 +419,18 @@ nodes:
|
|||||||
script: scripts/lane_fault.py
|
script: scripts/lane_fault.py
|
||||||
timeout: 30
|
timeout: 30
|
||||||
|
|
||||||
|
retry_gate:
|
||||||
|
id: retry_gate
|
||||||
|
type: script
|
||||||
|
description: Deterministic fault-only re-run gate. Stashes every lane report that ran this pass, restores reports a re-run pass overwrote with [] (a lane not being re-run maps over nothing), classifies each lane that ran as completed / faulted / skipped, counts attempts, and routes back into build_items for faulted lanes that still have attempts left inside the wall-clock budget; otherwise it records them in review_incomplete and falls through to verdict_gate. A real verdict is never re-run, and a gate crash never retries — it records a PIPELINE-FAULT and falls through.
|
||||||
|
script: scripts/retry_gate.py
|
||||||
|
timeout: 30
|
||||||
|
next: verdict_gate
|
||||||
|
|
||||||
verdict_gate:
|
verdict_gate:
|
||||||
id: verdict_gate
|
id: verdict_gate
|
||||||
type: script
|
type: script
|
||||||
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane.
|
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
|
||||||
script: scripts/verdict_gate.py
|
script: scripts/verdict_gate.py
|
||||||
timeout: 60
|
timeout: 60
|
||||||
next: done
|
next: done
|
||||||
@@ -424,6 +439,6 @@ nodes:
|
|||||||
id: done
|
id: done
|
||||||
type: end
|
type: end
|
||||||
output: |
|
output: |
|
||||||
GAUNTLET: {{gauntlet_verdict}}
|
GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
|
||||||
|
|
||||||
{{gauntlet_report}}
|
{{gauntlet_report}}
|
||||||
@@ -9,6 +9,10 @@ skip mechanism — no dynamic routing anywhere.
|
|||||||
The LLM's add_* flags can only ADD lanes. A flag that is false never
|
The LLM's add_* flags can only ADD lanes. A flag that is false never
|
||||||
removes a lane the hard rules selected.
|
removes a lane the hard rules selected.
|
||||||
|
|
||||||
|
A re-run pass (retry_gate emitted `retry_lanes`) skips selection: only the
|
||||||
|
named lanes get an item, stamped with their attempt number, and every other
|
||||||
|
lane maps over nothing — retry_gate restores their earlier reports.
|
||||||
|
|
||||||
Fail-closed: a builder crash emits all four lists empty plus a
|
Fail-closed: a builder crash emits all four lists empty plus a
|
||||||
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
|
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
|
||||||
never kills the graph.
|
never kills the graph.
|
||||||
@@ -16,6 +20,7 @@ never kills the graph.
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
|
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must
|
||||||
# canonicalize the same way on the degraded path as it does here.
|
# canonicalize the same way on the degraded path as it does here.
|
||||||
@@ -35,6 +40,18 @@ ALIASES = {
|
|||||||
"usage-pattern-testing": "probe",
|
"usage-pattern-testing": "probe",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Keep in sync with retry_gate.LANES
|
||||||
|
LANE_KEYS = {
|
||||||
|
"code-review": "code_review_items",
|
||||||
|
"adversary": "adversary_items",
|
||||||
|
"security": "security_items",
|
||||||
|
"probe": "probe_items",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Filled by main() so the crash guard can append to an upstream fault
|
||||||
|
# instead of overwriting it, and keep the first pass's lanes_summary.
|
||||||
|
crash_context = {"signals_error": "", "lanes_summary": ""}
|
||||||
|
|
||||||
|
|
||||||
def load_state():
|
def load_state():
|
||||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
@@ -45,11 +62,38 @@ def load_state():
|
|||||||
|
|
||||||
def main():
|
def main():
|
||||||
state = load_state()
|
state = load_state()
|
||||||
|
prior = state.get("signals_error")
|
||||||
|
crash_context["signals_error"] = prior if isinstance(prior, str) else ""
|
||||||
|
# retry_gate's wall-clock budget is measured from this stamp; signals.py
|
||||||
|
# is the normal stamper, this is the defensive fallback.
|
||||||
|
stamp = {} if state.get("gauntlet_started_at") else {"gauntlet_started_at": time.time()}
|
||||||
|
|
||||||
def text(key):
|
def text(key):
|
||||||
v = state.get(key)
|
v = state.get(key)
|
||||||
return v.strip() if isinstance(v, str) else ""
|
return v.strip() if isinstance(v, str) else ""
|
||||||
|
|
||||||
|
crash_context["lanes_summary"] = text("lanes_summary")
|
||||||
|
retry_lanes = [lane for lane in (state.get("retry_lanes") or []) if lane in LANE_KEYS]
|
||||||
|
if retry_lanes:
|
||||||
|
attempts = state.get("lane_attempts") or {}
|
||||||
|
retried = [(lane, int(attempts.get(lane) or 0) + 1) for lane in retry_lanes]
|
||||||
|
note = "- retried: [" + ", ".join(f"{lane} (attempt {n})" for lane, n in retried) + "]"
|
||||||
|
summary = text("lanes_summary")
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
**{key: [] for key in LANE_KEYS.values()},
|
||||||
|
**{
|
||||||
|
LANE_KEYS[lane]: [{"lane": lane, "attempt": n}] for lane, n in retried
|
||||||
|
},
|
||||||
|
"lanes_summary": f"{summary}\n{note}" if summary else note,
|
||||||
|
"retry_lanes": [],
|
||||||
|
**stamp,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
forced_raw = state.get("forced_lanes") or []
|
forced_raw = state.get("forced_lanes") or []
|
||||||
forced, unknown = [], []
|
forced, unknown = [], []
|
||||||
for lane in forced_raw:
|
for lane in forced_raw:
|
||||||
@@ -126,6 +170,7 @@ def main():
|
|||||||
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
"security_items": [{"lane": "security"}] if "security" in lanes else [],
|
||||||
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
|
||||||
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
"lanes_summary": "\n".join(f"- {r}" for r in reasons),
|
||||||
|
**stamp,
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -134,6 +179,9 @@ def main():
|
|||||||
try:
|
try:
|
||||||
main()
|
main()
|
||||||
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
|
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
|
||||||
|
fault = f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run"
|
||||||
|
if crash_context["signals_error"]:
|
||||||
|
fault = f"{crash_context['signals_error']}; {fault}"
|
||||||
print(
|
print(
|
||||||
json.dumps(
|
json.dumps(
|
||||||
{
|
{
|
||||||
@@ -141,8 +189,8 @@ except Exception as e: # noqa: BLE001 — a builder crash must never kill the g
|
|||||||
"adversary_items": [],
|
"adversary_items": [],
|
||||||
"security_items": [],
|
"security_items": [],
|
||||||
"probe_items": [],
|
"probe_items": [],
|
||||||
"lanes_summary": "",
|
"lanes_summary": crash_context["lanes_summary"],
|
||||||
"signals_error": f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run",
|
"signals_error": fault,
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -0,0 +1,206 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic fault-only re-run gate for the review gauntlet.
|
||||||
|
|
||||||
|
Sits between the four lane maps and the verdict gate. Every lane map runs
|
||||||
|
on every pass — a lane that is not being re-run maps over [] — so a re-run
|
||||||
|
pass overwrites the untouched lanes' collected results with []. The gate's
|
||||||
|
hard rules:
|
||||||
|
|
||||||
|
- every lane that ran this pass is stashed in kept_results, completed
|
||||||
|
and faulted alike, and a lane whose map ran nothing this pass gets
|
||||||
|
its stashed report restored — the verdict gate always sees each
|
||||||
|
lane's LAST report;
|
||||||
|
- a lane is FAULTED only when its report is missing (the map ran but
|
||||||
|
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
|
||||||
|
the verdict gate never reads it as SKIPPED), non-string, the
|
||||||
|
lane_fault.py marker, sentinel-less, or carries the nested graph's own
|
||||||
|
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
|
||||||
|
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
|
||||||
|
- a faulted lane is re-run only while it has attempts left AND the
|
||||||
|
gauntlet is inside its wall-clock budget; otherwise it is recorded in
|
||||||
|
review_incomplete and its fault flows to the verdict gate as-is;
|
||||||
|
- malformed review_incomplete bookkeeping (not a list, or non-str
|
||||||
|
entries) keeps its str entries and records a PIPELINE-FAULT in
|
||||||
|
signals_error — never silently dropped, never split into lane names;
|
||||||
|
- the gate itself must never retry by accident: any internal error
|
||||||
|
records a PIPELINE-FAULT in signals_error and falls through to the
|
||||||
|
verdict gate, which blocks on it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
|
||||||
|
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
|
||||||
|
# = 45600; 600 s of slack below that.
|
||||||
|
MAX_RETRY_ELAPSED_SECS = 45000
|
||||||
|
MAX_ATTEMPTS = 3
|
||||||
|
|
||||||
|
# lane name -> (items key, results key)
|
||||||
|
LANES = {
|
||||||
|
"code-review": ("code_review_items", "code_review_results"),
|
||||||
|
"adversary": ("adversary_items", "adversary_results"),
|
||||||
|
"security": ("security_items", "security_results"),
|
||||||
|
"probe": ("probe_items", "probe_results"),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Must stay identical to verdict_gate.py's sentinel regexes: a lane this
|
||||||
|
# gate calls completed must be one the verdict gate can parse, or a drift
|
||||||
|
# would re-run lanes the verdict gate accepts (or accept lanes it cannot
|
||||||
|
# read). A cross-pin test compares the literals in both files.
|
||||||
|
SENTINELS = {
|
||||||
|
"code-review": r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)",
|
||||||
|
"adversary": r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)",
|
||||||
|
"security": r"SECURITY_REVIEW:\s*(PASS|FAIL)",
|
||||||
|
"probe": r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Filled by main() so the crash guard can append to an upstream fault
|
||||||
|
# instead of overwriting it, and report the lanes it had already declined.
|
||||||
|
crash_context = {"signals_error": "", "review_incomplete": []}
|
||||||
|
|
||||||
|
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
|
||||||
|
ADVERSARY_DEGRADED_HEADER = re.compile(
|
||||||
|
r"Criteria: .*("
|
||||||
|
r"degraded run: pipeline fault recorded"
|
||||||
|
r"|degraded run: \d+ criterion check\(s\) died"
|
||||||
|
r"|verdict computation error:"
|
||||||
|
r")"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
|
with open(path) as f:
|
||||||
|
return json.load(f)
|
||||||
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def adversary_degraded(report):
|
||||||
|
# The adversary's verdict.py emits its degraded-run header on the line
|
||||||
|
# directly under the FIRST sentinel; anchoring there keeps a report that
|
||||||
|
# merely quotes the wording later (observations, complaints) a real verdict.
|
||||||
|
lines = report.splitlines()
|
||||||
|
for i, ln in enumerate(lines):
|
||||||
|
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
|
||||||
|
nxt = lines[i + 1] if i + 1 < len(lines) else ""
|
||||||
|
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def nested_degraded(lane, report):
|
||||||
|
# The nested graphs complete with a sentinel even when a stage inside
|
||||||
|
# them died; their own degraded-run wording marks a partial review,
|
||||||
|
# which is not a completed one. Same anchoring as verdict_gate.py.
|
||||||
|
if lane == "code-review":
|
||||||
|
verdict_line = re.search(r"^\*\*Verdict: .*$", report, re.MULTILINE)
|
||||||
|
return bool(
|
||||||
|
verdict_line
|
||||||
|
and re.search(
|
||||||
|
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
|
||||||
|
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
|
||||||
|
verdict_line.group(0),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if lane == "adversary":
|
||||||
|
return adversary_degraded(report)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def is_faulted(lane, report):
|
||||||
|
if not isinstance(report, str):
|
||||||
|
return True
|
||||||
|
if report.strip().startswith("PIPELINE-FAULT:"):
|
||||||
|
return True
|
||||||
|
if not re.search(SENTINELS[lane], report):
|
||||||
|
return True
|
||||||
|
return nested_degraded(lane, report)
|
||||||
|
|
||||||
|
|
||||||
|
def attempt_count(attempts, lane):
|
||||||
|
try:
|
||||||
|
return int(attempts.get(lane) or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
state = load_state()
|
||||||
|
prior = state.get("signals_error")
|
||||||
|
signals_error = prior if isinstance(prior, str) else ""
|
||||||
|
crash_context["signals_error"] = signals_error
|
||||||
|
out = {}
|
||||||
|
declared = state.get("review_incomplete")
|
||||||
|
if isinstance(declared, list):
|
||||||
|
incomplete = [name for name in declared if isinstance(name, str)]
|
||||||
|
malformed = len(incomplete) != len(declared)
|
||||||
|
else:
|
||||||
|
incomplete = []
|
||||||
|
malformed = declared is not None
|
||||||
|
if malformed:
|
||||||
|
fault = (
|
||||||
|
"PIPELINE-FAULT: malformed review_incomplete bookkeeping "
|
||||||
|
f"({repr(declared)[:80]})"
|
||||||
|
)
|
||||||
|
signals_error = f"{signals_error}; {fault}" if signals_error else fault
|
||||||
|
crash_context["signals_error"] = signals_error
|
||||||
|
out["signals_error"] = signals_error
|
||||||
|
crash_context["review_incomplete"] = incomplete
|
||||||
|
raw_attempts = state.get("lane_attempts")
|
||||||
|
attempts = dict(raw_attempts) if isinstance(raw_attempts, dict) else {}
|
||||||
|
kept = dict(state.get("kept_results") or {})
|
||||||
|
faulted = []
|
||||||
|
|
||||||
|
for lane, (items_key, results_key) in LANES.items():
|
||||||
|
results = state.get(results_key) or []
|
||||||
|
if not state.get(items_key):
|
||||||
|
if not results and lane in kept:
|
||||||
|
out[results_key] = [kept[lane]]
|
||||||
|
continue
|
||||||
|
report = results[0] if results else None
|
||||||
|
if report is None:
|
||||||
|
report = f"PIPELINE-FAULT: {lane} lane produced no report"
|
||||||
|
out[results_key] = [report]
|
||||||
|
kept[lane] = report
|
||||||
|
if is_faulted(lane, report):
|
||||||
|
faulted.append(lane)
|
||||||
|
attempts[lane] = attempt_count(attempts, lane) + 1
|
||||||
|
|
||||||
|
# A missing/zero stamp reads as an ancient start and declines every
|
||||||
|
# retry — the safe direction when the clock's origin is unknown.
|
||||||
|
started = state.get("gauntlet_started_at") or 0
|
||||||
|
within_budget = (time.time() - float(started)) <= MAX_RETRY_ELAPSED_SECS
|
||||||
|
retry = [lane for lane in faulted if within_budget and attempts[lane] < MAX_ATTEMPTS]
|
||||||
|
declined = [lane for lane in faulted if lane not in retry]
|
||||||
|
incomplete = sorted(set(incomplete) | set(declined))
|
||||||
|
crash_context["review_incomplete"] = incomplete
|
||||||
|
|
||||||
|
out.update(
|
||||||
|
{
|
||||||
|
"lane_attempts": attempts,
|
||||||
|
"kept_results": kept,
|
||||||
|
"retry_lanes": retry,
|
||||||
|
"review_incomplete": incomplete,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if retry:
|
||||||
|
out["_next"] = "build_items"
|
||||||
|
print(json.dumps(out))
|
||||||
|
|
||||||
|
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as e: # noqa: BLE001 — a gate crash must fall through, never retry
|
||||||
|
fault = f"PIPELINE-FAULT: retry gate crashed — {e}"
|
||||||
|
if crash_context["signals_error"]:
|
||||||
|
fault = f"{crash_context['signals_error']}; {fault}"
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"retry_lanes": [],
|
||||||
|
"review_incomplete": crash_context["review_incomplete"],
|
||||||
|
"signals_error": fault,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
@@ -12,6 +12,7 @@ import json
|
|||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import subprocess
|
import subprocess
|
||||||
|
import time
|
||||||
|
|
||||||
def load_state():
|
def load_state():
|
||||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
@@ -37,6 +38,10 @@ def git(*args):
|
|||||||
|
|
||||||
|
|
||||||
out = {"project_dir": proj}
|
out = {"project_dir": proj}
|
||||||
|
# retry_gate measures its wall-clock budget from this stamp; it is set
|
||||||
|
# before the git work so a failed diff still starts the clock.
|
||||||
|
if not state.get("gauntlet_started_at"):
|
||||||
|
out["gauntlet_started_at"] = time.time()
|
||||||
try:
|
try:
|
||||||
if spec in ("", "worktree"):
|
if spec in ("", "worktree"):
|
||||||
names = git("diff", "--name-only", "HEAD")
|
names = git("diff", "--name-only", "HEAD")
|
||||||
|
|||||||
@@ -8,14 +8,20 @@ re-reads, no judgment. The gate's hard rules:
|
|||||||
- a non-string lane result (the map collected an object instead of the
|
- a non-string lane result (the map collected an object instead of the
|
||||||
lane's rendered text) blocks — it cannot carry a sentinel;
|
lane's rendered text) blocks — it cannot carry a sentinel;
|
||||||
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
- a PIPELINE-FAULT lane result (the lane's agent died after retries;
|
||||||
detected only when the report IS the lane_fault.py marker, i.e. it
|
detected only when the report IS the lane_fault.py marker or
|
||||||
STARTS with it — a real review merely quoting the marker flows to the
|
retry_gate.py's no-report marker, i.e. it STARTS with it — a real
|
||||||
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse
|
review merely quoting the marker flows to the normal rules) or a
|
||||||
fault) blocks — a degraded pipeline is never a pass;
|
PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
|
||||||
|
degraded pipeline is never a pass;
|
||||||
- a code-review lane whose OWN verdict line records a pipeline fault
|
- a code-review lane whose OWN verdict line records a pipeline fault
|
||||||
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
(code-reviewer's verdict.py / render.py degraded-run wording) blocks —
|
||||||
the nested graph completed, but with a dead domain lane, verifier, or
|
the nested graph completed, but with a dead domain lane, verifier, or
|
||||||
synthesis behind it, so its critical count is not trustworthy;
|
synthesis behind it, so its critical count is not trustworthy;
|
||||||
|
- an adversary lane whose OWN header line (the line directly under its
|
||||||
|
sentinel) records a degraded run — a pipeline fault, died criterion
|
||||||
|
checks, or its verdict script's crash stub — blocks as a PIPELINE-FAULT,
|
||||||
|
not as a DIVERGES finding: the nested graph completed with a dead stage
|
||||||
|
behind it, so its verdict is not a judgment on the code;
|
||||||
- a non-zero critical count in the code-review report blocks regardless
|
- a non-zero critical count in the code-review report blocks regardless
|
||||||
of the lane's own verdict line — derived from render.py's deterministic
|
of the lane's own verdict line — derived from render.py's deterministic
|
||||||
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
|
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
|
||||||
@@ -24,14 +30,37 @@ re-reads, no judgment. The gate's hard rules:
|
|||||||
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
|
||||||
treated as PASS or FAIL (fix the environment per the plan's local-run
|
treated as PASS or FAIL (fix the environment per the plan's local-run
|
||||||
recipe, then re-run);
|
recipe, then re-run);
|
||||||
|
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
|
||||||
|
re-ran it, so the reader can tell a first-pass death from an exhausted
|
||||||
|
retry budget;
|
||||||
|
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
|
||||||
|
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
|
||||||
|
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
|
||||||
|
incomplete" section: that is a fault to re-run, never a code finding.
|
||||||
|
Every lane named in review_incomplete is forced to a BLOCKED row, so a
|
||||||
|
restored earlier report can never turn an incomplete review into a
|
||||||
|
pass: a non-empty machine line structurally implies GAUNTLET: BLOCKED.
|
||||||
|
Malformed review_incomplete bookkeeping (not a list, or non-str
|
||||||
|
entries) is itself recorded as a `pipeline` fault, never dropped;
|
||||||
- the gate itself must never crash into a silent pass: any internal
|
- the gate itself must never crash into a silent pass: any internal
|
||||||
error emits GAUNTLET: BLOCKED naming the error.
|
error emits GAUNTLET: BLOCKED naming the error, with the pipeline
|
||||||
|
marked incomplete.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
|
||||||
|
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
|
||||||
|
ADVERSARY_DEGRADED_HEADER = re.compile(
|
||||||
|
r"Criteria: .*("
|
||||||
|
r"degraded run: pipeline fault recorded"
|
||||||
|
r"|degraded run: \d+ criterion check\(s\) died"
|
||||||
|
r"|verdict computation error:"
|
||||||
|
r")"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def load_state():
|
def load_state():
|
||||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||||
with open(path) as f:
|
with open(path) as f:
|
||||||
@@ -39,10 +68,26 @@ def load_state():
|
|||||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
|
||||||
|
|
||||||
|
def adversary_degraded(report):
|
||||||
|
# The adversary's verdict.py emits its degraded-run header on the line
|
||||||
|
# directly under the FIRST sentinel; anchoring there keeps a report that
|
||||||
|
# merely quotes the wording later (observations, complaints) a real verdict.
|
||||||
|
lines = report.splitlines()
|
||||||
|
for i, ln in enumerate(lines):
|
||||||
|
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
|
||||||
|
nxt = lines[i + 1] if i + 1 < len(lines) else ""
|
||||||
|
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
state = load_state()
|
state = load_state()
|
||||||
blockers, attention, lane_lines, reports = [], [], [], []
|
blockers, attention, reports = [], [], []
|
||||||
|
rows = {}
|
||||||
malformed = set()
|
malformed = set()
|
||||||
|
lane_attempts = state.get("lane_attempts")
|
||||||
|
if not isinstance(lane_attempts, dict):
|
||||||
|
lane_attempts = {}
|
||||||
|
|
||||||
def lane_report(key):
|
def lane_report(key):
|
||||||
arr = state.get(key) or []
|
arr = state.get(key) or []
|
||||||
@@ -55,7 +100,17 @@ def main():
|
|||||||
return json.dumps(item)
|
return json.dumps(item)
|
||||||
|
|
||||||
def record(lane, status, detail=""):
|
def record(lane, status, detail=""):
|
||||||
lane_lines.append(f"| {lane} | {status} | {detail} |")
|
rows[lane] = (status, detail)
|
||||||
|
|
||||||
|
def attempt_count(lane):
|
||||||
|
try:
|
||||||
|
return int(lane_attempts.get(lane) or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def attempts_suffix(lane):
|
||||||
|
n = attempt_count(lane)
|
||||||
|
return f" ×{n}" if n > 1 else ""
|
||||||
|
|
||||||
def fault_excerpt(report):
|
def fault_excerpt(report):
|
||||||
line = next(
|
line = next(
|
||||||
@@ -64,10 +119,11 @@ def main():
|
|||||||
return line.strip()[:200]
|
return line.strip()[:200]
|
||||||
|
|
||||||
def is_lane_fault(report):
|
def is_lane_fault(report):
|
||||||
# A genuine lane fault is the ENTIRE report emitted by lane_fault.py,
|
# A genuine lane fault is the ENTIRE report — the lane_fault.py marker
|
||||||
# which always begins with the marker. Matching the marker anywhere
|
# or retry_gate.py's no-report marker — which always begins with
|
||||||
# would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass
|
# "PIPELINE-FAULT:". Matching the marker anywhere would let a real
|
||||||
# 🔴 counting and sentinel parsing.
|
# review that merely QUOTES it bypass 🔴 counting and sentinel
|
||||||
|
# parsing.
|
||||||
return report.strip().startswith("PIPELINE-FAULT:")
|
return report.strip().startswith("PIPELINE-FAULT:")
|
||||||
|
|
||||||
def lane_fault_blocker(lane, report):
|
def lane_fault_blocker(lane, report):
|
||||||
@@ -75,7 +131,7 @@ def main():
|
|||||||
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
|
||||||
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
|
||||||
)
|
)
|
||||||
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)")
|
record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
|
||||||
|
|
||||||
def malformed_blocker(lane, report):
|
def malformed_blocker(lane, report):
|
||||||
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
|
||||||
@@ -129,10 +185,11 @@ def main():
|
|||||||
blockers.append(
|
blockers.append(
|
||||||
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
|
||||||
)
|
)
|
||||||
|
suffix = attempts_suffix("code-review")
|
||||||
if m_fault and reds:
|
if m_fault and reds:
|
||||||
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴")
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
|
||||||
elif m_fault:
|
elif m_fault:
|
||||||
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)")
|
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
|
||||||
elif reds:
|
elif reds:
|
||||||
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
|
||||||
elif m and m.group(1) == "MERGE-READY":
|
elif m and m.group(1) == "MERGE-READY":
|
||||||
@@ -162,7 +219,17 @@ def main():
|
|||||||
else:
|
else:
|
||||||
reports.append(("adversary", adv))
|
reports.append(("adversary", adv))
|
||||||
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
|
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
|
||||||
if m and m.group(1) == "CONFORMS":
|
if adversary_degraded(adv):
|
||||||
|
blockers.append(
|
||||||
|
"adversary: PIPELINE-FAULT — degraded run, pipeline fault recorded "
|
||||||
|
"inside the adversary; a degraded lane is never a pass"
|
||||||
|
)
|
||||||
|
record(
|
||||||
|
"adversary",
|
||||||
|
"BLOCKED",
|
||||||
|
f"PIPELINE-FAULT (degraded lane{attempts_suffix('adversary')})",
|
||||||
|
)
|
||||||
|
elif m and m.group(1) == "CONFORMS":
|
||||||
record("adversary", "GREEN", "CONFORMS")
|
record("adversary", "GREEN", "CONFORMS")
|
||||||
elif m:
|
elif m:
|
||||||
blockers.append(
|
blockers.append(
|
||||||
@@ -228,16 +295,69 @@ def main():
|
|||||||
)
|
)
|
||||||
record("probe", "BLOCKED", "missing verdict sentinel")
|
record("probe", "BLOCKED", "missing verdict sentinel")
|
||||||
|
|
||||||
|
# retry_gate restores an exhausted lane's LAST report, which may be an
|
||||||
|
# earlier clean verdict — an incomplete lane must block regardless of
|
||||||
|
# what its row parsed to.
|
||||||
|
declared = state.get("review_incomplete")
|
||||||
|
incomplete = set()
|
||||||
|
malformed_bookkeeping = None
|
||||||
|
if isinstance(declared, list):
|
||||||
|
incomplete = {name for name in declared if isinstance(name, str)}
|
||||||
|
if any(not isinstance(name, str) for name in declared):
|
||||||
|
malformed_bookkeeping = repr(declared)[:80]
|
||||||
|
elif declared is not None:
|
||||||
|
malformed_bookkeeping = repr(declared)[:80]
|
||||||
|
for lane in sorted(incomplete - {"pipeline"}):
|
||||||
|
if rows.get(lane, ("", ""))[0] == "BLOCKED":
|
||||||
|
continue
|
||||||
|
after = f" after {n} attempt(s)" if (n := attempt_count(lane)) else ""
|
||||||
|
blockers.append(
|
||||||
|
f"{lane}: review incomplete{after} — the lane never produced a usable verdict"
|
||||||
|
)
|
||||||
|
record(lane, "BLOCKED", f"review incomplete{after}")
|
||||||
|
|
||||||
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
# A gauntlet-level fault (build_items crash, dead parse stage) means the
|
||||||
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
# lanes above were never built — SKIPPED rows alone must not read as PASS.
|
||||||
signals_error = state.get("signals_error")
|
signals_error = state.get("signals_error")
|
||||||
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error:
|
pipeline_fault = isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error
|
||||||
|
if pipeline_fault:
|
||||||
blockers.append(f"pipeline: {signals_error}")
|
blockers.append(f"pipeline: {signals_error}")
|
||||||
|
incomplete.add("pipeline")
|
||||||
|
if malformed_bookkeeping is not None:
|
||||||
|
blockers.append(
|
||||||
|
"pipeline: PIPELINE-FAULT — malformed review_incomplete bookkeeping "
|
||||||
|
f"({malformed_bookkeeping}); never a pass"
|
||||||
|
)
|
||||||
|
incomplete.add("pipeline")
|
||||||
|
if "pipeline" in incomplete and not any(b.startswith("pipeline:") for b in blockers):
|
||||||
|
blockers.append(
|
||||||
|
"pipeline: review incomplete — a pipeline stage never completed "
|
||||||
|
"(no fault text recorded)"
|
||||||
|
)
|
||||||
|
|
||||||
verdict = "BLOCKED" if blockers else "PASS"
|
verdict = "BLOCKED" if blockers or incomplete else "PASS"
|
||||||
|
|
||||||
|
incomplete_lines = []
|
||||||
|
for name in sorted(incomplete):
|
||||||
|
if name == "pipeline":
|
||||||
|
incomplete_lines.append(
|
||||||
|
f"- pipeline: {signals_error}"
|
||||||
|
if pipeline_fault
|
||||||
|
else "- pipeline: review incomplete (no fault text recorded)"
|
||||||
|
)
|
||||||
|
elif n := attempt_count(name):
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
|
||||||
|
else:
|
||||||
|
incomplete_lines.append(f"- {name}: no completed review")
|
||||||
|
incomplete_line = (
|
||||||
|
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
|
||||||
|
if incomplete
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
|
||||||
md = "\n".join(
|
md = "\n".join(
|
||||||
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines
|
["| lane | status | detail |", "|------|--------|--------|"]
|
||||||
|
+ [f"| {lane} | {status} | {detail} |" for lane, (status, detail) in rows.items()]
|
||||||
)
|
)
|
||||||
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
|
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
|
||||||
if blockers:
|
if blockers:
|
||||||
@@ -246,11 +366,21 @@ def main():
|
|||||||
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
|
||||||
if state.get("signals_error"):
|
if state.get("signals_error"):
|
||||||
md += f"\n\n> {state['signals_error']}"
|
md += f"\n\n> {state['signals_error']}"
|
||||||
|
if incomplete_lines:
|
||||||
|
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
|
||||||
md += "\n\n## Full lane reports\n"
|
md += "\n\n## Full lane reports\n"
|
||||||
for lane, rep in reports:
|
for lane, rep in reports:
|
||||||
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
|
||||||
|
|
||||||
print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md}))
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"gauntlet_verdict": verdict,
|
||||||
|
"gauntlet_report": md,
|
||||||
|
"gauntlet_incomplete_line": incomplete_line,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -261,6 +391,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash into a sile
|
|||||||
{
|
{
|
||||||
"gauntlet_verdict": "BLOCKED",
|
"gauntlet_verdict": "BLOCKED",
|
||||||
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
|
||||||
|
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -47,7 +47,7 @@ Spawnable sub-agents (from `config.yaml`):
|
|||||||
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
|
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
|
||||||
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
|
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
|
||||||
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
|
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
|
||||||
- **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed.
|
- **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. A `GAUNTLET_REVIEW_INCOMPLETE:` line under BLOCKED is an infrastructure fault (a lane exhausted its retries, or the pipeline faulted) that Sisyphus escalates to the user — retry / accept NEEDS-HUMAN / abort — never a finding to fix. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed.
|
||||||
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
|
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
|
||||||
- **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion.
|
- **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion.
|
||||||
|
|
||||||
|
|||||||
@@ -329,11 +329,27 @@ instructions: |
|
|||||||
parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`
|
parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`
|
||||||
with a lane table, blockers, attention items, and every lane's full report.
|
with a lane table, blockers, attention items, and every lane's full report.
|
||||||
|
|
||||||
- `GAUNTLET: BLOCKED` blocks completion. Route each blocker through the findings-handling rules in
|
- `GAUNTLET: BLOCKED` blocks completion. Route each blocker that is a real review finding through
|
||||||
the reviewer sections below (🔴 → coder fix in the SAME session; DIVERGES → the adversary rules;
|
the findings-handling rules in the reviewer sections below (🔴 → coder fix in the SAME session;
|
||||||
security FAIL → the security rules; probe INCONCLUSIVE → fix the ENVIRONMENT per the plan's
|
DIVERGES → the adversary rules; security FAIL → the security rules; probe INCONCLUSIVE → fix the
|
||||||
local-run recipe first — it is never a code verdict). Then spawn a FRESH gauntlet run; lane
|
ENVIRONMENT per the plan's local-run recipe first — it is never a code verdict). Then spawn a
|
||||||
selection re-runs from the new diff.
|
FRESH gauntlet run; lane selection re-runs from the new diff. A BLOCKED whose only blockers are
|
||||||
|
review faults (`PIPELINE-FAULT` / `review incomplete` entries) is NOT a code failure — the next
|
||||||
|
rule owns it.
|
||||||
|
- `GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]` (the line directly under `GAUNTLET: BLOCKED`)
|
||||||
|
means the review could NOT be completed: the named lanes still faulted after 3 attempts, or the
|
||||||
|
gauntlet pipeline itself faulted (`pipeline`). This is an infrastructure fault, not a code
|
||||||
|
finding. NEVER route it through the findings-handling rules. NEVER claim done while it stands.
|
||||||
|
Escalate to the user via `user__select` with exactly these three options:
|
||||||
|
- "Retry the review again"
|
||||||
|
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
|
||||||
|
- "Abort this task"
|
||||||
|
Retry → spawn a FRESH gauntlet run. Accept → finish the work and record the skipped review
|
||||||
|
(lane names or `pipeline`) in your final report and in the PR body when you author one. Abort →
|
||||||
|
stop; leave the code in a working state and report. Findings AND incomplete on the same output →
|
||||||
|
fix the findings first, then re-run the gauntlet (the re-run retries the lane) — escalate only if
|
||||||
|
the line persists. Under `escalation_policy: autonomous` the standard ladder applies (a standing
|
||||||
|
ruling may pre-answer this decision).
|
||||||
- `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those
|
- `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those
|
||||||
items into your final report verbatim.
|
items into your final report verbatim.
|
||||||
- The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the
|
- The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the
|
||||||
|
|||||||
+59
-63
@@ -11,74 +11,70 @@
|
|||||||
# - <agent-name>_AGENT_SESSION
|
# - <agent-name>_AGENT_SESSION
|
||||||
# - <agent-name>_VARIABLES (as JSON array of key-value pairs; e.g. '[{"name": "username", "value": "alex"}]')
|
# - <agent-name>_VARIABLES (as JSON array of key-value pairs; e.g. '[{"name": "username", "value": "alex"}]')
|
||||||
|
|
||||||
model: openai:gpt-4o # Specify the LLM to use
|
model: openai:gpt-4o # Specify the LLM to use
|
||||||
temperature: null # Set default temperature parameter, range (0, 1)
|
temperature: null # Set default temperature parameter, range (0, 1)
|
||||||
top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model
|
top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model
|
||||||
reasoning_effort:
|
reasoning_effort: null # Reasoning effort level for models that support it (e.g. low, medium, high).
|
||||||
null # Reasoning effort level for models that support it (e.g. low, medium, high).
|
# Only valid when the agent's model declares reasoning_levels.
|
||||||
# Only valid when the agent's model declares reasoning_levels.
|
agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session
|
||||||
agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session
|
name: <agent-name> # Name of the agent, used in the UI and logs
|
||||||
name: <agent-name> # Name of the agent, used in the UI and logs
|
description: <description> # Description of the agent, used in the UI
|
||||||
description: <description> # Description of the agent, used in the UI
|
version: 1 # Version of the agent
|
||||||
version: 1 # Version of the agent
|
|
||||||
# Auto-Continue (Todo System)
|
# Auto-Continue (Todo System)
|
||||||
# The auto-continue system provides built-in task tracking for improved reliability.
|
# The auto-continue system provides built-in task tracking for improved reliability.
|
||||||
# When enabled, the model can create todo lists and the system will automatically
|
# When enabled, the model can create todo lists and the system will automatically
|
||||||
# prompt it to continue when incomplete tasks remain.
|
# prompt it to continue when incomplete tasks remain.
|
||||||
# See the [Todo System documentation](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System) for more information
|
# See the [Todo System documentation](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System) for more information
|
||||||
auto_continue: false # Enable automatic continuation when incomplete todos remain
|
auto_continue: false # Enable automatic continuation when incomplete todos remain
|
||||||
max_auto_continues: 10 # Maximum number of automatic continuations before stopping
|
max_auto_continues: 10 # Maximum number of automatic continuations before stopping
|
||||||
inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt
|
inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt
|
||||||
continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null)
|
continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null)
|
||||||
|
|
||||||
# Sub-Agent Spawning System
|
# Sub-Agent Spawning System
|
||||||
# Enable this agent to spawn and manage child agents in parallel.
|
# Enable this agent to spawn and manage child agents in parallel.
|
||||||
# See https://github.com/Dark-Alex-17/coyote/wiki/Agents for detailed documentation.
|
# See https://github.com/Dark-Alex-17/coyote/wiki/Agents for detailed documentation.
|
||||||
can_spawn_agents: false # Enable the agent to spawn child agents
|
can_spawn_agents: false # Enable the agent to spawn child agents
|
||||||
# spawnable_agents: # Optional whitelist restricting which agents can be spawned via `agent__spawn`.
|
# spawnable_agents: # Optional whitelist restricting which agents can be spawned via `agent__spawn`.
|
||||||
# - explore # If omitted (the default), ALL installed agents are spawnable. This is the unrestricted default.
|
# - explore # If omitted (the default), ALL installed agents are spawnable. This is the unrestricted default.
|
||||||
# - coder # Provide a list to restrict. Match is exact and case-sensitive (use directory names).
|
# - coder # Provide a list to restrict. Match is exact and case-sensitive (use directory names).
|
||||||
# - oracle # An empty list ([]) means literally nothing spawnable.
|
# - oracle # An empty list ([]) means literally nothing spawnable.
|
||||||
# Also filters `agent__list_available` output so the LLM only sees what it can spawn.
|
# Also filters `agent__list_available` output so the LLM only sees what it can spawn.
|
||||||
# Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes.
|
# Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes.
|
||||||
max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously
|
max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously
|
||||||
max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning)
|
max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning)
|
||||||
max_concurrent_jobs:
|
max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once for this agent
|
||||||
5 # Max background jobs (`job__*` tools) running at once for this agent
|
# (overrides the global setting; 0 disables background jobs for this agent)
|
||||||
# (overrides the global setting; 0 disables background jobs for this agent)
|
inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt
|
||||||
inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt
|
summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model
|
||||||
summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model
|
summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent
|
||||||
summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent
|
escalation_timeout: 0 # 0 = wait indefinitely for the reply (default); N = give up after N seconds
|
||||||
escalation_timeout: 300 # Seconds a sub-agent waits for a user interaction response before timing out (default: 5 minutes)
|
mcp_servers: # Optional list of MCP servers that the agent utilizes
|
||||||
mcp_servers: # Optional list of MCP servers that the agent utilizes
|
- github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file
|
||||||
- github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file
|
mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers
|
||||||
mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers
|
github: # (glob patterns: * and ?). Intersects with the global config,
|
||||||
github: # (glob patterns: * and ?). Intersects with the global config,
|
- get_* # mcp.json `allowedTools`, and every other configured layer. It
|
||||||
- get_* # mcp.json `allowedTools`, and every other configured layer. It
|
- search_* # can only narrow access, never widen it.
|
||||||
- search_* # can only narrow access, never widen it.
|
global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
|
||||||
global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
|
|
||||||
- web_search
|
- web_search
|
||||||
- fs
|
- fs
|
||||||
- python
|
- python
|
||||||
skills_enabled:
|
skills_enabled: true # Master switch for skills in this agent (default: inherit from global).
|
||||||
true # Master switch for skills in this agent (default: inherit from global).
|
# Skills also require `function_calling_support: true` in the global config.
|
||||||
# Skills also require `function_calling_support: true` in the global config.
|
enabled_skills: # Optional list of skills available when this agent runs.
|
||||||
enabled_skills: # Optional list of skills available when this agent runs.
|
- git-master # Must be a subset of global `visible_skills`. Omit to inherit the global default.
|
||||||
# Must be a subset of global `visible_skills`. Omit to inherit the global default.
|
|
||||||
- git-master
|
|
||||||
- ai-slop-remover
|
- ai-slop-remover
|
||||||
inject_skill_instructions:
|
inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled
|
||||||
true # Inject a short hint pointing the model at `skill__list` when skills are enabled
|
# (default: true). Suppressed automatically when no skills are available.
|
||||||
# (default: true). Suppressed automatically when no skills are available.
|
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
|
||||||
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
|
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
|
||||||
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
|
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
|
||||||
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
|
memory: null # Per-agent memory override (default: inherit). Set to `false` to disable memory
|
||||||
memory:
|
# for this agent regardless of workspace/global presence. See the Memory wiki page.
|
||||||
null # Per-agent memory override (default: inherit). Set to `false` to disable memory
|
|
||||||
# for this agent regardless of workspace/global presence. See the Memory wiki page.
|
|
||||||
|
|
||||||
dynamic_instructions: false # Whether to use dynamic instructions for the agent; if false, static instructions are used
|
dynamic_instructions: false # Whether to use dynamic instructions for the agent; if false, static instructions are used
|
||||||
instructions:
|
instructions: # Static instructions for the agent; ignored if dynamic instructions are used
|
||||||
| # Static instructions for the agent; ignored if dynamic instructions are used
|
|
|
||||||
You are a AI agent designed to demonstrate agent capabilities.
|
You are a AI agent designed to demonstrate agent capabilities.
|
||||||
|
|
||||||
<tools>
|
<tools>
|
||||||
@@ -99,10 +95,10 @@ instructions:
|
|||||||
username: {{username}}
|
username: {{username}}
|
||||||
</user>
|
</user>
|
||||||
variables: # Optional variables for the agent
|
variables: # Optional variables for the agent
|
||||||
# The variables defined above like {{__variable_name__}} are automatically available
|
# The variables defined above like {{__variable_name__}} are automatically available
|
||||||
- name: username
|
- name: username
|
||||||
description: Your user name
|
description: Your user name
|
||||||
default: null # A default value for this variable; if null, the variable must be provided when starting the agent
|
default: null # A default value for this variable; if null, the variable must be provided when starting the agent
|
||||||
conversation_starters: # Optional conversation starters for the agent
|
conversation_starters: # Optional conversation starters for the agent
|
||||||
- What is the meaning of life?
|
- What is the meaning of life?
|
||||||
- Tell me a joke.
|
- Tell me a joke.
|
||||||
@@ -114,15 +110,15 @@ conversation_starters: # Optional conversation starters for the agent
|
|||||||
- How do I stay motivated?
|
- How do I stay motivated?
|
||||||
- What is the best way to exercise?
|
- What is the best way to exercise?
|
||||||
- How do I manage my time effectively?
|
- How do I manage my time effectively?
|
||||||
documents: # Optional documents to load for the agent
|
documents: # Optional documents to load for the agent
|
||||||
# To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval),
|
# To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval),
|
||||||
# set `rag_extractor_model` in your global config.yaml.
|
# set `rag_extractor_model` in your global config.yaml.
|
||||||
# See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag
|
# See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag
|
||||||
- git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path
|
- git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path
|
||||||
- pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path
|
- pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path
|
||||||
- https://some-website.com/some-page
|
- https://some-website.com/some-page
|
||||||
- some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file
|
- some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file
|
||||||
- ~/some-file.txt # File in the user's home directory
|
- ~/some-file.txt # File in the user's home directory
|
||||||
- /absolute/path/to/some-file.md # File with absolute path
|
- /absolute/path/to/some-file.md # File with absolute path
|
||||||
- /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories
|
- /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories
|
||||||
- /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1)
|
- /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1)
|
||||||
+2
-1
@@ -889,7 +889,7 @@ fn default_summarization_threshold() -> usize {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn default_escalation_timeout() -> u64 {
|
fn default_escalation_timeout() -> u64 {
|
||||||
300
|
0
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AgentConfig {
|
impl AgentConfig {
|
||||||
@@ -1341,6 +1341,7 @@ variables:
|
|||||||
assert_eq!(config.max_concurrent_agents, 4);
|
assert_eq!(config.max_concurrent_agents, 4);
|
||||||
assert_eq!(config.max_agent_depth, 3);
|
assert_eq!(config.max_agent_depth, 3);
|
||||||
assert_eq!(config.max_auto_continues, 10);
|
assert_eq!(config.max_auto_continues, 10);
|
||||||
|
assert_eq!(config.escalation_timeout, 0);
|
||||||
assert!(config.mcp_servers.is_empty());
|
assert!(config.mcp_servers.is_empty());
|
||||||
assert!(config.global_tools.is_empty());
|
assert!(config.global_tools.is_empty());
|
||||||
assert!(config.conversation_starters.is_empty());
|
assert!(config.conversation_starters.is_empty());
|
||||||
|
|||||||
@@ -1383,6 +1383,12 @@ clients:
|
|||||||
assert_eq!(cfg.clients.len(), 1);
|
assert_eq!(cfg.clients.len(), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn config_template_does_not_carry_the_per_agent_escalation_timeout_key() {
|
||||||
|
// `escalation_timeout` is a per-agent setting; the global template must never grow it.
|
||||||
|
assert!(!CONFIG_TEMPLATE.contains("escalation_timeout"));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn config_enabled_macros_empty_string_is_some_empty() {
|
fn config_enabled_macros_empty_string_is_some_empty() {
|
||||||
let cfg: Config = serde_yaml::from_str("enabled_macros: \"\"").unwrap();
|
let cfg: Config = serde_yaml::from_str("enabled_macros: \"\"").unwrap();
|
||||||
|
|||||||
@@ -200,7 +200,7 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {"
|
|||||||
1. Read the child's question and options.
|
1. Read the child's question and options.
|
||||||
2. If you can answer from context, call `agent__reply_escalation` with your answer.
|
2. If you can answer from context, call `agent__reply_escalation` with your answer.
|
||||||
3. If you need the user's input, call the appropriate `user__*` tool yourself, then relay the answer via `agent__reply_escalation`.
|
3. If you need the user's input, call the appropriate `user__*` tool yourself, then relay the answer via `agent__reply_escalation`.
|
||||||
4. **Respond promptly**; the child agent is blocked and waiting (5-minute timeout).
|
4. **Respond promptly**; the child blocks until answered — no timeout by default (`escalation_timeout` per agent).
|
||||||
"};
|
"};
|
||||||
|
|
||||||
pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {"
|
pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {"
|
||||||
|
|||||||
+1964
-9
File diff suppressed because it is too large.
Load diff
@@ -1711,7 +1711,7 @@ fn handle_reply_escalation(ctx: &mut RequestContext, args: &Value) -> Result<Val
|
|||||||
}
|
}
|
||||||
None => Ok(json!({
|
None => Ok(json!({
|
||||||
"status": "error",
|
"status": "error",
|
||||||
"message": format!("No pending escalation found with id '{escalation_id}'. It may have already been replied to or timed out."),
|
"message": format!("No pending escalation found with id '{escalation_id}'. It may have already been replied to."),
|
||||||
})),
|
})),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -3093,6 +3093,10 @@ mod tests {
|
|||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
assert_eq!(result["status"], "error");
|
assert_eq!(result["status"], "error");
|
||||||
|
assert_eq!(
|
||||||
|
result["message"],
|
||||||
|
"No pending escalation found with id 'missing'. It may have already been replied to."
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ use tokio::sync::oneshot;
|
|||||||
|
|
||||||
pub const USER_FUNCTION_PREFIX: &str = "user__";
|
pub const USER_FUNCTION_PREFIX: &str = "user__";
|
||||||
|
|
||||||
const DEFAULT_ESCALATION_TIMEOUT_SECS: u64 = 300;
|
const DEFAULT_ESCALATION_TIMEOUT_SECS: u64 = 0;
|
||||||
const CUSTOM_MULTI_CHOICE_ANSWER_OPTION: &str = "Other (custom)";
|
const CUSTOM_MULTI_CHOICE_ANSWER_OPTION: &str = "Other (custom)";
|
||||||
|
|
||||||
pub fn user_interaction_function_declarations() -> Vec<FunctionDeclaration> {
|
pub fn user_interaction_function_declarations() -> Vec<FunctionDeclaration> {
|
||||||
@@ -289,17 +289,32 @@ async fn handle_escalated(ctx: &RequestContext, action: &str, args: &Value) -> R
|
|||||||
|
|
||||||
root_queue.submit(request);
|
root_queue.submit(request);
|
||||||
|
|
||||||
let timeout = Duration::from_secs(timeout_secs);
|
await_escalation_reply(rx, timeout_secs).await
|
||||||
match tokio::time::timeout(timeout, rx).await {
|
}
|
||||||
Ok(Ok(reply)) => Ok(json!({ "answer": reply })),
|
|
||||||
Ok(Err(_)) => Ok(json!({
|
/// Waits for the parent's reply. `timeout_secs == 0` waits indefinitely; a
|
||||||
"error": "Escalation was cancelled. The parent agent dropped the request",
|
/// dropped sender (the parent cancelled the request) resolves either way.
|
||||||
"fallback": "Make your best judgment and proceed",
|
async fn await_escalation_reply(rx: oneshot::Receiver<String>, timeout_secs: u64) -> Result<Value> {
|
||||||
})),
|
let reply = if timeout_secs == 0 {
|
||||||
|
rx.await
|
||||||
|
} else {
|
||||||
|
match tokio::time::timeout(Duration::from_secs(timeout_secs), rx).await {
|
||||||
|
Ok(reply) => reply,
|
||||||
|
Err(_) => {
|
||||||
|
return Ok(json!({
|
||||||
|
"error": format!(
|
||||||
|
"Escalation timed out after {timeout_secs} seconds waiting for user response"
|
||||||
|
),
|
||||||
|
"fallback": "Make your best judgment and proceed",
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
match reply {
|
||||||
|
Ok(reply) => Ok(json!({ "answer": reply })),
|
||||||
Err(_) => Ok(json!({
|
Err(_) => Ok(json!({
|
||||||
"error": format!(
|
"error": "Escalation was cancelled. The parent agent dropped the request",
|
||||||
"Escalation timed out after {timeout_secs} seconds waiting for user response"
|
|
||||||
),
|
|
||||||
"fallback": "Make your best judgment and proceed",
|
"fallback": "Make your best judgment and proceed",
|
||||||
})),
|
})),
|
||||||
}
|
}
|
||||||
@@ -331,6 +346,8 @@ fn parse_options(args: &Value) -> Result<Vec<String>> {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use std::fs;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn headless_select_returns_structured_json() {
|
fn headless_select_returns_structured_json() {
|
||||||
@@ -351,4 +368,116 @@ mod tests {
|
|||||||
assert_eq!(v["action"], "confirm");
|
assert_eq!(v["action"], "confirm");
|
||||||
assert_eq!(v["options"], json!([]));
|
assert_eq!(v["options"], json!([]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn escalation_timeout_defaults_to_unlimited() {
|
||||||
|
assert_eq!(DEFAULT_ESCALATION_TIMEOUT_SECS, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn zero_timeout_waits_indefinitely_for_the_reply() {
|
||||||
|
let (tx, rx) = oneshot::channel();
|
||||||
|
let wait = tokio::spawn(await_escalation_reply(rx, 0));
|
||||||
|
|
||||||
|
tokio::time::advance(Duration::from_secs(3600)).await;
|
||||||
|
assert!(!wait.is_finished());
|
||||||
|
|
||||||
|
tx.send("approved".to_string()).unwrap();
|
||||||
|
let v = wait.await.unwrap().unwrap();
|
||||||
|
assert_eq!(v, json!({ "answer": "approved" }));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn nonzero_timeout_still_expires_with_the_existing_message() {
|
||||||
|
let (tx, rx) = oneshot::channel::<String>();
|
||||||
|
let wait = tokio::spawn(await_escalation_reply(rx, 5));
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
|
||||||
|
tokio::time::advance(Duration::from_secs(4)).await;
|
||||||
|
assert!(!wait.is_finished());
|
||||||
|
|
||||||
|
tokio::time::advance(Duration::from_secs(2)).await;
|
||||||
|
let v = wait.await.unwrap().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
v["error"],
|
||||||
|
"Escalation timed out after 5 seconds waiting for user response"
|
||||||
|
);
|
||||||
|
assert_eq!(v["fallback"], "Make your best judgment and proceed");
|
||||||
|
assert!(tx.send("too late".to_string()).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(start_paused = true)]
|
||||||
|
async fn zero_timeout_resolves_when_the_sender_is_dropped() {
|
||||||
|
let (tx, rx) = oneshot::channel::<String>();
|
||||||
|
drop(tx);
|
||||||
|
|
||||||
|
let v = tokio::time::timeout(Duration::from_secs(1), await_escalation_reply(rx, 0))
|
||||||
|
.await
|
||||||
|
.expect("the wait must complete once the sender is gone")
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
v["error"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.starts_with("Escalation was cancelled"),
|
||||||
|
"{v}"
|
||||||
|
);
|
||||||
|
assert_eq!(v["fallback"], "Make your best judgment and proceed");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn stale_timeout_wording_is_gone_from_shipped_text() {
|
||||||
|
// Split so this file does not match its own needles.
|
||||||
|
let needles = [
|
||||||
|
concat!("(5-minute", " timeout)"),
|
||||||
|
concat!("escalation_timeout:", " 300"),
|
||||||
|
concat!("default:", " 5 minutes"),
|
||||||
|
];
|
||||||
|
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||||
|
let example = root.join("config.agent.example.yaml");
|
||||||
|
let example_text = fs::read_to_string(&example).unwrap();
|
||||||
|
for needle in ["escalation_timeout: 0", "0 = wait indefinitely"] {
|
||||||
|
assert!(
|
||||||
|
example_text.contains(needle),
|
||||||
|
"{} lost the shipped wording {needle:?}",
|
||||||
|
example.display()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut files = vec![example];
|
||||||
|
collect_files(&root.join("src"), &mut files);
|
||||||
|
collect_files(&root.join("assets"), &mut files);
|
||||||
|
let this_file = root.join(file!());
|
||||||
|
assert!(
|
||||||
|
files.contains(&this_file),
|
||||||
|
"walk did not reach {}",
|
||||||
|
this_file.display()
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut hits = Vec::new();
|
||||||
|
for path in &files {
|
||||||
|
let text = String::from_utf8_lossy(&fs::read(path).unwrap()).into_owned();
|
||||||
|
for needle in needles {
|
||||||
|
if text.contains(needle) {
|
||||||
|
hits.push(format!("{}: {needle:?}", path.display()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
hits.is_empty(),
|
||||||
|
"stale timeout wording found:\n{}",
|
||||||
|
hits.join("\n")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn collect_files(dir: &Path, out: &mut Vec<PathBuf>) {
|
||||||
|
for entry in fs::read_dir(dir).unwrap().flatten() {
|
||||||
|
let path = entry.path();
|
||||||
|
if path.is_dir() {
|
||||||
|
collect_files(&path, out);
|
||||||
|
} else if path.is_file() {
|
||||||
|
out.push(path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -1325,6 +1325,117 @@ nodes:
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A script `_next` back-edge into a node with a static fan-out must
|
||||||
|
/// re-fire every fan-out target on the next pass, and a map that
|
||||||
|
/// re-runs over zero items must overwrite its `collect_into` with `[]`
|
||||||
|
/// rather than leave the previous pass's results in place.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn script_next_back_edge_refires_static_fan_out_and_empty_map_rerun_clears_collect_into()
|
||||||
|
{
|
||||||
|
if !cmd_available("python3") {
|
||||||
|
eprintln!("skipping: python3 not available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let ws = TestWorkspace::new();
|
||||||
|
ws.write_script(
|
||||||
|
"build.py",
|
||||||
|
r#"#!/usr/bin/env python3
|
||||||
|
import os, json
|
||||||
|
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
if state["pass"] == 0:
|
||||||
|
print(json.dumps({"a_items": [1], "b_items": [1]}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"a_items": [], "b_items": [2]}))
|
||||||
|
"#,
|
||||||
|
);
|
||||||
|
ws.write_script(
|
||||||
|
"worker.py",
|
||||||
|
r#"#!/usr/bin/env python3
|
||||||
|
import os, json
|
||||||
|
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
print(json.dumps({"output": f"report-{state['item']}"}))
|
||||||
|
"#,
|
||||||
|
);
|
||||||
|
ws.write_script(
|
||||||
|
"gate.py",
|
||||||
|
r#"#!/usr/bin/env python3
|
||||||
|
import os, json
|
||||||
|
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||||
|
if state["pass"] == 0:
|
||||||
|
print(json.dumps({"pass": 1, "_next": "build"}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({}))
|
||||||
|
"#,
|
||||||
|
);
|
||||||
|
|
||||||
|
let yaml = r#"
|
||||||
|
name: fan_out_rerun_test
|
||||||
|
start: build
|
||||||
|
initial_state:
|
||||||
|
pass: 0
|
||||||
|
a_items: []
|
||||||
|
b_items: []
|
||||||
|
a_results: []
|
||||||
|
b_results: []
|
||||||
|
nodes:
|
||||||
|
build:
|
||||||
|
type: script
|
||||||
|
script: build.py
|
||||||
|
next: [map_a, map_b]
|
||||||
|
map_a:
|
||||||
|
type: map
|
||||||
|
over: "{{a_items}}"
|
||||||
|
as: item
|
||||||
|
branch: worker
|
||||||
|
collect_into: a_results
|
||||||
|
next: gate
|
||||||
|
map_b:
|
||||||
|
type: map
|
||||||
|
over: "{{b_items}}"
|
||||||
|
as: item
|
||||||
|
branch: worker
|
||||||
|
collect_into: b_results
|
||||||
|
next: gate
|
||||||
|
worker:
|
||||||
|
type: script
|
||||||
|
script: worker.py
|
||||||
|
state_updates: {}
|
||||||
|
timeout: 60
|
||||||
|
gate:
|
||||||
|
type: script
|
||||||
|
script: gate.py
|
||||||
|
next: done
|
||||||
|
done:
|
||||||
|
type: end
|
||||||
|
output: "{{a_results}}|{{b_results}}"
|
||||||
|
"#;
|
||||||
|
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
|
||||||
|
let mut ctx = make_ctx();
|
||||||
|
let abort = create_abort_signal();
|
||||||
|
let result = GraphExecutor::new(graph, &ws.dir)
|
||||||
|
.execute(&mut ctx, abort)
|
||||||
|
.await
|
||||||
|
.unwrap_or_else(|e| panic!("executor failed: {e:#}"));
|
||||||
|
|
||||||
|
let (a, b) = result
|
||||||
|
.split_once('|')
|
||||||
|
.unwrap_or_else(|| panic!("expected `a|b`, got: {result}"));
|
||||||
|
let a: Value =
|
||||||
|
serde_json::from_str(a).unwrap_or_else(|_| panic!("a_results not JSON: {result}"));
|
||||||
|
let b: Value =
|
||||||
|
serde_json::from_str(b).unwrap_or_else(|_| panic!("b_results not JSON: {result}"));
|
||||||
|
assert_eq!(
|
||||||
|
b,
|
||||||
|
serde_json::json!(["report-2"]),
|
||||||
|
"the back-edge into build must re-fire both fan-out targets: {result}"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
a,
|
||||||
|
serde_json::json!([]),
|
||||||
|
"map_a must re-fire on the back-edge pass and clear its collect_into over zero items: {result}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn parallel_branch_error_aborts_super_step() {
|
async fn parallel_branch_error_aborts_super_step() {
|
||||||
if !cmd_available("bash") {
|
if !cmd_available("bash") {
|
||||||
|
|||||||
+9
-1
@@ -2281,6 +2281,7 @@ where
|
|||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
use std::time::{SystemTime, UNIX_EPOCH};
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
struct TempDir {
|
struct TempDir {
|
||||||
@@ -2289,10 +2290,17 @@ mod tests {
|
|||||||
|
|
||||||
impl TempDir {
|
impl TempDir {
|
||||||
fn new(tag: &str) -> Self {
|
fn new(tag: &str) -> Self {
|
||||||
let unique = SystemTime::now()
|
// Wall-clock nanos alone are not unique across parallel tests
|
||||||
|
// (coarse clocks hand the same tick to concurrent callers, so two
|
||||||
|
// tests would share a directory and overwrite each other's files);
|
||||||
|
// a process-wide counter makes every name distinct.
|
||||||
|
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||||||
|
let nanos = SystemTime::now()
|
||||||
.duration_since(UNIX_EPOCH)
|
.duration_since(UNIX_EPOCH)
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.as_nanos();
|
.as_nanos();
|
||||||
|
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||||||
|
let unique = format!("{nanos}-{seq}");
|
||||||
let path = env::temp_dir().join(format!("coyote-rag-{tag}-{unique}"));
|
let path = env::temp_dir().join(format!("coyote-rag-{tag}-{unique}"));
|
||||||
fs::create_dir_all(&path).unwrap();
|
fs::create_dir_all(&path).unwrap();
|
||||||
Self { path }
|
Self { path }
|
||||||
|
|||||||
Reference in new issue
Block a user