Merge pull request #28 from Dark-Alex-17/feat/PLAN-review-retry

PLAN-review-retry: self-healing review gate (fault-only lane retries) + untimed approvals
This commit is contained in:
Alex Clarke authored and GitHub committed 2026-09-14 11:46:15 -06:00
commit 9a6ffcb8ca
20 files changed
+2862 -147

No files matched your search

+1 -1
View File
@@ -3,7 +3,7 @@ name: CI
on: on:
pull_request: pull_request:
branches: branches:
- '*' - '**'
push: push:
branches: branches:
- main - main
+1 -1
View File
@@ -177,7 +177,7 @@ By default (`parallel_tasks: 0`) tasks run **sequentially** on the single run br
| [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. | | [`explore`](../explore/README.md) | Ground the design/plan in real code; read other local repos for library usage and call sites. |
| [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. | | [`librarian`](../librarian/README.md) | External docs / OSS examples for unfamiliar libraries. |
| [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. | | [`whetstone`](../whetstone/README.md) | Continuous improvement — spawned once before the final report when the run addressed PR review feedback from outside its own lanes; converts review misses into user-approved suite edits / ledger entries. Never blocks the run. |
| [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; direct adversary/probe spawns remain the fallback. | | [`review-gauntlet`](../review-gauntlet/README.md) | Preferred per-task conformance gate — one spawn running adversary (+ probe for consumer-facing tasks) with a machine-parsed GAUNTLET: PASS/BLOCKED verdict; a `GAUNTLET_REVIEW_INCOMPLETE:` line is an infrastructure fault escalated to the user (retry / accept NEEDS-HUMAN / abort), never a divergence; direct adversary/probe spawns remain the fallback. |
## Related skills ## Related skills
+22 -3
View File
@@ -341,9 +341,28 @@ instructions: |
prompt also carries pasted plan text from the repo under review, and whatever is declared prompt also carries pasted plan text from the repo under review, and whatever is declared
here is executed with a shell — a trust boundary the prose must not cross. here is executed with a shell — a trust boundary the prose must not cross.
It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED apply the same fix-and-rerun rules as below It returns `GAUNTLET: PASS/BLOCKED`. On BLOCKED with real review findings apply the same
(resume the SAME Sisyphus session with the findings verbatim, then a FRESH gauntlet run). fix-and-rerun rules as below (resume the SAME Sisyphus session with the findings verbatim,
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed. then a FRESH gauntlet run). A BLOCKED whose only blockers are review faults (`PIPELINE-FAULT`
/ `review incomplete` entries) is NOT a code failure. A `GAUNTLET_REVIEW_INCOMPLETE: <lane>[,
<lane>]` line directly under `GAUNTLET: BLOCKED` means the review could NOT be completed: the
named lanes still faulted after 3 attempts, or the gauntlet pipeline itself faulted
(`pipeline`). This is an infrastructure fault, not a code finding.
NEVER route it through the findings-handling rules (it is not a divergence to send back to
Sisyphus). NEVER claim done while it stands.
Escalate to the user via `user__select` with exactly these three options:
- "Retry the review again"
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
- "Abort this task"
Retry → a FRESH gauntlet run. Accept → close the task and record the skipped review (lane
names or `pipeline`) in the task's `log.md`, the Phase F digest, and the PR body. Abort →
leave the task `in-progress`, log the abort in its `log.md`, and stop to report. Findings AND
incomplete on the same output → fix the findings first, then re-run the gauntlet (the re-run
retries the lane) — escalate only if the line persists. Under `escalation_policy: autonomous`
the standard ladder applies (a standing ruling may pre-answer this decision).
The direct `adversary`/probe spawns below are the FALLBACK when the gauntlet is not installed;
on that path an adversary/probe result that is ONLY a `PIPELINE-FAULT` (no verdict on the
code) is likewise an infrastructure fault to retry, never a divergence.
- **Spawn `adversary`** with the diff base and the criteria pasted in: - **Spawn `adversary`** with the diff base and the criteria pasted in:
``` ```
+79 -14
View File
@@ -26,8 +26,10 @@ flowchart TD
radv -. "fallback" .-> lfault radv -. "fallback" .-> lfault
rsec -. "fallback" .-> lfault rsec -. "fallback" .-> lfault
rpb -. "fallback" .-> lfault rpb -. "fallback" .-> lfault
mcr & madv & msec & mpb --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"] mcr & madv & msec & mpb --> rgate["retry_gate<br/>(script: fault-only re-run gate;<br/>stash + restore, ≤3 attempts,<br/>wall-clock budget)"]
gate --> done(["GAUNTLET: PASS | BLOCKED"]) rgate -. "_next: retry faulted lanes" .-> build
rgate --> gate["verdict_gate<br/>(script: sentinel regex — deterministic)"]
gate --> done(["GAUNTLET: PASS | BLOCKED<br/>(+ GAUNTLET_REVIEW_INCOMPLETE: … when any lane or the pipeline was incomplete)"])
``` ```
## Why a graph ## Why a graph
@@ -49,12 +51,17 @@ those rules are *structure*:
- **Conditional lanes without dynamic routing**: each lane is a `map` node - **Conditional lanes without dynamic routing**: each lane is a `map` node
over a 0-or-1-item list — an empty list means the map runs zero branches over a 0-or-1-item list — an empty list means the map runs zero branches
(map-over-nothing), which *is* the skip mechanism. All four maps join at (map-over-nothing), which *is* the skip mechanism. All four maps join at
the verdict gate on equal-length paths. the retry gate on equal-length paths; it falls through to the verdict
gate, or routes back to `build_items` for faulted lanes only.
- **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing - **The verdict is a regex, not an opinion** (`verdict_gate.py`): a missing
sentinel is a lane FAILURE, never a pass; the critical count from the sentinel is a lane FAILURE, never a pass; the critical count from the
code-review report's summary line (raw 🔴 count if absent) blocks code-review report's summary line (raw 🔴 count if absent) blocks
regardless of its verdict line; a code-review lane whose own verdict line regardless of its verdict line; a code-review lane whose own verdict line
records a pipeline fault blocks (a degraded review is never a pass); records a pipeline fault blocks (a degraded review is never a pass); the
adversary's own degraded-run header (the `Criteria: …` line directly under
its sentinel recording a pipeline fault, died criterion checks, or its
verdict script's crash stub) blocks as a pipeline fault,
not as a DIVERGES finding;
`NEEDS-HUMAN` without 🔴 passes but `NEEDS-HUMAN` without 🔴 passes but
is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks is surfaced under *Human attention required*; probe `INCONCLUSIVE` blocks
with an environment note — it is never treated as PASS or FAIL. with an environment note — it is never treated as PASS or FAIL.
@@ -77,12 +84,59 @@ those rules are *structure*:
`GAUNTLET:` sentinel is always emitted: even a crashed gate prints `GAUNTLET:` sentinel is always emitted: even a crashed gate prints
`GAUNTLET: BLOCKED`, never silence. `GAUNTLET: BLOCKED`, never silence.
## Fault-only re-runs
`retry_gate.py` sits between the lane maps and the verdict gate and re-runs
only lanes that **faulted** — a missing report (stood in for by a synthetic
`PIPELINE-FAULT: <lane> lane produced no report` marker so it never renders as
SKIPPED), a non-string report, the `lane_fault` marker, a sentinel-less
report, or a nested graph's own degraded-run wording (the code-review verdict
line / the adversary's header line directly under its sentinel). A real verdict
(DIVERGES, FAIL, INCONCLUSIVE, NEEDS-HUMAN, …) is never re-run: the review's
judgment stands. Each lane gets up to 3 attempts, all inside a wall-clock
budget (`MAX_RETRY_ELAPSED_SECS`) sized to fit the graph timeout.
- **Stash + restore**: every lane report that ran this pass is stashed in
`kept_results`; a re-run pass overwrites the untouched lanes' collected
results with `[]`, so their stashed reports are restored before the gate.
The verdict gate always sees each lane's *last* report.
- **Retry never narrows selection**: lane selection is not re-run. A re-run
pass rebuilds items only for the retried lanes, and the stashed results
stand in for the rest.
- **Exhaustion is recorded, not hidden**: a lane still faulted after its
attempts or the budget lands in `review_incomplete`, and its fault flows to
the verdict gate as-is.
- **Crash guards append**: the `build_items.py` and `retry_gate.py` crash
guards append to an existing `signals_error` rather than overwrite it, and
`build_items.py` preserves the prior `lanes_summary`.
A non-empty `GAUNTLET_REVIEW_INCOMPLETE:` line structurally implies
`GAUNTLET: BLOCKED`: the verdict gate forces every lane named in
`review_incomplete` to a BLOCKED row (`review incomplete after N attempt(s)`)
and a matching blocker, so a restored earlier report can never turn an
incomplete review into a pass — and malformed `review_incomplete` bookkeeping
is itself recorded as a `pipeline` fault, never dropped.
Whenever at least one lane (or the pipeline) could not complete, the first
line reads `GAUNTLET: BLOCKED` followed immediately by
`GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]`. The names are the exhausted
lanes from `review_incomplete`, plus the `pipeline` pseudo-lane whenever a
pipeline-level `PIPELINE-FAULT:` fired — parse death, or a crash in the
builder, the retry gate, the default-lanes fallback, or the verdict gate
itself (the default-lanes fallback's crash still runs the widened lanes, so
`pipeline` appears alongside real lane rows). The report gains a
`## Review incomplete` section naming each exhausted lane with its
attempt count; `PIPELINE-FAULT` lane rows carry ` ×N` when the lane was
re-run (N > 1). The *named* entries are faults to re-run, never code
findings; the line can accompany real findings from the other lanes, which
still appear under `## Blockers` and must be fixed as usual.
## Sentinels parsed ## Sentinels parsed
| Lane | Sentinel | Blocks on | | Lane | Sentinel | Blocks on |
|------|----------|-----------| |------|----------|-----------|
| code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault | | code-reviewer | `**Verdict: MERGE-READY \| NEEDS-HUMAN**` | critical count from the report's summary line (raw 🔴 count if absent); missing sentinel; own verdict line records a pipeline fault |
| adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; missing sentinel | | adversary | `ADVERSARIAL_REVIEW: CONFORMS \| DIVERGES` | DIVERGES; own degraded-run header (pipeline fault, died criterion checks, or its verdict script's crash stub); missing sentinel |
| security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel | | security-reviewer | `SECURITY_REVIEW: PASS \| FAIL` | FAIL; missing sentinel |
| probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel | | probe | `USAGE_PROBE: PASS \| FAIL \| INCONCLUSIVE` | FAIL; INCONCLUSIVE (environment note); missing sentinel |
@@ -115,22 +169,33 @@ coyote -a review-gauntlet --agent-variable project_dir /abs/path/to/repo \
--agent-variable verification_commands '["cargo test --all"]' "<prompt>" --agent-variable verification_commands '["cargo test --all"]' "<prompt>"
``` ```
The reply ends with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`, a lane status The reply opens with `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` (plus the
`GAUNTLET_REVIEW_INCOMPLETE:` line when applicable), followed by a lane status
table, the lane-selection reasons, blockers, attention items, and each lane's table, the lane-selection reasons, blockers, attention items, and each lane's
full report in a collapsible section. On BLOCKED: fix the findings per your full report in a collapsible section. On BLOCKED with a
own findings-handling rules, then spawn a **fresh** gauntlet run. On PASS with `GAUNTLET_REVIEW_INCOMPLETE:` line: the named lanes never completed — this is an
*Human attention required*: complete the work but carry those items into your infrastructure fault, never a code finding. Fix any non-fault blockers listed
final report verbatim. under `## Blockers` first (the re-run retries the lane); if the line persists,
escalate to the user with three options — retry the review again (a **fresh**
gauntlet run), accept NEEDS-HUMAN and proceed without that lane (recorded in the
report / PR body), or abort — never route it through your findings-handling
rules. On BLOCKED without it: fix the findings per your own findings-handling
rules, then spawn a **fresh** gauntlet run. On PASS with *Human attention
required*: complete the work but carry those items into your final report
verbatim.
## Notes ## Notes
- The caller decides *whether* review is warranted at all (the gauntlet is for - The caller decides *whether* review is warranted at all (the gauntlet is for
non-trivial work — that's why code-review is unconditionally on); the non-trivial work — that's why code-review is unconditionally on); the
gauntlet decides *which* of the other lanes apply and what the verdicts mean. gauntlet decides *which* of the other lanes apply and what the verdicts mean.
- Lane sub-agents get generous finite timeouts (2h each; 3h whole-gauntlet) - Lane sub-agents get generous finite per-attempt timeouts (see the
and one retry (`max_attempts: 2`). A lane that still dies or times out falls `timeout:` on each `run_*` node) and one retry (`max_attempts: 2`) inside
back to the `lane_fault` marker — BLOCKED naming the lane, never a silent the lane, then up to 3 attempts of the whole lane from `retry_gate.py` while
pass. the wall-clock budget (`MAX_RETRY_ELAPSED_SECS`) allows; `settings.timeout`
is derived from that envelope in graph.yaml's comment. A lane that still
dies or times out falls back to the `lane_fault` marker — BLOCKED naming the
lane and `GAUNTLET_REVIEW_INCOMPLETE`, never a silent pass.
- The `verification_commands` variable is forwarded to the adversary lane as a - The `verification_commands` variable is forwarded to the adversary lane as a
structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template structured `inputs:` passthrough on `run_adversary` (a lone `{{…}}` template
lands the raw value in the child's state) — never through the lane prompt lands the raw value in the child's state) — never through the lane prompt
+29 -14
View File
@@ -41,13 +41,14 @@ variables:
settings: settings:
max_loop_iterations: 10 max_loop_iterations: 10
log_state_snapshots: true log_state_snapshots: true
# max over agent lanes of (max_attempts × lane timeout) = 2 × 11400 = 22800 # A faulted lane can be re-run by retry_gate for up to 3 passes, and each
# (agent-node retries each get a fresh per-attempt budget, # pass's agent node retries up to max_attempts = 2 with a fresh per-attempt
# src/graph/agent.rs retry_transient/bounded_attempt) + 1200 margin for # budget (src/graph/agent.rs retry_transient/bounded_attempt):
# the script stages (signals/select/build/gate) = 24000; the parse and # 3 passes × 2 attempts × 11400 (max lane timeout) = 68400, + 1200 margin
# select_lanes llm nodes carry no `timeout:` and are bounded by # for the script stages (signals/select/build/retry gate/verdict gate)
# max_iterations. # = 69600; the parse and select_lanes llm nodes carry no `timeout:` and
timeout: 24000 # are bounded by max_iterations.
timeout: 69600
initial_state: initial_state:
diff_spec: worktree diff_spec: worktree
@@ -84,9 +85,15 @@ initial_state:
adversary_results: [] adversary_results: []
security_results: [] security_results: []
probe_results: [] probe_results: []
lane_attempts: {}
kept_results: {}
retry_lanes: []
review_incomplete: []
gauntlet_started_at: 0
lanes_summary: '' lanes_summary: ''
gauntlet_verdict: '' gauntlet_verdict: ''
gauntlet_report: '' gauntlet_report: ''
gauntlet_incomplete_line: ''
start: parse start: parse
@@ -251,7 +258,7 @@ nodes:
build_items: build_items:
id: build_items id: build_items
type: script type: script
description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map. description: Deterministic lane builder — hard rules are the floor, LLM flags may only add, forced lanes override exactly. Emits one 0-or-1-item list per lane; an empty list skips that lane's map. On a re-run pass (retry_lanes non-empty) it skips selection and emits items only for those lanes, each carrying its attempt number.
script: scripts/build_items.py script: scripts/build_items.py
timeout: 30 timeout: 30
next: [map_code_review, map_adversary, map_security, map_probe] next: [map_code_review, map_adversary, map_security, map_probe]
@@ -265,7 +272,7 @@ nodes:
output_key: lane_out output_key: lane_out
collect_into: code_review_results collect_into: code_review_results
max_concurrency: 1 max_concurrency: 1
next: verdict_gate next: retry_gate
map_adversary: map_adversary:
id: map_adversary id: map_adversary
@@ -276,7 +283,7 @@ nodes:
output_key: lane_out output_key: lane_out
collect_into: adversary_results collect_into: adversary_results
max_concurrency: 1 max_concurrency: 1
next: verdict_gate next: retry_gate
map_security: map_security:
id: map_security id: map_security
@@ -287,7 +294,7 @@ nodes:
output_key: lane_out output_key: lane_out
collect_into: security_results collect_into: security_results
max_concurrency: 1 max_concurrency: 1
next: verdict_gate next: retry_gate
map_probe: map_probe:
id: map_probe id: map_probe
@@ -298,7 +305,7 @@ nodes:
output_key: lane_out output_key: lane_out
collect_into: probe_results collect_into: probe_results
max_concurrency: 1 max_concurrency: 1
next: verdict_gate next: retry_gate
run_code_review: run_code_review:
id: run_code_review id: run_code_review
@@ -412,10 +419,18 @@ nodes:
script: scripts/lane_fault.py script: scripts/lane_fault.py
timeout: 30 timeout: 30
retry_gate:
id: retry_gate
type: script
description: Deterministic fault-only re-run gate. Stashes every lane report that ran this pass, restores reports a re-run pass overwrote with [] (a lane not being re-run maps over nothing), classifies each lane that ran as completed / faulted / skipped, counts attempts, and routes back into build_items for faulted lanes that still have attempts left inside the wall-clock budget; otherwise it records them in review_incomplete and falls through to verdict_gate. A real verdict is never re-run, and a gate crash never retries — it records a PIPELINE-FAULT and falls through.
script: scripts/retry_gate.py
timeout: 30
next: verdict_gate
verdict_gate: verdict_gate:
id: verdict_gate id: verdict_gate
type: script type: script
description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. description: Deterministic sentinel parser and gate. A missing sentinel is a lane failure, never a pass; probe INCONCLUSIVE blocks with an environment note; the code-review critical count is read from render.py's summary line (raw 🔴 count if absent) and blocks regardless of the lane's own verdict line; a code-review verdict line that records the nested graph's own pipeline fault blocks as a degraded lane. Lanes retry_gate gave up on, and any pipeline-level fault, are named on a GAUNTLET_REVIEW_INCOMPLETE machine line so a caller can tell an incomplete review from a failed one.
script: scripts/verdict_gate.py script: scripts/verdict_gate.py
timeout: 60 timeout: 60
next: done next: done
@@ -424,6 +439,6 @@ nodes:
id: done id: done
type: end type: end
output: | output: |
GAUNTLET: {{gauntlet_verdict}} GAUNTLET: {{gauntlet_verdict}}{{gauntlet_incomplete_line}}
{{gauntlet_report}} {{gauntlet_report}}
@@ -9,6 +9,10 @@ skip mechanism — no dynamic routing anywhere.
The LLM's add_* flags can only ADD lanes. A flag that is false never The LLM's add_* flags can only ADD lanes. A flag that is false never
removes a lane the hard rules selected. removes a lane the hard rules selected.
A re-run pass (retry_gate emitted `retry_lanes`) skips selection: only the
named lanes get an item, stamped with their attempt number, and every other
lane maps over nothing — retry_gate restores their earlier reports.
Fail-closed: a builder crash emits all four lists empty plus a Fail-closed: a builder crash emits all four lists empty plus a
PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash PIPELINE-FAULT in signals_error — verdict_gate blocks on it and the crash
never kills the graph. never kills the graph.
@@ -16,6 +20,7 @@ never kills the graph.
import json import json
import os import os
import time
# Keep in sync with default_lanes.ALIASES — a caller-forced lane must # Keep in sync with default_lanes.ALIASES — a caller-forced lane must
# canonicalize the same way on the degraded path as it does here. # canonicalize the same way on the degraded path as it does here.
@@ -35,6 +40,18 @@ ALIASES = {
"usage-pattern-testing": "probe", "usage-pattern-testing": "probe",
} }
# Keep in sync with retry_gate.LANES
LANE_KEYS = {
"code-review": "code_review_items",
"adversary": "adversary_items",
"security": "security_items",
"probe": "probe_items",
}
# Filled by main() so the crash guard can append to an upstream fault
# instead of overwriting it, and keep the first pass's lanes_summary.
crash_context = {"signals_error": "", "lanes_summary": ""}
def load_state(): def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"): if path := os.environ.get("GRAPH_STATE_FILE"):
@@ -45,11 +62,38 @@ def load_state():
def main(): def main():
state = load_state() state = load_state()
prior = state.get("signals_error")
crash_context["signals_error"] = prior if isinstance(prior, str) else ""
# retry_gate's wall-clock budget is measured from this stamp; signals.py
# is the normal stamper, this is the defensive fallback.
stamp = {} if state.get("gauntlet_started_at") else {"gauntlet_started_at": time.time()}
def text(key): def text(key):
v = state.get(key) v = state.get(key)
return v.strip() if isinstance(v, str) else "" return v.strip() if isinstance(v, str) else ""
crash_context["lanes_summary"] = text("lanes_summary")
retry_lanes = [lane for lane in (state.get("retry_lanes") or []) if lane in LANE_KEYS]
if retry_lanes:
attempts = state.get("lane_attempts") or {}
retried = [(lane, int(attempts.get(lane) or 0) + 1) for lane in retry_lanes]
note = "- retried: [" + ", ".join(f"{lane} (attempt {n})" for lane, n in retried) + "]"
summary = text("lanes_summary")
print(
json.dumps(
{
**{key: [] for key in LANE_KEYS.values()},
**{
LANE_KEYS[lane]: [{"lane": lane, "attempt": n}] for lane, n in retried
},
"lanes_summary": f"{summary}\n{note}" if summary else note,
"retry_lanes": [],
**stamp,
}
)
)
return
forced_raw = state.get("forced_lanes") or [] forced_raw = state.get("forced_lanes") or []
forced, unknown = [], [] forced, unknown = [], []
for lane in forced_raw: for lane in forced_raw:
@@ -126,6 +170,7 @@ def main():
"security_items": [{"lane": "security"}] if "security" in lanes else [], "security_items": [{"lane": "security"}] if "security" in lanes else [],
"probe_items": [{"lane": "probe"}] if "probe" in lanes else [], "probe_items": [{"lane": "probe"}] if "probe" in lanes else [],
"lanes_summary": "\n".join(f"- {r}" for r in reasons), "lanes_summary": "\n".join(f"- {r}" for r in reasons),
**stamp,
} }
) )
) )
@@ -134,6 +179,9 @@ def main():
try: try:
main() main()
except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph except Exception as e: # noqa: BLE001 — a builder crash must never kill the graph
fault = f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run"
if crash_context["signals_error"]:
fault = f"{crash_context['signals_error']}; {fault}"
print( print(
json.dumps( json.dumps(
{ {
@@ -141,8 +189,8 @@ except Exception as e: # noqa: BLE001 — a builder crash must never kill the g
"adversary_items": [], "adversary_items": [],
"security_items": [], "security_items": [],
"probe_items": [], "probe_items": [],
"lanes_summary": "", "lanes_summary": crash_context["lanes_summary"],
"signals_error": f"PIPELINE-FAULT: lane builder crashed — {e}; no lanes were run", "signals_error": fault,
} }
) )
) )
@@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""Deterministic fault-only re-run gate for the review gauntlet.
Sits between the four lane maps and the verdict gate. Every lane map runs
on every pass — a lane that is not being re-run maps over [] — so a re-run
pass overwrites the untouched lanes' collected results with []. The gate's
hard rules:
- every lane that ran this pass is stashed in kept_results, completed
and faulted alike, and a lane whose map ran nothing this pass gets
its stashed report restored — the verdict gate always sees each
lane's LAST report;
- a lane is FAULTED only when its report is missing (the map ran but
collected nothing — replaced by a synthetic PIPELINE-FAULT marker so
the verdict gate never reads it as SKIPPED), non-string, the
lane_fault.py marker, sentinel-less, or carries the nested graph's own
degraded-run wording; a real verdict (DIVERGES, FAIL, INCONCLUSIVE,
NEEDS-HUMAN, …) is never re-run — the review's judgment stands;
- a faulted lane is re-run only while it has attempts left AND the
gauntlet is inside its wall-clock budget; otherwise it is recorded in
review_incomplete and its fault flows to the verdict gate as-is;
- malformed review_incomplete bookkeeping (not a list, or non-str
entries) keeps its str entries and records a PIPELINE-FAULT in
signals_error — never silently dropped, never split into lane names;
- the gate itself must never retry by accident: any internal error
records a PIPELINE-FAULT in signals_error and falls through to the
verdict gate, which blocks on it.
"""
import json
import os
import re
import time
# 69600 (graph timeout) − 2 × 11400 (largest lane envelope) − 1200 (script stages)
# = 45600; 600 s of slack below that.
MAX_RETRY_ELAPSED_SECS = 45000
MAX_ATTEMPTS = 3
# lane name -> (items key, results key)
LANES = {
"code-review": ("code_review_items", "code_review_results"),
"adversary": ("adversary_items", "adversary_results"),
"security": ("security_items", "security_results"),
"probe": ("probe_items", "probe_results"),
}
# Must stay identical to verdict_gate.py's sentinel regexes: a lane this
# gate calls completed must be one the verdict gate can parse, or a drift
# would re-run lanes the verdict gate accepts (or accept lanes it cannot
# read). A cross-pin test compares the literals in both files.
SENTINELS = {
"code-review": r"Verdict:\**\s*\**\s*(MERGE-READY|NEEDS-HUMAN)",
"adversary": r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)",
"security": r"SECURITY_REVIEW:\s*(PASS|FAIL)",
"probe": r"USAGE_PROBE:\s*(PASS|FAIL|INCONCLUSIVE)",
}
# Filled by main() so the crash guard can append to an upstream fault
# instead of overwriting it, and report the lanes it had already declined.
crash_context = {"signals_error": "", "review_incomplete": []}
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
ADVERSARY_DEGRADED_HEADER = re.compile(
r"Criteria: .*("
r"degraded run: pipeline fault recorded"
r"|degraded run: \d+ criterion check\(s\) died"
r"|verdict computation error:"
r")"
)
def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f:
return json.load(f)
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def adversary_degraded(report):
# The adversary's verdict.py emits its degraded-run header on the line
# directly under the FIRST sentinel; anchoring there keeps a report that
# merely quotes the wording later (observations, complaints) a real verdict.
lines = report.splitlines()
for i, ln in enumerate(lines):
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
nxt = lines[i + 1] if i + 1 < len(lines) else ""
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
return False
def nested_degraded(lane, report):
# The nested graphs complete with a sentinel even when a stage inside
# them died; their own degraded-run wording marks a partial review,
# which is not a completed one. Same anchoring as verdict_gate.py.
if lane == "code-review":
verdict_line = re.search(r"^\*\*Verdict: .*$", report, re.MULTILINE)
return bool(
verdict_line
and re.search(
r"^\*\*Verdict: NEEDS-HUMAN\*\* — .*"
r"(pipeline fault\(s\) recorded|PIPELINE-FAULT:|report rendering error)",
verdict_line.group(0),
)
)
if lane == "adversary":
return adversary_degraded(report)
return False
def is_faulted(lane, report):
if not isinstance(report, str):
return True
if report.strip().startswith("PIPELINE-FAULT:"):
return True
if not re.search(SENTINELS[lane], report):
return True
return nested_degraded(lane, report)
def attempt_count(attempts, lane):
try:
return int(attempts.get(lane) or 0)
except (TypeError, ValueError):
return 0
def main():
state = load_state()
prior = state.get("signals_error")
signals_error = prior if isinstance(prior, str) else ""
crash_context["signals_error"] = signals_error
out = {}
declared = state.get("review_incomplete")
if isinstance(declared, list):
incomplete = [name for name in declared if isinstance(name, str)]
malformed = len(incomplete) != len(declared)
else:
incomplete = []
malformed = declared is not None
if malformed:
fault = (
"PIPELINE-FAULT: malformed review_incomplete bookkeeping "
f"({repr(declared)[:80]})"
)
signals_error = f"{signals_error}; {fault}" if signals_error else fault
crash_context["signals_error"] = signals_error
out["signals_error"] = signals_error
crash_context["review_incomplete"] = incomplete
raw_attempts = state.get("lane_attempts")
attempts = dict(raw_attempts) if isinstance(raw_attempts, dict) else {}
kept = dict(state.get("kept_results") or {})
faulted = []
for lane, (items_key, results_key) in LANES.items():
results = state.get(results_key) or []
if not state.get(items_key):
if not results and lane in kept:
out[results_key] = [kept[lane]]
continue
report = results[0] if results else None
if report is None:
report = f"PIPELINE-FAULT: {lane} lane produced no report"
out[results_key] = [report]
kept[lane] = report
if is_faulted(lane, report):
faulted.append(lane)
attempts[lane] = attempt_count(attempts, lane) + 1
# A missing/zero stamp reads as an ancient start and declines every
# retry — the safe direction when the clock's origin is unknown.
started = state.get("gauntlet_started_at") or 0
within_budget = (time.time() - float(started)) <= MAX_RETRY_ELAPSED_SECS
retry = [lane for lane in faulted if within_budget and attempts[lane] < MAX_ATTEMPTS]
declined = [lane for lane in faulted if lane not in retry]
incomplete = sorted(set(incomplete) | set(declined))
crash_context["review_incomplete"] = incomplete
out.update(
{
"lane_attempts": attempts,
"kept_results": kept,
"retry_lanes": retry,
"review_incomplete": incomplete,
}
)
if retry:
out["_next"] = "build_items"
print(json.dumps(out))
try:
main()
except Exception as e: # noqa: BLE001 — a gate crash must fall through, never retry
fault = f"PIPELINE-FAULT: retry gate crashed — {e}"
if crash_context["signals_error"]:
fault = f"{crash_context['signals_error']}; {fault}"
print(
json.dumps(
{
"retry_lanes": [],
"review_incomplete": crash_context["review_incomplete"],
"signals_error": fault,
}
)
)
@@ -12,6 +12,7 @@ import json
import os import os
import re import re
import subprocess import subprocess
import time
def load_state(): def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"): if path := os.environ.get("GRAPH_STATE_FILE"):
@@ -37,6 +38,10 @@ def git(*args):
out = {"project_dir": proj} out = {"project_dir": proj}
# retry_gate measures its wall-clock budget from this stamp; it is set
# before the git work so a failed diff still starts the clock.
if not state.get("gauntlet_started_at"):
out["gauntlet_started_at"] = time.time()
try: try:
if spec in ("", "worktree"): if spec in ("", "worktree"):
names = git("diff", "--name-only", "HEAD") names = git("diff", "--name-only", "HEAD")
@@ -8,14 +8,20 @@ re-reads, no judgment. The gate's hard rules:
- a non-string lane result (the map collected an object instead of the - a non-string lane result (the map collected an object instead of the
lane's rendered text) blocks — it cannot carry a sentinel; lane's rendered text) blocks — it cannot carry a sentinel;
- a PIPELINE-FAULT lane result (the lane's agent died after retries; - a PIPELINE-FAULT lane result (the lane's agent died after retries;
detected only when the report IS the lane_fault.py marker, i.e. it detected only when the report IS the lane_fault.py marker or
STARTS with it — a real review merely quoting the marker flows to the retry_gate.py's no-report marker, i.e. it STARTS with it — a real
normal rules) or a PIPELINE-FAULT in signals_error (builder/parse review merely quoting the marker flows to the normal rules) or a
fault) blocks — a degraded pipeline is never a pass; PIPELINE-FAULT in signals_error (builder/parse fault) blocks — a
degraded pipeline is never a pass;
- a code-review lane whose OWN verdict line records a pipeline fault - a code-review lane whose OWN verdict line records a pipeline fault
(code-reviewer's verdict.py / render.py degraded-run wording) blocks — (code-reviewer's verdict.py / render.py degraded-run wording) blocks —
the nested graph completed, but with a dead domain lane, verifier, or the nested graph completed, but with a dead domain lane, verifier, or
synthesis behind it, so its critical count is not trustworthy; synthesis behind it, so its critical count is not trustworthy;
- an adversary lane whose OWN header line (the line directly under its
sentinel) records a degraded run — a pipeline fault, died criterion
checks, or its verdict script's crash stub — blocks as a PIPELINE-FAULT,
not as a DIVERGES finding: the nested graph completed with a dead stage
behind it, so its verdict is not a judgment on the code;
- a non-zero critical count in the code-review report blocks regardless - a non-zero critical count in the code-review report blocks regardless
of the lane's own verdict line — derived from render.py's deterministic of the lane's own verdict line — derived from render.py's deterministic
summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴 summary line (`*Reviewed N files, found X critical, …*`); the raw 🔴
@@ -24,14 +30,37 @@ re-reads, no judgment. The gate's hard rules:
- probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never - probe INCONCLUSIVE blocks with an ENVIRONMENT note — it is never
treated as PASS or FAIL (fix the environment per the plan's local-run treated as PASS or FAIL (fix the environment per the plan's local-run
recipe, then re-run); recipe, then re-run);
- a PIPELINE-FAULT lane row carries its attempt count (` ×N`) when retry_gate
re-ran it, so the reader can tell a first-pass death from an exhausted
retry budget;
- an incomplete review — lanes retry_gate gave up on (review_incomplete)
or a pipeline-level fault (reported under the `pipeline` pseudo-lane) —
emits the GAUNTLET_REVIEW_INCOMPLETE: machine line and a "Review
incomplete" section: that is a fault to re-run, never a code finding.
Every lane named in review_incomplete is forced to a BLOCKED row, so a
restored earlier report can never turn an incomplete review into a
pass: a non-empty machine line structurally implies GAUNTLET: BLOCKED.
Malformed review_incomplete bookkeeping (not a list, or non-str
entries) is itself recorded as a `pipeline` fault, never dropped;
- the gate itself must never crash into a silent pass: any internal - the gate itself must never crash into a silent pass: any internal
error emits GAUNTLET: BLOCKED naming the error. error emits GAUNTLET: BLOCKED naming the error, with the pipeline
marked incomplete.
""" """
import json import json
import os import os
import re import re
# Duplicated verbatim in the sibling gate script — the two copies must stay identical.
ADVERSARY_DEGRADED_HEADER = re.compile(
r"Criteria: .*("
r"degraded run: pipeline fault recorded"
r"|degraded run: \d+ criterion check\(s\) died"
r"|verdict computation error:"
r")"
)
def load_state(): def load_state():
if path := os.environ.get("GRAPH_STATE_FILE"): if path := os.environ.get("GRAPH_STATE_FILE"):
with open(path) as f: with open(path) as f:
@@ -39,10 +68,26 @@ def load_state():
return json.loads(os.environ.get("GRAPH_STATE", "{}")) return json.loads(os.environ.get("GRAPH_STATE", "{}"))
def adversary_degraded(report):
# The adversary's verdict.py emits its degraded-run header on the line
# directly under the FIRST sentinel; anchoring there keeps a report that
# merely quotes the wording later (observations, complaints) a real verdict.
lines = report.splitlines()
for i, ln in enumerate(lines):
if re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", ln):
nxt = lines[i + 1] if i + 1 < len(lines) else ""
return bool(ADVERSARY_DEGRADED_HEADER.match(nxt))
return False
def main(): def main():
state = load_state() state = load_state()
blockers, attention, lane_lines, reports = [], [], [], [] blockers, attention, reports = [], [], []
rows = {}
malformed = set() malformed = set()
lane_attempts = state.get("lane_attempts")
if not isinstance(lane_attempts, dict):
lane_attempts = {}
def lane_report(key): def lane_report(key):
arr = state.get(key) or [] arr = state.get(key) or []
@@ -55,7 +100,17 @@ def main():
return json.dumps(item) return json.dumps(item)
def record(lane, status, detail=""): def record(lane, status, detail=""):
lane_lines.append(f"| {lane} | {status} | {detail} |") rows[lane] = (status, detail)
def attempt_count(lane):
try:
return int(lane_attempts.get(lane) or 0)
except (TypeError, ValueError):
return 0
def attempts_suffix(lane):
n = attempt_count(lane)
return f" ×{n}" if n > 1 else ""
def fault_excerpt(report): def fault_excerpt(report):
line = next( line = next(
@@ -64,10 +119,11 @@ def main():
return line.strip()[:200] return line.strip()[:200]
def is_lane_fault(report): def is_lane_fault(report):
# A genuine lane fault is the ENTIRE report emitted by lane_fault.py, # A genuine lane fault is the ENTIRE report — the lane_fault.py marker
# which always begins with the marker. Matching the marker anywhere # or retry_gate.py's no-report marker — which always begins with
# would let a real review that merely QUOTES "PIPELINE-FAULT:" bypass # "PIPELINE-FAULT:". Matching the marker anywhere would let a real
# 🔴 counting and sentinel parsing. # review that merely QUOTES it bypass 🔴 counting and sentinel
# parsing.
return report.strip().startswith("PIPELINE-FAULT:") return report.strip().startswith("PIPELINE-FAULT:")
def lane_fault_blocker(lane, report): def lane_fault_blocker(lane, report):
@@ -75,7 +131,7 @@ def main():
f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced " f"{lane}: PIPELINE-FAULT — the lane failed after retries and produced "
f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})" f"no verdict; a degraded lane is never a pass ({fault_excerpt(report)})"
) )
record(lane, "BLOCKED", "PIPELINE-FAULT (lane failed)") record(lane, "BLOCKED", f"PIPELINE-FAULT (lane failed{attempts_suffix(lane)})")
def malformed_blocker(lane, report): def malformed_blocker(lane, report):
blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass") blockers.append(f"{lane}: malformed lane result (non-string item) — never a pass")
@@ -129,10 +185,11 @@ def main():
blockers.append( blockers.append(
f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done" f"code-review: {reds} 🔴 CRITICAL finding(s) — fix before claiming done"
) )
suffix = attempts_suffix("code-review")
if m_fault and reds: if m_fault and reds:
record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane); {reds} 🔴") record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix}); {reds} 🔴")
elif m_fault: elif m_fault:
record("code-review", "BLOCKED", "PIPELINE-FAULT (degraded lane)") record("code-review", "BLOCKED", f"PIPELINE-FAULT (degraded lane{suffix})")
elif reds: elif reds:
record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)") record("code-review", "BLOCKED", f"{reds} 🔴 finding(s)")
elif m and m.group(1) == "MERGE-READY": elif m and m.group(1) == "MERGE-READY":
@@ -162,7 +219,17 @@ def main():
else: else:
reports.append(("adversary", adv)) reports.append(("adversary", adv))
m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv) m = re.search(r"ADVERSARIAL_REVIEW:\s*(CONFORMS|DIVERGES)", adv)
if m and m.group(1) == "CONFORMS": if adversary_degraded(adv):
blockers.append(
"adversary: PIPELINE-FAULT — degraded run, pipeline fault recorded "
"inside the adversary; a degraded lane is never a pass"
)
record(
"adversary",
"BLOCKED",
f"PIPELINE-FAULT (degraded lane{attempts_suffix('adversary')})",
)
elif m and m.group(1) == "CONFORMS":
record("adversary", "GREEN", "CONFORMS") record("adversary", "GREEN", "CONFORMS")
elif m: elif m:
blockers.append( blockers.append(
@@ -228,16 +295,69 @@ def main():
) )
record("probe", "BLOCKED", "missing verdict sentinel") record("probe", "BLOCKED", "missing verdict sentinel")
# retry_gate restores an exhausted lane's LAST report, which may be an
# earlier clean verdict — an incomplete lane must block regardless of
# what its row parsed to.
declared = state.get("review_incomplete")
incomplete = set()
malformed_bookkeeping = None
if isinstance(declared, list):
incomplete = {name for name in declared if isinstance(name, str)}
if any(not isinstance(name, str) for name in declared):
malformed_bookkeeping = repr(declared)[:80]
elif declared is not None:
malformed_bookkeeping = repr(declared)[:80]
for lane in sorted(incomplete - {"pipeline"}):
if rows.get(lane, ("", ""))[0] == "BLOCKED":
continue
after = f" after {n} attempt(s)" if (n := attempt_count(lane)) else ""
blockers.append(
f"{lane}: review incomplete{after} — the lane never produced a usable verdict"
)
record(lane, "BLOCKED", f"review incomplete{after}")
# A gauntlet-level fault (build_items crash, dead parse stage) means the # A gauntlet-level fault (build_items crash, dead parse stage) means the
# lanes above were never built — SKIPPED rows alone must not read as PASS. # lanes above were never built — SKIPPED rows alone must not read as PASS.
signals_error = state.get("signals_error") signals_error = state.get("signals_error")
if isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error: pipeline_fault = isinstance(signals_error, str) and "PIPELINE-FAULT:" in signals_error
if pipeline_fault:
blockers.append(f"pipeline: {signals_error}") blockers.append(f"pipeline: {signals_error}")
incomplete.add("pipeline")
if malformed_bookkeeping is not None:
blockers.append(
"pipeline: PIPELINE-FAULT — malformed review_incomplete bookkeeping "
f"({malformed_bookkeeping}); never a pass"
)
incomplete.add("pipeline")
if "pipeline" in incomplete and not any(b.startswith("pipeline:") for b in blockers):
blockers.append(
"pipeline: review incomplete — a pipeline stage never completed "
"(no fault text recorded)"
)
verdict = "BLOCKED" if blockers else "PASS" verdict = "BLOCKED" if blockers or incomplete else "PASS"
incomplete_lines = []
for name in sorted(incomplete):
if name == "pipeline":
incomplete_lines.append(
f"- pipeline: {signals_error}"
if pipeline_fault
else "- pipeline: review incomplete (no fault text recorded)"
)
elif n := attempt_count(name):
incomplete_lines.append(f"- {name}: no completed review after {n} attempt(s)")
else:
incomplete_lines.append(f"- {name}: no completed review")
incomplete_line = (
"\nGAUNTLET_REVIEW_INCOMPLETE: " + ", ".join(sorted(incomplete))
if incomplete
else ""
)
md = "\n".join( md = "\n".join(
["| lane | status | detail |", "|------|--------|--------|"] + lane_lines ["| lane | status | detail |", "|------|--------|--------|"]
+ [f"| {lane} | {status} | {detail} |" for lane, (status, detail) in rows.items()]
) )
md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)") md += "\n\n## Lane selection\n" + (state.get("lanes_summary") or "(none recorded)")
if blockers: if blockers:
@@ -246,11 +366,21 @@ def main():
md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention) md += "\n\n## Human attention required\n" + "\n".join(f"- {a}" for a in attention)
if state.get("signals_error"): if state.get("signals_error"):
md += f"\n\n> {state['signals_error']}" md += f"\n\n> {state['signals_error']}"
if incomplete_lines:
md += "\n\n## Review incomplete\n" + "\n".join(incomplete_lines)
md += "\n\n## Full lane reports\n" md += "\n\n## Full lane reports\n"
for lane, rep in reports: for lane, rep in reports:
md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n" md += f"\n<details><summary>{lane}</summary>\n\n{rep}\n\n</details>\n"
print(json.dumps({"gauntlet_verdict": verdict, "gauntlet_report": md})) print(
json.dumps(
{
"gauntlet_verdict": verdict,
"gauntlet_report": md,
"gauntlet_incomplete_line": incomplete_line,
}
)
)
try: try:
@@ -261,6 +391,7 @@ except Exception as e: # noqa: BLE001 — the gate must never crash into a sile
{ {
"gauntlet_verdict": "BLOCKED", "gauntlet_verdict": "BLOCKED",
"gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.", "gauntlet_report": f"verdict gate error: {e} — treat as a failed gate, not a pass.",
"gauntlet_incomplete_line": "\nGAUNTLET_REVIEW_INCOMPLETE: pipeline",
} }
) )
) )
+1 -1
View File
@@ -47,7 +47,7 @@ Spawnable sub-agents (from `config.yaml`):
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries. - **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run. - **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run. - **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
- **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed. - **[review-gauntlet](../review-gauntlet/README.md)** — preferred single entry point for the post-implementation review passes: a graph that computes diff signals deterministically, runs the applicable lanes (code-reviewer / adversary / security-reviewer / probe) in parallel, and machine-parses their verdict sentinels into GAUNTLET: PASS/BLOCKED — skipping a lane or misreading a verdict becomes structurally impossible. A `GAUNTLET_REVIEW_INCOMPLETE:` line under BLOCKED is an infrastructure fault (a lane exhausted its retries, or the pipeline faulted) that Sisyphus escalates to the user — retry / accept NEEDS-HUMAN / abort — never a finding to fix. The reviewer bullets above describe the lanes it runs and remain the spawn fallback when it is not installed.
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review. - **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
- **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion. - **[whetstone](../whetstone/README.md)** — continuous improvement; spawned ONCE at the end of a run that addressed review feedback from outside Sisyphus's own lanes (humans, or a teammate's independent review agent — never its own reviewers' output). Classifies each piece of feedback CATCH/PARTIAL/MISS against what the run's own reviewers flagged and converts the misses into user-approved suite improvements (skill edits, ledger entries). Never blocks run completion.
+21 -5
View File
@@ -329,11 +329,27 @@ instructions: |
parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED` parallel verdict-independent, and returns a machine-parsed `GAUNTLET: PASS` or `GAUNTLET: BLOCKED`
with a lane table, blockers, attention items, and every lane's full report. with a lane table, blockers, attention items, and every lane's full report.
- `GAUNTLET: BLOCKED` blocks completion. Route each blocker through the findings-handling rules in - `GAUNTLET: BLOCKED` blocks completion. Route each blocker that is a real review finding through
the reviewer sections below (🔴 → coder fix in the SAME session; DIVERGES → the adversary rules; the findings-handling rules in the reviewer sections below (🔴 → coder fix in the SAME session;
security FAIL → the security rules; probe INCONCLUSIVE → fix the ENVIRONMENT per the plan's DIVERGES → the adversary rules; security FAIL → the security rules; probe INCONCLUSIVE → fix the
local-run recipe first — it is never a code verdict). Then spawn a FRESH gauntlet run; lane ENVIRONMENT per the plan's local-run recipe first — it is never a code verdict). Then spawn a
selection re-runs from the new diff. FRESH gauntlet run; lane selection re-runs from the new diff. A BLOCKED whose only blockers are
review faults (`PIPELINE-FAULT` / `review incomplete` entries) is NOT a code failure — the next
rule owns it.
- `GAUNTLET_REVIEW_INCOMPLETE: <lane>[, <lane>]` (the line directly under `GAUNTLET: BLOCKED`)
means the review could NOT be completed: the named lanes still faulted after 3 attempts, or the
gauntlet pipeline itself faulted (`pipeline`). This is an infrastructure fault, not a code
finding. NEVER route it through the findings-handling rules. NEVER claim done while it stands.
Escalate to the user via `user__select` with exactly these three options:
- "Retry the review again"
- "Accept NEEDS-HUMAN: proceed without the <lane or pipeline> review (recorded in the report / PR body)"
- "Abort this task"
Retry → spawn a FRESH gauntlet run. Accept → finish the work and record the skipped review
(lane names or `pipeline`) in your final report and in the PR body when you author one. Abort →
stop; leave the code in a working state and report. Findings AND incomplete on the same output →
fix the findings first, then re-run the gauntlet (the re-run retries the lane) — escalate only if
the line persists. Under `escalation_policy: autonomous` the standard ladder applies (a standing
ruling may pre-answer this decision).
- `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those - `GAUNTLET: PASS` with a "Human attention required" section: finish the work and carry those
items into your final report verbatim. items into your final report verbatim.
- The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the - The reviewer sections below remain authoritative for HANDLING each lane's findings, and are the
+59 -63
View File
@@ -11,74 +11,70 @@
# - <agent-name>_AGENT_SESSION # - <agent-name>_AGENT_SESSION
# - <agent-name>_VARIABLES (as JSON array of key-value pairs; e.g. '[{"name": "username", "value": "alex"}]') # - <agent-name>_VARIABLES (as JSON array of key-value pairs; e.g. '[{"name": "username", "value": "alex"}]')
model: openai:gpt-4o # Specify the LLM to use model: openai:gpt-4o # Specify the LLM to use
temperature: null # Set default temperature parameter, range (0, 1) temperature: null # Set default temperature parameter, range (0, 1)
top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model top_p: null # Set default top-p parameter, with a range of (0, 1) or (0, 2) depending on the model
reasoning_effort: reasoning_effort: null # Reasoning effort level for models that support it (e.g. low, medium, high).
null # Reasoning effort level for models that support it (e.g. low, medium, high). # Only valid when the agent's model declares reasoning_levels.
# Only valid when the agent's model declares reasoning_levels. agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session
agent_session: null # Set a session to use when starting the agent. (e.g. temp, default); defaults to globally set agent_session name: <agent-name> # Name of the agent, used in the UI and logs
name: <agent-name> # Name of the agent, used in the UI and logs description: <description> # Description of the agent, used in the UI
description: <description> # Description of the agent, used in the UI version: 1 # Version of the agent
version: 1 # Version of the agent
# Auto-Continue (Todo System) # Auto-Continue (Todo System)
# The auto-continue system provides built-in task tracking for improved reliability. # The auto-continue system provides built-in task tracking for improved reliability.
# When enabled, the model can create todo lists and the system will automatically # When enabled, the model can create todo lists and the system will automatically
# prompt it to continue when incomplete tasks remain. # prompt it to continue when incomplete tasks remain.
# See the [Todo System documentation](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System) for more information # See the [Todo System documentation](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System) for more information
auto_continue: false # Enable automatic continuation when incomplete todos remain auto_continue: false # Enable automatic continuation when incomplete todos remain
max_auto_continues: 10 # Maximum number of automatic continuations before stopping max_auto_continues: 10 # Maximum number of automatic continuations before stopping
inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt inject_todo_instructions: true # Inject the default todo tool usage instructions into the agent's system prompt
continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null) continuation_prompt: null # Custom prompt used when auto-continuing (optional; uses default if null)
# Sub-Agent Spawning System # Sub-Agent Spawning System
# Enable this agent to spawn and manage child agents in parallel. # Enable this agent to spawn and manage child agents in parallel.
# See https://github.com/Dark-Alex-17/coyote/wiki/Agents for detailed documentation. # See https://github.com/Dark-Alex-17/coyote/wiki/Agents for detailed documentation.
can_spawn_agents: false # Enable the agent to spawn child agents can_spawn_agents: false # Enable the agent to spawn child agents
# spawnable_agents: # Optional whitelist restricting which agents can be spawned via `agent__spawn`. # spawnable_agents: # Optional whitelist restricting which agents can be spawned via `agent__spawn`.
# - explore # If omitted (the default), ALL installed agents are spawnable. This is the unrestricted default. # - explore # If omitted (the default), ALL installed agents are spawnable. This is the unrestricted default.
# - coder # Provide a list to restrict. Match is exact and case-sensitive (use directory names). # - coder # Provide a list to restrict. Match is exact and case-sensitive (use directory names).
# - oracle # An empty list ([]) means literally nothing spawnable. # - oracle # An empty list ([]) means literally nothing spawnable.
# Also filters `agent__list_available` output so the LLM only sees what it can spawn. # Also filters `agent__list_available` output so the LLM only sees what it can spawn.
# Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes. # Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes.
max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously
max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning) max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning)
max_concurrent_jobs: max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once for this agent
5 # Max background jobs (`job__*` tools) running at once for this agent # (overrides the global setting; 0 disables background jobs for this agent)
# (overrides the global setting; 0 disables background jobs for this agent) inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt
inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model
summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent
summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent escalation_timeout: 0 # 0 = wait indefinitely for the reply (default); N = give up after N seconds
escalation_timeout: 300 # Seconds a sub-agent waits for a user interaction response before timing out (default: 5 minutes) mcp_servers: # Optional list of MCP servers that the agent utilizes
mcp_servers: # Optional list of MCP servers that the agent utilizes - github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file
- github # Corresponds to the name of an MCP server in the `<coyote-config-dir>/mcp.json` file mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers
mcp_tools: # Optional per-server tool allowlist for the agent's MCP servers github: # (glob patterns: * and ?). Intersects with the global config,
github: # (glob patterns: * and ?). Intersects with the global config, - get_* # mcp.json `allowedTools`, and every other configured layer. It
- get_* # mcp.json `allowedTools`, and every other configured layer. It - search_* # can only narrow access, never widen it.
- search_* # can only narrow access, never widen it. global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
global_tools: # Optional list of additional global tools to enable for the agent; i.e. not tools specific to the agent
- web_search - web_search
- fs - fs
- python - python
skills_enabled: skills_enabled: true # Master switch for skills in this agent (default: inherit from global).
true # Master switch for skills in this agent (default: inherit from global). # Skills also require `function_calling_support: true` in the global config.
# Skills also require `function_calling_support: true` in the global config. enabled_skills: # Optional list of skills available when this agent runs.
enabled_skills: # Optional list of skills available when this agent runs. - git-master # Must be a subset of global `visible_skills`. Omit to inherit the global default.
# Must be a subset of global `visible_skills`. Omit to inherit the global default.
- git-master
- ai-slop-remover - ai-slop-remover
inject_skill_instructions: inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled
true # Inject a short hint pointing the model at `skill__list` when skills are enabled # (default: true). Suppressed automatically when no skills are available.
# (default: true). Suppressed automatically when no skills are available. skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null) enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL. - generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default. memory: null # Per-agent memory override (default: inherit). Set to `false` to disable memory
memory: # for this agent regardless of workspace/global presence. See the Memory wiki page.
null # Per-agent memory override (default: inherit). Set to `false` to disable memory
# for this agent regardless of workspace/global presence. See the Memory wiki page.
dynamic_instructions: false # Whether to use dynamic instructions for the agent; if false, static instructions are used dynamic_instructions: false # Whether to use dynamic instructions for the agent; if false, static instructions are used
instructions: instructions: # Static instructions for the agent; ignored if dynamic instructions are used
| # Static instructions for the agent; ignored if dynamic instructions are used |
You are a AI agent designed to demonstrate agent capabilities. You are a AI agent designed to demonstrate agent capabilities.
<tools> <tools>
@@ -99,10 +95,10 @@ instructions:
username: {{username}} username: {{username}}
</user> </user>
variables: # Optional variables for the agent variables: # Optional variables for the agent
# The variables defined above like {{__variable_name__}} are automatically available # The variables defined above like {{__variable_name__}} are automatically available
- name: username - name: username
description: Your user name description: Your user name
default: null # A default value for this variable; if null, the variable must be provided when starting the agent default: null # A default value for this variable; if null, the variable must be provided when starting the agent
conversation_starters: # Optional conversation starters for the agent conversation_starters: # Optional conversation starters for the agent
- What is the meaning of life? - What is the meaning of life?
- Tell me a joke. - Tell me a joke.
@@ -114,15 +110,15 @@ conversation_starters: # Optional conversation starters for the agent
- How do I stay motivated? - How do I stay motivated?
- What is the best way to exercise? - What is the best way to exercise?
- How do I manage my time effectively? - How do I manage my time effectively?
documents: # Optional documents to load for the agent documents: # Optional documents to load for the agent
# To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval), # To enable graph-based RAG (entity/relationship extraction + knowledge graph retrieval),
# set `rag_extractor_model` in your global config.yaml. # set `rag_extractor_model` in your global config.yaml.
# See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag # See https://github.com/Dark-Alex-17/coyote/wiki/RAG#graph-based-rag
- git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path - git:/some/repo # Explicitly tell Coyote to use the 'git' document loader using an absolute path
- pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path - pdf:some-pdf-file.pdf # Explicitly tell Coyote to use the 'pdf' document loader using a relative path
- https://some-website.com/some-page - https://some-website.com/some-page
- some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file - some-file.pdf # File with relative path to the <coyote-config-dir>/agents/<agent-name> directory; i.e. file in the same directory as this config file
- ~/some-file.txt # File in the user's home directory - ~/some-file.txt # File in the user's home directory
- /absolute/path/to/some-file.md # File with absolute path - /absolute/path/to/some-file.md # File with absolute path
- /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories - /absolute/path/**/NAME.txt # Find all NAME.txt files in the specified directory and all its subdirectories
- /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1) - /absolute/path/to/*/README.md # Find all README.md files in all immediate subdirectories of the specified directory (depth=1)
+2 -1
View File
@@ -889,7 +889,7 @@ fn default_summarization_threshold() -> usize {
} }
fn default_escalation_timeout() -> u64 { fn default_escalation_timeout() -> u64 {
300 0
} }
impl AgentConfig { impl AgentConfig {
@@ -1341,6 +1341,7 @@ variables:
assert_eq!(config.max_concurrent_agents, 4); assert_eq!(config.max_concurrent_agents, 4);
assert_eq!(config.max_agent_depth, 3); assert_eq!(config.max_agent_depth, 3);
assert_eq!(config.max_auto_continues, 10); assert_eq!(config.max_auto_continues, 10);
assert_eq!(config.escalation_timeout, 0);
assert!(config.mcp_servers.is_empty()); assert!(config.mcp_servers.is_empty());
assert!(config.global_tools.is_empty()); assert!(config.global_tools.is_empty());
assert!(config.conversation_starters.is_empty()); assert!(config.conversation_starters.is_empty());
+6
View File
@@ -1383,6 +1383,12 @@ clients:
assert_eq!(cfg.clients.len(), 1); assert_eq!(cfg.clients.len(), 1);
} }
#[test]
fn config_template_does_not_carry_the_per_agent_escalation_timeout_key() {
// `escalation_timeout` is a per-agent setting; the global template must never grow it.
assert!(!CONFIG_TEMPLATE.contains("escalation_timeout"));
}
#[test] #[test]
fn config_enabled_macros_empty_string_is_some_empty() { fn config_enabled_macros_empty_string_is_some_empty() {
let cfg: Config = serde_yaml::from_str("enabled_macros: \"\"").unwrap(); let cfg: Config = serde_yaml::from_str("enabled_macros: \"\"").unwrap();
+1 -1
View File
@@ -200,7 +200,7 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {"
1. Read the child's question and options. 1. Read the child's question and options.
2. If you can answer from context, call `agent__reply_escalation` with your answer. 2. If you can answer from context, call `agent__reply_escalation` with your answer.
3. If you need the user's input, call the appropriate `user__*` tool yourself, then relay the answer via `agent__reply_escalation`. 3. If you need the user's input, call the appropriate `user__*` tool yourself, then relay the answer via `agent__reply_escalation`.
4. **Respond promptly**; the child agent is blocked and waiting (5-minute timeout). 4. **Respond promptly**; the child blocks until answered — no timeout by default (`escalation_timeout` per agent).
"}; "};
pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {"
File diff suppressed because it is too large. Load diff
+5 -1
View File
@@ -1711,7 +1711,7 @@ fn handle_reply_escalation(ctx: &mut RequestContext, args: &Value) -> Result<Val
} }
None => Ok(json!({ None => Ok(json!({
"status": "error", "status": "error",
"message": format!("No pending escalation found with id '{escalation_id}'. It may have already been replied to or timed out."), "message": format!("No pending escalation found with id '{escalation_id}'. It may have already been replied to."),
})), })),
} }
} }
@@ -3093,6 +3093,10 @@ mod tests {
) )
.unwrap(); .unwrap();
assert_eq!(result["status"], "error"); assert_eq!(result["status"], "error");
assert_eq!(
result["message"],
"No pending escalation found with id 'missing'. It may have already been replied to."
);
} }
#[test] #[test]
+140 -11
View File
@@ -13,7 +13,7 @@ use tokio::sync::oneshot;
pub const USER_FUNCTION_PREFIX: &str = "user__"; pub const USER_FUNCTION_PREFIX: &str = "user__";
const DEFAULT_ESCALATION_TIMEOUT_SECS: u64 = 300; const DEFAULT_ESCALATION_TIMEOUT_SECS: u64 = 0;
const CUSTOM_MULTI_CHOICE_ANSWER_OPTION: &str = "Other (custom)"; const CUSTOM_MULTI_CHOICE_ANSWER_OPTION: &str = "Other (custom)";
pub fn user_interaction_function_declarations() -> Vec<FunctionDeclaration> { pub fn user_interaction_function_declarations() -> Vec<FunctionDeclaration> {
@@ -289,17 +289,32 @@ async fn handle_escalated(ctx: &RequestContext, action: &str, args: &Value) -> R
root_queue.submit(request); root_queue.submit(request);
let timeout = Duration::from_secs(timeout_secs); await_escalation_reply(rx, timeout_secs).await
match tokio::time::timeout(timeout, rx).await { }
Ok(Ok(reply)) => Ok(json!({ "answer": reply })),
Ok(Err(_)) => Ok(json!({ /// Waits for the parent's reply. `timeout_secs == 0` waits indefinitely; a
"error": "Escalation was cancelled. The parent agent dropped the request", /// dropped sender (the parent cancelled the request) resolves either way.
"fallback": "Make your best judgment and proceed", async fn await_escalation_reply(rx: oneshot::Receiver<String>, timeout_secs: u64) -> Result<Value> {
})), let reply = if timeout_secs == 0 {
rx.await
} else {
match tokio::time::timeout(Duration::from_secs(timeout_secs), rx).await {
Ok(reply) => reply,
Err(_) => {
return Ok(json!({
"error": format!(
"Escalation timed out after {timeout_secs} seconds waiting for user response"
),
"fallback": "Make your best judgment and proceed",
}));
}
}
};
match reply {
Ok(reply) => Ok(json!({ "answer": reply })),
Err(_) => Ok(json!({ Err(_) => Ok(json!({
"error": format!( "error": "Escalation was cancelled. The parent agent dropped the request",
"Escalation timed out after {timeout_secs} seconds waiting for user response"
),
"fallback": "Make your best judgment and proceed", "fallback": "Make your best judgment and proceed",
})), })),
} }
@@ -331,6 +346,8 @@ fn parse_options(args: &Value) -> Result<Vec<String>> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use std::fs;
use std::path::{Path, PathBuf};
#[test] #[test]
fn headless_select_returns_structured_json() { fn headless_select_returns_structured_json() {
@@ -351,4 +368,116 @@ mod tests {
assert_eq!(v["action"], "confirm"); assert_eq!(v["action"], "confirm");
assert_eq!(v["options"], json!([])); assert_eq!(v["options"], json!([]));
} }
#[test]
fn escalation_timeout_defaults_to_unlimited() {
assert_eq!(DEFAULT_ESCALATION_TIMEOUT_SECS, 0);
}
#[tokio::test(start_paused = true)]
async fn zero_timeout_waits_indefinitely_for_the_reply() {
let (tx, rx) = oneshot::channel();
let wait = tokio::spawn(await_escalation_reply(rx, 0));
tokio::time::advance(Duration::from_secs(3600)).await;
assert!(!wait.is_finished());
tx.send("approved".to_string()).unwrap();
let v = wait.await.unwrap().unwrap();
assert_eq!(v, json!({ "answer": "approved" }));
}
#[tokio::test(start_paused = true)]
async fn nonzero_timeout_still_expires_with_the_existing_message() {
let (tx, rx) = oneshot::channel::<String>();
let wait = tokio::spawn(await_escalation_reply(rx, 5));
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(4)).await;
assert!(!wait.is_finished());
tokio::time::advance(Duration::from_secs(2)).await;
let v = wait.await.unwrap().unwrap();
assert_eq!(
v["error"],
"Escalation timed out after 5 seconds waiting for user response"
);
assert_eq!(v["fallback"], "Make your best judgment and proceed");
assert!(tx.send("too late".to_string()).is_err());
}
#[tokio::test(start_paused = true)]
async fn zero_timeout_resolves_when_the_sender_is_dropped() {
let (tx, rx) = oneshot::channel::<String>();
drop(tx);
let v = tokio::time::timeout(Duration::from_secs(1), await_escalation_reply(rx, 0))
.await
.expect("the wait must complete once the sender is gone")
.unwrap();
assert!(
v["error"]
.as_str()
.unwrap()
.starts_with("Escalation was cancelled"),
"{v}"
);
assert_eq!(v["fallback"], "Make your best judgment and proceed");
}
#[test]
fn stale_timeout_wording_is_gone_from_shipped_text() {
// Split so this file does not match its own needles.
let needles = [
concat!("(5-minute", " timeout)"),
concat!("escalation_timeout:", " 300"),
concat!("default:", " 5 minutes"),
];
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
let example = root.join("config.agent.example.yaml");
let example_text = fs::read_to_string(&example).unwrap();
for needle in ["escalation_timeout: 0", "0 = wait indefinitely"] {
assert!(
example_text.contains(needle),
"{} lost the shipped wording {needle:?}",
example.display()
);
}
let mut files = vec![example];
collect_files(&root.join("src"), &mut files);
collect_files(&root.join("assets"), &mut files);
let this_file = root.join(file!());
assert!(
files.contains(&this_file),
"walk did not reach {}",
this_file.display()
);
let mut hits = Vec::new();
for path in &files {
let text = String::from_utf8_lossy(&fs::read(path).unwrap()).into_owned();
for needle in needles {
if text.contains(needle) {
hits.push(format!("{}: {needle:?}", path.display()));
}
}
}
assert!(
hits.is_empty(),
"stale timeout wording found:\n{}",
hits.join("\n")
);
}
fn collect_files(dir: &Path, out: &mut Vec<PathBuf>) {
for entry in fs::read_dir(dir).unwrap().flatten() {
let path = entry.path();
if path.is_dir() {
collect_files(&path, out);
} else if path.is_file() {
out.push(path);
}
}
}
} }
+111
View File
@@ -1325,6 +1325,117 @@ nodes:
); );
} }
/// A script `_next` back-edge into a node with a static fan-out must
/// re-fire every fan-out target on the next pass, and a map that
/// re-runs over zero items must overwrite its `collect_into` with `[]`
/// rather than leave the previous pass's results in place.
#[tokio::test]
async fn script_next_back_edge_refires_static_fan_out_and_empty_map_rerun_clears_collect_into()
{
if !cmd_available("python3") {
eprintln!("skipping: python3 not available");
return;
}
let ws = TestWorkspace::new();
ws.write_script(
"build.py",
r#"#!/usr/bin/env python3
import os, json
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
if state["pass"] == 0:
print(json.dumps({"a_items": [1], "b_items": [1]}))
else:
print(json.dumps({"a_items": [], "b_items": [2]}))
"#,
);
ws.write_script(
"worker.py",
r#"#!/usr/bin/env python3
import os, json
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
print(json.dumps({"output": f"report-{state['item']}"}))
"#,
);
ws.write_script(
"gate.py",
r#"#!/usr/bin/env python3
import os, json
state = json.loads(os.environ.get("GRAPH_STATE", "{}"))
if state["pass"] == 0:
print(json.dumps({"pass": 1, "_next": "build"}))
else:
print(json.dumps({}))
"#,
);
let yaml = r#"
name: fan_out_rerun_test
start: build
initial_state:
pass: 0
a_items: []
b_items: []
a_results: []
b_results: []
nodes:
build:
type: script
script: build.py
next: [map_a, map_b]
map_a:
type: map
over: "{{a_items}}"
as: item
branch: worker
collect_into: a_results
next: gate
map_b:
type: map
over: "{{b_items}}"
as: item
branch: worker
collect_into: b_results
next: gate
worker:
type: script
script: worker.py
state_updates: {}
timeout: 60
gate:
type: script
script: gate.py
next: done
done:
type: end
output: "{{a_results}}|{{b_results}}"
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
let mut ctx = make_ctx();
let abort = create_abort_signal();
let result = GraphExecutor::new(graph, &ws.dir)
.execute(&mut ctx, abort)
.await
.unwrap_or_else(|e| panic!("executor failed: {e:#}"));
let (a, b) = result
.split_once('|')
.unwrap_or_else(|| panic!("expected `a|b`, got: {result}"));
let a: Value =
serde_json::from_str(a).unwrap_or_else(|_| panic!("a_results not JSON: {result}"));
let b: Value =
serde_json::from_str(b).unwrap_or_else(|_| panic!("b_results not JSON: {result}"));
assert_eq!(
b,
serde_json::json!(["report-2"]),
"the back-edge into build must re-fire both fan-out targets: {result}"
);
assert_eq!(
a,
serde_json::json!([]),
"map_a must re-fire on the back-edge pass and clear its collect_into over zero items: {result}"
);
}
#[tokio::test] #[tokio::test]
async fn parallel_branch_error_aborts_super_step() { async fn parallel_branch_error_aborts_super_step() {
if !cmd_available("bash") { if !cmd_available("bash") {
+9 -1
View File
@@ -2281,6 +2281,7 @@ where
mod tests { mod tests {
use super::*; use super::*;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
struct TempDir { struct TempDir {
@@ -2289,10 +2290,17 @@ mod tests {
impl TempDir { impl TempDir {
fn new(tag: &str) -> Self { fn new(tag: &str) -> Self {
let unique = SystemTime::now() // Wall-clock nanos alone are not unique across parallel tests
// (coarse clocks hand the same tick to concurrent callers, so two
// tests would share a directory and overwrite each other's files);
// a process-wide counter makes every name distinct.
static SEQ: AtomicU64 = AtomicU64::new(0);
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH) .duration_since(UNIX_EPOCH)
.unwrap() .unwrap()
.as_nanos(); .as_nanos();
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
let unique = format!("{nanos}-{seq}");
let path = env::temp_dir().join(format!("coyote-rag-{tag}-{unique}")); let path = env::temp_dir().join(format!("coyote-rag-{tag}-{unique}"));
fs::create_dir_all(&path).unwrap(); fs::create_dir_all(&path).unwrap();
Self { path } Self { path }