feat(TASK-020): finding-verifier + step-runner fail-closed hardening
finding-verifier: parse gains max_attempts 2 + parse_failure capture +
fallback to a new parse_fault marker that rewrites verdicts to a single
UNVERIFIABLE PIPELINE-FAULT entry and continues to done (sentinel
preserved — the caller's verify lane degrades instead of dying);
verify_one gains a branch-local verify_fault marker normalizing the
engine failure text into a schema-conformant UNVERIFIABLE verdict.
step-runner: implement gains fallback end_failure (no retry — reruns
against a mutated tree are unsafe; STEP_FAILED renders the failure via
coder_result); independent_review gains fallback write_handoff with the
handoff instructions and route_review treating an "Agent node failed:"
review_report as review-did-not-run (fail-visible, guard ahead of the
🔴 grep); orient/edge_case_sweep/write_handoff capture their failure
text via state_updates, shrinking the bundled fallback-capture warning
baseline back to zero.
R5 tests: fault-marker normalization + crash paths (raw non-JSON
GRAPH_STATE), route_review.sh fault/critical/clean routing via bash+jq,
and structural wiring tests pinning fallbacks, sentinels, and
end_failure's initial_state-backed template keys.
This commit is contained in:
6 files changed
+585
-24
No files matched your search
@@ -32,6 +32,7 @@ initial_state:
|
||||
verdicts: []
|
||||
gate_feedback: ''
|
||||
gate_attempts: 0
|
||||
parse_failure: ''
|
||||
|
||||
start: parse
|
||||
|
||||
@@ -92,8 +93,20 @@ nodes:
|
||||
type: string
|
||||
description: Repository directory named in the request (echo the prompt default when none).
|
||||
required: [findings_list, project_dir]
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
parse_failure: '{{output}}'
|
||||
fallback: parse_fault
|
||||
next: verify_each
|
||||
|
||||
parse_fault:
|
||||
id: parse_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead parse stage (reached only via parse's fallback) — rewrites `verdicts` to a single UNVERIFIABLE PIPELINE-FAULT entry and jumps straight to done, so the FINDING_VERIFIER_RESULTS sentinel still renders and the caller's verify lane degrades instead of dying.
|
||||
script: scripts/parse_fault.py
|
||||
timeout: 30
|
||||
next: done
|
||||
|
||||
verify_each:
|
||||
id: verify_each
|
||||
type: map
|
||||
@@ -165,6 +178,7 @@ nodes:
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
verdict: '{{output}}'
|
||||
fallback: verify_fault
|
||||
next: verdict_gate
|
||||
|
||||
verdict_gate:
|
||||
@@ -174,6 +188,13 @@ nodes:
|
||||
script: scripts/verdict_gate.py
|
||||
timeout: 30
|
||||
|
||||
verify_fault:
|
||||
id: verify_fault
|
||||
type: script
|
||||
description: Fail-closed marker for a dead verify branch (reached only via verify_one's fallback; branch-local). Normalizes the engine's failure text in `verdict` into a schema-conformant UNVERIFIABLE verdict the map collects — a dead verifier lane never sinks the whole map or masquerades as verified.
|
||||
script: scripts/verify_fault.py
|
||||
timeout: 30
|
||||
|
||||
done:
|
||||
id: done
|
||||
type: end
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead parse stage of the finding-verifier graph.
|
||||
|
||||
Reached ONLY via parse's `fallback:` route. parse's state_updates wrote an
|
||||
"LLM node …failed: <chain>" string into `parse_failure` before routing here;
|
||||
this node rewrites `verdicts` to a single schema-conformant UNVERIFIABLE
|
||||
entry naming the fault and the graph jumps straight to `done` — the
|
||||
FINDING_VERIFIER_RESULTS sentinel still renders, so the caller (e.g.
|
||||
code-reviewer's verify lane) degrades instead of dying. Findings that were
|
||||
never extracted cannot be verified; the caller's "UNVERIFIABLE = keep the
|
||||
finding, marked unverified" semantics do the right thing with the entry.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def fault_verdicts(detail):
|
||||
return {
|
||||
"verdicts": [
|
||||
{
|
||||
"id": "pipeline-fault",
|
||||
"verdict": "UNVERIFIABLE",
|
||||
"evidence": "",
|
||||
"note": (
|
||||
"PIPELINE-FAULT: parse failed — findings could not be extracted "
|
||||
f"from the verification request; no finding was verified: {detail}"
|
||||
),
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
raw = state.get("parse_failure")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the parse stage died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps(fault_verdicts(detail)))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(json.dumps(fault_verdicts(f"fault-marker script error: {e}")))
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed marker for a dead verify branch of the finding-verifier graph.
|
||||
|
||||
Reached ONLY via verify_one's `fallback:` route inside the map branch.
|
||||
Grounded in engine behavior: on llm-node failure the engine writes
|
||||
"LLM node failed: <chain>" into `verdict` via the node's state_updates
|
||||
BEFORE routing to the fallback, so the raw failure text is already in
|
||||
state. This node normalizes it into a schema-conformant UNVERIFIABLE
|
||||
verdict (authoritative id stamped from the finding item, mirroring
|
||||
verdict_gate.py) and the chain ends there — the map collects a valid
|
||||
verdict, so a dead verifier lane never sinks the whole map or masquerades
|
||||
as verified.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
MAX_DETAIL_CHARS = 500
|
||||
|
||||
|
||||
def load_state():
|
||||
if path := os.environ.get("GRAPH_STATE_FILE"):
|
||||
with open(path) as f:
|
||||
return json.load(f)
|
||||
return json.loads(os.environ.get("GRAPH_STATE", "{}"))
|
||||
|
||||
|
||||
def finding_id(state):
|
||||
finding = state.get("finding")
|
||||
if isinstance(finding, dict):
|
||||
fid = finding.get("id")
|
||||
if isinstance(fid, str) and fid.strip():
|
||||
return fid.strip()
|
||||
return "unknown"
|
||||
|
||||
|
||||
def fault_verdict(fid, detail):
|
||||
return {
|
||||
"verdict": {
|
||||
"id": fid,
|
||||
"verdict": "UNVERIFIABLE",
|
||||
"evidence": "",
|
||||
"note": f"PIPELINE-FAULT: verifier lane failed after retries — {detail}",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
state = load_state()
|
||||
fid = finding_id(state)
|
||||
raw = state.get("verdict")
|
||||
detail = raw.strip().replace("\n", " ") if isinstance(raw, str) else ""
|
||||
if not detail.startswith("LLM node"):
|
||||
detail = "the verifier lane died without recording a failure detail"
|
||||
if len(detail) > MAX_DETAIL_CHARS:
|
||||
detail = detail[: MAX_DETAIL_CHARS - 1] + "…"
|
||||
print(json.dumps(fault_verdict(fid, detail)))
|
||||
|
||||
|
||||
try:
|
||||
main()
|
||||
except Exception as e: # noqa: BLE001 — the fault marker itself must never crash
|
||||
print(json.dumps(fault_verdict("unknown", f"fault-marker script error: {e}")))
|
||||
@@ -64,6 +64,7 @@ initial_state:
|
||||
plan_summary: ''
|
||||
implementation_brief: ''
|
||||
staleness_report: ''
|
||||
orient_failure: ''
|
||||
has_major_deviation: false
|
||||
deviation_summary: ''
|
||||
user_feedback: ''
|
||||
@@ -80,12 +81,14 @@ initial_state:
|
||||
tests_output: ''
|
||||
edge_case_report: ''
|
||||
downstream_updates: ''
|
||||
sweep_failure: ''
|
||||
needs_independent_review: false
|
||||
review_report: ''
|
||||
review_attempts: 0
|
||||
max_review_attempts: 1
|
||||
handoff_attempts: 0
|
||||
handoff_fix: ''
|
||||
handoff_failure: ''
|
||||
step_summary: ''
|
||||
|
||||
start: resolve_step
|
||||
@@ -191,6 +194,8 @@ nodes:
|
||||
has_major_deviation,
|
||||
deviation_summary,
|
||||
]
|
||||
state_updates:
|
||||
orient_failure: '{{output}}'
|
||||
fallback: end_failure
|
||||
next: route_staleness
|
||||
|
||||
@@ -270,6 +275,12 @@ nodes:
|
||||
timeout: 3600
|
||||
state_updates:
|
||||
coder_result: '{{output}}'
|
||||
# No max_attempts: rerunning a coder that died mid-edit against an
|
||||
# already-mutated tree is unsafe. On failure the "Agent node failed:"
|
||||
# text lands in coder_result (via state_updates above) and end_failure
|
||||
# renders it; end_failure's other keys all have initial_state defaults,
|
||||
# so the STEP_FAILED output degrades to empty build/tests sections.
|
||||
fallback: end_failure
|
||||
next: route_coder_result
|
||||
|
||||
route_coder_result:
|
||||
@@ -376,6 +387,8 @@ nodes:
|
||||
needs_independent_review:
|
||||
type: boolean
|
||||
required: [edge_case_report, downstream_updates, needs_independent_review]
|
||||
state_updates:
|
||||
sweep_failure: '{{output}}'
|
||||
fallback: write_handoff
|
||||
next: route_sweep
|
||||
|
||||
@@ -408,6 +421,7 @@ nodes:
|
||||
max_attempts: 2
|
||||
state_updates:
|
||||
review_report: '{{output}}'
|
||||
fallback: write_handoff
|
||||
next: route_review
|
||||
|
||||
route_review:
|
||||
@@ -451,6 +465,15 @@ nodes:
|
||||
|
||||
If "Gate feedback" below is non-empty, a previous handoff attempt
|
||||
failed validation - fix exactly what it lists.
|
||||
|
||||
PIPELINE-FAULT VISIBILITY: an "Independent review report" below that
|
||||
starts with "Agent node failed:" means the independent review DID NOT
|
||||
RUN - the graph routed here through the reviewer's failure fallback.
|
||||
Never present that text as review findings. Flag it prominently
|
||||
instead: add a Deviations entry "PIPELINE-FAULT: independent review
|
||||
did not run - <failure text>", repeat it under Notes for next step,
|
||||
and carry it into `step_summary` so the user sees it at the approval
|
||||
gate.
|
||||
prompt: |
|
||||
## Step
|
||||
{{step_number}} ({{step_title}}) - plan at {{step_plan_path}}
|
||||
@@ -502,6 +525,8 @@ nodes:
|
||||
type: string
|
||||
description: 3-6 sentence summary of the step for the user's approval decision - what was done, deviations, anything needing their attention
|
||||
required: [step_summary]
|
||||
state_updates:
|
||||
handoff_failure: '{{output}}'
|
||||
fallback: end_failure
|
||||
next: check_handoff
|
||||
|
||||
|
||||
@@ -13,6 +13,16 @@ review_report=$(echo "$state" | jq -r '.review_report // ""')
|
||||
review_attempts=$(echo "$state" | jq -r '.review_attempts // 0')
|
||||
max_review_attempts=$(echo "$state" | jq -r '.max_review_attempts // 1')
|
||||
|
||||
# Fail-visible guard: a review_report holding the engine's "Agent node
|
||||
# failed:" text means the independent review DID NOT RUN (its fallback
|
||||
# normally routes straight to write_handoff, bypassing this script). Never
|
||||
# treat that text as findings or spend a fix-loop attempt on it -
|
||||
# write_handoff flags the fault prominently in the handoff instead.
|
||||
if [[ "$review_report" == "Agent node failed:"* ]]; then
|
||||
jq -nc '{"_next": "write_handoff"}'
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! grep -qF "🔴" <<< "$review_report"; then
|
||||
jq -nc '{"_next": "write_handoff"}'
|
||||
exit 0
|
||||
|
||||
Reference in new issue
Block a user