test(suite): replace plan-label references with self-contained wording
Rust test comments/assert messages and deep-research script docstrings cited numbered requirement labels (R3/R5) and task labels (TASK-016, TASK-021) that are defined only in an external plan document. Reword each to say what it means on its own: fail-safe crash guards, the validator's fallback-capture warning, task-queue anchors.
This commit is contained in:
8 files changed
+24
-23
No files matched your search
@@ -263,7 +263,7 @@ pass — every fault is visible to the reviewer or the caller — and some
|
|||||||
end node always renders its sentinel, so the graph degrades instead of
|
end node always renders its sentinel, so the graph degrades instead of
|
||||||
dying.
|
dying.
|
||||||
|
|
||||||
All 11 scripts also carry a top-level crash guard (R3): a script that
|
All 11 scripts also carry a top-level crash guard: a script that
|
||||||
throws records a fault and fails forward where the flow allows it (a
|
throws records a fault and fails forward where the flow allows it (a
|
||||||
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
|
crashed `reflexion_gate` proceeds to `synthesize`; a crashed
|
||||||
`verify_sources` reports that sources were NOT checked) instead of
|
`verify_sources` reports that sources were NOT checked) instead of
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ the RAG knowledge lookup) as a single super-step. The validator requires
|
|||||||
declared parallel-branch script outputs, so we emit an empty JSON object
|
declared parallel-branch script outputs, so we emit an empty JSON object
|
||||||
explicitly here.
|
explicitly here.
|
||||||
|
|
||||||
Fail-safe (R3): the node reads no state and computes nothing, so its
|
Fail-safe: the node reads no state and computes nothing, so its
|
||||||
happy-path output IS the sane degraded output — the guard emits the same
|
happy-path output IS the sane degraded output — the guard emits the same
|
||||||
empty object on any crash. No PIPELINE-FAULT is recorded: appending one
|
empty object on any crash. No PIPELINE-FAULT is recorded: appending one
|
||||||
would require reading state (the very thing that crashed) or clobbering
|
would require reading state (the very thing that crashed) or clobbering
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ surface their collected output_key). This script lifts such findings into
|
|||||||
`pipeline_faults` (deduplicated, since a reflexion or feedback loop can
|
`pipeline_faults` (deduplicated, since a reflexion or feedback loop can
|
||||||
re-run the map) so dead lanes surface in the report's Pipeline notes.
|
re-run the map) so dead lanes surface in the report's Pipeline notes.
|
||||||
|
|
||||||
Fail-safe (R3): on a crash (e.g. malformed GRAPH_STATE), `findings`
|
Fail-safe: on a crash (e.g. malformed GRAPH_STATE), `findings`
|
||||||
becomes the fault text itself and a PIPELINE-FAULT entry is recorded —
|
becomes the fault text itself and a PIPELINE-FAULT entry is recorded —
|
||||||
the pipeline proceeds visibly degraded, and the human approval gate is
|
the pipeline proceeds visibly degraded, and the human approval gate is
|
||||||
the backstop. Existing faults are preserved best-effort.
|
the backstop. Existing faults are preserved best-effort.
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ budget.
|
|||||||
|
|
||||||
Routing (`_next`): always research_each_question.
|
Routing (`_next`): always research_each_question.
|
||||||
|
|
||||||
Fail-safe (R3): a crash here means the user's feedback text cannot be
|
Fail-safe: a crash here means the user's feedback text cannot be
|
||||||
recovered, but their INTENT — another research pass — is unambiguous
|
recovered, but their INTENT — another research pass — is unambiguous
|
||||||
(they chose neither accept nor reject). The degraded route therefore
|
(they chose neither accept nor reject). The degraded route therefore
|
||||||
still loops to `research_each_question`, with a generic feedback note
|
still loops to `research_each_question`, with a generic feedback note
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Routing (`_next`):
|
|||||||
- prompt present -> (no _next; static next: plan)
|
- prompt present -> (no _next; static next: plan)
|
||||||
- prompt empty -> ask_topic
|
- prompt empty -> ask_topic
|
||||||
|
|
||||||
Fail-safe (R3): a crash here (e.g. malformed GRAPH_STATE) means the
|
Fail-safe: a crash here (e.g. malformed GRAPH_STATE) means the
|
||||||
caller's prompt cannot be read at all, so proceeding silently is
|
caller's prompt cannot be read at all, so proceeding silently is
|
||||||
meaningless — the degraded route is `ask_topic`, which asks the user for
|
meaningless — the degraded route is `ask_topic`, which asks the user for
|
||||||
the topic directly, plus a PIPELINE-FAULT entry so the loss is visible in
|
the topic directly, plus a PIPELINE-FAULT entry so the loss is visible in
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Reflexion is a best-effort quality booster, not a hard gate: once the
|
|||||||
budget is spent the workflow proceeds anyway, and the human approval
|
budget is spent the workflow proceeds anyway, and the human approval
|
||||||
step is the final backstop.
|
step is the final backstop.
|
||||||
|
|
||||||
Fail-safe (R3): a crash here fails FORWARD to `synthesize` — the same
|
Fail-safe: a crash here fails FORWARD to `synthesize` — the same
|
||||||
direction as the existing malformed-critique PASS default — with a
|
direction as the existing malformed-critique PASS default — with a
|
||||||
PIPELINE-FAULT entry recorded (best-effort preserving existing faults),
|
PIPELINE-FAULT entry recorded (best-effort preserving existing faults),
|
||||||
so a broken gate costs the run its automated critique routing, never the
|
so a broken gate costs the run its automated critique routing, never the
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ rendered section when `pipeline_faults` is non-empty and the empty
|
|||||||
string otherwise, keeping the happy-path `{{report}}{{pipeline_notes}}`
|
string otherwise, keeping the happy-path `{{report}}{{pipeline_notes}}`
|
||||||
output byte-identical to the pre-hardening `{{report}}`.
|
output byte-identical to the pre-hardening `{{report}}`.
|
||||||
|
|
||||||
Fail-safe (R3): per-URL probes were already guarded; the top level now
|
Fail-safe: per-URL probes were already guarded; the top level now
|
||||||
is too. On a crash the summary says sources were NOT checked, a
|
is too. On a crash the summary says sources were NOT checked, a
|
||||||
PIPELINE-FAULT is recorded (best-effort preserving existing faults), and
|
PIPELINE-FAULT is recorded (best-effort preserving existing faults), and
|
||||||
pipeline_notes still renders — the approval gate remains the backstop.
|
pipeline_notes still renders — the approval gate remains the backstop.
|
||||||
|
|||||||
@@ -8181,7 +8181,7 @@ mod tests {
|
|||||||
] {
|
] {
|
||||||
assert!(
|
assert!(
|
||||||
architect.contains(anchor),
|
architect.contains(anchor),
|
||||||
"architect config lost TASK-021 anchor: {anchor:?}"
|
"architect config lost task-queue mirroring anchor: {anchor:?}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -8199,7 +8199,7 @@ mod tests {
|
|||||||
] {
|
] {
|
||||||
assert!(
|
assert!(
|
||||||
sisyphus.contains(anchor),
|
sisyphus.contains(anchor),
|
||||||
"sisyphus config lost TASK-021 anchor: {anchor:?}"
|
"sisyphus config lost task-queue chain anchor: {anchor:?}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -8214,7 +8214,7 @@ mod tests {
|
|||||||
] {
|
] {
|
||||||
assert!(
|
assert!(
|
||||||
spawn.contains(anchor),
|
spawn.contains(anchor),
|
||||||
"DEFAULT_SPAWN_INSTRUCTIONS lost TASK-021 anchor: {anchor:?}"
|
"DEFAULT_SPAWN_INSTRUCTIONS lost task-queue semantics anchor: {anchor:?}"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
assert!(
|
assert!(
|
||||||
@@ -8492,7 +8492,7 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- adversary suite-script regression tests (R5) ----
|
// ---- adversary suite-script regression tests ----
|
||||||
//
|
//
|
||||||
// Fault paths of the adversary's verdict/gate scripts, exercised by
|
// Fault paths of the adversary's verdict/gate scripts, exercised by
|
||||||
// invoking `python3 <script>` with a synthetic GRAPH_STATE env — the same
|
// invoking `python3 <script>` with a synthetic GRAPH_STATE env — the same
|
||||||
@@ -9427,7 +9427,7 @@ mod tests {
|
|||||||
assert_eq!(holistic.max_iterations, 20);
|
assert_eq!(holistic.max_iterations, 20);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- review-gauntlet suite-script regression tests (R5) ----
|
// ---- review-gauntlet suite-script regression tests ----
|
||||||
//
|
//
|
||||||
// Degradation paths of the gauntlet's lane/builder/gate scripts,
|
// Degradation paths of the gauntlet's lane/builder/gate scripts,
|
||||||
// exercised the same way as the adversary suite above: `python3 <script>`
|
// exercised the same way as the adversary suite above: `python3 <script>`
|
||||||
@@ -9914,7 +9914,7 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- code-reviewer suite-script regression tests (R5) ----
|
// ---- code-reviewer suite-script regression tests ----
|
||||||
//
|
//
|
||||||
// Fail-closed fault paths of the code-reviewer's verdict/fault/render
|
// Fail-closed fault paths of the code-reviewer's verdict/fault/render
|
||||||
// scripts, exercised the same way as the suites above: `python3 <script>`
|
// scripts, exercised the same way as the suites above: `python3 <script>`
|
||||||
@@ -10503,7 +10503,7 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- finding-verifier suite-script regression tests (R5) ----
|
// ---- finding-verifier suite-script regression tests ----
|
||||||
//
|
//
|
||||||
// Fail-closed fault paths of the finding-verifier's marker scripts,
|
// Fail-closed fault paths of the finding-verifier's marker scripts,
|
||||||
// exercised the same way as the suites above: `python3 <script>` with a
|
// exercised the same way as the suites above: `python3 <script>` with a
|
||||||
@@ -10738,7 +10738,7 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- step-runner fault-wiring regression tests (R5) ----
|
// ---- step-runner fault-wiring regression tests ----
|
||||||
//
|
//
|
||||||
// route_review.sh is a bash script node; the graph's script executor runs
|
// route_review.sh is a bash script node; the graph's script executor runs
|
||||||
// `.sh` scripts through bash with the same GRAPH_STATE env contract, so
|
// `.sh` scripts through bash with the same GRAPH_STATE env contract, so
|
||||||
@@ -10899,7 +10899,7 @@ mod tests {
|
|||||||
if skip_step_runner_bash_harness() {
|
if skip_step_runner_bash_harness() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// R3: a fault-noting script must never itself kill the pipeline —
|
// A fault-noting script must never itself kill the pipeline —
|
||||||
// the harness asserts exit 0 + JSON stdout even for a bare state.
|
// the harness asserts exit 0 + JSON stdout even for a bare state.
|
||||||
let out = run_step_runner_script("note_llm_fault.sh", &json!({}));
|
let out = run_step_runner_script("note_llm_fault.sh", &json!({}));
|
||||||
assert_eq!(out, json!({"fault_note": ""}));
|
assert_eq!(out, json!({"fault_note": ""}));
|
||||||
@@ -10949,8 +10949,9 @@ mod tests {
|
|||||||
assert_eq!(review.fallback.as_deref(), Some("write_handoff"));
|
assert_eq!(review.fallback.as_deref(), Some("write_handoff"));
|
||||||
|
|
||||||
// The llm nodes with fallbacks capture their failure text so the
|
// The llm nodes with fallbacks capture their failure text so the
|
||||||
// fallback path can see why it was reached (the TASK-016 warning —
|
// fallback path can see why it was reached (the validator's
|
||||||
// this is what shrank the step-runner warning baseline to zero).
|
// fallback-capture warning — this is what shrank the step-runner
|
||||||
|
// warning baseline to zero).
|
||||||
// orient and write_handoff route their fallbacks through
|
// orient and write_handoff route their fallbacks through
|
||||||
// note_llm_fault, which distills the capture into fault_note;
|
// note_llm_fault, which distills the capture into fault_note;
|
||||||
// edge_case_sweep still falls back to write_handoff, which renders
|
// edge_case_sweep still falls back to write_handoff, which renders
|
||||||
@@ -11037,9 +11038,9 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- deep-research suite-script regression tests (R5) ----
|
// ---- deep-research suite-script regression tests ----
|
||||||
//
|
//
|
||||||
// The deep-research graph's R3 guards and fault markers, exercised by
|
// The deep-research graph's crash guards and fault markers, exercised by
|
||||||
// invoking `python3 <script>` with a synthetic GRAPH_STATE env. The raw
|
// invoking `python3 <script>` with a synthetic GRAPH_STATE env. The raw
|
||||||
// runner feeds deliberately malformed JSON to prove every guarded script
|
// runner feeds deliberately malformed JSON to prove every guarded script
|
||||||
// emits a sane degraded output instead of crashing the node.
|
// emits a sane degraded output instead of crashing the node.
|
||||||
@@ -11077,7 +11078,7 @@ mod tests {
|
|||||||
eprintln!("skipping: python3 not available");
|
eprintln!("skipping: python3 not available");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// R3: every guarded script and fault marker must exit 0 with JSON
|
// Fail-safe: every guarded script and fault marker must exit 0 with JSON
|
||||||
// stdout even when GRAPH_STATE is not JSON at all (the runner
|
// stdout even when GRAPH_STATE is not JSON at all (the runner
|
||||||
// asserts both); per-script degraded shapes are pinned below.
|
// asserts both); per-script degraded shapes are pinned below.
|
||||||
for script in [
|
for script in [
|
||||||
@@ -11516,8 +11517,8 @@ mod tests {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
// plan: fail-closed. Its output_schema + fallback combination requires
|
// plan: fail-closed. Its output_schema + fallback combination requires
|
||||||
// a state_updates capture (the TASK-016 warning) so plan_fault can see
|
// a state_updates capture (the validator's fallback-capture warning) so
|
||||||
// why it was reached.
|
// plan_fault can see why it was reached.
|
||||||
let NodeType::Llm(plan) = &graph.get_node("plan").unwrap().node_type else {
|
let NodeType::Llm(plan) = &graph.get_node("plan").unwrap().node_type else {
|
||||||
panic!("plan must be an llm node")
|
panic!("plan must be an llm node")
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in new issue
Block a user